JPH09323280A - Manipulator control method and system - Google Patents

Abstract

(57) [Abstract] [PROBLEMS] Control method of manipulator for enabling operator to appropriately operate and position a tip effector with respect to a remote target object without relying on visual observation or experience, and control for implementing the same. Provide the system. When manipulating the position / orientation of a manipulator with respect to a target object O, a tip effector 2 equipped with a 3D marker 3 in advance is installed at the tip of an arm 1, and a stereo camera 4 is prepared and placed. 3D marker 3
And the target object O are photographed by the stereo camera 4 at the same time, and the controller 5 uses the feature points of the target object O and the 3D marker 3
Of the manipulator so that the reference coordinate system Σf arbitrarily selected on the tip effector 2 matches the reference coordinate system ΣB defined by the feature points on the target object O. Positioning control of the arm 1 is performed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to control of a manipulator which enables positioning control when a tip effector such as a robot arm is operated to a desired relative position / posture with respect to a remote object using a stereo camera. It relates to a method and a control system for implementing this.

[0002]

2. Description of the Related Art Conventionally, in order to remotely operate a large crane or a ladder of a fire engine (manipulator) to appropriately position a tip (tip effector), it has been largely dependent on experience based on visual observation by an operator. Therefore, a high degree of skill is required to freely operate the manipulator, which is a difficult task for beginners. Therefore, in order to facilitate the positioning of the manipulator arm, a servo control system that simultaneously captures the target object and the tip effector of the arm with a camera, tracks these characteristic points, and matches them on the image plane. The control system of the manipulator which utilized it is proposed.

[0003]

However, in the control system of the manipulator, it is necessary to recognize both the feature points of the target object and the tip effector from the image taken by the camera, so that the processing takes a lot of time. However, the current situation is that it has not been put to practical use.

The main objects to be solved by the present invention are as follows. A first object of the present invention is to provide a control method for a manipulator which enables an operator to appropriately operate a tip effector with respect to a remote target object without relying on visual observation or experience to perform positioning, and control for implementing the same. It is intended to provide a system.

A second object of the present invention is to easily grasp the position / orientation of the tip effector three-dimensionally with respect to a reference coordinate system defined on the target object, and to perform relative translation / rotational movement. It is intended to provide a manipulator control method capable of efficiently performing a highly flexible positioning operation and a control system for implementing the same.

Other objects of the present invention include the specification, drawings,
Especially, it will be apparent from the description of each claim.

[0007]

According to the present invention, when an operator operates a tip effector at a desired relative position / orientation, the position / orientation of the tip effector is image-recognized with respect to a target object captured by a stereo camera. The feature is that feedback control for error correction is performed by stereoscopically capturing with a 3D marker that is easy to perform. As a result, the operator can appropriately operate the tip effector with respect to the remote target object without relying on visual observation or experience to perform positioning. More specifically, in order to solve the problem, the above object is achieved by adopting each novel characteristic construction method and means listed in the following by the present invention.

That is, the first feature of the method of the present invention is that when the position / orientation of the manipulator with respect to the target object is manipulated, the tip effector at the end of the arm is adjusted to the proper position / orientation in relation to the target object. When performing positioning control, in order to facilitate photographing of the position / orientation of the relative relationship and analysis of projective geometry, etc., the tip effector is obtained through imaging of the position / orientation of the 3D marker integrally attached to the tip effector and analysis of projective geometry, etc. Is a method of controlling a manipulator that indirectly calculates the position and orientation of the.

A second feature of the method of the present invention is the control of a manipulator in which a stereo camera captures the relative relationship between the target object and the tip effector in the first feature of the method of the present invention. On the way.

A third feature of the method of the present invention is that the projective geometric analysis of the relative relationship between the target object and the tip effector in the first or second feature of the method of the present invention is performed by the image of the stereo camera. In the control method of the manipulator, the characteristic points of the target object and the coordinate position of the reference pattern of the 3D marker are numerically indexed and calculated through a predetermined relational expression.

A fourth feature of the method of the present invention is that, when operating the position / orientation of the manipulator with respect to a target object, a tip effector equipped with a 3D marker in advance is installed at the tip of the arm of the manipulator, and a stereo camera is used. Is prepared and placed, the 3D marker and the target object are simultaneously photographed by the stereo camera, the feature points of the target object and the position of the reference pattern of the 3D marker on the screen are tracked, and the manipulator's The control method of the manipulator controls the positioning of the arm of the manipulator such that a reference coordinate system arbitrarily selected on the tip effector matches a reference coordinate system defined from the feature points on the target object.

A fifth feature of the method of the present invention is that, when operating the position / orientation of the manipulator with respect to the target object, a tip effector equipped with a 3D marker in advance is installed at the tip of the arm of the manipulator, and the 3D marker is used. A stereo camera that simultaneously captures the target object and the target object is prepared and placed. First, a reference coordinate system is defined from a plurality of feature points of the target object on the image plane of the stereo camera, and on the tip effector. The position / orientation of the reference coordinate system is designated, and then the feature points of the target object and the positions of the reference patterns of the 3D markers on the screen are tracked to refer to the 3D markers on the left and right screens by the stereo camera. After obtaining the position of the pattern in the camera coordinate system, while calculating the relative position / orientation of the 3D marker coordinate system with respect to the camera coordinate system, The relative position of the serial 3D marker coordinate system,
The projection of the reference coordinate system on the tip effector is calculated based on the posture, the calculated projection of the reference coordinate system is squeezed on both left and right screens of the stereo camera, and then the reference on the tip effector is calculated. The projection image of the coordinate system is mapped to a 6-dimensional arm feature amount vector, while the projection image of the reference coordinate system of the target object is mapped to a 6-dimensional object feature amount vector to generate an arm feature amount vector and an object feature amount. This is a control method of a manipulator that controls the manipulator so that the two vectors match after obtaining the vector.

A sixth feature of the method of the present invention is that the 3D marker in the first, second, third, fourth or fifth feature of the method of the present invention is on the same plane of an oblique hexahedron or a rectangular parallelepiped. In the control method of the manipulator, reference patterns are arranged at four or more vertices not found in.

The seventh feature of the method of the present invention is that the 3D in the third, fourth, fifth or sixth feature of the method of the present invention.
The position of the reference pattern of the marker on the screen is calculated by storing the image of the reference pattern previously attached to the 3D marker as a comparison image and correlating it with the image captured by the stereo camera. Control method.

The eighth feature of the method of the present invention is that the manipulator in the first, second, third, fourth, fifth, sixth or seventh feature of the method of the present invention is an arm of an industrial robot. , A control method for manipulators such as cranes or ladders of fire trucks.

A ninth feature of the method of the present invention is that the reference pattern of the 3D marker in the third, fourth, fifth, sixth, seventh or eighth feature of the method of the present invention is patterned. And a light source that brightens the pattern plate.

The first feature of the device of the present invention is, in a manipulator control system in which a tip effector is provided at the tip of an arm, a 3D marker equipped on the tip effector, a target object and the 3D marker are simultaneously provided. A stereo camera for photographing, tracking the characteristic points of the target object and the position on the screen of the reference pattern of the 3D marker when operating the manipulator, and a reference coordinate system arbitrarily selected on the tip effector of the manipulator is used. A controller for controlling the manipulator so as to match a reference coordinate system defined by the characteristic points on the target object, and a control system for the manipulator.

The second feature of the device of the present invention is, in a manipulator control system in which a tip effector is provided at the tip of an arm, a 3D marker equipped on the tip effector, a target object and the 3D marker are simultaneously provided. A stereo camera for photographing and a controller are provided, and the controller reads the image information from the stereo camera at the time of operating the manipulator to determine the feature points of the target object and the position of the reference pattern of the 3D marker on the image surface. An image tracker for tracking, a 3D marker position / orientation estimator for calculating the relative position / orientation of the 3D marker coordinate system with respect to the camera coordinate system based on the tracking information from the image tracker, and a 3D marker coordinate system The projection of the reference coordinate system on the tip effector is calculated based on the relative position and orientation, and the projection of the reference coordinate system is calculated. A reference coordinate drawing device which is scratched on both left and right screens of the stereo camera, and a feature amount vector having a one-to-one correspondence between the projection of the reference coordinate system on the tip effector and the position / orientation of the tip effector with respect to the camera coordinate system. And an arm feature amount calculation unit for converting the feature point of the target object output from the image tracker into a target feature amount vector having a one-to-one correspondence with the position / orientation of the target object with respect to camera coordinates. A manipulator including an object feature amount calculation unit, and a feedback controller that calculates a velocity / angular velocity command of a tip portion of the manipulator based on a difference between outputs from the arm feature amount calculation unit and the object feature amount calculation unit. Control system.

A third feature of the device of the present invention is that the image of the stereo camera is used as an input instead of the image tracker and the 3D marker position / orientation estimator of the controller in the second feature of the device of the present invention. An image tracker that tracks the position of the reference pattern of the 3D marker on the image plane while correlating it with a comparison pattern, and a 3D marker for the camera coordinate system based on the output from the image tracker. A control system for a manipulator having a 3D marker position / orientation estimator for estimating the relative position / orientation of a coordinate system.

A fourth feature of the device of the present invention is that the 3D marker in the first, second or third feature of the device of the present invention is not on the same plane of an oblique hexahedron or a rectangular parallelepiped.
This is a control system of a manipulator in which reference patterns are arranged at one or more vertices.

A fifth feature of the device of the present invention is that the manipulator in the first, second, third or fourth feature of the device of the present invention is an arm of an industrial robot, a crane or a ladder of a fire engine. It is in the control system of the manipulator.

A sixth feature of the device of the present invention is a pattern plate in which the reference pattern of the 3D marker in the first, second, third, fourth or fifth feature of the device of the present invention is drawn. And a light source that brightens the pattern board.

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below based on apparatus examples, method examples, and application examples. (Device Example) FIG. 1 shows a configuration of a control system α of the manipulator of this device example, and FIG. 2 is a 3D used in this device example.
The structure of a marker is shown. Note that the target object O described below
Of the target object O in the three-dimensional space (3
Dimension) and direction (3D).

The manipulator used in this device example is
A tip effector 2 is provided at the tip of the arm 1. As shown in FIG. 1, the control system α of the manipulator of this device example includes a 3D marker 3, a stereo camera 4, and a controller 5. The manipulator may be a crane, a ladder of a fire engine, or the like, as well as an arm of an industrial robot, and the present invention can be applied to a system equipped with an arm for remote control.

The 3D marker 3 is mounted on the tip effector 2. As shown in FIG. 2, the configuration is such that a total of four reference patterns 7 to be tracked by the stereo camera 4 are provided at the tips and origins of the three arms 6a, 6b, 6c having lengths Lm orthogonal to each other. . That is, the reference pattern 7 has four vertices (origin, x axis, y) that are not on the same plane of the cube.
Axis, z-axis).

The 3D marker 3 may have a structure in which the reference pattern 7 is arranged at the apex of a rectangular parallelepiped or an oblique hexahedron. However, in order to obtain the position / orientation of the 3D marker 3 with respect to the camera coordinate system, 4 is used. More than one reference pattern 7
Need. The reference pattern 7 of the 3D marker 3 of this device example is a checkerboard pattern plate 7a and a pattern plate 7a.
And a light emitting diode 7b for illuminating.

The stereo camera 4 photographs the target object O and the 3D marker 3 at the same time. The controller 5 includes an image tracker 8, a 3D marker position / orientation estimator 9, a reference coordinate drawing device 10, an arm feature amount calculator 11, an object feature amount calculator 12, and a feedback controller 13.

The image tracker 8 reads the image information from the stereo camera 4 when the manipulator is operated, and tracks the characteristic points of the target object O on the left and right screens and the position of the reference pattern 7 of the 3D marker 3 on the image plane. To do. The 3D marker position / orientation estimator 9 estimates the relative position / orientation of the 3D marker coordinate system with respect to the camera coordinate system based on the tracking information from the image tracker 8.

The reference coordinate drawing unit 10 has three preset coordinates.
Relative position / orientation information of D marker coordinate system, image tracker 8
Output information of the 3D marker position / orientation estimator 9 to obtain the projection of the reference coordinate system on the tip effector 2, and the projection of the reference coordinate system of the image obtained from the stereo camera 4. Scratch on both left and right screens. The arm feature quantity calculator 11 converts the projection of the reference coordinate system on the tip effector 2 into a feature quantity vector having a one-to-one correspondence with the position / orientation of the tip effector 2 with respect to the camera coordinate system.

The object feature quantity calculator 12 is an image tracker 8
The feature points of the target object O output from the target object O are converted into a target feature amount vector having a one-to-one correspondence with the position / orientation of the target object O with respect to the camera coordinates. Feedback controller 1
3 is an arm feature amount calculation unit 11 and an object feature amount calculation unit 1
The velocity / angular velocity command of the tip portion of the arm 1 of the manipulator is calculated based on the difference between the outputs from 2.

According to the manipulator control system α of the present apparatus example described above, as shown in FIG. 4, a stereo image obtained by simultaneously capturing the target object O and the 3D marker 3 is used to display on the tip effector 2. An arbitrarily selected reference coordinate system Σf
Arm 1 of the manipulator so that is aligned with the reference coordinate system ΣB defined by the feature points f1, f2, f3 on the target object O.
Control. Here, FIG. 4 shows the 3D marker 3 and the target object O captured by the stereo camera 4.

(Example of Method) Next, an example of the method of controlling the manipulator using the above-mentioned apparatus example will be specifically described. (1) First, the left and right image planes Im photographed by the stereo camera 4 when operating the manipulator are displayed on the monitor screen, and the operator selects feature points of the target object O while referring to the monitor screen. Then, the image plane Im
From the plurality of feature points of the target object O on the reference coordinate system Σ
B is defined, and the position of the reference coordinate system Σf on the tip effector 2
The posture is specified, and the position / posture information of the reference coordinate system Σf is input to the reference coordinate drawing device 10.

(2) Next, the image tracker 8 tracks the feature points of the target object O and the position of the reference pattern 7 of the 3D marker 3 on the screen. At this time, for example, 3 shown in FIG.
The image of the reference pattern 7 attached to the D marker 3 is stored in memory as a comparison image, and the position on the image surface is calculated by correlating with the input image. It should be noted that two orthogonal diameter lines included in the reference pattern 7 may be detected and the position on the surface image plane may be calculated as the intersection thereof.

(3) Subsequently, the 3D marker position / orientation estimator 9 calculates (estimates) the relative position / orientation of the 3D marker coordinate system with respect to the camera coordinate system based on the output information of the image tracker 8. 3D marker position / orientation estimator 9
The processing performed in is as follows.

FIG. 5 shows the projection of the 3D marker 3 by the stereo camera on one image plane Im.

[Outside 1]

[0036]

[Outside 2]

[0037]

[Outside 3]

[0038]

[Outside 4]

[0039]

[Outside 5]

[0040]

[Outside 6]

[0041]

[Outside 7]

[0042]

[Outside 8]

[0043]

[Equation 1]

[0044]

[Equation 2]

[0045]

(Equation 3)

Here,

(Equation 4)

[0047]

[Outside 9]

(4) Next, the reference coordinate drawing unit 10 performs 3D
The projection of the reference coordinate system on the tip effector 2 is calculated based on the relative position / orientation information of the marker coordinate system, the output information of the image tracker 8, and the output information of the 3D marker position / orientation estimator 9, and The projection of the reference coordinate system is scratched on the left and right screens of the image obtained from the stereo camera 4.

[0049]

[Outside 10]

Here, the left and right camera coordinate systems are re-established by Σl.
, Σr. By replacing Σc with Σl and Σr, the projection of the reference coordinate system Σf on the left and right image planes Im can be obtained. The origin of the reference coordinate system Σf synthesized on each image plane Im is defined as p _Oi , x, y, and z-axis end points as p _Xi , p _Yi , and p _Zi (i = 1, r), respectively.

(5) Subsequently, the arm feature amount calculation unit 11 has a one-to-one correspondence between the projection of the reference coordinate system Σf on the tip effector 2 and the position / orientation of the tip effector 2 with respect to the camera coordinate system. A target object having a one-to-one correspondence with the position / orientation of the target object O with respect to the camera coordinates while converting the feature point of the target object O output from the image tracker 8 by the target object feature amount calculation unit 12 into a quantity vector. Convert to feature vector. Then, the feedback controller 13 calculates the velocity / angular velocity command of the tip of the manipulator based on the difference between the outputs from the arm feature amount calculation unit 11 and the object feature amount calculation unit 12.

For simplification of explanation, it is assumed that the tip effector 2 of the arm 1 of the manipulator can be driven so as to realize arbitrary translational velocity and rotational angular velocity with respect to the reference coordinate system ΣB. Therefore, the visual feedback rule to be designed here is to generate a translational velocity and rotational angular velocity command such that the reference coordinate ΣB of the target object O and the reference coordinate Σf of the arm 1 match on the image plane. Where p _Oi (i
Considering the first derivative of = 1, r), it is given as follows.

[0053]

[Outside 11]

[0054]

[Outside 12]

Also,

(Equation 5)

The first derivative of p _Yi and p _Zi can be similarly obtained.

[Outside 13]

Positioning is performed so that the reference coordinate system Σf of the tip effector 2 coincides with the reference coordinate system ΣB defined on the target object O on the image plane, and the origin of the reference coordinate system Σf and x, y, The feature point of the target object O that can directly correspond to the end point of the z axis does not always exist. Therefore,

[Outside 14]

This mapping is a 6-dimensional vector (feature vector) corresponding to the one-to-one correspondence between the projection of a certain coordinate system in the three-dimensional space on the stereo image plane Im and the position / orientation of that coordinate system with respect to the camera coordinate system. It is to convert. Therefore, if this mapping is performed on the projections of the reference coordinate system Σf on the tip effector 2 and the reference coordinate system ΣB on the target object O, positioning is realized by matching the feature amount vectors of both.

Here, the vector relating to the reference coordinate Σf of the tip effector 2 is an arm feature vector.

[Outside 15]

[0060]

[Outside 16]

[0061]

[Outside 17]

At this time, the entire system including the mapping can be described as follows.

(Equation 6)

Here,

(Equation 7)

At this time, the following visual feedback control law (feedback controller 13) can be constructed by using the inverse matrix of the image Jacobian J.

[Outside 18]

[0065]

[Outside 19]

(Application Example) Next, an application example of this embodiment will be described. In the present application example, as shown in FIG. 7, the image tracker 8 correlates the image of the reference pattern of the 3D marker 3 stored as the comparison image Im ′ with the input image to determine the image surface of the reference pattern 7. The position above shall be calculated. In this case, the projection 7 ′ of the reference pattern 7 on the image plane Im initially matches the comparison image Im ′, but changes according to the movement of the arm 1 of the manipulator.

On the other hand, if the 3D marker position / orientation estimator 9 described above is configured, the position / orientation of the marker coordinate system with respect to the camera coordinate system can be calculated, and thus the comparison image Im ′ is used for the 3D marker at that time. It is possible to sequentially obtain the projections produced by the reference pattern 7 on the image plane Im at the positions and orientations of 3 and use this as the comparison pattern. Alternatively, the reference pattern 7 is obtained by correlating with the comparison pattern.
After obtaining the position of, the image near the position may be sequentially updated as a new image for comparison.

When it becomes difficult to observe the four reference patterns 7 of the 3D marker 3 with the stereo camera, the direction of the 3D marker 3 or the reference pattern (for example, as shown in FIG. 7) may be changed. However, since the position / orientation of the reference coordinate system Σf of the tip effector 2 is not changed, when drawing the reference coordinate Σf on the image surface, the position / orientation information of the reference coordinate system Σf with respect to the marker coordinate system is changed. calculate.

Although the representative apparatus example, method example and application example of the present embodiment have been described above, the present embodiment is not necessarily limited to the means of the apparatus example and the method of the method example. The present embodiment achieves that purpose,
The present invention can be appropriately modified and implemented within a range having the effects described below.

[0070]

As described above, according to the present invention, a desired relative position / position with respect to a target object captured by a stereo camera can be obtained.
When operating the tip effector in the posture, the position / posture of the tip effector is stereoscopically grasped by a 3D marker that is easy to image-recognize, and the error is corrected by feedback control. Therefore, it is defined on the target object. The tip effector can be translated and rotated relative to the reference coordinate system, and moreover, highly flexible positioning work can be efficiently performed. Therefore, the operator can appropriately position the arm of the manipulator without relying on visual observation or experience.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a control system of a manipulator of an apparatus example of the present invention.

FIG. 2 is an explanatory diagram showing a configuration of a 3D marker used in the device example of the same.

FIG. 3 is a schematic diagram showing a configuration of a reference pattern of a 3D marker.

FIG. 4 is an explanatory diagram showing a 3D marker and a target object observed by a stereo camera.

FIG. 5 is an explanatory diagram showing a projection of a 3D marker on one image plane by a stereo camera.

FIG. 6 is an explanatory diagram showing a process of relative position estimation between a stereo camera and a 3D marker.

FIG. 7 is an explanatory diagram showing how the projection of the reference pattern changes depending on the relative positional relationship between the arm of the manipulator and the stereo camera.

[Explanation of symbols]

 α ... Manipulator control system 1 ... Arm 2 ... Tip effector 3 ... 3D marker 4 ... Stereo camera 5 ... Controller 6a, 6b, 6c ... Arm 7 ... Reference pattern 7a ... Pattern plate 7b ... Light emitting diode 7 '... Reference pattern Projection 8 ... Image tracker 9 ... 3D marker position / orientation estimator 10 ... Reference coordinate drawing device 11 ... Arm feature amount calculator 12 ... Object feature amount calculator 13 ... Feedback controller Im ... Image plane Im '... For comparison Image Σf… Reference coordinate system ΣB… Standard coordinate system F… Focus O… Target object R… Optical axis

Claims (15)

[Claims] 1. A position of a manipulator with respect to a target object
When manipulating the posture, when performing positioning control of the tip effector at the tip of the arm to the proper position / posture in relation to the target object, it is easy to capture the position / posture of the relative relation and analyze the projective geometry, etc. In order to do so, the position / orientation of the tip effector is indirectly calculated through imaging of the position / orientation of the 3D marker integrally attached to the tip effector and analysis of projective geometry and the like. 2. The manipulator control method according to claim 1, wherein the photographing of the relative relationship between the target object and the tip effector is performed by a stereo camera. 3. The projective geometric analysis of the relative relationship between the target object and the tip effector numerically determines the coordinate points of the feature points of the target object in the image of the stereo camera and the reference pattern of the 3D marker. The manipulator control method according to claim 1 or 2, wherein indexing is performed and calculation processing is performed through a predetermined relational expression. 4. The position of the manipulator with respect to the target object
When manipulating the posture, a tip effector equipped with a 3D marker in advance is installed at the arm tip of the manipulator, and a stereo camera is prepared and placed, and the 3D marker and the target object are simultaneously displayed by the stereo camera. After capturing the image, the characteristic points of the target object and the 3D
Tracking the position of the reference pattern of the marker on the screen, the reference coordinate system arbitrarily selected on the tip effector of the manipulator so as to match the reference coordinate system defined from the feature points on the target object. A method for controlling a manipulator, comprising controlling the positioning of an arm of the manipulator. 5. The position of the manipulator with respect to the target object
When manipulating the posture, a tip effector equipped with a 3D marker in advance is installed on the arm tip of the manipulator, and a stereo camera for simultaneously photographing the 3D marker and the target object is prepared and placed. A standard coordinate system is defined from a plurality of feature points of the target object on the image plane of the stereo camera, and a position / orientation of a reference coordinate system on the tip effector is specified. The position of the reference pattern of the 3D marker on the screen is tracked, the positions of the reference patterns of the 3D markers on the left and right screens of the stereo camera are obtained in the camera coordinate system, and then the 3D marker with respect to the camera coordinate system. While calculating the relative position and orientation of the coordinate system, reference coordinates on the tip effector based on the relative position and orientation of the 3D marker coordinate system The projection of the system is calculated, the calculated projection of the reference coordinate system is squeezed onto the left and right screens of the stereo camera, and then the projection image of the reference coordinate system on the tip effector is displayed on the screen 6.
While mapping to the three-dimensional arm feature amount vector, the projection image of the reference coordinate system of the target object is mapped to the six-dimensional target feature amount vector to obtain the arm feature amount vector and the target feature amount vector. A method for controlling a manipulator, comprising controlling the manipulator so that the vectors match. 6. The 3D marker has reference patterns arranged at four or more apexes that are not on the same plane of an orthorhombic hexahedron or a rectangular parallelepiped. Alternatively, the manipulator control method according to the fifth aspect. 7. The position of the reference pattern of the 3D marker on the screen is stored as an image for comparison in advance of the image of the reference pattern attached to the 3D marker, and is correlated with the image taken by the stereo camera. It calculates by taking. The control method of the manipulator according to claim 3, 4, 5, or 6. 8. The manipulator is an arm of an industrial robot, a crane, or a ladder of a fire engine, wherein the manipulator is an arm of a robot or a ladder of a fire engine.
A method for controlling the manipulator described in. 9. The reference pattern of the 3D marker comprises a pattern plate on which a pattern is drawn, and a light source for making the pattern plate brighter. Alternatively, the method of controlling the manipulator according to Item 8. 10. A manipulator control system in which a tip effector is provided at the tip of an arm, wherein a 3D marker mounted on the tip effector, a stereo camera for simultaneously photographing a target object and the 3D marker, and a manipulator During operation, the feature points of the target object and the positions of the reference patterns of the 3D markers on the screen are tracked, and a reference coordinate system arbitrarily selected on the tip effector of the manipulator is selected from the feature points on the target object. A controller for controlling the manipulator so as to match a defined reference coordinate system, and a control system for the manipulator. 11. A control system for a manipulator in which a tip effector is provided at the tip of an arm, wherein a 3D marker equipped on the tip effector, a stereo camera for simultaneously photographing a target object and the 3D marker, and a controller. The controller reads the image information from the stereo camera when operating the manipulator, and the characteristic points of the target object and the 3
An image tracker that tracks the position of the reference pattern of the D marker on the image plane, and a relative position of the 3D marker coordinate system with respect to the camera coordinate system based on the tracking information from the image tracker.
A 3D marker position / orientation estimator for calculating the attitude, and a projection of the reference coordinate system on the tip effector is calculated based on the relative position / orientation of the 3D marker coordinate system, and the projection of the reference coordinate system is performed by the stereo. Converting a reference coordinate drawing device to be scratched on both left and right screens of the camera and a projection of the reference coordinate system on the tip effector into a feature amount vector having a one-to-one correspondence with the position / orientation of the tip effector with respect to the camera coordinate system An arm feature amount calculation unit for converting the feature points of the target object output from the image tracker into a target feature amount vector having a one-to-one correspondence with the position / orientation of the target object with respect to the camera coordinates. A characteristic amount calculation unit, and a feedback controller that calculates a velocity / angular velocity command of the tip portion of the manipulator based on the difference between the outputs from the arm characteristic amount calculation unit and the object characteristic amount calculation unit. Control system comprising a controller, and wherein the manipulator. 12. An image tracker and 3D of the controller.
Instead of a marker position / orientation estimator, an image tracker that inputs the image of the stereo camera and tracks the position of the reference pattern of the 3D marker on the image plane while correlating it with a comparison pattern prepared in advance. And a 3D marker position / orientation estimator that estimates the relative position / orientation of the 3D marker coordinate system with respect to the camera coordinate system based on the output from the image tracker. Manipulator control system. 13. The 3D marker has reference patterns arranged at four or more apexes that are not on the same plane of an orthorhombic hexahedron or a rectangular parallelepiped, and the 3D marker is characterized in that: Manipulator control system. 14. The manipulator control system according to claim 10, 11, 12 or 13, wherein the manipulator is an arm of an industrial robot, a crane, or a ladder of a fire engine. 15. The reference pattern of the 3D marker comprises: a pattern plate on which a pattern is drawn; and a light source for making the pattern plate brighter, 10, 11, 12, 13 or 1.
4. The control system for the manipulator described in 4.