WO2024181702A1 - Dispositif électronique comportant un dispositif de préhension permettant de saisir un objet devant être saisi et procédé de commande associé - Google Patents

Dispositif électronique comportant un dispositif de préhension permettant de saisir un objet devant être saisi et procédé de commande associé Download PDF

Info

Publication number
WO2024181702A1
WO2024181702A1 PCT/KR2024/001660 KR2024001660W WO2024181702A1 WO 2024181702 A1 WO2024181702 A1 WO 2024181702A1 KR 2024001660 W KR2024001660 W KR 2024001660W WO 2024181702 A1 WO2024181702 A1 WO 2024181702A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target object
gripper
neural network
network model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2024/001660
Other languages
English (en)
Korean (ko)
Inventor
윤재민
안준모
정락준
하창수
한흥우
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR1020230094078A external-priority patent/KR20240133506A/ko
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of WO2024181702A1 publication Critical patent/WO2024181702A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
    • B25J19/02—Sensing devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/02—Program-controlled manipulators characterised by movement of the arms, e.g. cartesian coordinate type
    • B25J9/04—Program-controlled manipulators characterised by movement of the arms, e.g. cartesian coordinate type by rotating at least one arm, excluding the head movement itself, e.g. cylindrical coordinate type or polar coordinate type
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the present invention relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device including a gripper and a method for controlling the same.
  • robots for various purposes are being deployed in factories and homes, and these robots can perform actions such as grasping external objects and moving them from one location to another or transporting them.
  • industrial robots deployed in factories can pick up objects that are difficult for humans to pick up and transport them to specific locations, or perform assembly.
  • robots deployed in homes can perform indoor cleaning, such as transporting foreign substances, or move objects that are difficult for humans to pick up or dangerous objects to specific locations.
  • the electronic device includes a first sensor, a second sensor, a gripper, and one or more processors configured to input a first image acquired through the first sensor into a first neural network model to acquire a first feature map corresponding to the first image, input a second image acquired through the second sensor into a second neural network model to acquire a second feature map corresponding to the second image, merge the first feature map and the second feature map to acquire a third image, identify a grip target object based on the third image, and control the gripper to grasp the grip target object.
  • a method for controlling an electronic device including a gripper for achieving the above-described purpose includes the steps of: inputting a first image acquired through a first sensor into a first neural network model to acquire a first feature map corresponding to the first image; inputting a second image acquired through a second sensor into a second neural network model to acquire a second feature map corresponding to the second image; acquiring a third image by merging the first feature map and the second feature map; identifying a grip target object based on the third image; and controlling the gripper to grasp the grip target object.
  • a computer-readable recording medium including a program for executing a control method for an electronic device including a gripper
  • the control method for the electronic device includes a step of inputting a first image acquired through a first sensor into a first neural network model to acquire a first feature map corresponding to the first image, a step of inputting a second image acquired through a second sensor into a second neural network model to acquire a second feature map corresponding to the second image, a step of merging the first feature map and the second feature map to acquire a third image, a step of identifying a grip target object based on the third image, and a step of controlling the gripper to grasp the grip target object.
  • FIG. 1 is a drawing schematically illustrating a gripper and a grip target object according to one embodiment of the present disclosure.
  • FIG. 2 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure.
  • FIG. 3 is a diagram for explaining a first neural network model and a second neural network model according to one embodiment of the present disclosure.
  • FIG. 4 is a diagram for explaining a third neural network model and movement information according to one embodiment of the present disclosure.
  • FIG. 5 is a diagram for explaining reliability according to one embodiment of the present disclosure.
  • FIG. 6 is a drawing for explaining the result of an electronic device moving a grip target object according to an embodiment of the present disclosure.
  • FIG. 7 is a flowchart for explaining a method for controlling an electronic device according to an embodiment of the present disclosure.
  • first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another.
  • a “module” or “part” performs at least one function or operation, and may be implemented by hardware or software, or by a combination of hardware and software.
  • a plurality of “modules” or a plurality of “parts” may be integrated into at least one module and implemented by at least one processor (not shown), except for a “module” or “part” that needs to be implemented by a specific hardware.
  • FIG. 1 is a drawing schematically illustrating a gripper and a grip target object according to one embodiment of the present disclosure.
  • an electronic device (100) may be implemented as various types of devices, such as a user terminal device, a display device, a set-top box, a tablet personal computer (PC), a smart phone, an e-book reader, a desktop PC, a laptop PC, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a kiosk, and the like.
  • a user terminal device such as a user terminal device, a display device, a set-top box, a tablet personal computer (PC), a smart phone, an e-book reader, a desktop PC, a laptop PC, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a kiosk, and the like.
  • PDA personal digital assistant
  • PMP portable multimedia player
  • MP3 player an MP3 player
  • the electronic device (100) may be implemented as various types of electronic devices, such as a wearable device in the form of at least one of an accessory type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, a contact lens, or a head-mounted device (HMD)), a fabric or clothing-integrated type (e.g., an electronic garment), a robot including a driving unit, a projector, a server, etc.
  • an accessory type e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, a contact lens, or a head-mounted device (HMD)
  • a fabric or clothing-integrated type e.g., an electronic garment
  • a robot including a driving unit, a projector, a server, etc.
  • An electronic device (100) may be implemented as a robot.
  • a robot may mean various types of machines that have the ability to perform work functions on their own.
  • a robot may mean a smart machine that, in addition to simple repetitive functions, detects the surrounding environment in real time based on sensors, cameras, etc., collects information, and operates autonomously.
  • An electronic device (100) may include a gripper capable of grasping or moving an external object, like a human finger.
  • the gripper may also be called a robot hand, a grasper, etc., which is an end portion of a multi-joint robot, but for convenience of explanation, it will be collectively called a gripper hereinafter.
  • the electronic device (100) may be classified into industrial, medical, household, military, and exploration robots depending on the field of use or the functions that can be performed.
  • the industrial robot may be subdivided into a robot used in the product manufacturing process in a factory, a robot that performs customer service, order reception, and serving in a store or restaurant, etc.
  • the electronic device (100) according to an embodiment of the present disclosure may be implemented as a robot that can grip an external object by controlling a gripper provided in the electronic device (100) in various places such as a factory, a hotel, a mart, a hospital, and a clothing store, and then transport the external object to a specific location.
  • the robot may be classified in various ways depending on the field of use, function, and purpose of use, and is of course not limited to the above-described examples.
  • Conventional robots calculate the distance between an external object and a gripper as a specific numerical value based on a depth image or RGB image containing the external object, move the gripper based on the calculated distance, and then grip the external object.
  • conventional robots could position a gripper adjacent to an external object only when the external object was recognized, and could control the gripper to grip the external object.
  • This gripping method has a disadvantage in that it can smoothly grip an external object only when the robot recognizes the external object with relatively high accuracy (or reliability). For example, if the robot calculates the distance between the external object and the gripper relatively accurately within the error range, it can grip the external object only when it properly identifies the grip pose according to the shape of the external object. If the calculated distance exceeds the error range or the grip pose is not appropriate for the shape of the external object, there is a problem in that it cannot grip the external object or it has to grip the external object after several trials and errors.
  • trial and error means that if the robot fails to grasp the external object, it recalculates the distance between the external object and the gripper or re-identifies the grip pose and then moves the gripper again to grip the external object.
  • a grip target object an external object that the electronic device (100) grips through the gripper is collectively referred to as a grip target object.
  • an electronic device (100) can recognize a grip target object by appropriately using an RGB image and a depth image.
  • the electronic device (100) identifies the grip target object by using both the first image that detects (or captures) a first characteristic of the surrounding environment including the grip target object and the second image that detects a second characteristic of the surrounding environment, so that the electronic device (100) can identify the grip target object with higher accuracy (or reliability) than a conventional robot.
  • the electronic device (100) can identify the distance between the grip target object and the gripper, the grip pose for gripping the grip target object, etc. with high accuracy.
  • FIG. 2 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure.
  • the electronic device (100) includes a first sensor (110), a second sensor (120), a gripper (130), and one or more processors (140).
  • the first sensor (110) is configured to capture a still image or a moving image.
  • the first sensor (110) may include an RGB camera sensor.
  • the RGB camera sensor may collect visible light (400 to 700 nm), convert it into an electric signal, and transmit it to one or more processors (140).
  • the one or more processors (140) may obtain a first image including a first characteristic of the surrounding environment based on the electric signal received from the RGB camera sensor.
  • the first characteristic may include an RGB characteristic.
  • the second sensor (120) is configured to obtain information about the distance.
  • the second sensor (120) may include a depth camera sensor.
  • the second sensor (120) is a stereo camera, and based on the same principle that when a person's two eyes about 6.5 cm apart look at the same object, different images are formed in the left and right eyes, the stereo vision camera can obtain a left-eye image and a right-eye image that are differently captured of the same grip target object.
  • one or more processors (140) can obtain a second image including a second characteristic of at least one object (e.g., a grip target object) included in the surrounding environment based on the disparity between the left-eye image and the right-eye image.
  • the second characteristic can include a depth characteristic.
  • the second sensor (120) may include a ToF (Time of Flight) camera sensor.
  • the ToF camera sensor may be a sensor that measures the distance between the ToF camera sensor and the grip target object by irradiating a signal (e.g., near-infrared, ultrasonic, laser, etc.) and measuring the time until the irradiated signal is reflected by the grip target object and received by the ToF sensor.
  • a signal e.g., near-infrared, ultrasonic, laser, etc.
  • the gripper (130) may be controllable in three degrees of freedom (3 DOF) or six degrees of freedom (6 DOF) by one or more processors (140).
  • 3 degrees of freedom means left and right rotation (roll) around the X-axis, forward and backward rotation (pitch) around the Y-axis, and up and down rotation (yaw) around the Z-axis in a 3D rectangular coordinate system
  • 6 degrees of freedom can mean forward/back (surge), left/right (sway), and up/down (heave) translational motion in addition to 3DOF.
  • the gripper (130) can be positioned adjacent to a grip target object by one or more processors (140), and then grasp the grip target object in a grip pose under the control of one or more processors (140).
  • One or more processors (140) control the overall operation of the electronic device (100).
  • the processor (140) may be implemented as a digital signal processor (DSP) for processing a digital signal, a microprocessor, a timing controller (TCON).
  • DSP digital signal processor
  • the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, and an artificial intelligence (AI) processor, or may be defined by the corresponding terms.
  • the processor (140) may be implemented as a system on chip (SoC) having a processing algorithm built in, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA).
  • SoC system on chip
  • LSI large scale integration
  • FPGA field programmable gate array
  • the processor (140) may perform various functions by executing computer executable instructions stored in a memory.
  • the one or more processors (140) may include one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), an MIC (Many Integrated Core), a DSP (Digital Signal Processor), an NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator.
  • the one or more processors (140) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing.
  • the one or more processors (140) may execute one or more programs or instructions stored in a memory. For example, the one or more processors (140) may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.
  • the plurality of operations may be performed by one processor or may be performed by a plurality of processors.
  • the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-only processor).
  • One or more processors (140) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core).
  • processors (140) are implemented as a multi-core processor
  • each of the multiple cores included in the multi-core processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multi-core processor.
  • each of the multiple cores (or some of the multiple cores) included in the multi-core processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.
  • the plurality of operations may be performed by one core among the plurality of cores included in the multi-core processor, or may be performed by the plurality of cores.
  • the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.
  • a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single core processor, a multi-core processor, or a core included in a single core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.
  • SoC system on a chip
  • one or more processors (140) can input a first image acquired through a first sensor (110) into a first neural network model to obtain a first feature map corresponding to the first image, and input a second image acquired through a second sensor (120) into a second neural network model to obtain a second feature map corresponding to the second image.
  • one or more processors (140) may merge (or fuse) the first feature map and the second feature map to obtain a third image.
  • the third image may be a fused image of the first feature map and the second feature map.
  • One or more processors (140) can identify a grip target object based on the third image and obtain movement information of the gripper (130) for grasping the grip target object.
  • the movement information of the gripper (130) can include a movement direction, a movement distance, and a rotation angle of the gripper (130) (e.g., a grip pose of the gripper (130) according to shape information of the grip target object).
  • FIG. 3 is a diagram for explaining a first neural network model and a second neural network model according to one embodiment of the present disclosure.
  • one or more processors (140) can input a first image (1) acquired through a first sensor (110) into a first neural network model (10) to acquire a first feature map (1') corresponding to the first image (1).
  • the first neural network model (10) may be a model learned to output RGB feature information according to an object in a sample image based on RGB information included in the sample image.
  • one or more processors (140) may input a second image (2) acquired through a second sensor (120) into a second neural network model (20) to acquire a second feature map (2') corresponding to the second image (2).
  • a second neural network model (20) may be a model learned to output depth feature information according to an object in a sample image based on depth information included in the sample image.
  • the artificial intelligence related function according to the present disclosure is operated through one or more processors (140) and memory of the electronic device (100).
  • the one or more processors (140) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an NPU (Neural Processing Unit), but are not limited to the examples of the processors described above.
  • a CPU Central Processing Unit
  • GPU Graphic Processing Unit
  • NPU Neurological Processing Unit
  • CPU is a general-purpose processor that can perform not only general operations but also artificial intelligence operations, and can efficiently execute complex programs through a multi-layer cache structure.
  • CPU is advantageous in a serial processing method that enables organic linkage between the previous calculation result and the next calculation result through sequential calculation.
  • General-purpose processors are not limited to the above-mentioned examples, except in the case where they are specified as CPUs.
  • GPU is a processor for large-scale operations such as floating point operations used in graphic processing, and can perform large-scale operations in parallel by integrating a large number of cores.
  • GPU may be advantageous over CPU in parallel processing methods such as convolution operations.
  • GPU may be used as a co-processor to supplement the function of CPU.
  • Processors for large-scale operations are not limited to the examples described above, except in the case where they are specified as GPUs described above.
  • An NPU is a processor specialized in artificial intelligence operations using an artificial neural network, and each layer constituting the artificial neural network can be implemented with hardware (e.g., silicon).
  • hardware e.g., silicon
  • an NPU can be implemented in various forms such as a TPU (Tensor Processing Unit), an IPU (Intelligence Processing Unit), a VPU (Vision processing unit), etc.
  • the artificial intelligence processor is not limited to the examples described above, except in the case where it is specified as an NPU described above.
  • processors (140) may be implemented as a SoC (System on Chip).
  • SoC System on Chip
  • the SoC may further include, in addition to one or more processors (140), a memory, and a network interface such as a bus for data communication between the processor and the memory.
  • the electronic device (100) may perform operations related to artificial intelligence (for example, operations related to learning or inference of an artificial intelligence model) by using some of the plurality of processors.
  • the electronic device (100) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors.
  • this is only one embodiment, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor.
  • the electronic device (100) can perform operations for functions related to artificial intelligence by using multiple cores (e.g., dual cores, quad cores, etc.) included in one processor.
  • the electronic device (100) can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel by using multiple cores included in the processor.
  • One or more processors (140) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in memory.
  • the predefined operation rules or artificial intelligence models are characterized by being created through learning.
  • being created through learning means that a predetermined operation rule or artificial intelligence model with desired characteristics is created by applying a learning algorithm to a large number of learning data.
  • This learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server/system.
  • the artificial intelligence model may be composed of a plurality of neural network layers. At least one layer has at least one weight value and performs the operation of the layer through the operation result of the previous layer and at least one defined operation.
  • Examples of the neural network include a CNN (Convolutional Neural Network), a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), an RBM (Restricted Boltzmann Machine), a DBN (Deep Belief Network), a BRDNN (Bidirectional Recurrent Deep Neural Network), and Deep Q-Networks, and a Transformer, and the neural network in the present disclosure is not limited to the above-described examples unless otherwise specified.
  • a learning algorithm is a method of training a given target device (e.g., a robot) using a plurality of learning data so that the given target device can make decisions or predictions on its own.
  • Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in the present disclosure is not limited to the above-described examples unless otherwise specified.
  • FIG. 4 is a diagram for explaining a third neural network model and movement information according to one embodiment of the present disclosure.
  • one or more processors (140) may input a first image (1) into a first neural network model (10) to obtain a first feature map (1') corresponding to the first image (1).
  • one or more processors (140) may input a first image (1) including RGB information to a first neural network model (10) based on ResNet-50 and a Feature Pyramid Network (FPN).
  • the first neural network model (10) may output a first feature map (1') including RGB feature information.
  • one or more processors (140) can input a second image (2) into a second neural network model (20) to obtain a second feature map (2') corresponding to the second image (2).
  • one or more processors (140) can input a second image (2) including depth information to a second neural network model (20) based on set abstraction and feature propagation layers of PointNet++.
  • the second neural network model (20) can output a second feature map (2') including depth feature information.
  • one or more processors (140) may pre-process the second image (2) when the second image (2) is received and input the pre-processed second image into the second neural network model (20).
  • the preprocessed second image may include point cloud information about the grip target object.
  • the point cloud information may mean information representing the grip target object by assuming the grip target object (e.g., a 3D object) as a collection of multiple points.
  • one or more processors (140) can obtain point cloud information by applying uniform grid sampling to the image space within the second image (2).
  • one or more processors (140) can input point cloud information into the second neural network model (20), thereby reducing the input size and thus reducing the computational load.
  • One or more processors (140) can merge the first feature map (1') and the second feature map (2') to obtain a third image (3).
  • one or more processors (140) may obtain a third image (3) including a fused feature by concatenating the RGB feature information and the depth feature information in the channel direction to utilize the RGB feature information included in the first feature map (1') and the depth feature information included in the second feature map (2').
  • One or more processors (140) can input the third image (3) into the third neural network model (30) to obtain movement information of the gripper (130) to control the gripper (130) to grasp the grip target object.
  • one or more processors (140) can input a third image (3) into a third neural network model (30) based on a multi-layer perceptron (MLP) to obtain movement information of the gripper (130).
  • MLP multi-layer perceptron
  • the third neural network model (30) may be a model trained to output a grasp probability (or a grasp score of 0 to 1) to predict whether a grip target object can be grasped.
  • the third neural network model (30) can output movement information including at least one of a movement direction, a movement distance, and a rotation angle of the gripper (130) based on shape information of a grip target object included in the third image.
  • the third neural network model (30) may include a movement distance head (Head) for inferring a movement distance, a movement direction head (or, Approach Direction Head) of a gripper (130) for grasping a grip target object, and a lateral direction head (Lateral Direction Head) for inferring a rotation angle (hereinafter, lateral direction with respect to the grip target object).
  • a movement distance head Head
  • a movement direction head or, Approach Direction Head
  • a gripper for grasping a grip target object
  • a lateral direction head Lateral Direction Head
  • the moving distance head can output the moving distance of the gripper (130) to move the gripper (130) adjacent to the grip target object and the grip width of the gripper (130) to grasp the grip target object.
  • one or more processors (140) can position the gripper (130) adjacent to a grip target object based on the movement direction and movement distance.
  • One or more processors (140) can identify a lateral direction for a grip target object based on a rotation angle and control the gripper (130) to grasp the grip target object in the identified lateral direction.
  • the lateral direction for the grip target object can include a grasp pose for appropriately grasping the grip target object.
  • FIG. 5 is a diagram for explaining reliability according to one embodiment of the present disclosure.
  • the third neural network model (30) can obtain multiple movement information of the gripper (130) for grasping the grip target object based on the shape information of the grip target object.
  • the third neural network model (30) can obtain a reliability corresponding to each of multiple pieces of movement information and output the movement information with the highest reliability.
  • the third neural network model (30) includes a movement distance head, an approach direction head, and a lateral direction head that are learned to output movement information of a gripper (130) for grasping a grip target object, and may include a collision head for inferring whether or not the grip target object collides with another object while being grasped.
  • the third neural network model (30) may be a model learned using loss functions suitable for the outputs of each of the moving distance head, the approach direction head, and the lateral direction head.
  • the third neural network model (30) may be a model that learns a regression-classification task between the predicted value of the head and the actual value (ground truth) using a binary cross-entropy loss function.
  • the third neural network model (30) can obtain the output value of the binary cross-entropy loss function for all pixels in the third image (3) and identify points at which the grip target object can be grasped.
  • the third neural network model (30) can identify a point cloud, which is a collection of points capable of grasping a grip target object.
  • the third neural network model (30) may be a model trained on a regression task using a regression loss function.
  • the regression loss function may include a mean absolute error.
  • the third neural network model (30) may define a regression loss function using the mean absolute error for graspable points.
  • the third neural network model (30) outputs a plurality of pieces of movement information, and can obtain reliability by considering the probability of grasping the grip target object and the probability of collision with another object according to each piece of movement information.
  • the third neural network model (30) outputs movement information having the highest reliability among multiple reliabilities
  • one or more processors (130) are positioned adjacent to the grip target object and can grasp the grip target object based on the movement information output by the third neural network model (30).
  • FIG. 6 is a drawing for explaining the result of an electronic device moving a grip target object according to an embodiment of the present disclosure.
  • one or more processors (140) control the gripper (130) to grasp a grip target object, and when the gripper (130) grasps the grip target object, the grip target object can be moved to another location.
  • one or more processors (140) identify a grip target object based on a third image (3) that fuses a first feature map (1') with an enhanced first characteristic and a second feature map (2') with an enhanced second characteristic, and obtains movement information of a gripper (130) for grasping the grip target object. Therefore, even when the grip target object is flat, such as a plate on a table, or when it is difficult to obtain depth feature information, such as a knife, the grasping success rate for the grip target object can be increased.
  • one or more processors (140) obtain movement information of a gripper (130) for grasping a grip target object based on a third image (3) that fuses a first feature map (1') with an enhanced first characteristic and a second feature map (2') with an enhanced second characteristic, so that even when it is difficult to obtain the first characteristic (for example, when the surrounding environment is dark), the grasping success rate for the grip target object may not decrease.
  • conventional robots A and B do not input a first image (1) including first feature information into a first neural network model (10), and do not input a second image (2) including second feature information into a second neural network model (20) to obtain a first feature map (1') and a second feature map (2'), and do not grasp a grip target object using a third image (3) that is a fusion of the first feature map (1') and the second feature map (2'), so conventional robots A and B can grasp a grip target object with a lower success probability than the grasping success probability of an electronic device (100) according to an example of the present disclosure.
  • One or more processors (140) can obtain an updated first image (1) from a first sensor (110) in real time or at preset intervals, and can obtain an updated second image (2) from a second sensor (120).
  • one or more processors (140) may input an updated first image (1) into a first neural network model (10) to obtain an updated first feature map (1'), and input an updated second image (2) into a second neural network model (20) to obtain an updated second feature map (2').
  • One or more processors (140) can merge the updated first feature map (1') and the updated second feature map (2') to obtain an updated third image (3).
  • One or more processors (140) can update movement information of the gripper (130) to control the gripper (130) to grasp the grip target object based on the updated third image (3), and control the gripper (130) based on the updated movement information.
  • FIG. 7 is a flowchart for explaining a method for controlling an electronic device according to an embodiment of the present disclosure.
  • a first image acquired through a first sensor is input into a first neural network model to acquire a first feature map corresponding to the first image (S710).
  • the control method inputs a second image acquired through a second sensor into a second neural network model to acquire a second feature map corresponding to the second image (S720).
  • the control method acquires a third image by merging the first feature map and the second feature map (S730).
  • the control method identifies a grip target object based on a third image (S740).
  • the control method controls the gripper to grasp a grip target object (S750).
  • the first sensor is an RGB sensor
  • the step S710 of obtaining the first feature map includes the step of, when the first image including RGB information is received through the RGB sensor, inputting the first image into the first neural network model, and the first feature map obtained from the first neural network model may include RGB feature information for the grip target object.
  • the second sensor is a depth sensor
  • the step S720 of obtaining the second feature map includes the step of, when the second image including depth information is received through the depth sensor, inputting the second image into the second neural network model, and the second feature map obtained from the second neural network model may include depth feature information for the grip target object.
  • the step S720 of obtaining the second feature map includes the step of preprocessing the second image when the second image is received, and the step of inputting the preprocessed second image into the second neural network model, wherein the preprocessed second image may include point cloud information for the grip target object.
  • the step S740 of identifying the grip target object includes a step of inputting the third image into a third neural network model to obtain movement information of the gripper for controlling the gripper to grasp the grip target object, and the third neural network model can output the movement information including at least one of a movement direction, a movement distance, and a rotation angle of the gripper based on shape information of the grip target object included in the third image.
  • the step S750 of controlling the gripper may include the steps of positioning the gripper adjacent to the grip target object according to the movement direction and the movement distance, the step of identifying a lateral direction for the grip target object according to the rotation angle, and the step of controlling the gripper to grip the grip target object in the identified lateral direction.
  • the third neural network model obtains, based on the shape information, a plurality of movement information of the gripper for the gripper to grasp the grip target object, obtains a reliability corresponding to each of the plurality of movement information, and outputs movement information having the highest reliability, wherein the reliability may include a probability of the gripper grasping the grip target object.
  • a control method may further include the steps of: acquiring an updated first image from the first sensor in real time or at preset time intervals, and acquiring an updated second image from the second sensor; inputting the updated first image into the first neural network model to acquire an updated first feature map; inputting the updated second image into the second neural network model to acquire an updated second feature map; merging the updated first feature map and the updated second feature map to acquire an updated third image; updating movement information of the gripper based on the updated third image to control the gripper to grasp the grip target object; and controlling the gripper based on the updated movement information.
  • each of the first image and the second image may be an image including at least a portion of the gripper and the grip target object.
  • the step S750 of controlling the gripper may include a step of controlling the gripper with 6 degrees of freedom.
  • the various embodiments described above may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof.
  • the embodiments described in this specification may be implemented by the processor itself.
  • embodiments such as the procedures and functions described in this specification may be implemented by separate software modules. Each of the software modules may perform one or more functions and operations described in this specification.
  • computer instructions for performing processing operations of an electronic device may be stored in a non-transitory computer-readable medium.
  • the computer instructions stored in the non-transitory computer-readable medium are executed by a processor of a specific device, they cause the specific device to perform processing operations in the electronic device (100) according to various embodiments described above.
  • a non-transitory computer-readable medium is not a medium that stores data for a short period of time, such as a register, cache, or memory, but a medium that permanently stores data and can be read by a device.
  • Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray disks, USBs, memory cards, and ROMs.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Robotics (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Image Analysis (AREA)

Abstract

Un dispositif électronique est divulgué. Le dispositif électronique comporte un premier capteur, un second capteur, un dispositif de préhension et un ou plusieurs processeurs qui : entrent une première image obtenue par l'intermédiaire du premier capteur, dans un premier modèle de réseau neuronal pour obtenir une première carte de caractéristiques correspondant à la première image ; entrent une deuxième image obtenue par l'intermédiaire du second capteur, dans un second modèle de réseau neuronal pour obtenir une seconde carte de caractéristiques correspondant à la deuxième image ; fusionnent la première carte de caractéristiques et la seconde carte de caractéristiques pour obtenir une troisième image ; identifient, sur la base de la troisième image, un objet devant être saisi ; et commandent le dispositif de préhension pour saisir l'objet devant être saisi.
PCT/KR2024/001660 2023-02-27 2024-02-05 Dispositif électronique comportant un dispositif de préhension permettant de saisir un objet devant être saisi et procédé de commande associé Ceased WO2024181702A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
KR20230026026 2023-02-27
KR10-2023-0026026 2023-02-27
KR10-2023-0094078 2023-07-19
KR1020230094078A KR20240133506A (ko) 2023-02-27 2023-07-19 그립 대상 객체를 파지하기 위한 그리퍼를 포함하는 전자 장치 및 그 제어 방법

Publications (1)

Publication Number Publication Date
WO2024181702A1 true WO2024181702A1 (fr) 2024-09-06

Family

ID=92590804

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2024/001660 Ceased WO2024181702A1 (fr) 2023-02-27 2024-02-05 Dispositif électronique comportant un dispositif de préhension permettant de saisir un objet devant être saisi et procédé de commande associé

Country Status (1)

Country Link
WO (1) WO2024181702A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019093461A (ja) * 2017-11-20 2019-06-20 株式会社安川電機 把持システム、学習装置、把持方法、及び、モデルの製造方法
JP2020001127A (ja) * 2018-06-28 2020-01-09 勇貴 高橋 ピッキングシステム，ピッキング処理装置及びプログラム
KR102220304B1 (ko) * 2020-02-28 2021-02-25 주식회사 두산 로봇 제어 장치 및 로봇 제어 방법
JP2022077725A (ja) * 2020-11-12 2022-05-24 オムロン株式会社 計測システム、計測装置、計測方法及び計測プログラム
JP2022179118A (ja) * 2021-05-21 2022-12-02 株式会社Preferred Networks 把持推定装置、モデル生成方法、モデル学習装置、及びグリッパ

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019093461A (ja) * 2017-11-20 2019-06-20 株式会社安川電機 把持システム、学習装置、把持方法、及び、モデルの製造方法
JP2020001127A (ja) * 2018-06-28 2020-01-09 勇貴 高橋 ピッキングシステム，ピッキング処理装置及びプログラム
KR102220304B1 (ko) * 2020-02-28 2021-02-25 주식회사 두산 로봇 제어 장치 및 로봇 제어 방법
JP2022077725A (ja) * 2020-11-12 2022-05-24 オムロン株式会社 計測システム、計測装置、計測方法及び計測プログラム
JP2022179118A (ja) * 2021-05-21 2022-12-02 株式会社Preferred Networks 把持推定装置、モデル生成方法、モデル学習装置、及びグリッパ

Similar Documents

Publication Publication Date Title
CN111055279B (zh) 基于触觉与视觉结合的多模态物体抓取方法与系统
EP3557521B1 (fr) Processeur et contrôleur de reconnaissance d'images pour dispositifs industriels
CN108908334A (zh) 一种基于深度学习的智能抓取系统及方法
CN108290286A (zh) 用于教导工业机器人拾取零件的方法
WO2020149493A1 (fr) Dispositif électronique et son procédé de commande
CN106256512A (zh) 包括机器视觉的机器人装置
CN115578460A (zh) 基于多模态特征提取与稠密预测的机器人抓取方法与系统
WO2024072001A1 (fr) Appareil et procédé de partage et d'élagage de poids pour des modèles de vision et de langage
WO2019124728A1 (fr) Appareil et procédé d'identification d'objet
CN114820796A (zh) 基于自监督表示学习的视觉抓取检测方法及系统
WO2024122845A1 (fr) Robot de service
WO2024181702A1 (fr) Dispositif électronique comportant un dispositif de préhension permettant de saisir un objet devant être saisi et procédé de commande associé
WO2021246645A1 (fr) Procédé d'optimisation d'algorithmes de traitement d'images et dispositif électronique le prenant en charge
KR20240133506A (ko) 그립 대상 객체를 파지하기 위한 그리퍼를 포함하는 전자 장치 및 그 제어 방법
WO2023167490A1 (fr) Appareil et procédé d'apprentissage auto-supervisé pour représentation de caractéristiques visuelles d'images egocentriques
WO2024053876A1 (fr) Dispositif électronique effectuant un étalonnage de caméra, et son procédé de fonctionnement
WO2025159384A1 (fr) Robot pour déplacer un objet en fonction de la taille de l'objet et procédé de commande pour celui-ci
CN112288819B (zh) 多源数据融合的视觉引导机器人抓取及分类系统及方法
WO2023063570A1 (fr) Dispositif électronique permettant d'obtenir des données d'image concernant un mouvement de la main et son procédé de fonctionnement
Barstuğan et al. Deep Learning Based Human Robot Interaction With 5G Communication
Chiha et al. Improving NAO Robot Grasps with Deep Learning-Based Rectangle Detection
Štursa et al. Multiple objects localization using image segmentation with U-net
Osmanpaşaoğlu et al. Deep Learning Based Human Robot Interaction With 5g Communication
US20240351198A1 (en) Trained model generation method, trained model generation device, trained model, and holding mode inference device
WO2026059073A1 (fr) Module central, dispositif robotisé le comprenant et son procédé de commande

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24764102

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 24764102

Country of ref document: EP

Kind code of ref document: A1