WO2024032040A1 - 一种物体拾取方法以及相关设备 - Google Patents

一种物体拾取方法以及相关设备 Download PDF

Info

Publication number
WO2024032040A1
WO2024032040A1 PCT/CN2023/091430 CN2023091430W WO2024032040A1 WO 2024032040 A1 WO2024032040 A1 WO 2024032040A1 CN 2023091430 W CN2023091430 W CN 2023091430W WO 2024032040 A1 WO2024032040 A1 WO 2024032040A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
cloud data
target
information
pickup
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/091430
Other languages
English (en)
French (fr)
Inventor
顾启鹏
金鑫
赵夕朦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Cloud Computing Technologies Co Ltd
Original Assignee
Huawei Cloud Computing Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Cloud Computing Technologies Co Ltd filed Critical Huawei Cloud Computing Technologies Co Ltd
Priority to EP23851270.1A priority Critical patent/EP4552808A4/en
Publication of WO2024032040A1 publication Critical patent/WO2024032040A1/zh
Priority to US19/050,588 priority patent/US20250187178A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1602—Program controls characterised by the control system, structure, architecture
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1612—Program controls characterised by the hand, wrist, grip control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1602—Program controls characterised by the control system, structure, architecture
    • B25J9/161—Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1628—Program controls characterised by the control loop
    • B25J9/163—Program controls characterised by the control loop learning, adaptive, model based, rule based expert control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1628—Program controls characterised by the control loop
    • B25J9/1653—Program controls characterised by the control loop parameters identification, estimation, stiffness, accuracy, error analysis
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Program controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1679—Program controls characterised by the tasks executed
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/37—Measurements
    • G05B2219/37555—Camera detects orientation, position workpiece, points of workpiece
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/39—Robotics, robotics to robotics hand
    • G05B2219/39536—Planning of hand motion, grasping
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40053—Pick 3-D object from pile of objects

Definitions

  • This application relates to the field of automation control technology, and specifically to an object picking method and related equipment.
  • automation control technologies such as robots are widely used in fields such as cargo handling, cargo transmission, intelligent manufacturing, and smart medical care.
  • autonomous picking technology is the basic capability of automated control such as robots.
  • automated control such as robots.
  • This application provides an object picking method to solve the problem that in scenes with many types and shapes of objects, it is difficult to achieve adaptive picking of objects in different situations, thereby making it difficult to meet the requirements of actual application scenarios.
  • This application also provides corresponding devices, equipment, computer-readable storage media, computer program products, etc.
  • the first aspect of this application provides an object picking method.
  • point cloud data about a target object can be obtained; target picking posture information obtained by processing the point cloud data by a neural network and the type of the target object are obtained.
  • Information, the target pickup posture information is used to describe the target pickup posture of the pickup device for the target object; according to the target pickup posture information and the type information, the pickup device is controlled to pick up the target object.
  • the neural network can obtain the target picking attitude information and the type information of the target object based on the point cloud data about the target object, so that it can adaptively use an appropriate picking method and picking attitude according to the situation of the target object.
  • Target objects are picked up to successfully perform the picking task.
  • the picking device includes multiple types of end effectors, and different types of end effectors are used to pick up different types of objects; the above step: picking up posture information according to the target and The type information controls the pickup device to pick up the target object, including: determining a target pickup mode of the pickup device according to the type information, and the target pickup mode is used to indicate the method used to pick up the target object.
  • An end effector controlling the pickup device to pick up the target object based on the target pickup posture indicated by the target pickup posture information in the target pickup mode.
  • multiple types of end effectors are provided in the picking device, so that an appropriate end effector can be selected for picking based on the type of object to be picked up (such as a target object), so that it can be picked based on actual scenarios.
  • a more appropriate picking mode is adaptively selected for picking.
  • the accurate picking attitude can also be determined through the neural network. In this way, based on the actual scene, the picking device can be adaptively controlled to pick up the target object in an appropriate picking mode and accurate attitude, realizing the collaboration of software and hardware. Control can successfully pick up objects in various situations and adaptively complete picking tasks in more complex scenes.
  • the neural network includes a first convolutional network and a second convolutional network network; the above step: obtaining the target picking attitude information and the type information of the target object obtained by processing the point cloud data by the neural network, including: obtaining, based on the point cloud data, through the first convolutional network The target picks up posture information; through the second convolution network, the type information of the target object is obtained based on the point cloud data.
  • both the first convolution network and the second convolution network may include convolution operations, and the convolution operations in the first convolution network and the second convolution network are not limited.
  • Specific methods in product operations may include one or more of the convolution operation on the matrix, the graph convolution operation on the graph structure, the convolution operation on the point cloud data, etc.
  • the specific methods of corresponding convolution operations may be different.
  • the specific methods of graph convolution operations for graph structures and convolution operations for point cloud data may be different. You can refer to the current or subsequent developments. related technologies to achieve.
  • two branch structures i.e., the first convolutional network and the second convolutional network
  • the way to pick up target objects is to adaptively select the appropriate way to pick up objects in various situations, so as to meet the needs of various application scenarios.
  • the above step: obtaining the target picking posture information based on the point cloud data through the first convolutional network includes: executing according to the end of the picking device Structural information of the device, and at least two sets of local point cloud data matching the end effector are obtained from the point cloud data, and each set of local point cloud data corresponds to a candidate pickup posture; according to the at least two sets of local point cloud data Cloud data, through the first convolutional network, obtain target picking attitude information.
  • each set of local point cloud data may include information on at least one candidate picking point to describe a corresponding candidate picking posture.
  • the at least one candidate picking point may be used to describe a candidate location for picking the target object through the end effector.
  • the candidate picking gesture may be characterized by at least one candidate picking point.
  • the local point cloud data includes not only the information of the candidate picking points, but also the structural information of the end effector, the candidate picking posture can be described more comprehensively.
  • the local point cloud data can not only describe a candidate pickup pose of the target object by the pickup device, but also include information such as the contact area when the pickup device picks up the target object.
  • Each set of local point cloud data corresponds to a candidate pickup posture, so that the first convolutional network determines the target candidate posture according to the candidate pickup postures corresponding to the multiple sets of local point cloud data, so as to output the target candidate posture information.
  • the first convolutional network is a graph convolutional network; the above step: according to the at least two sets of local point cloud data, through the first convolutional network, obtain
  • the target picking posture information includes: obtaining a graph structure according to the at least two sets of local point cloud data, each node in the graph structure corresponds to one local point cloud data; and using the first convolution network to obtain a graph structure. Structure processing to obtain target picking attitude information.
  • each set of local point cloud data is converted into a graph structure form through graph construction, so as to describe at least two sets of local point cloud data through structured data, which satisfies the requirements of the graph convolution network on the input data. formal requirements. It can be seen that multiple sets of local point cloud data can be effectively integrated through the graph structure, so that the target pickup posture information can be obtained through the graph convolution network based on multiple candidate pickup postures described in the graph structure.
  • the first convolutional network includes a first feature extraction network and a first classifier, and the first feature extraction network is located before the first classifier; the above steps : Obtaining the type information of the target object based on the point cloud data through the second convolution network, including: using the second convolution network, based on the point cloud according to the first convolution network When data is processed, the first feature extraction network outputs A feature information and the point cloud data are used to obtain the type information of the target object.
  • the first convolutional network in the process of determining the target pickup posture of the target object by the pickup device, the first convolutional network often needs to extract the local features corresponding to the target pickup posture.
  • the first convolutional network The first feature information output by the first feature extraction network when processing based on point cloud data may include local feature information corresponding to the target pickup posture.
  • the type of the target object is not only identified based on the global point cloud data, but also combined with the local feature information contained in the first feature information to identify the type of the target object.
  • the second convolutional network includes a second feature extraction network and a second classifier; the above steps: through the second convolutional network, according to the first convolutional network Obtaining the type information of the target object based on the first feature information output by the first feature extraction network when the product network processes the point cloud data and the point cloud data includes: using the second feature extraction The network processes the point cloud data to obtain second feature information; aggregates the first feature information and the second feature information to obtain an aggregation result; and processes the aggregation result through a second classifier, Obtain type information of the target object.
  • the second feature information usually includes global feature information of the target object in the point cloud data.
  • the first feature information may be feature information obtained based on multiple sets of local point cloud data.
  • the first feature information usually includes local feature information of the target object in the point cloud data.
  • the second classifier can aggregate the local feature information and global feature information of the target object to determine the type of the target object, thereby accurately classifying the target object based on relatively complete information.
  • a second aspect of the present application provides an object pickup device, which is applied to computer equipment.
  • the device has the function of implementing the method of the above-mentioned first aspect or any possible implementation of the first aspect.
  • This function can be implemented by hardware, or it can be implemented by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above functions, such as: acquisition module, processing module and control module.
  • a third aspect of the present application provides a computer device.
  • the computer device includes at least one processor, a memory, and computer-executable instructions stored in the memory and executable on the processor.
  • the processor executes the method of the above first aspect or any possible implementation of the first aspect.
  • a fourth aspect of the present application provides a computer-readable storage medium that stores one or more computer-executable instructions.
  • the processor executes any of the above-mentioned first aspects or possible methods of the first aspect. Ways to implement it.
  • a fifth aspect of the present application provides a computer program product that stores one or more computer-executable instructions.
  • the processor executes the above-mentioned first aspect or any possible implementation of the first aspect. Methods.
  • a sixth aspect of the present application provides a chip system.
  • the chip system includes a processor and is used to support a terminal to implement the functions involved in the above-mentioned first aspect or any possible implementation manner of the first aspect.
  • the chip system may also include a memory, which is used to store necessary program instructions and data for the computer device.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • Figure 1 is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present application.
  • Figure 2 is an exemplary schematic diagram of an object picking method provided by an embodiment of the present application.
  • Figure 3 is a schematic diagram of an exemplary control flow provided by an embodiment of the present application.
  • Figure 4a is an exemplary schematic diagram of the equivalent shape corresponding to the two-finger gripper provided by the embodiment of the present application;
  • Figure 4b is an exemplary schematic diagram of obtaining local point cloud data provided by the embodiment of the present application.
  • Figure 5 is an exemplary schematic diagram of obtaining the target picking posture of a target object provided by an embodiment of the present application
  • Figure 6 is an exemplary diagram of a neural network provided by an embodiment of the present application.
  • Figure 7 is a schematic diagram of an object pickup device provided by an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
  • picking includes grabbing or sucking, etc.
  • objects can be moved in different ways.
  • the end effector can be a manipulator, and the object can be grasped by the manipulator.
  • the end effector can be a suction cup, in which case the object can be sucked through the suction cup.
  • the end effector may also have other structural forms, which are not limited in the embodiments of this application.
  • the end effector is a magnetic end effector, and at this time, the object can be attracted by magnetic attraction.
  • embodiments of the present application provide an object picking method, which can be combined with a neural network to adaptively pick up objects of different types and situations.
  • Artificial Intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the nature of intelligence and produce a new class of intelligent machines that can respond in a manner similar to human intelligence.
  • Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.
  • Figure 1 shows a structural schematic diagram of the main framework of artificial intelligence.
  • the following is an analysis of the above artificial intelligence theme from the two dimensions of “intelligent information chain” (horizontal axis) and “IT value chain” (vertical axis). framework is explained.
  • the "intelligent information chain” reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensation process of "data-information-knowledge-wisdom".
  • the "IT value chain” reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (providing and processing technology implementation) to the systematic industrial ecological process.
  • Infrastructure provides computing power support for artificial intelligence systems, enables communication with the external world, and supports it through basic platforms.
  • computing power consists of smart chips (central processing unit (CPU), neural network processing unit (NPU), graphics processing unit (GPU), dedicated integrated Circuits (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA), tensor processing unit (TPU) and other hardware acceleration chips are provided;
  • the basic platform includes a distributed computing framework and network and other related platform guarantees and support, which can include cloud storage and computing, interconnection networks, etc.
  • sensors communicate with the outside world to obtain data, which are provided to smart chips in the distributed computing system provided by the basic platform for calculation.
  • Data from the upper layer of the infrastructure is used to represent data sources in the field of artificial intelligence.
  • the data involves graphics, images, voice, and text, as well as IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.
  • Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.
  • machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.
  • Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formal information to perform machine thinking and problem solving based on reasoning control strategies. Typical functions are search and matching.
  • Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.
  • some general capabilities can be formed based on the results of further data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, and image processing. identification, etc.
  • Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Its application fields mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, smart cities, etc.
  • the neural network can be composed of neural units.
  • the neural unit can refer to an operation unit that takes xs (ie, input data) and intercept 1 as input.
  • the output of the operation unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • Ws is the weight of xs
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field.
  • the local receptive field can be an area composed of several neural units.
  • DNN deep neural network
  • CNN convolutional neural network
  • This application does not limit the specific types of neural networks involved.
  • Deep Neural Network can be understood as a neural network with many hidden layers. There is no special metric for "many” here. The essence of what we often call multi-layer neural networks and deep neural networks is It's the same thing. From the division of DNN according to the position of different layers, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.
  • each layer can be understood as the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just a pair of input vectors After such a simple operation, the output vector is obtained Since there are many DNN layers, the coefficient W and offset vector There will also be a lot of them. The following describes how specific parameters are defined in DNN. First, the definition of coefficient W is introduced.
  • the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the number of layers where the coefficient W is located, while the subscript is The corresponding output is the third layer index 2 and the input second layer index 4.
  • the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as Note that the input layer has no W parameter.
  • more hidden layers make the network more capable of describing complex situations in the real world. Theoretically, a model with more parameters has higher complexity and greater "capacity", which means it can complete more complex learning tasks.
  • Convolutional neural network (Convosutionas Neuras Network, CNN) is a deep neural network with a convolutional structure.
  • the convolutional neural network contains a feature extractor composed of convolutional layers and subsampling layers.
  • the feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map).
  • the convolutional layer refers to the neuron layer in the convolutional neural network that convolves the input signal.
  • a neuron can be connected to only some of the neighboring layer neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units.
  • Neural units in the same feature plane share weights, and the shared weights here are convolution kernels.
  • Shared weights can be understood as a way to extract image information independent of position. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that the image information learned in one part can also be used in another part. So for all positions on the image, we can use the same learned image information.
  • multiple convolution kernels can be used to extract different image information. Generally, the greater the number of convolution kernels, the richer the image information reflected by the convolution operation.
  • the convolution kernel can be initialized in the form of a random-sized matrix. During the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning.
  • the convolutional neural network can use the error back propagation (BP) algorithm to modify the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller.
  • BP error back propagation
  • forward propagation of the input signal until the output will produce an error loss
  • the parameters in the initial super-resolution model are updated by back-propagating the error loss information, so that the error loss converges.
  • the backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.
  • the object picking method of the embodiment of the present application can be applied to computer equipment.
  • the computer equipment can be used as a host computer of the picking device to control the picking device.
  • the specific type of the computer equipment can be multiple, which is not limited here.
  • the computer device may be a terminal device, a server, a container or a virtual machine.
  • the computer device may be a terminal device.
  • the terminal device can be a mobile phone (mobile phone), a tablet computer (pad), a computer with wireless transceiver functions, a virtual reality (virtual reality, VR) terminal, an augmented reality (AR) terminal, an industrial control
  • the terminal device can directly or indirectly control the movement of the pickup device.
  • the terminal device can directly send an instruction signal to the pickup device to instruct the pickup device to move.
  • other devices such as motors, etc.
  • the terminal device may drive the end effector in the pickup device to move by controlling the motor, solenoid valve, and other devices.
  • the computer device may be a cloud device, such as a cloud server or server cluster, or a cloud container or virtual machine, etc.
  • a neural network can be deployed in the cloud device, and the cloud device can execute embodiments of the present application to obtain control information for controlling the pickup device, and then send the control information to the pickup device to control the movement of the pickup device, Alternatively, the control information can also be sent to a local terminal device corresponding to the pickup device, so that the movement of the pickup device can be controlled through the local terminal device.
  • the pick-up device may include an end effector.
  • the structure and type of the pickup device can be determined based on the actual application scenario, which is not limited in the embodiments of the present application.
  • the picking device may include multiple types of end effectors, and different end effectors are used to pick up different types of objects.
  • the number of end effectors of the same type in the pickup device may be one or more.
  • the picking device may include multiple suction cups, wherein the multiple suction cups may work independently to suck different objects, or may be used to suck the same object.
  • the end effectors can be arranged in various ways. For example, multiple end effectors may be arranged in an array in the pickup device, or may be arranged irregularly in the pickup device. In addition, in some examples, the arrangement of the multiple end effectors on the pickup device can also change based on changes in the application scenario.
  • the computer device and the picking device can be located in the same physical device, for example, on the same robot.
  • the computer device and the picking device can be modules on the robot; or they can also be located in different physical devices.
  • the computer equipment is located in the cloud, and the picking device is located on the local robot. At this time, the computer equipment can control the picking device through communication connection.
  • the object picking method implemented in combination with a neural network includes steps 201-203.
  • Step 201 Obtain point cloud data about the target object.
  • the target object may be an object to be picked up.
  • the target object can be of various types, sizes and/or materials. There are no restrictions here.
  • the material of the target object may be one or more of a carton, a paper bag, a foam bag, a foam box, a plastic bag, etc. in which the cargo is wrapped.
  • the material of the target object surface is usually used as the material of the target object.
  • Point cloud data about the target object may refer to point cloud data containing feature points about the target object.
  • the point cloud data can also be considered as scene point cloud data of the scene where the target object is located, or operations such as filtering, downsampling, plane removal based on plane equations, etc. are performed on the scene point cloud data to Delete some point clouds in the scene point cloud data that are not helpful for subsequent processing to obtain point cloud data about the target object.
  • the point cloud data may be pre-stored in the computer device that executes the embodiment of the present application, or, It may also be transmitted to the computer device through other devices, or it may be generated by the computer device based on data collected by a sensor device such as a camera in the computer device.
  • the point cloud data may be generated based on data collected by a depth camera or lidar.
  • one exemplary way of generating point cloud data may include the following steps:
  • the scene point cloud data corresponding to the scene is obtained
  • Preprocess the scene point cloud data to obtain point cloud data about the target object.
  • the depth image can be obtained in various ways.
  • the depth image can be obtained through a depth camera, lidar, or structured light.
  • scene point cloud data can be obtained based on the depth image.
  • the method of obtaining the scene point cloud data corresponding to the scene according to the depth image can be implemented based on related technologies, which is not limited in the embodiments of the present application.
  • the depth image is acquired by a depth camera.
  • the coordinates of the pixels in the depth image in the camera coordinate system of the depth camera can be calculated.
  • the coordinates of the pixels in the depth image in the world coordinate system can be calculated.
  • the scene point cloud data can be obtained.
  • the preprocessing may include processing operations such as filtering, downsampling, background plane removal based on plane equations, etc., to delete some point cloud information in the scene point cloud data that is not helpful for subsequent processing (such as scene point cloud some noise and/or background point cloud information in the data), thereby obtaining point cloud data about the target object.
  • the preprocessing can also include a uniform sampling operation (for example, the last step of the preprocessing is a uniform sampling operation).
  • the density of the scene point cloud data can be reduced through uniform sampling, and the amount of data can be reduced to reduce the subsequent cost. reduce the amount of data processing and reduce the consumption of computing resources.
  • the point cloud data may only include feature points about the target object. In this way, the point cloud data may describe the outline of the target object for subsequent type recognition and determination of the target pickup posture. The operation provides a better data foundation, reduces the impact of interfering data, and improves the accuracy of subsequent processing results.
  • Step 202 Obtain the target picking posture information and the type information of the target object obtained by processing the point cloud data by the neural network.
  • the target pickup posture information is used to describe the target pickup posture of the pickup device for the target object.
  • the point cloud data can be processed through a neural network to obtain the target pickup attitude information and the type information of the target object.
  • the point cloud data can be directly input into the neural network to be processed by the neural network; at this time, the neural network includes a neural network structure that can process the point cloud data, for example, it can include networks such as Pointnet.
  • the point cloud data can also be input to the neural network for processing after specified processing.
  • the neural network includes a graph convolution network
  • the point cloud data can be processed to obtain the corresponding graph structure, and then Use this graph structure as input to this graph convolutional network.
  • the point cloud number can be calculated based on the parameters of the end effector. Search and match the data, obtain local point cloud data used to characterize the candidate picking pose from the point cloud data, and generate a graph structure based on each set of local point cloud data. In this way, the graph convolution network can be based on the graph structure described in The information of each candidate pickup posture is used to determine the target pickup posture.
  • the neural network may include a dual-branch structure, the output of one branch structure of the dual-branch structure is the target pickup posture information, and the output of the other branch structure is the type information of the target object.
  • the two branch structures included may be two parallel structures, or there may be a part of a shared structure and a part of a parallel structure.
  • the data contained in the two branch structures can be executed independently of each other, and there can also be some data that can be used in the two branch structures, for example, the output of a certain layer in a certain branch structure
  • the data can be input as part of a layer of another branch structure for processing in a layer of another branch structure.
  • the inputs of the two branch structures may be the same, different, or partially the same.
  • the target pickup posture information is used to indicate the posture of the pickup device when picking up the target object, that is, the target pickup posture.
  • the target pickup posture information may include 6-dimensional (6D) pose information, where 6D includes 6 degrees of freedom (DoF), specifically including 3 degrees of freedom translation (translation), And 3 degrees of freedom space rotation (rotation).
  • 6D includes 6 degrees of freedom
  • DoF degrees of freedom
  • translation translation
  • rotation degrees of freedom space rotation
  • the target pickup posture information can be described by the three-dimensional coordinates and normal vector of the target pickup point where the target object is picked up by the pickup device.
  • the neural network to pick up the attitude information of the target obtained by processing the point cloud data.
  • the output regarding the target pickup posture obtained by processing the point cloud data through the neural network may be the three-dimensional coordinates of the target pickup point.
  • the normal vector corresponding to the target pickup point can be obtained from the point cloud data.
  • the three-dimensional coordinates and normal vector can be in the camera coordinate system of the depth camera corresponding to the point cloud data or in the world coordinate system.
  • the described target pickup posture is also in the camera coordinate system of the corresponding depth camera or in the world coordinate system. under the coordinate system.
  • the three-dimensional coordinates and normal vector need to be converted to the coordinate system corresponding to the pickup device, and the pickup device is controlled based on the converted three-dimensional coordinates and normal vector.
  • the output of the target picking posture obtained by processing the point cloud data by the neural network can be the three-dimensional coordinates and normal vector of the target picking point.
  • the three-dimensional coordinates and normal vector of the target picking point output by the neural network can be The coordinates and normal vector are converted to the coordinate system corresponding to the picking device, and the picking device is controlled based on the converted three-dimensional coordinates and normal vector.
  • the type information of the target object is used to indicate the type of the target object.
  • the target object may include multiple materials, and accordingly, the target object may correspond to multiple types of information.
  • the type information of the target object can be used to indicate the type of the entire target object, or can be used to indicate the type of the part in contact with the target object when the picking device picks up the target object. .
  • Step 203 Control the pickup device to pick up the target object according to the target pickup posture information and type information.
  • the pickup device After obtaining the target pickup posture information and the target object type information, the pickup device can be controlled to determine a suitable pickup mode based on the target object type information, and a better grasp of the target object can be determined based on the target pickup posture information. posture, so as to stably pick up the target object with a suitable picking mode and a better grasping posture. to successfully perform the pickup task.
  • a suitable end effector can be determined from the picking device based on the type information of the target object, and suitable picking parameters can also be determined based on the type information of the target object. For example, when the size of the target object is within a preset size range, if the type of the target object is metal, the value of the pickup parameter of the end effector (such as the air flow rate of the suction cup) can be determined so that the pickup of the end effector The force is larger; and if the type of the target object is plastic, the value of the pickup parameter of the end effector can be determined so that the pickup force of the end effector is smaller. Then, the picking device can be controlled to pick up the target object with the appropriate end effector, appropriate picking parameters, and accurate attitude.
  • the pickup parameter of the end effector such as the air flow rate of the suction cup
  • the neural network can obtain the target picking posture information and the type information of the target object based on the point cloud data about the target object, so that it can adaptively use a suitable picking method and pickup method according to the situation of the target object. Pose to pick up the target object.
  • the picking device can be controlled to pick up the target object based on the target picking attitude information and type information obtained by the neural network, thereby realizing collaborative control combining software and hardware, which can detect objects in various situations. Perform successful picking and adaptively complete picking tasks with more complex scenes.
  • the picking device includes multiple types of end effectors, and different types of end effectors are used to pick up different types of objects;
  • the above step 203 includes:
  • the target pickup mode is used to indicate the end effector used to pick up the target object
  • the pickup device is controlled to pick up the target object based on the target pickup attitude indicated by the target pickup attitude information in the target pickup mode.
  • the multiple types of end effectors may include at least two of a manipulator, a suction cup, and a magnetic end effector.
  • the multiple types of end effectors may include multiple types of manipulators, multiple types of suction cups, or multiple types of magnetic end effectors.
  • various types of suction cups can include single-tower suction cups and multi-corrugated suction cups.
  • the specific classification method of types is not limited here.
  • a certain end effector can be a manipulator, and the object can be grasped by the manipulator.
  • a certain end effector can be a suction cup, in which case the object can be sucked through the suction cup.
  • a certain end effector is a magnetic end effector. At this time, the object can be attracted by magnetic attraction.
  • the number of end effectors of the same type may be one or more. Moreover, multiple end effectors can work independently to pick up different objects, or they can be used to pick up the same object.
  • the end effectors can be arranged in various ways. For example, multiple end effectors may be arranged in an array in the pickup device, or may be arranged irregularly in the pickup device. In addition, in some examples, the arrangement of the multiple end effectors on the pickup device can also change based on changes in the application scenario. In addition, the spacing between end effectors can also be determined based on information such as the size and shape of the objects to be picked up in actual application scenarios.
  • the computer equipment can process the point cloud data through the neural network to obtain the type information of the target object and the target pickup posture, and then control the pickup device to switch to the target pickup mode according to the type information of the target object, thereby switching to the method used to pick up the target object.
  • the end effector controls the picking device to perform the picking task in the target picking attitude.
  • multiple types of end effectors are provided in the picking device, so that an appropriate end effector can be selected for picking according to the type of object to be picked up (such as the target object in the embodiment of the present application), so that Based on the type of objects to be picked up in the actual scene, a more appropriate picking mode can be adaptively selected for picking.
  • the accurate picking attitude can also be determined through the neural network. In this way, based on the actual scene, the picking device can be adaptively controlled to pick up the target object in an appropriate picking mode and accurate attitude, thereby successfully performing the picking task.
  • a neural network may be deployed in the computer device, so that the type of object to be picked up and the picking posture of the picking device are determined based on the neural network.
  • neural networks Some examples of neural networks are introduced below.
  • the neural network includes a first convolutional network and a second convolutional network
  • the above step 202 includes:
  • the target picking attitude information is obtained
  • the type information of the target object is obtained based on the point cloud data.
  • both the first convolution network and the second convolution network may include convolution operations, and the specific methods of the convolution operations in the first convolution network and the second convolution network are not limited. .
  • the convolution operation in the neural network may include one or more of the convolution operation on the matrix, the graph convolution operation on the graph structure, the convolution operation on the point cloud data, etc.
  • the specific methods of corresponding convolution operations may be different.
  • the specific methods of graph convolution operations for graph structures and convolution operations for point cloud data may be different. You can refer to the current or subsequent developments. This is achieved by using related technologies, which will not be described again in the embodiments of this application.
  • the first convolutional network and the second convolutional network may be two parallel structures, or may have a part of a shared structure and a part of a parallel structure.
  • the data of the first convolutional network and the second convolutional network can be executed independently of each other, or there can be some data that can be used for the first convolutional network and the second convolutional network, for example, in the first convolutional network
  • the data output by a certain layer of the second convolutional network can be used as part of the input of a certain layer of the second convolutional network.
  • the inputs of the first convolutional network and the second convolutional network may be the same, may be different, or may be partially the same.
  • the target picking pose information can be directly obtained through the output of the first convolutional network.
  • the output of the first convolution network can also be processed through information processing operations such as information conversion to obtain the target picking posture information.
  • the output of the first convolutional network may be the three-dimensional coordinates of the target pickup point in the camera coordinate system of the depth camera corresponding to the point cloud data.
  • the normal vector of the target pickup point in the camera coordinate system can be obtained from the point cloud data.
  • two branch structures i.e., the first convolutional network and the second convolutional network
  • the picking method adaptively selects the appropriate method for picking objects in various situations, thus meeting the needs of various application scenarios.
  • the above steps obtaining target picking posture information based on point cloud data through the first convolutional network, including:
  • At least two sets of local point cloud data matching the end effector are obtained from the point cloud data, and each set of local point cloud data corresponds to a candidate pickup posture;
  • the target picking attitude information is obtained through the first convolutional network.
  • the structural information of the end effector is used to describe the structure of the end effector. For example, it may include information on the structure of the part of the end effector that is associated with the object when it picks up the object.
  • the structural information can be different for different types of end effectors.
  • a certain end effector is a suction cup.
  • the contact surface with the object is usually circular. Therefore, the structural information of the suction cup can include the radius or diameter of the contact part between the suction cup and the object. In this way, the suction cup can be The suction cup is equivalent to a circular flat area.
  • a certain end effector is a manipulator.
  • the manipulator picks up an object, it can use multiple end fingers on the manipulator to achieve semi-encircled or fully enclosed grasping. Therefore, the structural information of the manipulator can describe the end of the manipulator.
  • the structural information of the manipulator may include the size of the end clamps of the manipulator (such as the length, width, height, etc. of each end clamp), the distance between each end clamp, and other information.
  • the method of obtaining at least two sets of local point cloud data matching the end effector from the point cloud data based on the structural information is not limited here.
  • an equivalent shape can be constructed based on the structural information of the end effector of the pickup device to describe the corresponding end effector based on the equivalent shape.
  • local point cloud data can be obtained in the point cloud data through the equivalent shape.
  • the point cloud data can be converted into the form of a k-dimensional tree (kd-tree), and then the equivalent shape can be projected on the outline of the target object described by the kd-tree. If the projection area meets the specified conditions, then According to the point cloud corresponding to the projection area in the point cloud data, a set of local point cloud data is obtained.
  • kd-tree k-dimensional tree
  • the specified condition can be determined based on the actual application scenario and the type of the end effector.
  • the specified conditions may include constraints that satisfy force closure and shape closure.
  • shape closure refers to a state in which an object cannot change its posture due to a set of static constraints imposed on the surface of the object, and this set of static constraints is completely determined by the picking position of the end effector such as a manipulator. , then this state is called a shape-closed state, and this set of picking operations is called a shape-closed picking operation.
  • Force closure refers to a state in which an object cannot change its posture due to a set of static constraints imposed on the surface of the object, and this set of static constraints is entirely due to the end part of the end effector such as the fingers of the manipulator exerting on the surface of the object.
  • the force spiral at the contact point of the object is determined, then this state is called a force-closed state, and this set of picking operations is called a force-closed picking operation.
  • the specific calculation process based on constraints such as form closure and/or force closure may refer to existing and subsequently developed related technologies, which are not limited in the embodiments of the present application.
  • the relative pose between the equivalent shape and the point cloud data can be adjusted and projected.
  • the candidate picking up attitude corresponding to the set of local point cloud data can be used as the target picking up attitude. That is to say, the target picking up attitude information can be obtained based on the set of local point cloud data.
  • At least two sets of local point cloud data can be obtained, so that the target picking posture can be obtained through the first convolutional network based on at least two sets of local point cloud data. information.
  • the equivalent shape may be a two-dimensional shape.
  • the equivalent shape corresponding to the suction cup may be a circular planar area; or the equivalent shape may be a three-dimensional shape.
  • the end effector is a two-finger gripper.
  • an exemplary equivalent shape corresponding to the two-finger gripper is shown in Figure 4a.
  • the structural information of the two-finger gripper may include the length, width, and height of each gripper as well as the distance between the grippers.
  • the equivalent shape can be projected on the outline of the target object described in the point cloud data. If the If the projection area meets the specified conditions (for example, it satisfies the constraints of force closure and shape closure), then a set of local point cloud data can be obtained based on the point cloud corresponding to the projection area in the point cloud data.
  • the equivalent shape corresponding to the suction cup is a circular planar area.
  • the circular planar area can be projected on the point cloud corresponding to the outline of the target object. If the projection area on the point cloud corresponding to the outline of the target object meets the specified conditions, a set of local point cloud data is obtained based on the point cloud corresponding to the projection area in the point cloud data. For example, if the projection area on the point cloud corresponding to the outline of the target object is a circle, the center of the circle is used as a candidate pickup point, and a set of local point cloud data is obtained based on the information of the candidate pickup point.
  • the pickup device includes multiple types of end effectors.
  • the equivalent shapes corresponding to the multiple types of end effectors may be the same or different.
  • the pick-up device can include a single-tower suction cup and a multi-corrugated suction cup, and the size of the contact surfaces of these two suction cups when sucking objects can be the same. Therefore, the single-tower suction cup and the multi-corrugated suction cup are equivalent.
  • the shape can be the same. Therefore, each end effector in the pickup device can be described by this equivalent shape.
  • the candidate pickup postures described by the multiple sets of local point cloud data obtained can be applied to each suction cup in the pickup device. .
  • the equivalent shapes corresponding to the multiple types of end effectors included in the pickup device are different.
  • a set of outputs can be obtained through the neural network based on the structural information of each type of end effector.
  • the output is used to describe the corresponding end effector's pickup posture of the target object and the corresponding type.
  • the pickup postures and corresponding types of target objects of various end effectors can be obtained.
  • the appropriate end effector can be selected from the various end effectors based on the preset priority, control difficulty, control resource consumption and other information.
  • the end effector, and the appropriate end effector's pickup posture of the target object is used as the target pickup posture, and the corresponding type is used as the type of the target object.
  • Each set of local point cloud data may include information on at least one candidate pickup point to describe a corresponding candidate pickup gesture.
  • the at least one candidate picking point may be used to describe a candidate location for picking the target object through the end effector.
  • the end effector is a suction cup
  • the structural part of the suction cup that contacts the target object is a circular planar area, that is to say, the circular planar area can represent a candidate position for the suction cup to pick up the target object. Therefore, the at least one candidate pickup point may include the center of the corresponding circular planar area.
  • the candidate pickup gesture may be characterized by at least one candidate pickup point. And if the local point cloud data includes not only the information of the candidate picking points, but also the structural information of the end effector, the candidate picking posture can be described more comprehensively. At this time, the local point cloud data can not only describe a candidate pickup pose of the target object by the pickup device, but also include information such as the contact area when the pickup device picks up the target object.
  • the target picking attitude information can be obtained through the first convolutional network based on at least two sets of local point cloud data.
  • the first convolutional network can determine the target candidate pose according to the candidate pickup poses respectively corresponding to the multiple sets of local point cloud data, so as to output the target candidate pose information.
  • the form of the input data of the first convolutional network is not limited here.
  • the plurality of sets of local point cloud data may be described based on a specified data structure (such as a graph structure), and the specified data structure may be input into the first convolutional network.
  • the first convolutional network is a graph convolutional network
  • the graph structure is processed to obtain the target picking attitude information.
  • the graph structure may include nodes and edges.
  • each node in the graph structure can uniquely correspond to a local point cloud data. Specifically, if the local point cloud data is not filtered during the graph construction process, the nodes in the graph structure and the local point cloud data can correspond one to one.
  • the node information of each node in the graph structure may include corresponding local point cloud data, for example, may include three-dimensional coordinate information of the corresponding candidate pickup point and/or structural information of the corresponding end effector.
  • the edges between each node can be generated through random initialization or other methods. , at this time, the edges between each node in the graph structure cannot reflect the real correlation between each set of local point cloud data.
  • each set of local point cloud data is converted into a graph structure form through graph construction, so as to describe at least two sets of local point cloud data through structured data, which satisfies the requirements of the graph convolution network on the input data. formal requirements.
  • the connection relationship between nodes in the graph structure can be considered to be initialized and does not reflect the relationship between each group of local point cloud data. the real relationship between them.
  • the correlation between each set of local point cloud data can be determined during the processing of the graph structure by the graph convolution network.
  • the graph convolution network can determine the graph structure. The edges between nodes and the weight of each edge, and based on the edges between nodes and the weight of each edge in the graph structure, the target pickup posture is determined based on the candidate pickup posture corresponding to each set of local point cloud data.
  • the graph convolution network may include a first feature extraction network and a first classifier to extract semantic features from each local point cloud data through the first feature extraction network, and then, the first classifier extracts semantic features from each local point cloud data. The features are processed to output the target picking pose.
  • the local point cloud data may include information of corresponding candidate pickup points. In different stages of subsequent processing based on the information of the candidate picking points, different information of the candidate picking points can be applied.
  • the node information of the graph structure may include information about the three-dimensional coordinates of the corresponding candidate pickup points.
  • you can use the graph convolution network The graph structure is processed to output the three-dimensional coordinates of the target pickup point.
  • the 6D pickup pose of the target object by the pickup device can be determined based on the normal vector and three-dimensional coordinates of the target pickup point in the point cloud data.
  • the normal vector and three-dimensional coordinates of the target pickup point in the coordinate system corresponding to the point cloud data can be converted into the normal vector and three-dimensional coordinates in the coordinate system corresponding to the pickup device, and then the target is determined according to the coordinate system corresponding to the pickup device.
  • the three-dimensional coordinates and normal vector of the pickup point determine the target pickup pose of the target object by the pickup device.
  • FIG. 5 it is an exemplary schematic diagram for obtaining the target picking posture of the target object.
  • multiple sets of local point cloud data can be obtained based on the point cloud data and the structural information of the end effector, and then graph construction is performed on the multiple sets of local point cloud data to obtain the corresponding graph structure, and Input the graph structure into the graph convolution network to obtain the target picking attitude information output by the graph convolution network.
  • multiple sets of local point cloud data can be effectively integrated through the graph structure, so that the target pickup posture information can be obtained through the graph convolution network based on multiple candidate pickup postures described in the graph structure.
  • the first convolutional network includes a first feature extraction network and a first classifier, and the first feature extraction network is located before the first classifier;
  • the type information of the target object is obtained according to the first feature information output by the first feature extraction network when the first convolutional network processes the point cloud data, and the point cloud data.
  • the first feature extraction network is used to extract features from the corresponding input (such as graph structure).
  • the number and type of layers included in the first feature extraction network are not limited here.
  • the first feature extraction network may include one or more of a convolution layer, an activation layer, a pooling layer, and the like.
  • the first classifier is used to classify the corresponding input (such as the feature information output by the previous layer of the first classifier) and obtain the corresponding classification result. If the first classifier is the last layer of the first convolutional network, the output of the first classifier can be the target pickup posture information.
  • the number and type of layers included in the first classifier are not limited here.
  • the first classifier may include a fully connected layer.
  • the specific positional relationship between the first feature extraction network and the first classifier in the first convolutional network is not specifically limited here.
  • the first feature extraction network is the previous layer network of the first classifier, and the output of the first feature extraction network is the input of the first classifier; and in another example, the first feature extraction network Other layers can also be included between the first classifier and the first classifier.
  • the first convolutional network may also include other structures besides the first feature extraction network and the first classifier.
  • the input of the second convolutional network includes the output of the first feature extraction network when the first convolutional network processes the point cloud data, and the point cloud data.
  • the first convolutional network in the process of determining the target pickup posture of the target object by the pickup device, the first convolutional network often needs to extract the local features corresponding to the target pickup posture. That is to say, the first convolutional network is based on The first feature information output by the first feature extraction network when processing point cloud data may include local feature information corresponding to the target pickup posture.
  • the type of the target object is not only identified based on the global point cloud data, but also combined with the local feature information contained in the first feature information to identify the type of the target object.
  • the second convolutional network there can be multiple ways of combining point cloud data and first feature information.
  • the first feature information output by the first feature extraction network is the intermediate layer output of the first convolutional network, it is different from the point cloud number.
  • the data formats and corresponding semantics of data are usually quite different, so they usually cannot be combined directly.
  • the point cloud data and the first feature information may not be directly combined, but the point cloud data may be used as the input of the second convolutional neural network, and during the processing of the point cloud data by the second convolutional network, in the third An intermediate layer of the second convolutional network combines the output of the previous layer of the intermediate layer with the first feature information output by the first feature extraction network and uses it as input for processing, so that in the intermediate layer and subsequent layers, the combination
  • the characteristics of the global point cloud data and the local feature information extracted by the first feature extraction network are processed to combine with the more comprehensive feature information to achieve accurate identification of the type of the target object.
  • the second convolutional network includes a second feature extraction network and a second classifier
  • the aggregation result is processed through the second classifier to obtain the type information of the target object.
  • the second convolutional network can process point cloud data.
  • the second convolutional network can be a PointNet network or other open source network that can process point cloud data, or it can also be a network developed by researchers themselves. .
  • the second feature extraction network is used to extract features from the point cloud data to obtain global feature information about the point cloud data.
  • the number and type of layers included in the second feature extraction network are not limited here.
  • the second feature extraction network may include one or more of a convolution layer, an activation layer, a pooling layer, and the like.
  • the second classifier is used to classify the corresponding input (for example, the feature information output by the previous layer of the second classifier) to obtain the corresponding classification result. If the second classifier is the last layer of the second convolutional network, the output of the second classifier may be the type information of the target object.
  • the number and type of layers included in the second classifier are not limited here.
  • the second classifier may include a fully connected layer.
  • the second convolutional network may also include other structures besides the second feature extraction network and the second classifier, which are not limited in the embodiments of the present application.
  • the second feature extraction network may be a layer network before the second classifier.
  • the input of the second classifier is the aggregation result of the first feature information and the second feature information.
  • the first feature information and the second feature information may be spliced to obtain an aggregation result; or the first feature information and the second feature information may be multiplied by their corresponding weights, and then spliced to obtain the aggregation result.
  • the aggregation result is obtained; or, the corresponding elements of the first feature information and the second feature information can be added together to obtain the aggregation result.
  • the second feature information usually includes global feature information of the target object in the point cloud data.
  • the first feature information may be feature information obtained based on multiple sets of local point cloud data.
  • the first feature information usually includes local feature information of the target object in the point cloud data. It can be seen that in the embodiment of the present application, the second classifier can aggregate the local feature information and global feature information of the target object to determine the type of the target object, thereby accurately classifying the target object based on the relatively complete information.
  • FIG. 6 it is an exemplary diagram of a neural network.
  • multiple sets of local point cloud data can be obtained based on the point cloud data and the structural information of the end effector, and then the multiple sets of local point cloud data can be graph constructed to obtain the corresponding graph structure, and the graph structure can be input to the third In a convolutional network, the target picking posture information output by the first convolutional network is obtained.
  • the first convolutional network specifically includes a first feature extraction network and a first classifier, and the output of the first feature extraction network regarding the graph structure is first feature information, and the first feature information may include information about the target object. local feature information.
  • the point cloud data can be input into the second feature extraction network of the second convolutional network to obtain second feature information, which can include global feature information about the target object.
  • the first feature information and the second feature information are aggregated to obtain an aggregation result; and then the aggregation result is processed by the second classifier to obtain the type information of the target object.
  • the neural network may be a trained model.
  • the following is an exemplary introduction to the training process of the first convolutional network and the second convolutional network.
  • a training data set can be constructed in advance.
  • Each training sample in the training data set includes a set of training point cloud data, and each training sample corresponds to a picking posture label and a type label.
  • the training point cloud data can be obtained by collecting a depth image through a depth camera and converting it based on information such as the depth image and internal parameters of the depth camera.
  • the picking posture label may be obtained based on mechanical criteria such as shape closure and/or force closure.
  • the specific calculation process of obtaining the picking posture label based on mechanical criteria such as shape closure and/or force closure may refer to existing and subsequently developed related technologies, which is not limited in the embodiments of the present application.
  • This type of label can be obtained through manual annotation or other methods.
  • the neural network After obtaining the training data set, the neural network can be trained end-to-end based on the training data set.
  • the following processing process is called a designated function: according to the structural information of the end effector, obtain at least two sets of local training point cloud data matching the end effector from the training point cloud data, and then obtain at least two sets of local training point cloud data according to the Cloud data,processing process to obtain training graph structure. Then, in each iteration process, the training point cloud data can be used as the input of the second convolutional network to be trained, and the training point cloud data can be processed by the specified function to obtain the training graph structure, and then the training graph structure can be As input to the first convolutional network to be trained.
  • the neural network to be trained can be trained through the back propagation algorithm, and after the neural network to be trained converges, the above-mentioned neural network is obtained.
  • the neural network may be trained in the computer device that executes the embodiments of the present application, or may be transferred and deployed to the computer device after the training of other devices is completed.
  • the embodiments of the present application have introduced the object picking up method from many aspects.
  • the object picking up device 70 of the present application will be introduced below with reference to FIG. 7 .
  • the object picking up device 70 can be applied to the above computer equipment.
  • the object pickup device 70 may include:
  • Acquisition module 701 is used to obtain point cloud data about the target object
  • the processing module 702 is used to obtain the target pickup posture information and the type information of the target object obtained by processing the point cloud data by the neural network.
  • the target pickup posture information is used to describe the target pickup posture of the target object by the pickup device;
  • the control module 703 is used to control the picking device to pick up the target object according to the target picking posture information and type information.
  • the picking device includes multiple types of end effectors, and different types of end effectors are used to pick up different types of objects;
  • Control module 703 is used for:
  • the target pickup mode is used to indicate the end effector used to pick up the target object
  • the pickup device is controlled to pick up the target object based on the target pickup attitude indicated by the target pickup attitude information in the target pickup mode.
  • the neural network includes a first convolutional network and a second convolutional network
  • Processing module 702 is used to:
  • the target picking attitude information is obtained
  • the type information of the target object is obtained based on the point cloud data.
  • processing module 702 is used to:
  • At least two sets of local point cloud data matching the end effector are obtained from the point cloud data, and each set of local point cloud data corresponds to a candidate pickup posture;
  • the target picking attitude information is obtained through the first convolutional network.
  • the first convolutional network is a graph convolutional network
  • Processing module 702 is used to:
  • the graph structure is processed to obtain the target picking attitude information.
  • the first convolutional network includes a first feature extraction network and a first classifier, and the first feature extraction network is located before the first classifier;
  • Processing module 702 is used to:
  • the type information of the target object is obtained according to the first feature information output by the first feature extraction network when the first convolutional network processes the point cloud data, and the point cloud data.
  • the second convolutional network includes a second feature extraction network and a second classifier
  • Processing module 702 is used to:
  • the aggregation result is processed through the second classifier to obtain the type information of the target object.
  • FIG. 8 is a schematic diagram of a possible logical structure of the computer device 80 provided by the embodiment of the present application.
  • the computer device 80 is used to implement the functions of the computer device involved in any of the above embodiments.
  • the computer device 80 includes: a memory 801, a processor 802, a communication interface 803 and a bus 804. Among them, the memory 801, the processor 802, and the communication interface 803 implement communication connections between each other through the bus 804.
  • the memory 801 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 801 can store a program. When the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to execute steps 201-203 and so on of the above-mentioned object picking method embodiment.
  • the processor 802 may adopt a central processing unit (CPU), a microprocessor, a specific Application-specific integrated circuit (ASIC), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA) Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components or any combination thereof, used to execute relevant programs to implement the acquisition module, processing module and control module required in the object pickup device of the above embodiments. Functions performed, or steps 201-203, etc. of the object picking method embodiment of the method embodiment of the present application. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • CPU central processing unit
  • ASIC Application-specific integrated circuit
  • GPU graphics processing unit
  • DSP digital signal processor
  • FPGA field-programmable gate array
  • the software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field.
  • the storage medium is located in the memory 801.
  • the processor 802 reads the information in the memory 801, and performs steps 201-203 and so on of the above object picking method embodiment in conjunction with its hardware.
  • the communication interface 803 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the computer device 80 and other devices or communication networks.
  • a transceiver device such as, but not limited to, a transceiver to implement communication between the computer device 80 and other devices or communication networks.
  • Bus 804 may implement a path for transferring information between various components of computer device 80 (eg, memory 801, processor 802, and communication interface 803).
  • the bus 804 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc.
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • the bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in Figure 8, but it does not mean that there is only one bus or one type of bus.
  • a computer-readable storage medium is also provided.
  • Computer-executable instructions are stored in the computer-readable storage medium.
  • the processor of the device executes the computer-executable instructions
  • the device executes the above-mentioned steps in Figure 8 The steps performed by the processor.
  • a computer program product includes computer-executable instructions, and the computer-executable instructions are stored in a computer-readable storage medium; when the processor of the device executes the computer-executed instructions When , the device performs the steps performed by the processor in Figure 8 above.
  • a chip system in another embodiment, is also provided.
  • the chip system includes a processor, and the processor is configured to implement the steps performed by the processor in FIG. 8 .
  • the chip system may also include a memory, a memory, a device for storing data writing, necessary program instructions and data.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • the disclosed systems, devices and methods can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of units is only a logical function division.
  • the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
  • a unit described as a separate component may or may not be physically separate.
  • a component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.
  • Functions may be stored in a computer-readable storage medium when implemented in the form of software functional units and sold or used as independent products.
  • the technical solutions of the embodiments of the present application are essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the embodiments of this application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Health & Medical Sciences (AREA)
  • Orthopedic Medicine & Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Fuzzy Systems (AREA)
  • Artificial Intelligence (AREA)
  • Physics & Mathematics (AREA)
  • Image Analysis (AREA)

Abstract

一种物体拾取方法，包括：获取关于目标物体的点云数据；获得神经网络对点云数据处理而得到的目标拾取姿态信息和目标物体的类型信息，目标拾取姿态信息用于描述拾取装置对目标物体的目标拾取姿态；根据目标拾取姿态信息和类型信息，控制拾取装置拾取目标物体。通过该方法，可以结合神经网络对不同类型等不同情况的物体进行自适应地拾取。还提供一种物体拾取设备、计算机设备及计算机可读存储介质。

Description

一种物体拾取方法以及相关设备
本申请要求于2022年08月11日提交的、中国申请号为CN202210962888.1、发明名称为“一种物体拾取方法以及相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及自动化控制技术领域，具体涉及一种物体拾取方法以及相关设备。
背景技术
随着自动化控制技术的不断发展，诸如机器人等自动化控制技术在货物搬运、货物传送、智能制造、智慧医疗等领域都有着广泛应用。
其中，自主拾取技术是机器人等自动化控制的基础能力。而在诸如货物搬运等场景中，需要拾取的物体的情况可能有多种。因此，亟需一种可以对不同类型、尺寸等不同情况的物体进行自适应地拾取的方法。
发明内容
本申请提供一种物体拾取方法，以解决在物体种类、形态较多的场景中，难以实现对不同情况的物体的自适应拾取，从而难以满足实际应用场景的要求的问题。本申请还提供了相应的装置、设备、计算机可读存储介质和计算机程序产品等。
本申请第一方面提供一种物体拾取方法，该方法中，可以获取关于目标物体的点云数据；获得神经网络对所述点云数据处理而得到的目标拾取姿态信息和所述目标物体的类型信息，所述目标拾取姿态信息用于描述拾取装置对目标物体的目标拾取姿态；根据所述目标拾取姿态信息和所述类型信息，控制所述拾取装置拾取所述目标物体。
在第一方面中，神经网络可以基于关于目标物体的点云数据，获得目标拾取姿态信息和目标物体的类型信息，从而可以针对目标物体的情况，适应性地以合适的拾取方式和拾取姿态对目标物体进行拾取，以成功执行拾取任务。
在第一方面的一种可能的实现方式中，所述拾取装置包括多种类型的末端执行器，不同类型的末端执行器用于拾取不同类型的物体；上述步骤：根据所述目标拾取姿态信息和所述类型信息，控制拾取装置拾取所述目标物体，包括：根据所述类型信息，确定所述拾取装置的目标拾取模式，所述目标拾取模式用于指示对所述目标物体进行拾取所采用的末端执行器；控制所述拾取装置在所述目标拾取模式下，基于所述目标拾取姿态信息所指示的目标拾取姿态，拾取所述目标物体。
该种可能的实现方式中，拾取装置中设置有多种类型的末端执行器，从而可以根据待拾取的物体(如目标物体)的类型选择合适的末端执行器来进行拾取，从而可以基于实际场景中待拾取的物体的类型，适应性地选择较为合适的拾取模式来进行拾取。此外，还可以通过神经网络确定准确的拾取姿态，这样，可以基于实际场景，自适应地控制拾取装置以合适的拾取模式以及准确的姿态对目标物体进行拾取，实现了软件与硬件相结合的协同控制，可以对于各种情况的物体进行成功拾取，自适应地完成场景较为复杂的拾取任务。
在第一方面的一种可能的实现方式中，所述神经网络包括第一卷积网络和第二卷积网 络；上述步骤：获得神经网络对所述点云数据处理而得到的目标拾取姿态信息和所述目标物体的类型信息，包括：通过所述第一卷积网络，基于所述点云数据，获得所述目标拾取姿态信息；通过所述第二卷积网络，基于所述点云数据，获得所述目标物体的类型信息。
该种可能的实现方式中，本申请实施例中，第一卷积网络和第二卷积网络均可以包含卷积运算，并且并不限定第一卷积网络和第二卷积网络中的卷积运算中的具体方式。例如，神经网络中的卷积运算可以包括对矩阵的卷积运算、对图结构的图卷积运算、对点云数据的卷积运算等中的一种或多种。针对不同的输入数据形式，对应的卷积运算的具体方式可能存在差异，例如，对图结构的图卷积运算和对点云数据的卷积运算的具体方式可能不同，可以参考目前或后续发展的相关技术来实现。
该种可能的实现方式中，通过两个分支结构(即第一卷积网络和第二卷积网络)来分别获得目标拾取姿态信息和目标物体的类型信息，从而基于较为全面的信息来评估对目标物体的拾取方式，以针对各种情况的物体，自适应地选择合适的方式进行拾取，从而可以满足多种应用场景的需要。
在第一方面的一种可能的实现方式中，上述步骤：通过所述第一卷积网络，基于所述点云数据，获得所述目标拾取姿态信息，包括：根据所述拾取装置的末端执行器的结构信息，从所述点云数据中获得与所述末端执行器相匹配的至少两组局部点云数据，每组局部点云数据对应一个候选拾取姿态；根据所述至少两组局部点云数据，通过所述第一卷积网络，获得目标拾取姿态信息。
该种可能的实现方式中，每组局部点云数据可以包括至少一个候选拾取点的信息，以描述对应的一个候选拾取姿态。该至少一个候选拾取点可以用于描述通过末端执行器对目标物体进行拾取的候选位置。此时，可以通过至少一个候选拾取点表征候选拾取姿态。而若该局部点云数据中不仅包括候选拾取点的信息，还包括末端执行器的结构信息，则可以更为全面地描述该候选拾取姿态。此时，局部点云数据不仅可以描述拾取装置对目标物体的一种候选的拾取位姿，还可以包含该拾取装置对该目标物体进行拾取时的接触区域等信息。每组局部点云数据对应一个候选拾取姿态，以便于第一卷积网络根据多组局部点云数据分别对应的候选拾取姿态而确定目标候选姿态，以输出目标候选姿态信息。
在第一方面的一种可能的实现方式中，所述第一卷积网络为图卷积网络；上述步骤：根据所述至少两组局部点云数据，通过所述第一卷积网络，获得目标拾取姿态信息，包括：根据所述至少两组局部点云数据，获得图结构，所述图结构中的每个节点对应一个局部点云数据；通过所述第一卷积网络对所述图结构处理，获得目标拾取姿态信息。
该种可能的实现方式中，通过图构建的方式将各组局部点云数据转换为图结构的形式，以通过结构化数据来描述至少两组局部点云数据，满足图卷积网络对输入数据的形式要求。可见，通过图结构可以有效整合多组局部点云数据，从而通过图卷积网络，根据图结构中描述的多个候选拾取姿态，获得目标拾取姿态信息。
在第一方面的一种可能的实现方式中，所述第一卷积网络包括第一特征提取网络和第一分类器，所述第一特征提取网络位于所述第一分类器之前；上述步骤：通过所述第二卷积网络，基于所述点云数据，获得所述目标物体的类型信息，包括：通过所述第二卷积网络，根据所述第一卷积网络基于所述点云数据进行处理时所述第一特征提取网络输出的第 一特征信息，以及所述点云数据，获得所述目标物体的类型信息。
该种可能的实现方式中，第一卷积网络在确定拾取装置对目标物体的目标拾取姿态的过程中，往往需要提取该目标拾取姿态对应的局部特征，也即是说，第一卷积网络中基于点云数据进行处理时的第一特征提取网络输出的第一特征信息可以包含目标拾取姿态对应的局部特征信息。基于此，该种可能的实现方式中，不仅根据全局的点云数据来识别目标物体的类型，还结合第一特征信息中包含的局部特征信息来识别目标物体的类型。
在第一方面的一种可能的实现方式中，所述第二卷积网络包括第二特征提取网络和第二分类器；上述步骤：通过所述第二卷积网络，根据所述第一卷积网络基于所述点云数据进行处理时所述第一特征提取网络输出的第一特征信息，以及所述点云数据，获得所述目标物体的类型信息，包括：通过所述第二特征提取网络对所述点云数据进行处理，获得第二特征信息；将所述第一特征信息与所述第二特征信息进行聚合，获得聚合结果；通过第二分类器对所述聚合结果进行处理，获得所述目标物体的类型信息。
该种可能的实现方式中，第二特征信息通常包括点云数据中目标物体的全局特征信息。而第一特征信息可以是基于多组局部点云数据而得到的特征信息，此时，第一特征信息通常包括点云数据中目标物体的局部特征信息。可见，该种可能的实现方式中，第二分类器可以聚合目标物体的局部特征信息和全局特征信息来确定目标物体的类型，从而结合较为完整的信息，对目标物体进行准确分类。
本申请第二方面提供一种物体拾取装置，应用于计算机设备，该装置具有实现上述第一方面或第一方面任意一种可能实现方式的方法的功能。该功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块，例如：获取模块、处理模块以及控制模块。
本申请第三方面提供一种计算机设备，该计算机设备包括至少一个处理器、存储器以及存储在存储器中并可在处理器上运行的计算机执行指令，当计算机执行指令被处理器执行时，处理器执行如上述第一方面或第一方面任意一种可能的实现方式的方法。
本申请第四方面提供一种存储一个或多个计算机执行指令的计算机可读存储介质，当计算机执行指令被处理器执行时，处理器执行如上述第一方面或第一方面任意一种可能的实现方式的方法。
本申请第五方面提供一种存储一个或多个计算机执行指令的计算机程序产品，当计算机执行指令被处理器执行时，处理器执行如上述第一方面或第一方面任意一种可能的实现方式的方法。
本申请第六方面提供了一种芯片系统，该芯片系统包括处理器，用于支持终端实现上述第一方面或第一方面任意一种可能的实现方式中所涉及的功能。在一种可能的设计中，芯片系统还可以包括存储器，存储器用于保存计算机设备必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包含芯片和其他分立器件。
其中，第二方面至第六方面或者其中任一种可能实现方式所带来的技术效果可参见第一方面或第一方面的相关可能实现方式所带来的技术效果，此处不再赘述。
附图说明
图1是本申请实施例提供的一种人工智能主体框架示意图；
图2是本申请实施例提供的物体拾取方法的一示例性示意图；
图3是本申请实施例提供的一种示例性的控制流程示意图；
图4a是本申请实施例提供的二指夹爪对应的等效形状的一种示例性示意图；
图4b是本申请实施例提供的获得局部点云数据的一种示例性示意图；
图5是本申请实施例提供的获得目标物体的目标拾取姿态的一种示例性示意图；
图6是本申请实施例提供的神经网络的一种示例性示例图；
图7是本申请实施例提供的物体拾取装置的一实施例示意图；
图8是本申请实施例提供的计算机设备的一结构示意图。
具体实施方式
下面结合本申请实施例中的附图对本申请实施例进行描述。本申请的实施方式部分使用的术语仅用于对本申请的具体实施例进行解释，而非旨在限定本申请。
本领域普通技术人员可知，随着技术的发展和新场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
本申请中，“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B的情况，其中A,B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项”或其类似表达，是指的这些项中的任意组合，包括单项或复数项的任意组合。本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的术语在适当情况下可以互换，这仅仅是描述本申请的实施例中对相同属性的对象在描述时所采用的区分方式。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，以便包含一系列单元的过程、方法、系统、产品或设备不必限于那些单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它单元。
如背景技术中所描述的，在货物搬运、货物传送、智能制造、智慧医疗等领域，常常需要通过机器人等自动控制的机械设备来完成对物品的自主拾取任务。
其中，拾取包括抓取或者吸取等。采用不同的末端执行器时，搬运物体的方式可以不同。
在一种示例中，末端执行器可以为机械手，此时可以通过机械手抓取物体。在另一种示例中，末端执行器可以为吸盘，此时可以通过吸盘吸取物体。当然，末端执行器还可以有其他结构形式，本申请实施例中对此不做限制。在又一种示例中，末端执行器为磁吸式末端执行器，此时，可以通过磁吸的方式吸取物体。
然而，目前，机械设备通常固定地用于对一种物体进行拾取，而不能适用于对种类、形态较多的不同物体进行拾取。
可见，目前，在物体种类、形态较多的场景中，难以实现对不同情况的物体的自适应 拾取，从而难以满足实际应用场景的要求。
基于上述问题，本申请实施例中提供了一种物体拾取方法，可以结合神经网络对不同类型等不同情况的物体进行自适应地拾取。
由于本申请涉及人工智能，下面首先对人工智能进行介绍。
人工智能(Artificial Intelligence,AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。人工智能领域的研究包括机器人，自然语言处理，计算机视觉，决策与推理，人机交互，推荐与搜索，AI基础理论等。
为了便于理解，下面先对本申请实施例涉及的相关术语及神经网络等相关概念进行介绍。
请参见图1，图1示出的为人工智能主体框架的一种结构示意图，下面从“智能信息链”(水平轴)和“IT价值链”(垂直轴)两个维度对上述人工智能主题框架进行阐述。其中，“智能信息链”反映从数据的获取到处理的一列过程。举例来说，可以是智能信息感知、智能信息表示与形成、智能推理、智能决策、智能执行与输出的一般过程。在这个过程中，数据经历了“数据—信息—知识—智慧”的凝练过程。“IT价值链”从人智能的底层基础设施、信息(提供和处理技术实现)到系统的产业生态过程，反映人工智能为信息技术产业带来的价值。
(1)基础设施
基础设施为人工智能系统提供计算能力支持，实现与外部世界的沟通，并通过基础平台实现支撑。通过传感器与外部沟通；计算能力由智能芯片(中央处理器(central processing unit，CPU)、神经网络处理器(neural-network processing unit，NPU)、图形处理器(graphics processing unit，GPU)、专用集成电路(application specific integrated circuit，ASIC)、现场可编程逻辑门阵列(field programmable gate array，FPGA)、张量处理器(tensor processing unit，TPU)等硬件加速芯片)提供；基础平台包括分布式计算框架及网络等相关的平台保障和支持，可以包括云存储和计算、互联互通网络等。举例来说，传感器和外部沟通获取数据，这些数据提供给基础平台提供的分布式计算系统中的智能芯片进行计算。
(2)数据
基础设施的上一层的数据用于表示人工智能领域的数据来源。数据涉及到图形、图像、语音、文本，还涉及到传统设备的物联网数据，包括已有系统的业务数据以及力、位移、液位、温度、湿度等感知数据。
(3)数据处理
数据处理通常包括数据训练，机器学习，深度学习，搜索，推理，决策等方式。
其中，机器学习和深度学习可以对数据进行符号化和形式化的智能信息建模、抽取、预处理、训练等。
推理是指在计算机或智能系统中，模拟人类的智能推理方式，依据推理控制策略，利用形式化的信息进行机器思维和求解问题的过程，典型的功能是搜索与匹配。
决策是指智能信息经过推理后进行决策的过程，通常提供分类、排序、预测等功能。
(4)通用能力
对数据经过上面提到的数据处理后，进一步基于数据处理的结果可以形成一些通用的能力，比如可以是算法或者一个通用系统，例如，翻译，文本的分析，计算机视觉的处理，语音识别，图像的识别等等。
(5)智能产品及行业应用
智能产品及行业应用指人工智能系统在各领域的产品和应用，是对人工智能整体解决方案的封装，将智能信息决策产品化、实现落地应用，其应用领域主要包括：智能终端、智能交通、智能医疗、自动驾驶、智慧城市等。
(6)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以xs(即输入数据)和截距1为输入的运算单元，该运算单元的输出可以为：
其中，s＝1、2、……n，n为大于1的自然数，Ws为xs的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
神经网络有多种类型，例如，深度神经网络(deep neural network，DNN)，也称多层神经网络，也就是具有多层隐含层的神经网络；再例如，卷积神经网络(convolutional neural network，CNN)，是带有卷积结构的深度神经网络。本申请不限定涉及的神经网络的具体类型。
(7)深度神经网络
深度神经网络(Deep Neural Network，DNN)，可以理解为具有很多层隐含层的神经网络，这里的“很多”并没有特别的度量标准，我们常说的多层神经网络和深度神经网络其本质上是同一个东西。从DNN按不同层的位置划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。简单来说每一层的工作可以理解为如下线性关系表达式：其中，是输入向量，是输出向量，是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量经过如此简单的操作得到输出向量由于DNN层数多，则系数W和偏移向量的数量也会很多。下面介绍具体的参数在DNN是如何定义的。首先介绍系数W的定义。以一个三层的DNN为例，如：第二层的第4个神经元到第三层的第2个神经元的线性系数定义为上标3代表系数W所在的层数，而下标对 应的是输出的第三层索引2和输入的第二层索引4。总结下，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为注意，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。
(8)卷积神经网络(Convosutionas Neuras Network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器。该特征抽取器可以看作是滤波器，卷积过程可以看作是使用一个可训练的滤波器与一个输入的图像或者卷积特征平面(feature map)做卷积。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。这其中隐含的原理是：图像的某一部分的统计信息与其他部分是一样的。即意味着在某一部分学习的图像信息也能用在另一部分上。所以对于图像上的所有位置，我们都能使用同样的学习得到的图像信息。在同一卷积层中，可以使用多个卷积核来提取不同的图像信息，一般地，卷积核数量越多，卷积操作反映的图像信息越丰富。
卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。
(9)反向传播算法
卷积神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的超分辨率模型中参数的大小，使得超分辨率模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的超分辨率模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的超分辨率模型的参数，例如权重矩阵。
本申请实施例的物体拾取方法可以应用于计算机设备，在一些示例中，该计算机设备可以作为拾取装置的上位机，以用于控制拾取装置。
其中，该计算机设备的具体类型可以有多种，在此不做限制。
示例性地，该计算机设备可以是终端设备、服务器、容器(container)或者虚拟机(virtual machine)。
在一种示例中，该计算机设备可以为终端设备。示例性地，该终端设备可以是手机(mobile phone)、平板电脑(pad)、带无线收发功能的电脑、虚拟现实(virtual reality，VR)终端、增强现实(augmented reality，AR)终端、工业控制(industrial control)中的终端、无人驾驶(self driving)中的终端、远程医疗(remote medical)中的终端、智能电网(smart grid)中的终端、运输安全(transportation safety)中的终端、智慧城市(smart city)中的终端、智慧家庭(smart home)中的终端、以及物联网(Internet of things，IoT)中的终端等。
当计算机设备为终端设备时，该终端设备可以直接或者间接地控制拾取装置进行运动。
例如，该终端设备可以直接向拾取装置发送指示信号，以指示拾取装置进行运动。或者，该终端设备与拾取装置之间还可以包括其他器件(例如电机等)，此时，该终端设备可以通过控制电机、电磁阀等器件来驱动拾取装置中的末端执行器进行运动。
在另一种示例中，该计算机设备可以为云端设备，例如可以为云端的服务器或者服务器集群，或者云端的容器或者虚拟机等。
此时，云端设备中可以部署有神经网络，该云端设备可以执行本申请实施例，以获得用于控制拾取装置的控制信息，再将该控制信息发送给拾取装置，以控制拾取装置的运动，或者，也可以将该控制信息发送给拾取装置对应的本地的终端设备，以通过该本地的终端设备，控制拾取装置的运动。
拾取装置可以包括末端执行器。
该拾取装置的结构和类型可以基于实际应用场景来确定，本申请实施例中对此不做限定。
示例性地，该拾取装置中的可以包括多种类型的末端执行器，不同的末端执行器用于拾取不同类型的物体。
需要说明的是，拾取装置中，同一类型的末端执行器的数量可以是一个或者多个。
例如，在一种示例中，拾取装置中可以包括多个吸盘，其中，多个吸盘可以各自独立地工作以吸取不同的物体，也可以用于对同一个物体进行吸取。
该拾取装置中，末端执行器的排布方式可以有多种。示例性地，多个末端执行器可以阵列式地排布于拾取装置中，也可以不规则地排布于拾取装置中。此外，在一些示例中，该多个末端执行器在拾取装置上的排布方式还可以基于应用场景的变化而变化。
本申请实施例中，计算机设备和拾取装置可以位于同一物理设备中，例如，位于同一机器人上，此时，计算机设备和拾取装置可以为该机器人上的模块；或者，也可以位于不同物理设备中，例如，计算机设备位于云端，而拾取装置位于本地的机器人上，此时，计算机设备可以通过通信连接，实现对拾取装置的控制。
如图2所示，本申请实施例中，结合神经网络而实现的物体拾取方法包括步骤201-203。
步骤201，获取关于目标物体的点云数据。
本申请实施例中，该目标物体可以为待拾取的物体。该目标物体的类型、尺寸和/或材质可以有多种。在此不做限制。
示例性地，在货物搬运的应用场景中，该目标物体的材质可以为包裹有货物的纸箱、纸袋、泡沫袋、泡沫箱、塑料袋等中的一种或多种。通常来说，在实际应用中对物体进行拾取时采用的是非破坏式的拾取方式，因此，在许多示例中，通常将目标物体表面的材质作为目标物体的材质。
关于目标物体的点云数据可以指包含关于目标物体的特征点的点云数据。在一些示例中，该点云数据也可以认为是目标物体所在场景的场景点云数据，或者是对该场景点云数据进行诸如滤波、下采样、基于平面方程进行平面移除等等操作，以删除场景点云数据中一些对后续处理没有帮助的点云，从而获得关于目标物体的点云数据。
获取关于目标物体的点云数据的方式可以有多种。
示例性地，该点云数据可以是执行本申请实施例的计算机设备中预先存储的，或者， 也可以是通过其他设备传输至该计算机设备中的，或者，也可以是基于计算机设备中的摄像头等传感设备采集到的数据而通过计算机设备生成的。
示例性地，该点云数据可以是基于深度相机或者激光雷达等采集到的数据而生成。
例如，一种生成点云数据的示例性方式可以包括以下步骤：
获取包含目标物体的场景的深度图像；
根据深度图像，获得场景对应的场景点云数据；
对场景点云数据进行预处理，获得关于目标物体的点云数据。
该深度图像的获取方式可以有多种。例如，可以通过深度相机、激光雷达或者结构光获得该深度图像。在获得该深度图像之后，可以根据该深度图像获得场景点云数据。
其中，根据深度图像，获得场景对应的场景点云数据的方式可以基于相关技术而实现，本申请实施例中对此不做限定。
示例性地，深度图像通过深度相机采集而得到。此时，根据深度图像的信息和深度相机的内部参数，可以计算出深度图像中的像素点在深度相机的相机坐标系下的坐标。而根据深度图像的信息和深度相机的内部参数与外部参数，可以计算出深度图像中的像素点在世界坐标系下的坐标。然后，根据深度图像中的像素点在世界坐标系下的坐标，可以获得场景点云数据。
在获得场景点云数据之后，可以对场景点云数据进行预处理，例如，可以减少噪声以及一些对后续处理无帮助的数据信息，以减少干扰信息，使得获得的点云数据更有利于后续的数据处理操作。示例性地，该预处理可以包括诸如滤波、下采样、基于平面方程进行背景平面移除等等处理操作，以删除场景点云数据中一些对后续处理没有帮助的点云信息(例如场景点云数据中的一些噪声和/或背景点云信息)，从而获得关于目标物体的点云数据。此外，该预处理中还可以包括均匀采样操作(例如，预处理的最后一步为均匀采样操作)，这样，可以通过均匀采样来降低场景点云数据的稠密程度，减小数据量，以降低后续的数据处理量，减小对计算资源的消耗。在一些示例中，在预处理之后，该点云数据中可以仅包括关于目标物体的特征点，这样，该点云数据可以描述目标物体的轮廓，以为后续的类型识别以及确定目标拾取姿态的处理操作提供较好的数据基础，减少干扰数据的影响，提高后续处理结果的准确性。
步骤202，获得神经网络对点云数据处理而得到的目标拾取姿态信息和目标物体的类型信息。
目标拾取姿态信息用于描述拾取装置对目标物体的目标拾取姿态。
在获得点云数据之后，可以通过神经网络对该点云数据进行处理，以获得目标拾取姿态信息和目标物体的类型信息。
神经网络对该点云数据进行处理的方式可以有多种。
例如，该点云数据可以直接输入神经网络，以通过神经网络进行处理；此时，神经网络中包括可以对点云数据进行处理的神经网络结构，例如可以包括Pointnet等网络。此外，该点云数据还可以通过指定的处理之后，再输入神经网络进行处理，例如，该神经网络中包括图卷积网络，则可以对该点云数据进行处理以获得相应的图结构，再将该图结构作为该图卷积网络的输入。具体来说，在一种示例中，可以根据末端执行器的参数对该点云数 据进行搜索匹配，从点云数据中获得用于表征候选拾取姿态的局部点云数据，并根据各组局部点云数据，生成图结构，这样，图卷积网络可以根据该图结构中描述的各个候选拾取姿态的信息而确定目标拾取姿态。
神经网络的具体结构可以有多种，本申请实施例中对此不做限定。
在一些示例中，该神经网络中可以包括双分支结构，该双分支结构中的一个分支结构的输出为目标拾取姿态信息，而另一个分支结构的输出为目标物体的类型信息。
该双分支结构中，所包含的两个分支结构可以是两个并行的结构，也可以是存在一部分共享的结构，还存在一部分并行的结构。此外，该双分支结构中，所包含的两个分支结构的数据可以相互独立地执行，也可以存在一些可以用于两个分支结构的数据，例如，某一分支结构中的某一层所输出的数据可以作为另一分支结构的某一层的一部分输入，以在另一分支结构的某一层进行处理。并且，该双分支结构中，两个分支结构的输入可以相同，也可以不同，还可以部分相同。
该目标拾取姿态信息用于指示拾取装置对目标物体进行拾取时的姿态，即目标拾取姿态。
示例性地，该目标拾取姿态信息可以包括6维(6dimension，6D)位姿信息，其中，6D包括6个自由度(degree of freedom，DoF)，具体包括3个自由度的平移(translation)，以及3个自由度的空间旋转(rotation)。举例来说，可以通过拾取装置对目标物体进行拾取的目标拾取点的三维坐标以及法向量，来描述该目标拾取姿态信息。
需要说明的是，神经网络对点云数据处理而得到的目标拾取姿态信息的方式可以有多种。
例如，在一些示例中，神经网络对点云数据处理而得到的关于目标拾取姿态的输出可以是目标拾取点的三维坐标。然后，可以从点云数据中获得该目标拾取点对应的法向量。此时，该三维坐标和法向量可以是点云数据对应的深度相机的相机坐标系下或者是世界坐标系下，所描述的目标拾取姿态也是在对应的深度相机的相机坐标系下或者是世界坐标系下。而为了实现对拾取装置的控制，需要将该三维坐标和法向量转换至拾取装置对应的坐标系下，并根据转换后的三维坐标和法向量，对拾取装置进行控制。
或者，在一些示例中，该神经网络对点云数据处理而得到的关于目标拾取姿态的输出可以是目标拾取点的三维坐标和法向量，此时，可以将神经网络输出的目标拾取点的三维坐标和法向量转换至拾取装置对应的坐标系下，并根据转换后的三维坐标和法向量，对拾取装置进行控制。
目标物体的类型信息用于指示目标物体的类型。其中，在一些情况下，目标物体上可以包括多种材质，相应地，该目标物体可以对应有多种类型信息。此时，在不同的场景中，该目标物体的类型信息可以用于指示目标物体整体的类型，也可以用于指示在拾取装置对目标物体进行拾取时，拾取装置与目标物体接触的部分的类型。
步骤203，根据目标拾取姿态信息和类型信息，控制拾取装置拾取目标物体。
在获得目标拾取姿态信息和目标物体的类型信息之后，可以控制拾取装置基于目标物体的类型信息，确定合适的拾取模式，并且，可以根据目标拾取姿态信息，确定对目标物体的较佳的抓取姿态，从而以合适的拾取模式和较佳的抓取姿态对目标物体进行稳定地拾 取，以成功执行拾取任务。
例如，可以基于目标物体的类型信息，从拾取装置中确定合适的末端执行器，还可以根据目标物体的类型信息确定合适的拾取参数。例如，目标物体的尺寸在预设的尺寸范围内时，若目标物体的类型为金属，则可以确定末端执行器的拾取参数的取值(例如吸盘的气流量)，以使得末端执行器的拾取力较大；而若目标物体的类型为塑料，则可以确定末端执行器的拾取参数的取值，以使得末端执行器的拾取力较小。然后，可以控制拾取装置以合适的末端执行器、合适的拾取参数以及准确的姿态对目标物体进行拾取。
可见，本申请实施例中，神经网络可以基于关于目标物体的点云数据，获得目标拾取姿态信息和目标物体的类型信息，从而可以针对目标物体的情况，适应性地以合适的拾取方式和拾取姿态对目标物体进行拾取。
本申请实施例中，可以根据神经网络所获得的目标拾取姿态信息和类型信息，对拾取装置进行控制以拾取目标物体，从而实现了软件与硬件相结合的协同控制，可以对于各种情况的物体进行成功拾取，自适应地完成场景较为复杂的拾取任务。
下面对软硬结合的方案进行示例性介绍。
在一些实施例中，拾取装置包括多种类型的末端执行器，不同类型的末端执行器用于拾取不同类型的物体；
上述步骤203，包括：
根据类型信息，确定拾取装置的目标拾取模式，目标拾取模式用于指示对目标物体进行拾取所采用的末端执行器；
控制拾取装置在目标拾取模式下，基于目标拾取姿态信息所指示的目标拾取姿态，拾取目标物体。
拾取装置的末端执行器的类型可以有多种选择。举例来说，多种类型的末端执行器可以包括机械手、吸盘以及磁吸式末端执行器中的至少两种。或者，多种类型的末端执行器可以包括多种类型的机械手、多种类型的吸盘或者多种类型的磁吸式末端执行器。以吸盘为例，多种类型的吸盘可以包括单塔式吸盘和多波纹吸盘等。本申请实施例中，类型的具体划分方式在此不作限定。
采用不同的末端执行器时，拾取物体的方式可以不同。
例如，某一末端执行器可以为机械手，此时可以通过机械手抓取物体。又如，某一末端执行器可以为吸盘，此时可以通过吸盘吸取物体。再如，某一末端执行器为磁吸式末端执行器，此时，可以通过磁吸的方式吸取物体。
拾取装置中，同一类型的末端执行器的数量可以是一个或者多个。并且，多个末端执行器可以各自独立地工作以拾取不同的物体，也可以用于对同一个物体进行拾取。
该拾取装置中，末端执行器的排布方式可以有多种。示例性地，多个末端执行器可以阵列式地排布于拾取装置中，也可以不规则地排布于拾取装置中。此外，在一些示例中，该多个末端执行器在拾取装置上的排布方式还可以基于应用场景的变化而变化。此外，末端执行器之间的间隔也可以基于实际应用场景中待拾取的物体的尺寸和形态等信息来确定。
如图3所示，为一种示例性的控制流程示意图。
其中，计算机设备可以通过神经网络处理点云数据，获得目标物体的类型信息和目标拾取姿态，然后控制拾取装置根据目标物体的类型信息切换至目标拾取模式，从而切换至对目标物体进行拾取所采用的末端执行器，并控制拾取装置以目标拾取姿态执行拾取任务。
本申请实施例中，拾取装置中设置有多种类型的末端执行器，从而可以根据待拾取的物体(如本申请实施例中的目标物体)的类型选择合适的末端执行器来进行拾取，从而可以基于实际场景中待拾取的物体的类型，适应性地选择较为合适的拾取模式来进行拾取。此外，还可以通过神经网络确定准确的拾取姿态，这样，可以基于实际场景，自适应地控制拾取装置以合适的拾取模式以及准确的姿态对目标物体进行拾取，从而成功执行拾取任务。
本申请实施例中，计算机设备中可以部署有神经网络，从而基于神经网络来确定待拾取的物体的类型以及拾取装置的拾取姿态。
下面对神经网络的一些实施例进行示例性介绍。
在一些实施例中，神经网络包括第一卷积网络和第二卷积网络；
上述步骤202，包括：
通过第一卷积网络，基于点云数据，获得目标拾取姿态信息；
通过第二卷积网络，基于点云数据，获得目标物体的类型信息。
本申请实施例中，第一卷积网络和第二卷积网络的具体结构在此不做限定。并且，本申请实施例中，第一卷积网络和第二卷积网络均可以包含卷积运算，并且并不限定第一卷积网络和第二卷积网络中的卷积运算中的具体方式。
例如，神经网络中的卷积运算可以包括对矩阵的卷积运算、对图结构的图卷积运算、对点云数据的卷积运算等中的一种或多种。针对不同的输入数据形式，对应的卷积运算的具体方式可能存在差异，例如，对图结构的图卷积运算和对点云数据的卷积运算的具体方式可能不同，可以参考目前或后续发展的相关技术来实现，本申请实施例中对此不再赘述。
本申请实施例中，第一卷积网络和第二卷积网络可以是两个并行的结构，也可以是存在一部分共享的结构，还存在一部分并行的结构。此外，第一卷积网络和第二卷积网络的数据可以相互独立地执行，也可以存在一些可以用于第一卷积网络和第二卷积网络的数据，例如，第一卷积网络中的某一层所输出的数据可以作为第二卷积网络的某一层的一部分输入。并且，第一卷积网络和第二卷积网络的输入可以相同，也可以不同，还可以部分相同。
在一些示例中，通过第一卷积网络的输出可以直接获得目标拾取姿态信息。而在另一种示例中，也可以通过信息转换等信息处理操作对第一卷积网络的输出进行处理，以获得该目标拾取姿态信息。例如，第一卷积网络的输出可以是目标拾取点在点云数据对应的深度相机的相机坐标系下的三维坐标。然后，可以从点云数据中获得该目标拾取点在相机坐标系下的法向量。而为了实现对拾取装置的控制，需要将目标拾取点在相机坐标系下的三维坐标和法向量转换至拾取装置对应的坐标系下，并根据转换后的三维坐标和法向量，对拾取装置进行控制。
本申请实施例中，通过两个分支结构(即第一卷积网络和第二卷积网络)来分别获得目标拾取姿态信息和目标物体的类型信息，从而基于较为全面的信息来评估对目标物体的 拾取方式，以针对各种情况的物体，自适应地选择合适的方式进行拾取，从而可以满足多种应用场景的需要。
在一些实施例中，上述步骤：通过第一卷积网络，基于点云数据，获得目标拾取姿态信息，包括：
根据拾取装置的末端执行器的结构信息，从点云数据中获得与末端执行器相匹配的至少两组局部点云数据，每组局部点云数据对应一个候选拾取姿态；
根据至少两组局部点云数据，通过第一卷积网络，获得目标拾取姿态信息。
其中，末端执行器的结构信息用于描述末端执行器的结构，例如，可以包括末端执行器在拾取物体时与物体关联的部分的结构的信息。对于不同类型的末端执行器，结构信息可以不同。
举例来说，某一末端执行器为吸盘，吸盘在拾取物体时与物体的接触面通常为圆形，因此，吸盘的结构信息可以包括吸盘与物体接触部分的半径或者直径等，这样，可以将吸盘等效为圆形平面区域的形式。
又如，某一末端执行器为机械手，机械手在拾取物体时，可以通过机械手上的多个末端手指来实现诸如半包围或者全包围式的抓取，因此，机械手的结构信息可以描述机械手的末端所形成的抓取姿态。此时，机械手的结构信息可以包括机械手的末端夹爪的尺寸(如每个末端夹爪的长、宽、高等)、各个末端夹爪之间的距离等信息。
在获得该结构信息之后，根据该结构信息从点云数据中获得与末端执行器相匹配的至少两组局部点云数据的方式在此不做限定。
示例性地，可以根据拾取装置的末端执行器的结构信息，构建等效形状，以根据该等效形状来描述相应的末端执行器。
在得到等效形状之后，可以通过该等效形状在点云数据中获得局部点云数据。例如，可以将点云数据转换为k-dimensional树(kd-tree)的形式，然后将该等效形状在kd-tree所描述的目标物体的轮廓上进行投影，若投影区域满足指定条件，则根据此次投影区域在点云数据中对应的点云，获得一组局部点云数据。
其中，该指定条件可以基于实际应用场景以及末端执行器的类型等来确定。示例性地，该指定条件可以包括满足力封闭和形封闭的约束条件。其中，形封闭指这样一种状态，即如果一个物体因一组施加在物体表面的静态约束而不能发生任何位姿的改变，而这组静态约束完全由于机械手等末端执行器的拾取位置而决定，那么就成这种状态为形封闭状态，同时称这组拾取操作为形封闭拾取操作。力封闭指这样一种状态，即如果一个物体因一组施加在物体表面的静态约束而不能发生任何位姿的改变，而这组静态约束完全由于机械手的手指等末端执行器的末端部分施加在物体接触点处的力螺旋决定，那么就称这种状态为力封闭状态，同时称这组拾取操作为力封闭拾取操作。基于形封闭和/或力封闭等约束条件的具体计算过程可以参考现有的以及后续发展的相关技术，本申请实施例中对此不做限定。
通过指定的搜索规则，可以调整等效形状与点云数据之间的相对位姿并进行投影。
若只存在一次投影所对应的投影区域满足指定条件，则仅获得一组局部点云数据。此时，可以将该组局部点云数据对应的候选拾取姿态作为目标拾取姿态，也即是说，可以根据该组局部点云数据，获得目标拾取姿态信息。
若其中存在至少两次投影所对应的投影区域满足指定条件，则可以获得至少两组局部点云数据，以在后续根据至少两组局部点云数据，通过第一卷积网络，获得目标拾取姿态信息。
其中，基于不同的应用场景，该等效形状可以是二维的图形，例如，吸盘对应的等效形状可以是圆形平面区域；或者，该等效形状可以是三维形状。
下面以具体示例来描述获得局部点云数据的一种示例性过程。
在一种示例中，末端执行器为二指夹爪，那么，该二指夹爪对应的一种示例性等效形状如图4a所示。其中，二指夹爪的结构信息可以包括每个夹爪的长、宽、高以及夹爪之间的距离。
如图4b所示获得局部点云数据的一种示例性示意图中，在获得该等效形状之后，可以将该等效形状在点云数据中所描述的目标物体的轮廓上进行投影，若该投影区域满足指定的条件(例如，满足力封闭和形封闭的约束条件)，则可以根据此次投影区域在点云数据中对应的点云，获得一组局部点云数据。
在另一种示例中，吸盘对应的等效形状为圆形平面区域，此时，可以将该圆形平面区域在目标物体的轮廓对应的点云上进行投影。若在目标物体的轮廓对应的点云上的投影区域满足指定条件，则根据此次投影区域在点云数据中对应的点云，获得一组局部点云数据。例如，目标物体的轮廓对应的点云上的投影区域为圆形，则将该圆形的圆心作为一个候选拾取点，并根据该候选拾取点的信息获得一组局部点云数据。
在一些场景中，拾取装置包括多种类型的末端执行器，此时，该多种类型的末端执行器对应的等效形状可以相同，也可以不同。
举例来说，拾取装置中可以包括单塔式吸盘和多波纹吸盘，而这两种吸盘对物体进行吸取时的接触面的大小可以相同，因此，单塔式吸盘和多波纹吸盘对应的等效形状可以相同，因此，可以通过这一等效形状来描述拾取装置中的各个末端执行器，相应地，获得的多组局部点云数据所描述的候选拾取姿态可以适用于拾取装置中的各个吸盘。
或者，拾取装置中包括的该多种类型的末端执行器对应的等效形状不同。此时，可以分别针对每一种末端执行器的结构信息，通过神经网络来获得一组输出，此时，该输出用于描述相应的末端执行器对目标物体的拾取姿态和对应的类型。这样，可以获得多种末端执行器对目标物体的拾取姿态和对应的类型，然后，可以基于预设的优先级、控制难度、控制资源消耗等信息，从该多种末端执行器中选择合适的末端执行器，并该合适的末端执行器对目标物体的拾取姿态作为目标拾取姿态，将对应的类型作为目标物体的类型。
每组局部点云数据可以包括至少一个候选拾取点的信息，以描述对应的一个候选拾取姿态。
该至少一个候选拾取点可以用于描述通过末端执行器对目标物体进行拾取的候选位置。
举例来说，若末端执行器为吸盘，由于吸盘与目标物体接触的结构部分是一个圆形平面区域，也即是说，该圆形平面区域可以表示吸盘对目标物体进行拾取的一个候选位置，因此，该至少一个候选拾取点可以包括相应的圆形平面区域的圆心。
可以通过至少一个候选拾取点表征候选拾取姿态。而若该局部点云数据中不仅包括候选拾取点的信息，还包括末端执行器的结构信息，则可以更为全面地描述该候选拾取姿态。 此时，局部点云数据不仅可以描述拾取装置对目标物体的一种候选的拾取位姿，还可以包含该拾取装置对该目标物体进行拾取时的接触区域等信息。
在获得至少两组局部点云数据之后，可以根据至少两组局部点云数据，通过第一卷积网络，获得目标拾取姿态信息。其中，第一卷积网络可以根据多组局部点云数据分别对应的候选拾取姿态而确定目标候选姿态，以输出目标候选姿态信息。
本申请实施例中，第一卷积网络的输入数据的形式在此不做限定。可以基于指定的数据结构(例如图结构)来描述该多组局部点云数据，并将该指定的数据结构输入第一卷积网络中。
在一些实施例中，第一卷积网络为图卷积网络；
上述步骤：根据至少两组局部点云数据，通过第一卷积网络，获得目标拾取姿态信息，包括：
根据至少两组局部点云数据，获得图结构，图结构中的每个节点对应一个局部点云数据；
通过第一卷积网络对图结构处理，获得目标拾取姿态信息。
本申请实施例中，图结构中可以包括节点和边。
其中，图结构中的每个节点可以唯一地对应一个局部点云数据。具体来说，若在图构建过程中，没有对局部点云数据进行筛选，则图结构中的节点与局部点云数据可以一一对应。
图结构中每个节点的节点信息可以包括对应的局部点云数据，例如，可以包括对应的候选拾取点的三维坐标信息和/或对应的末端执行器的结构信息。
而在获得各组局部点云数据之后，通常来说还并不能确定各组局部点云数据之间的关联关系，因此图结构中，各个节点之间的边可以是通过随机初始化等方式生成的，此时，该图结构中各个节点之间的边并不能反映各组局部点云数据之间的真实的关联关系。
可见，本申请实施例中，通过图构建的方式将各组局部点云数据转换为图结构的形式，以通过结构化数据来描述至少两组局部点云数据，满足图卷积网络对输入数据的形式要求。
然而，由于此时并不能确定各组局部点云数据之间的关联关系，因此，该图结构中的节点之间的连接关系可以认为是初始化生成的，并不反映各组局部点云数据之间的真实的关联关系。此时，各组局部点云数据之间的关联关系可以是在图卷积网络对图结构的处理过程中来确定的，也即是说，可以认为该图卷积网络能够确定图结构中的节点之间的边以及各边的权重，并根据图结构中的节点之间的边以及各边的权重，根据各组局部点云数据对应的候选拾取姿态确定目标拾取姿态。
该图卷积网络的具体类型在此不做限制。示例性地，该图卷积网络中可以包括第一特征提取网络以及第一分类器，以通过第一特征提取网络从各个局部点云数据中提取语义特征，然后，第一分类器对该语义特征进行处理，以输出目标拾取姿态。
在一些示例中，局部点云数据中可以包括对应的候选拾取点的信息。而在后续基于候选拾取点的信息进行处理时的不同阶段，可以对候选拾取点的不同信息进行应用。
举例来说，在一种示例中，根据候选拾取点的信息生成图结构时，图结构的节点信息中可以包括对应的候选拾取点的三维坐标的信息。获得图结构之后，可以通过图卷积网络 对该图结构进行处理以输出目标拾取点的三维坐标，此时，可以根据该目标拾取点在点云数据中的法向量以及三维坐标，确定拾取装置对目标物体的6D拾取位姿。具体地，可以将该目标拾取点在点云数据对应的坐标系下的法向量以及三维坐标转换为拾取装置对应的坐标系下的法向量以及三维坐标，再根据拾取装置对应的坐标系下目标拾取点的三维坐标和法向量，确定拾取装置对目标物体的目标拾取位姿。
如图5所示，为获得目标物体的目标拾取姿态的一种示例性示意图。
在图5所示的示例中，可以根据点云数据和末端执行器的结构信息，获得多组局部点云数据，然后对该多组局部点云数据进行图构建，获得对应的图结构，并将该图结构输入至图卷积网络中，获得该图卷积网络输出的目标拾取姿态信息。
本申请实施例中，通过图结构可以有效整合多组局部点云数据，从而通过图卷积网络，根据图结构中描述的多个候选拾取姿态，获得目标拾取姿态信息。
在一些实施例中，第一卷积网络包括第一特征提取网络和第一分类器，第一特征提取网络位于第一分类器之前；
上述步骤：通过第二卷积网络，基于点云数据，获得目标物体的类型信息，包括：
通过第二卷积网络，根据第一卷积网络基于点云数据进行处理时第一特征提取网络输出的第一特征信息，以及点云数据，获得目标物体的类型信息。
第一特征提取网络用于对相应的输入(例如图结构)进行特征提取。第一特征提取网络中所包含的层的个数和类型在此不做限制。示例性地，该第一特征提取网络可以包括卷积层、激活层、池化层等中的一种或多种。第一分类器用于对相应的输入(例如第一分类器的前一层输出的特征信息)进行分类，获得相应的分类结果。若该第一分类器为第一卷积网络的最后一层，则该第一分类器的输出可以为目标拾取姿态信息。第一分类器所包含的层的个数和类型在此不做限制。示例性地，该第一分类器中可以包括全连接层。
其中，第一特征提取网络和第一分类器在第一卷积网络中的具体位置关系在此不做具体限定。
在一种示例中，第一特征提取网络为第一分类器的前一层网络，第一特征提取网络的输出为第一分类器的输入；而在另一种示例中，第一特征提取网络和第一分类器之间也可以包含其他层。此外，在一些示例中，第一卷积网络还可以包括除第一特征提取网络和第一分类器之外的其他结构。
第二卷积网络的输入包括第一卷积网络基于点云数据进行处理时的第一特征提取网络的输出，以及点云数据。
本申请实施例中，第一卷积网络在确定拾取装置对目标物体的目标拾取姿态的过程中，往往需要提取该目标拾取姿态对应的局部特征，也即是说，第一卷积网络中基于点云数据进行处理时的第一特征提取网络输出的第一特征信息可以包含目标拾取姿态对应的局部特征信息。
基于此，本申请实施例中，不仅根据全局的点云数据来识别目标物体的类型，还结合第一特征信息中包含的局部特征信息来识别目标物体的类型。
其中，第二卷积网络中，结合点云数据和第一特征信息的方式可以有多种。
由于第一特征提取网络输出的第一特征信息是第一卷积网络的中间层输出，与点云数 据的数据形式和对应的语义通常差异较大，因此通常无法直接结合。此时，可以不直接结合点云数据和第一特征信息，而是将点云数据作为第二卷积神经网络的输入，并在第二卷积网络对点云数据的处理过程中，在第二卷积网络的某一中间层，将该中间层的前一层的输出与第一特征提取网络输出的第一特征信息结合并作为输入以进行处理，从而在该中间层以及后续层，结合全局的点云数据的特征以及第一特征提取网络提取出的局部特征信息来进行处理，以结合较为全面的特征信息，实现对目标物体的类型的准确识别。
在一些实施例中，第二卷积网络包括第二特征提取网络和第二分类器；
上述步骤：通过第二卷积网络，根据第一卷积网络基于点云数据进行处理时第一特征提取网络输出的第一特征信息，以及点云数据，获得目标物体的类型信息，包括：
通过第二特征提取网络对点云数据进行处理，获得第二特征信息；
将第一特征信息与第二特征信息进行聚合，获得聚合结果；
通过第二分类器对聚合结果进行处理，获得目标物体的类型信息。
本申请实施例中，第二卷积网络的具体类型在此不做限定。该第二卷积网络可以对点云数据进行处理，例如，第二卷积网络可以是PointNet网络或者其他开源的且能够对点云数据进行处理的网络，或者也可以是研发人员自行开发的网络。
其中，第二特征提取网络用于对点云数据进行特征提取，以获得关于点云数据的全局的特征信息。第二特征提取网络中所包含的层的个数和类型在此不做限制。示例性地，该第二特征提取网络可以包括卷积层、激活层、池化层等中的一种或多种。第二分类器用于对相应的输入(例如第二分类器的前一层输出的特征信息)进行分类，获得相应的分类结果。若该第二分类器为第二卷积网络的最后一层，则该第二分类器的输出可以为目标物体的类型信息。第二分类器所包含的层的个数和类型在此不做限制。示例性地，该第二分类器中可以包括全连接层。当然，在一些示例中，第二卷积网络中还可以包括第二特征提取网络和第二分类器之外的其他结构，本申请实施例中对此不做限制。
本申请实施例中，第二特征提取网络可以为第二分类器的前一层网络，此时，第二分类器的输入为第一特征信息与第二特征信息的聚合结果。
具体的聚合方式可以有多种，在此不做限定。例如，可以是将第一特征信息和第二特征信息进行拼接，以获得聚合结果；或者，可以是将第一特征信息和第二特征信息分别与各自对应的权重相乘之后，再进行拼接以获得聚合结果；或者，可以是将第一特征信息与第二特征信息进行对应元素的加和，以获得聚合结果。
其中，第二特征信息通常包括点云数据中目标物体的全局特征信息。而第一特征信息可以是基于多组局部点云数据而得到的特征信息，此时，第一特征信息通常包括点云数据中目标物体的局部特征信息。可见，本申请实施例中，第二分类器可以聚合目标物体的局部特征信息和全局特征信息来确定目标物体的类型，从而结合较为完整的信息，对目标物体进行准确分类。
如图6所示，为神经网络的一种示例性示例图。
其中，可以根据点云数据和末端执行器的结构信息，获得多组局部点云数据，然后对该多组局部点云数据进行图构建，获得对应的图结构，并将该图结构输入至第一卷积网络中，获得第一卷积网络输出的目标拾取姿态信息。
其中，第一卷积网络中具体包括第一特征提取网络和第一分类器，而第一特征提取网络关于该图结构的输出为第一特征信息，该第一特征信息可以包括关于目标物体的局部特征信息。
此外，点云数据可以输入至第二卷积网络的第二特征提取网络中，获得第二特征信息，该第二特征信息可以包括关于目标物体的全局特征信息。
然后，将第一特征信息与第二特征信息进行聚合，获得聚合结果；再通过第二分类器对聚合结果进行处理，获得目标物体的类型信息。
本申请实施例中，神经网络可以是训练完成的模型。
下面对第一卷积网络和第二卷积网络的训练过程进行示例性介绍。
本申请实施例中，可以预先构建训练数据集，该训练数据集中的每个训练样本均包括一组训练点云数据，并且，每个训练样本均对应有拾取姿态标签和类型标签。
示例性地，该训练点云数据可以是通过深度相机采集深度图像，并根据深度图像以及深度相机的内部参数等信息转换而得到的。
该拾取姿态标签可以是基于形封闭和/或力封闭等力学准则而得到的。基于形封闭和/或力封闭等力学准则而得到该拾取姿态标签的具体计算过程可以参考现有的以及后续发展的相关技术，本申请实施例中对此不做限定。
该类型标签可以是通过人工标注等方式而得到的。
在获得训练数据集之后，可以基于该训练数据集端到端地对待训练神经网络。
其中，将以下处理过程称为指定函数：根据末端执行器的结构信息，从训练点云数据中获得与末端执行器相匹配的至少两组局部训练点云数据，再根据至少两组局部训练点云数据，获得训练图结构的处理过程。那么，在每次迭代过程中，可以将训练点云数据作为待训练的第二卷积网络的输入，并通过指定函数对训练点云数据进行处理而获得训练图结构，然后将该训练图结构作为待训练的第一卷积网络的输入。
这样，可以基于该训练数据集，通过反向传播算法对待训练神经网络进行训练，并在待训练神经网络收敛之后，获得上述神经网络。
该神经网络可以是在执行本申请实施例的计算机设备中训练的，也可以是在其他设备训练完成之后，传输并部署至该计算机设备中的。
以上，本申请实施例从多个方面介绍了物体拾取方法，下面结合附图7，介绍本申请的物体拾取装置70，该物体拾取装置70可以应用于上述计算机设备。
如图7所示，该物体拾取装置70可以包括：
获取模块701，用于获取关于目标物体的点云数据；
处理模块702，用于获得神经网络对点云数据处理而得到的目标拾取姿态信息和目标物体的类型信息，目标拾取姿态信息用于描述拾取装置对目标物体的目标拾取姿态；
控制模块703，用于根据目标拾取姿态信息和类型信息，控制拾取装置拾取目标物体。
可选地，拾取装置包括多种类型的末端执行器，不同类型的末端执行器用于拾取不同类型的物体；
控制模块703用于：
根据类型信息，确定拾取装置的目标拾取模式，目标拾取模式用于指示对目标物体进行拾取所采用的末端执行器；
控制拾取装置在目标拾取模式下，基于目标拾取姿态信息所指示的目标拾取姿态，拾取目标物体。
可选地，其特征在于，神经网络包括第一卷积网络和第二卷积网络；
处理模块702用于：
通过第一卷积网络，基于点云数据，获得目标拾取姿态信息；
通过第二卷积网络，基于点云数据，获得目标物体的类型信息。
可选地，处理模块702用于：
根据拾取装置的末端执行器的结构信息，从点云数据中获得与末端执行器相匹配的至少两组局部点云数据，每组局部点云数据对应一个候选拾取姿态；
根据至少两组局部点云数据，通过第一卷积网络，获得目标拾取姿态信息。
可选地，第一卷积网络为图卷积网络；
处理模块702用于：
根据至少两组局部点云数据，获得图结构，图结构中的每个节点对应一个局部点云数据；
通过第一卷积网络对图结构处理，获得目标拾取姿态信息。
可选地，第一卷积网络包括第一特征提取网络和第一分类器，第一特征提取网络位于第一分类器之前；
处理模块702用于：
通过第二卷积网络，根据第一卷积网络基于点云数据进行处理时第一特征提取网络输出的第一特征信息，以及点云数据，获得目标物体的类型信息。
可选地，第二卷积网络包括第二特征提取网络和第二分类器；
处理模块702用于：
通过第二特征提取网络对点云数据进行处理，获得第二特征信息；
将第一特征信息与第二特征信息进行聚合，获得聚合结果；
通过第二分类器对聚合结果进行处理，获得目标物体的类型信息。
图8所示，是本申请实施例提供的计算机设备80的一种可能的逻辑结构示意图。该计算机设备80用于实现上述任一实施例中所涉及的计算机设备的功能。该计算机设备80包括：存储器801、处理器802、通信接口803以及总线804。其中，存储器801、处理器802、通信接口803通过总线804实现彼此之间的通信连接。
存储器801可以是只读存储器(read-only memory，ROM)、静态存储设备、动态存储设备或者随机存取存储器(random access memory，RAM)。存储器801可以存储程序，当存储器801中存储的程序被处理器802执行时，处理器802和通信接口803用于执行上述的物体拾取方法实施例的步骤201-203等。
处理器802可以采用中央处理器(central processing unit，CPU)、微处理器、特定 应用集成电路(application-specific integrated circuit，ASIC)、图形处理器(graphics processing unit，GPU)、数字信号处理器(digital signal processor，DSP)、现成可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件或者其任意组合，用于执行相关程序，以实现上述实施例的物体拾取装置中的获取模块、处理模块以及控制模块等所需执行的功能，或者执行本申请方法实施例的物体拾取方法实施例的步骤201-203等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器801，处理器802读取存储器801中的信息，结合其硬件执行上述的物体拾取方法实施例的步骤201-203等。
通信接口803使用例如但不限于收发器一类的收发装置，来实现计算机设备80与其他设备或通信网络之间的通信。
总线804可实现在计算机设备80各个部件(例如，存储器801、处理器802以及通信接口803)之间传送信息的通路。总线804可以是外设部件互连标准(Peripheral Component Interconnect，PCI)总线或扩展工业标准结构(Extended Industry Standard Architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图8中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
在本申请的另一实施例中，还提供一种计算机可读存储介质，计算机可读存储介质中存储有计算机执行指令，当设备的处理器执行该计算机执行指令时，设备执行上述图8中的处理器所执行的步骤。
在本申请的另一实施例中，还提供一种计算机程序产品，该计算机程序产品包括计算机执行指令，该计算机执行指令存储在计算机可读存储介质中；当设备的处理器执行该计算机执行指令时，设备执行上述图8中的处理器所执行的步骤。
在本申请的另一实施例中，还提供一种芯片系统，该芯片系统包括处理器，该处理器用于实现上述图8的处理器所执行的步骤。在一种可能的设计中，芯片系统还可以包括存储器，存储器，用于保存数据写入的装置必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包含芯片和其他分立器件。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请实施例的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请实施例所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或 组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请实施例各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请实施例各个实施例方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上，仅为本申请实施例的具体实施方式，但本申请实施例的保护范围并不局限于此。

Claims (16)

  1. 一种物体拾取方法，其特征在于，包括：
    获取关于目标物体的点云数据；
    获得神经网络对所述点云数据处理而得到的目标拾取姿态信息和所述目标物体的类型信息，所述目标拾取姿态信息用于描述拾取装置对目标物体的目标拾取姿态；
    根据所述目标拾取姿态信息和所述类型信息，控制所述拾取装置拾取所述目标物体。
  2. 根据权利要求1所述的方法，其特征在于，所述拾取装置包括多种类型的末端执行器，不同类型的末端执行器用于拾取不同类型的物体；
    所述根据所述目标拾取姿态信息和所述类型信息，控制拾取装置拾取所述目标物体，包括：
    根据所述类型信息，确定所述拾取装置的目标拾取模式，所述目标拾取模式用于指示对所述目标物体进行拾取所采用的末端执行器；
    控制所述拾取装置在所述目标拾取模式下，基于所述目标拾取姿态信息所指示的目标拾取姿态，拾取所述目标物体。
  3. 根据权利要求1或2所述的方法，其特征在于，所述神经网络包括第一卷积网络和第二卷积网络；
    所述获得神经网络对所述点云数据处理而得到的目标拾取姿态信息和所述目标物体的类型信息，包括：
    通过所述第一卷积网络，基于所述点云数据，获得所述目标拾取姿态信息；
    通过所述第二卷积网络，基于所述点云数据，获得所述目标物体的类型信息。
  4. 根据权利要求3所述的方法，其特征在于，所述通过所述第一卷积网络，基于所述点云数据，获得所述目标拾取姿态信息，包括：
    根据所述拾取装置的末端执行器的结构信息，从所述点云数据中获得与所述末端执行器相匹配的至少两组局部点云数据，每组局部点云数据对应一个候选拾取姿态；
    根据所述至少两组局部点云数据，通过所述第一卷积网络，获得目标拾取姿态信息。
  5. 根据权利要求4所述的方法，其特征在于，所述第一卷积网络为图卷积网络；
    所述根据所述至少两组局部点云数据，通过所述第一卷积网络，获得目标拾取姿态信息，包括：
    根据所述至少两组局部点云数据，获得图结构，所述图结构中的每个节点对应一个局部点云数据；
    通过所述第一卷积网络对所述图结构处理，获得目标拾取姿态信息。
  6. 根据权利要求3-5任一项所述的方法，其特征在于，所述第一卷积网络包括第一特征提取网络和第一分类器，所述第一特征提取网络位于所述第一分类器之前；
    所述通过所述第二卷积网络，基于所述点云数据，获得所述目标物体的类型信息，包括：
    通过所述第二卷积网络，根据所述第一卷积网络基于所述点云数据进行处理时所述第一特征提取网络输出的第一特征信息，以及所述点云数据，获得所述目标物体的类型信息。
  7. 根据权利要求6所述的方法，其特征在于，所述第二卷积网络包括第二特征提取网络和第二分类器；
    所述通过所述第二卷积网络，根据所述第一卷积网络基于所述点云数据进行处理时所述第一特征提取网络输出的第一特征信息，以及所述点云数据，获得所述目标物体的类型信息，包括：
    通过所述第二特征提取网络对所述点云数据进行处理，获得第二特征信息；
    将所述第一特征信息与所述第二特征信息进行聚合，获得聚合结果；
    通过第二分类器对所述聚合结果进行处理，获得所述目标物体的类型信息。
  8. 一种物体拾取装置，其特征在于，包括：
    获取模块，用于获取关于目标物体的点云数据；
    处理模块，用于获得神经网络对所述点云数据处理而得到的目标拾取姿态信息和所述目标物体的类型信息，所述目标拾取姿态信息用于描述拾取装置对目标物体的目标拾取姿态；
    控制模块，用于根据所述目标拾取姿态信息和所述类型信息，控制所述拾取装置拾取所述目标物体。
  9. 根据权利要求8所述的装置，其特征在于，所述拾取装置包括多种类型的末端执行器，不同类型的末端执行器用于拾取不同类型的物体；
    所述控制模块用于：
    根据所述类型信息，确定所述拾取装置的目标拾取模式，所述目标拾取模式用于指示对所述目标物体进行拾取所采用的末端执行器；
    控制所述拾取装置在所述目标拾取模式下，基于所述目标拾取姿态信息所指示的目标拾取姿态，拾取所述目标物体。
  10. 根据权利要求8或9所述的装置，其特征在于，所述神经网络包括第一卷积网络和第二卷积网络；
    所述处理模块用于：
    通过所述第一卷积网络，基于所述点云数据，获得所述目标拾取姿态信息；
    通过所述第二卷积网络，基于所述点云数据，获得所述目标物体的类型信息。
  11. 根据权利要求10所述的装置，其特征在于，所述处理模块用于：
    根据所述拾取装置的末端执行器的结构信息，从所述点云数据中获得与所述末端执行 器相匹配的至少两组局部点云数据，每组局部点云数据对应一个候选拾取姿态；
    根据所述至少两组局部点云数据，通过所述第一卷积网络，获得目标拾取姿态信息。
  12. 根据权利要求11所述的装置，其特征在于，所述第一卷积网络为图卷积网络；
    所述处理模块用于：
    根据所述至少两组局部点云数据，获得图结构，所述图结构中的每个节点对应一个局部点云数据；
    通过所述第一卷积网络对所述图结构处理，获得目标拾取姿态信息。
  13. 根据权利要求10-12任一项所述的装置，其特征在于，所述第一卷积网络包括第一特征提取网络和第一分类器，所述第一特征提取网络位于所述第一分类器之前；
    所述处理模块用于：
    通过所述第二卷积网络，根据所述第一卷积网络基于所述点云数据进行处理时所述第一特征提取网络输出的第一特征信息，以及所述点云数据，获得所述目标物体的类型信息。
  14. 根据权利要求13所述的装置，其特征在于，所述第二卷积网络包括第二特征提取网络和第二分类器；
    所述处理模块用于：
    通过所述第二特征提取网络对所述点云数据进行处理，获得第二特征信息；
    将所述第一特征信息与所述第二特征信息进行聚合，获得聚合结果；
    通过第二分类器对所述聚合结果进行处理，获得所述目标物体的类型信息。
  15. 一种计算机设备，其特征在于，所述计算机设备包括至少一个处理器、存储器及存储在所述存储器上并可被所述至少一个处理器执行的指令，所述至少一个处理器执行所述指令，以实现权利要求1-7任一项所述的方法的步骤。
  16. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该计算机程序被处理器执行时实现权利要求1-7任一项所述的方法。
PCT/CN2023/091430 2022-08-11 2023-04-28 一种物体拾取方法以及相关设备 Ceased WO2024032040A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23851270.1A EP4552808A4 (en) 2022-08-11 2023-04-28 METHOD FOR COLLECTING OBJECTS AND ASSOCIATED DEVICE
US19/050,588 US20250187178A1 (en) 2022-08-11 2025-02-11 Object Pickup Method and Related Device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210962888.1A CN117621032A (zh) 2022-08-11 2022-08-11 一种物体拾取方法以及相关设备
CN202210962888.1 2022-08-11

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/050,588 Continuation US20250187178A1 (en) 2022-08-11 2025-02-11 Object Pickup Method and Related Device

Publications (1)

Publication Number Publication Date
WO2024032040A1 true WO2024032040A1 (zh) 2024-02-15

Family

ID=89850636

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/091430 Ceased WO2024032040A1 (zh) 2022-08-11 2023-04-28 一种物体拾取方法以及相关设备

Country Status (4)

Country Link
US (1) US20250187178A1 (zh)
EP (1) EP4552808A4 (zh)
CN (1) CN117621032A (zh)
WO (1) WO2024032040A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119115957A (zh) * 2024-10-28 2024-12-13 北京邮电大学 一种多臂空间机器人的抓取规划方法
CN119660058A (zh) * 2024-11-26 2025-03-21 武汉理工大学 基于柔性机械手的汽车冲压件动态抓取与装箱方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180336690A1 (en) * 2017-05-19 2018-11-22 Faro Technologies, Inc. Three-dimensional measurement device with annotation features
CN110027005A (zh) * 2019-05-27 2019-07-19 北京瑞盈智拓科技发展有限公司 一种快换工具的工具库及快换工具
CN111085997A (zh) * 2019-12-17 2020-05-01 清华大学深圳国际研究生院 基于点云获取和处理的抓取训练方法及系统
US20210086358A1 (en) * 2019-09-25 2021-03-25 Toyota Research Institute, Inc. Object location analysis
CN112894377A (zh) * 2021-01-15 2021-06-04 廊坊市亿创科技有限公司 带有工具自动切换功能的机器人柔性打磨系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275063B (zh) * 2018-12-04 2023-06-09 深圳市中科德睿智能科技有限公司 一种基于3d视觉的机器人智能抓取控制方法及系统
US11813758B2 (en) * 2019-04-05 2023-11-14 Dexterity, Inc. Autonomous unknown object pick and place
US11559885B2 (en) * 2020-07-14 2023-01-24 Intrinsic Innovation Llc Method and system for grasping an object
AU2021391551A1 (en) * 2020-12-02 2023-06-22 Ocado Innovation Limited Pixelwise predictions for grasp generation

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180336690A1 (en) * 2017-05-19 2018-11-22 Faro Technologies, Inc. Three-dimensional measurement device with annotation features
CN110027005A (zh) * 2019-05-27 2019-07-19 北京瑞盈智拓科技发展有限公司 一种快换工具的工具库及快换工具
US20210086358A1 (en) * 2019-09-25 2021-03-25 Toyota Research Institute, Inc. Object location analysis
CN111085997A (zh) * 2019-12-17 2020-05-01 清华大学深圳国际研究生院 基于点云获取和处理的抓取训练方法及系统
CN112894377A (zh) * 2021-01-15 2021-06-04 廊坊市亿创科技有限公司 带有工具自动切换功能的机器人柔性打磨系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4552808A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119115957A (zh) * 2024-10-28 2024-12-13 北京邮电大学 一种多臂空间机器人的抓取规划方法
CN119660058A (zh) * 2024-11-26 2025-03-21 武汉理工大学 基于柔性机械手的汽车冲压件动态抓取与装箱方法及系统
CN119660058B (zh) * 2024-11-26 2025-09-16 武汉理工大学 基于柔性机械手的汽车冲压件动态抓取与装箱方法及系统

Also Published As

Publication number Publication date
US20250187178A1 (en) 2025-06-12
CN117621032A (zh) 2024-03-01
EP4552808A1 (en) 2025-05-14
EP4552808A4 (en) 2025-11-12

Similar Documents

Publication Publication Date Title
Fujita et al. What are the important technologies for bin picking? Technology analysis of robots in competitions based on a set of performance metrics
Schmidt et al. Grasping of unknown objects using deep convolutional neural networks based on depth images
CN115861999B (zh) 一种基于多模态视觉信息融合的机器人抓取检测方法
CN110948492B (zh) 一种基于深度学习的三维抓取平台和抓取方法
Araki et al. MT-DSSD: Deconvolutional single shot detector using multi task learning for object detection, segmentation, and grasping detection
CN109584298B (zh) 面向机器人自主物体拾取任务的在线自学习方法
CN113752255B (zh) 一种基于深度强化学习的机械臂六自由度实时抓取方法
CN117773920A (zh) 一种自然语言驱动的机械臂抓取方法
US20250187178A1 (en) Object Pickup Method and Related Device
US12377549B2 (en) System and method for determining a grasping hand model
WO2023092519A1 (zh) 抓取控制方法、装置、电子设备和存储介质
Zeng Learning visual affordances for robotic manipulation
JP7836423B2 (ja) ヒューリスティックに基づくロボット把持
Zhang et al. A robotic grasp detection method based on auto-annotated dataset in disordered manufacturing scenarios
CN115256377A (zh) 一种基于多源信息融合的机器人抓取方法和装置
US20240198515A1 (en) Transformation for covariate shift of grasp neural networks
CN113538576A (zh) 基于双臂机器人的抓取方法、装置及双臂机器人
CN118493372A (zh) 一种基于深度学习的机械臂交叉反馈抓取方法及抓取系统
CN116079723A (zh) 基于视觉的机器人抓取和装配技能深度强化学习方法
CN117772648A (zh) 基于具身智能的零件分拣处理方法、装置、设备及介质
Yildiz et al. Convolutional Neural Network Based Hand Gesture Recognition in Sophisticated Background for Humanoid Robot Control.
Tang et al. A reinforcement learning method for rearranging scattered irregular objects inside a crate
Jhang et al. Optimizing parameters of YOLO model through uniform experimental design for gripping tasks performed by an internet of things–based robotic arm
Geng et al. A novel Real-time Grasping Method Cobimbed with YOLO and GDFCN
Mohammed et al. Color matching based approach for robotic grasping

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23851270

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023851270

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2023851270

Country of ref document: EP

Effective date: 20250205

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 2023851270

Country of ref document: EP