CN116152702A - Method, device, electronic device and self-driving vehicle for obtaining point cloud tags - Google Patents

Abstract

The disclosure provides a method and a device for acquiring a point cloud tag and an automatic driving vehicle, and belongs to the technical field of artificial intelligence, in particular to the technical field of deep learning, semantic segmentation and automatic driving, and the specific implementation scheme is as follows: acquiring M point cloud frames including a current point cloud frame, and respectively projecting point clouds in the M point cloud frames into grids of a bird's eye view, wherein M is an integer greater than or equal to 2; acquiring target fusion characteristics of M point cloud frames corresponding to each grid according to the projected aerial view; aiming at each grid, based on target fusion characteristics of the grid, respectively acquiring semantic segmentation labels and dynamic and static labels of the grid; and carrying out back projection on the grid, obtaining the grid where the point cloud is located in the current point cloud frame, and determining the label of the grid where the point cloud is located as the label of the point cloud. According to the method and the device for obtaining the labels of the point cloud, feature fusion is carried out on the multi-frame point cloud, the labels of the point cloud are obtained based on the fusion features, the time delay between the labels of the obtained point cloud is reduced, and the accuracy and the reliability of the labels of the point cloud are improved.

Description

Method, device, electronic device and self-driving vehicle for obtaining point cloud tags

technical field

The present disclosure relates to the technical field of artificial intelligence, specifically to the technical fields of deep learning, semantic segmentation and automatic driving, and in particular to a method and device for acquiring point cloud tags, and a storage medium for electronic equipment.

Background technique

With the popularity and development of deep learning and lidar, it is gradually becoming possible to perform semantic segmentation and dynamic and static estimation of point cloud background through deep learning methods. In related technologies, the two tasks of point cloud background semantic segmentation and dynamic and static estimation are independent of each other. When you want to obtain semantic segmentation results and dynamic and static estimation results at the same time, you need to use two deep learning models to generate semantic segmentation results and dynamic and static estimation results respectively. Static estimation results, if the application scenario is an automatic driving scenario, the above method will often increase the delay of the automatic driving perception link. Therefore, how to reduce the delay between the semantic segmentation labels and dynamic and static labels of each point cloud in the point cloud frame at the same time, and at the same time ensure the accuracy and reliability of the label of each point cloud in the point cloud frame, has become an urgent solution. The problem.

Contents of the invention

The present disclosure provides a point cloud label acquisition method, device, electronic equipment, storage medium and program product.

According to the first aspect, a method for acquiring point cloud tags is provided, including: acquiring M point cloud frames including the current point cloud frame, and projecting the point clouds in the M point cloud frames to a bird's-eye view respectively In the grid of the figure, wherein M is an integer greater than or equal to 2; according to the bird's eye view after projection, obtain the target fusion features of the M point cloud frames corresponding to each grid; for each grid based on the grid Target fusion features, respectively obtain the semantic segmentation label and dynamic and static label of the grid; perform back projection on each grid, determine the grid of the point cloud in the current point cloud frame, and place the point cloud in The semantic segmentation label and dynamic and static label of the mesh are determined as the label of the point cloud.

According to the second aspect, there is provided a device for acquiring point cloud tags, including: a projection module, configured to acquire M point cloud frames including the current point cloud frame, and convert points in the M point cloud frames into The clouds are respectively projected into the grid of the bird's-eye view, wherein M is an integer greater than or equal to 2; the first acquisition module is used to obtain the target fusion of the M point cloud frames corresponding to each grid according to the bird's-eye view after projection feature; the second acquisition module, for each grid, based on the target fusion feature of the grid, respectively acquire the semantic segmentation label and the dynamic and static label of the grid; the third acquisition module, for each grid Perform back projection, determine the grid of the point cloud in the current point cloud frame, and determine the semantic segmentation label and dynamic and static label of the grid where the point cloud is located as the label of the point cloud.

According to a third aspect, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor , the instructions are executed by the at least one processor, so that the at least one processor can execute the method for acquiring point cloud tags described in the first aspect of the present disclosure.

According to a fourth aspect, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method for obtaining point cloud labels according to the first aspect of the present disclosure .

According to a fifth aspect, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for obtaining point cloud tags according to the first aspect of the present disclosure is implemented.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The accompanying drawings are used to better understand the present solution, and do not constitute a limitation to the present disclosure. in:

1 is a schematic flow diagram of a method for obtaining point cloud tags according to a first embodiment of the present disclosure;

2 is a schematic flow diagram of a method for obtaining point cloud tags according to a second embodiment of the present disclosure;

3 is a schematic flow diagram of a method for obtaining point cloud tags according to a third embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of a backbone network according to the present disclosure;

5 is a schematic flow diagram of a method for obtaining point cloud tags according to a fourth embodiment of the present disclosure;

6 is a schematic flowchart of a method for obtaining point cloud tags according to a fifth embodiment of the present disclosure;

7 is a schematic diagram of a method for obtaining point cloud tags according to the present disclosure;

FIG. 8 is a block diagram of an acquisition device for point cloud tags used to implement an embodiment of the present disclosure;

Fig. 9 is a block diagram of an electronic device used to implement the method for acquiring a point cloud tag according to an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Artificial Intelligence (AI for short) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

Deep Learning (DL) is a new research direction in the field of Machine Learning (ML). It is introduced into machine learning to make it closer to the original goal—artificial intelligence. Deep learning is to learn sample data The internal laws and representation levels of these learning processes are of great help to the interpretation of data such as text, images and sounds. Its ultimate goal is to enable machines to have the ability to analyze and learn like humans, and to be able to recognize data such as text, images, and sounds.

Semantic segmentation (semantic segmentation) is a basic task in computer vision. In semantic segmentation, visual input needs to be divided into different semantically interpretable categories, that is, the classification categories are meaningful in the real world.

Automatic driving generally refers to the automatic driving system, which uses advanced communication, computer, network and control technologies to realize real-time and continuous control of trains. Using modern means of communication, directly facing the train, can realize two-way data communication between the train and the ground, with fast transmission rate and large amount of information. Follow-up tracking of the train and the control center can know the exact position of the preceding train in time, making the operation management more flexible. The control is more effective and more suitable for the needs of automatic train driving.

A method for acquiring point cloud tags according to an embodiment of the present disclosure will be described below with reference to the accompanying drawings.

Fig. 1 is a schematic flowchart of a method for acquiring point cloud tags according to a first embodiment of the present disclosure.

As shown in Figure 1, the method for obtaining point cloud tags in the embodiment of the present disclosure may specifically include the following steps:

S101. Acquire M point cloud frames including the current point cloud frame, and respectively project the point clouds in the M point cloud frames to the grid of the bird's-eye view, where M is an integer greater than or equal to 2.

Specifically, the execution body of the method for obtaining point cloud tags in the embodiment of the present disclosure may be the processing device provided in the embodiment of the present disclosure, and the processing device may be a hardware device with data information processing capabilities and/or a device that drives the hardware device to work. The necessary software is required. Optionally, the execution subject may include workstations, servers, computers, user terminals and other devices. Wherein, the user terminal includes, but is not limited to, a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, and the like.

It should be noted that, the present disclosure does not limit the specific manner of acquiring the point cloud frame, which can be selected according to the actual situation.

Optionally, a laser radar acquisition device can be used to acquire point cloud frames.

For example, image acquisition devices such as laser line scan cameras and binocular structured light cameras can be used to obtain point cloud frames.

Optionally, a laser radar (Light Detection And Ranging, LiDAR for short) may be used to acquire the point cloud frame.

Among them, the Bird's Eye View (BEV for short) is a three-dimensional map drawn from a certain point on a high place looking down on the undulations of the ground based on the principle of perspective.

It should be noted that projecting the point clouds in the M point cloud frames to the grid of the bird's-eye view is to project the three-dimensional (x, y, z) coordinates of the point cloud to the two-dimensional (x, y) grid coordinates.

It should be noted that the present disclosure does not limit the specific setting of the grid of the bird's-eye view, which can be set according to actual conditions.

Optionally, 10m can be set as a grid; optionally, 20m can be set as a grid.

It should be noted that after obtaining the three-dimensional (x, y, z) coordinates of the point cloud, the point cloud can be projected to the bird's-eye view corresponding to in the grid.

Optionally, when M is 2, the point clouds in the point cloud frame of the current frame and the point cloud frame of the previous frame can be respectively projected into the grid of the bird's-eye view.

S102. According to the projected bird's-eye view, acquire target fusion features of M point cloud frames corresponding to each grid.

It should be noted that after projecting the point clouds in the M point cloud frames to the grid of the bird's-eye view, multiple initial feature information can be obtained according to the projected bird's-eye view, and then multiple initial feature information can be processed , to obtain the target fusion features.

For example, according to the bird's-eye view after projection, various initial features such as the number information of the point cloud, the reflectivity information of the point cloud, the height information of the point cloud, and the height difference information of the point cloud can be obtained, and then the various initial features can be obtained. Splicing to obtain spliced features, and then perform feature extraction on the spliced features to obtain target fusion features.

S103. Obtain semantic segmentation labels and dynamic and static labels of the grid based on the target fusion feature of the grid.

For example, after the target fusion feature of the grid is obtained, the target fusion feature can be input into the corresponding model to obtain the semantic segmentation label and dynamic and static label of the grid respectively. The model has two output branches, Semantic segmentation labels and dynamic and static labels of the mesh can be output at the same time.

S104. Perform reverse projection on each grid, determine the grid where the point cloud is located in the current point cloud frame, and determine the semantic segmentation label and dynamic and static label of the grid where the point cloud is located as the label of the point cloud.

In the embodiment of the present disclosure, after obtaining the semantic segmentation labels and dynamic and static labels of each grid, each grid can be back-projected, that is, each grid can be back-projected to the current point cloud frame , and then the point cloud in the current point cloud frame covered by each grid can be determined, so that the grid of each point cloud in the current point cloud frame can be determined. Further, the semantic segmentation label and dynamic and static label of the grid where the point cloud is located are determined as the label of the point cloud.

For example, for grid 1, when the semantic segmentation label of grid 1 is 1, and the dynamic and static label is 0, then the semantic segmentation label of each point cloud in the current point cloud frame in grid 1 is 1, and the dynamic and static label is 0. Static tags are all 0.

To sum up, the method for obtaining point cloud tags in the embodiment of the present disclosure obtains M point cloud frames including the current point cloud frame, and projects the point clouds in the M point cloud frames to the grid of the bird's-eye view Among them, M is an integer greater than or equal to 2. According to the bird's-eye view after projection, the target fusion features of M point cloud frames corresponding to each grid are obtained. For each grid, based on the target fusion features of the grid, the network The semantic segmentation label and dynamic and static label of the grid are back-projected to each grid to obtain the label of each point cloud in the current point cloud frame. The disclosure realizes the aggregation of point clouds in multiple point cloud frames through the grid by projecting the point clouds in multiple point cloud frames into the grid, so that the same grid can have the characteristics of multiple point cloud frames, and then Based on the target fusion feature of the grid, the semantic segmentation label and dynamic and static label of the grid are obtained, and then the grid of the point cloud in the current point cloud frame is determined by back projection, so as to determine the semantic segmentation label of the point cloud in the current point cloud frame And dynamic and static labels, not only reduces the time delay between acquiring point cloud semantic segmentation labels and dynamic and static labels, but also improves the accuracy and reliability of point cloud labels.

Fig. 2 is a schematic flowchart of a method for acquiring point cloud tags according to a second embodiment of the present disclosure.

As shown in FIG. 2 , on the basis of the embodiment shown in FIG. 1 , the method for obtaining a point cloud tag in an embodiment of the present disclosure may specifically include the following steps:

S201. Acquire M point cloud frames including the current point cloud frame, and respectively project the point clouds in the M point cloud frames to the grid of the bird's-eye view, where M is an integer greater than or equal to 2.

Specifically, step S201 in this embodiment is the same as step S101 in the foregoing embodiment, and will not be repeated here.

The step S102 in the above embodiment of "obtaining the target fusion features of the M point cloud frames corresponding to each grid according to the projected bird's-eye view" may specifically include the following steps S202-S204.

S202. For each of the M point cloud frames, according to the projected bird's-eye view of the point cloud frame, acquire initial feature information of the point cloud frame corresponding to each grid.

As a possible implementation, as shown in FIG. 3, on the basis of the above-mentioned embodiment, in the above-mentioned step S202, according to the projected bird's-eye view of the point cloud frame, the initial feature information of the point cloud frame corresponding to each grid is obtained. The specific process includes the following steps:

S301. Obtain point cloud sets located in each grid in the projected bird's-eye view.

Optionally, according to the two-dimensional (x, y) coordinates of the grid, the point cloud set located in each grid in the projected bird's eye view may be determined, wherein the point cloud set is composed of multiple point clouds.

S302. Determine initial feature information of each grid based on the point cloud set of each grid.

Optionally, the number of point clouds in the point cloud set, the height value of the point cloud in the point cloud set can be obtained, and the height difference and/or average height of the point cloud set can be determined according to the height value, the reflectivity of the point cloud in the point cloud set, and Determines the average reflectance of the point cloud set from the reflectance.

Optionally, the number of point clouds in the point cloud set, the height difference and/or average height of the point cloud set, and the average reflectivity of the point cloud set may be used as the initial feature information of each grid.

S203. Concatenate the initial feature information of each point cloud frame of the same grid to obtain candidate spliced features.

For example, the initial feature information of each point cloud frame of the same grid can be concatenated in the row dimension to obtain candidate concatenated features.

S204. Process the candidate splicing features of the grid through the backbone network to obtain the target fusion features of the grid.

Optionally, an initial backbone network structure can be established, the initial backbone network structure can be trained using known data sets and verification sets, a total loss function can be set to supervise the initial backbone network structure, and a trained backbone network structure can be obtained.

For example, the structure of the backbone network, as shown in Figure 4, can input candidate splicing features (concat) into the backbone network, and the convolutional layer and deconvolution layer of the backbone network process the candidate splicing features to Output the target fusion features.

The step S103 "acquire the semantic segmentation label of the grid based on the target fusion feature of the grid" in the above embodiment may specifically include the following steps S205-S207.

S205, perform semantic segmentation on the target fusion features of the grid, and obtain multiple semantic segmentation probabilities of the grid.

As a possible implementation, as shown in FIG. 5 , on the basis of the above embodiment, in the above step S205, the target fusion feature of the grid is semantically segmented, and the specific process of obtaining multiple semantic segmentation probabilities of the grid is as follows: Include the following steps:

S501. Perform first convolution processing on the target fusion feature to obtain the fusion feature after the first convolution.

Optionally, two-dimensional convolution may be used on the target fusion feature to encode the target fusion feature to obtain the first convolutional fusion feature.

S502. Perform a first probability function mapping on the fusion features after the first convolution to obtain multiple semantic segmentation probabilities.

Optionally, the first probability function can be a normalized exponential function, that is, a softmax function, through which a numerical vector can be normalized into a probability distribution vector, and the sum of the probabilities is one, which can be mapped to (0 , the value of 1).

For example, by performing softmax function mapping on the fusion features after the first convolution, multiple semantic segmentation probabilities can be obtained.

S206. Determine the largest semantic segmentation probability from multiple semantic segmentation probabilities.

It should be noted that the present disclosure does not limit the specific manner of determining the maximum semantic segmentation probability from multiple semantic segmentation probabilities, which may be selected according to actual conditions.

Optionally, the maximum semantic segmentation probability can be determined from multiple semantic segmentation probabilities through the maximum independent variable point set argmax function.

S207. Determine the semantic segmentation label corresponding to the maximum semantic segmentation probability as the semantic segmentation label of the grid.

In the embodiment of the present disclosure, when determining as the semantic segmentation label of the grid, the semantic segmentation label corresponding to the maximum semantic segmentation probability may be determined as the semantic segmentation label of the grid.

The step S103 "acquire the dynamic and static labels of the grid based on the target fusion features of the grid" in the above embodiment may specifically include the following steps S208-S209.

S208, perform binary classification recognition on the target fusion feature of the grid, and obtain the type identification probability of the grid.

As a possible implementation, as shown in FIG. 6, on the basis of the above-mentioned embodiment, in the above-mentioned step S208, the target fusion feature of the grid is classified and identified, and the specific process of obtaining the type identification probability of the grid includes: The following steps:

S601. Perform a second convolution process on the target fusion feature to obtain the fusion feature after the second convolution.

Optionally, two-dimensional convolution may be used on the target fusion feature to encode the target fusion feature to obtain the second convolutional fusion feature.

S602. Perform second probability function mapping on the fusion features after the second convolution to obtain the type identification probability of the grid.

Optionally, the second probability function may be a sigmiod function, through which variables can be mapped to values of (0, 1).

For example, the sigmiod function mapping is performed on the fusion feature after the second convolution to obtain the type identification probability of the grid.

S209, comparing the type identification probability with a preset probability threshold, and determining the dynamic and static labels of the grid based on the comparison result.

Optionally, the probability threshold may be preset to be 0.5.

For example, if the type identification probability is greater than 0.5, the dynamic and static label of the grid may be determined as 1, and if the type identification probability is less than 0.5, the dynamic and static label of the grid may be determined as 0.

S210. Reverse project each grid to obtain the label of each point cloud in the current point cloud frame.

Specifically, step S210 in this embodiment is the same as step S104 in the above embodiment, and will not be repeated here.

The disclosure realizes the aggregation of point clouds in multiple point cloud frames through the grid by projecting the point clouds in multiple point cloud frames into the grid, so that the same grid can have the characteristics of multiple point cloud frames, and then Based on the target fusion feature of the grid, the semantic segmentation label and dynamic and static label of the grid are obtained, and then the grid of the point cloud in the current point cloud frame is determined by back projection, so as to determine the semantic segmentation label of the point cloud in the current point cloud frame and static and dynamic labels, improving the accuracy and reliability of the labels for each point cloud in the point cloud frame. Further, the present disclosure can obtain semantic segmentation labels and dynamic and static labels of each point cloud in a point cloud frame at the same time through a complete neural network, reducing the time delay between acquiring semantic segmentation labels and dynamic and static labels of point clouds .

The following explains how to obtain point cloud tags.

For example, as shown in Figure 7, the current frame (current frame) point cloud frame (point cloud) and its previous frame (previous frame) point cloud frame (point cloud) can be projected onto the Bev grid to obtain the projection After the bird's-eye view (Bev projection), the initial feature information of the point cloud frame corresponding to each grid is generated (handcraft feature), and the initial feature information generated by the two frames of point clouds are spliced (concat) and input to In the model, the model includes a backbone network, a semantic segmentation network and a dynamic and static estimation network. Optionally, an initial backbone network structure can be established, and the initial backbone network structure can be trained using a known data set and a verification set, and the total loss function can be set Supervise the initial backbone network structure to obtain a trained backbone network structure. Optionally, an initial semantic segmentation network structure can be established, and the initial semantic segmentation network structure can be trained using known data sets and verification sets, and the total loss function can be set to The initial semantic segmentation network structure is supervised to obtain the trained semantic segmentation network structure. Optionally, the initial dynamic and static estimation network structure can be established, and the initial dynamic and static estimation network structure can be trained by using the known data set and verification set. The loss function supervises the initial dynamic and static estimation network structure to obtain the trained dynamic and static estimation network structure. First, the candidate splicing features are down-sampled through the backbone network to obtain the target fusion feature (Deconv2d, s=4). In the output branch of the segmentation, the target fusion feature is encoded using two-dimensional convolution, the output is converted into a probability value using the softmax function, and the maximum semantic segmentation probability is determined from multiple semantic segmentation probabilities through the argmax function, and the maximum The semantic segmentation label corresponding to the semantic segmentation probability is determined as the semantic segmentation label of the grid. In the output branch of dynamic and static estimation, the target fusion feature is encoded using two-dimensional convolution, and then the output is converted into a probability using the sigmiod function Value, by setting a probability threshold of 0.5, compare the type recognition probability with the preset probability threshold, and determine the dynamic and static labels of the grid based on the comparison results. After obtaining the semantic segmentation labels and dynamic and static labels in the bev grid, you can The label of each point cloud in the current frame point cloud frame is obtained by back projection.

The disclosure realizes the aggregation of point clouds in multiple point cloud frames through the grid by projecting the point clouds in multiple point cloud frames into the grid, so that the same grid can have the characteristics of multiple point cloud frames, and then Based on the target fusion feature of the grid, the semantic segmentation label and dynamic and static label of the grid are obtained, and then the grid of the point cloud in the current point cloud frame is determined by back projection, so as to determine the semantic segmentation label of the point cloud in the current point cloud frame and static and dynamic labels, improving the accuracy and reliability of the labels for each point cloud in the point cloud frame. Further, the present disclosure can obtain semantic segmentation labels and dynamic and static labels of each point cloud in a point cloud frame at the same time through a complete neural network, reducing the time delay between acquiring semantic segmentation labels and dynamic and static labels of point clouds .

It should be noted that the acquisition, storage and application of the user's personal information involved in the technical solution of the present disclosure all comply with relevant laws and regulations and do not violate public order and good customs.

Fig. 8 is a schematic structural diagram of an apparatus for acquiring point cloud tags according to an embodiment of the present disclosure.

As shown in FIG. 8 , the device 800 for obtaining point cloud tags includes: a projection module 810 , a first obtaining module 820 , a second obtaining module 830 and a third obtaining module 840 . in:

The projection module 810 is configured to obtain M point cloud frames including the current point cloud frame, and project the point clouds in the M point cloud frames into the grid of the bird's-eye view, wherein M is greater than or equal to an integer of 2;

The first acquisition module 820 is configured to acquire the target fusion features of the M point cloud frames corresponding to each grid according to the projected bird's-eye view;

The second acquiring module 830 is configured to acquire, for each grid, the semantic segmentation label and the dynamic and static label of the grid based on the target fusion feature of the grid;

The third acquisition module 840 is used to reverse project each grid, determine the grid where the point cloud is located in the current point cloud frame, and determine the semantic segmentation label and dynamic and static label of the grid where the point cloud is located as A label for the point cloud.

Wherein, the first acquisition module 820 is also used for:

For each point cloud frame in the M point cloud frames, according to the bird's-eye view after projection of the point cloud frame, the initial feature information of the point cloud frame corresponding to each grid is obtained;

Concatenate the initial feature information of each point cloud frame of the same grid to obtain candidate splicing features;

The candidate splicing features are processed through a backbone network to obtain the target fusion features.

Wherein, the first obtaining module 820 is also used for:

Obtain the point cloud set located in each grid in the bird's-eye view after the projection;

Based on the point cloud set of each grid, the initial feature information of each grid is determined.

Wherein, the first acquisition module 820 is also used for:

Obtain the number of point clouds in the point cloud set;

Obtain the height value of the point cloud in the point cloud set, and determine the height difference and/or average height of the point cloud set according to the height value;

Obtain the reflectance of the point cloud in the point cloud set, and determine the average reflectance of the point cloud set according to the reflectance.

Wherein, the second acquiring module 830 is also used for:

For each grid, perform semantic segmentation on the target fusion feature of the grid, and obtain multiple semantic segmentation probabilities of the grid;

determining a maximum semantic segmentation probability from the plurality of semantic segmentation probabilities;

The semantic segmentation label corresponding to the maximum semantic segmentation probability is determined as the semantic segmentation label of the grid.

Wherein, the second acquiring module 830 is also used for:

performing a first convolution process on the target fusion feature to obtain the fusion feature after the first convolution;

Performing a first probability function mapping on the first convolutional fusion feature to obtain the plurality of semantic segmentation probabilities.

Wherein, the second acquiring module 830 is also used for:

For each grid, perform binary classification recognition on the target fusion feature of the grid, and obtain the type recognition probability of the grid;

The type identification probability is compared with a preset probability threshold, and the dynamic and static labels of the grid are determined based on the comparison result.

Wherein, the second acquiring module 830 is also used for:

performing a second convolution process on the target fusion feature to obtain the fusion feature after the second convolution;

A second probability function mapping is performed on the second convolutional fusion feature to obtain the type identification probability of the grid.

It should be noted that the above explanations on the embodiment of the method for obtaining point cloud tags are also applicable to the device for obtaining point cloud tags in the embodiment of the present disclosure, and the specific process will not be repeated here.

The disclosure realizes the aggregation of point clouds in multiple point cloud frames through the grid by projecting the point clouds in multiple point cloud frames into the grid, so that the same grid can have the characteristics of multiple point cloud frames, and then Based on the target fusion feature of the grid, the semantic segmentation label and dynamic and static label of the grid are obtained, and then the grid of the point cloud in the current point cloud frame is determined by back projection, so as to determine the semantic segmentation label of the point cloud in the current point cloud frame and static and dynamic labels, improving the accuracy and reliability of the labels for each point cloud in the point cloud frame. Further, the present disclosure can obtain semantic segmentation labels and dynamic and static labels of each point cloud in a point cloud frame at the same time through a complete neural network, reducing the time delay between acquiring semantic segmentation labels and dynamic and static labels of point clouds .

According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

FIG. 9 shows a schematic block diagram of an example electronic device 900 that may be used to implement embodiments of the present disclosure. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 9 , the device 900 includes a computing unit 901 that can be executed according to a computer program stored in a read-only memory (ROM) 902 or loaded from a storage unit 907 into a random access memory (RAM) 903. Various appropriate actions and treatments. In the RAM 903, various programs and data necessary for the operation of the device 900 can also be stored. The computing unit 901 , ROM 902 , and RAM 903 are connected to each other through a bus 904 . An input/output (I/O) interface 905 is also connected to the bus 904 .

Multiple components in the device 900 are connected to the I/O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc. ; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 909 allows the device 900 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.

The computing unit 901 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 901 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 executes various methods and processes described above, such as the method for acquiring point cloud tags. For example, in some embodiments, a method for obtaining point cloud tags. may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 908 . In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 900 via the ROM 902 and/or the communication unit 909 . When the computer program is loaded into the RAM 903 and executed by the computing unit 901 , one or more steps of the method for model training or point cloud label acquisition described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured in any other appropriate way (for example, by means of firmware) to execute the method for acquiring point cloud tags.

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

The present disclosure also provides a computer program product, including a computer program. When the computer program is executed by a processor, the above-mentioned method for obtaining point cloud tags is realized.

The present disclosure also provides an automatic driving vehicle, and the automatic driving vehicle may include the electronic device as in the above embodiment, and the electronic device is used to execute the method for acquiring point cloud tags in the above embodiment. The self-driving vehicle is provided with a point cloud acquisition device, and the point cloud frame is collected by the point cloud acquisition device, and the collected point cloud frame can be input into the electronic device, and the electronic device executes the input point cloud frame as in the above-mentioned embodiment The method for obtaining the point cloud label of .

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, each step described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (20)

1. The method for acquiring the point cloud label comprises the following steps:

acquiring M point cloud frames including a current point cloud frame, and respectively projecting point clouds in the M point cloud frames into grids of a bird's eye view, wherein M is an integer greater than or equal to 2;

acquiring target fusion characteristics of the M point cloud frames corresponding to each grid according to the projected aerial view;

aiming at each grid, acquiring a semantic segmentation label and a dynamic and static label of the grid based on target fusion characteristics of the grid;

and carrying out back projection on each grid, determining the grid where the point cloud is located in the current point cloud frame, and determining semantic segmentation labels and dynamic and static labels of the grid where the point cloud is located as labels of the point cloud.

2. The method of claim 1, wherein the obtaining, according to the projected aerial view, the target fusion features of the M point cloud frames corresponding to each grid includes:

aiming at each point cloud frame in the M point cloud frames, acquiring initial characteristic information of the point cloud frames corresponding to each grid according to the projected aerial view of the point cloud frames;

splicing the initial characteristic information of each point cloud frame of the same grid to obtain candidate splicing characteristics;

And processing the candidate splicing characteristics of the grids through a backbone network aiming at each grid to obtain target fusion characteristics of the grids.

3. The method of claim 2, wherein the obtaining initial feature information of the point cloud frame corresponding to each grid according to the projected bird's eye view of the point cloud frame comprises:

acquiring point clouds in each grid in the projected aerial view;

the initial characteristic information of each grid is determined based on the point cloud set of each grid.

4. A method according to claim 3, wherein said determining said initial characteristic information for each grid based on the point clouds of each grid comprises:

acquiring the quantity of point clouds in the point cloud set;

acquiring a height value of point clouds in the point cloud set, and determining a height difference and/or an average height of the point cloud set according to the height value;

and acquiring the reflectivity of the point cloud in the point cloud set, and determining the average reflectivity of the point cloud set according to the reflectivity.

5. The method of any of claims 1-4, wherein for each mesh, obtaining semantic segmentation tags for the mesh based on target fusion features of the mesh, comprises:

Performing semantic segmentation on the target fusion features of the grid to obtain a plurality of semantic segmentation probabilities of the grid;

determining a maximum semantic segmentation probability from the plurality of semantic segmentation probabilities;

and determining the semantic segmentation label corresponding to the maximum semantic segmentation probability as the semantic segmentation label of the grid.

6. The method of claim 5, wherein semantically segmenting the target fusion feature of the mesh to obtain a plurality of semantic segmentation probabilities for the mesh, comprises:

performing first convolution processing on the target fusion feature to obtain a first convolution post-fusion feature;

and performing first probability function mapping on the first convolution fusion characteristic to obtain the semantic segmentation probabilities.

7. The method of any of claims 1-4, wherein for each grid, obtaining an dynamic and static tag for the grid based on target fusion features of the grid, comprises:

performing classification recognition on the target fusion characteristics of the grid to obtain the type recognition probability of the grid;

and comparing the type recognition probability with a preset probability threshold value, and determining the dynamic and static labels of the grid based on a comparison result.

8. The method of claim 6, wherein the classifying the target fusion feature of the grid to obtain a type recognition probability of the grid comprises:

performing second convolution processing on the target fusion feature to obtain a second convolution post-fusion feature;

and performing second probability function mapping on the fusion features after the second convolution to obtain the type recognition probability of the grid.

9. An acquisition device of a point cloud tag, comprising:

the projection module is used for acquiring M point cloud frames including the current point cloud frame, and respectively projecting the point clouds in the M point cloud frames into grids of the aerial view, wherein M is an integer greater than or equal to 2;

the first acquisition module is used for acquiring target fusion characteristics of the M point cloud frames corresponding to each grid according to the projected aerial view;

the second acquisition module is used for respectively acquiring semantic segmentation labels and dynamic and static labels of the grids based on target fusion characteristics of the grids aiming at each grid;

and the third acquisition module is used for carrying out back projection on each grid, determining the grid where the point cloud is located in the current point cloud frame, and determining the semantic segmentation label and the dynamic and static label of the grid where the point cloud is located as the label of the point cloud.

10. The apparatus of claim 9, wherein the first acquisition module is further configured to:

aiming at each point cloud frame in the M point cloud frames, acquiring initial characteristic information of the point cloud frames corresponding to each grid according to the projected aerial view of the point cloud frames;

splicing the initial characteristic information of each point cloud frame of the same grid to obtain candidate splicing characteristics;

and processing the candidate splicing characteristics of the grids through a backbone network aiming at each grid to obtain target fusion characteristics of the grids.

11. The apparatus of claim 10, wherein the first acquisition module is further configured to:

acquiring point clouds in each grid in the projected aerial view;

the initial characteristic information of each grid is determined based on the point cloud set of each grid.

12. The apparatus of claim 11, wherein the first acquisition module is further configured to:

acquiring the quantity of point clouds in the point cloud set;

acquiring a height value of point clouds in the point cloud set, and determining a height difference and/or an average height of the point cloud set according to the height value;

and acquiring the reflectivity of the point cloud in the point cloud set, and determining the average reflectivity of the point cloud set according to the reflectivity.

13. The apparatus of any of claims 9-12, wherein the second acquisition module is further to:

for each grid, carrying out semantic segmentation on target fusion features of the grid to obtain a plurality of semantic segmentation probabilities of the grid;

determining a maximum semantic segmentation probability from the plurality of semantic segmentation probabilities;

and determining the semantic segmentation label corresponding to the maximum semantic segmentation probability as the semantic segmentation label of the grid.

14. The apparatus of claim 13, wherein the second acquisition module is further configured to:

performing first convolution processing on the target fusion feature to obtain a first convolution post-fusion feature;

and performing first probability function mapping on the first convolution fusion characteristic to obtain the semantic segmentation probabilities.

15. The apparatus of any one of claims 9-12, wherein the second acquisition module is further configured to:

aiming at each grid, carrying out classification recognition on the target fusion characteristics of the grid, and obtaining the type recognition probability of the grid;

and comparing the type recognition probability with a preset probability threshold value, and determining the dynamic and static labels of the grid based on a comparison result.

16. The apparatus of claim 15, wherein the second acquisition module is further configured to:

performing second convolution processing on the target fusion feature to obtain a second convolution post-fusion feature;

and performing second probability function mapping on the fusion features after the second convolution to obtain the type recognition probability of the grid.

17. An electronic device, comprising a processor and a memory;

wherein the processor runs a program corresponding to executable program code stored in the memory by reading the executable program code for implementing the method according to claims 1-8.

18. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to claims 1-8.

19. A computer program product comprising a computer program which, when executed by a processor, implements the method according to claims 1-8.

20. An autonomous vehicle comprising the electronic device of claim 17.

Images