CN109118564B

CN109118564B - A three-dimensional point cloud marking method and device based on fused voxels

Info

Publication number: CN109118564B
Application number: CN201810861715.4A
Authority: CN
Inventors: 马燕新; 鲁敏; 涂兵; 郭裕兰; 雷印杰
Original assignee: Shandong Jiayin Information Technology Co ltd
Current assignee: National University of Defense Technology
Priority date: 2018-08-01
Filing date: 2018-08-01
Publication date: 2023-09-19
Anticipated expiration: 2038-08-01
Also published as: CN109118564A

Abstract

The embodiment of the invention provides a three-dimensional point cloud marking method and device based on fusion voxels. The three-dimensional point cloud marking method based on the fusion voxels comprises the steps of carrying out voxelization on a three-dimensional point cloud data set, and carrying out voxel feature extraction in the voxels based on a processing result to form a first voxel feature matrix; taking the first voxel feature matrix as the input of a three-dimensional convolutional neural network to calculate multi-scale features of voxels, and carrying out feature series fusion on the multi-scale features to obtain a second voxel feature matrix; expanding voxel features in the second voxel feature matrix into points in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix; and inputting the point cloud characteristic matrix into a multi-layer sensor to realize attribute marking of the three-dimensional point cloud. The method can realize finer point-by-point classification and identification so as to further improve the point cloud marking performance.

Description

Three-dimensional point cloud marking method and device based on fusion voxels

Technical Field

The invention relates to the technical field of data marking, in particular to a three-dimensional point cloud marking method and device based on fusion voxels.

Background

With the wide application of three-dimensional sensors (such as LiDAR, microsoft Kinect, ASUS Xtion, etc.) in various fields (such as mobile robot, autopilot, remote sensing interpretation, virtual reality, augmented reality, battlefield situation awareness, etc.), three-dimensional data is also growing significantly in recent years, and three-dimensional point cloud labeling is an important means for processing three-dimensional data, which is to identify the category attribute of each point in observed scene point cloud data and assign a unique category label to each point, such as a building, a road, an automobile, etc. However, the traditional semantic labeling based on the voxel convolutional neural network requires that all points in a voxel share the same semantic label, and regular data input is required, and meanwhile, the obtained labeling result is also a rough labeling result at the voxel level.

Disclosure of Invention

In view of the above, the present invention provides a three-dimensional point cloud labeling method and apparatus based on fusion voxels to solve the above-mentioned problems.

In one aspect, a preferred embodiment of the present invention provides a three-dimensional point cloud labeling method based on fused voxels, the method comprising:

voxel processing is carried out on the three-dimensional point cloud data set, and voxel feature extraction is carried out in voxels based on a processing result to form a first voxel feature matrix;

Taking the first voxel feature matrix as the input of a three-dimensional convolutional neural network to calculate multi-scale features of voxels, and carrying out feature series fusion on the multi-scale features to obtain a second voxel feature matrix;

expanding voxel features in the second voxel feature matrix into points in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix;

and inputting the point cloud characteristic matrix into a multi-layer sensor to realize attribute marking of the three-dimensional point cloud.

On the other hand, the preferred embodiment of the invention also provides a three-dimensional point cloud marking device based on the fusion voxels, which comprises:

the voxel processing and feature extraction module is used for carrying out voxelization processing on the three-dimensional point cloud data set and carrying out voxel feature extraction in the voxels based on the processing result to form a first voxel feature matrix;

the multi-scale voxel feature calculation module is used for taking the first voxel feature matrix as the input of a three-dimensional convolutional neural network to calculate multi-scale features of voxels, and carrying out feature series fusion on the multi-scale features to obtain a second voxel feature matrix;

the feature expansion module is used for expanding the voxel features in the second voxel feature matrix to each point in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix;

And the point cloud marking module is used for inputting the point cloud characteristic matrix into the multi-layer sensor so as to mark the attribute of the three-dimensional point cloud.

Compared with the prior art, the three-dimensional point cloud marking method and device based on the fusion voxels, provided by the embodiment of the invention, have the advantages that the multi-scale space is built on the regularized voxel model based on the voxel convolution neural network to extract multi-scale voxel characteristics, then the voxel characteristics are expanded to the point characteristics in a characteristic interpolation mode, and further finer point-by-point classification identification and point cloud marking are realized.

In addition, the invention also utilizes CRF-RNN to optimize the marking result of the marked point cloud data so as to improve the marking precision.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of an application scenario of a three-dimensional point cloud marking device based on fusion voxels according to an embodiment of the present invention.

Fig. 2 is a schematic flow chart of a three-dimensional point cloud labeling method based on fusion voxels according to an embodiment of the present invention.

Fig. 3 is a schematic view of the sub-flow of step S11 shown in fig. 2.

Fig. 4 is a schematic diagram of a three-dimensional point cloud labeling method framework structure based on a voxel convolutional neural network according to an embodiment of the present invention.

Fig. 5 is a schematic structural diagram of a local and global feature fusion module according to an embodiment of the present invention.

Fig. 6 is a schematic diagram of a voxel feature extraction network according to an embodiment of the present invention.

Fig. 7 is a schematic view of the sub-flow of step S12 shown in fig. 2.

Fig. 8 is a schematic diagram of a three-dimensional convolution operation according to an embodiment of the present disclosure.

FIG. 9 is a two-dimensional schematic of convolution and deconvolution provided by an embodiment of the present invention.

Fig. 10 is a schematic diagram of a multi-scale feature extraction structure based on 3D CNN according to an embodiment of the present invention.

Fig. 11 is a schematic diagram of point cloud feature interpolation based on voxel features according to an embodiment of the present invention.

Fig. 12 is a schematic diagram of a full-connection CRF connection according to an embodiment of the present invention.

Fig. 13 is a schematic diagram of a CRF implementation structure based on RNN according to an embodiment of the present invention.

Fig. 14 is a schematic diagram of a three-dimensional point cloud label network structure for fusing voxel convolutional neural network and CRF back-end optimization provided in an embodiment of the present invention.

Fig. 15 is a schematic diagram of a labeling result on an S3DIS dataset based on XYZ coordinate information according to an embodiment of the present invention.

Fig. 16 is a schematic diagram of a labeling result on a vKITTI data set based on XYZ coordinate information according to an embodiment of the present invention.

Fig. 17 is a class confusion matrix for Point-VoxelNet on two data sets provided by an embodiment of the present invention.

Fig. 18 is a class confusion matrix for PVCRF provided in an embodiment of the present invention on two data sets.

Fig. 19 is a schematic block structure diagram of a three-dimensional point cloud labeling device based on fusion voxels according to an embodiment of the present invention.

Icon: 10-an electronic terminal; 100-a three-dimensional point cloud marking device based on fusion voxels; 110-voxel processing and feature extraction module; a 111-voxel dividing unit; 112-a point cloud classifying unit; 113-a point cloud sampling unit; 120-a multi-scale voxel feature calculation module; 130-feature extension module; 140-a point cloud marking module; 200-memory; 300-a memory controller; 400-processor.

Detailed Description

The inventor has found that on the global context information modeling of existing scene point clouds, the representation capability of graph models (graphic models) can be generally utilized to be solved, as a more common way is to combine classifiers with conditional random fields (Conditional Random Fields, CRF) to estimate semantic labels for each data point. However, the classifier classification stage and the CRF optimization stage are usually operated independently as separate modules, and have no interaction with each other, so that information exchange between the modules is limited.

Among these, three-dimensional voxel convolutional neural networks are a good choice for the classifier. The three-dimensional voxel convolution neural network is expanded from a two-dimensional convolution neural network, has good performance in the three-dimensional target classification and identification task, and has the advantages of clear network structure, easy acceleration implementation and the like compared with a depth neural network based on point cloud. However, the voxel neural network requires a regularized data input and its labeling results are also rough labeling at the voxel level.

Aiming at the problems in the prior art, the embodiment of the invention provides a three-dimensional point cloud marking method and a three-dimensional point cloud marking device based on fusion voxels, wherein a multi-scale space is built on a regularized voxel model based on a voxel convolution neural network to extract multi-scale voxel characteristics, and then the voxel characteristics are expanded to point characteristics by utilizing a characteristic interpolation mode, so that finer point-by-point classification and identification are realized, and the marking performance is further improved. For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.

Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.

Fig. 1 is a schematic diagram of an application scenario of a three-dimensional point cloud labeling apparatus 100 based on fusion voxels according to an embodiment of the present invention. The electronic terminal 10 includes, among other things, a fused voxel-based three-dimensional point cloud marking apparatus 100, a memory 200, a memory controller 300, and a processor 400. The electronic terminal 10 may be, but not limited to, an electronic device having a processing function such as a computer or a mobile internet device (mobile Internet device, MID), or may be a server.

Optionally, the memory 200, the memory controller 300, and the processor 400 are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically connected by one or more communication buses or signal lines. The fused voxel based three-dimensional point cloud marking apparatus 100 comprises at least one software functional module that may be stored in the memory 200 in the form of software or firmware or cured in the operating system of the electronic terminal 10. The processor 400 accesses the memory 200 under the control of the memory controller 300 for executing executable modules stored in the memory 200, such as software functional modules and computer programs included in the fused voxel-based three-dimensional point cloud marking apparatus 100.

It is to be understood that the configuration shown in fig. 1 is merely illustrative and that the electronic terminal 10 may also include more or fewer components than those shown in fig. 1 or have a different configuration than that shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.

Further, referring to fig. 2 in combination, the embodiment of the present invention further provides a three-dimensional point cloud labeling method based on a fused voxel, and the three-dimensional point cloud labeling method based on the fused voxel is described below in combination with fig. 2.

Step S11, voxelization processing is carried out on the three-dimensional point cloud data set, and voxel feature extraction is carried out in voxels based on a processing result to form a first voxel feature matrix;

step S12, taking the first voxel feature matrix as the input of a three-dimensional convolutional neural network to calculate multi-scale features of voxels, and carrying out feature series fusion on the multi-scale features to obtain a second voxel feature matrix;

step S13, extending voxel features in the second voxel feature matrix to points in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix;

and S14, inputting the point cloud feature matrix into the multi-layer sensor to realize attribute marking of the three-dimensional point cloud.

In the embodiment, the three-dimensional point cloud is subjected to voxelization firstly, then the feature extraction is performed on the point cloud in the voxels, then the voxels model taking the voxels feature as the element is input into the three-dimensional convolutional neural network to perform multi-scale feature extraction and fusion, and then the feature interpolation algorithm is utilized to expand the voxels feature to the point cloud feature, so that the marking of the three-dimensional point cloud is realized, and the marking precision of the point cloud can be effectively improved.

In detail, referring to fig. 3, the process of voxelization of the point cloud in step S11 may be implemented by the following steps S111-S113:

step S111, dividing a point cloud coordinate space into a plurality of voxels according to a preset voxel size;

step S112, classifying each point in the three-dimensional point cloud data set into a corresponding voxel according to the grid parameters of the voxel;

step S113, sampling the points in each voxel after classification so that the number of points in the voxels reaches a first preset value.

In this embodiment, a point cloud voxelization model is introduced in the steps S111 to S113 to voxelize the point cloud. Specifically, as shown in FIG. 4, the point cloud is voxelized, i.e., in terms of a given voxel sizeThe point cloud coordinate space is segmented into a plurality of voxels. Wherein, the size of the input point cloud in three coordinate axes, namely X, Y, Z axis direction is W, H, E, and the size of each voxel is lambda _W 、λ _H 、λ _E The model after voxelization is of size W' =w/λ _W 、H'＝H/λ _H 、E'＝E/λ _E . In this embodiment, W ', H ', E ' may be integers and powers of 2 in order to facilitate the subsequent convolution operation.

After the voxel grid is implemented in the point cloud coordinate space in step S111, each point in the point cloud may be further classified according to the grid parameters of each voxel, so that each point is assigned to each voxel. However, when the point cloud is classified, the acquired point cloud is often uneven due to the influence of measurement errors, distance, shielding and other factors when the three-dimensional point cloud data is acquired, for example, the point cloud of a part of areas is concentrated, and the point cloud of the part of areas is sparse. In addition, the collection of the point cloud data is equivalent to the sampling of the surface of the target, so that the interior of the target is empty, the point cloud data does not exist, and therefore, after the voxel is formed in the point cloud space, the distribution of the point cloud in each voxel is uneven, as shown in fig. 4, wherein the voxels in the lower left corner do not contain the point cloud, and the voxels in the upper right corner contain fewer points. Therefore, in order to facilitate the subsequent unified voxel feature extraction, the same number of points need to be sampled from each voxel after the point cloud segmentation, such as the first preset value T (T is determined according to the point cloud resolution and the storage capacity).

It should be noted that, when sampling is performed, if the number of points included in a voxel is greater than a first preset value, randomly sampling the first preset value of points from the current voxel so that the number of points in the voxel reaches the first preset value; if the number of points contained in the voxel is smaller than the first preset value, randomly selecting one or more points from the current voxel to copy so that the number of points in the voxel reaches the first preset value. For example, assuming that the first preset value is T, for voxels with the number of points in the voxel exceeding T, randomly sampling T points, for voxels with the number of points in the voxel being less than T, randomly copying the corresponding number of points to obtain a set of T points, performing point cloud blocking and sampling to obtain a set of voxels containing T points, and further performing feature learning by using point cloud data in the voxels to obtain effective feature expression of each voxel containing point cloud.

Further, the step of extracting the point cloud feature in the voxel based on the processing result in the step S11 to form a first voxel feature matrix includes: calculating the center coordinates of the point cloud aiming at the point cloud in each voxel, and carrying out center normalization processing on the point cloud data in the voxels based on the center coordinates to obtain an initial data matrix; inputting the initial data matrix into an LGAB module to realize point-by-point local feature description, and carrying out point-by-point pooling operation on the local feature set in the voxel by adopting maximum pooling to obtain the global feature of the voxel and taking the global feature as a first voxel feature matrix.

Specifically, in the embodiment of the present invention, a local and global feature fusion module (LGAB) as shown in fig. 5 is used to stack to build a feature learning network for voxel sign extraction. Wherein, as shown in FIG. 6, suppose V _x For non-empty voxels containing T points, i.e. V _x ＝{p _i ＝(x _i ,y _i ,z _i (v) i=1, 2,3, …, T), then center normalization is performed before the point cloud data (initial data matrix) is input to the LGAB module, i.e. the center coordinates (c) of the point cloud in the voxel are calculated first _x ,c _y ,c _z ) Using the central coordinates (c _x ,c _y ,c _z ) Performing center normalization on the point cloud data to obtain a final initial data matrix of the input dataThat is, the input to the voxel feature extraction module is a t×6-dimensional initial data matrix.

Further, the stacked LGAB modules can be used for obtaining point-by-point local feature description, and the maximum value Pooling (MP) is used for Pooling the point-by-point feature set in the voxels, so that the global features of the voxels can be obtained. An example of feature extraction for non-empty voxels is given in fig. 6, where the remaining non-empty voxels may share the same network parameters in performing feature extraction in order to reduce the number of parameters. In practical implementation, because the LGAB module can well integrate the local neighborhood sharing information of the point cloud and the respective difference information, in this embodiment, the point cloud information inside the voxel can be effectively extracted by using cascade connection of a plurality of LGAB modules.

Further, as shown in fig. 7, the process of taking the first voxel feature matrix as an input of the three-dimensional convolutional neural network to calculate the multi-scale feature of the voxel in step S12 may be implemented by the following steps.

Step S120, converting the voxel feature matrix into 4-dimensional tensors, and respectively inputting the 4-dimensional tensors into three-dimensional convolution neural networks with convolution kernels of different sizes to calculate voxel features under different scales;

step S121, respectively inputting the voxel features under different scales into three-dimensional deconvolution neural networks with convolution kernels of different sizes to obtain a plurality of voxel features with different scales, wherein the convolution kernel size of each three-dimensional deconvolution neural network during convolution is correspondingly the same as the convolution kernel size of each three-dimensional deconvolution neural network during deconvolution.

In detail, because the space geometric information is important information of the three-dimensional object, the three-dimensional data is directly processed to extract effective feature description of the object, the invention uses the great success of the two-dimensional convolution neural network in image processing to expand the two-dimensional convolution neural network into the three-dimensional convolution neural network, namely, when the three-dimensional convolution neural network is adopted to process the three-dimensional data, the three-dimensional data needs to be subjected to regularization processing, namely, voxel processing, and two-dimensional convolution operation, pooling operation and the like need to be expanded to the three-dimensional voxel data. Wherein, the three-dimensional convolution formula is:

In the formula (1), the components are as follows,for the input three-dimensional voxel data, +.>In the form of a three-dimensional convolution kernel template,for which a response is output. Similar to the two-dimensional convolution operation, the three-dimensional convolution operation simply expands the two-dimensional convolution kernel to the three-dimensional convolution kernel, and includes three dimensions of length, width and height, and correspondingly, as shown in fig. 8, the local sense of the three-dimensional convolution kernel is also converted from a local neighborhood on the two-dimensional plane to a local neighborhood in the three-dimensional space. In practical implementation, the three-dimensional convolution operation reduces the spatial size of the three-dimensional data, i.e. the spatial size of the three-dimensional feature map is smaller than the size of the input voxel data. However, for three-dimensional data labeling, feature information of each data point needs to be acquired, that is, features of each voxel need to be acquired when voxel feature extraction is performed, so that a feature map after a convolution operation needs to be mapped back into an original input voxel. To solve this problem, the present embodiment uses deconvolution (deconvolution) to process the obtained three-dimensional voxel feature map to obtain a feature map of the same size as the input voxel data. The core of deconvolution operation is still convolution operation, but the edge 0-supplementing operation is carried out on the input characteristic data before the convolution operation so as to ensure the size requirement of the output characteristic diagram. For example, FIG. 9 shows a two-dimensional example of a convolution and deconvolution operation, where blue mark data is the input data for both operations, gray mark data is the convolution kernel (the same size), and green mark data is the response output for both. It can be seen that the input to the deconvolution layer is the output of the convolution layer, which has the same size as the input to the convolution layer.

Further, after processing all non-empty voxels based on the foregoing operations, a series of D-dimensional voxel features can be obtained. Since each voxel feature uniquely corresponds to voxel coordinates in three-dimensional space, the acquired feature can be expressed as a 4-dimensional tensor whose size is W ' x H ' x E ' x D, (for empty voxels, a D-dimensional zero vector is used as its feature description). After the feature representation is converted into the feature representation based on the 4-dimensional tensor, a three-dimensional convolutional neural network (3D CNN) can be adopted for further feature optimization. Considering that feature extraction under a fixed scale (convolution kernel size) is insufficient to completely express local neighborhood structure information of voxels, the method adopts multi-scale feature extraction and fusion to extract more abundant local neighborhood information.

As shown in fig. 10, the three-dimensional convolutional neural network structure is a specific three-dimensional convolutional neural network structure, that is, the 3D CNN-based multi-scale feature extraction mainly includes a three-dimensional convolutional operation (Conv 3D), a three-dimensional deconvolution operation (DeConv 3D), and a feature tandem operation (Concat). For the input W ' ×H ' ×E ' ×D dimensional feature, three different convolution kernels were used to perform convolution and deconvolution operations, denoted Conv3D (f _in ；f _out ；ker；st；pad)，DeConv3D(f _in ；f _out The method comprises the steps of carrying out a first treatment on the surface of the A ker; st; pad), where f _in 、f _out Representing the dimension of the input-output characteristic matrix, and the ker; st; the pad represents the size of the convolution kernel, the moving step size of the convolution kernel during convolution and the data filling size during data expansion, and are all three-dimensional vectors. To obtain features at different scales, the three convolution kernels may be, but are not limited to, (1; 2); (2; 1; 2); (2; 1) the convolution uses the same convolution kernel as in the deconvolution operation, in which the corresponding edge-fill 0 operation is performed. In addition, the convolution layer and the deconvolution layer not only comprise convolution and deconvolution operations, but also comprise a normalization layer (Batch Normalization layer, BN) and a ReLU activation operation after each convolution operation.

The 3D CNN-based multi-scale feature extraction adopts different convolution kernels to extract and fuse voxel features along three mutually perpendicular directions (namely X, Y, Z directions) of a three-dimensional space, so that the learned features can contain more local structure information, and more complete expression of point cloud is realized.

Further, in step S13, since the feature description of each point needs to be acquired when the labeling of the three-dimensional point cloud is implemented, and the three-dimensional convolutional neural network just can acquire the feature description of each voxel, the feature description of each point in the input point cloud is acquired by interpolating the voxel features. As shown in FIG. 11 For a given target point p, a nearest preset number (such as 8) of neighborhood voxels in the voxel space formed by the second voxel feature matrix is found, and the feature descriptions corresponding to the neighborhood voxels are respectivelyWhere j=1, 2, …,8, then the target point p is characterized as:

in the formula (2), the amino acid sequence of the compound,representing the center point c in the voxels according to the target point p and the jth neighborhood _j Weight parameter derived from Euclidean distance between them, < ->And representing the voxel characteristics of the jth neighborhood voxel, and repeatedly executing the process on each point in the three-dimensional point cloud, so that the voxel characteristics in the second voxel characteristic matrix can be expanded to each point in the three-dimensional point cloud data set to obtain the point cloud characteristic matrix.

Further, in step S14, the point cloud feature matrix of each point obtained in step S13 is input into a multi-layer perceptron (MLP) to realize point classification, i.e. three-dimensional point cloud labeling, and the specific network structure thereof is shown in fig. 4 again.

According to the actual requirement, in order to further optimize the point cloud marking result obtained by adopting the convolutional neural network and improve the marking precision, the invention further comprises a step S15.

And S15, inputting the three-dimensional point cloud with the completed attribute mark into a CRF-RNN network to perform point cloud attribute mark optimization.

Specifically, in the embodiment, the FC-CRF realized based on the CNN basic operation is embedded into the point cloud marking network based on the three-dimensional convolutional neural network, so that the three-dimensional point cloud fine marking network which is end-to-end and integrates coarse marking and rear end optimization is realized, and the accuracy of the point cloud marking, particularly the smoothness of the target boundary and the contour, is further improved.

In the embodiment, the CRF for the three-dimensional point cloud mark is modeled first, then CRF based on CNN operation is approximately realized, and finally the CRF optimized three-dimensional point cloud mark is fused.

The traditional semantic marks are modeled as point-by-point classification and identification, and the classification and identification are carried out by adopting local features and deep neural networks. However, point-by-point classification usually brings about some clearly unacceptable marking errors, for example, a part of points inside a certain target may be identified as other categories, because point-by-point classification does not consider the point-to-point adjacency relationship, and only uses the neighborhood information of the part and small size of the point to be marked. If the object structure information can be modeled in advance (for example, all objects are continuous, and adjacent points with similar characteristics should be marked as the same kind of objects), and the marking result is optimized and limited based on the modeling result, some obvious errors can be effectively removed, and then the marking result with high precision is obtained. Conditional Random Fields (CRFs) are efficient methods of modeling object continuity and its neighborhood structure information, and have been widely used in two-dimensional image labeling. Wherein the conditional random field is a model for calculating the conditional probability distribution of a given set of random variables for another set of output random variables, the main feature of which is to assume that the output variables constitute a markov random field (Markov Random Field, MRF).

In detail, the CRF is a discriminant probability undirected graph model, can model global context information and cross characteristics in data, and is a probability graph model capable of well processing sequence data segmentation and marking. Let x= { X given a set of random variables ₁ ,X ₂ ,…,X _N And p= { P ₁ ,P ₂ ,…,P _N (wherein X is _i ∈L＝{l ₁ ,l ₁ ,…,l _M For a three-dimensional point cloud label, P is an input point cloud containing N points, P _j For the observation vector of the j-th point, X is the semantic mark of the input point cloudAs a result, X _i For the semantic label of the ith point, which takes one of M semantic labels, the corresponding CRF model can be represented by Gibbs probability distribution, namely:

in the formula (3), G is a probability undirected graph constructed on a random variable set X, _O is a group in the graph G, wherein, _ο each pair of nodes in the network is adjacent, _OG then it is the set of all groups in G, Z (P) is a normalization function, Λ (x _ο P) is an energy function on a radical, also known as a potential function.

For any marking result x ε L ^N The overall potential function of (2) is:

solving based on the maximum posterior probability algorithm to obtain an optimal marking result:

from the foregoing optimal solution, it can be seen that the maximization of the posterior probability of the labeling result, i.e., the minimization of the overall potential function. It can be seen that the conditional random field firstly realizes modeling of local context information through the group potential function, and then utilizes the graph structure to transfer the context information, so that modeling of the context information in a large range is realized.

For the fully connected CRF model, each node in graph G is connected to the rest of the nodes, as shown in FIG. 12, corresponding groups _O As a group containing a single node or containing paired nodes, the overall potential function corresponding to x can be expressed as:

wherein, for convenience of description, the condition P part in the conditional posterior probability is removed in the formula (6), that is, Λ (x) =Λ (x|p),as a unit potential function>I, j=1, 2, …, N as a pairwise binary potential function. Elementary potential function->Indicating that the ith node in graph G is marked as x _i At the cost, the function is usually defined by the probability output of a discriminant classifier, the estimation result at this time often contains more noise points, and the segmentation result often is discontinuous at the edge of the target. The paired binary potential function gives the simultaneous marking of the ith observation point and the j observation points as x _i 、x _j The cost of the method is that the method has the effects of keeping the consistency of marks of adjacent observation points and reducing inconsistency, and the smoothness of marking results can be improved.

The idea of gaussian weighting is used to define a pair binary potential function as:

in equation (7), the function ψ (x) _i ,x _j ) Calculating a function, w, for consistency between different labels ^(m) As the weight value of the weight value,for a gaussian kernel based smoothing filter function, there is a total of M _G A Gaussian kernel function f _i ,f _j Feature vectors representing observation points i, j, respectively, and have +.>Each Gaussian kernel function->Can pass through a symmetrical positive definite matrix lambda ^m Is defined. So far, the full-connection CRF modeling for the three-dimensional point cloud mark is completed, and the next step is how to solve to obtain the optimal mark result.

The three-dimensional point cloud label optimization process based on the full-connection CRF is a process of maximizing posterior probability phi (X) based on input point cloud data. Solving the accurate posterior probability is difficult and has huge calculation amount, and the average field-based approximation method can convert the posterior probability phi (X) into a series of mutually independent products of marginal probabilities, namely phi (X) and theta (X) and II _i＝1 Θ _i (X _i ) Any labeling result x can be obtained by combining formulas (3) - (7) _i Is (are) marginal probability theta _i The method comprises the following steps:

based on equation (8), an iterative inference algorithm for CRF can be constructed, as shown in algorithm 5.1. The convergence of the iterative algorithm is mainly measured by the difference between the estimated Q and the estimated P, and can be obtained by evaluating the convergence of the algorithm, and when the iteration number is 10, the estimation error is small, so that the algorithm has good convergence.

Algorithm 1: CRF iterative inference algorithm based on average field approximation

1. Initializing: initializing all nodes

While is convergence do

3. Information transfer: calculate all Gaussian filter results

4. And (5) weighted filtering:

5. and (3) consistency detection:

6. adding a unitary potential function:

7. normalization:

8.end while

how the above algorithm can be implemented by using correlation operations in the CNN to iterate the CRF is described below. The biggest problem in reconstructing the algorithm based on CNN operation is that whether the back propagation of errors can be realized or not, namely, the BP algorithm is adopted for parameter learning training.

(1) Initializing operations

The initialization operation in algorithm 1 is:

wherein,,i.e. all values may be summed. Recording deviceThere is->Z _i ＝∑ _l exp(U _i (l) It can be seen that this operation is equivalent to marking the results U for all possible sites _i (l) An activation operation based on the Softmax function is performed. The Softmax function is a commonly used activation function in CNN networks, which does not contain any parameters, whose error derivatives can be counter-propagatedThus, the Back-Propagation (BP) algorithm can also be utilized for learning training.

(2) Information transfer

As shown in algorithm 1, the information transfer in CRF is performed by M _G A Gaussian filter pair Θ _j Smoothing filtering is performed. The kernel function of the Gaussian filter is obtained according to the characteristics of the point cloud, such as coordinate information or color and intensity information of points, and expresses the association relation between the points of each scene. In the fully-connected CRF model, each filter needs to cover all points in the point cloud, and the data volume and the calculation amount of each filter are large, so that each filter cannot be directly implemented. The method (permutohedral laTTice) based on the full-freedom polyhedral lattice is adopted to realize the rapid Gaussian convolution, the calculated amount is O (N), N is the number of points for filtering, and compared with the traditional Gaussian convolution, the method has the advantages of higher speed and better filtering effect. The fast gaussian convolution based on the full-degree-of-freedom polyhedral lattice method comprises four stages, namely a polyhedral lattice construction stage, an extended (splat) mapping, a slice (slice) mapping, a fuzzy (blur) stage and the like.

In back propagation, the input (error derivative) of the current convolution layer is the output of the previous filter passing M in the opposite direction _G And outputting results after the Gaussian filters. In gaussian convolution based on the full-degree-of-freedom polyhedral lattice method, this back propagation can be achieved by reversing the order of filters in the blur stage on the basis of maintaining the same polyhedral lattice construction, expansion mapping and slice mapping as in the forward propagation. The calculated amount of the implementation method is still O (N), so that the calculated amount is obviously reduced, and the calculation efficiency is improved.

(3) Weighted filtering

The following calculation is the aforementioned M for each semantic label/ _G The output results are weighted and summed. In the point cloud label, each semantic label is independent, so that the weighted filtering operation can be realized through M _G A convolution operation with a convolution kernel of l×l is implemented, wherein the input of the convolution operation is a convolution operation containing M _G The feature matrix of each channel is output as a feature matrix containing l channels. In the back propagation, due to thisThe input and output of the step operation are known, and the convolution kernels are mutually independent, so that error derivatives of the convolution kernel parameters and the input data can be calculated, and further the BP algorithm can be utilized to learn and train the convolution kernel parameters.

(4) Consistency detection

In the consistency detection, compatibility calculation is carried out on the output results of different labels in the last step by using a PoTTs model. The compatibility calculation is mainly to compare whether the labels of two similar observation points are the same, wherein the consistency detection is 0 when the semantic labels of the two points are the same, and the penalty item sigma is introduced when the semantic labels of the two points are different, and the calculation is as follows:

compared to using a fixed penalty term σ, the present invention contemplates using a penalty value learned based on data because the degree of association between different labels is different and thus the effect of marking different labels at adjacent points on the overall marking result is different. Therefore, the consistency detection can also be regarded as a convolution layer, the number of input and output channels of the layer is M (label number), the convolution kernel size is l×l, and the learned neuron connection weight parameter is the value of the conversion function. This step is also back-propagated, since it is implemented using a basic convolution operation.

(5) Increasing the unitary potential function

By applying a unitary potential functionAnd combining the output result obtained in the consistency detection element by element to obtain a complete potential function result. In the step of adding the unitary potential function, no parameter is included, so that the error in the output can be simply copied to the input end to realize back propagation.

(6) Normalization

Similar to the initialization procedure, the normalization step can also be implemented by an activation operation based on a Softmax function, whose back propagation is consistent with the back propagation in CNN based on the Softmax function. So far, the basic operation in the CNN network is adopted to realize each step of single iteration in the algorithm 1, and the steps are stacked to realize the solving algorithm of multiple iterations.

Based on the above description, in this embodiment, the CRF model is first approximately modeled by using an average field approximation method, and then each step in the average field approximation method is equivalently implemented by using the basic operation in the CNN, that is, the average field approximation algorithm of a single iteration is implemented. For the iterative average field approximation method, only related steps are needed to be stacked, namely iterative average field approximation calculation can be realized by adopting a recursive CNN structure (RNN), the structure is shown in fig. 13, the given input point cloud data is P, and the point-by-point unitary potential function is U=U _i (l) The marginal probability obtained in the previous iteration is H ₁ The marginal probability obtained by the current iteration is H ₂ The single average field approximation is denoted as f _Ω (U,P,H ₁ ) Ω is its parameter set (including all parameters in weighted filtering and consistency detection) and Ω= { w ^(m) ψ (l, l'). For H ₁ Initializing the output as a softmax function with U as input at the beginning of the iteration, equal to H in subsequent iterations ₂ The outputs of (1) are:

wherein T' is the number of iterations.

After obtaining H ₁ Then H is performed based on the average field approximation algorithm ₂ Namely:

H ₂ (t')＝f _Ω (U,P,H ₁ (t')),0<t'≤T'

for output Y, the estimation result is output only at the last iteration, i.e., y=h ₂ (T')。

Based on the above analysis, it can be known that the error derivative about the parameter in the whole network structure (denoted CRF-RNN) is available, so that it can be solved by using standard BP algorithm, and thus can be embedded into other neural networks for learning training.

Further, a three-dimensional Point cloud labeling network (Point voxelnet+crf-RNN, PVCRF) fused with the three-dimensional voxel convolution neural network and CRF back-end optimization is constructed based on the aforementioned Point cloud labeling network Point-VoxelNet network, and the specific structure is shown in fig. 14, wherein, for an input Point cloud, voxelNet is first voxelized according to the scene size of the input Point cloud and a fixed number of Point clouds are randomly selected in the voxels for subsequent feature extraction, then feature extraction is performed in the voxels based on the LGAB module to obtain simple voxel features, after all non-empty voxel features are obtained (the features of empty voxels adopt the complementary 0 operation), multi-scale voxel feature extraction and fusion are performed based on the three-dimensional convolution neural network (Conv 3D, deConv 3D), then the multi-scale voxel features are expanded to all points by an interpolation method to obtain Point-by-Point feature, then the Point feature is input into a multi-layer perceptron to obtain a preliminary Point cloud labeling result, and finally back-end optimization is performed based on the CRF-n network structure. The network structure well realizes information interaction between the classification and identification stage and the CRF optimization stage in the point cloud marking, and has obvious effect on improving the precision of the point cloud marking.

Based on the description of the three-dimensional point cloud marking method based on the fusion voxels, the inventor also verifies the performance of the three-dimensional point cloud marking method based on the fusion voxels, such as evaluation indexes of point set intersection ratio (Intersection over Union, ioU), overall Accuracy (OA) and the like, and evaluates the point cloud marking performance.

(1) Network implementation and parameter setting

In the point cloud data rasterization and sampling stage, different data sets need to be processed differently.

S3DIS: for the S3DIS dataset, the maximum size range of the scene along the Z, Y, X axis direction is e=8m, h=16m, and w=50m, respectively. To cover the entire scene, the size of the entire grid is 8×16×50, and the size of each voxel is λ _E ＝0.5m、λ _H ＝0.25m、λ _W The voxel model sizes constructed were E ' =16, h ' =64, w ' =256, with more voxels being empty, =0.2m. T=32 points were chosen for each voxel.

vKITTI: for the vKITTI data set, the maximum size ranges of the scenes along the Z, Y, X axis direction are E=33m, H=193 m and W=148 m respectively. Here, the size of each voxel is set to lambda _E ＝2m、λ _H ＝1.6m、λ _W The voxel model sizes constructed are E ' =16, h ' =128, w ' =128, =1.2m. Likewise, t=32 points are selected in each voxel.

For the CRF-RNN network, to prevent overfitting and gradient lapse, the number of iterations is set to T '=5 in the training phase, and T' =10 in the testing phase. The gaussian filter size is consistent with the point cloud data size.

The invention adopts a strategy of two-step training, wherein the first step is to train the Point-VoxelNet independently, and the second step is to finely tune the PVCRF of the network combining the Point-VoxelNet and the CRF-RNN. In the Point-VoxelNet network, an Adam optimization algorithm with a dynamic value of 0.9 is adopted for training and optimization, the initial learning rate is 0.001, and the training batch size is 16. For the PVCRF network, an Adam optimization algorithm with a dynamic value of 0.6 is adopted for training and optimization, the initial learning rate is 0.0001, and the training batch size is 16. During training, an early-stop strategy is also employed to obtain optimal network parameters, the maximum number of training rounds being 100, and training is stopped if the network parameters remain un-updated after 10 consecutive rounds of training. In the test phase, the algorithm is verified by adopting 6-fold cross verification, and the grouping situation of the training data and the test data is shown in table 1.

The proposed three-dimensional voxel neural network structure and CRF-RNN network structure are realized by adopting a Tensorflow deep learning framework based on Python language. The experimental hardware environment is as follows: intel Core i76700KCPU,48G memory, GTX 1080Ti graphic card (supporting CUDA 8.0, cuDNN 5.1).

(2) Quantitative result analysis

Based on the two data sets, the application effect of the deep neural network model (Point VoxelNet and PVCRF) in the three-dimensional Point cloud marking is compared, and the comparison analysis is carried out with the currently better marking algorithms PointNet, MS+CU (2), SEGCloud and 3 DContextNet. The rectangular coordinate information of the point cloud, namely the XYZ coordinates, is only adopted for processing.

TABLE 1

TABLE 2

The statistics marking results of different network models on two small data sets are shown in tables 1 and 2, wherein the PVCRF model provided by the embodiment obtains better average IoU on S3DIS and vKITTI data sets, namely 51.8% and 39.1% respectively, and obtains better results on the overall marking accuracy OA, namely 81.2% and 82.6% respectively, which shows that the PVCRF model can better acquire the complete feature expression of the point cloud, and proves that the multi-scale feature extraction based on the three-dimensional voxel space has equivalent functions to the multi-scale feature extraction based on the European space, and can provide the detail information of the point cloud scene for the three-dimensional point cloud marking.

Comparing PVCRF with SEGCLloud, SEGCLloud, directly adopting binary voxel model to make voxel feature learning, using rasterized point cloud voxel model for PVCRF, and utilizing point cloud in the voxels to make voxel feature learning. Both add back-end optimization based on fully connected CRF model. In addition, the PVCRF comprises a multi-scale feature extraction module based on a three-dimensional voxel space, so that the PVCRF obtains higher average IoU, and the processing by utilizing the point cloud data in the voxels and the feature description with stronger characterization capability can be extracted by the multi-scale feature extraction module are fully described.

Comparing the Point-VoxelNet with the PVCRF model, the PVCRF achieves better performance in both average IoU and overall accuracy OA, because the fully connected CRF model can model a larger range of context information, and has stronger characterization capability for Point cloud adjacencies.

On different data sets, taking PointNet as a benchmark, because the point clouds in the S3DIS data set are concentrated, the point cloud density is generally higher, the network model PVCRF provided by the embodiment obtains larger performance improvement, and in the vKITTI data set, the performance improvement is smaller because the point cloud distribution is generally sparse.

The marking results obtained by the different algorithms on the S3DIS and vKITTI data sets and the true marking results are shown in fig. 15 and 16. The steps are as follows from left to right: and inputting a color point cloud, and truly marking the result based on the marking result of PointNet, point-VoxelNet, PVCRF. As can be seen from the labeling results in fig. 15 and 16, the results obtained based on PVCRF are better than the points net and the Point-VoxelNet, and the labeling results are closer to the real results, thus proving the effectiveness of multi-scale feature learning and CRF back-end optimization in PVCRF. Compared with PointNet and Point-VoxelNet, the marking result obtained based on Point-VoxelNet is superior to PointNet, which is mainly because the introduction of multi-scale feature learning in Point-VoxelNet improves the feature learning capability of the network model.

In indoor and outdoor scenarios, the Point-VoxelNet, PVCRF model is still difficult to separate for objects with high overlap or tight connection, such as the wall panel (board) and window (window) in fig. 15, the road (road) and terrain (terrain) in fig. 16. Because the Point-VoxelNet and PVCRF models both consider multi-scale feature extraction, objects larger than the object size in the scene can be marked well, such as a table in fig. 15 and a building in fig. 16.

Compared with Point-VoxelNet and PVCRF, the full-connection CRF can extract large-scale context information in the scene Point cloud, so that details in the labeling result are more prominent, as shown by the segmentation result of chair (chair) and sofa (sofa) in the first row of fig. 15, and road (road) and terrain (terrain) in fig. 16.

(4) Category confusion matrix analysis

Fig. 17 and fig. 18 respectively show category confusion matrices obtained by two network models of Point-VoxelNet and PVCRF on the S3DIS and vKITTI data sets, wherein the numerical values in the matrix grids are category marking accuracy, and the grid colors also represent the accuracy. Comparing the results of the two results can find that for the indoor data set S3DIS, the segmentation precision of the Point-VoxelNet model and the PVCRF model on various targets is equivalent, and the introduction of the CRF model mainly reduces confusion between tables and floors and the like. For the outdoor data set vKITTI, the introduction of the CRF model mainly reduces the confusion between buildings (building) and trucks (van) and the like.

As can be seen from fig. 17 (a), in the S3DIS data set, the recognition accuracy of the Point-VoxelNet network model on ceilings (ceilings), floors (floor), doors (door), columns (columns), beams (beams), windows (windows), bookcases (book case), wallboards (board) and chairs (chair) is above 52%, the accuracy of the marks is low, namely sofas (sofa), walls (wall) and shadows (cluTTer), the accuracy is between 35% and 46%, and the worst of 13 targets is tables (table), and the accuracy is 22%. The average class accuracy obtained by PVCRF on S3DIS dataset is similar to the Point-VoxelNet network model, but the general accuracy is improved, wherein the accuracy of the table is improved to 30%, as shown in fig. 18 (a).

Similar comparisons are also found in vKITTI data sets, as shown in FIGS. 17 (b) and 18 (b). This comparison demonstrates the accurate knowledge and modeling of the CRF model for the target details in the scene. Comparing the different results in the two data sets in fig. 17 and fig. 18, the marking precision on the vKITTI data set is generally lower, mainly because the point cloud in the vKITTI data set is generally sparse and uneven relative to the point cloud data in the S3DIS data set, the neighborhood structure of the point is not obvious, and thus, the information extraction is insufficient, and the realization of high-precision point cloud marking is not facilitated.

(5) Calculation time statistical analysis

Experimental analysis was performed on the calculation efficiency of the different algorithms on the S3DIS dataset. The S3DIS dataset contains 272 independent point clouds, each point cloud is labeled by using different algorithms, and the average test time (only the time consumed by the neural network calculation and the time consumed by the data preparation stage are not counted) is counted, and the statistical result is shown in table 3. As can be seen from the statistics in Table 3, the calculation time of PointNet is about 1.8s. The time consumed by the calculation of the PVCRF, which is a network model for fusing the three-dimensional voxel convolutional neural network and the back-end optimization of the CRF, is maximum and is 4.52s, and the calculated amount is obviously increased due to the introduction of the CRF.

TABLE 3 Table 3

And finally, further optimizing by adopting FC-CRF realized based on convolutional neural network to obtain a fine marking result.

Further, referring to fig. 19 in combination, the page building apparatus 100 provided in the embodiment of the present invention includes a voxel processing and feature extraction module 110, a multi-scale voxel feature calculation module 120, a feature expansion module 130, and a point cloud labeling module 140.

The voxel processing and feature extracting module 110 is configured to voxel a three-dimensional point cloud data set, and perform voxel feature extraction in a voxel based on a processing result to form a first voxel feature matrix; in this embodiment, the step S11 may be performed by the voxel processing and feature extraction module 110, that is, the specific description of the voxel processing and feature extraction module 110 may refer to the step S11, and the description of this embodiment is omitted herein. Alternatively, as shown in fig. 19, in the present embodiment, the voxel processing and feature extraction module 110 includes a voxel dividing unit 111, a point cloud classifying unit 112, and a point cloud sampling unit 113.

A voxel dividing unit 111, configured to divide the point cloud coordinate space into a plurality of voxels according to a preset voxel size; in this embodiment, the step S111 may be performed by the voxel dividing unit 111, that is, the specific description of the voxel dividing unit 111 refers to the step S111, and the embodiment is not repeated here.

A point cloud classifying unit 112, configured to classify each point in the three-dimensional point cloud dataset into a corresponding voxel according to a grid parameter of the voxel; in this embodiment, the step S112 may be performed by the point cloud classifying unit 112, that is, the specific description of the point cloud classifying unit 112 may refer to the step S112, and the description of this embodiment is omitted herein.

And the point cloud sampling unit 113 is configured to sample the points in each voxel after classification so that the number of points in the voxels reaches a first preset value. In this embodiment, the step S113 may be performed by the point cloud sampling unit 113, that is, the specific description of the point cloud sampling unit 113 may refer to the step S113, and the description of this embodiment is omitted herein.

The multi-scale voxel feature calculation module 120 is configured to take the first voxel feature matrix as an input of the three-dimensional convolutional neural network to calculate a multi-scale feature of the voxel, and perform feature series fusion on the multi-scale feature to obtain a second voxel feature matrix; in this embodiment, the step S12 may be performed by the multi-scale voxel feature calculation module 120, that is, the specific description of the multi-scale voxel feature calculation module 120 refers to the step S12, and the description of this embodiment is omitted herein.

The feature expansion module 130 is configured to expand voxel features in the second voxel feature matrix into points in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix; in this embodiment, the step S13 may be performed by the feature expansion module 130, that is, the specific description of the feature expansion module 130 refers to the step S13, and the description of this embodiment is omitted here.

The point cloud marking module 140 is configured to input the point cloud feature matrix into the multi-layer sensor to implement attribute marking of the three-dimensional point cloud. In this embodiment, the step S14 may be performed by the point cloud marking module 140, that is, the specific description of the point cloud marking module 140 refers to the step S14, and the description of this embodiment is omitted herein.

In summary, the three-dimensional point cloud marking method and device based on the fused voxels provided by the embodiment of the invention are characterized in that a multi-scale voxel characteristic is extracted by constructing a multi-scale space based on a voxel convolution neural network on a regularized voxel model, and then the voxel characteristic is expanded to a point characteristic by utilizing a characteristic interpolation mode, so that finer point-by-point classification identification and point cloud marking are realized.

In the description of the present invention, the terms "configured," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art. In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. The apparatus and method embodiments described above are merely illustrative, for example, flow charts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to preset numbers of embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code. The modules, segments, or portions of code contain one or a predetermined number of logic functions for achieving the specified functions.

It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional point cloud labeling method based on fusion voxels, the method comprising:

inputting the point cloud feature matrix into a multi-layer sensor to realize attribute marking of the three-dimensional point cloud;

the step of extracting the point cloud features in the voxels based on the processing results to form a first voxel feature matrix comprises the following steps:

calculating the center coordinates of the point cloud aiming at the point cloud in each voxel, and carrying out center normalization processing on the point cloud data in the voxels based on the center coordinates to obtain an initial data matrix;

inputting the initial data matrix into an LGAB module to realize point-by-point local feature description, and carrying out point-by-point pooling operation on a local feature set in a voxel by adopting maximum pooling to obtain a global feature of the voxel and taking the global feature as a first voxel feature matrix;

the step of expanding the voxel features in the second voxel feature matrix into points in the three-dimensional point cloud data set based on a feature interpolation algorithm comprises the following steps:

Each point in the three-dimensional point cloud data set is used as a target point, and a preset number of voxels closest to the target point are searched in a voxel space formed by the second voxel characteristic matrix to be used as neighborhood voxels;

and performing extension calculation on the voxel characteristics of the neighborhood voxels to obtain the point cloud characteristics of the target point.

2. The method for labeling a three-dimensional point cloud based on fused voxels according to claim 1, wherein the step of voxelizing the point cloud comprises:

dividing the point cloud coordinate space into a plurality of voxels according to the preset voxel size;

classifying each point in the three-dimensional point cloud data set into a corresponding voxel according to the grid parameters of the voxel;

and sampling the points in each voxel after classification so that the number of the points in the voxels reaches a first preset value.

3. The fused voxel-based three-dimensional point cloud labeling method of claim 2, wherein the step of sampling points in each of the categorized voxels comprises:

if the number of the points contained in the voxel is larger than a first preset value, randomly sampling the first preset value points from the current voxel to enable the number of the points in the voxel to reach the first preset value;

If the number of points contained in the voxel is smaller than a first preset value, randomly selecting one or more points from the current voxel to copy so that the number of points in the voxel reaches the first preset value.

4. The method of claim 1, wherein the step of using the first voxel feature matrix as an input to a three-dimensional convolutional neural network to calculate multi-scale features of voxels comprises:

converting the voxel characteristic matrix into 4-dimensional tensors, and respectively inputting the 4-dimensional tensors into three-dimensional convolution neural networks with convolution kernels of different sizes to calculate voxel characteristics under different scales;

and respectively inputting the voxel features under different scales into three-dimensional deconvolution neural networks with convolution kernels of different sizes to obtain a plurality of voxel features with different scales, wherein the convolution kernel size of each three-dimensional deconvolution neural network during convolution is correspondingly the same as the convolution kernel size of each three-dimensional deconvolution neural network during deconvolution.

5. The fusion voxel-based three-dimensional point cloud labeling method of claim 1, wherein the point cloud features F of the target point p _p Is thatWherein (1)>Representing the center point c in the voxels according to the target point p and the jth neighborhood _j Weight parameter obtained from Euclidean distance between F _v ^j Representing the voxel characteristics of the jth neighborhood voxel.

6. The fused voxel-based three-dimensional point cloud labeling method of claim 1, further comprising:

and inputting the three-dimensional point cloud with the completed attribute mark into a CRF-RNN network to perform point cloud attribute mark optimization.

7. A three-dimensional point cloud marking device based on fused voxels, the device comprising:

The point cloud marking module is used for inputting the point cloud characteristic matrix into the multi-layer sensor so as to mark the attribute of the three-dimensional point cloud;

the voxel processing and feature extraction module is specifically configured to calculate a center coordinate of a point cloud in each voxel for the point cloud, and perform center normalization processing on point cloud data in the voxel based on the center coordinate to obtain an initial data matrix; inputting the initial data matrix into an LGAB module to realize point-by-point local feature description, and carrying out point-by-point pooling operation on a local feature set in a voxel by adopting maximum pooling to obtain a global feature of the voxel and taking the global feature as a first voxel feature matrix;

the feature expansion module is specifically configured to, for each point in the three-dimensional point cloud data set, respectively serve as a target point, and search a voxel space formed by the second voxel feature matrix for a preset number of voxels closest to the target point to serve as neighborhood voxels; and performing extension calculation on the voxel characteristics of the neighborhood voxels to obtain the point cloud characteristics of the target point.

8. The fused voxel-based three-dimensional point cloud labeling apparatus of claim 7, wherein the voxel processing and feature extraction module comprises:

The voxel dividing unit is used for dividing the point cloud coordinate space into a plurality of voxels according to the preset voxel size;

the point cloud classifying unit is used for classifying each point in the three-dimensional point cloud data set into a corresponding voxel according to the grid parameters of the voxel;

and the point cloud sampling unit is used for sampling the points in each voxel after classification so that the number of the points in the voxels reaches a first preset value.