CN109118564A - A kind of three-dimensional point cloud labeling method and device based on fusion voxel - Google Patents

Abstract

The embodiment of the present invention provides a kind of three-dimensional point cloud labeling method and device based on fusion voxel.Wherein, the three-dimensional point cloud labeling method based on fusion voxel includes voxelization processing being carried out to three dimensional point cloud collection, and carry out voxel feature extraction in voxel based on processing result and form the first voxel eigenmatrix；First voxel eigenmatrix is calculated to the Analysis On Multi-scale Features of voxel as the input of Three dimensional convolution neural network, and feature fused in tandem is carried out to obtain the second voxel eigenmatrix to the Analysis On Multi-scale Features；The voxel feature in the second voxel eigenmatrix is extended in each point of three dimensional point cloud concentration to obtain a cloud eigenmatrix based on feature interpolation algorithm；It will be marked in cloud eigenmatrix input multilayer perceptron with attribute of the realization to three-dimensional point cloud.The present invention can be realized point-by-point more sophisticated category identification, with further hoist point cloud labeling properties.

Description

A three-dimensional point cloud labeling method and device based on fusion voxels

technical field

The present invention relates to the technical field of data labeling, and in particular, to a method and device for labeling a three-dimensional point cloud based on fusion voxels.

Background technique

With the wide application of 3D sensors (such as LiDAR, Microsoft Kinect, ASUS Xtion, etc.) in various fields (such as mobile robots, autonomous driving, remote sensing interpretation, virtual reality, augmented reality and battlefield situational awareness, etc.) in recent years, 3D data It is also growing significantly, and 3D point cloud tagging, as an important means of processing 3D data, refers to identifying the category attribute of each point in the observed scene point cloud data, and assigning a unique category to each point. Labels, such as buildings, roads, or cars. However, the traditional semantic labeling based on voxel convolutional neural network requires all points in the voxel to share the same semantic label, and requires regularized data input, and the obtained labeling result is also a rough labeling result at the voxel level.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a fused voxel-based three-dimensional point cloud marking method and device to solve the above problems.

On the one hand, a preferred embodiment of the present invention provides a fusion voxel-based three-dimensional point cloud labeling method, the method comprising:

Perform voxelization processing on the 3D point cloud data set, and perform voxel feature extraction in the voxel based on the processing result to form a first voxel feature matrix;

The first voxel feature matrix is used as the input of the three-dimensional convolutional neural network to obtain the multi-scale feature of the voxel, and the multi-scale feature is subjected to feature series fusion to obtain the second voxel feature matrix;

Extending the voxel features in the second voxel feature matrix to each point in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix;

The point cloud feature matrix is input into the multi-layer perceptron to realize the attribute labeling of the three-dimensional point cloud.

On the other hand, a preferred embodiment of the present invention also provides a three-dimensional point cloud marking device based on fusion voxels, the device comprising:

The voxel processing and feature extraction module is used to voxelize the three-dimensional point cloud data set, and perform voxel feature extraction in the voxel based on the processing result to form a first voxel feature matrix;

The multi-scale voxel feature calculation module is used to use the first voxel feature matrix as the input of the three-dimensional convolutional neural network to calculate the multi-scale features of the voxels, and perform feature series fusion on the multi-scale features to obtain the first voxel feature matrix. Two-voxel feature matrix;

A feature extension module, configured to extend the voxel feature in the second voxel feature matrix to each point in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix;

The point cloud labeling module is used to input the point cloud feature matrix into the multi-layer perceptron to realize the attribute labeling of the three-dimensional point cloud.

Compared with the prior art, the embodiment of the present invention provides a method and device for labeling a 3D point cloud based on fusion voxels, by constructing a multi-scale space based on a voxel convolutional Multi-scale voxel features, and then use feature interpolation to extend voxel features to point features, so as to achieve point-by-point finer classification and identification and point cloud marking.

In addition, the present invention also utilizes CRF-RNN to optimize the labeling result of the marked point cloud data, so as to improve the labeling accuracy.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 is a schematic diagram of an application scenario of a fused voxel-based three-dimensional point cloud labeling device according to an embodiment of the present invention.

FIG. 2 is a schematic flowchart of a method for labeling a 3D point cloud based on fusion voxels according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a sub-flow of step S11 shown in FIG. 2 .

FIG. 4 is a schematic diagram of a framework structure of a method for labeling a three-dimensional point cloud based on a voxel convolutional neural network according to an embodiment of the present invention.

FIG. 5 is a schematic structural diagram of a local and global feature fusion module provided by an embodiment of the present invention.

FIG. 6 is a schematic structural diagram of a voxel feature extraction network provided by an embodiment of the present invention.

FIG. 7 is a schematic diagram of a sub-flow of step S12 shown in FIG. 2 .

FIG. 8 is a schematic diagram of a three-dimensional convolution operation provided by an embodiment of the present invention.

FIG. 9 is a two-dimensional schematic diagram of convolution and deconvolution according to an embodiment of the present invention.

FIG. 10 is a schematic structural diagram of a 3D CNN-based multi-scale feature extraction provided by an embodiment of the present invention.

FIG. 11 is a schematic diagram of point cloud feature interpolation based on voxel features according to an embodiment of the present invention.

FIG. 12 is a schematic diagram of a fully connected CRF connection provided by an embodiment of the present invention.

FIG. 13 is a schematic structural diagram of an RNN-based CRF implementation provided by an embodiment of the present invention.

FIG. 14 is a schematic structural diagram of a three-dimensional point cloud labeling network optimized by fusing a voxel convolutional neural network and a CRF back-end according to an embodiment of the present invention.

FIG. 15 is a schematic diagram of a labeling result on an S3DIS dataset based on XYZ coordinate information provided by an embodiment of the present invention.

FIG. 16 is a schematic diagram of a labeling result on the vKITTI dataset based on XYZ coordinate information provided by an embodiment of the present invention.

FIG. 17 is a class confusion matrix of Point-VoxelNet on two datasets provided by an embodiment of the present invention.

FIG. 18 is a class confusion matrix of PVCRF on two data sets provided by an embodiment of the present invention.

FIG. 19 is a schematic block diagram of a 3D point cloud labeling device based on fusion voxels according to an embodiment of the present invention.

Icons: 10 - electronic terminal; 100 - 3D point cloud labeling device based on fusion voxels; 110 - voxel processing and feature extraction module; 111 - voxel division unit; 112 - point cloud classification unit; 113 - point cloud sampling unit; 120-multi-scale voxel feature calculation module; 130-feature expansion module; 140-point cloud marking module; 200-memory; 300-storage controller; 400-processor.

Detailed ways

The inventor's research found that in the global context information modeling of the existing scene point cloud, the representation ability of graphical models can usually be used to solve the problem. For example, the more common way is to combine the classifier with the conditional random field ( Conditional Random Fields, CRF) are combined to estimate the semantic label of each data point. However, the classification and recognition stage of the classifier and the CRF optimization stage are usually operated independently as separate modules without interaction with each other, thus limiting the information exchange between the modules.

Among them, for the classifier, 3D voxel convolutional neural network is a better choice. The three-dimensional voxel convolutional neural network is extended from the two-dimensional convolutional neural network, and it has also achieved good performance in the task of three-dimensional target classification and recognition. Compared with the point cloud-based deep neural network, the three-dimensional voxel Convolutional neural networks also have the advantages of clear network structure and easy acceleration. But voxel neural networks require regularized data input, and their labeling results are also roughly labelled at the voxel level.

In view of the above problems in the prior art, the embodiments of the present invention provide a method and device for labeling a 3D point cloud based on fusion voxels, by constructing a multi-scale space based on a voxel convolutional neural network on a regularized voxel model To extract multi-scale voxel features, and then use feature interpolation to extend the voxel features to point features, so as to achieve a more refined point-by-point classification and recognition to further improve the labeling performance. In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. The components of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings are not intended to limit the scope of the invention as claimed, but are merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

As shown in FIG. 1 , it is a schematic diagram of an application scenario of the fused voxel-based three-dimensional point cloud labeling apparatus 100 according to an embodiment of the present invention. The electronic terminal 10 includes a fusion voxel-based three-dimensional point cloud labeling device 100 , a memory 200 , a storage controller 300 and a processor 400 . The electronic terminal 10 may be, but not limited to, an electronic device with processing functions such as a computer, a mobile Internet device (MID), or the like, or a server or the like.

Optionally, the elements of the memory 200, the storage controller 300, and the processor 400 are directly or indirectly electrically connected to each other, so as to realize data transmission or interaction. For example, these components are electrically connected through one or more communication buses or signal lines. The fused voxel-based three-dimensional point cloud labeling device 100 includes at least one software function module that can be stored in the memory 200 in the form of software or firmware or fixed in the operating system of the electronic terminal 10 . The processor 400 accesses the memory 200 under the control of the memory controller 300 to execute executable modules stored in the memory 200, such as software function modules and computer programs included in the fused voxel-based three-dimensional point cloud labeling apparatus 100. .

It can be understood that the structure shown in FIG. 1 is only for illustration, and the electronic terminal 10 may further include more or less components than those shown in FIG. 1 , or have different configurations from those shown in FIG. 1 . Each component shown in FIG. 1 may be implemented in hardware, software, or a combination thereof.

Further, referring to FIG. 2 , an embodiment of the present invention further provides a method for labeling a 3D point cloud based on fusion voxels. The method for labeling a 3D point cloud based on fusion voxels is described below with reference to FIG. 2 .

Step S11, performing voxelization processing on the three-dimensional point cloud data set, and performing voxel feature extraction in the voxels based on the processing result to form a first voxel feature matrix;

Step S12, using the first voxel feature matrix as the input of the three-dimensional convolutional neural network to obtain the multi-scale feature of the voxel, and performing feature series fusion on the multi-scale feature to obtain the second voxel feature matrix;

Step S13, based on the feature interpolation algorithm, the voxel feature in the second voxel feature matrix is extended to each point in the three-dimensional point cloud data set to obtain the point cloud feature matrix;

Step S14, the point cloud feature matrix is input into the multi-layer perceptron to realize the attribute marking of the three-dimensional point cloud.

In this embodiment, the three-dimensional point cloud is first subjected to voxelization, and then the feature extraction is performed on the point cloud within the voxel, and then the voxel model with voxel features as elements is input into the three-dimensional convolutional neural network for multi-scale feature extraction. Then, the feature interpolation algorithm is used to extend the voxel feature to the point cloud feature to realize the labeling of the 3D point cloud, which can effectively improve the accuracy of the point cloud labeling.

In detail, please refer to FIG. 3 , the process of performing voxelization processing on the point cloud in step S11 can be implemented through the following steps S111 to S113:

Step S111, dividing the point cloud coordinate space into a plurality of voxels according to a preset voxel size;

Step S112, according to the grid parameter of the voxel, classify each point in the three-dimensional point cloud data set into the corresponding voxel;

Step S113: Sampling points in each of the classified voxels so that the number of points in the voxels reaches a first preset value.

In this embodiment, a point cloud voxelization model is introduced in the above-mentioned steps S111 to S113 to perform voxelization processing on the point cloud. Specifically, as shown in FIG. 4 , the point cloud voxelization is to divide the point cloud coordinate space into multiple voxels according to a given voxel size. Among them, it is assumed that the dimensions of the input point cloud in the directions of the three coordinate axes, namely the X, Y, and Z axes, are W, H, and E respectively, and the dimensions of each voxel are λ _W , λ _H , λ _E , then the voxelization process The dimensions of the resulting model are W'=W/λ _W , H'=H/λ _H , E'=E/λ _E . In this embodiment, in order to facilitate subsequent convolution operations, W', H', and E' may be integers and powers of 2.

After the voxelization grid is implemented for the point cloud coordinate space in step S111, each point in the point cloud can be further classified according to the grid parameters of each voxel, so that each point belongs to each voxel. However, when classifying point clouds, due to the influence of measurement error, distance, occlusion and other factors when collecting 3D point cloud data, the collected point clouds are often uneven. The regional point cloud is relatively sparse. In addition, the point cloud data collection is equivalent to the sampling of the target surface, so the interior of the target is empty and there is no point cloud data, which leads to the uneven distribution of point clouds inside each voxel after voxelization of the point cloud space. , as shown in Figure 4, where the voxels in the lower left corner do not contain point clouds, and the voxels in the upper right corner contain fewer points. Therefore, in order to facilitate subsequent unified voxel feature extraction, after the point cloud is divided into blocks, the same number of points need to be sampled from each voxel, such as the first preset value T (T is based on the point cloud resolution and storage capacity. to make sure).

It should be noted that, during sampling, if the number of points contained in a voxel is greater than the first preset value, the first preset value points are randomly sampled from the current voxel to make the points in the voxel have a higher number of points than the first preset value. The number reaches the first preset value; if the number of points contained in the voxel is less than the first preset value, one or more points are randomly selected from the current voxel for copying so that the number of points in the voxel reaches The first preset value. For example, assuming that the first preset value is T, for voxels whose number of points in a voxel exceeds T, T points can be randomly sampled. A set of T points is obtained, and after the point cloud is divided and sampled, no more than T'×H'×E' voxel sets containing T points can be obtained, and then the point cloud data in the voxels are used for feature learning. Obtain a valid feature representation for each voxel containing the point cloud.

Further, the step of performing point cloud feature extraction in the voxel based on the processing result to form the first voxel feature matrix in the above step S11 includes: calculating the center coordinates of the point cloud for the point cloud in each voxel, and based on the center coordinates The coordinates of the point cloud data in the voxel are normalized to the center to obtain the initial data matrix; the initial data matrix is input into the LGAB module to realize the point-by-point local feature description, and the maximum value pooling is used to analyze the local features within the voxel. The feature set is subjected to a point-by-point pooling operation to obtain the global feature of the voxel as the first voxel feature matrix.

Specifically, in the embodiment of the present invention, a local and global feature fusion module (LGAB) as shown in FIG. 5 is used for stacking to build a feature learning network to perform voxel sign extraction. Among them, as shown in Figure 6, it is assumed that V _x is a non-empty voxel containing T points, that is, V _x ={pi =(x _i ,y _i ,z _i ,)}, _i =1,2,3 ,...,T, then normalize the center before inputting the point cloud data (initial data matrix) into the LGAB module, that is, first calculate the center coordinates (c _x , c _y , c _z ) of the point cloud in the voxel, and use the center The coordinates (c _x , c _y , c _z ) are normalized to the center of the point cloud data to obtain the final input data initial data matrix That is, the input of the voxel feature extraction module is an initial data matrix of T × 6 dimensions.

Further, the stacked LGAB modules can be used to obtain the point-by-point local feature description, and the global feature of the voxel can be obtained by pooling the point-by-point feature set within the voxel by using Max-Pooling (MP). Figure 6 shows an example of feature extraction for non-empty voxels. In order to reduce the number of parameters, the remaining non-empty voxels can share the same network parameters during feature extraction. In actual implementation, since the LGAB module can well fuse the local neighborhood common information of the point cloud and the respective difference information, in this embodiment, the cascade of multiple LGAB modules can effectively extract the point cloud information inside the voxel.

Further, as shown in FIG. 7 , the process of using the first voxel feature matrix as the input of the three-dimensional convolutional neural network in step S12 to obtain the multi-scale features of the voxels can be realized by the following steps.

Step S120, converting the voxel feature matrix into a 4-dimensional tensor, and inputting the 4-dimensional tensor into a three-dimensional convolutional neural network with convolution kernels of different sizes to calculate the voxel features at different scales;

In step S121, the voxel features at different scales are respectively input into the three-dimensional deconvolution neural network with convolution kernels of different sizes to obtain a plurality of voxel features of different scales. The size of the convolution kernel of the convolutional neural network corresponds to the size of the convolution kernel of each three-dimensional deconvolutional neural network when performing deconvolution.

In detail, since the spatial geometric information is the important information of the three-dimensional target, the effective feature description of the target can be extracted by directly processing the three-dimensional data. The convolutional neural network is extended to the three-dimensional convolutional neural network, that is, when the three-dimensional convolutional neural network is used to process the three-dimensional data, the three-dimensional data needs to be regularized, that is, voxelized, and the two-dimensional convolution operation, Pooling operations etc. are extended to 3D voxel data. Among them, the three-dimensional convolution formula is:

In formula (1), is the input 3D voxel data, is the three-dimensional convolution kernel template, output a response for it. Similar to the two-dimensional convolution operation, the three-dimensional convolution operation just expands the two-dimensional convolution kernel to a three-dimensional convolution kernel, including three dimensions of length, width and height. Correspondingly, as shown in Figure 8, its local feeling is also determined by the two-dimensional plane. The local neighborhoods on are converted to local neighborhoods in 3D space. In actual implementation, since the three-dimensional convolution operation will reduce the spatial size of the three-dimensional data, that is, the spatial size of the three-dimensional feature map is smaller than the size of the input voxel data. However, for 3D data labeling, the feature information of each data point needs to be obtained, that is, the feature of each voxel needs to be obtained when performing voxel feature extraction, so the feature map after the convolution operation needs to be reversely mapped back to the original input volume in the element. To solve this problem, in this embodiment, a deconvolution operation is used to process the obtained three-dimensional voxel feature map to obtain a feature map of the same size as the input voxel data. The core of the deconvolution operation is still the convolution operation, but before the convolution operation, the edge padding 0 operation is performed on the input feature data to ensure the size requirements of the output feature map. For example, Figure 9 shows a two-dimensional example of convolution and deconvolution operations, where the data marked in blue is the input data of the two operations, the data marked in gray is the convolution kernel (both have the same size), and the green Mark data as the response output for both. It is not difficult to see that the input of the deconvolution layer is the output of the convolution layer, and the output of the deconvolution layer has the same size as the input of the convolution layer.

Further, after processing all non-empty voxels based on the foregoing operations, a series of D-dimensional voxel features can be obtained. Since each voxel feature uniquely corresponds to the voxel coordinates of the three-dimensional space, the acquired feature can be expressed as a 4-dimensional tensor, and its size is W'×H'×E'×D, (for empty voxels, Use D-dimensional zero vector as its feature description). After converting the feature representation to a 4-dimensional tensor-based feature representation, a three-dimensional convolutional neural network (3D CNN) can be used for further feature optimization. Considering that feature extraction at a fixed scale (convolution kernel size) is not enough to fully express the local neighborhood structure information of voxels, the present invention adopts multi-scale feature extraction and fusion to extract more abundant local neighborhood information.

Among them, as shown in Figure 10, it is a specific three-dimensional convolutional neural network structure, that is, the multi-scale feature extraction based on 3D CNN mainly includes three-dimensional convolution operation (Conv3D), three-dimensional deconvolution operation (DeConv3D) and feature series operation. (Concat). For the input W'×H'×E'×D dimension features, three different convolution kernels are used to perform convolution and deconvolution operations respectively, which are recorded as Conv3D(f _in ; f _out ; ker; st; pad) , DeConv3D(f _in ; f _out ; ker; st; pad), where f _in and f _out represent the dimension of the input and output feature matrix, ker; st; pad represent the size of the convolution kernel, the convolution kernel moving step during convolution The long size and the data padding size during data expansion are all three-dimensional vectors. In order to obtain features at different scales, the three convolution kernels can be but are not limited to (1; 2; 2); (2; 1; 2); (2; 2; 1), in the convolution and deconvolution operations The same convolution kernel is used, and the corresponding edge padding 0 operation is performed in the deconvolution operation. In addition, the convolution layer and the deconvolution layer not only include convolution and deconvolution operations, but each convolution operation also includes a normalization layer (Batch Normalization layer, BN) and ReLU activation operations.

The multi-scale feature extraction based on 3D CNN uses different convolution kernels to extract and fuse voxel features along three mutually perpendicular directions (ie, X, Y, and Z directions) in three-dimensional space, so that the learned features can contain more To achieve a more complete expression of the point cloud.

Further, in step S13, since it is necessary to obtain the feature description of each point when realizing the marking of the three-dimensional point cloud, and the aforementioned three-dimensional convolutional neural network can only obtain the feature description of each voxel, therefore, in the present invention, through Interpolate voxel features to obtain feature descriptions for each point in the input point cloud. As shown in Figure 11, for a given target point p, find the nearest preset number (eg 8) neighborhood voxels in the voxel space formed by the second voxel feature matrix, and each neighborhood voxel corresponds to The feature descriptions are Where j=1,2,...,8, then the feature description of the target point p is:

In formula (2), represents the weight parameter obtained according to the Euclidean distance between the target point p and the center point c _j in the j-th neighborhood voxel, Represents the voxel feature of the jth neighborhood voxel, and repeats the above process for each point in the 3D point cloud, so that the voxel feature in the second voxel feature matrix can be extended to each point in the 3D point cloud dataset to get the point cloud feature matrix.

Further, in step S14, the point cloud feature matrix of each point obtained in step S13 is input into the multi-layer perceptron (MLP) to realize the category recognition of the point, that is, the three-dimensional point cloud mark. Please refer to the figure again for its specific network structure. 4.

According to actual requirements, in order to further optimize the above-mentioned point cloud labeling result obtained by using the convolutional neural network and improve the labeling accuracy, the present invention further includes step S15.

In step S15, the three-dimensional point cloud whose attribute labeling has been completed is input into the CRF-RNN network for optimization of the attribute labeling of the point cloud.

Specifically, this embodiment embeds the FC-CRF based on the basic operation of CNN into the aforementioned point cloud labeling network based on 3D convolutional neural network, to realize an end-to-end 3D point cloud fine labeling network that integrates coarse labeling and back-end optimization. , to further improve the accuracy of point cloud labeling, especially the smoothness of the target boundary and contour.

In this embodiment, the CRF oriented to the 3D point cloud tag is first modeled, then the CRF based on the CNN operation is approximated, and finally the 3D point cloud tag optimized by the CRF is fused.

Traditional semantic tags are all modeled as point-by-point classification and recognition, some use local features for classification and recognition, and some use deep neural networks for classification and recognition. However, point-by-point classification and recognition usually bring some obvious unacceptable labeling errors. For example, some points within a certain target may be recognized as other categories. This is because point-by-point classification and recognition does not consider the difference between points. The adjacency relationship of , which only uses part of the point to be marked, the neighborhood information of small size. If the target structure information can be modeled in advance (for example: all targets are continuous, and its adjacent points with similar characteristics should be marked as the same type of target) and the labeling results can be optimized and limited based on the modeling results, some obvious The errors can be effectively eliminated, and then high-precision marking results can be obtained. Conditional random fields (CRFs) are effective methods for modeling object continuity and its neighborhood structure information, and have been widely used in 2D image labeling. Among them, the conditional random field is a model that calculates the conditional probability distribution of another set of output random variables under the condition of a given set of random variables. Its main feature is to assume that the output variables constitute a Markov Random Field (MRF). .

In detail, CRF is a discriminative probabilistic undirected graph model, which can model global context information and cross-cutting features in data, and is a probabilistic graph model that can handle sequence data segmentation and labeling well. Suppose a set of random variables X={X ₁ ,X ₂ ,…,X _N } and P={P ₁ ,P ₂ ,…,P _N }, where X _i ∈L={l ₁ ,l ₁ , ...,l _M ,}, for 3D point cloud labeling, P is the input point cloud containing N points, P _j is the observation vector of the jth point, X is the semantic labeling result of the input point cloud, X _i is the ith point cloud The semantic label of a point is one of the M semantic labels, then the corresponding CRF model can be represented by the Gibbs probability distribution, that is:

In formula (3), G is the probability undirected graph constructed on the random variable set X, _O is the group in the graph G, wherein, each pair of nodes in _o is adjacent, and _OG is all the bases in G. The set of groups, Z(P) is the normalized function, Λ(x _ο |P) is the energy function on the group, also known as the potential function.

The overall potential function for any kind of labeling result x∈L ^N is:

Based on the maximum posterior probability algorithm, the optimal labeling result is obtained as follows:

It can be seen from the above optimal solution solution process that the maximization of the posterior probability of the labeling result is the minimization of the overall potential function. It is not difficult to see that the conditional random field first uses the group potential function to realize the modeling of local context information, and then uses the graph structure to transmit the context information, and then realizes the modeling of large-scale context information.

For the fully connected CRF model, each node in graph G is connected to other nodes. As shown in Figure 12, the corresponding group _O is a group containing a single node or a pair of nodes, so the overall potential function corresponding to x can be Expressed as:

Among them, for the convenience of description, the conditional part P in the conditional posterior probability is removed from formula (6), that is, Λ(x)=Λ(x|P), is the unit potential function, is a pairwise binary potential function, i,j=1,2,...,N. unit potential function Indicates that the i-th node in graph G is marked as _xi cost. This function is usually defined by the probability output of a discriminant classifier. At this time, the estimation result often contains more noise, and the segmentation result is often discontinuous at the target edge. . The pairwise binary potential function gives the cost of simultaneously labeling the i and jth observation points as x _i , x _j , which has the effect of preserving the labeling consistency of adjacent observation points and reducing inconsistency, which can improve the accuracy of labeling results. smoothness.

Using the idea of Gaussian weighting, the pairwise binary potential function is defined as:

In formula (7), the function ψ(x _i , x _j ) is the consistency calculation function between different labels, w ^(m) is the weight, is a smoothing filter function based on Gaussian kernel, there are _MG Gaussian kernel functions in total, f _i , f _j represent the feature vectors of observation points i and j respectively, and there are Each Gaussian kernel function It can be defined by a symmetric, positive definite matrix ^Λm . So far, the fully connected CRF modeling for 3D point cloud labeling has been completed, and the next step is how to solve the optimal labeling results.

The optimization process of 3D point cloud labeling based on fully connected CRF is the process of maximizing the posterior probability Φ(X) based on the input point cloud data. It is difficult to find the exact posterior probability, and the amount of calculation is huge. The approximation method based on the mean field can convert the posterior probability Φ(X) into a product of a series of independent marginal probabilities, that is, Φ(X)≈Θ (X)=∏ _i=1 Θ _i (X _i ), combined with equations (3)-(7), the marginal probability Θ _i of any marked result x _i can be obtained as:

Based on equation (8), an iterative inference algorithm for CRF can be constructed, as shown in Algorithm 5.1. The convergence of the iterative algorithm is mainly measured by the difference between the estimated Q and P. It can be obtained by evaluating the convergence of the algorithm. When the number of iterations is 10, the estimated error is already very small, which proves that the algorithm has good convergence. .

Algorithm 1: CRF Iterative Inference Algorithm Based on Mean Field Approximation

1. Initialization: Initialize all nodes

2.while is the convergence do

3. Information transfer: Calculate all Gaussian filter results

4. Weighted filtering:

5. Consistency detection:

6. Increase the unary potential function:

7. Normalization:

8. end while

The following describes how to use the relevant operations in CNN to implement the CRF iterative inference algorithm. The biggest problem in the reconstruction of the algorithm based on the CNN operation is whether the back propagation of the error can be realized, that is, the BP algorithm is used for parameter learning and training.

(1) Initialization operation

The initialization operation in Algorithm 1 is:

in, That is, all possible values can be summed. remember then there are Z _i =∑ _l exp(U _i (l)), it can be seen that this operation is equivalent to performing an activation operation based on the Softmax function on U _i (l) of all possible labelling results at each scene point. The Softmax function is a commonly used activation function in the CNN network. It does not contain any parameters, and its error derivative can be back-propagated, so the Back-Propagation (BP) algorithm can also be used for learning and training.

(2) Information transmission

As shown in Algorithm 1, the information transfer in CRF uses _MG Gaussian filters to smooth Θ _j . The kernel function of the Gaussian filter is obtained according to the characteristics of the point cloud, such as the coordinate information or color and intensity information of the point, which expresses the relationship between each scene point. In the fully connected CRF model, each filter needs to cover all points in the point cloud, and the amount of data and calculation is very large, so it cannot be directly implemented. Here, the permutohedral laTTice method based on the full degree of freedom polyhedron lattice is used to realize fast Gaussian convolution, and its calculation amount is O(N), where N is the number of points to be filtered. speed and better filtering. The fast Gaussian convolution based on the full-degree-of-freedom polyhedral lattice method consists of four stages, the polyhedral lattice construction stage, the expansion (splat) mapping, the slice mapping and the blur (blur) stage.

In backpropagation, the input (error derivative) of the current convolutional layer is the output of the previous layer of filters after passing through _MG Gaussian filters in the reverse direction. In Gaussian convolution based on the full-degree-of-freedom polyhedral lattice method, this back-propagation can reverse the filter in the blurring stage while maintaining the same polyhedral lattice construction, expansion mapping and slice mapping as in the forward propagation. order to achieve. The calculation amount of this implementation method is still O(N), which significantly reduces the calculation amount and improves the calculation efficiency.

(3) Weighted filtering

The next computation is a weighted sum of the previous _MG outputs for each semantic label l. In the point cloud labeling, each semantic label is independent of each other, so this weighted filtering operation can be implemented by MG _convolution operations with a convolution kernel of l×l, where the input of the convolution operation is _A feature matrix containing MG channels, and the output is a feature matrix containing l channels. In backpropagation, since the input and output of this step are known, and the convolution kernels are independent of each other, the parameters of the convolution kernel and the error derivative of the input data can be calculated, and then The BP algorithm can be used to learn and train the parameters of the convolution kernel.

(4) Consistency detection

In the consistency detection, the PoTTs model is used to calculate the compatibility of the output results of different labels in the previous step. The compatibility calculation is mainly to compare whether the labels of two similar observation points are the same. When the semantic labels of the two points are the same, the consistency detection is 0. When the semantic labels of the two points are different, a penalty term σ is introduced. The calculation is as follows :

Compared with using a fixed penalty term σ, the present invention considers using the penalty value learned based on data, because the degree of association between different labels is different, so marking adjacent points with different labels is important for the entire labeling result. The impact is different. Therefore, consistency detection can also be regarded as a convolutional layer. The number of input and output channels of this layer is M (the number of labels), the size of the convolution kernel is l×l, and the learned neuron connection weight parameter is The value of the conversion function. This step can also be back-propagated due to the use of basic convolution operations.

(5) Increase the unary potential function

The unary potential function Element-by-element combination with the output results obtained in the consistency check is performed to obtain the complete potential function result. In the step of adding a one-variable potential function, no parameters are included, so the error in the output can be simply copied to the input to achieve backpropagation.

(6) Normalization

Similar to the initialization process, the normalization step can also be implemented by the activation operation based on the Softmax function, and its backpropagation is consistent with the backpropagation based on the Softmax function in CNN. So far, the basic operations in the CNN network have been used to implement each step of a single iteration in Algorithm 1, and the solution algorithm for multiple iterations can be implemented by stacking the above steps.

Based on the above description, in this embodiment, the CRF model is first approximated by the mean field approximation method, and then each step in the mean field approximation method is equivalently realized by using the basic operations in CNN, that is, the single Iterative mean-field approximation algorithm. For the iterative mean field approximation method, it is only necessary to stack the relevant steps, that is, the recursive CNN structure (RNN) can be used to realize the iterative mean field approximation calculation. Its structure is shown in Figure 13. Given the input point The cloud data is P, the point-by-point unary potential function is U=U _i (l), the marginal probability obtained by the previous iteration is H ₁ , the marginal probability obtained by the current iteration is H ₂ , and the approximate estimation of the single mean field is denoted as f _Ω (U, P, H ₁ ), Ω is its parameter set (including all parameters in weighted filtering and consistency detection) and Ω={w ^(m) ,ψ(l,l')}. For H ₁ , it is initialized as the output of the softmax function with U as the input at the beginning of the iteration, and is equal to the output of H ₂ in the subsequent iterations, namely:

where T' is the number of iterations.

After obtaining H1, estimate _H2 based _on the mean field approximation algorithm, namely:

H ₂ (t')=f _Ω (U,P,H ₁ (t')),0<t'≤T'

For output Y, the estimation result is output only at the last iteration, ie Y=H ₂ (T').

Based on the above analysis, it can be known that the error derivative of the parameter in the entire network structure (referred to as CRF-RNN) is obtainable, so it can be solved by the standard BP algorithm, so it can also be embedded in other neural networks for learning and training .

Further, based on the aforementioned point cloud labeling network Point-VoxelNet network, a 3D point cloud labeling network (Point VoxelNet+CRF-RNN, PVCRF) that integrates 3D voxel convolutional neural network and CRF back-end optimization is constructed. The specific structure is as follows. As shown in Figure 14, where, for the input point cloud, firstly perform voxelization according to the scene size of the input point cloud and randomly select a fixed number of point clouds in the voxel for subsequent feature extraction, and then use the LGAB module in the body Perform feature extraction within the pixels to obtain simple voxel features, and after obtaining all non-empty voxel features (the features of empty voxels are complemented by 0 operations), multi-scale voxel features are performed based on three-dimensional convolutional neural networks (Conv3D, DeConv3D). Extraction and fusion, and then use the interpolation method to extend the multi-scale voxel features to all points to obtain point-by-point point features, then input the point features into the multilayer perceptron to obtain the preliminary point cloud labeling results, and finally based on CRF- RNN network structure for back-end optimization. This network structure well realizes the information interaction between the classification and recognition stage and the CRF optimization stage in the point cloud labeling, and has a significant effect on improving the accuracy of point cloud labeling.

Based on the description of the above-mentioned fusion voxel-based 3D point cloud labeling method, the inventor also verified the performance of the fusion voxel-based 3D point cloud labeling method, such as Intersection over Union (IoU) And the overall accuracy (Overall Accuracy, OA) and other evaluation indicators to evaluate the performance of point cloud labeling.

(1) Network implementation and parameter setting

In the point cloud data rasterization and sampling stage, different data sets need to be processed differently.

· S3DIS: For the S3DIS dataset, the maximum size ranges of the scene along the Z, Y, and X axis directions are E=8m, H=16m, and W=50m, respectively. In order to cover the entire scene, the size of the entire grid is 8×16×50, the size of each voxel is λ _E = 0.5m, λ _H = 0.25m, λ _W = 0.2m, and the size of the constructed voxel model is E '=16, H'=64, W'=256, and the extra voxels are left blank. T=32 points are selected in each voxel.

vKITTI: For the vKITTI dataset, the maximum size ranges of the scene along the Z, Y, and X axes are E=33m, H=193m, and W=148m, respectively. Here, the size of each voxel is set as λ _E =2m, _λH =1.6m, _λW =1.2m, and the size of the constructed voxel model is E'=16, H'=128, and W'=128. Similarly, T=32 points are selected in each voxel.

For the CRF-RNN network, in order to prevent overfitting and gradient disappearance, the number of iterations is set to T'=5 in the training phase, and T'=10 in the testing phase. The Gaussian filter size is the same as the point cloud data size.

The present invention adopts a two-step training strategy. The first step is to train Point-VoxelNet separately, and the second step is to fine-tune the network PVCRF that combines Point-VoxelNet and CRF-RNN. In the Point-VoxelNet network, the Adam optimization algorithm with a momentum value of 0.9 is used for training optimization, the initial learning rate is 0.001, and the training batch size is 16. For the PVCRF network, the Adam optimization algorithm with a momentum value of 0.6 is used for training optimization, the initial learning rate is 0.0001, and the training batch size is 16. During training, an early stopping strategy is also used to obtain the optimal network parameters. The maximum number of training rounds is 100. If the network parameters are still not updated after 10 consecutive rounds of training, the training will be stopped. In the testing phase, similarly, 6-fold cross-validation is used to verify the algorithm, and the grouping of training data and test data is shown in Table 1.

Based on the Python language, the deep learning framework of Tensorflow is used to realize the proposed 3D voxel neural network structure and CRF-RNN network structure. The experimental hardware environment is: Intel Core i76700KCPU, 48G memory, GTX1080Ti graphics card (support CUDA 8.0, cuDNN 5.1).

(2) Analysis of quantitative results

Based on the above two datasets, the application effects of deep neural network models (Point VoxelNet and PVCRF) in 3D point cloud labeling were compared, and compared with the current better labeling algorithms PointNet, MS+CU(2), SEGCloud, 3DContextNet. Comparative analysis. Among them, only the Cartesian coordinate information of the point cloud, that is, the XYZ coordinates, is used for processing.

Table 1

Table 2

Table 1 and Table 2 respectively show the statistical labeling results of different network models on two small datasets. The PVCRF model proposed in this example has achieved better average IoU on the S3DIS and vKITTI datasets, which are 51.8 respectively. %, 39.1%, and good results were also achieved in the overall labeling accuracy OA, which were 81.2% and 82.6%, respectively, indicating that the PVCRF model can better obtain the complete feature expression of the point cloud, which proves that the 3D voxel space-based Multi-scale feature extraction and Euclidean space-based multi-scale feature extraction have equivalent functions, both of which can provide detailed information of point cloud scenes for 3D point cloud labeling.

Comparing PVCRF and SEGCLloud, SEGCLloud directly uses a binary voxel model for voxel feature learning, while PVCRF uses a rasterized point cloud voxel model, and uses the point cloud in the voxel for voxel feature learning. Both add backend optimization based on the fully connected CRF model. In addition, PVCRF includes a multi-scale feature extraction module based on three-dimensional voxel space, so PVCRF achieves a higher average IoU, which fully shows that the processing of point cloud data in voxels and the multi-scale feature extraction module can extract stronger features. Characterization of the ability to characterize.

Comparing the Point-VoxelNet and PVCRF models, PVCRF achieves better performance in both average IoU and overall accuracy OA. This is because the fully connected CRF model can model a wider range of contextual information, and the Adjacency relationships have stronger representational power.

On different datasets, taking PointNet as the benchmark, since the point clouds in the S3DIS dataset are relatively concentrated and the density of point clouds is generally high, the network model PVCRF proposed in this embodiment has achieved a greater performance improvement, while in the vKITTI dataset , because its point cloud distribution is generally sparse, the performance improvement is smaller.

The labeling results obtained by different algorithms on the S3DIS and vKITTI datasets and the ground-truth labeling results are shown in Figure 15 and Figure 16. From left to right: input color point cloud, labeling results based on PointNet, Point-VoxelNet, PVCRF, real labeling results. From the labeling results in Figure 15 and Figure 16, it can be seen that the results obtained based on PVCRF are better than PointNet and Point-VoxelNet, and the labeling results are closer to the real results, proving the multi-scale feature learning in PVCRF and CRF back-end optimization effectiveness. Compared with PointNet and Point-VoxelNet, the labeling results obtained based on Point-VoxelNet are better than PointNet, mainly because the introduction of multi-scale feature learning in Point-VoxelNet improves the feature learning ability of the network model.

In indoor and outdoor scenes, the Point-VoxelNet and PVCRF models are still difficult to separate for objects with high overlap or close connections, such as the board and window in Figure 15, and the road in Figure 16 ( road) and terrain. Since both Point-VoxelNet and PVCRF models consider multi-scale feature extraction, objects larger than the target size in the scene can be well marked, such as the table in Figure 15 and the building in Figure 16. )Wait.

Compared with Point-VoxelNet and PVCRF, since the fully connected CRF can extract large-scale context information in the scene point cloud, the details in the labeling results are more prominent, as shown in the first row of the chair and sofa in Figure 15. ), and the segmentation result of road and terrain in Figure 16.

(4) Category confusion matrix analysis

Figure 17 and Figure 18 show the category confusion matrix obtained by the Point-VoxelNet and PVCRF network models on the S3DIS and vKITTI datasets, respectively. The values in the matrix grid are the category labeling accuracy, and the grid color also represents the accuracy. the size of the rate. Comparing the results of the two, it can be found that for the indoor data set S3DIS, the segmentation accuracy of Point-VoxelNet and PVCRF model for various targets is equivalent. The introduction of CRF model mainly reduces the difference between table and floor, etc. confused. For the outdoor dataset vKITTI, the introduction of the CRF model mainly reduces the confusion between buildings and vans.

As can be seen from Figure 17(a), in the S3DIS dataset, the Point-VoxelNet network model has a The recognition accuracy rate of bookcase (bookcase), wall board (board) and chair (chair) are all above 52%, and the labeling accuracy rate is low for sofa (sofa), wall (wall) and shadow (cluTTer), and its accuracy is in Between 35% and 46%, the worst of the 13 class targets is table, which has an accuracy of 22%. The distribution of average class accuracy obtained by PVCRF on the S3DIS dataset is similar to that of the Point-VoxelNet network model, but the general accuracy has been improved, and the accuracy of the table has increased to 30%, as shown in Figure 18(a). Show.

Similar comparison results are also found in the vKITTI dataset, as shown in Figures 17(b) and 18(b). This comparison result proves that the CRF model can accurately grasp and model the target details in the scene. Comparing the different results on the two datasets in Figure 17 and Figure 18, the labeling accuracy on the vKITTI dataset is generally lower, mainly because the point cloud data in the vKITTI dataset is generally sparser than the point cloud data in the S3DIS dataset. Therefore, the neighborhood structure of points is not so obvious, which leads to insufficient information extraction, which is not conducive to the realization of high-precision point cloud labeling.

(5) Statistical analysis of computing time

The computational efficiency of different algorithms is experimentally analyzed on the S3DIS dataset. The S3DIS data set contains a total of 272 independent point clouds. Different algorithms are used to mark each point cloud and the average test time is calculated (only the calculation time of the neural network is calculated, and the time consumption of the data preparation stage is not calculated). The statistical results See Table 3. From the statistical results in Table 3, it can be seen that the computation time of PointNet is about 1.8s. The network model PVCRF that integrates the three-dimensional voxel convolutional neural network and the CRF back-end optimization given in this embodiment takes the most computation time, which is 4.52s, because the introduction of the CRF significantly increases the computation amount.

table 3

Finally, FC-CRF based on convolutional neural network is used for further optimization to obtain fine labeling results.

Further, referring to FIG. 19 , the page construction apparatus 100 provided by the embodiment of the present invention includes a voxel processing and feature extraction module 110 , a multi-scale voxel feature calculation module 120 , a feature expansion module 130 and a point cloud marking module 140 .

The voxel processing and feature extraction module 110 is used to perform voxelization processing on the three-dimensional point cloud data set, and perform voxel feature extraction in the voxels based on the processing results to form a first voxel feature matrix; in this embodiment, the above Step S11 may be performed by the voxel processing and feature extraction module 110 , that is, for the specific description of the voxel processing and feature extraction module 110 , reference may be made to step S11 , which will not be repeated in this embodiment. Optionally, as shown in FIG. 19 , in this embodiment, the voxel processing and feature extraction module 110 includes a voxel division unit 111 , a point cloud classification unit 112 and a point cloud sampling unit 113 .

The voxel division unit 111 is used to divide the point cloud coordinate space into a plurality of voxels according to the preset voxel size; in this embodiment, the above step S111 can be performed by the voxel division unit 111, that is, the specific details about the voxel division unit 111 For description, reference may be made to step S111, which is not repeated in this embodiment.

The point cloud classification unit 112 is used to classify each point in the 3D point cloud data set into corresponding voxels according to the grid parameters of the voxels; in this embodiment, the above step S112 can be performed by the point cloud classification unit 112 , that is, for a specific description of the point cloud classification unit 112 , reference may be made to step S112 , which will not be repeated in this embodiment.

The point cloud sampling unit 113 is configured to sample the points in each of the classified voxels so that the number of points in the voxels reaches a first preset value. In this embodiment, the above-mentioned step S113 may be performed by the point cloud sampling unit 113 , that is, for the specific description of the point cloud sampling unit 113 , reference may be made to step S113 , which will not be repeated in this embodiment.

The multi-scale voxel feature calculation module 120 is configured to use the first voxel feature matrix as the input of the three-dimensional convolutional neural network to obtain the multi-scale features of the voxels, and perform feature series fusion on the multi-scale features to obtain the second voxel feature matrix; in this embodiment, the above step S12 can be performed by the multi-scale voxel feature calculation module 120, that is, for the specific description of the multi-scale voxel feature calculation module 120, please refer to step S12, which is not repeated in this embodiment. .

The feature expansion module 130 is used to expand the voxel features in the second voxel feature matrix to each point in the three-dimensional point cloud data set based on the feature interpolation algorithm to obtain the point cloud feature matrix; in this embodiment, the above step S13 can be performed by The feature extension module 130 executes, that is, for a specific description of the feature extension module 130, reference may be made to step S13, which is not repeated in this embodiment.

The point cloud labeling module 140 is used to input the point cloud feature matrix into the multilayer perceptron to realize the attribute labeling of the three-dimensional point cloud. In this embodiment, the above-mentioned step S14 may be performed by the point cloud marking module 140 , that is, for a specific description of the point cloud marking module 140 , reference may be made to step S14 , which will not be repeated in this embodiment.

To sum up, the embodiments of the present invention provide a method and device for 3D point cloud labeling based on fusion voxels, which extracts multi-scale voxel features by constructing multi-scale space based on voxel convolutional neural network on a regularized voxel model. , and then extend the voxel features to point features by means of feature interpolation, so as to achieve point-by-point finer classification and identification and point cloud marking.

In the description of the present invention, the terms "arranged", "connected" and "connected" should be understood in a broad sense, for example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection, or a It is an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in specific situations. In the several embodiments provided by the embodiments of the present invention, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus and method embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show possible architectures of the apparatuses, methods and computer program products according to a preset number of embodiments of the present invention , function and operation. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code. The module, program segment or part of the code contains one or a preset number for implementing the specified logic function.

It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. a three-dimensional point cloud marking method based on fusion voxel, is characterized in that, described method comprises: Perform voxelization processing on the 3D point cloud data set, and perform voxel feature extraction in the voxel based on the processing result to form a first voxel feature matrix; The first voxel feature matrix is used as the input of the three-dimensional convolutional neural network to obtain the multi-scale feature of the voxel, and the multi-scale feature is subjected to feature series fusion to obtain the second voxel feature matrix; Extending the voxel features in the second voxel feature matrix to each point in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix; The point cloud feature matrix is input into the multi-layer perceptron to realize the attribute labeling of the three-dimensional point cloud. 2. The three-dimensional point cloud marking method based on fusion voxels according to claim 1, wherein the step of performing voxelization on the point cloud comprises: Divide the point cloud coordinate space into multiple voxels according to the preset voxel size; Classify each point in the 3D point cloud dataset into a corresponding voxel according to the grid parameters of the voxel; The points in each of the classified voxels are sampled so that the number of points in the voxels reaches a first preset value. 3. The three-dimensional point cloud marking method based on fusion voxel according to claim 2, is characterized in that, the step of sampling the point in each voxel after classification comprises: If the number of points contained in the voxel is greater than the first preset value, randomly sampling the first preset value points from the current voxel so that the number of points in the voxel reaches the first preset value; If the number of points included in the voxel is less than the first preset value, one or more points are randomly selected from the current voxel for copying so that the number of points in the voxel reaches the first preset value. 4. The three-dimensional point cloud marking method based on fusion voxels according to claim 1, wherein the step of performing point cloud feature extraction in voxels based on processing results to form the first voxel feature matrix comprises: Calculate the center coordinate of the point cloud for the point cloud in each voxel, and perform center normalization on the point cloud data in the voxel based on the center coordinate to obtain an initial data matrix; The initial data matrix is input into the LGAB module to realize the point-by-point local feature description, and the maximum pooling is used to perform a point-by-point pooling operation on the local feature set in the voxel to obtain the global feature of the voxel as the first one. Voxel feature matrix. 5. The three-dimensional point cloud labeling method based on fusion voxels according to claim 1, wherein the first voxel feature matrix is used as the input of the three-dimensional convolutional neural network to calculate the multi-scale feature of the voxel The steps include: Converting the voxel feature matrix into a 4-dimensional tensor, and inputting the 4-dimensional tensor into a three-dimensional convolutional neural network with convolution kernels of different sizes to calculate voxel features at different scales; The voxel features at different scales are respectively input into three-dimensional deconvolutional neural networks with convolution kernels of different sizes to obtain a plurality of voxel features of different scales, wherein each of the three-dimensional The size of the convolution kernel of the convolutional neural network corresponds to the size of the convolution kernel of each of the three-dimensional deconvolutional neural networks when performing deconvolution. 6 . The 3D point cloud labeling method based on fusion voxels according to claim 1 , wherein the voxel features in the second voxel feature matrix are extended to the 3D point cloud data based on a feature interpolation algorithm. 7 . The steps in the various points of concentration include: For each point in the 3D point cloud data set as a target point, search for a preset number of voxels closest to the target point in the voxel space formed by the second voxel feature matrix as a neighborhood volume white; Perform extended calculation on the voxel features of the neighborhood voxels to obtain point cloud features of the target point. 7. The three-dimensional point cloud marking method based on fusion voxels according to claim 6, wherein the point cloud feature F _p of the target point p is in, represents the weight parameter obtained according to the Euclidean distance between the target point p and the center point c _j in the j-th neighborhood voxel, Represents the voxel feature of the jth neighborhood voxel. 8. The fusion voxel-based three-dimensional point cloud marking method according to claim 1, wherein the fusion voxel-based three-dimensional point cloud marking method further comprises: The 3D point cloud with completed attribute labeling is input into the CRF-RNN network for point cloud attribute labeling optimization. 9. A three-dimensional point cloud marking device based on fusion voxels, wherein the device comprises: The voxel processing and feature extraction module is used to voxelize the three-dimensional point cloud data set, and perform voxel feature extraction in the voxel based on the processing result to form a first voxel feature matrix; The multi-scale voxel feature calculation module is used to use the first voxel feature matrix as the input of the three-dimensional convolutional neural network to calculate the multi-scale features of the voxels, and perform feature series fusion on the multi-scale features to obtain the first voxel feature matrix. Two-voxel feature matrix; A feature extension module, configured to extend the voxel feature in the second voxel feature matrix to each point in the three-dimensional point cloud data set based on a feature interpolation algorithm to obtain a point cloud feature matrix; The point cloud labeling module is used to input the point cloud feature matrix into the multi-layer perceptron to realize the attribute labeling of the three-dimensional point cloud. 10. The three-dimensional point cloud labeling device based on fusion voxels according to claim 9, wherein the voxel processing and feature extraction module comprises: A voxel division unit, which is used to divide the point cloud coordinate space into multiple voxels according to the preset voxel size; a point cloud classification unit, configured to classify each point in the three-dimensional point cloud data set into corresponding voxels according to the grid parameters of the voxels; A point cloud sampling unit, configured to sample points in each of the classified voxels so that the number of points in the voxels reaches a first preset value.