CN121437726A

CN121437726A - A Real-Time Semantic-Geometric Joint Mapping Method and System Based on an Open 3D Prior Model

Info

Publication number: CN121437726A
Application number: CN202511435203.8A
Authority: CN
Inventors: 黄晨光; 李兵; 陈凯; 张康; 桂峥嵘; 朱佳俊; 王文远; 谭贻导
Original assignee: China Construction Fourth Engineering Division Corp Ltd
Current assignee: China Construction Fourth Engineering Division Corp Ltd
Priority date: 2025-10-09
Filing date: 2025-10-09
Publication date: 2026-01-30

Abstract

This invention discloses a real-time semantic-geometric joint mapping method and system based on an open 3D prior model, relating to the field of autonomous driving technology. The system integrates multiple sensor sources, including LiDAR, RGB-D cameras, stereo cameras, image recognition cameras, OCR devices, millimeter-wave radar, and acoustic arrays, to acquire geometric, semantic, and dynamic features of roads and the environment, generating compliant metadata. Based on geometric similarity, semantic matching degree, and task scenario parameters, it calculates prior selection comprehensive coefficients and determines model adaptability. It performs differential comparison based on point cloud, radar, and acoustic data to improve model consistency. Based on temporal granularity memory factors, device computing power factors, and compliance privacy factors, it assesses and optimizes the mapping process's adaptability. Through joint evaluation results, it performs model compression and texture correction, supporting real-time output and closed-loop feedback in autonomous driving, augmented reality, and collaborative scenarios, improving mapping efficiency and application adaptability.

Description

Real-time semantic-geometric joint mapping method and system based on open three-dimensional prior model

Technical Field

The invention relates to the technical field of automatic driving, in particular to a real-time semantic-geometric joint mapping method and system based on an open three-dimensional prior model.

Background

With the rapid development of applications such as autopilot and augmented reality AR, higher and higher requirements are being put on real-time perception of the environment and high-precision three-dimensional reconstruction. The existing three-dimensional mapping technology mainly takes visual SLAM, laser radar point cloud splicing and semantic-based scene reconstruction as main, and realizes positioning and modeling of the environment by methods of extracting geometric features, splicing point clouds, introducing semantic constraints and the like. However, the prior art still faces several key bottlenecks in open, diverse real world scenarios, and it is difficult to meet engineering deployment and cross-scenario generalization requirements.

Some researches have been developed in terms of semantic-geometric coupling, for example, some schemes reconstruct indoor layout through image semantics and geometric constraints, so that wall and structure contours can be recovered more accurately in a controlled indoor scene, other schemes work in dynamic scenes, semantic segmentation and feature point weighting are introduced to improve pose estimation robustness, and a method is used for establishing a lightweight semantic map for object level, distributing object point clouds through geometric SLAM and semantic detection and simplifying modeling. Each of these schemes has advantages, but there are generally the following limitations:

The method is characterized in that a plurality of existing methods focus on single or closed data/model sources (for example, RGB images are used only, laser radars are used only or CAD libraries are used only), the coverage capability of long-tail objects (rare or non-standardized objects) is limited in the open world, the failure rate of object completion and semantic reconstruction is high, scale and pose recovery is unstable under traditional vision or sparse deep observation, scale drift and point cloud cavities are easy to occur particularly in weak texture, unstructured or night low signal-to-noise ratio environments, even if the model sources are searched or matched to similar priori models, a fine grid correction flow based on observation residual errors is lacked, obvious geometric deviation exists between a model after direct replacement or splicing and real observation, the existing research is focused on a single side in a long-term static map or short-term object modeling, a scene-level static-dynamic dual-memory management and time attenuation/version control mechanism is lacked, long-term stability and prior response are difficult to guarantee in incremental updating, most of the realization systems of the operations/engineering are not considered to consider privacy, scale drift and revocable management (for example, digital license and scale and fuzzy records are not suitable for being deployed on a large scale and a full scale, and practical scale is suitable for calculating and real-time scale, and practical scale limits are not suitable for registering.

It is seen from the comprehensive existing representative literature that, although breakthroughs are made in terms of semantic-geometric coupling, dynamic scene pose optimization and light-weight object modeling, a set of end-to-end schemes capable of simultaneously solving engineering problems such as open priori search coverage, scale robust Sim (3) registration, observation driven residual grid correction, static-dynamic dual-memory incremental fusion, rule compliance and federal erasure, and embedded platform-oriented algorithm force optimization are not formed. The sources of the defects mainly comprise limited model library size and lack of updating mechanism, dimension blurring caused by single sensing observation, lack of grid correction flow driven by residual errors, lack of time management and version control strategies at scene level, and neglect of constraint on the rule of management of involution/privacy and calculation force at a vehicle end.

Disclosure of Invention

Aiming at the defects of the prior art, the invention provides a real-time semantic-geometric joint mapping method and system based on an open three-dimensional prior model, which are used for solving the problems in the background art.

In order to achieve the purpose, the invention is realized by the following technical scheme that the real-time semantic-geometric joint mapping system based on the open three-dimensional prior model comprises:

The system comprises a data acquisition module, a dynamic target and environment sound source feature acquisition module, a compliance privacy tag acquisition and verification module, a power calculation monitoring module, a data acquisition module and a data acquisition module, wherein the data acquisition module is used for acquiring geometrical point cloud and structure information of an automatic driving road and environment through a laser radar, acquiring a depth map and a color image through an RGB-D camera and a stereoscopic camera, acquiring semantic element and task scene information through an image recognition camera and an OCR recognition device;

The prior model active selection module is used for extracting geometric features, semantic elements and task scene parameters of the point cloud and the image, acquiring geometric similarity Sgeom, semantic matching Ssem and task scene parameters Ttask, comprehensively calculating prior selection coefficients PSI in an open three-dimensional prior model library, comparing and analyzing with a prior selection judgment threshold Pth, judging whether the candidate model meets the current task scene requirement, and giving corresponding strategies if the candidate model does not meet the current task scene requirement;

the cross-modal residual error correction module is used for extracting point cloud residual errors by carrying out difference ratio pair on point cloud, radar and acoustic data Difference of radar appearance boundarySum acoustic reflection residualCalculating a cross-modal residual error comprehensive coefficient MRC, comparing and analyzing with a difference correction threshold Mth, judging whether the cross-modal residual error correction is qualified or not, and giving a corresponding strategy if the cross-modal residual error correction is unqualified;

the map construction adaptation optimization module is used for calculating a map construction adaptation comprehensive coefficient ZMAI by extracting a time granularity memory factor Tmanagement, an equipment calculation force factor Ccompute and a compliance privacy factor Lcompliance, comparing and analyzing with a map construction adaptation threshold Zth, judging whether the current map construction process meets the adaptation requirement, and giving a corresponding strategy if the current map construction process does not meet the adaptation requirement;

And the real scene output module is used for carrying out self-adaptive compression and texture correction on the three-dimensional model by adopting a graph optimization and space-time fusion technology according to the joint evaluation results of PSI, MRC and ZMAI, generating an optimization model adapting to the computing force of the terminal, and realizing real-time output and closed-loop feedback of the automatic driving, augmented reality and collaborative scene through multi-line Cheng Yuyi task allocation.

Preferably, the data acquisition module comprises a geometric point cloud image acquisition unit, a semantic element task scene acquisition unit and a multi-mode signal compliance information acquisition unit;

The geometrical point cloud image acquisition unit is used for acquiring dense point cloud data of an automatic driving road scene, including road boundaries, building elevation, obstacle outlines and environment structures, by installing a vehicle-mounted three-dimensional laser radar;

The semantic element task scene acquisition unit is used for acquiring semantic visual object images in road marks, traffic marks, building entrance numbers, virtual interactive marks and man-machine interaction instructions through an image recognition camera and OCR recognition equipment;

The multi-mode signal compliance information acquisition unit is used for acquiring motion track information of a dynamic target through millimeter wave radar equipment, acquiring environment sound source characteristics and structure echo signals through an acoustic array microphone, acquiring processor occupancy rate, memory and bandwidth state information in real time through an internal computing power monitoring module of the equipment, and performing copyright license filtering, privacy labeling and revocable traceability on acquired image and model data through deployment of compliance privacy tag acquisition and verification equipment to form compliance metadata.

Preferably, the prior model active selection module comprises a first parameter extraction unit, a first calculation unit and a first analysis unit;

The first parameter extraction unit is used for comparing geometric structural features of road boundaries, building elevation and obstacle outlines with corresponding geometric features of candidate prior models by adopting a point cloud feature extraction and geometric registration algorithm based on dense point cloud data, depth map and synchronous color image data of an automatic driving road scene, obtaining geometric similarity parameters Sgeom between an environment real structure and the prior models, carrying out task demand and parameterization expression on a vehicle running target and a man-machine interaction instruction by adopting a semantic recognition and label alignment technology based on road marks, traffic marking, building entrance numbers, virtual interaction marks and semantic visual object images in man-machine interaction instructions, adopting a semantic recognition and label alignment technology to comprise target detection, OCR text recognition, semantic segmentation and label mapping, carrying out one-to-one correspondence matching on semantic elements in the semantic visual object images and the open three-dimensional prior models, obtaining consistent matching degree parameters Ssem of the environment actual semantic elements and the model semantic elements, inputting the scene task parameters based on running target information, adopting a task analysis and parameter modeling technology to comprise path constraint modeling, operation area decomposition and task semantic abstraction, and carrying out task demand and parameterization expression on the vehicle running target and the man-machine interaction instruction, and generating a scene parameter Ttask reflecting the adaptability of the task target and the building map.

Preferably, the first calculation unit is configured to search a candidate model in an open three-dimensional prior model library, and calculate and obtain a prior selection comprehensive coefficient PSI after dimensionless processing by combining the extracted geometric similarity parameter Sgeom, the consistency matching parameter Ssem and the task scene parameter Ttask, where the formula is as follows:

wherein w1, w2 and w3 represent weight coefficients;

The first analysis unit is configured to perform a comparison analysis on the prior selection integrated coefficient PSI and the prior selection decision threshold Pth by presetting the prior selection decision threshold Pth, and the obtaining a first evaluation result includes:

when the first-check selection comprehensive coefficient PSI is more than or equal to the first-check selection judgment threshold Pth, the candidate model accords with the current task scene requirement, and the current model is loaded to enter a graph building flow to serve as a basic model of geometric and semantic combined graph building;

When the prior selection comprehensive coefficient PSI is smaller than the prior selection judgment threshold Pth, the candidate model is not in accordance with the current task scene requirement, a first early warning instruction is triggered, a first strategy is generated, the other candidate prior models are retrieved again, weight coefficient distribution is dynamically adjusted or a temporary complement model is generated based on local observation point cloud, and the calculation is performed again until the prior selection comprehensive coefficient PSI is larger than or equal to the prior selection judgment threshold Pth.

Preferably, the cross-modal residual error correction module comprises a second parameter extraction unit, a second calculation unit and a second analysis unit;

The second parameter extraction unit is used for performing difference ratio pair on the acquired point cloud and the prior model point cloud by adopting an iterative closest point ICP registration and geometric feature comparison method based on dense point cloud data of the automatic driving road scene to acquire point cloud residual errors Based on the motion trail information of the acquired dynamic target, adopting a target contour extraction and boundary matching technology, comparing the radar perception appearance with the point cloud geometric boundary to acquire a radar appearance boundary difference valueBased on the environment sound source characteristics and the structure echo signals, adopting a waveform characteristic analysis and echo delay comparison method to compare the acoustic echo with the predicted echo of the geometric structure model so as to obtain an acoustic reflection residual error。

Preferably, the second computing unit is configured to introduce a visual point cloud, a millimeter wave radar and an acoustic signal to perform cross-mode residual correction based on a basic model of geometric and semantic joint mapping, and combine the extracted point cloud residualDifference of radar appearance boundarySum acoustic reflection residualAfter dimensionless processing, calculating and obtaining a cross-mode residual error comprehensive coefficient MRC, wherein the formula is as follows:

wherein a1, a2 and a3 represent weight coefficients;

The second analysis unit is configured to perform a comparative analysis on the cross-modal residual error comprehensive coefficient MRC and the residual error correction threshold Mth by presetting the residual error correction threshold Mth, and the obtaining a second evaluation result includes:

When the cross-modal residual error comprehensive coefficient MRC is more than or equal to the residual error correction threshold Mth, judging that the cross-modal residual error correction is qualified, enabling the geometric and physical consistency of the current three-dimensional model to meet the precision requirement, and entering a subsequent graph building flow;

When the cross-modal residual error comprehensive coefficient MRC is smaller than the residual error correction threshold Mth, judging that the cross-modal residual error correction is unqualified, triggering a second early warning instruction, and generating a second strategy, namely automatically starting local encryption point cloud resampling, improving the resolution of the millimeter wave radar by 15%, and simultaneously implementing grid local modification on the area with deviation by combining acoustic echo information so as to achieve residual error compression and consistency improvement.

Preferably, the mapping adaptation optimization module comprises a third parameter extraction unit, a third calculation unit and a third analysis unit;

The third parameter extraction unit is used for carrying out time sequence modeling on dynamic target changes in a task scene based on dynamic target motion track data by adopting a time sequence feature analysis and target state updating algorithm to obtain a time granularity memory factor Tmanagement, carrying out performance analysis on equipment operation states by adopting a calculation load assessment and resource utilization rate normalization method based on processor occupancy rate, memory and bandwidth state information to obtain equipment calculation force factors Ccompute, and carrying out comprehensive analysis on formed compliance metadata by adopting a compliance feature statistics and tag consistency assessment method based on compliance metadata to obtain compliance privacy factors Lcompliance.

Preferably, the third calculation unit is configured to calculate, after dimensionless processing, an obtained mapping adaptation comprehensive coefficient ZMAI through the obtained time granularity memory factor Tmemory, the device calculation force factor Ccompute and the compliance privacy factor Lcompliance, where the formula is as follows:

wherein s1, s2 and s3 represent weight coefficients;

The third analysis unit is configured to perform a comparison analysis on the map-building adaptation comprehensive coefficient ZMAI and the map-building adaptation threshold Zth by presetting the map-building adaptation threshold Zth, and obtain a third evaluation result includes:

When the mapping adaptation comprehensive coefficient ZMAI is more than or equal to the mapping adaptability threshold value Zth, the current mapping process is indicated to meet the adaptability requirement, the final mapping result is output, and the model fingerprint information is recorded;

When the map construction adaptation comprehensive coefficient ZMAI is smaller than the map construction adaptation threshold value Zth, the current map construction process is not satisfied with the adaptation requirement, a third strategy is triggered, namely, time attenuation fusion is implemented on the dynamic object part, pruning or lightweight substitution is carried out on the computation intensive module, and the compliance label is automatically updated to ensure that the privacy and copyright requirements are satisfied, and meanwhile, the real-time property and long-term stability of the map construction are improved.

Preferably, the real scene output module is configured to execute dynamic adaptation and multitask instruction generation on the three-dimensional model output by the map-building adaptive optimization module according to a joint evaluation result of a perceived semantic consistency index PSI, a cross-modal residual error comprehensive coefficient MRC and a model adaptability index ZMAI, call a model reprojection technology based on map optimization and space-time fusion when PSI and ZMAI reach a preset threshold at the same time and MRC meets a residual error correction condition, perform adaptive compression and texture mapping correction on a three-dimensional geometric structure to generate an optimized model adapting to the current terminal computing power and task scene, automatically generate a corresponding execution instruction set according to a scene category by adopting a multi-thread parallel semantic task allocation method, realize path planning and obstacle recognition of a three-dimensional map-building result in automatic driving obstacle avoidance navigation, viewpoint reconstruction and illumination matching in augmented reality rendering, and environment sharing and path synchronization output in collaborative operation, and write model optimization parameters and execution feedback data into a system log by an output result feedback mechanism.

Preferably, a real-time semantic-geometric joint mapping method based on an open three-dimensional prior model comprises the following steps:

The method comprises the steps of firstly, collecting geometrical point cloud and structure information of an automatic driving road and an environment through a laser radar, collecting a depth image and a color image through an RGB-D camera and a stereo camera, obtaining semantic element and task scene information through an image recognition camera and OCR recognition equipment, collecting dynamic target and environment sound source characteristics through a millimeter wave radar and an acoustic array, collecting the running state of the equipment through a computing power monitoring module, and collecting and verifying the running state through a compliance privacy tag to form compliance metadata;

Step two, geometrical similarity Sgeom, semantic matching Ssem and task scene parameters Ttask are obtained by extracting geometrical characteristics, semantic elements and task scene parameters of the point cloud and the image, a priori selection coefficient PSI is comprehensively calculated in an open three-dimensional priori model library, and is compared and analyzed with a priori selection judgment threshold Pth to judge whether a candidate model meets the current task scene requirement, and if not, a corresponding strategy is given;

step three, extracting point cloud residual errors by carrying out differential comparison on point cloud, radar and acoustic data Difference of radar appearance boundarySum acoustic reflection residualCalculating a cross-modal residual error comprehensive coefficient MRC, comparing and analyzing with a difference correction threshold Mth, judging whether the cross-modal residual error correction is qualified or not, and giving a corresponding strategy if the cross-modal residual error correction is unqualified;

Step four, calculating a graph construction adaptation comprehensive coefficient ZMAI by extracting a time granularity memory factor Tmanagement, an equipment calculation force factor Ccompute and a compliance privacy factor Lcompliance, comparing and analyzing with a graph construction adaptation threshold value Zth, judging whether the current graph construction process meets the adaptation requirement, and giving a corresponding strategy if the current graph construction process does not meet the adaptation requirement;

And fifthly, performing self-adaptive compression and texture correction on the three-dimensional model by adopting a graph optimization and space-time fusion technology according to the combined evaluation result of PSI, MRC and ZMAI to generate an optimization model adapting to the computing power of the terminal, and realizing real-time output and closed-loop feedback of automatic driving, augmented reality and collaborative scenes by multi-line Cheng Yuyi task allocation.

The invention provides a real-time semantic-geometric joint mapping method and system based on an open three-dimensional prior model. The beneficial effects are as follows:

(1) The real-time semantic-geometric joint mapping method and system based on the open three-dimensional prior model are used for collecting road geometric, semantic and dynamic information through fusion of multi-mode sensors such as a laser radar, an RGB-D camera, a stereo camera, a millimeter wave radar and an acoustic array, and remarkably improving the structural precision and semantic consistency of the three-dimensional mapping by combining with compliance privacy label processing, and providing high-precision and reliable basic data for automatic driving and augmented reality.

(2) According to the real-time semantic-geometric joint mapping method and system based on the open three-dimensional prior model, the best matched open three-dimensional prior model is automatically selected by introducing a comprehensive evaluation mechanism of geometric similarity, semantic matching degree and task scene parameters, geometric and physical consistency of the model is improved by cross-modal residual correction, mapping results are ensured to adapt to complex scenes and dynamic changes, and model deviation risks are reduced.

(3) According to the real-time semantic-geometric joint mapping method and system based on the open three-dimensional prior model, the real-time adaptability assessment and optimization of the mapping process are realized through the comprehensive calculation of the time granularity memory factor, the equipment calculation force factor and the compliance privacy factor, the mapping precision and the calculation resource consumption are effectively balanced, and the real-time performance and the long-term stability of the system are improved.

(4) The real-time semantic-geometric joint mapping method and system based on the open three-dimensional prior model are based on the perception semantic consistency, the cross-modal residual error and the model adaptability comprehensive evaluation result, the three-dimensional model is subjected to self-adaptive compression and texture correction by adopting a graph optimization and space-time fusion technology, automatic driving obstacle avoidance navigation, augmented reality viewpoint reconstruction and real-time output of collaborative operation are realized through multithreading task allocation, closed-loop updating of model optimization parameters and execution feedback is supported, and the overall efficiency of the system is improved.

Drawings

FIG. 1 is a block diagram flow chart of a real-time semantic-geometric joint mapping system based on an open three-dimensional prior model;

fig. 2 is a schematic diagram of steps of a real-time semantic-geometric joint mapping method based on an open three-dimensional prior model.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Example 1

Referring to fig. 1, the present invention provides a real-time semantic-geometric joint mapping system based on an open three-dimensional prior model, comprising:

In the embodiment, through a collaborative mechanism of multi-mode data acquisition, prior model active selection, cross-mode residual error correction and map building adaptive optimization, high precision, semantic consistency and dynamic adaptive capacity of three-dimensional map building are realized, real-time performance and reliability of path planning, obstacle recognition and environment sharing in automatic driving, augmented reality and collaborative operation scenes are effectively improved, and meanwhile, the method has calculation perception and compliance guarantee.

Example 2

The embodiment is explained in embodiment 1, please refer to fig. 1, specifically, the data acquisition module includes a geometric point cloud image acquisition unit, a semantic element task scene acquisition unit, and a multi-mode signal compliance information acquisition unit;

In the embodiment, the geometric point cloud, the depth map, the color image, the semantic visual information and the multi-mode sensing signal are fused through the data acquisition module, so that the high-precision synchronous acquisition of the structure and the semantic information in the complex automatic driving scene can be realized, the integrity, the instantaneity and the legality of the map building data are ensured by combining the calculation force monitoring and the compliance verification, and a reliable basis is provided for the follow-up high-precision map building and intelligent decision.

Example 3

In this embodiment, for the explanation in embodiment 1, please refer to fig. 1, specifically, the prior model active selection module includes a first parameter extraction unit, a first calculation unit, and a first analysis unit;

In the embodiment, the candidate models are accurately matched and comprehensively evaluated based on geometric features, semantic elements and task scene parameters, so that the high-consistency selection of the prior models and the actual environment is realized, the accuracy and adaptability of automatic driving map construction are remarkably improved, unnecessary calculation cost is reduced, and the response efficiency of the system is improved.

Example 4

In this embodiment, for explanation in embodiment 3, please refer to fig. 1, specifically, the first calculating unit is configured to search a candidate model in an open three-dimensional prior model library, and calculate and obtain a prior selection comprehensive coefficient PSI after dimensionless processing by combining the extracted geometric similarity parameter Sgeom, the consistency matching parameter Ssem and the task scene parameter Ttask, where the formula is as follows:

wherein w1, w2 and w3 represent weight coefficients;

Representing the influence of geometric similarity Sgeom on the prior model selection, occupying higher weight, being a key index, and directly reflecting the contribution of the matching degree of the three-dimensional geometric structure to the mapping precision and task adaptation;

representing the influence of semantic matching degree Ssem on the selection of a priori model, occupying medium weight, and reflecting the sensitivity of semantic information on task scene adaptation;

Characterizing the influence of task scene parameters Ttask on the selection of a priori model, and reflecting the importance of task demand on the matching of the model;

The prior selection judgment threshold Pth is obtained by carrying out prior model selection test and simulation analysis on a large number of automatic driving road scenes and multi-mode perception data under different task working conditions, counting the comprehensive distribution range of geometric similarity Sgeom, semantic matching degree Ssem and task scene parameters Ttask, and determining a reasonable prior selection threshold by combining model library coverage capability and scene switching response characteristics. The threshold is formulated by referring to related automatic driving mapping standards, three-dimensional model matching precision specifications and industry expert experience to accurately reflect the sensitivity of the system to the prior model matching degree, the model selection mismatch risk is timely identified, and mapping precision and task execution reliability are guaranteed.

In the embodiment, the first calculation unit and the first analysis unit are used for accurately calculating the priori selection comprehensive coefficients through weighted comprehensive geometric similarity, semantic matching degree and task scene parameters, and dynamically screening the optimal model based on the judgment threshold, so that the graph construction precision and efficiency can be effectively improved, redundant calculation is reduced, and the three-dimensional graph construction process is ensured to be highly matched with an actual task scene.

Example 5

In this embodiment, as explained in embodiment 1, please refer to fig. 1, specifically, the cross-mode residual error correction module includes a second parameter extraction unit, a second calculation unit, and a second analysis unit;

In this embodiment, by iterative closest point ICP registration and geometric feature comparison methods, differential analysis is performed on the acquired dense point cloud and the open three-dimensional prior model, and point cloud residuals are accurately extractedBased on the motion trail information of the dynamic target, a target contour extraction and boundary matching technology is adopted to obtain the boundary difference value of the radar appearanceBased on the characteristics of the environmental sound source and the structural echo signals, extracting acoustic reflection residual errors by adopting a waveform characteristic analysis and echo delay comparison method. The cross-mode residual error correction method can comprehensively integrate visual, radar and acoustic data characteristics, and remarkably improves matching precision and geometric consistency among different sensors, so that drawing errors are effectively reduced, and stability and reliability of the system under a complex road scene are enhanced.

Example 6

The embodiment is explained in embodiment 5, referring to fig. 1, specifically, the second computing unit is configured to introduce a visual point cloud, a millimeter wave radar and an acoustic signal to perform cross-mode residual correction based on a basic model of geometric and semantic joint mapping, and combine the extracted point cloud residualDifference of radar appearance boundarySum acoustic reflection residualAfter dimensionless processing, calculating and obtaining a cross-mode residual error comprehensive coefficient MRC, wherein the formula is as follows:

wherein a1, a2 and a3 represent weight coefficients;

The influence of the point cloud residual error on the cross-modal consistency is characterized, the point cloud residual error occupies higher weight and is a core index, and the contribution of the geometric error to the model consistency is directly reflected;

Representing the influence of radar appearance boundary difference on cross-modal consistency, occupying medium weight, and reflecting the sensitivity of appearance identification to model correction;

representing the influence of acoustic reflection residual errors on cross-modal consistency, occupying the next highest weight, and reflecting the auxiliary effect of environmental acoustic characteristics on residual error correction;

The residual correction threshold Mth is obtained by performing cross-modal residual test and statistical analysis on various road environments, dynamic targets and multi-modal sensing data, extracting the point cloud residual, radar appearance boundary difference and the amplitude distribution range of acoustic reflection residual, and determining a reasonable residual correction critical value by combining the geometric consistency requirement and real-time correction capability of the system. The threshold is formulated by referring to the related multi-mode fusion standard, the automatic driving mapping accuracy specification and the suggestion of an expert on the error tolerance so as to accurately reflect the sensitivity degree of the system to the cross-mode residual error, and the geometrical and semantic mapping error risk is timely identified, so that the stability and accuracy of a mapping model are improved.

In this embodiment, by introducing visual point cloud, millimeter wave radar and acoustic signals, cross-modal residual correction is performed on a basic model of geometric and semantic joint mapping, and point cloud residual is combinedDifference of radar appearance boundarySum acoustic reflection residualThe system can automatically start local encryption point cloud resampling, improve millimeter wave radar resolution and combine acoustic echo to implement local modification when the geometric consistency of the mapping model meets the precision requirement, thereby obviously improving the precision and consistency of three-dimensional mapping, enhancing the capturing capability of the system to scene details in complex environments, and ensuring the reliability and safety of applications such as automatic driving, augmented reality and the like.

Example 7

In this embodiment, for the explanation in embodiment 1, please refer to fig. 1, specifically, the mapping adaptation optimization module includes a third parameter extraction unit, a third calculation unit, and a third analysis unit;

In the embodiment, a time sequence modeling is performed on a dynamic target motion track through a third parameter extraction unit to obtain a time granularity memory factor Tmanagement, so that time sequence characteristics of target changes in a scene are effectively captured, meanwhile, based on processor occupancy rate, memory and bandwidth state information, a calculation load assessment and resource utilization rate normalization method is adopted to extract equipment calculation force factors Ccompute, real-time quantitative analysis on system operation performance is realized, and based on compliance metadata, a compliance privacy factor Lcompliance is obtained through a compliance feature statistics and tag consistency assessment method, so that copyright, privacy and compliance requirements are met in a data acquisition and mapping process, and accordingly, self-adaptability, stability and compliance of the mapping process are improved, and continuous and reliable operation capability of the system in a complex task scene is enhanced.

Example 8

In this embodiment, for the explanation in embodiment 7, please refer to fig. 1, specifically, the third calculating unit is configured to calculate and obtain the map-building adaptation comprehensive coefficient ZMAI through the obtained time granularity memory factor Tmemory, the device calculation force factor Ccompute and the compliance privacy factor Lcompliance after dimensionless processing, where the formula is as follows:

wherein s1, s2 and s3 represent weight coefficients;

Representing the influence of a time granularity memory factor Tmanagement on the graph construction adaptability, wherein the influence occupies higher weight and is a core index reflecting the continuous adaptability of dynamic scene change to the graph construction process;

Representing the influence of the calculation force factor Ccompute of the equipment on the graph construction adaptability, occupying medium weight, and reflecting the guarantee effect of calculation force resources on the graph construction efficiency and precision;

Characterizing the influence of a compliance privacy factor Lcompliance on the map construction adaptability, occupying the next highest weight, and reflecting the influence of compliance requirements on the feasibility and stability of the map construction process;

The acquisition mode of the map construction adaptability threshold value Zth is that a reasonable map construction adaptability threshold value is determined by carrying out experimental analysis on map construction processes under different calculation force conditions, dynamic scene changes and compliance requirements, counting the change distribution ranges of a time granularity memory factor Tmanagement, a device calculation force factor Ccompute and a compliance privacy factor Lcompliance and combining the map construction adaptability and real-time processing performance of the system. And (3) referring to related automatic driving map construction adaptability standards, equipment performance test specifications and system optimization experience of field experts, and formulating the threshold value to accurately reflect the map construction adaptability of the system under different scenes and calculation conditions, so as to ensure the stability, instantaneity and compliance of the map construction process.

In the embodiment, the third calculation unit is used for fusing the time granularity memory factor Tm, the equipment calculation force factor Ccompute and the compliance privacy factor Lcompliance, calculating the graph construction adaptation comprehensive coefficient ZMAI and realizing quantitative evaluation of the adaptability of the graph construction process, and the third analysis unit is used for intelligently judging whether the graph construction meets the adaptability requirement or not by comparing the third analysis unit with a preset graph construction adaptability threshold Zth, ensuring the output of high-quality graph construction results, and automatically triggering strategies such as time attenuation fusion, pruning or lightweight substitution of a calculation module, and compliance label updating when the conditions are not met, so that the instantaneity, stability and compliance of the graph construction process are remarkably improved, and continuous and efficient operation of the system under a complex dynamic scene is ensured and a reliable three-dimensional model is output.

Example 9

The embodiment is explained in embodiment 1, please refer to fig. 1, specifically, the real scene output module is configured to execute dynamic adaptation and multitasking instruction generation on the three-dimensional model output by the map-building adaptive optimization module according to the joint evaluation result of the perceived semantic consistency index PSI, the cross-modal residual error comprehensive coefficient MRC and the model adaptability index ZMAI, call a model reprojection technology based on map optimization and space-time fusion when PSI and ZMAI reach a preset threshold at the same time and MRC meets the residual error correction condition, perform adaptive compression and texture mapping correction on the three-dimensional geometric structure to generate an optimized model adapting to the current terminal computing power and task scene, automatically generate a corresponding execution instruction set according to the scene category by adopting a multithreading parallel semantic task allocation method, realize path planning and obstacle recognition of the three-dimensional map-building result in automatic driving obstacle avoidance navigation, viewpoint reconstruction and illumination matching in augmented reality rendering, and environment sharing and path synchronization output in collaborative operation, and write model optimization parameters and execution feedback data into the system log through an output result feedback mechanism.

In the embodiment, through joint evaluation of a perception semantic consistency index PSI, a cross-modal residual error comprehensive coefficient MRC and a model adaptability index ZMAI, an intelligent triggering model reprojection technology based on graph optimization and space-time fusion is used for carrying out self-adaptive compression and texture mapping correction on a three-dimensional geometric structure to generate an optimization model considering terminal computing power and task scene requirements, meanwhile, an execution instruction meeting scene requirements is automatically generated by adopting a multi-line Cheng Yuyi task allocation method, automatic driving obstacle avoidance navigation path planning and obstacle recognition, augmented reality viewpoint reconstruction and illumination matching are realized, and collaborative operation environment sharing and path synchronization are realized, so that the instantaneity, adaptability and multi-task collaborative capability of a graph building system are remarkably improved, a complete model optimization and execution feedback closed loop is formed through a feedback mechanism, and the reliability and traceability of the system are improved.

Example 10

Referring to fig. 2, the real-time semantic-geometric joint mapping method based on the open three-dimensional prior model specifically includes the following steps:

In the embodiment, a multi-mode sensing and mapping method based on laser radar, RGB-D camera, stereo camera, millimeter wave radar and acoustic array is constructed through multi-step collaborative processing, geometric features, semantic elements and task scene parameters are utilized to conduct prior model selection, model precision is improved through cross-mode residual correction, time granularity memory factors, equipment calculation force factors and compliance privacy factors are introduced to conduct mapping adaptability optimization, finally a high-precision optimization model adapting to terminal calculation force is generated through diagram optimization and space-time fusion technology, automatic driving obstacle avoidance, real-time output and closed-loop feedback of augmented reality rendering and collaborative operation are achieved through multi-line Cheng Yuyi task distribution, and therefore mapping precision, adaptability and real-time collaborative capability of a system are remarkably improved, and meanwhile data compliance and traceability are guaranteed.

The size of the threshold is set for convenience of comparison, and depends on the number of sample data and the number of cardinalities set for each group of sample data by a person skilled in the art, so long as the proportional relationship between the parameter and the quantized value is not affected.

The above formulas are all formulas obtained by collecting a large amount of data for software simulation and selecting a formula close to the true value, and coefficients in the formulas are set by a person skilled in the art according to practical situations, and the above is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto, and any person skilled in the art is within the technical scope of the present invention, and the technical scheme and the inventive concept according to the present invention are equivalent or changed, and are all covered in the protection scope of the present invention.

Claims

1. A real-time semantic-geometric joint mapping system based on an open three-dimensional prior model, comprising:

2. The real-time semantic-geometric joint mapping system based on the open three-dimensional prior model according to claim 1, wherein the data acquisition module comprises a geometric point cloud image acquisition unit, a semantic element task scene acquisition unit and a multi-mode signal compliance information acquisition unit;

3. The real-time semantic-geometric joint mapping system based on the open three-dimensional prior model according to claim 1, wherein the prior model active selection module comprises a first parameter extraction unit, a first calculation unit and a first analysis unit;

4. The real-time semantic-geometric joint mapping system based on an open three-dimensional prior model according to claim 3, wherein the first computing unit is configured to search a candidate model in an open three-dimensional prior model library, and calculate and obtain a prior selection comprehensive coefficient PSI after dimensionless processing by combining the extracted geometric similarity parameter Sgeom, the consistency matching parameter Ssem and the task scene parameter Ttask;

5. The real-time semantic-geometric joint mapping system based on the open three-dimensional prior model according to claim 1, wherein the cross-modal residual correction module comprises a second parameter extraction unit, a second calculation unit and a second analysis unit;

6. The system for real-time semantic-geometric joint mapping based on open three-dimensional prior model according to claim 5, wherein the second computing unit is used for introducing visual point cloud, millimeter wave radar and acoustic signals to perform cross-modal residual correction based on a basic model of geometric-semantic joint mapping, and combining the extracted point cloud residualDifference of radar appearance boundarySum acoustic reflection residualAfter dimensionless processing, calculating and obtaining a cross-mode residual error comprehensive coefficient MRC;

7. The real-time semantic-geometric joint mapping system based on the open three-dimensional prior model according to claim 1, wherein the mapping adaptation optimization module comprises a third parameter extraction unit, a third calculation unit and a third analysis unit;

8. The real-time semantic-geometric joint mapping system based on the open three-dimensional prior model according to claim 7, wherein the third calculation unit is configured to calculate and obtain a mapping adaptation comprehensive coefficient ZMAI after dimensionless processing through the obtained time granularity memory factor Tmemory, the device calculation force factor Ccompute and the compliance privacy factor Lcompliance;

9. The real-time semantic-geometric joint mapping system based on the open three-dimensional prior model according to claim 1 is characterized in that the real scene output module is used for executing dynamic adaptation and multitasking instruction generation on the three-dimensional model output by the mapping adaptation optimization module according to a joint evaluation result of a perception semantic consistency index PSI, a cross-modal residual error comprehensive coefficient MRC and a model adaptability index ZMAI, when PSI and ZMAI reach a preset threshold value at the same time and MRC meets residual error correction conditions, invoking a model reprojection technology based on graph optimization and space-time fusion, performing self-adaptive compression and texture mapping correction on the three-dimensional geometric structure to generate an optimization model adapting to the current terminal computing power and task scene, adopting a semantic task allocation method parallel to threads, automatically generating a corresponding execution instruction set according to scene types, realizing path planning and obstacle recognition of the three-dimensional mapping result in automatic driving obstacle avoidance navigation, viewpoint reconstruction and illumination matching in augmented reality rendering and environment sharing and path synchronization output in collaborative operation, and writing model optimization parameters and execution feedback data system through an output result feedback log returning mechanism.

10. The real-time semantic-geometric joint mapping method based on the open three-dimensional prior model is applied to the real-time semantic-geometric joint mapping system based on the open three-dimensional prior model, and is characterized by comprising the following steps: