WO2024259614A1 - Procédés et processeurs pour une modélisation implicite avec des nuages de points multi-balayage pour une reconstruction de véhicule 3d dans une conduite autonome - Google Patents

Procédés et processeurs pour une modélisation implicite avec des nuages de points multi-balayage pour une reconstruction de véhicule 3d dans une conduite autonome Download PDF

Info

Publication number
WO2024259614A1
WO2024259614A1 PCT/CN2023/101585 CN2023101585W WO2024259614A1 WO 2024259614 A1 WO2024259614 A1 WO 2024259614A1 CN 2023101585 W CN2023101585 W CN 2023101585W WO 2024259614 A1 WO2024259614 A1 WO 2024259614A1
Authority
WO
WIPO (PCT)
Prior art keywords
network
point cloud
features
shape
point
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/101585
Other languages
English (en)
Inventor
Yuan Ren
Yibo Liu
Kelly ZHU
Bingbing LIU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to PCT/CN2023/101585 priority Critical patent/WO2024259614A1/fr
Publication of WO2024259614A1 publication Critical patent/WO2024259614A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/56—Particle system, point based geometry or rendering

Definitions

  • the present technology relates generally to 3D object reconstruction; and in particular, to electronic devices and methods for executing 3D reconstruction of objects.
  • Autonomous vehicles also known as self-driving cars, have gained attention in recent years due to their potential to revolutionize transportation and enhance road safety. These vehicles rely on a combination of sensors, algorithms, and artificial intelligence to navigate through complex environments without human intervention.
  • LiDAR technology has emerged as an important component for autonomous vehicles, offering a robust and accurate perception system.
  • LiDAR systems employ laser beams to measure distances and create detailed three-dimensional representations of the surrounding environment. For example, by emitting laser pulses and measuring the time it takes for the pulses to bounce back after hitting objects, LiDAR sensors can precisely calculate the distance and location of obstacles, pedestrians, vehicles, and other elements in the vehicle's vicinity.
  • LIDAR sensors provide a rich and high-resolution point cloud representation of the environment, enabling precise detection and localization of objects. This detailed perception allows autonomous vehicles to make informed decisions in real-time, enhancing their ability to navigate complex scenarios and avoid potential hazards.
  • LiDAR sensors are capable of operating in various environmental conditions, including low-light situations, rain, fog, and snow. This versatility ensures that autonomous vehicles equipped with LiDAR can operate reliably and safely in different weather conditions, enhancing their overall robustness and dependability.
  • LiDAR sensors can complement other sensor modalities used in autonomous vehicles, such as cameras and radar. While cameras provide visual information and radar can detect objects based on their reflective properties, LiDAR sensors offer precise depth information that is independent of lighting conditions, making it valuable for object detection and tracking.
  • LiDAR sensors may experience difficulties in accurately detecting certain objects with low reflectivity, such as non-metallic or transparent materials.
  • 3D vehicle reconstruction from sparse and partial point clouds may be required for autonomous driving, where sensor simulators require 3D foreground object models as their input.
  • a sensor simulator In the autonomous driving industry, a sensor simulator usually requires 3D foreground object models as input. In order to guarantee the diversity of foreground objects, a large number of models may be needed. Developers have realized that generating an object library is an expensive task and using 3D reconstruction to that end can greatly save costs. Using partial LiDAR sweeps of vehicles to reconstruct an object library can reduce domain differences and therefore maintain the original distributions of the object size, type and pose, for example.
  • Explicit modeling-based approaches can be further classified into point-based, voxel-based, and mesh-based and directly represent 3D object shape structure with points, voxels, and meshes respectively.
  • Point-based 3D reconstruction methods such as Point Completion Network (PCN) , GRNet, and PoinTr, output a point cloud as a shape representation of the 3D reconstruction.
  • Voxel-based modeling described volumes by subdividing the space into a 3D grip and mesh-based approaches generate meshes using a fixed reference template from a given object class.
  • Conventional explicit reconstruction approaches, including point-based, voxel-based, and mesh-based suffer from problems such as low resolution and high memory costs.
  • “explicit” features may refer to features generated using one or more explicit modeling techniques. It can be said that explicit features represent features of a first type and which are extracted from an explicitly modeled object.
  • intermediate features of PCN models, GRNet models, and PoinTr models may be used as features of the first type.
  • features of the first type can be generated using (i) a voxel-based technique for shape completion using 3D-Encoder-Predictor CNNs and shape synthesis, and (ii) a mesh-based technique for multi-chart generative surface modeling.
  • the PoinTr model short for "Point Transformer, " is a deep learning model designed for image recognition and object detection tasks.
  • the PoinTr model uses convolutional neural networks (CNNs) and transformers to effectively capture spatial relationships and contextual information within images. It uses self-attention mechanisms to model long-range dependencies between image regions, allowing it to attend to relevant image features and objects.
  • CNNs convolutional neural networks
  • the PoinTr model replaces the convolutional layers with transformer layers, enabling it to perform both local and global image analysis. By considering the entire image as a sequence of patches, the model can learn patterns and relationships between different parts of the image.
  • SDF is an implicit function that specifies whether a querying point is inside, or outside, the surface of an object. The value of an SDF represents the distance of a querying point from the surface of the object and its sign, either positive or negative, indicates whether the querying point lies on the exterior or interior of the surface, respectively.
  • “implicit” features may refer to features generated using one or more implicit modeling techniques. It can be said that implicit features represent features of a second type and which are extracted from an implicitly modeled object. In some embodiments, features of the second type may be extracted using occupancy networks by learning 3D reconstruction in function space, NeuS models by learning neural implicit surfaces by volume rendering for multi-view reconstruction, NeAT models by learning neural implicit surfaces with arbitrary topologies from multi-view Images, and the like.
  • NeAT Neuro-Evolution of Augmenting Topologies
  • NeAT models combine evolutionary algorithms with neural network training. They evolve both the structure and weights of neural networks through mutation and crossover operations. By evaluating fitness and selecting networks with higher scores, NeAT models can evolve architectures well-suited to specific tasks.
  • NeuS can be implemented as a combination of a DeepSDF model and a NeRF model. It can be said that a DeepSDF model may be a MLP portion which is used to represent the 3D shape of the object, and NeRF model may be an other MLP portion which represents the texture of the object.
  • the 3D shape can be stored as a one-dimensional memory-saving latent code and the continuous SDF supports 3D mesh extraction at different resolutions.
  • the input of a DeepSDF network is a latent code and a query point.
  • the output is a signed distance of this query point.
  • the parameters in the DeepSDF network are fixed after training, so the 3D shape is represented by the latent code.
  • the DeepSDF method mainly focuses on recovering 3D shapes from a single-view partial point cloud and fails to leverage the multi-sweep information of vehicles.
  • single-view methods usually produce low-fidelity results.
  • the DeepSDF method only takes the given surface points of a partial point cloud into consideration when generating 3D shapes. This can lead to suboptimal reconstruction results in areas not captured in the partial point cloud, since the DeepSDF method will perform reconstruction arbitrarily based on prior knowledge learned during training. Let it be assumed that a point cloud only captures a front of a pickup truck.
  • a DeepSDF method may be used to reconstruct the object as a pickup truck, or alternatively as an SUV, since the front of the two object models is similar.
  • the process of reconstructing the rear of the object can be said to be “arbitrary” and/or involve uncertainty.
  • PC1 and PC2 are point clouds of a same object from different viewpoints.
  • the DeepSDF method reconstruction results Mesh 1 and Mesh 2 are reconstruction outputs of the DeepSDF method applied onto PC1 and PC2, respectively. It should be noted that although Mesh 1 and Mesh 2 are reconstruction outputs of the same object, they are notably different. Moreover, due to the noise and registration error, surface reconstructed by the stacked multi-sweep point cloud is usually not smooth, as seen in FIG. 2B.
  • conventional geometry methods e.g. Iterative closest point (ICP)
  • ICP Iterative closest point
  • a stacking operation may be performed based on the annotations (e.g., bounding box) .
  • a neural network can be trained for performing the stacking operation. For example, a processor may use a neural network to learn the relative poses among point clouds (e.g., weakly-supervised 3D shape completion in the wild) .
  • 3D vehicle reconstruction from sparse and partial point clouds may be used in variety of applications in the autonomous driving industry.
  • a processor configured to infer from sparse and/or partial point clouds 3D structure of one or more vehicles at a variety of resolutions during in-use. This information may be further employed in one or more downstream tasks executed by the processor during operation the autonomous vehicle.
  • a processor configured to perform an implicit modeling technique using multi-sweep point clouds.
  • multi-sweep point clouds refer to a collection of point clouds obtained from multiple scanning or sensing passes over the same area or object.
  • Each individual point cloud in the multi-sweep dataset represents a separate scan or sweep taken at a different time or from a different perspective.
  • the point cloud data is typically acquired using technologies such as LIDAR, for example.
  • LIDAR LIDAR
  • the multi-sweep point clouds provide a detailed representation of the scanned area. It can capture different viewpoints, occluded areas, or changes that may occur over time.
  • a stacked point cloud refers to a combination or fusion of multiple point clouds into a single composite point cloud. Unlike multi-sweep point clouds, where each individual scan or sweep is preserved separately, a stacked point cloud merges the data from different sources or scans into a unified representation. The process of creating a stacked point cloud involves aligning and combining the individual point clouds, sometimes obtained from different sensors, scanning devices, or viewpoints. The alignment is necessary to ensure that all the points are registered correctly in a common coordinate system. In some embodiments, multi-sweep point clouds may be stacked into a stacked multi-sweep point cloud.
  • a processor configured to use multi-sweep point cloud input to (i) extract implicit features in the latent space, and (ii) extract global features, and (iii) combine the extracted implicit features and extracted explicit features into a common output.
  • a processor configured to run a model architecture for leveraging multi-sweep LiDAR point cloud data as input, and which is trained on both explicit global features and implicit features in the latent space.
  • a model architecture for leveraging multi-sweep LiDAR point cloud data as input, and which is trained on both explicit global features and implicit features in the latent space.
  • Such a model may allow generating robust and higher fidelity reconstruction results on real-world autonomous driving datasets.
  • a processor may be configured to perform a stacking operation onto one or more input point clouds (e.g., the first and second point clouds) , and thereby generate a stacked point cloud.
  • a stacked point cloud refers to a combination or fusion of multiple point clouds into a single point cloud.
  • a stacked point cloud is generated from merging the data from different sources or scans into a unified representation.
  • the process of creating a stacked point cloud involves aligning and combining the individual point clouds, sometimes obtained from different sensors, scanning devices, and/or viewpoints. The alignment may be necessary to ensure that all the points are registered correctly in a common coordinate system.
  • multi-sweep point clouds may be stacked into a stacked multi-sweep point cloud.
  • a method of performing 3D reconstruction of an object comprising acquiring a first point cloud and a second point cloud, the first and second point clouds being multi-sweep point clouds representing at least the object.
  • the method comprises extracting, using a first network, explicit features about a shape of the object based on the first and second point clouds, the first network having been trained to explicitly model the shape of the object.
  • the method comprises extracting, using a second network, implicit features about the shape of the object based on the first and second point clouds, the second network having been trained to implicitly model the shape of the object.
  • the method comprises generating, using a third network, a latent code by projecting the implicit features and the explicit features into a latent space.
  • the method comprises generating a 3D shape representative of the object using the latent code.
  • the first network is at least one of a PCN encoder, and a PointNet encoder.
  • the second network is at least one of a DeepSDF encoder, and a ONet encoder.
  • the method further comprises generating a first modified point cloud and a second modified point cloud by applying a Farthest Point Sampling (FPS) function onto the first point cloud and the second point cloud, respectively.
  • the first modified point cloud and the second modified point cloud have N respective number of points, N being an integer.
  • the extracting the explicit features comprises extracting the explicit features based on the first modified point cloud and the second modified point cloud.
  • the third network comprises a pooling layer for combining the implicit features and the explicit features and a fully-connected layer for outputting the latent code.
  • the input further comprises images, voxels, meshes, and/or any combination thereof.
  • the method further comprises, generating a database including the latent code.
  • an electronic device for performing 3D reconstruction of an object.
  • the electronic device comprises a processor and a memory, the memory storing instructions that when executed by the processor cause the electronic device to acquire a first point cloud and a second point cloud, the first and second point clouds being multi-sweep point clouds representing at least the object, extract, using a first network, explicit features about a shape of the object based on the first and second point clouds, the first network having been trained to explicitly model the shape of the object, extract, using a second network, implicit features about the shape of the object based on the first and second point clouds, the second network having been trained to implicitly model the shape of the object, generate, using a third network, a latent code by projecting the implicit features and the explicit features into a latent space, and generate a 3D shape representative of the object using the latent code.
  • the first network is at least one of a PCN encoder, and a PointNet encoder.
  • the second network is at least one of a DeepSDF encoder, and a ONet encoder.
  • the electronic device further generates a first modified point cloud and a second modified point cloud by applying a Farthest Point Sampling (FPS) function onto the first point cloud and the second point cloud, respectively, the first modified point cloud and the second modified point cloud have N respective number of points, N being an integer.
  • the electronic device extracting the explicit features comprises the extracting the explicit features based on the first modified point cloud and the second modified point cloud.
  • the third network comprises a pooling layer for combining the implicit features and the explicit features and a fully-connected layer for outputting the latent code.
  • the input further comprises images, voxels, meshes, and/or any combination thereof.
  • the electronic device further generates a database including the latent code.
  • a “server” is a computer program that is running on appropriate hardware and is capable of receiving requests (e.g., from devices) over a network, and carrying out those requests, or causing those requests to be carried out.
  • the hardware may be one physical computer or one physical computer system, but neither is required to be the case with respect to the present technology.
  • a “server” is not intended to mean that every task (e.g., received instructions or requests) or any particular task will have been received, carried out, or caused to be carried out, by the same server (i.e., the same software and/or hardware) ; it is intended to mean that any number of software elements or hardware devices may be involved in receiving/sending, carrying out or causing to be carried out any task or request, or the consequences of any task or request; and all of this software and hardware may be one server or multiple servers, both of which are included within the expression “at least one server” .
  • device is any computer hardware that is capable of running software appropriate to the relevant task at hand.
  • devices include personal computers (desktops, laptops, netbooks, etc. ) , smartphones, and tablets, as well as network equipment such as routers, switches, and gateways.
  • network equipment such as routers, switches, and gateways.
  • a device acting as a device in the present context is not precluded from acting as a server to other devices.
  • the use of the expression “adevice” does not preclude multiple devices being used in receiving/sending, carrying out or causing to be carried out any task or request, or the consequences of any task or request, or steps of any method described herein.
  • a “database” is any structured collection of data, irrespective of its particular structure, the database management software, or the computer hardware on which the data is stored, implemented or otherwise rendered available for use.
  • a database may reside on the same hardware as the process that stores or makes use of the information stored in the database or it may reside on separate hardware, such as a dedicated server or plurality of servers. It can be said that a database is a logically ordered collection of structured data kept electronically in a computer system
  • information includes information of any nature or kind whatsoever capable of being stored in a database.
  • information includes, but is not limited to audiovisual works (images, movies, sound records, presentations etc. ) , data (location data, numerical data, etc. ) , text (opinions, comments, questions, messages, etc. ) , documents, spreadsheets, lists of words, etc.
  • component is meant to include software (appropriate to a particular hardware context) that is both necessary and sufficient to achieve the specific function (s) being referenced.
  • computer usable information storage medium is intended to include media of any nature and kind whatsoever, including RAM, ROM, disks (CD-ROMs, DVDs, floppy disks, hard drivers, etc. ) , USB keys, solid state-drives, tape drives, etc.
  • first server and “third server” is not intended to imply any particular order, type, chronology, hierarchy or ranking (for example) of/between the server, nor is their use (by itself) intended imply that any “second server” must necessarily exist in any given situation.
  • references to a “first” element and a “second” element does not preclude the two elements from being the same actual real-world element.
  • a “first” server and a “second” server may be the same software and/or hardware, in other cases they may be different software and/or hardware.
  • Implementations of the present technology each have at least one of the above-mentioned object and/or aspects, but do not necessarily have all of them. It should be understood that some aspects of the present technology that have resulted from attempting to attain the above-mentioned object may not satisfy this object and/or may satisfy other objects not specifically recited herein.
  • FIG. 1 illustrates an example of a computing device that may be used to implement any of the methods described herein.
  • FIG. 2A shows point clouds of a same ground-truth object from different viewpoints and reconstruction results using a conventional reconstruction technique.
  • FIG. 2B shows a stacked point cloud and a mesh obtained directly from the stack point cloud using a conventional reconstruction technique.
  • FIG. 3 illustrates a schematic representation of a model configured to extract both implicit and explicit features for generating a representation in a latent space, in accordance with at least some non-limiting embodiments of the present technology.
  • FIG. 4 illustrates a non-limiting example of a neural architecture of a model in accordance with at least some non-limiting embodiments of the present technology.
  • FIG. 5 illustrates a comparison of different methods using the Waymo dataset for 3D object reconstruction.
  • FIG. 6 illustrates a comparison of different methods using the KITTI dataset for 3D object reconstruction.
  • FIG. 7 is a scheme-block illustration of a method executed by a processor of the computing device of FIG. 1, in accordance with at least some non-limiting embodiments of the present technology.
  • processor may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software.
  • the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared.
  • the processor may be a general purpose processor, such as a central processing unit (CPU) or a processor dedicated to a specific purpose, such as a digital signal processor (DSP) .
  • CPU central processing unit
  • DSP digital signal processor
  • processor should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, application specific integrated circuit (ASIC) , field programmable gate array (FPGA) , read-only memory (ROM) for storing software, random access memory (RAM) , and non-volatile storage.
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • ROM read-only memory
  • RAM random access memory
  • non-volatile storage non-volatile storage.
  • Other hardware conventional and/or custom, may also be included.
  • modules may be represented herein as any combination of flowchart elements or other elements indicating performance of process steps and/or textual description. Such modules may be executed by hardware that is expressly or implicitly shown. Moreover, it should be understood that module may include for example, but without being limitative, computer program logic, computer program instructions, software, stack, firmware, hardware circuitry or a combination thereof which provides the required capabilities.
  • FIG. 1 illustrates a diagram of a computing environment 100 in accordance with an embodiment of the present technology is shown.
  • the computing environment 100 may be implemented by any of a conventional personal computer, a computer dedicated to operating and/or monitoring systems relating to a data center, a controller and/or an electronic device (such as, but not limited to, a mobile device, a tablet device, a server, a controller unit, a control device, a monitoring device etc. ) and/or any combination thereof appropriate to the relevant task at hand.
  • the computing environment 100 comprises various hardware components including one or more single or multi-core processors collectively represented by a processor 110, a solid-state drive 120, a random access memory 130 and an input/output interface 150.
  • the computing environment 100 may also be a sub-system of one of the above-listed systems. In some other embodiments, the computing environment 100 may be an “off the shelf” generic computer system. In some embodiments, the computing environment 100 may also be distributed amongst multiple systems. The computing environment 100 may also be specifically dedicated to the implementation of the present technology. As a person in the art of the present technology may appreciate, multiple variations as to how the computing environment 100 is implemented may be envisioned without departing from the scope of the present technology.
  • Communication between the various components of the computing environment 100 may be enabled by one or more internal and/or external buses 160 (e.g. a PCI bus, universal serial bus, IEEE 1394 “Firewire” bus, SCSI bus, Serial-ATA bus, ARINC bus, etc. ) , to which the various hardware components are electronically coupled.
  • internal and/or external buses 160 e.g. a PCI bus, universal serial bus, IEEE 1394 “Firewire” bus, SCSI bus, Serial-ATA bus, ARINC bus, etc.
  • the input/output interface 150 may allow enabling networking capabilities such as wire or wireless access.
  • the input/output interface 150 may comprise a networking interface such as, but not limited to, a network port, a network socket, a network interface controller and the like. Multiple examples of how the networking interface may be implemented will become apparent to the person skilled in the art of the present technology.
  • the networking interface may implement specific physical layer and data link layer standard such as Ethernet, Fibre Channel, Wi-Fi or Token Ring.
  • the specific physical layer and the data link layer may provide a base for a full network protocol stack, allowing communication among small groups of computers on the same local area network (LAN) and large-scale network communications through routable protocols, such as Internet Protocol (IP) .
  • IP Internet Protocol
  • the solid-state drive 120 stores program instructions suitable for being loaded into the random access memory 130 and executed by the processor 110 for executing operating data centers based on a generated machine learning pipeline.
  • the program instructions may be part of a library or an application.
  • the computing environment 100 may be implemented as part of a cloud computing environment.
  • a cloud computing environment is a type of computing that relies on a network of remote servers hosted on the internet, for example, to store, manage, and process data, rather than a local server or personal computer. This type of computing allows users to access data and applications from remote locations, and provides a scalable, flexible, and cost-effective solution for data storage and computing.
  • Cloud computing environments can be divided into three main categories: Infrastructure as a Service (IaaS) , Platform as a Service (PaaS) , and Software as a Service (SaaS) .
  • IaaS Infrastructure as a Service
  • PaaS Platform as a Service
  • SaaS Software as a Service
  • users can rent virtual servers, storage, and other computing resources from a third-party provider, for example.
  • a PaaS environment users have access to a platform for developing, running, and managing applications without having to manage the underlying infrastructure.
  • users can access pre-built software applications that are hosted by a third-party provider, for example.
  • cloud computing environments offer a range of benefits, including cost savings, scalability, increased agility, and the ability to quickly deploy and manage applications.
  • FIG. 3 there is depicted a schematic pipeline 300 of the processor 110 generating instance features 314.
  • the processor 110 is configured to provide input 302 to an implicit information extractor 308 and to an implicit information extractor 304.
  • the content of the input 302 may vary depending on inter alia various implementations of the present technology.
  • the input 302 may comprise point clouds, voxels, meshes, images or any combination thereof.
  • the input 302 may comprise point clouds and images of a given scene.
  • the explicit information extractor 308 is configured to extract explicit features 310 from an explicit shape of an object in the input 302 (e.g., point clouds, meshes, voxels, and images) . How the explicit information extractor 308 can be implemented is not particularly limiting. A choice of machine learning model for implement the explicit information extractor 308 may be made based on inter alia a format of the input 302. For example, if the input 302 comprises multiple point clouds, the explicit information extractor 308 can be implemented as PointNet encoder, Point Completion Network (PCN) encoder, and the like.
  • PCN Point Completion Network
  • a PointNet encoder is a deep learning architecture designed for processing point cloud data, which represents the geometric structure of objects or environments.
  • the PointNet encoder can employ shared Multi-Layer Perceptrons (MLPs) to capture local geometric information, while max-pooling aggregates the local features to generate a global feature vector.
  • MLPs Multi-Layer Perceptrons
  • the PointNett encoder can also comprise a transformation network to handle rotation and translation invariance.
  • the PointNet encoder may process unstructured point clouds.
  • a PCN encoder is a deep learning architecture designed for point cloud completion tasks.
  • Point cloud completion aims to reconstruct missing or occluded parts of a 3D object based on the available partial point cloud data.
  • the PCN encoder addresses the challenge of completing partial point clouds by effectively capturing the underlying structure and geometry. It takes in the incomplete point cloud as input and generates a complete and coherent representation of the object.
  • the PCN encoder employs a hierarchical feature extraction process to capture local and global contextual information. It consists of multiple layers that progressively refine the feature representation of the input point cloud. This hierarchical structure allows the network to capture both fine-grained details and higher-level semantics.
  • the encoder may comprise MLPs and convolutional layers, for example.
  • the PCN encoder may be trained using a combination of supervised and unsupervised learning approaches.
  • Supervised learning utilizes paired datasets, where complete and partial point clouds are available, to provide direct supervision for training.
  • Unsupervised learning techniques such as adversarial training or reconstruction loss, are also employed to encourage the model to generate plausible and realistic completions.
  • the implicit information extractor 304 is configured to extract implicit features 306 from an implicit shape of the object in the input 302, such as via SDFs, Neural Radiance Field (NeRF) model, and occupancy functions, for example.
  • SDFs Seural Radiance Field
  • NeRF Neural Radiance Field
  • NeRF model comprises a deep learning architecture that enables synthesis of detailed 3D scenes from a sparse set of 2D images or 2D-to-3D correspondences.
  • NeRF models are configured to model the volumetric scene as a continuous function that predicts radiance values for different 3D points within the scene. By training a neural network on a set of observed images, the NeRF model can learn to infer the volumetric density and color properties of the scene. This allows for the generation of novel views and renderings from different viewpoints.
  • the NeRF model can capture scene details, including reflections, refractions, and complex lighting effects, during 3D reconstruction.
  • the NeRF model can operate by sampling rays from the camera through the scene and using a deep neural network to estimate the radiance along each ray.
  • the NeRF model may enable synthesis of novel views by integrating information from multiple rays and/or viewpoints.
  • occupancy functions are configured to model an occupancy status of space in a 3D environment. These functions may be used to infer the presence, or absence, of object (s) and/or structure (s) within a given volume.
  • the occupancy function may be used to discretize the 3D space into voxels or cells and assign a binary value to each voxel indicating whether it is occupied or unoccupied. This binary representation forms a voxel grid or occupancy grid, which serves as a compact and structured representation of the 3D scene.
  • Occupancy functions can be estimated from sensor data, such as point clouds, depth maps, or volumetric data obtained from techniques like LiDAR or structured light scanning. Various algorithms and techniques can be employed to process the sensor data and determine the occupancy status of each voxel. In some cases, a probabilistic occupancy estimation may be performed where the occupancy function assigns a probability value to each voxel, representing the likelihood of it being occupied. This probabilistic representation may be selected to facilitate fusion of multiple sensor modalities or data sources.
  • the implicit information extractor 304 can be implemented is not particularly limiting.
  • a choice of machine learning model for implement the implicit information extractor 304 may be made based on inter alia a format of the input 302. For example, if the input is multiple point clouds, the implicit information extractor 304 can be implemented as a DeepSDF model, ONet model, and the like.
  • DeepSDF model comprises a deep learning architecture that models the surface of 3D objects as a continuous implicit function and estimates the signed distance to the object surface from any given point in space.
  • DeepSDF is a neural network trained to learn the implicit representation of 3D shapes from a set of example shapes. The network is provided with 3D coordinates as input and predicts the signed distance to the surface.
  • the DeepSDF model learns to capture the underlying shape structure and generalizes to reconstruct new shapes.
  • the DeepSDF model may comprise one or more deep neural networks, such as MLPs and/or convolutional networks for learning shape representation.
  • an optimization process can be used to minimize the difference between the predicted signed distances and ground truth distances from the training shapes.
  • the ONet model comprises a deep learning architecture configured to model the occupancy status of 3D space. It provides a compact and structured representation of the 3D scene by estimating the occupancy probabilities of a grid of 3D voxels.
  • the ONet model comprises neural networks that are used to model the 3D occupancy of space based on observed data, such as point clouds, depth maps, or images. It estimates the probability of each voxel being occupied, which indicates whether an object or structure exists at that location in space.
  • the ONet model comprises of an encoder-decoder architecture where the encoder processes the input data and extracts features, while the decoder predicts the occupancy probabilities for each voxel in the 3D grid.
  • the network is trained on labeled data, typically using supervised learning techniques, to learn the relationship between the observed data and the corresponding occupancy probabilities.
  • the processor 110 is configured to provide the explicit features 310 and the implicit features 306 are provided to a pooling module 312.
  • the pooling module 312 is configured to pool explicit and implicit information and project the instance features 314 to a space and/or size.
  • the output may be in a form of a 1 1x256 vector, called a latent code.
  • the pooling module 312 may be a neural architecture with a max pooling layer and two fully-connected layers. In other embodiments, if the input format is meshes and/or images, neural architecture of the pooling module 312 may be implemented differently to be suitable for use with the input format.
  • the input 402 comprises multi-view LiDAR sweeps of a given object.
  • the object is a same vehicle shown in the multi-view LiDAR sweep data.
  • the processor 110 provides the input 402 to a Farthest Point Sampling (FPS) function.
  • the FPS function is configured to provide a network 410 with processed point clouds, all including N respective number of points.
  • the FPS function may be executed with a number of centroids parameter set to “256" .
  • the processor 110 is configured to perform explicit information extraction via the network 410.
  • the network 410 is a shared PCN model employed to extract global features from respective processed point clouds.
  • the processor 110 is configured to perform implicit information extraction from the input 402 using a network 406.
  • the network 406 may be a DeepSDF model for generate latent code (s) 408 for respective partial point cloud (s) from the input 402.
  • Outputs of the network 410 and of the network 406 e.g., tensors of latent codes and global features
  • the network 414 transforms the concatenated data into instance features via a max pooling operation (s) .
  • the network 414 projects the instance features to the size of a latent code 416 by two fully-connected layers.
  • post-FPS point clouds are used as input into the network 410 to extract explicit global features for each post-FPS point cloud.
  • the original partial point clouds from the input 402 are used as input into the network 406 to extract latent codes for each partial point cloud.
  • the processor 110 may be configured execute a combination operation 412 to combine outputs of the networks 410 and 406.
  • the combination operation 412 may be a concatenation operation performed by the processor 110 on the output data of the networks 410 and 406.
  • a combined output is the provided to an average pooling operation to aggregate the information into a single instance tensor.
  • this instance tensor is transformed into the predicted latent code 416 by one or more fully-connected layers.
  • the processor 110 is configured to use a pre-trained DeepSDF decoder 418 to map the latent code 416 to the SDF for the given object (vehicle) .
  • the processor 110 may be configured to mesh extrication function to extract a 3D mesh at a variety of different resolutions.
  • a predicted latent code may be converted to an SDF using a pre-trained DeepSDF decoder.
  • the iso-surface composed of querying points with zero SDF values represents the surface of the given object and the implicit surface can be rasterized to a 3D mesh using Marching Cubes, for example. Since the SDF is a continuous function, 3D mesh extraction can be supported at a variety of different resolutions.
  • the neural architecture 400 can be trained by the processor 110 in a two-stage process.
  • the processor 110 is configured to train the networks 406 and 418.
  • the processor 110 may be configured to train a DeepSDF decoder using watertight CAD models.
  • the processor 110 may be configured to obtain a DeepSDF decoder pre-trained based on a ShapeNetV2 dataset.
  • a CAD model may be stored by the processor 110 in memory as an STL file.
  • an STL file describes a raw, unstructured triangulated surface by the unit normal and vertices (ordered by the right-hand rule) of the triangles using a 3D cartesian system. It should be noted that in some cases, to perform a 3D analysis on an STL model, the model may need to be watertight before being imported.
  • a watertight model may be defined as a model with at least some of the following features: (i) every triangle edge in the model has exactly two neighbors, thereby there are no holes or non-manifold edges, (ii) every node in the triangle is connected to only one fan of triangles around it-that is, for a given node, every triangle that shares that node must be accessible from any other triangle that shares the same node by moving across triangle edges (this may also be a condition on the domain being a proper manifold) , (iii) there are no geometric overlaps or intersections in the model, where the model may be properly manifold but may still have overlaps and/or intersections due to triangles intersecting or overlapping with each other geometrically, and (iv) there are no geometric errors that produce unrealistically thin areas, where although similar to an overlap, may not be severe enough to be picked up as an overlap.
  • a partial point cloud generation method can be used for generating the training set from the CAD models.
  • at least some partial point cloud generation methods allow to position a virtual LiDAR with real-world parameters (resolution and sampling pattern) about the object and simulates the point cloud captured by the virtual LiDAR from that position. These methods differ from a method typically used for DeepSDF decoder training, where a simulated depth camera is used as the virtual sensor instead of a LiDAR.
  • the processor 110 can use the pre-trained DeepSDF decoder to generate a latent code for a corresponding input partial point cloud.
  • the processor 110 is configured to train the networks 410 and 414 to reconstruct 3D objects based on partial multi-sweep point clouds and their corresponding latent codes.
  • multi-sweep point clouds refer to data representing an object that has been scanned by a LiDAR sensor, for example, in several frames. However, the point of view of these scans may or may not be the same. It should be noted that multi-view point clouds are a subset of multi-sweep clouds where the point clouds are taken from different points of view.
  • g ( ⁇ ) represents the function of the implicit shape prediction network 406 and ⁇ represents the parameters of the networks 419 and 414.
  • the training objective of second training stage can be defined as:
  • an Adam optimizer may be employed with a learning rate of 1e -5 and network training includes 20 epochs.
  • an Adam optimizer is an optimization algorithm used in machine learning for updating the parameters of a model during the training process. It is an extension of the stochastic gradient descent (SGD) optimization method that incorporates adaptive learning rates.
  • FIG. 5 there is depicted experimental results comparing different 3D reconstruction methods including some 3D reconstructions methods contemplated in the context of the present technology.
  • a set of input sweeps 502 a set of ground-truth shapes 504 associated with the set of input sweeps 502, outputs 506 of a plurality of conventional methods, and outputs 508 in accordance with some implementations of the present technology.
  • experimentation has been performed on Waymo dataset which comprises the set of input sweeps 502 and ground-truth shapes 504.
  • FIG. 6 there is depicted experimental results comparing different 3D reconstruction methods including some 3D reconstructions methods contemplated in the context of the present technology.
  • a set of input sweeps 602 a set of ground-truth shapes 604 associated with the set of input sweeps 602, outputs 606 of a plurality of conventional methods, and outputs 608 in accordance with some implementations of the present technology.
  • experimentation has been performed on KITTI dataset which comprises the set of input sweeps 602 and ground-truth shapes 604.
  • the network 410 with a PCN-based architecture can be replaced with PointNet, PointNet++, Dynamic Graph Convolutional Neural Network (DGCNN) , or Frustum PointNets to perform global feature extraction.
  • the network 406 with a DeepSDF architecture can be replaced by other frameworks that extract features from an implicit shape representation, such as ONet, NeRF, and the like.
  • the network 414 may be configured to combine the outputs of the networks 410 and 414 and my perform concatenations, average pooling, and mapping.
  • the network 414 may be replaced a network with other operations that also merge/combine outputs of the network 410 and 414 and changes tensor dimensions, such as max pooling and bit-wise multiplication or addition operations, for example.
  • PointNet++ is a hierarchical architecture involving multiple stages of sampling, grouping, and transforming operations.
  • a point cloud is downsampled, and local features are extracted at each level. The features are then upsampled and combined with higher-level features to capture more detailed information.
  • This process allows PointNet++ to capture both local and global context in a scalable manner.
  • the PointNet++ architecture performs feature learning hierarchically, enabling it to capture fine-grained geometric patterns and achieve improved performance on tasks like semantic segmentation and object classification compared to the original PointNet.
  • DGCNN is a deep learning architecture designed for processing point cloud data and exploits inherent spatial relationships between points in a point cloud by constructing a dynamic graph representation.
  • the dynamic graph is built by connecting each point to its k nearest neighbors based on Euclidean distance. This graph structure captures the local geometric context of each point.
  • the DGCNN architecture can comprise several components. Local feature extraction can be performed where per-point features are computed by passing each point's position information through shared MLPs.
  • Edge convolution can be performed where a dynamic graph is used to perform graph convolutions on the local feature representations. This operation aggregates the information from neighboring points and captures the local geometric relationships.
  • DGCNN leverages the graph convolutional neural network (GCN) framework, which enables it to capture and process the local and global geometric features of point cloud data.
  • GCN graph convolutional neural network
  • Frustum PointNets are deep learning architectures designed for 3D object detection in point cloud data by leveraging both 2D image data and 3D point cloud information.
  • Frustum PointNets use 2D image-based object proposals (such as bounding boxes) generated by a 2D object detector as a starting point. These proposals are then transformed into 3D frustums (conical frustum shapes) in the point cloud data corresponding to the visible region in the image. The frustums serve as the regions of interest for further analysis.
  • the Frustum PointNets architecture can comprises of different components. Frustum Extraction can be performed where the 2D object proposals are projected into the 3D point cloud, and the corresponding points within the frustum are extracted.
  • PointNet is then applied to these frustums to learn local geometric features.
  • a PointNet can also be employed to extract per-point features within the frustum. This helps capture fine-grained details and local object characteristics.
  • Voting and object detection can be performed where a voting mechanism is utilized to estimate the 3D bounding box parameters for each object within the frustum. These estimated boxes are refined using a fully connected network, and a final classification and localization step is performed to detect and localize objects in 3D space.
  • the network architecture 400 can be tuned to work with point-based, mesh-based, or voxel-based input and output formats. It should also be noted that the network architecture 400 can also be used for other taxonomies in a variety of industries that benefit from 3D reconstruction. This can be achieved by using other application-specific datasets on which one or more networks of the network architecture 400 are trained.
  • the processor 110 is configured to execute a method 700 a scheme-block illustration of which is depicted in FIG. 7. It is contemplated that the method 700 can be executed by an electronic device implemented similarly to what has been described above with reference to FIG. 1. In some embodiments, one or more steps of the method 700 may be executed by more than one physical processors. For example, more than one physical processors may be communicatively coupled over a network for performing one or more steps in a distributed manner. It is therefore contemplated that one or more steps from the method 700 may be executed by distinct electronic devices, without departing from the scope of the present technology.
  • STEP 702 acquiring a first point cloud and a second point cloud, the first and second point clouds being multi-sweep point clouds representing at least the object
  • the method 700 begins with a processor acquiring a first point cloud and a second point cloud.
  • the first and second point clouds are multi-sweep point clouds representing an object.
  • multi-sweep point clouds refer to one or more point clouds obtained from multiple scanning or sensing passes over a same area and/or object.
  • a processor may be configured to acquire a multi-sweep dataset comprising point clouds each of which represent a separate scan or sweep taken at a different time and/or from a different perspective.
  • the point cloud data can be acquired from a LIDAR system, for example. By combining multiple sweeps or scans, the multi-sweep point clouds provide a detailed representation of the scanned area and/or object.
  • STEP 704 extracting, using a first network, explicit features about a shape of the object based on the first and second point clouds, the first network having been trained to explicitly model the shape of the object
  • the method 700 continues to step 704 with extracting, using a first network, explicit features about a shape of the object based on the first and second point clouds.
  • the first network has been trained to explicitly model the shape of the object.
  • “explicit” features may refer to features generated using one or more explicit modeling techniques.
  • Explicit modeling techniques comprise point-based, voxel-based, and mesh-based techniques and are configured to directly represent 3D object shape structure with points, voxels, and meshes, respectively.
  • a point-based 3D reconstruction model such as a PCN model, a GRNet model, and a PoinTr model, for example, can output a point cloud as a shape representation of the 3D reconstruction.
  • a voxel-based model can model volumes by subdividing the space into a 3D grid.
  • Mesh-based models can generate meshes using a fixed reference template from a given object class.
  • Conventional explicit reconstruction approaches, including point-based, voxel-based, and mesh-based models suffer from problems such as low resolution and high memory costs.
  • explicit features represent features of a first type and which are extracted from an explicitly modeled object as opposed to an implicitly modeled object.
  • explicit features may include one or more intermediate features of at least one of a PCN model, GRNet model, and PoinTr model, and the like.
  • the first network may be configured to extract explicit features from an explicitly modeled shape of an object in the input (e.g., point clouds, meshes, voxels, and images) .
  • a choice of machine learning model for implement the explicit information extractor 308 may be made based on inter alia a format of the input 302.
  • the method 700 may further comprise a step of generating a first modified point cloud and a second modified point cloud by applying an FPS function onto the first point cloud and the second point cloud, respectively. This may allow generating modified points clouds having a same number N of points.
  • the step 704 may be performed on the first modified and second modified point clouds, as opposed to the first and second point clouds.
  • STEP 706 extracting, using a second network, implicit features about the shape of the object based on the first and second point clouds, the second network having been trained to implicitly model the shape of the object
  • the method 700 continues to step 706 with extracting, using a second network, implicit features about the shape of the object based on the first and second point clouds.
  • the second network has been trained to implicitly model the shape of the object.
  • “implicit” features may refer to features generated using one or more implicit modeling techniques. Contrary to explicit modeling techniques, implicit modeling uses function-based decision boundaries to implicitly define surfaces for 3D representations. For example, DeepSDF model trained to map the 3D shape to a low- dimensional latent space and learn the projection from the latent space to an SDF that describes that 3D shape. Broadly speaking, SDF is an implicit function that specifies whether a querying point is inside, or outside, the surface of an object. The value of an SDF represents the distance of a querying point from the surface of the object and its sign, either positive or negative, indicates whether the querying point lies on the exterior or interior of the surface, respectively.
  • the conventional DeepSDF model mainly focuses on recovering 3D shapes from a single-view partial point cloud and fails to leverage the multi-sweep information of vehicles.
  • single-view techniques e.g., conventional implicit techniques
  • the conventional DeepSDF model only takes the given surface points of a partial point cloud into consideration when generating 3D shapes. This can lead to suboptimal reconstruction results in areas not captured in the partial point cloud, since the DeepSDF method will perform reconstruction arbitrarily based on prior knowledge learned during training.
  • implicit features represent features of a second type and which are extracted from an implicitly modeled object.
  • implicit features may comprise one or more features extracted using occupancy networks, NeuS models, NeAT models, and the like.
  • a given machine learning model for implementing the second network may be made based on inter alia a format of the input (e.g., point clouds, meshes, voxels, and images) .
  • STEP 708 generating, using a third network, a latent code by projecting the implicit features and the explicit features into a latent space
  • the method 700 continues to step 708 with generating, using a third network, a lateener code by projecting the implicit features of the explicit features into a latent space.
  • the third network is configured to combine (e.g., pool) explicit and implicit information and project instance features to a latent space.
  • the output of the third network may be in a form of a given vector, called a latent code.
  • the third network may be a neural network with at least one max pooling layer and at least two fully-connected layers.
  • the neural architecture used to implement the third neural network may be selected based on inter alia the format of the input (e.g., point clouds, meshes, voxels, and images) .
  • a combined neural network may comprise one or more of the first network, the second network, and the third network for performing one or more steps of the method 700.
  • the combined neural network may be a distributed network, without departing from the scope of the present technology.
  • STEP 710 generating a 3D shape representative of the object using the latent code
  • the method 700 continues to step 710 with generating a 3D shape representative of the object using the latent code.
  • a processor may use a pre-trained DeepSDF decoder to map the latent code to the SDF for the given object (e.g., vehicle) . It is further contemplated that a processor may be configured to execute a mesh extrication function to extract a 3D mesh at one or more resolutions representing the given object.
  • a predicted latent code may be converted to an SDF using a pre-trained DeepSDF decoder.
  • the iso-surface composed of querying points with zero SDF values represents the surface of the given object and the implicit surface can be rasterized to a 3D mesh using Marching Cubes, for example. Since the SDF is a continuous function, 3D mesh extraction can be supported at a variety of different resolutions.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

La divulgation concerne des procédés et des dispositifs électroniques pour effectuer une reconstruction 3D d'un objet. Le procédé consiste à acquérir un premier nuage de points et un second nuage de points, les premier et second nuages de points étant des nuages de points multi-balayage représentant au moins l'objet, à extraire, à l'aide d'un premier réseau, des caractéristiques explicites concernant une forme de l'objet sur la base des premier et second nuages de points, le premier réseau ayant été entraîné pour modéliser explicitement la forme de l'objet, à extraire, à l'aide d'un deuxième réseau, des caractéristiques implicites concernant la forme de l'objet sur la base des premier et second nuages de points, le second réseau ayant été entraîné pour modéliser implicitement la forme de l'objet, à générer, à l'aide d'un troisième réseau, un code latent par projection des caractéristiques implicites et des caractéristiques explicites dans un espace latent, et à générer une forme 3D représentative de l'objet à l'aide du code latent.
PCT/CN2023/101585 2023-06-21 2023-06-21 Procédés et processeurs pour une modélisation implicite avec des nuages de points multi-balayage pour une reconstruction de véhicule 3d dans une conduite autonome Ceased WO2024259614A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2023/101585 WO2024259614A1 (fr) 2023-06-21 2023-06-21 Procédés et processeurs pour une modélisation implicite avec des nuages de points multi-balayage pour une reconstruction de véhicule 3d dans une conduite autonome

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2023/101585 WO2024259614A1 (fr) 2023-06-21 2023-06-21 Procédés et processeurs pour une modélisation implicite avec des nuages de points multi-balayage pour une reconstruction de véhicule 3d dans une conduite autonome

Publications (1)

Publication Number Publication Date
WO2024259614A1 true WO2024259614A1 (fr) 2024-12-26

Family

ID=93934690

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/101585 Ceased WO2024259614A1 (fr) 2023-06-21 2023-06-21 Procédés et processeurs pour une modélisation implicite avec des nuages de points multi-balayage pour une reconstruction de véhicule 3d dans une conduite autonome

Country Status (1)

Country Link
WO (1) WO2024259614A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120070746A (zh) * 2025-01-21 2025-05-30 西安电子科技大学芜湖研究院 基于改进knn的dgcnn模型的建筑物点云重建方法
CN120381273A (zh) * 2025-06-27 2025-07-29 中南民族大学 基于稀疏动态图卷积神经网络的情绪状态识别方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111161364A (zh) * 2019-12-24 2020-05-15 东南大学 一种针对单视角深度图的实时形状补全和姿态估计方法
CN112184899A (zh) * 2020-11-06 2021-01-05 中山大学 一种基于符号距离函数的三维重建方法
CN115690324A (zh) * 2022-11-15 2023-02-03 广州中思人工智能科技有限公司 一种基于点云的神经辐射场重建优化方法及装置
EP4152274A1 (fr) * 2021-09-21 2023-03-22 Toyota Jidosha Kabushiki Kaisha Système et procédé permettant de prédire une probabilité d'occupation d'un point dans un environnement, et son procédé de formation
CN115840971A (zh) * 2021-06-01 2023-03-24 达索系统公司 用于3d形状优化的深度参数化

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111161364A (zh) * 2019-12-24 2020-05-15 东南大学 一种针对单视角深度图的实时形状补全和姿态估计方法
CN112184899A (zh) * 2020-11-06 2021-01-05 中山大学 一种基于符号距离函数的三维重建方法
CN115840971A (zh) * 2021-06-01 2023-03-24 达索系统公司 用于3d形状优化的深度参数化
EP4152274A1 (fr) * 2021-09-21 2023-03-22 Toyota Jidosha Kabushiki Kaisha Système et procédé permettant de prédire une probabilité d'occupation d'un point dans un environnement, et son procédé de formation
CN115690324A (zh) * 2022-11-15 2023-02-03 广州中思人工智能科技有限公司 一种基于点云的神经辐射场重建优化方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120070746A (zh) * 2025-01-21 2025-05-30 西安电子科技大学芜湖研究院 基于改进knn的dgcnn模型的建筑物点云重建方法
CN120381273A (zh) * 2025-06-27 2025-07-29 中南民族大学 基于稀疏动态图卷积神经网络的情绪状态识别方法及装置

Similar Documents

Publication Publication Date Title
US12586362B2 (en) Method and apparatus with multi-modal feature fusion
Rajapaksha et al. Deep learning-based depth estimation methods from monocular image and videos: A comprehensive survey
Huang et al. Autonomous driving with deep learning: A survey of state-of-art technologies
CN112602116B (zh) 使用视频数据映射对象实例
US12051261B2 (en) Semantic segmentation of 2D floor plans with a pixel-wise classifier
US11636234B2 (en) Generating 3D models representing buildings
Chen et al. 3d point cloud processing and learning for autonomous driving
Tesema et al. Point cloud completion: A survey
JP2025120177A (ja) 構造的制約及び物理的制約を伴う３ｄ自動ラベル付け
US11810311B2 (en) Two-stage depth estimation machine learning algorithm and spherical warping layer for equi-rectangular projection stereo matching
WO2015123122A1 (fr) Parachèvement de contour permettant d'augmenter des reconstructions de surface
WO2024259614A1 (fr) Procédés et processeurs pour une modélisation implicite avec des nuages de points multi-balayage pour une reconstruction de véhicule 3d dans une conduite autonome
CN118891656A (zh) 用于改进的对象姿态估计的语义slam框架
Ward et al. RGB-D image-based object detection: from traditional methods to deep learning techniques
US20250118009A1 (en) View synthesis for self-driving
CN115719476A (zh) 图像的处理方法、装置、电子设备和存储介质
CN116206273A (zh) 一种车道线检测方法、装置、设备及存储介质
EP3992909A1 (fr) Algorithme d'apprentissage automatique d'estimation de profondeur en deux étapes et couche de déformation sphérique pour correspondance stéréo avec projection rectangulaire équiforme
WO2024243725A1 (fr) Segmentation sémantique de nuages de points utilisant un regroupement adaptatif de caractéristiques basé sur la confiance et une augmentation de caractéristiques de voisinage 3d
Saeed et al. ASPPMVSNet: A high‐receptive‐field multiview stereo network for dense three‐dimensional reconstruction
CN120032339B (zh) 基于bev与全稀疏架构的自动驾驶场景的目标检测方法
Tan et al. 3D detection transformer: Set prediction of objects using point clouds
Zhang et al. High‐order multilayer attention fusion network for 3D object detection
US20250336186A1 (en) System and method with adaptive resolution for semantic occupancy
CN119600206B (zh) 一种基于gru和3dcnn的多视图三维重建方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23941918

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE