WO2017003424A1 - Assemblage tridimensionnel (3d) de métriques pour des données rvb-d - Google Patents

Assemblage tridimensionnel (3d) de métriques pour des données rvb-d Download PDF

Info

Publication number
WO2017003424A1
WO2017003424A1 PCT/US2015/038278 US2015038278W WO2017003424A1 WO 2017003424 A1 WO2017003424 A1 WO 2017003424A1 US 2015038278 W US2015038278 W US 2015038278W WO 2017003424 A1 WO2017003424 A1 WO 2017003424A1
Authority
WO
WIPO (PCT)
Prior art keywords
rgb
dataset
new
reconstructed scene
datasets
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2015/038278
Other languages
English (en)
Inventor
Johannes BOPP
Stefan Kluckner
Terrence Chen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens AG
Siemens Corp
Original Assignee
Siemens AG
Siemens Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens AG, Siemens Corp filed Critical Siemens AG
Priority to PCT/US2015/038278 priority Critical patent/WO2017003424A1/fr
Publication of WO2017003424A1 publication Critical patent/WO2017003424A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • the present disclosure relates generally to methods, systems, and apparatuses, for metric 3D stitching of RGB-D image data.
  • the disclosed technology may be applicable to, for example, the improved modeling of organs in medical applications, 4D modeling of physical objects, and support matching computer aided design (CAD) models to photo data.
  • CAD computer aided design
  • Kinect Fusion (KinFu) a recent concept proposed by Microsoft.
  • KinFu stitches a 3D scene by using a continuous data stream from a RGB-D sensor like the Microsoft Kinect to create a meshed 3D model by fusion of data within a volume.
  • KinFu requires a continuous data stream, otherwise the algorithm can lose track and won't be able to recover. Additionally, KinFu requires a volume like representation in order to handle the arriving point cloud data.
  • SfM may be implemented in an online or an offline manner. For an offline implementation, all data is acquired in advance of processing. Conversely, in an online implementation, a subset of the data is acquired in advance and the remainder is acquired during processing.
  • An offline implementation of SfM uses a set of redundant (overlapping) photos to create a 3D model using feature based camera pose estimation with bundle adjustment.
  • Embodiments of the present invention address and overcome one or more of the above shortcomings and drawbacks, by providing methods, systems, and apparatuses related to metric 3D stitching of RGB-D data. More specifically, the disclosed technology enables the automatic online 3D stitching of an ordered or unordered RGB-D sequence into a metric dense 3D model with interactive response times.
  • a computer-implemented method for metric 3D stitching of RGB-D image data includes an image processing computer determining an initial reconstructed scene representative of one or more physical objects based on a previously received RGB-D datasets.
  • the image processing computer receives a new RGB- D dataset comprising image data and depth information corresponding to the one or more physical objects and updates the initial reconstructed scene using the new RGB-D dataset, yielding an updated reconstructed scene.
  • the image processing computer determines 2D-3D correspondences between the new RGB-D dataset and the updated reconstructed scene, estimates a scaling factor for each 2D-3D correspondence based on depth information included in the new RGB-D dataset, and augments the updated reconstructed scene based on the scaling factor estimated for each 2D-3D correspondence, yielding a scaled metric reconstructed scene. Then, the image processing computer may create a 3D model of the one or more physical objects using the scaled metric reconstructed scene.
  • the initial reconstructed scene is updated by identifying a subset of the previously received RGB-D datasets comprising RGB-D datasets that are visually similar to the new RGB-D dataset.
  • the subset is identified by calculating a similarity score for each previously received RGB-D dataset (e.g., using an image indexing and search system) and ranking the previously received RGB-D datasets according to score. The highest ranked datasets may then be designated as the subset that is visually similar.
  • feature correspondences are determined by matching the new RGB-D dataset with each previously received RGB-D dataset included in the subset.
  • These feature correspondences are used to determine a relative pose of the new RGB-D dataset (e.g., using a Five Point Algorithm in combination with Random Sample Consensus (RANSAC)).
  • RANSAC Random Sample Consensus
  • new 3D points are determined using the feature correspondences and the relative pose of the new RGB-D dataset.
  • the initial reconstructed scene may then be updated using the new 3D points.
  • the aforementioned method may be applied to medical applications.
  • the one or more physical objects comprises portions of anatomy of an individual and acquisition of the new RGB-D dataset is triggered based the individual's heart rate and/or breathing rate.
  • the new RGB-D dataset may be acquired using a medical imaging device such as an endoscope.
  • the 3D model generated by the aforementioned method may be used to perform registration to preoperative 3D data acquired from an imaging scanner.
  • the aforementioned method may be used to develop 4D models.
  • the method is repeated a plurality of times to generate multiple 3D models.
  • Each of the 3D models is associated with a time value.
  • a 4D model of the physical objects may then be created based on the 3D models and the time value corresponds to each of the 3D models.
  • the modeling technique provided by the aforementioned method may also be applied to Product Lifecycle Management or Supply Chain Management domains.
  • a computer-aided design model related to the one or more physical objects may be identified using the 3D model.
  • an article of manufacture for metric 3D stitching of RGB-D image data comprises a non-transitory, tangible computer-readable medium holding computer-executable instructions for performing the aforementioned method.
  • the computer-readable medium may hold additional instructions for performing the various additional features of the aforementioned method described above.
  • a system for methods for metric 3D stitching of RGB-D image data includes a database, a camera, and a processor.
  • the database is configured to store previously received RGB-D datasets representative of one or more physical objects, and an initial reconstructed scene based on the previously received RGB-D datasets.
  • the camera is configured to acquire a new RGB-D dataset comprising image data and depth information corresponding to the one or more physical objects.
  • the camera is an endoscope.
  • the processor in the system is configured to update the initial reconstructed scene using the new RGB-D dataset, determine 2D-3D correspondences between the new RGB- D dataset and the updated reconstructed scene.
  • the processor is further configured to estimate a scaling factor for each 2D-3D correspondence based on depth information included in the new RGB-D dataset.
  • the processor augments the updated reconstructed scene based on the scaling factor estimated for each 2D-3D correspondence and creates a 3D model of the one or more physical objects using the scaled reconstructed scene.
  • FIG. 1 provides an illustration of a system for performing metric 3D stitching of
  • RGB-D data according to some embodiments.
  • FIG. 2 illustrates a process for creating an initial reconstructed scene using RGB-D data, according to some embodiments
  • FIG. 3 illustrates a process for expanding a reconstructed scene using a newly acquired RGB-D image, according to some embodiments.
  • FIG. 4 illustrates an exemplary computing environment within which embodiments of the invention may be implemented.
  • RGB-D data includes RGB (visual) images along with per-pixel depth information.
  • the disclosed technology enables the automatic 3D stitching of a RGB-D sequence into a metric dense 3D model with interactive response times. This technology does not rely on continuous data stream of image data and, thus, it does not risk losing track of feature placement during data collection. Since the depth information provides metric measurements, the stitching procedure gets augmented with the correct scene scale automatically.
  • FIG. 1 provides an illustration of a system 100 for performing metric 3D stitching of
  • RGB-D data according to some embodiments.
  • a Calibrated Image Source 105 provides a stream of RGB-D data representative of one or more physical objects to Image Processing Computer 1 10.
  • the RGB-D data includes both visual information (in RGB format), along with depth information related to the physical objects.
  • the Image Processing Computer 1 10 calculates a metric dense 3D model based on the received data. This model is then presented on Display 1 15.
  • the Calibrated Image Source 105 in the example of FIG. 1 can be any device generally known in the art capable of capturing RGB-D data. Calibration of the Calibrated Image Source 105 may be performed prior to imaging using any manual, semi-automatic, or automatic calibration technique known in the art and suitable to the type of sensor used as the Calibrated Image Source 105.
  • the Calibrated Image Source 105 is a stationary sensor device such as MicrosoftTM Kinect.
  • handheld devices such as smartphones and tablets equipped with RGB-D sensors may be used as the Calibrated Image Source 105.
  • the Calibrated Image Source 105 is a medical device such as a RGB-D endoscopic/laparoscopic device.
  • the type of Calibrated Image Source 105 may be selected based on the level of detail required for analysis. For example, the aforementioned mobile devices may be ill suited for surgical procedures where fine detail is required of anatomical features, thus a device such as an endoscope can be used.
  • data from different image sources may be combined to create the stream of RGB-D received at the Image Processing Computer 1 10. It should also be noted that the stream of images provided to the Image Processing Computer 1 10 by the
  • Calibrated Image Source 105 is not necessarily continuous. Thus, images may be acquired at different time intervals (e.g., triggered by an EKG and/or a breath trigger).
  • the Image Processing Computer 1 10 receives the RGB-D image data and creates a metric dense 3D model. More specifically, the Image Processing Computer 1 10 recovers the scene captured in the image data, along with its corresponding scale. Collectively, the scene and scale data is referred to herein as the model.
  • Component 1 1 OA performs feature extraction and relative pose estimation using the
  • RGB-D image data For example, as a new RGB-D image is received, a feature exaction technique is used to detect and describe local features in the image.
  • the feature extraction technique is scale-invariant feature transform (SIFT).
  • SIFT scale-invariant feature transform
  • a matching procedure obtains pairs of images to identify corresponding features in the image regions. These features may then be used by pose estimation algorithm such as the Five-Point Algorithm generally known in the art.
  • Component HOB performs localization of the RGB-D image data and structure expansion of the existing reconstructed scene.
  • the Image Processing Computer 1 10 maintains a database (not shown in FIG. 1) of all previously received RGB-D images and their
  • a group of previously received images are identified which are visually similar to the new RGB-D image. For example, in some embodiments, a similarity score is calculated for each previously received image (e.g., using an image indexing and similarity search system). The images are then ranked according to score and the highest ranked images are identified as the most similar. Next, the RGB-D image is matched pairwise against the image features of the images in the identified group to determine feature correspondences. Finally, the scene is expanded by triangulating new 3D points using the relative pose information. If the localization of an image fails, the user may be notified that a new image must be acquired. For example, in some embodiments, a visual indicator is presented on a display indicating that additional images are required.
  • 2D-3D image correspondences may be determined. Given a set of 2D-3D correspondences and Calibrated Image Source 105, the images may be localized, for example, by solving the absolute pose problem robustly in a Random Sample Consensus (RANSAC) loop. After each model update, the model is augmented with a scaling factor. Based on the depth cue provided in the new RGB-D image, a metric distance from the camera center to a related 3D point in the real world may be determined. Thus, Component HOB can estimate a scaling factor for each 2D-3D correspondence which can be integrated for all 2D-3D relations in a frame.
  • RANSAC Random Sample Consensus
  • bundle adjustment may be performed after the scaling step.
  • the bundle adjustment is performed in a separate processing thread to optimize execution time of the overall process. This bundle adjustment may be performed, for example, using a conventional SfM pipeline generally known in the art.
  • Component 1 IOC of the Image Processing Computer 1 10 performs 3D stitching of the data.
  • image stitching is the process of combining multiple RGB- D images with overlapping fields of view to produce a segmented 3D panorama or high- resolution 3D image.
  • Component 1 IOC aligns each conserved depth image (i.e., XYZ coordinates and RGB colors) within the common coordinates system by considering the estimated pose.
  • Display 1 15 is then presented on Display 1 15.
  • This Display 1 15 may be, for example, a computer monitor used during a surgical operation.
  • visual indicators related to the fidelity of the resultant metric dense 3D model may be presented along with (or as an alternative) the model itself.
  • a color system is used to alert the user that additional images are required for model generation. For example, an indicator may remain red until enough images have been acquired to provide sufficient coverage for the model.
  • the metric dense 3D model is also stored, either locally at the
  • Image Processing Computer 1 10 or on a remote computer-readable storage medium (not shown in FIG. 1).
  • the stored model may be updated as new images are processed.
  • the processing performed by the system 100 does not necessarily require a continuous stream of image data.
  • FIG. 2 illustrates a process 200 for creating an initial reconstructed scene using
  • RGB-D data a set of initial RGB-D images taken from different viewpoints is received.
  • step 210 feature extraction and matching occurs. During this step, distinctive key points are extracted from each RGB-D image frame and assigned feature descriptors.
  • Various feature descriptor extraction routines may be used at step 210 including, without limitation, Scale-Invariant Feature Transform (or SIFT).
  • the relative pose of the camera between the initial RGB-D images is determined using the feature descriptors.
  • the Five-Point Algorithm is for relative pose determination.
  • the Five-Point Algorithm is used in the case of a calibrated camera with at least five point correspondences. In the case that the image frames are not calibrated, more complicated algorithms may be employed (e.g., requiring additional points). Some of the matched features may be incorrectly matched. Thus, for robustness, feature matching may be performed using an iterative technique such as RANSAC to estimate parameters of the algorithm.
  • the initial reconstructed scene is determined using the set of initial RGB-D images and the relative camera poses determined at step 215.
  • Certain error metrics may be applied during the process 200. For example, a threshold can be placed on the distance between triangulated points. Additionally (or alternatively), the number of triangulated points in the initial reconstructed scene may be required to be greater than a predetermined number. If the initial reconstructed scene has fewer points, the user may be prompted to take additional images until the threshold number of points has been acquired.
  • FIG. 3 illustrates a process 300 for expanding a reconstructed scene using a newly acquired RGB-D image, according to some embodiments of the present invention.
  • This process 300 may be performed, for example, using the initial reconstructed scene produced via the process 200 outlined in FIG. 2. Additionally, after the initial reconstructed scene has been expanded, the process 300 may be continually used to further refine the scene.
  • a new RGB-D image is received by the image processing computer (see, e.g., FIG. 1) that is storing (or has access to) the reconstructed scene.
  • the image processing computer see, e.g., FIG. 1 that is storing (or has access to) the reconstructed scene.
  • 2D-3D image correspondences are estimated using the original features in the new RGB-D image and the triangulated 3D points in the reconstructed scene.
  • the set of 2D- 3D correspondences are used to localize the image, for example, by solving the absolute pose problem using a three-point algorithm in a random sample consensus (RANSAC) loop.
  • RANSAC random sample consensus
  • a report is presented to the user at step 320.
  • This report provides a visual indication that a new image should be acquired.
  • a color system is used where a visual indicator changes from one color (e.g., red) to another color (e.g., green) when a suitable RGB-D been acquired.
  • the metric scale of the model is estimated based on the 2D-3D correspondences between the new RGB-D image and the reconstructed scene.
  • each 2D-3D correspondence represents a ray in 3D space intersecting camera center, the 2D point on the image plane, and the reconstructed 3D point.
  • a metric distance from the 3D camera center to a related 3D point in the real world is obtained.
  • a scaling factor for each 2D-3D correspondence is estimated which can be integrated for all 2D-3D relations in the frame of the new RGB-D image.
  • the reconstructed scene is augmented with the scaling factor.
  • in the depth cue in the RGB-D depth data is only used as a supporting cue. Additionally, scale updates may only be considered before bundle adjustment when scaling reports consistent scaling (i.e., consideration of robust statistics over the 3D measurements of 2D-3D correspondences). Thus, stable optimization from 2D photo measurements may be utilized, while employing 3D information to improve real world capability (i.e., estimating the metric scale of final 3D model).
  • the techniques described herein may be applied to create a dense 3D model from unordered RGB-D data.
  • the sensing device gathering the RGB-D data is triggered by an EKG and/or a breath trigger to collect synchronized data. Therefore, the disclosed techniques may be extended time synchronized stitching.
  • frames are collected during medical procedures and include natural movements like breathing or cardiac motions.
  • the stitched models may be used to perform registration to preoperative 3D data collected by scanner modalities such as CT or MRI.
  • the disclosed techniques may also be used to support the generation of 4D models.
  • the models document progress over time, i.e., it helps to overlay 3D models of at different times to show the progress and compare it to the original CAD plans or segmented preoperative data.
  • the techniques described herein are applied to CAD modeling applications.
  • CAD CAD modeling applications.
  • FIG. 4 illustrates an exemplary computing environment 400 within which embodiments of the invention may be implemented.
  • this computing environment 400 may be used to implement the processes 200 and 300 as described in FIGS. 2 and 3, respectively.
  • the computing environment 400 may be used to implement one or more of the components illustrated in system 100 of FIG. 1.
  • the computing environment 400 may include computer system 410, which is one example of a computing system upon which embodiments of the invention may be implemented.
  • Computers and computing environments, such as computer system 410 and computing environment 400, are known to those of skill in the art and thus are described briefly here.
  • the computer system 410 may include a communication mechanism such as a bus 421 or other communication mechanism for communicating information within the computer system 410.
  • the computer system 410 further includes one or more processors 420 coupled with the bus 421 for processing the information.
  • the processors 420 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art.
  • the computer system 410 also includes a system memory 430 coupled to the bus 421 for storing information and instructions to be executed by processors 420.
  • the system memory 430 may include computer readable storage media in the form of volatile and/or nonvolatile memory, such as read only memory (ROM) 431 and/or random access memory (RAM) 432.
  • the system memory RAM 432 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM).
  • the system memory ROM 431 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM).
  • the system memory 430 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 420.
  • a basic input/output system (BIOS) 433 containing the basic routines that help to transfer information between elements within computer system 410 (e.g., during start-up) may be stored in ROM 431.
  • BIOS basic input/output system
  • RAM 432 may contain data and/or program modules that are immediately accessible to and/or presently being operated on by the processors 420.
  • System memory 430 may additionally include, for example, operating system 434, application programs 435, other program modules 436 and program data 437.
  • the computer system 410 also includes a disk controller 440 coupled to the bus 421 to control one or more storage devices for storing information and instructions, such as a hard disk 441 and a removable media drive 442 (e.g., floppy disk drive, compact disc drive, tape drive, and/or solid state drive).
  • the storage devices may be added to the computer system 410 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or Fire Wire.
  • SCSI small computer system interface
  • IDE integrated device electronics
  • USB Universal Serial Bus
  • the computer system 410 may also include a display controller 465 coupled to the bus 421 to control a display 466, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user.
  • the computer system includes an input interface 460 and one or more input devices, such as a keyboard 462 and a pointing device 461 , for interacting with a computer user and providing information to the processor 420.
  • the pointing device 461 for example, may be a mouse, a trackball, or a pointing stick for communicating direction information and command selections to the processor 420 and for controlling cursor movement on the display 466.
  • the display 466 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 461.
  • the computer system 410 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 420 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 430.
  • a memory such as the system memory 430.
  • Such instructions may be read into the system memory 430 from another computer readable medium, such as a hard disk 441 or a removable media drive 442.
  • the hard disk 441 may contain one or more datastores and data files used by embodiments of the present invention. Datastore contents and data files may be encrypted to improve security.
  • the processors 420 may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory 430.
  • hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.
  • the computer system 410 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein.
  • the term "computer readable medium” as used herein refers to any medium that participates in providing instructions to the processor 420 for execution.
  • a computer readable medium may take many forms including, but not limited to, non-volatile media, volatile media, and transmission media.
  • Non-limiting examples of non- volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as hard disk 441 or removable media drive 442.
  • Non-limiting examples of volatile media include dynamic memory, such as system memory 430.
  • Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the bus 421.
  • Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
  • the computing environment 400 may further include the computer system 410 operating in a networked environment using logical connections to one or more remote computers, such as remote computer 480.
  • Remote computer 480 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system 410.
  • computer system 410 may include modem 472 for establishing communications over a network 471 , such as the Internet. Modem 472 may be connected to bus 421 via user network interface 470, or via another appropriate mechanism.
  • Network 471 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a
  • LAN local area network
  • WAN wide area network
  • MAN metropolitan area network
  • MAN direct connection or series of connections
  • cellular telephone network or any other network or medium capable of facilitating communication between computer system 410 and other computers (e.g., remote computer 480).
  • the network 471 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-1 1 or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and
  • Bluetooth infrared
  • cellular networks satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 471.
  • the embodiments of the present disclosure may be implemented with any combination of hardware and software.
  • the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, computer-readable, non-transitory media.
  • the media has embodied therein, for instance, computer readable program code for providing and facilitating the mechanisms of the embodiments of the present disclosure.
  • the article of manufacture can be included as part of a computer system or sold separately.
  • An executable application comprises code or machine readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input.
  • An executable procedure is a segment of code or machine readable instruction, sub-routine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and/or parameters, performing operations on received input data and/or performing functions in response to received input parameters, and providing resulting output data and/or parameters.
  • a graphical user interface comprises one or more display images, generated by a display processor and enabling user interaction with a processor or other device and associated data acquisition and processing functions.
  • the GUI also includes an executable procedure or executable application.
  • the executable procedure or executable application conditions the display processor to generate signals representing the GUI display images. These signals are supplied to a display device which displays the image for viewing by the user.
  • the processor under control of an executable procedure or executable application, manipulates the GUI display images in response to signals received from the input devices. In this way, the user may interact with the display image using the input devices, enabling user interaction with the processor or other device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un procédé mis en œuvre par ordinateur pour un assemblage tridimensionnel (3D) de métriques de données d'image RVB-D dans un ordinateur de traitement d'image déterminant une scène reconstruite initiale représentative d'un ou plusieurs objets physiques sur la base d'une pluralité d'ensembles de données RVB-D reçus précédemment. L'ordinateur de traitement d'image reçoit un nouvel ensemble de données RVB-D comprenant des données d'image et des informations de profondeur correspondant au ou aux objets physiques et met à jour la scène reconstruite initiale à l'aide du nouvel ensemble de données RVB-D, produisant une scène reconstruite mise à jour. L'ordinateur de traitement d'image détermine une pluralité de correspondances bidimensionnelles (2D)-tridimensionnelles (3D) entre le nouvel ensemble de données RVB-D et la scène reconstruite mise à jour, estime un facteur de mise à l'échelle pour chaque correspondance 2D-3D sur la base d'informations de profondeur incluses dans le nouvel ensemble de données RVB-D, et augmente la scène reconstruite mise à jour sur la base du facteur de mise à l'échelle estimé pour chaque correspondance 2D-3D, produisant une scène reconstruite de métrique mise à l'échelle. Ensuite, l'ordinateur de traitement d'image peut créer un modèle 3D du ou des objets physiques à l'aide de la scène reconstruite de métrique mise à l'échelle.
PCT/US2015/038278 2015-06-29 2015-06-29 Assemblage tridimensionnel (3d) de métriques pour des données rvb-d Ceased WO2017003424A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/US2015/038278 WO2017003424A1 (fr) 2015-06-29 2015-06-29 Assemblage tridimensionnel (3d) de métriques pour des données rvb-d

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2015/038278 WO2017003424A1 (fr) 2015-06-29 2015-06-29 Assemblage tridimensionnel (3d) de métriques pour des données rvb-d

Publications (1)

Publication Number Publication Date
WO2017003424A1 true WO2017003424A1 (fr) 2017-01-05

Family

ID=53719959

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2015/038278 Ceased WO2017003424A1 (fr) 2015-06-29 2015-06-29 Assemblage tridimensionnel (3d) de métriques pour des données rvb-d

Country Status (1)

Country Link
WO (1) WO2017003424A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018046094A1 (fr) * 2016-09-09 2018-03-15 Siemens Aktiengesellschaft Procédé pour faire fonctionner un endoscope, endoscope et moniteur de paramètres vitaux
CN110349249A (zh) * 2019-06-26 2019-10-18 华中科技大学 基于rgb-d数据的实时稠密重建方法及系统
CN111667522A (zh) * 2020-06-04 2020-09-15 上海眼控科技股份有限公司 三维激光点云密集化方法及设备
CN116433493A (zh) * 2023-06-07 2023-07-14 湖南大学 一种基于度量学习的工件点云集合拼接方法

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
"Field and Service Robotics", vol. 79, 1 January 2014, SPRINGER BERLIN HEIDELBERG, Berlin, Heidelberg, ISSN: 1610-7438, article PETER HENRY ET AL: "RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments", pages: 477 - 491, XP055192425 *
D. PAGLIARI ET AL: "Kinect Fusion improvement using depth camera calibration", ISPRS - INTERNATIONAL ARCHIVES OF THE PHOTOGRAMMETRY, REMOTE SENSING AND SPATIAL INFORMATION SCIENCES, vol. XL-5, 6 June 2014 (2014-06-06), pages 479 - 485, XP055254962 *
NEWCOMBE R A ET AL: "KinectFusion: Real-time dense surface mapping and tracking", MIXED AND AUGMENTED REALITY (ISMAR), 2011 10TH IEEE INTERNATIONAL SYMPOSIUM ON, IEEE, 26 October 2011 (2011-10-26), pages 127 - 136, XP032114483 *
NIKOLAS ENGELHARD ET AL: "Real-time 3D visual SLAM with a hand-held RGB-D camera", 1 April 2011 (2011-04-01), XP055140023, Retrieved from the Internet <URL:http://vision.informatik.tu-muenchen.de/_media/spezial/bib/engelhard11euron.pdf> [retrieved on 20140912] *
SHAHRAM IZADI ET AL: "KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera", 16 October 2011 (2011-10-16), XP055246992, Retrieved from the Internet <URL:http://research.microsoft.com/pubs/155416/kinectfusion-uist-comp.pdf> [retrieved on 20160202] *
THOMAS WHELAN ET AL: "Kintinuous: Spatially Extended KinectFusion", RSS WORKSHOP ON RGB-D: ADVANCED REASONING WITH DEPTH CAMERAS, 19 July 2012 (2012-07-19), XP055140839, Retrieved from the Internet <URL:http://www.cs.nuim.ie/research/vision/data/rgbd2012/Whelan12rssw.pdf> [retrieved on 20140917] *
ZHENGYOU ZHANG: "Microsoft Kinect Sensor and Its Effect", IEEE MULTIMEDIA, IEEE SERVICE CENTER, NEW YORK, NY, US, vol. 19, no. 2, 1 February 2012 (2012-02-01), pages 4 - 10, XP011442655 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018046094A1 (fr) * 2016-09-09 2018-03-15 Siemens Aktiengesellschaft Procédé pour faire fonctionner un endoscope, endoscope et moniteur de paramètres vitaux
CN110349249A (zh) * 2019-06-26 2019-10-18 华中科技大学 基于rgb-d数据的实时稠密重建方法及系统
CN111667522A (zh) * 2020-06-04 2020-09-15 上海眼控科技股份有限公司 三维激光点云密集化方法及设备
CN116433493A (zh) * 2023-06-07 2023-07-14 湖南大学 一种基于度量学习的工件点云集合拼接方法
CN116433493B (zh) * 2023-06-07 2023-09-22 湖南大学 一种基于度量学习的工件点云集合拼接方法

Similar Documents

Publication Publication Date Title
US11900541B2 (en) Method and system of depth determination with closed form solution in model fusion for laparoscopic surgical guidance
US9665936B2 (en) Systems and methods for see-through views of patients
Zheng et al. Hybridfusion: Real-time performance capture using a single depth sensor and sparse imus
CN107430686B (zh) 用于移动设备定位的区域描述文件的众包创建和更新
US8712016B2 (en) Three-dimensional shape data processing apparatus and three-dimensional shape data processing method
CN104000655B (zh) 用于腹腔镜外科手术的组合的表面重构和配准
JP6850399B2 (ja) 単眼画像の深度回復方法及び装置、コンピュータ機器
WO2020206903A1 (fr) Procédé et dispositif de mise en correspondance d&#39;images et support de mémoire lisible par ordinateur
US20170193693A1 (en) Systems and methods for generating time discrete 3d scenes
US10078906B2 (en) Device and method for image registration, and non-transitory recording medium
JP6376873B2 (ja) 画像処理装置、画像処理方法及びプログラム
JP2020524355A5 (fr)
CN107025661B (zh) 一种实现增强现实的方法、服务器、终端及系统
CN111273772A (zh) 基于slam测绘方法的增强现实交互方法、装置
CN107665507A (zh) 基于平面检测实现增强现实的方法及装置
WO2017003424A1 (fr) Assemblage tridimensionnel (3d) de métriques pour des données rvb-d
Guzov et al. Visually plausible human-object interaction capture from wearable sensors
JP6228239B2 (ja) プリミティブの組を用いてデータをレジストレーションする方法
WO2020194663A1 (fr) Dispositif de suivi, modèle pré-retenu, système d&#39;endoscope et procédé de suivi
Hsieh et al. Markerless Augmented Reality via Stereo Video See‐Through Head‐Mounted Display Device
CN116403243A (zh) 一种多视角多人3d姿态估计方法、装置及设备
CN116309819A (zh) 位姿估计方法、系统、装置及存储介质
US20260026893A1 (en) Registration for spine navigation
US12456267B2 (en) Automatic registration of landmarks for augmented reality assisted surgery
US10021364B2 (en) Method of building stereoscopic model with kalman filtering

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15741683

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15741683

Country of ref document: EP

Kind code of ref document: A1