WO2020168094A1 - Mesure simultanée de profil de profondeur et de spectre - Google Patents

Mesure simultanée de profil de profondeur et de spectre Download PDF

Info

Publication number
WO2020168094A1
WO2020168094A1 PCT/US2020/018133 US2020018133W WO2020168094A1 WO 2020168094 A1 WO2020168094 A1 WO 2020168094A1 US 2020018133 W US2020018133 W US 2020018133W WO 2020168094 A1 WO2020168094 A1 WO 2020168094A1
Authority
WO
WIPO (PCT)
Prior art keywords
spectral
sequence
frames
images
multispectral
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2020/018133
Other languages
English (en)
Inventor
Brian Andrew SPARANO
Goldie Lynne GOLDSTEIN
Juan Carlos IVALDI
Daniel Gene Smith
Abdelmoula HABOUB
Stephen Paul RENWICK
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nikon Corp
Original Assignee
Nikon Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nikon Corp filed Critical Nikon Corp
Publication of WO2020168094A1 publication Critical patent/WO2020168094A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10141—Special mode during image acquisition
    • G06T2207/10152—Varying illumination
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection

Definitions

  • FIG. 1 is a block diagram of an embodiment of a 3D spectral scanning system for simultaneous depth profile and spectral measurement.
  • FIG. 2 is a flowchart of an embodiment of a method for 3D spectral scanning system for simultaneous depth profile and spectral measurement that can be performed by the exemplary system depicted in FIG. 1.
  • FIG. 3 shows a sequence of multi-shot binary coded structured patterns projected on an object or target of interest.
  • FIG. 4 shows a sequence of multi-shot gray code structured patterns projected on an object or target of interest.
  • FIG. 5 is a flowchart of an embodiment of a method of performing spectral segmentation using the images to identify material differences in the object.
  • FIG. 6 is a flowchart of another embodiment of a method of performing spectral segmentation using the images to identify material differences in the object.
  • FIG. 7 depicts a processing chain for 3D spectral scanning data acquisition and processing flow as described herein.
  • FIG. 8 shows the representation of four induced endmembers extracted by PPI.
  • FIG. 9 is a flowchart of an embodiment of a method of performing spectral segmentation that includes generating a lull scan data cube.
  • FIG. 10A is a flowchart of an embodiment of a method of performing spectral segmentation that includes matching spectrally distinct signatures or endmembers to a particular reference signature associated with a particular material.
  • FIG. 10B is a flowchart of an embodiment of a method of performing spectral segmentation for generating a 3D reconstruction or 3D point cloud based on or in response to a target spectral signature associated with a material of interest.
  • FIG. IOC is a flowchart of an embodiment of a method of performing spectral segmentation using a segmentation algorithm.
  • FIG. 11 is a diagram illustrating an embodiment of a measurement device.
  • FIG. 12 is a diagram illustrating an embodiment of a measurement device.
  • FIG. 13 is a diagram illustrating an embodiment of a measurement device.
  • FIG. 14 is a diagram illustrating an embodiment of a measurement device.
  • FIG. 15 is a diagram illustrating an embodiment of a measurement device.
  • FIG. 16 is a diagram illustrating an embodiment of a measurement system.
  • FIG. 17 is a flow diagram illustrating an embodiment of a process for making a simultaneous depth profile and spectral measurement.
  • FIG. 18 is a flow diagram illustrating an embodiment of a process for determining a depth calibration.
  • FIG. 19 is a flow diagram illustrating an embodiment of a process for determining a depth measurement.
  • FIG. 20 is a flow diagram illustrating an embodiment of a process for determining a spectral calibration.
  • FIG. 21 is a flow diagram illustrating an embodiment of a process for determining a spectral measurement.
  • FIG. 22 is a diagram illustrating an embodiment of data from a device measuring height and a spectrum of a target simultaneously.
  • FIG. 23 is a diagram illustrating an embodiment of data from a device measuring height and a spectrum of a target simultaneously.
  • FIG. 24 is a diagram illustrating an embodiment of a twyman green structured light system using a broadband source.
  • FIG. 25 is a diagram illustrating an embodiment of a twyman green structured light system using a series of laser diode sources.
  • FIG. 26 depicts an example of finding the mean background irradiance by calculating the mean of the two images.
  • FIG. 27 depicts a synthetic wavelength example.
  • FIG. 28 depicts equivalent synthetic wavelengths.
  • FIG. 29 depicts a diagram of a structured light system using laser sources and a polarization grating.
  • FIG. 30 depicts a diagram of a structured light system using fiber coupled LEDs and polarization grating.
  • FIG. 31 shows a table listing examples of various industrial applications for 3D spectral scanning as disclosed herein.
  • FIG. 32 is a block diagram of an embodiment of a material processing system or platform 3200 that uses 3D spectral scanning to execute material processing steps.
  • FIG. 33 is a flowchart of an embodiment of a method for material processing using
  • 3D spectral scanning that can be performed by the exemplary system depicted in FIG. 32.
  • FIG. 34 illustrates an application of 3D spectral scanning to provide 3D segmentation by material for automated optical inspection.
  • FIG. 35 shows the difference between traditional 3D scanning and 3D spectral scanning as disclosed herein.
  • the invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and or provided by a memory coupled to the processor.
  • these implementations, or any other form that the invention may take, may be referred to as techniques.
  • processor refers to one or more devices, circuits, and or processing cores configured to process data, such as computer program instructions.
  • the machine vision market is evolving rapidly with increasing demands for speed, complex identification, inspection, and sorting.
  • the drivers come from a need for automation of industrial processes.
  • the performance of existing machine vision systems can be enhanced by extracting more information available in images taken of objects in industrial processes.
  • spectral data and 3D shape information can be leveraged through advanced data analysis (such as support vector machine analysis, K-means clustering, and neural networks) to enable decisions.
  • chemometric algorithms can be applied to perform true unknown analysis to identify materials from large stock libraries.
  • the rendering of its components as a 3D point cloud segmented by material can be used to inform detection and diagnosis of issues to automate material processing steps or actions in industrial manufacturing or other applications (e.g., additive manufacturing applications, removal processing).
  • techniques are described for reverse engineering components in an assembly by providing an integrated 3D spectral scanning system configured to provide 3D surface data and to distinguish sub-component parts in an assembly by their material spectral signatures.
  • an integrated inspection technique is disclosed that can perform both 3D shape and material identification simultaneously such that spatial and spectral data are automatically co-registered and interpreted as an aggregate data set.
  • an automated system for parts inspection and reverse engineering configured to distinguish and label components made of different materials within an assembly.
  • the disclosed system comprises a 3D imaging system, a spectral imaging camera, and engineering software configured to represent an object in a computer (e.g., using CAD).
  • the 3D imaging system comprises a structured light (SL) camera for extracting a detailed 3D geometric mesh of a part under inspection.
  • the spectral imaging camera is configured to provide images of the object at different wavelengths, thereby capturing spectral signatures that enable detection of different materials in the part under inspection.
  • the software is configured to merge the 3D surface and material differences to output a computer representation of the part showing spatial dimensions and different components based on a material distinction.
  • a 3D spectral scanning or imaging technique can discriminate material composition and 3D object shape in industrial machine vision applications.
  • a 3D spectral scanning computer system for simultaneous depth profile and spectral measurement comprises a processor and a memory coupled to the processor configured to provide the processor with instructions that when executed cause the processor to: receive multispectral or hyperspectral images of an object from which 3D data can be extracted, perform spectral segmentation using the images to identify material differences in the object; perform 3D reconstruction using the images to identify a depth profile based on the object; and output a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.
  • the disclosed techniques for 3D spectral scanning using structured illumination are not limited to any particular method of performing either spectral segmentation or 3D reconstruction.
  • an important element is that the images of the object or target of interest used as a basis for spectral segmentation and for 3D reconstruction are acquired or captured in a manner so as to include both spectral data and 3D data or data from which a depth profile can be extracted.
  • spectral segmentation and 3D reconstruction are performed using the images to provide a 3D point cloud segmented by material.
  • 3D reconstruction is performed based on or in response to spectral segmentation.
  • spectral segmentation is performed based on or in response to 3D reconstruction.
  • the images are acquired or captured such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images.
  • Performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images. Extracting a 3D point cloud or 3D geometric mesh of the object from the images is in some cases based on the spectral segmentation and identified material differences.
  • the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene.
  • a 3D spectral scanning system for simultaneous depth profile and spectral measurement comprises a structured light source and a spectral imaging device configured to acquire or capture images of an object illuminated by the structured light source at different frequencies.
  • the system fiirther comprises a processor and a memory coupled to the processor configured to provide the processor with instructions that when executed cause the processor to: perform spectral segmentation using the images to identify material differences in the object; perform 3D reconstruction using the images to identify a depth profile based on the object; and output a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.
  • One area for the application of 3D spectral scanning is for material processing applications such as additive manufacturing and removal processing.
  • material processing steps may be controlled or initiated based on or in response to the 3D shape and material composition data output by a 3D scanning system.
  • a material processing system comprises obtaining 3D shape and material composition data output by 3D scanning an object using the 3D spectral scanning techniques as described herein.
  • a method for material processing using 3D spectral scanning comprises performing 3D spectral scanning on an object.
  • the 3D spectral scanning may be performed according to any of the disclosed techniques as described herein.
  • the method also includes performing detection and diagnosis to identify issues associated with the object based on the 3D spectral scanning; generating a decision in response to the detection and diagnosis;
  • a material processing system or platform is configured to obtain both the 3D shape and the material of an object (such as a part being manufactured) using the 3D spectral scanning techniques disclosed herein, detect and diagnose issues associated with the object (e.g., recognize a contaminated, deteriorated, degraded, or otherwise substandard or abnormal portion of the object, provide a recommended action to address the issues, and execute the action (e.g., clean up contaminated portions or overlay or replace deteriorated or degraded portions while leaving normal portions intact).
  • FIG. 1 is a block diagram of an embodiment of a 3D spectral scanning system for simultaneous depth profile and spectral measurement.
  • the system 100 can be configured to provide 3D data and to distinguish parts in an object by their material spectral signatures.
  • the system 100 comprises a structured light source (e.g., Structured Light Source 110) and a spectral imaging device (e.g., Spectral Imaging Device 120) configured to acquire or capture images of an object (e.g., Object 101).
  • a structured light source e.g., Structured Light Source 110
  • a spectral imaging device e.g., Spectral Imaging Device 120
  • the structured light source includes a projector.
  • the structured light source can also include other structured light systems and sources, some of which are discussed in more detail below with respect to various embodiments.
  • the structured light source includes a broadband source, multiple individual sources, laser-line sources, a series of laser diode sources that can be coupled using a grating or a combination of fiber-coupled LEDs.
  • the structured light source comprises a multi-spectral fringe projection system. The fringes can be generated interferometrically, creating different fringe frequencies per illumination wavelength.
  • the fringes are generated using laser sources and a polarization grating.
  • the fringes are generated by interfering the +1 and -1 orders of a polarization grating which are orthogonally circularly polarized.
  • Imaging is performed using a spectral imaging device.
  • an image is obtained or acquired such that the image captures spectral and 3D data of an object, target of interest, or a scene comprising an object or target of interest.
  • the spectral imaging device can be a multispectral or a hyperspectral camera.
  • the spectral imaging device comprises a detector.
  • a detector can take many forms (e.g., 2-dimensional imaging device, multi wavelength detector, snapshot mosaic sensor, spectrum detector, multi-wavelength imaging spectrometer, pushbroom spectrometer, etc.).
  • the spectral imaging device can also include an imaging spectrometer such as an IRIS imaging spectrometer, a multiple-image Fourier transform spectrometer, or a compact birefringent Fourier transform spectrometer.
  • imaging is performed with a multi-wavelength imaging spectrometer through multiple phase shifts achieved by the rotation of a polarizer.
  • a structured light camera (e.g, shown at 102 of FIG. 1) comprises a structured light source and a spectral imaging device.
  • a structured light camera comprises or provides a structured light source and the 3D spectral scanning system comprises the structured light camera and a spectral imaging device.
  • the structured light source is configured to project a sequence of structured light patterns on a scene comprising an object.
  • the sequence of structured light patterns projected by the structured light source is configured to provide 3D data of the scene used to perform 3D reconstruction from images acquired or captured by the spectral imaging device.
  • the object is illuminated by the structured light source at different frequencies such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images.
  • the spectral imaging device can be a multispectral or hyperspectral camera.
  • 3D spectral scanning system 100 includes a computer system 130 made up of various subsystems including least one microprocessor subsystem also referred to as a central processing unit, or CPU (not shown).
  • the CPU can be implemented by a single-chip processor or by multiple processors.
  • the CPU is a general purpose digital processor that controls the operation of the computer system 130. Using instructions retrieved from a memory (not shown) coupled to the processor, the CPU controls the reception and manipulation of input data (e.g., spectral images received from Spectral Imaging Camera 120), and the output and display of data on output devices.
  • the memory is configured to provide the processor with instructions that when executed cause the processor to: perform spectral segmentation using the images to identify material differences in the object; perform 3D reconstruction using the images to identify a depth profile based on the object; and output a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.
  • performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images and a computer representation of the object showing spatial dimensions and material differences based on the spectral segmentation and the 3D point cloud or 3D geometric mesh is output by the system.
  • extracting a 3D point cloud or 3D geometric mesh of the object from the images is based on the spectral segmentation and identified material differences.
  • Spectral Segmentation Module 140 is configured to perform spectral segmentation using the images to identify material differences in the object. For example, spectral segmentation can be performed based on spectral imaging data of an object (e.g., spectral images received from Spectral Imaging Device 120). Spectral imaging data includes multispectral or hyperspectral images received from a multispectral camera or hyperspectral camera respectively.
  • 3D Reconstruction Module 150 is configured to perform 3D reconstruction based on or using the images acquired or captured by the spectral imaging device to identify a depth profile based on the object.
  • 3D Reconstruction Module 150 is configured to perform 3D reconstruction including by extracting a 3D point cloud or 3D geometric mesh of the object from the images.
  • extracting a 3D point cloud or 3D geometric mesh of the object from the images is based on the spectral segmentation and identified material differences.
  • computer system 130 outputs a computer representation of the object (e.g.,
  • Output 160 showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.
  • performing spectral segmentation includes obtaining spectral signatures from multispectral or hyperspectral images to identify material differences in the object.
  • performing 3D reconstruction includes extracting a 3D point cloud or 3D geometric mesh of the object from the images
  • the system is configured to output a computer representation of the object (e.g., Output 160) showing spatial dimensions and material differences based on the spectral signatures and the 3D point cloud or 3D geometric mesh.
  • FIG. 2 is a flowchart of an embodiment of a method 200 for 3D spectral scanning for simultaneous depth profile and spectral measurement that can be performed by the exemplary system 100 depicted in FIG. 1.
  • the method 200 includes illuminating an object with a structured light source at 210 and capturing images of the object illuminated by the structured light source at different frequencies at 220.
  • a structured light source e.g., Structured Light Source 110 of FIG. 1 illuminates the object at different frequencies.
  • the object is illuminated such that a 3D point cloud or 3D geometric mesh of the object can be extracted from images taken of the object.
  • the images are acquired or captured using a spectral imaging device (e.g., Spectral Imaging Device 120 of FIG. 1).
  • the spectral imaging device can be a multispectral or hyperspectral camera.
  • spectral segmentation is performed using the images to identify material differences in the object.
  • Spectral Segmentation Module 140 can perform spectral segmentation using the images to identify material differences in the object.
  • spectral segmentation can be performed by obtaining spectral signatures from the images based on spectral imaging data of an object (e.g., spectral images received from Spectral Imaging Device 120).
  • Spectral imaging data includes multispectral or hyperspectral images received from a multispectral camera or hyperspectral camera respectively.
  • 3D reconstruction is performed using the images to identify a depth profile based on the object.
  • the images are acquired or captured such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images.
  • 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images.
  • extracting a 3D point cloud or 3D geometric mesh of the object from the images is based on the spectral segmentation and identified material differences.
  • a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction is output by the system.
  • outputting a computer representation of the object showing spatial dimensions and material differences is based on spectral segmentation and the 3D point cloud or 3D geometric mesh.
  • An essential part of structured light imaging is the calibration of both the spectral imaging device (e.g., a multispectral or hyperspectral camera) and the structured light source (e.g., a projector) to ensure the two elements are properly configured to acquire or capture images of an object or target of interest.
  • the camera and projector calibration procedures establish the relationship between a pixel on a 2D image (in camera coordinates) and a straight line in 3D space (world coordinates) along with the object location by taking the lens distortion into consideration. Calibration procedures are performed before the acquisition of data (e.g., capturing images of an object or target of interest).
  • Camera calibration a key aspect in 3D imaging technology as the 3D image reconstruction depends on the camera calibration approach.
  • the aim of the calibration of both the camera and projector is to obtain their intrinsic and extrinsic parameters including parameters of lens distortion and parameters of rigid transformations from camera and projector coordinates to world coordinates.
  • the intrinsic parameters include the focal length, principle point, pixel skew factor or skew coefficient, distortion coefficients, and pixel size.
  • the extrinsic parameters include the rotation and translation from a world coordinate system to a camera or projector coordinate system.
  • a camera calibration procedure is performed using a set of images taken at different angles and distances of a known calibration object such as a planar checkboard (calibration image) that has distinctive comers that are easy to detect.
  • the comers of each square are detected and extracted in each perspective image scene of the checkboard. All the intrinsic and extrinsic parameters of the camera are determined and used in the 3D reconstruction of scanned objects.
  • the next step is to calibrate the projector and refine the camera calibration against any imperfect planarity of the checkerboard that is used in this calibration.
  • the calibration of a projector can be performed in a similar manner as for the camera by projecting a pre-calibrated checkboard pattern and extracting the square comers. The projected points are evaluated to match those of the camera calibration images. To find the intrinsic and extrinsic parameters of the projector, as the spatial relationship between the camera and the planar checkboard is already known, only the correspondence between projector and camera needs to be established.
  • a Single-Shot Spatial Structured Illumination (SI) Light method is performed to calibrate the projector.
  • This method requires one shot per calibration pose and adopts spatial multiplexed patterns, in which the feature point correspondences are uniquely encoded in a single structured light (SL) pattern.
  • An advantage of this method is that it refines imperfectly calibrated camera and projector parameters due to imperfect planar calibration board using a bundle adjustment method. To proceed with this calibration, after completing the selection of four extreme comers in each image, all the comers of all squares in the projected image are selected and extracted. The intersection is computed between the rays from the projector toward the camera and used to establish the matching between the camera and projector.
  • the calibration process described herein for the camera and structured light source or structured light imaging system is an example of how these elements may be calibrated.
  • Other calibration methods and systems including various algorithms and techniques for determination of the intrinsic and extrinsic parameters of structured light imaging system can be applied without limiting the scope of the disclosed 3D spectral scanning techniques.
  • the system can be used for data acquisition to acquire or capture images of the scene comprising the object or target of interest.
  • structured light patterns can be projected on the object or target to enable 3D geometric shape reconstruction from the images.
  • STRUCTURED LIGHT FOR 3D SHAPE MEASUREMENT is a non- contact, high-resolution optical method to extract the 3D profiles of objects.
  • the method is based on camera imaging of an object while illuminated by a projected light pattern, such as a grid of bright and dark lines. When observed at an angle, the projected patterns are deformed by the object’s surface shape. Surface height information is extracted directly from the analysis of the deformed grid pattern recorded in the image.
  • the structured light source comprises a structured light illumination system.
  • a structured illumination system for projecting a sequence of structured light (SL) patterns from a structured light source on a scene comprising an object or target of interest includes a 2D array detector, a light projector, and a processing unit or processor for projecting and acquiring SL patterned scenes.
  • the projector can use preloaded SL patterns or user defined patterns that can be generated and loaded into the projector.
  • a camera is used to acquire or capture a sequence of 2D images of the object under the structured light illumination.
  • the projected SL patterns are distorted by the object’s surface features and therefore the 3D information of scanned objects can inferred by correcting the distortion.
  • Depth extraction can be performed in a number of different ways depending on the nature of the projected patterns.
  • a single shot one projected pattern and one image
  • a multi-shot approach can be used in which a sequence of multiple, distinct patterns can be projected and the associated images collected. Multi-shot methods can improve the information content and resolution of the 3D profile produced.
  • the projected pattern can be modulated in numerous ways. For example, simple binary patterns of alternating bright and dark stripes or even a projected laser line can be used. The latter is an example of one-dimensional pattern. Alternatively, patterns can be modulated in two dimensions, as is the case with a binary checkerboard pattern. More complex two-dimensional patterns can be formed such as arrays of segmented stripes, dot arrays, pseudo-random 2D arrays, and random speckle patterns. Another variation in SLP methodology is the use of grayscale indexing or the introduction of color coding to the projected pattern. The modulation depth of projected patterns can be binary (light or dark) or it may have a gradual grading throughout the pattern.
  • the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source (e.g., Structured Light Source 110 of FIG. 1) on a scene comprising the object.
  • the sequence of structured light patterns is configured to provide 3D data of the scene. Multiple frames of data are acquired in order to complete the full depth/spectrum measurement of the scene.
  • multi-shot sequential binary coded or gray coded patterns can be used in the 3D image reconstruction of an object or scanned target.
  • a multi-shot sequential binary coded or gray code structured light illumination 3D imaging technique is used. These imaging techniques are based on the use of structured light with special designed 2D intensity patterns that allow the estimation of depth maps of objects by measuring the deformation in the projected SL patterns compared to SL reference patterns.
  • FIG. 3 shows an example of a sequence 300 of structured light patterns configured to provide 3D data of an object in a scene.
  • FIG. 3 shows a sequence of multi-shot binary coded structured patterns 300 projected on an object or target of interest.
  • the sequence 300 consists of black and white stripes that form a sequence of projection patterns such that each point on the surface of the object or target possesses a unique binary code that differs from any other code for a different point on the surface.
  • These binary SL codifications encode a scene by using just two intensity values: all black (intensity 0) and all white (intensity 1).
  • the 3D coordinates (x, y, z) of the scanned object or target can be computed along each horizontal and vertical line, thus reconstructing the 3D geometric shape of the scanned object or target.
  • these pixel entities can be converted into a world coordinate system and then 3D coordinates (x, y, z) of the scanned objects can be recovered via triangulation.
  • FIG. 4 shows another example of a sequence 400 of structured light patterns configured to provide 3D data of an object in a scene.
  • FIG. 4 shows a sequence of multi-shot gray code structured patterns 400 projected on an object or target of interest.
  • the multi shot sequential gray code structure illumination (SI) light can be advantageous in that it tends to be more robust with respect to noise and varying surface characteristics as compared to binary encoding.
  • the gray code designed 2D intensity patterns allow the estimation of the depth maps of objects by measuring the deformation in the projected SI light patterns compared to the SI light reference patterns. As shown in FIG.
  • the sequence 400 consists of black and white stripes that form a sequence of projection patterns such that each point on the surface of the object possesses a unique gray code that differs from any other codes of different points.
  • the 3D coordinates (x, y, z) of the scanned target can be computed along each horizontal and vertical line via triangulation, thus reconstructing the 3D geometric shape of the scanned target.
  • the system performs spectral segmentation (e.g., as shown at step 230 of FIG. 2) using the images to identify material differences in the object. Exemplary methods for performing spectral segmentation are discussed below with respect to FIGS. 5-10.
  • FIG. 5 is a flowchart of an embodiment of a method 500 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) using the images to identify material differences in the object.
  • the method 500 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 510 and spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember at 520.
  • the acquired or captured images comprise both the structured illumination (SI) and spectral data (e.g., multispectral or hyperspectral data).
  • the acquired or captured images are stored as hypercubes.
  • Each hypercube is a three dimensional matrix that contains spectral information in the third dimension organized by x, y pixel position in the first and second dimensions respectively.
  • an RGB image can be thought of as a data cube with R, G and B values for each x, y pixel position
  • a spectral image can be thought of as a hypercube with raw spectral data (for different wavelengths) for each x, y pixel position.
  • the different wavelengths comprise more than RGB wavelengths (e.g., multispectral and hyperspectral wavelengths).
  • FIG. 6 is a flowchart of an embodiment of a method 600 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) using the images to identify material differences in the object.
  • the method 600 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 610.
  • the method includes extracting one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes.
  • the method includes spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to an extracted spectrally distinct signature or endmember.
  • the spectral data (e.g., multispectral or hyperspectral data) is processed by (e.g., by Spectral Segmentation Module 140 of FIG. 1) by extracting spectral signatures of materials and their associated abundance maps.
  • a processing pipeline illustrating the data acquisition and processing flow is shown in the FIG. 7.
  • FIG. 7 depicts a processing chain 700 for 3D spectral scanning data acquisition and processing flow as described herein.
  • the processing chain 700 comprises different components and blocks that include a data acquisition block (e.g., shown at 701), a spectral segmentation block (e.g., shown at 702), and a 3D reconstruction block (e.g., shown at 703).
  • the data acquisition block 701 depicts the processing flow for acquiring or capturing images of an object illuminated by the structured light source at different frequencies.
  • structured light patterns necessary for 3D reconstruction are preloaded (e.g., into a DLP projector) for projection to illuminate an object or target of interest. Each pattern is projected for a period of time configured to synchronize with the camera burst exposure.
  • start of the scanning process can be initiated by manually starting the DLP sequence and camera burst using their respective software applications.
  • Synchronization between the DLP projector and the camera can be performed using open loop control (e.g., manually by a human operator) or it can be automated through the use of an external “master” device such as an industrial programmable logic controller (PLC) capable of sub millisecond timing.
  • PLC programmable logic controller
  • the spectral segmentation block 702 depicts the flow for performing spectral segmentation by processing the acquired images or frames an input source of raw data (e.g., spectral imaging camera), including by performing data filtration, masking, pre-processing, endmember extraction (EE), and abundance map reconstructions (AMR) to identify or detect different materials.
  • an input source of raw data e.g., spectral imaging camera
  • data filtration, masking, pre-processing, endmember extraction (EE), and abundance map reconstructions (AMR) to identify or detect different materials.
  • the processing chain includes acquiring the images or frames from the spectral imaging camera at 721 to generate a sequence of frames or complete scan at 722, wherein a frame (e.g., an image of an individual pattern) comprises a hypercube corresponding to the acquisition of one structured illumination pattern from the sequence of patterns necessary to produce a scan.
  • a frame e.g., an image of an individual pattern
  • a scan is an acquisition of the full set of structured illumination patterns necessary to reconstruct the 3D point cloud of one pose of the target object, wherein a pose is one specific location and field of view of a target object.
  • a pose is one specific location and field of view of a target object.
  • an object on a turn table will have one pose per rotational step or height change. It is often necessary to scan multiple poses of an object to build a 3D model.
  • a scan can be thought of as the images of a complete sequence, wherein a sequence is a series of structured illumination patterns necessary to reconstruct a 3D point cloud.
  • a sequence contains multiple patterns, wherein a pattern comprises a ID series of black and white stripes (binary coded or gray code) or grayscale sinusoids (phase) programmed into a structured light source (e.g., a DLP projector for projecting structured illumination patterns).
  • a structured light source e.g., a DLP projector for projecting structured illumination patterns.
  • Each pattern is one of the sequence necessary to reconstruct 3D positions.
  • each frame of the sequence of frames (i.e., each hypercube in the sequence of hypercubes) is processed to extract one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes.
  • An endmember can be thought of as a material that is spectrally unique in the wavelength bands.
  • An endmember is associated to the pixel spectra or spectral signature of materials after the pixel un-mixing and extraction or spectral segmentation process to generate abundance maps that display the fractional amount of material present at each pixel.
  • PCA principal component analysis
  • identification and extraction of endmembers from hypercubes and reconstruction of abundance maps is based on a linear spectral mixture analysis (LSMA) model.
  • LSMA linear spectral mixture analysis
  • the Pixel Purity Index (PPI) algorithm an unsupervised spectral algorithm implemented in commercially available hyperspectral machine vision software, enables the determination and extraction of distinct spectral signatures existing in hypercubes.
  • the PPI algorithm determines the spectral signatures associated to each pixel vector by calculating the spectral response of multiple components and by decomposing them into a collection of spectrally pure constituent spectra.
  • FIG. 8 shows the representation of four induced endmembers extracted by PPI.
  • FIG. 8 is a plot of the reflectance of four extracted endmembers (e.g., shown at 801, 802, 803, and 804 respectively) from a hyperspectral data set using the PPI algorithms as a function of wavelength, where each line is one distinct spectrum in the image.
  • each abundance map (Data Cube) is generated in response to an extracted spectrally distinct signature or endmember and the set of abundance maps or data cubes are combined into a single file at 725 to generate a full scan data cube (e.g., Full Scan Data Cube shown at 730).
  • the Full Scan Data Cube (one data cube per frame) at 730 is a larger Data Cube representing each frame in a scan as an individual data cube.
  • the Full Scan Data Cube is generated based on the abundance maps and captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.
  • FIG. 9 is a flowchart of an embodiment of a method 900 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) that includes generating a full scan data cube (e.g., Full Scan Data Cube shown 730) as shown in the spectral segmentation block 702 of FIG. 7.
  • the method 900 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 910.
  • one or more spectrally distinct signatures or endmembers are extracted based on the spectral data from each of the hypercubes.
  • each hypercube is spectrally segmented to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to an extracted spectrally distinct signature or endmember.
  • a full scan data cube is generated based on the abundance maps. The full scan data cube captures spectrally distinct signature or endmember data and xy- pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.
  • FIG. 10A is a flowchart of an embodiment of a method 1000 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) that includes matching spectrally distinct signatures or endmembers to a particular reference signature associated with a particular material.
  • the method 1000 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 1010.
  • the method includes extracting one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes.
  • the method includes matching one or more of the spectrally distinct signatures or endmembers to a particular reference signature, wherein the particular reference signature identifies or is associated with a particular material.
  • the method includes spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to an extracted spectrally distinct signature or endmember.
  • the method includes generating a lull scan data cube based on the abundance maps that captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.
  • each abundance map is generated in response to a spectrally distinct signature or endmember associated with a material identified based on matching the one or more spectrally distinct signatures or endmembers to a particular reference signature identifying or associated with the material.
  • each of the reference signatures is associated with a different material and the reference signatures associated with different materials are stored in a spectra library (e.g., Library of spectra for materials shown at 724 of FIG. 7).
  • the system is configured to perform 3D spectral scans, extract spectral signatures or endmembers and match these to a library entry or reconstructed endmember corresponding to a specific spectral signature associated with a material.
  • materials common to a selected use case are spectrally distinct and can be distinguished by their spectral fingerprints using even simple algorithms.
  • the system is configured to extract spectral data from the hypercube based on or in response to the particular library entry or target spectral signature to generate an abundance map.
  • the system is configured to match a target spectral signature to spectral data in the hypercube and spectrally segment the hypercube to generate an abundance map based on xy-pixel data associated with the target spectral signature.
  • the target spectral signature is associated with a material of interest.
  • FIG. 10B is a flowchart of an embodiment of a method 1001 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) for generating a 3D reconstruction or 3D point cloud based on or in response to a target spectral signature associated with a material of interest.
  • the method 1001 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 1011.
  • the method includes matching a target spectral signature to the spectral data in the hypercube.
  • the target spectral signature is associated with a material of interest.
  • the method includes spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to the identified xy-pixel data based on matching the target spectral signature to the spectral data in the hypercube.
  • the method includes generating a full scan data cube based on the abundance maps.
  • FIG. IOC is a flowchart of an embodiment of a method 1002 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) using a segmentation algorithm.
  • the method 1002 includes storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 1012 and spectrally segmenting each hypercube using a segmentation algorithm to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember at 1022.
  • the segmentation algorithm includes unsupervised learning algorithms such as K-means clustering and neural networks.
  • the 3D reconstruction block depicts the processing flow for generating 3D point cloud data from the abundance maps.
  • the full scan data cube e.g., Full Scan Data Cube at 730
  • the spectral segmentation block e.g., shown at 702
  • the full scan data cube captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.
  • one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames is identified or associated with a particular material.
  • a frame sequence or scan of one or more abundance maps associated with a particular material is generated for each identified material.
  • a 3D point cloud or 3D geometric mesh of the object is extracted or generated from the images based on the spectral segmentation and identified material differences.
  • a 3D point cloud or 3D geometric mesh of the object is extracted or generated using a sequence of the one or more abundance maps associated with the particular material (e.g., from each frame sequence shown at 732).
  • different abundance maps associated to different extracted endmembers are reconstructed and using the sequences of reconstructed abundance maps, 3D spectral point clouds associated to each extracted endmember are reconstructed.
  • the 3D point cloud or 3D geometric mesh for each material are included as segments to form or generate a final point cloud (e.g., a segmented point cloud showing segmentation of the object by material).
  • the segmented point cloud is displayed at 735.
  • spectral resolution Due to computational considerations there is generally a tradeoff between spectral resolution and 3D resolution.
  • a spectral imaging component e.g., multispectral versus hyperspectral camera
  • smaller datasets associated with multispectral versus hyperspectral reduce computational complexity with some implications to the selection of processing algorithms used for performing spectral segmentation.
  • a broadband illumination source and multispectral imaging camera or system having an NIR (600-1000nm) wavelength range and a fairly limited wavelength resolution (e.g., 25 bands) is typically sufficient. This configuration provides a lower- risk optical layout as compared to hyperspectral imaging.
  • the spectral resolution of a lull hyperspectral system may be required to differentiate certain materials.
  • the use of a broadband illumination source may not be able to achieve the same level of spatial accuracy as laser-line sources used in laser-based 3D analysis products.
  • the low light intensity of continuous halogen light sources in the NIR region is generally insufficient to enable 3D reconstruction (e.g., the generation of a 3D spectral point cloud) due to exposure time limitations.
  • a more powerful light source providing a higher light intensity may be needed in certain applications where reducing 3D spectral scanning time is a concern.
  • a high intensity light source is preferable to permit multispectral or hyperspectral image collection.
  • Projection brightness can also be improved by optimizing projector field of view with an appropriate selection of projector lens focal length (e.g., through a projector lens retrofit or supplementary optical system).
  • Signal intensity and dynamic range are both critical parameters in design of a 3D spectral scanning system.
  • the 3D point cloud reconstruction step is performed after the spectral segmentation step.
  • the 3D point cloud reconstruction algorithms use gray code (or phase) to calculate distance and thus, dynamic range or depth of signal modulation is critical for accurate depth calculation.
  • the structured illumination source comprises a line source and a physical dimension measurement is performed by measuring a lateral offset of a line projected on a target surface at an angle.
  • system for simultaneous depth profile and spectral measurement includes one or more illumination sources, one or more detectors, and a processor.
  • the one or more detectors is/are configured to capture one or more frames of a target illuminated using the one or more illumination sources.
  • the processor is configured to determine a depth measurement based at least in part on the one or more frames; determine a spectral measurement based at least in part on the one or more frames; and determine a corrected map of depth and spectral information by correcting the spectral measurement using the depth measurement.
  • the system enables inspection of a target both for physical dimensions (e.g., height profiles over a 2-dimensional plane) and material properties (e.g., a spectral measurement over 2- dimensional plane).
  • the system enables the two types of inspections to be performed simultaneously.
  • the physical dimension measurement is performed by measuring a lateral offset of a line projected on a target surface at an angle.
  • the spectral measurement is performed by measuring the back reflected spectrum or forward reflected/scattered spectrum of the target surface illuminated using a broadband source.
  • FIG. 11 is a diagram illustrating an embodiment of a measurement device.
  • illumination source 1100 e.g., a broadband illumination source, a white laser source, etc.
  • Cylindrical lens 1104 transforms the beam to be focused in one dimension (e.g., a line).
  • the one-dimensional beam passes through beam splitter 1106, optic 1112, and dichroic splitter 1114 to be projected onto the surface of target 1118 at an angle off of normal to the surface of target 1118.
  • the surface of target 1118 with the line projected e.g., line in and out of the page of FIG.
  • a backscattered beam path 1126 passes through dichroic splitter 1114 and optic 1112 and is reflected using beam splitter 1106 to be measured by detector 1110 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1108.
  • detector 1110 e.g., spectrum detector, pushbroom spectrometer, etc.
  • Illumination source 1116 (e.g., a monochromatic source, a narrowband laser source, etc.) generates a line that is projected on the surface of target 1118 (e.g., line in and out of the page of FIG. 11) along path 1128 after being reflected by dichroic splitter 1114 (e.g., dichroic beam splitter).
  • the surface of target 1118 is imaged normal to the surface of target 1118 using detector 1122 (e.g., a 2-dimensional imaging device) using optic 1120 following beam path 130.
  • FIG. 12 is a diagram illustrating an embodiment of a measurement device.
  • illumination source 1200 e.g., a broadband illumination source, a white laser source, etc.
  • Cylindrical lens 1204 transforms the beam to be focused in one dimension (e.g., a line).
  • the one-dimensional beam passes through beamsplitter 1206 and optic 1212 to be projected onto the surface of target 1218 at an angle off of normal to the surface of target 1218.
  • the surface of target 1218 with the line projected (e.g., line in and out of the page of FIG. 12) is translated along direction 1232 (e.g., left and right in the page of FIG.
  • a backscattered beam path 1226 passes through optic 1212 and is reflected using beamsplitter 1206 to be measured by detector 1210 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1208.
  • detector 1210 e.g., spectrum detector, pushbroom spectrometer, etc.
  • FIG. 12 is translated along direction 1232 (e.g., left and right in the page of FIG. 12) so that a measurement can be taken over the entire surface of target 1218.
  • the surface of target 1218 is imaged normal to the surface of target 1218 using detector 1222 (e.g., a 2-dimensional imaging device) using optic 1220 following beam path 1230.
  • detector 1222 e.g., a 2-dimensional imaging device
  • FIG. 13 is a diagram illustrating an embodiment of a measurement device.
  • illumination source 1300 e.g., a broadband illumination source, a white laser source, etc.
  • Cylindrical lens 1304 transforms the beam to be focused in one dimension (e.g., a line).
  • the one-dimensional beam passes through optic 1312 to be projected onto the surface of target 1318 at an angle off of normal to the surface of target 1318.
  • the surface of target 1318 with the line projected (e.g., line in and out of the page of FIG. 13) is translated along direction 1332 (e.g., left and right in the page of FIG.
  • Backscattered beam path 1326 is measured by detector 1310 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1308.
  • detector 1310 e.g., spectrum detector, pushbroom spectrometer, etc.
  • FIG. 13 is translated along direction 1332 (e.g., left and right in the page of FIG. 13) so that a measurement can be taken over the entire surface of target 1318.
  • the surface of target 1318 is imaged normal to the surface of target 1318 using detector 1322 (e.g., a 2-dimensional imaging device) using optic 1320 following beam path 1330.
  • detector 1322 e.g., a 2-dimensional imaging device
  • Illumination beam path 1336 (e.g., also path 1324) is incident to target 1318 at angle
  • Incidence imaging direction 1332 (e.g., also beam path 1330) is normal to the surface of target 1318.
  • Backscatter path 1340 e.g., also beam path 1326) is at angle 1338 (e.g., 1 degree) off of illumination beam path 1336.
  • FIG. 14 is a diagram illustrating an embodiment of a measurement device.
  • illumination source 1400 e.g., a broadband illumination source, a white laser source, etc.
  • Cylindrical lens 1404 transforms the beam to be focused in one dimension (e.g., a line).
  • the one-dimensional beam passes through optic 1412 to be projected onto the surface of target 1418 at an angle off of normal to the surface of target 1418.
  • the surface of target 1418 with the line projected (e.g., line in and out of the page of FIG. 14) is translated along direction 1432 (e.g., left and right in the page of FIG.
  • Forward reflected/scattered beam path 1426 is measured by detector 1410 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1408.
  • detector 1410 e.g., spectrum detector, pushbroom spectrometer, etc.
  • FIG. 14 is translated along direction 1432 (e.g., left and right in the page of FIG. 14) so that a measurement can be taken over the entire surface of target 1418.
  • the surface of target 1418 is imaged normal to the surface of target 1418 using detector 1422 (e.g., a 2-dimensional imaging device) using optic 1420 following beam path 1430.
  • detector 1422 e.g., a 2-dimensional imaging device
  • Illumination beam path 1436 (e.g., also path 1424) is incident to target 1418 at angle
  • Incidence imaging direction 1432 (e.g., also beam path 1430) is normal to the surface of target 1418.
  • Backscatter path 1440 e.g., also beam path 1426
  • angle 1438 e.g., theta2 off of illumination beam path 1436.
  • FIG. 15 is a diagram illustrating an embodiment of a measurement device.
  • illumination source 1500 e.g., a broadband illumination source, a white laser source, etc.
  • optic 1502 exits a fiber and is collimated using optic 1502 along path 1524.
  • Cylindrical lens 1504 transforms the beam to be focused in one dimension (e.g., a line).
  • the one-dimensional beam passes through optic 1512 to be projected onto the surface of target 1518 at an angle off of normal to the surface of target 1518.
  • the surface of target 1518 with the line projected e.g., line in and out of the page of FIG. 15
  • direction 1532 e.g., left and right in the page of FIG.
  • Forward reflected/scattered beam path 1526 is split off beam path 1530 using diffraction grating 1521 and measured by detector 1510 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1508.
  • detector 1510 e.g., spectrum detector, pushbroom spectrometer, etc.
  • FIG. 15 is translated along direction 1532 (e.g., left and right in the page of FIG. 15) so that a measurement can be taken over the entire surface of target 1518.
  • the surface of target 1518 is imaged off normal to the surface of target 1518 using detector 1522 (e.g., a 2-dimensional imaging device) using optic 1520 following beam path 1530 and beam path 1531 after passing through diffraction grating 1521.
  • detector 1522 e.g., a 2-dimensional imaging device
  • FIG. 16 is a diagram illustrating an embodiment of a measurement system.
  • measurement device 1614 in the measurement system of FIG. 16 is the measurement device of FIG. 11, FIG. 12, FIG. 13, FIG. 14, or FIG. 15.
  • measurement device 1614 is used to measure a target.
  • Controller 1602 controls measurement device 1614 using signals 1610 (e.g., laser control signals, detector control signals, etc.) for operating measurement device 1614.
  • Detector data signals 1612 are transmitted from measurement device 1614 and controller 1602.
  • Controller 1602 includes interface 1604 and processor 1606 for controlling measurement device 1614 to take a measurement of a target and provide data for processing.
  • User device 1600 interacts with controller 1602 to request and provide measurement of a target.
  • FIG. 17 is a flow diagram illustrating an embodiment of a process for making a simultaneous depth profile and spectral measurement. In some embodiments, the process of FIG.
  • a relative position (in X) is set between a target and a sensor.
  • a position is set for a stage that translates the target being measured or the device measuring the target.
  • a measurement is taken by capturing frames.
  • one or more detectors e.g., an imaging detector for a height measurement and/or a spectrometer for a spectral measurement
  • a depth measurement (for Y) is determined.
  • a height map is determined by processing the captured frames and using a calibration that corresponds a lateral offset of a portion of a line as imaged in the captured frames with a height.
  • a spectral measurement (for Y) is determined.
  • a spectrum is determined by processing the captured frames and using a calibration that corresponds a backscattered light spectrum for the line as imaged in the captured frames with calibrated wavelengths.
  • a corrected map (for XY) of spectral and depth information is determined using depth measurement and spectral measurement.
  • the spectral measurement is corrected using information from the depth measurement.
  • the spectral measurement is corrected for the shift caused by the depth measurement (e.g., using a triangulation measurement).
  • FIG. 18 is a flow diagram illustrating an embodiment of a process for determining a depth calibration.
  • the depth calibration of FIG. 18 is used to determine a depth measurement from a frame as in 1704 of FIG. 17.
  • a height calibration target is positioned.
  • a measurement is taken of a calibration standard at a position on the height calibration target.
  • multiple measurements are taken at multiple positions to determine a height calibration.
  • a measurement is taken by capturing frames.
  • the image of a height standard is imaged using a 2-dimensional imaging device (e.g., a camera).
  • a depth measurement for Y
  • depth calibration is determined using the depth measurement and calibration height.
  • the frames are analyzed to determine a height or depth and the measurement is corresponded to the standard’s calibrated height.
  • FIG. 19 is a flow diagram illustrating an embodiment of a process for determining a depth measurement.
  • the process of FIG. 19 is used to implement 1704 of FIG. 17.
  • frames are received.
  • frames are corrected using the depth calibration.
  • the depth measurement, the corrected frames, and correction are provided.
  • the frames are analyzed and determined heights are corrected using the depth calibration.
  • the correction and corrected frames are provided to later correct a spectral measurement of the target.
  • FIG. 20 is a flow diagram illustrating an embodiment of a process for determining a spectral calibration.
  • the spectral calibration of FIG. 20 is used to determine a spectral measurement from a frame as in 1706 of FIG. 17.
  • a spectral calibration target is positioned.
  • a measurement is taken of a spectral standard at a position on the spectral calibration target.
  • multiple measurements are taken at multiple positions to determine a spectral calibration.
  • a measurement is taken by capturing frames.
  • the image of a spectral standard is imaged using a spectral measurement device (e.g., a pushbroom spectrometer).
  • a spectral measurement device e.g., a pushbroom spectrometer
  • a spectral measurement (for Y) is determined.
  • a spectral calibration is determined using the spectral measurement and spectral calibration. For example, the frames are analyzed to determine a spectrum and the measurement is corresponded to the spectral standard’s calibrated spectrum.
  • FIG. 21 is a flow diagram illustrating an embodiment of a process for determining a spectral measurement.
  • the process of FIG. 21 is used to implement 1706 of FIG. 17.
  • frames are received.
  • frames are corrected using the spectral calibration.
  • the spectral measurement and the corrected frames are provided.
  • the frames are analyzed and spectra are corrected using the spectral calibration.
  • FIG. 22 is a diagram illustrating an embodiment of data from a device measuring height and a spectrum of a target simultaneously.
  • the target frame data of FIG. 22 comprises target 1118, target 1218, target 1318, target 1418, or target 1518.
  • a line from an illumination source is projected onto a target shown in 2200 with a profile as shown.
  • the illumination source is projected at an angle so that a surface that is at a different height is shifted.
  • an illumination line incident at an angle 9 to a feature of height z appears shifted to the right.
  • the frame image is then converted to heights by using the shift to determine a height map in 2202.
  • a spectral response of the line is shown in 2204 also with the lateral shifts for each of the spectral responses (e.g., shown as the lines in different colors).
  • the corrected spectral response of the lines is shown in 2206 as shifted using the height information from 2202 and 2200 so that a height mapping and spectral mapping can be generated and overlaid.
  • FIG. 23 is a diagram illustrating an embodiment of data from a device measuring height and a spectrum of a target simultaneously.
  • the target frame data of FIG. 23 comprises data related to target 1118, target 1218, target 1318, target 1418, or target 1518.
  • image frame 2301 shows an o-ring in a depression.
  • the line cut is indicated by 2303 through image frame 2301 is shown in height profile 2300.
  • Height profile 2300 is a one dimensional cut along the X-dimension and shows a step and the surface cut of the o-ring.
  • 2302 indicates the depression and the corresponding spectrum line as well as the center of the step 2304 and its corresponding spectrum line in the spectrum versus position graph 2306.
  • the spectrum related to 2302 is shown in a line graph of 2312 as indicated by 2308, and the spectrum related to 2304 is shown in a line graph of 2312 as indicated by 2310. Note that the spectra are different implying that the material of the o-ring is different from the material at the center of the step.
  • the structured light source comprises a multi-spectral fringe projection system.
  • the fringes are generated interferometrically, creating different fringe frequencies per illumination wavelength. Imaging is done through multiple phase shifts with a multi-wavelength imaging spectrometer.
  • fringes are generated using an interferometer with a tilted mirror. Fringes are imaged onto an object, target of interest, or part under test. At an angle, Q, a multi-wavelength detector/imaging spectrometer views the object. By combining phase-shifted images, the system is capable of measuring both spectral content of the object as well as the depth map.
  • FIG. 24 is a diagram illustrating an embodiment of a twyman green structured light system using a broadband source.
  • the system 2400 comprises a broadband source 2410 (e.g.
  • an interferometer 2420 comprising a mirror on PZT 2422 and a tilted mirror 2424, a detector 2430, and lenses (e.g., shown at 2426 and 2436 respectively).
  • broadband source 2410 is incident on a beamsplitter 2415.
  • the signal is divided into two beams using an interferometer 2420, such as a twyman green as is shown in FIG. 24.
  • interferometer configurations such as a Mach-Zender
  • Flat mirrors are in both arms of the interferometer 2420, but in one arm, one of the mirrors is tilted (e.g., tilted mirror 2424) to create tilt fringes.
  • the other mirror e.g., mirror on PZT 2422
  • PZT phase-shifting
  • a series of laser diode sources coupled using a grating (as shown in FIG. 25) or a combination of fiber-coupled LEDs (not shown) are used.
  • FIG. 25 is a diagram illustrating an embodiment of a twyman green structured light system 2500 having components as shown in FIG. 24 but using a series of laser diode sources 2510 in place of the broadband source 2410 as a structured light source.
  • the system 2500 comprises a series of laser diode sources 2510, an interferometer 2520 comprising a mirror on PZT 2522 and a tilted mirror 2524, a detector 2530 and lenses (e.g., shown at 2526 and 2536 respectively).
  • the series of laser diode sources 2510 are incident on a beamsplitter 2515.
  • the signal is divided into two beams using an interferometer 2520, such as a twyman green as is shown in FIG. 25.
  • an interferometer 2520 such as a twyman green as is shown in FIG. 25.
  • Other interferometer configurations such as a Mach- Zender, can also be implemented.
  • Flat mirrors are in both arms of the interferometer 2520, but in one arm, one of the mirrors is tilted (e.g., tilted mirror 2524) to create tilt fringes.
  • the other mirror e.g., mirror on PZT 2522
  • it is important that the paths are matched, a.k.a.
  • the distance from the beamsplitter face to the mirror in each arm of the interferometer (e.g., shown at 2501 and 2502) must be equal for highest contract interference fringes. Also important to note that the maximum tilt angle of the mirror where high contrast fringes will occur will be limited by the coherence properties of the source.
  • the generated interference pattern is imaged using a lens (e.g., shown at 2426 of
  • FIG. 24 and at 2526 of FIG. 25 onto an object or target of interest, which in this case is a sample under test (e.g., shown at 2440 of FIG. 24 and at 2540 of FIG. 25). Any height deviation on the sample will cause a local shift in the fringe pattern that can be detected by the imaging system.
  • the imaging system consists of a lens (e.g., shown at 2436 of FIG. 24 and at 2536 of FIG. 25) that images the part under test onto a multi-wavelength 2D detector (e.g., shown at 2430 of FIG. 24 and at 2530 of FIG. 25).
  • This detector could take many forms (e.g., a snapshot mosaic sensor). In this case, the part can be directly imaged onto the detector as shown in FIGS. 24 and 25.
  • an imaging spectrometer such as an IRIS imaging spectrometer, a multiple- image Fourier transform spectrometer, or a compact birefringent Fourier transform spectrometer.
  • an imaging spectrometer such as an IRIS imaging spectrometer, a multiple- image Fourier transform spectrometer, or a compact birefringent Fourier transform spectrometer.
  • the result is a 2D image for a given sampled wavelength band (also referred to as a channel).
  • FIG. 26 shows a first projected pattern 2610, a second projected pattern 2620 that is out of phase with respect to first projected pattern 2610, and the resultant image 2630 if the two images (e.g., shown at 2610 and 2620 respectively) are averaged. No fringes are visible in the result. [0161] In some embodiments, many images are acquired with small phase shifts such that the net sum of all images appears to have no residual interference pattern.
  • a LSPSI least squares phase shifting interferometry
  • the result of the mean image is a map of the reflectance as a function of wavelength for the part under test. If the part is comprised of multiple materials, for example a metal part with rubber o- rings, the spectrum can be used to identify the different materials present in the measurement. As discussed herein, a spectra library may be used to store reference signatures is associated with different materials to be matched against a spectrum generated by the above-disclosed techniques.
  • the projected fringe pattern for each wavelength band has a different frequency.
  • the measured phase at least three phase shifted measurements are required. The number of phase shifts and phase shift distance must be sufficient to recover the phase for all wavelengths in the system.
  • phase maps can be combined to generate synthetic wavelengths. This is commonly done in multi-wavelength interferometry in order to measure surfaces where the phase difference due to a large feature exceeds l/2. In this case where fringes are projected onto the object, the phase difference cannot be combined.
  • the equivalent wavelength for such a system is X ep
  • FIG. 27 depicts a synthetic wavelength example using 0.8 and 0.9um fringes to generate a source of effective 7.2um fringes.
  • a simulation was performed using 5 wavelengths (0.5-0.9um in O.lum increments) measuring a small bump of height 0.0796um.
  • An input OPD map is shown at 2710.
  • the simulated fringe pattern for wavelengths of 0.8um and 0.9um are shown at 2720 with their corresponding wrapped phases.
  • the fringe pattern and wrapped phase for a 7.2um source is also calculated and compared to the synthetic result shown at 2730.
  • This example also highlights that the wrapped phase of the 7.2um result requires no phase unwrapping, only tilt removal to detect the height of the object.
  • a system of this kind can retain good depth resolution while also having large dynamic range (be able to measure objects » l RGo] «Lk ⁇ /2).
  • FIG. 28 depicts equivalent synthetic wavelengths for all combinations of wavelength from 0.5-0.9um.
  • the fringe patterns are phase shifted in many different frequencies. This is to reduce the complexity of the unwrapping problem and be able to measure greater height differences.
  • the synthetic wavelength approach used here achieves the same end goal.
  • a challenge in multi-l interferometry is that the accuracy of the measurement is highly dependent on the wavelength stability of the source. Most systems of this nature require expensive and highly stable laser sources. In this method, there is an opportunity to filter the wavelength at the detector and restrict the measured bandwidth, thus enforcing the wavelength stability of the system.
  • a synthetic wavelength can be generated by combining wavelengths of different frequencies.
  • This large synthetic wavelength can be used to guide the phase unwrapping of the small wavelengths, and a system of this kind can retain excellent depth resolution while also having large dynamic range, a.k.a. be able to measure objects » 7pro
  • the disclosed techniques allow for simultaneous measurement of height/depth measurement and spectral response. Additionally, a design of this type allows for the ability to confirm a part was built correctly, with both the proper mechanical dimensions as well as confirm or identify that the proper materials were utilized during the manufacturing process.
  • This design is advantageous because there is no scanning of the part under test required (as opposed to laser line triangulation systems that require the laser line to scan the part to build up a depth map over time).
  • the fringes that are generated are caused by interference. This results in a larger depth of focus than systems that project the image of fringes onto the sample. Additionally, this technique can use a broadband source, or multiple individual sources for versatility.
  • the structured light source comprises a multi-spectral fringe projection system, where the fringes are generated by interfering the +1 and -1 orders of a polarization grating which are orthogonally circularly polarized, and imaging is performed with a multi-wavelength imaging spectrometer through multiple phase shifts achieved by the rotation of a polarizer.
  • FIG. 29 depicts a diagram of a structured light system 2900 using laser sources and a polarization grating.
  • laser sources 2910 are coupled using a grating 2920 (e.g. to coalign different wavelengths shown at 2901, 2902 and 2903) and are used and after the illumination light passes through a polarizer 2922 to illuminate a polarization grating 2924, it is imaged onto an object, target of interest, or part under test (e.g., shown at 2930).
  • a multi-wavelength camera 2940 views the object 2930 through a polarizer 2942 that can be rotated to achieve different phase shifts of the fringe pattern.
  • FIG. 30 depicts a diagram of a structured light system 3000 using fiber coupled
  • a multi-l fiber-coupled LED 3010 is used as the structured light source and the illumination light goes through a polarizer 3022 to become linearly polarized. The illumination light then goes through a polarization grating 3024 which is imaged onto an object, target of interest, or part under test (e.g., shown at 3030). At an angle, 0 (e.g., shown at 3025), a multi-wavelength camera 3040 views the object 3030 through a polarizer 3042 that can be rotated to achieve different phase shifts of the fringe pattern.
  • the source of the system can be for example lasers coupled with a grating (FIG. 29) or fiber-coupled LEDs (FIG. 30).
  • interference fringes are generated using a polarization grating which sends orthogonally circularly polarized light into the +1 and -1 orders. Virtual fringes are imaged onto part under test (they are virtual because the two beams are orthogonally polarized). At an angle, 0, a multi-wavelength detector/imaging spectrometer views the part under test. A polarizer sits in front of the camera and is rotated to achieve phase shifts of the fringe pattern. By combining phase shifted images, the system is capable of measuring both spectral content of the part under test as well as the depth map.
  • the multi-wavelength 2D detector can take many forms.
  • One example is a snapshot mosaic sensor.
  • the part could be directly imaged onto the detector as shown in FIG. 29 and FIG 30.
  • an imaging spectrometer such as an IRIS imaging spectrometer, a multiple-image Fourier transform spectrometer, or a compact birefringent Fourier transform spectrometer can be used.
  • the projected fringe frequency for all wavelengths in this system will be the same since a grating is used to generate the interference pattern.
  • at least three phase-shifted images must be acquired with varying the rotation angle of the imaging polarizer.
  • a common choice is acquiring 4 images, rotating the polarizer by an angle of 45° between measurements such that the phase shift corresponds to a shift of l/4, a quarter of a period.
  • the disclosed techniques allow for simultaneous measurement of height/depth measurement and spectral response. Additionally, a design of this type allows for the ability to confirm a part was built correctly, with both the proper mechanical dimensions as well as confirm or identify that the proper materials were utilized during the manufacturing process.
  • This design is advantageous because there is no scanning of the part under test required (as opposed to laser line triangulation systems that require the laser line to scan the part to build up a depth map over time). In this case, the fringes that are generated are caused by interference. This results in a larger depth of focus than systems that project the image of fringes onto the sample.
  • This design can use a broadband source, or multiple individual sources.
  • FIG. 31 shows a table 3100 listing examples of various industrial applications for applying 3D spectral scanning as disclosed herein.
  • 3D spectral scanning applications that span various fields and major industries including automotive design and manufacturing, material processing, and plant inspection applications.
  • automotive industry has fully adopted automation and robotics in their manufacturing lines.
  • automated optical inspection is a crucial application for automotive and other industries such as pharma and food.
  • the added value of spectral imaging is the availability of material composition information. The need is driven by reducing the cost of correcting errors in manufacturing. Finding errors early saves money.
  • 3D spectral scanning is for material processing applications such as additive manufacturing and removal processing.
  • material processing steps may be controlled or initiated based on or in response to the 3D shape and material composition data output by a 3D spectral scanning system as described herein.
  • the 3D spectral scanning system can also be a component of a material processing platform designed or configured to achieve various objectives (e.g., material inspection, detection and diagnosis, automated removal processing based on or in response to 3D point cloud representations segmented by material).
  • inspection using the 3D spectral scanning techniques as described herein can be part of a feedback loop to inform a material processing system in making decisions for next steps and actions.
  • a material processing method comprises receiving 3D shape and material composition data output by scanning an object using the 3D spectral scanning techniques as described herein and taking material processing steps in response to the received output.
  • the material processing steps include
  • FIG. 32 is a block diagram of an embodiment of a material processing system or platform 3200 that uses 3D spectral scanning to execute material processing steps.
  • the material processing platform 3200 comprises a 3D spectral scanning system (e.g., 3D Spectral Scanner 100) for simultaneous depth profile and spectral measurement as shown in FIG. 1 and a Material Processing Unit 3201.
  • 3D spectral scanning system e.g., 3D Spectral Scanner 100
  • FIG. 1 the various components of the 3D spectral scanning system are described herein with respect to FIG. 1.
  • Material Processing Unit 3201 is configured to execute various material processing steps in response to the output (e.g. Output 160) received from 3D Spectral Scanner 100.
  • Material Processing Unit 3201 comprises various modules including Diagnostic Module 3210, Decision Module 3220, and Control Module 3210.
  • Diagnostic Module 3210 performs detection and diagnosis to identify issues associated with the object based on the 3D point cloud segmented by material. For example, based on a 3D point cloud representation of the object segmented by material, Diagnostic Module 3210 detects abnormal or unexpected areas on an object (e.g., contamination, deterioration, leakage, deformities in shape etc.) based on the 3D depth profile and material composition provided by the output of 3D Spectral Scanner 100.
  • a database or library e.g., Data 3215
  • the database can be used to store data associated with common contaminants, or data associated with the shape of common manufacturing defects.
  • Decision Module 3220 is configured to generate a decision in response to the detection and diagnosis performed by Diagnostic Module 3210. In some embodiments, Decision Module 3220 generates a decision using machine learning techniques. Control Module 3230 outputs a control action in response to the decision generated by Decision Module 3220. In some embodiments, Control Module 3230 actuates or controls one or more devices (e.g., Devices 3240) in response to the control action to address the issue related to the object. In some cases, the one or more devices (e.g., Devices 3240) perform the control action received from Control Module 3230 to address the issue related to the object.
  • devices e.g., Devices 3240
  • Decision Module 3220 outputs a decision to apply a cleaning process to the contaminated area based on the 3D point cloud showing the shape of the contaminated area and providing the composition of the contaminated material.
  • Control Module 2330 addresses the issue by instructing a device to spray the contaminated area with a cleaning element suitable to address the contamination.
  • the Diagnostic Module 3210 detects and diagnosis an incorrect part on an object or a manufacturing defect on the object.
  • Decision Module 3220 outputs a decision to remove and replace the incorrect part, or to reject the object due to the manufacturing defect.
  • Control Module addresses the issue by instructing a device (e.g., a robotic arm with end effector) to remove and replace the incorrect part, or by instructing a device to dispose of the object having the manufacturing defect.
  • FIG. 33 is a flowchart of an embodiment of a method for material processing using
  • the method includes performing 3D spectral scanning on an object.
  • the 3D spectral scanning may be performed according to any of the disclosed techniques as described herein.
  • the method includes performing detection and diagnosis to identify issues associated with the object based on the 3D spectral scanning.
  • detection and diagnosis is based on the 3D point cloud segmented by material provided by the 3D spectral scanning system.
  • the method includes generating a decision in response to the detection and diagnosis.
  • the decision can include a recommended course of action to address an issue associated with the object.
  • the method includes outputting a control action in response to the decision.
  • one or more instructions are generated in order to produce an action.
  • the method includes actuating or controlling a device in response to the control action to address an issue associated with the object.
  • the one or more instructions can be used to generate an action taken by a device or system configured to address a particular issue.
  • the method includes repeating the process starting at the 3D spectral scanning step. For example, after an issue associated with an object has been detected, diagnosed, and a recommended action has been executed, the system performs 3D spectral scanning on the same or a subsequent object and repeats the process to identify and address other issues.
  • the material processing system or platform is configured to obtain both the 3D shape and the material of an object (such as a part being manufactured) using the 3D spectral scanning techniques disclosed herein, detect and diagnose issues associated with the object (e.g., recognize a contaminated, deteriorated, degraded, or otherwise substandard or abnormal portion of the object, provide a recommended action to address the issues, and execute the action (e.g., clean up contaminated portions or overlay or replace deteriorated or degraded portions while leaving normal portions intact).
  • Another area for the application of 3D spectral scanning is for automotive inspection applications. For example, automated inspection of module assemblies in a production line verifies an assembly for the correct installation of components and confirms the correct materials.
  • FIG. 34 illustrates an application of 3D spectral scanning to provide 3D segmentation by material for automated optical inspection.
  • Another example is automated inspection of auto interior upholstering and fitting.
  • 3D spectral scanning Another area the application of 3D spectral scanning is for food quality analysis and control. For natural as well as processed foods, quality and safety analysis are required by federal institutions in most countries. 3D spectral scanning can provide non-destructive analysis and automation to identify many quality items such as: ripeness and quality (bruising and texture) of fruits and vegetables, contaminants, bacterial content, and process control in food processing plants.
  • 3D spectral scanning can also be of use in the pharmaceutical industry.
  • Pharmaceutical drugs and vitamins are commonly configured in the form of a solid pill.
  • the pills have standard shapes.
  • the shape and composition of pills can be readily verified with 3D spectral scanning.
  • the applications in pharma are in manufacturing: pill conveyor belt sorting, quality control, defects, contamination, and dose. Additional applications are possible when drugs are dispensed to patients. Detection of the correct medication and security against counterfeit or contaminated sources can be done at the point of distribution or dispensing.
  • the 3D spectral scanning techniques as disclosed herein can be used to acquire or capture images inside a surgical site.
  • Such images can be acquired or taken once before closing the incision, during or throughout the course of a special scanning procedure wherein a surgeon or a robot methodically probes tissue to expose more area to the imager, or periodically or continuously during the surgery (e.g., attached to a surgical robot, an endoscope, a dedicated scanning robot arm, a surgery light fixture or simply as a handheld imager).
  • the 3D spectral scanning techniques as disclosed herein can be used to outline and classify all objects or targets of interest in the images by material and depth profile or 3D shape.
  • a 3D spectral scanning system can be used to identify all shapes that are made of foreign substances not naturally present in the body (e.g., using spectral segmentation and material identification), and can fiirther identify foreign shapes that are not naturally present in the body.
  • a library of CAD shapes of various different instruments is used to look for partial matches, such as the visible end of a clamp nestled in tissue (which is a subset of the stored CAD rendering of the clamp).
  • AI or similar techniques can be used to classify 3D shapes for example as“naturally present in the body” or“not naturally present.” Glass fragments, bullets and fragments and dirt all have shape characteristics that are readily differentiated from the shapes of components of the human body.
  • a display system displays and highlights all foreign objects and suspected foreign objects, color coded and/or labeled as to type and shown against the color-coded surrounding tissue to make them easy to be located. After attempted retrieval of the foreign object by the surgeon or surgical robot, the entire sequence can be repeated until a clean scan is obtained.
  • a 3D spectral scanning system as disclosed herein can be used by a surgical robot to methodically remove all foreign objects meeting certain criteria during a surgical procedure by performing a sequence of steps including (1) probe and image objects in a scene (e.g., the surgical site); (2) classify or identify objects by depth profile and material composition; (3) remove identified objects; and (4) repeat steps (l)-(3) to probe, image, classify, identify and remove additional objects throughout the course of the surgical procedure. Examples of applications of this method are automated cleaning of a wound site or removal of shrapnel or glass fragments.
  • the 3D spectral scanning system can be operated in a video mode, with the surgeon (or robot) using the video feed (which may be displayed on goggles or a screen for a surgeon) to“navigate” in real time, probing a surgical site and removing each tagged object as it becomes visible.
  • the 3D spectral scanning system can be used to distinguish normal from abnormal tissues (such as certain tumors, growths, gall stones, kidney stones, etc.) ⁇
  • the system uses a combination of shape (morphological) and material information to classify the tissue.
  • the material differences between tissues are enhanced by the introduction into the body of dyes or other materials that are differentially absorbed or retained by normal vs. abnormal tissue.
  • FIG. 35 shows the difference between traditional 3D scanning and 3D spectral scanning as disclosed herein.
  • the disclosed method can automatically differentiate components and add material identity. This improves the information content and value of the system by saving time, enhancing productivity, and improving quality.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Theoretical Computer Science (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)

Abstract

L'invention concerne des techniques de balayage spectral 3D pour une mesure simultanée d'un profil de profondeur et d'un spectre et pour l'utilisation d'un balayage spectral 3D pour un traitement de matériau. Une source de lumière structurée et un dispositif d'imagerie spectrale sont conçus pour acquérir des images d'un objet éclairé par la source de lumière structurée à des fréquences différentes. Une segmentation spectrale et une reconstruction 3D sont réalisées à l'aide des images pour identifier des différences de matériau et un profil de profondeur de l'objet. Une représentation informatique de l'objet montrant des dimensions spatiales et des différences de matériau en réponse à une segmentation spectrale et une reconstruction 3D est générée en tant que sortie. La sortie renseigne la détection et le diagnostic de problèmes pour automatiser les étapes de traitement de matériau dans la fabrication industrielle et d'autres applications.
PCT/US2020/018133 2019-02-15 2020-02-13 Mesure simultanée de profil de profondeur et de spectre Ceased WO2020168094A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
US201962806418P 2019-02-15 2019-02-15
US201962806384P 2019-02-15 2019-02-15
US62/806,384 2019-02-15
US62/806,418 2019-02-15
US201962893537P 2019-08-29 2019-08-29
US62/893,537 2019-08-29

Publications (1)

Publication Number Publication Date
WO2020168094A1 true WO2020168094A1 (fr) 2020-08-20

Family

ID=70005710

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2020/018133 Ceased WO2020168094A1 (fr) 2019-02-15 2020-02-13 Mesure simultanée de profil de profondeur et de spectre

Country Status (1)

Country Link
WO (1) WO2020168094A1 (fr)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112669362A (zh) * 2021-01-12 2021-04-16 四川深瑞视科技有限公司 基于散斑的深度信息获取方法、装置及系统
CN113188450A (zh) * 2021-04-23 2021-07-30 封泽希 基于结构光的场景深度检测方法及系统
CN113506348A (zh) * 2021-07-15 2021-10-15 吉林省凯迪科技有限公司 一种基于格雷码辅助的三维坐标计算方法
CN113609907A (zh) * 2021-07-01 2021-11-05 奥比中光科技集团股份有限公司 一种多光谱数据的获取方法、装置及设备
CN113705288A (zh) * 2020-11-25 2021-11-26 天翼智慧家庭科技有限公司 一种基于3d人脸重建的公众人物识别系统和方法
CN113959961A (zh) * 2021-12-22 2022-01-21 广东省农业科学院动物科学研究所 一种基于高光谱图像的单宁添加剂防伪检测方法及系统
CN114095715A (zh) * 2021-11-18 2022-02-25 中国科学院长春光学精密机械与物理研究所 一种动态目标的结构光扫描成像方法及其成像装置
CN115585755A (zh) * 2022-10-20 2023-01-10 大连理工大学 一种基于伪随机二进制序列编码投影的平直度检测方法
US11605216B1 (en) 2022-02-10 2023-03-14 Elementary Robotics, Inc. Intelligent automated image clustering for quality assurance
US11605159B1 (en) 2021-11-03 2023-03-14 Elementary Robotics, Inc. Computationally efficient quality assurance inspection processes using machine learning
US11675345B2 (en) 2021-11-10 2023-06-13 Elementary Robotics, Inc. Cloud-based multi-camera quality assurance architecture
US20230298189A1 (en) * 2020-07-28 2023-09-21 Shenzhen Institutes Of Advanced Technology Chinese Academy Of Sciences Method for reconstructing three-dimensional object combining structured light and photometry and terminal device
US11937019B2 (en) 2021-06-07 2024-03-19 Elementary Robotics, Inc. Intelligent quality assurance and inspection device having multiple camera modules
US12051186B2 (en) 2021-11-03 2024-07-30 Elementary Robotics, Inc. Automatic object detection and changeover for quality assurance inspection
US12050454B2 (en) * 2021-11-10 2024-07-30 Elementary Robotics, Inc. Cloud-based multi-camera quality assurance lifecycle architecture
CN119379736A (zh) * 2024-10-08 2025-01-28 重庆移通学院 基于模式识别的目标跟踪方法、系统、设备及存储介质
CN119880135A (zh) * 2024-12-26 2025-04-25 聚时科技(上海)有限公司 一种光源辐射照度测量方法、辐射效率测量方法及系统
CN120689407A (zh) * 2025-06-04 2025-09-23 江苏盛邦印务包装有限公司 一种基于图像分析的烟包卡纸的定位方法
US12518391B2 (en) * 2022-02-09 2026-01-06 eBots Inc. System and method for improving image segmentation

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130324875A1 (en) * 2012-06-01 2013-12-05 Xerox Corporation Processing a video for respiration rate estimation
US20140028800A1 (en) * 2012-07-30 2014-01-30 Canon Kabushiki Kaisha Multispectral Binary Coded Projection
WO2015077493A1 (fr) * 2013-11-20 2015-05-28 Digimarc Corporation Imagerie lumineuse à synchronisation par capteur et structuration spectrale
US20160173855A1 (en) * 2014-12-16 2016-06-16 Commissariat A L'energie Atomique Et Aux Energies Alternatives Structured-Light Projector And Three-Dimensional Scanner Comprising Such A Projector
US20190011705A1 (en) * 2017-07-05 2019-01-10 Microsoft Technology Licensing, Llc Compact optical system with mems scanners for image generation and object tracking

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130324875A1 (en) * 2012-06-01 2013-12-05 Xerox Corporation Processing a video for respiration rate estimation
US20140028800A1 (en) * 2012-07-30 2014-01-30 Canon Kabushiki Kaisha Multispectral Binary Coded Projection
WO2015077493A1 (fr) * 2013-11-20 2015-05-28 Digimarc Corporation Imagerie lumineuse à synchronisation par capteur et structuration spectrale
US20160173855A1 (en) * 2014-12-16 2016-06-16 Commissariat A L'energie Atomique Et Aux Energies Alternatives Structured-Light Projector And Three-Dimensional Scanner Comprising Such A Projector
US20190011705A1 (en) * 2017-07-05 2019-01-10 Microsoft Technology Licensing, Llc Compact optical system with mems scanners for image generation and object tracking

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KIM M ET AL: "3D Imaging Spectroscopy for Measuring 3D Hyperspectral Patterns on Solid Objects", ACM TRANSACTIONS ON GRAPHICS, ACM, NY, US, vol. 31, no. 4, 1 July 2012 (2012-07-01), pages 1 - 11, XP002790241, ISSN: 0730-0301, DOI: 10.1145/2185520.2185534 *

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230298189A1 (en) * 2020-07-28 2023-09-21 Shenzhen Institutes Of Advanced Technology Chinese Academy Of Sciences Method for reconstructing three-dimensional object combining structured light and photometry and terminal device
US12586223B2 (en) * 2020-07-28 2026-03-24 Shenzhen Institutes Of Advanced Technology Chinese Academy Of Sciences Method for reconstructing three-dimensional object combining structured light and photometry and terminal device
CN113705288A (zh) * 2020-11-25 2021-11-26 天翼智慧家庭科技有限公司 一种基于3d人脸重建的公众人物识别系统和方法
CN112669362A (zh) * 2021-01-12 2021-04-16 四川深瑞视科技有限公司 基于散斑的深度信息获取方法、装置及系统
CN112669362B (zh) * 2021-01-12 2024-03-29 四川深瑞视科技有限公司 基于散斑的深度信息获取方法、装置及系统
CN113188450A (zh) * 2021-04-23 2021-07-30 封泽希 基于结构光的场景深度检测方法及系统
CN113188450B (zh) * 2021-04-23 2023-03-14 封泽希 基于结构光的场景深度检测方法及系统
US11937019B2 (en) 2021-06-07 2024-03-19 Elementary Robotics, Inc. Intelligent quality assurance and inspection device having multiple camera modules
CN113609907A (zh) * 2021-07-01 2021-11-05 奥比中光科技集团股份有限公司 一种多光谱数据的获取方法、装置及设备
CN113609907B (zh) * 2021-07-01 2024-03-12 奥比中光科技集团股份有限公司 一种多光谱数据的获取方法、装置及设备
CN113506348A (zh) * 2021-07-15 2021-10-15 吉林省凯迪科技有限公司 一种基于格雷码辅助的三维坐标计算方法
CN113506348B (zh) * 2021-07-15 2023-02-28 吉林省凯迪科技有限公司 一种基于格雷码辅助的三维坐标计算方法
US12051186B2 (en) 2021-11-03 2024-07-30 Elementary Robotics, Inc. Automatic object detection and changeover for quality assurance inspection
US11605159B1 (en) 2021-11-03 2023-03-14 Elementary Robotics, Inc. Computationally efficient quality assurance inspection processes using machine learning
US11675345B2 (en) 2021-11-10 2023-06-13 Elementary Robotics, Inc. Cloud-based multi-camera quality assurance architecture
US12050454B2 (en) * 2021-11-10 2024-07-30 Elementary Robotics, Inc. Cloud-based multi-camera quality assurance lifecycle architecture
CN114095715A (zh) * 2021-11-18 2022-02-25 中国科学院长春光学精密机械与物理研究所 一种动态目标的结构光扫描成像方法及其成像装置
CN113959961B (zh) * 2021-12-22 2022-03-08 广东省农业科学院动物科学研究所 一种基于高光谱图像的单宁添加剂防伪检测方法及系统
CN113959961A (zh) * 2021-12-22 2022-01-21 广东省农业科学院动物科学研究所 一种基于高光谱图像的单宁添加剂防伪检测方法及系统
US12518391B2 (en) * 2022-02-09 2026-01-06 eBots Inc. System and method for improving image segmentation
US11605216B1 (en) 2022-02-10 2023-03-14 Elementary Robotics, Inc. Intelligent automated image clustering for quality assurance
CN115585755A (zh) * 2022-10-20 2023-01-10 大连理工大学 一种基于伪随机二进制序列编码投影的平直度检测方法
CN119379736B (zh) * 2024-10-08 2025-08-22 重庆移通学院 基于模式识别的目标跟踪方法、系统、设备及存储介质
CN119379736A (zh) * 2024-10-08 2025-01-28 重庆移通学院 基于模式识别的目标跟踪方法、系统、设备及存储介质
CN119880135A (zh) * 2024-12-26 2025-04-25 聚时科技(上海)有限公司 一种光源辐射照度测量方法、辐射效率测量方法及系统
CN120689407A (zh) * 2025-06-04 2025-09-23 江苏盛邦印务包装有限公司 一种基于图像分析的烟包卡纸的定位方法

Similar Documents

Publication Publication Date Title
WO2020168094A1 (fr) Mesure simultanée de profil de profondeur et de spectre
US10788318B2 (en) Three-dimensional shape measurement apparatus
KR102048793B1 (ko) 표면 컬러를 이용한 표면 토포그래피 간섭측정계
US8649024B2 (en) Non-contact surface characterization using modulated illumination
CN112798617A (zh) 类镜面物体的缺陷检测装置及其检测方法
JP7413293B2 (ja) 複数タイプの光を用いるビジョン検査のためのシステム及び方法
KR20120088773A (ko) 검사 장치, 3차원 형상 측정 장치, 구조물의 제조 방법
WO2012125706A2 (fr) Système de mesure de forme 3d en temps réel
JP4090860B2 (ja) ３次元形状測定装置
WO2013105922A2 (fr) Caractérisation de surface sans contact utilisant un éclairage modulé
JP6918395B1 (ja) 撮像装置
CN107655421A (zh) 采用立体扫描相机对表面进行扫描的工艺和装置
US9042963B2 (en) System and method for acquiring images from within a tissue
US12601589B2 (en) Measuring system providing shape from shading
WO2009141838A1 (fr) Procédé de mesure sans contact d’objets à forme de surface, notamment pour des parties d’arcade dentaire ou de parties dentaires
JP4756148B2 (ja) 光沢・色再現システム、光沢・色再現プログラム
WO2024009454A1 (fr) Appareil d'inspection
CN111750799A (zh) 基于干涉照明的光谱偏振形貌五维信息测量装置及方法
WO2019238583A1 (fr) Techniques de déflectométrie
JP7574000B2 (ja) 鏡面ヘテロダイン光学位相測定装置
EP4389001A1 (fr) Alignement d'images acquises simultanément
US20040047517A1 (en) Shadow-free 3D and 2D measurement system and method
CN120831360B (zh) 一种基于线扫成像的2.5d缺陷检测方法及装置
JP2024043331A (ja) 光学検査プログラム、光学検査システム、光学検査システムのための処理装置、及び、光学検査方法
JP6860090B2 (ja) 分光データ管理装置、測定装置および測定システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20714328

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20714328

Country of ref document: EP

Kind code of ref document: A1