Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
  • G01B11/00—Measuring arrangements characterised by the use of optical techniques
  • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
  • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
  • G01B11/2545—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with one projection direction and several detection directions, e.g. stereo
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
  • G01B11/00—Measuring arrangements characterised by the use of optical techniques
  • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
  • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
  • G01B11/2513—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with several lines being projected in more than one direction, e.g. grids, patterns
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
  • G02B7/28—Systems for automatic generation of focusing signals
  • G02B7/36—Systems for automatic generation of focusing signals using image sharpness techniques, e.g. image processing techniques for generating autofocus signals

Definitions

• defocusing is one in which large data structures are generated by light capture with a CMOS, CCD or other imaging apparatus through restricted areas positioned at different radial locations along a common optical axis.
• defocusing is based on computing depth from the predictable way images go out-of-focus when an object is imaged off of the focal plane.
• the shift of matched-up image point features or patterns imaged through the offset apertures is used to measure depth.
• US Publication No. 2009/0295908 discloses a defocusing imaging system modified to use a projected pattern of laser dots specifically to acquire dense point cloud profilometry (i.e., 3D profile measurement) data for the observed object.
• Use of these laser point dot "markers" permit relatively high resolution imaging of a surface as the light is reflected from the object to be imaged, captured on a sensor, matched-up between apertures, and used to make z-axis determinations.
• Different color-coded apertures are employed to code the reflected light and make defocusing determinations for image aggregation.
• a projector projects an optical pattern toward a surface.
• a camera with at least two off-axis apertures is arranged to obtain an image of the projected pattern with the defocused information.
• the camera is movable between different positions to image the surface from the different positions and the projector is set at a specified angle of at least 5 degrees relative to the camera.
• a processor carries out a first operation using information received through the apertures to determine a pose of the camera and to determine three dimensional information about the object based on a degree of deformation of the optical pattern on the imaged surface.
• USPN 7,916,309 discloses various defocusing system options in which a light
• projection system may be provided to project a predetermined pattern onto a surface of the desired object to allow mapping of unmarked surfaces in three dimensions.
• the pattern is shown in the form of a (regular/consistent) grid with a distinguishable center point. As stated, however, the pattern need not be symmetrical or contain a regular sequence of markers or images.
• suitable addressable-pattern information which are a color sequence pattern, a pattern comprising differently shaped object, a position sequence pattern, distinguishable object features or object landmarks, or any combination thereof. Yet, whatever the properties of the projected pattern, in use it is static with respect to the contours of the imaged object.
• the camera moves and takes images from multiple viewpoints, each containing at least a portion of the addressable pattern such that an address can be assigned to each point in the pattern.
• a prerequisite for the system is that the patterns of captured light from each viewpoint are physically separate.
• light through the apertures must be separated by prisms, fiber optics, or color filters so as to record separate image information from each viewpoint.
• Recognizing image triplets vs. doublets can offer certain advantages in matching points.
• Triplet, quadruplet, etc. grouping is more distinctive than that of two points defining a line. Also, inclusion of more pattern points enables the averaging out of errors when locating the points.
• computational intensity increases, as well as the prospect for image crowding as discussed above.
• each recorded point for a given sensor area is known to correspond to a given aperture thereby greatly assisting in point match-up without interference. Such distinction also assists in point/feature matching and, thus, defocusing as a process.
• none of these achieve the benefits of reduced image crowding without some sort of optical tract modification or severe functional compromise (as in the static projection example).
• inventions described herein offer image-crowding solutions without such complications or limitations. Especially in connection with object scanning to measure and/or define surface topography, these embodiments offer marked advantages, such as the potential for dramatically increased point density acquisition relative to known systems. Other advantages are provided, or will become apparent to those of ordinary skill in the art, in the description set forth herein.
• the present inventions offer improvement to known defocusing systems, in essence, by embedding information in a projected light pattern sent to the object to be imaged.
• the imager is mobile, so too is the projector associated therewith. Otherwise, the camera and the projector may be stationary and the imaged object moved.
• the projected information is not static with respect to the object being imaged or tracked, and may be transmitted as reflected light through a plurality of un-coded apertures and received on a sensor with the light patterns from each aperture overlapping the other.
• the pattern includes different color points; in another embodiment, the pattern includes different intensity points; in yet another, different size points; and in still another, different shaped points/dots may be used. Likewise, a combination of such differentiation may be employed.
• the elements of the pattern referred to as points, dots, markers, features, etc.
• the "different-color” light may be in and/or out of the visible range of frequencies.
• the different intensity light may be visible, but may more advantageously be in the IR or near-IR spectrum so that it will go unnoticed by a user.
• the patterns will differ slightly due to aperture offset from the optical axis. And even though the patterns may overlap (when images received through a given aperture are not routed to a dedicated sensor or sensor portions), it can be determined which portion of the overall captured signal is provided by one pattern through a first aperture and another pattern through a second aperture, etc.
• the selection is a compromise between complexity (system and computational complexity) and the spatial resolution the system can achieve (i.e., how tight a point mesh or how short a nodal distance can be achieved between points defining a surface).
• Providing a greater number of intensities, colors, sizes, shapes, combinations, etc., allows subdivision of the space between each feature into finer granulation.
• greater control is required for the structured lighting, better resolution between image intensities, colors, etc., more computational power for resolving 3D data, etc.
• the variation in a differentiating feature provides information about which imaged dots within the pattern should be matched by the computer. This information is what reduces the crowding problem noted above. As a secondary effect of reduced crowding, the front-to-back "range" of depth that can be determined may also increase.
• a known challenge in practicing defocusing concerns the ability to determine which of multiple simultaneously imaged points captured on an imager correspond to one another. For defocusing to be possible, matches must be made. As such, the additional information of the projected patterns can be of assistance. For example, in the case where it is known that a green marker should be to the right of a red one, then the program can ignore nearby red or blue dots in making matches.
• the color information offered with a colored projected pattern may be resolved using a typical color image sensor (e.g., a CMOS or CCD unit incorporating a Bayer filter). For signals of different intensity, a CMOS or CCD without any filter may be preferred.
• a typical color image sensor e.g., a CMOS or CCD unit incorporating a Bayer filter.
• a CMOS or CCD without any filter may be preferred.
• any of the single-class differentiation methods i.e., by color, intensity, size, shape, polarization
• points will be known overlaps and the information can be used in more complex computer processing.
• the various advantages of the subject pattern matching to avoid the above-referenced "ghost" particles may generally be more important. Mismatched points or particles can result in large errors, whereas image overlap generally results in fewer image points for data.
• the matching advantages can be employed as described above for complex three- dimensional (or four-dimensional tracking involving object movement in time) surface mapping calculations and associated data structure output. However, the advantages can be used as effectively in simpler systems as well. In any event, the advantages available from the teachings herein allow for precise object mapping and tracking. As such, it can capture fine hand gestures, enabling a new range of applications. For such purposes, it may be preferred to use IR projection so as not to distract the user. In addition, a display may picture "virtual" tools, handle interfaces, etc., moving in coordination with the captured hand gestures.
• the hardware includes computer electronics.
• the hardware further includes an optical system with one or more sensors as described above provided in an arrangement suitable for defocusing imaging.
• the apertures may be filtered with the same "color" to eliminate signal noise from other reflected light without introducing aperture-to-aperture differentiation.
• a projection system is provided. It is adapted to project a pattern of different size, intensity, shape and/or color light markers.
• the grid pattern is regular in order to achieve maximized (for a given application) spatial resolution - though not necessarily so.
• the pattern optionally, and advantageously, includes no individually distinguishable point(s) - as in an addressable center point per the '309 patent.
• a laser may be used in the embodiments , as well as an LED or any other kind of other light generator for the light source.
• the light patterns may be provided by multiplying these entities or using beam splitters, holographic lenses, or diffraction gratings (i.e., as in a diffractive optical element (DOE)).
• DOE diffractive optical element
• the illumination and the sensor need not be on the same side of the surface such as, for example, if measuring on a semi-opaque film. It should also be clear, from those embodiments where the depth is extracted from the position of the projected dots, that the detector need not be concentric with the central axis of the illumination as any offset can be compensated for in the initial calibration. Aspects of the inventions include the subject hardware, software, and related methods of operation and manufacture.
• FIGs. 1 , 2, 3A, and 4 illustrate imaging system embodiments employing the subject patterns including differentiated elements within the same;
• Fig. 3B illustrates example variations in the shape of a point in a pattern; and
• Fig. 5 is a flowchart concerning imaged point matching.
• Fig. 1 depicts a structured light projection 100 that is a regular grid with a regular variation in the size of the points.
• the known variation in light point size can be used to determine which same-size points correspond to each other in the two (or more) cameras (such determination of same-type point pairs at different locations within the pattern indicated by example pointer pairs ⁇ , ⁇ , ⁇ ).
• the distance between thusly-matched points can be used to measure depth through defocusing.
• the hardware employed may include cameras 120 (or representative camera portions of a single camera) a projector 130 including its light source(s) - of any type as noted above or as otherwise implemented - for creating the pattern. Any such hardware may be housed separately or grouped together as indicated with dashed line including with a computing device 140.
• the computing device can be a microprocessor and graphics processing unit (GPU) included with a video card for optional output display 150, operating according to a stored program instructions in connection with digital storage media 160. (Various internal component connections now shown.)
• GPU graphics processing unit
• the reflected images captured on the imager are not color or otherwise filtered or time-wise separated to alter the incoming signal with respect to each other.
• the only differences intended in the captured light passed through associated len(s) and aperture(s) to the sensor(s) derive from differences in viewpoint (e.g., as determined by offset aperture position along a common optical axis) recorded on the sensor(s).
• the captured light elements of the pattern differ only (i.e., "only" in any significant sense where image processing is possible) in their position in an absolute sense and/or with the given pattern.
• it is contemplate that some elements of the pattern may not be recognizable or are otherwise corrupted by incomplete reflection, etc.
• the different offset images captured will be superimposed on the camera sensor.
• the captured light patterns may be combined if properly registered. The same holds true for examples with altogether separate lens arrangements that share a common optical axis.
• such hardware is suitable for capturing a plurality of offset images of reflected light patterns from an object to provide image data in which the captured images differ only in element position.
• an embodiment may be described as comprising at least one
• a camera imaging reflected light from a pattern provided by a projector directed at an object, the light enters the at least one camera passing through at least one lens and a plurality of offset restricted regions as may be provided by apertures offset from a common optical axis.
• the entire sensor or a portion of a sensor within each such camera captures one or more offset images.
• Elements within the projected patern are distinguishable from one another and these are matched between separate images from different cameras that may be combined or laid over images captured in connection with a single camera.
• a structured light projection 102 having a grid with regular variation in the intensity of points is shown (the different intensities being indicated by gray-scale or per formal drawing standards - stippling densities).
• a structured light projection 104 that is a regular grid with a regular variation in the shape of points is shown (exemplary shapes of "+” "x” and “o” being employed). However, any recognizable variation in individual element shape may be employed such as those shown in Fig. 3B.
• Fig. 4 depicts use of a structured light projection 106 that is a regular grid with a regular variation in the color of points (the different colors indicated as red, blue and green - advantageously selected for any coordinated use with an off the shelf BAYER filter sensor or - or by different hatching per formal drawing standards).
• each such pattern 100, 102, 104 and 106 three differentiatable elements are used in a regular pattern or in a repeating sequence, in repeating lines. In each successive line (as viewed vertically or horizontally) within the grid pattern, the elements are indexed by one position.
• the pattern may include additional colors, intensities, sizes and/or shapes for distinction within the pattern to the three-distinct-element embodiments pictured. It may alternatively include only two such elements in the projected pattern. In which case, a polarization-based variation with a custom polarizing filter associated with the sensor may be employed in spite of know difficulties that can be encountered with changes in polarity of reflected light.
• a method of point matching and continuation of the method for defocusing- based operation is shown.
• imaging is performed, optionally with the noted hardware. This may produce a combined scene on a single imager or separate scenes for different cameras.
• a distinguishable feature e.g., imaged color point
• the program looks to comparable same- type featured points in the same scene (e.g., based on a triplet relationship) or a different scene captured in connection with a different camera or portion (e.g., based on the relation provided by a calibration set). Matching points are thus provided at 230.
• the process may repeat as indicated by the dashed line.
• any further processing for defocusing-based depth finding for a given match occurs at 300.
• Image aggregation from multiple scenes taken with different camera pose or position may be aggregated at 310 (e.g., as per below).
• Final output of surface data or control signals for another unit such as a gaming machine, etc. are output at 320.
• the process stops after point matching and defocusing depth determination.
• any or all of the depth determination for a given imaged "scene" may be aggregated with other related image scene data taken for an object larger than the imager field of view.
• teachings for approaches to calculating camera pose, transforming and aggregating image data as presented in PCT/US10/57532 may be applied or others as may be apparent to those with skill in the art.
• the cameras described herein can be handheld portable units, or machine vision cameras, or underwater units. Or the camera may be mounted in a stationary position with an object moved relative to them or otherwise configured. Still further, the camera may be worn by a user to record facial expressions or gestures to be blended with animation. Other possibilities exist as well such as noted in the Summary above. [0048] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. Indeed, given the type of pixel-to-pixel matching for imaged points and associated calculations required with the data structures recorded and manipulated, computer use is necessary. In imaging any object, vast sets of data are collected and stored in a data structure requiring significant manipulation in accordance with imaging principles - including defocusing principles/equations - as noted herein and as incorporated by reference.
• DSP Digital Signal Processor
• ASIC Application Specific Integrated Circuit
• FPGA Field Programmable Gate Array
• a general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
• the processor can be part of a computer system that also has a user interface port that communicates with a user interface, and which receives commands entered by a user, has at least one memory (e.g., hard drive or other comparable storage, and random access memory) that stores electronic information including a program that operates under control of the processor and with communication via the user interface port, and a video output that produces its output via any kind of video output format, e.g., VGA, DVI, HDMI, display port, or any other form.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. These devices may also be used to select values for devices as described herein.
• a software module may reside in Random Access Memory (RAM), flash memory, Read Only Memory (ROM), Electrically
• An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium.
• the storage medium may be integral to the processor.
• the processor and the storage medium may reside in an ASIC.
• the ASIC may reside in a user terminal.
• the processor and the storage medium may reside as discrete components in a user terminal.
• Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
• a storage media may be any available media that can be accessed by a computer.
• such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
• the memory storage can also be rotating magnetic hard disk drives, optical disk drives, or flash memory based storage drives or other such solid state, magnetic, or optical storage devices.
• any connection is properly termed a computer-readable medium.
• the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium.
• DSL digital subscriber line
• Disk and disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
• the computer readable media can be an article comprising a machine-readable non-transitory tangible medium embodying information indicative of instructions that when performed by one or more machines result in computer implemented operations comprising the actions described throughout this specification.
• Operations as described herein can be carried out on or over a website.
• the website can be operated on a server computer, or operated locally, e.g., by being downloaded to the client computer, or operated via a server farm.
• the website can be accessed over a mobile phone or a PDA, or on any other client.
• the website can use HTML code in any form, e.g., MHTML, or XML, and via any form such as cascading style sheets (“CSS”) or other.
• the computers described herein may be any kind of computer, either general purpose, or some specific purpose computer such as a workstation.
• the programs may be written in C, or Java, Brew or any other programming language.
• the programs may be resident on a storage medium, e.g., magnetic or optical, e.g. the computer hard drive, a removable disk or media such as a memory stick or SD media, or other removable medium.
• the programs may also be run over a network, for example, with a server or other machine sending signals to the local machine, which allows the local machine to carry out the operations described herein.

Landscapes

• Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• General Physics & Mathematics (AREA)
• Optics & Photonics (AREA)
• Length Measuring Devices By Optical Means (AREA)

Priority Applications (1)

Applications Claiming Priority (1)

Publications (1)

Family

ID=49916442

Family Applications (1)

Country Status (1)

Cited By (6)

Citations (4)

• 2012
  • 2012-07-12 WO PCT/US2012/046484 patent/WO2014011179A1/fr not_active Ceased

Patent Citations (4)

Similar Documents

Legal Events