WO2017063435A1 - 一种组合深度图获得方法及深度相机 - Google Patents

一种组合深度图获得方法及深度相机 Download PDF

Info

Publication number
WO2017063435A1
WO2017063435A1 PCT/CN2016/093546 CN2016093546W WO2017063435A1 WO 2017063435 A1 WO2017063435 A1 WO 2017063435A1 CN 2016093546 W CN2016093546 W CN 2016093546W WO 2017063435 A1 WO2017063435 A1 WO 2017063435A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
tof
data
camera
processor
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/093546
Other languages
English (en)
French (fr)
Inventor
王靖雄
毛慧
沈林杰
俞海
浦世亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hikvision Digital Technology Co Ltd
Original Assignee
Hangzhou Hikvision Digital Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hikvision Digital Technology Co Ltd filed Critical Hangzhou Hikvision Digital Technology Co Ltd
Priority to US15/767,607 priority Critical patent/US10713804B2/en
Priority to EP16854817.0A priority patent/EP3364643B1/en
Publication of WO2017063435A1 publication Critical patent/WO2017063435A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/87—Combinations of systems using electromagnetic waves other than radio waves
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G01S17/894—Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10141—Special mode during image acquisition
    • G06T2207/10152—Varying illumination
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present application relates to the field of video surveillance technologies, and in particular, to a combined depth map obtaining method and a depth camera.
  • a depth camera has the same resolution as a normal camera, but its pixel points correspond to the distance from the object corresponding to the pixel to the camera. It can be called depth, that is, the depth camera outputs a depth map.
  • the value of a pixel represents the distance from the pixel to the camera.
  • a depth camera using a ToF (Time of Flight) sensor uses the ToF technique to output the depth map.
  • the ToF technology measures the distance between an object that reflects a light signal and the transmitting and receiving ends by transmitting and receiving a modulated optical signal and analyzing the time difference between transmitting and receiving the optical signal.
  • the ToF sensor is an optical signal receiving and analyzing component in the practical application of the ToF technology, which cooperates with a modulated light emitting component to achieve depth measurement.
  • the output of the ToF sensor can be obtained by a certain mathematical transformation to obtain a depth map.
  • a single ToF sensor's depth camera is generally small in its ability to measure depth, and may not be able to meet the needs of applications that require large field of view depth maps.
  • An object of the embodiments of the present application is to provide a combined depth map obtaining method and a depth camera to solve the problem of mutual interference between depth cameras of a plurality of single ToF sensors.
  • the technical solutions are as follows:
  • the present application provides a depth camera, including: a processor, at least one a light emitting element and at least two time-of-flight ranging ToF sensors, the superimposed illumination range of the at least one light-emitting element covering a superimposed field of view of the at least two ToF sensors, wherein
  • the processor is configured to generate a modulated signal and a demodulated signal, and output the modulated signal to each of the light emitting elements, output the demodulated signal to each ToF sensor; and receive depth data input by each ToF sensor And performing data fusion processing on all the received depth data to generate combined depth data; and obtaining a combined depth map according to the combined depth data;
  • Each of the light-emitting elements is configured to receive a modulation signal input by the processor; use the modulation signal to modulate its own optical signal, and emit a modulated light signal to an object within its own illumination range;
  • Each ToF sensor is configured to receive a demodulated signal input by the processor; receive a modulated optical signal reflected by an object within a range of its own field of view; and demodulate the received optical signal using the demodulated signal, Generating depth data; outputting the generated depth data to the processor.
  • the at least two ToF sensors are arranged in a sensor matrix, the sensor matrix comprising at least one row and at least one column;
  • the geometric center of the target surface of each ToF sensor in the row is in a straight line, and the distance between two adjacent ToF sensors in the row meets the preset first distance requirement.
  • the angle between the target faces of two adjacent ToF sensors in the row satisfies a preset angle requirement;
  • the geometric center of the target surface of each ToF sensor in the column is located on a straight line, and the distance between two adjacent ToF sensors in the column meets the preset second distance requirement.
  • the target faces of two adjacent ToF sensors in the column are coplanar or parallel to each other.
  • the target faces of two adjacent ToF sensors in each row of the sensor matrix are coplanar or parallel to each other.
  • the at least two ToF sensors are arranged on a preset spherical surface, and different ToF sensors have different positions on the preset spherical surface.
  • the processor includes: at least one field programmable gate array FPGA and a ToF controller TFC corresponding to each ToF sensor, where
  • the FPGA is configured to receive depth data of each TFC input, and perform data fusion processing on all received depth data to generate combined depth data; according to the combined depth data, obtain Combine depth maps;
  • the first TFC of the TFC included in the processor is configured to generate a modulated signal and a demodulated signal, and output the modulated signal to each of the light emitting elements, and output the demodulated signal to each ToF sensor;
  • Each TFC in the TFC included by the processor is configured to receive depth data of the ToF sensor input corresponding thereto, and output the received depth data to the FPGA.
  • the method further includes: a color camera, the field of view of the color camera covering a field of view superimposed by the at least two ToF sensors;
  • the color camera is configured to collect color data, and output the collected color data to the processor;
  • the processor is further configured to receive color data input by the color camera, and align the received color data with the combined depth data to obtain a color image that incorporates depth information.
  • the present application provides a combined depth map obtaining method for a processor in a depth camera, the depth camera including the processor, at least one light emitting element, and at least two time-of-flight ranging ToF sensors,
  • the superimposed illumination range of the at least one illuminating element covers a superimposed field of view of the at least two ToF sensors, the method comprising:
  • each ToF sensor For each ToF sensor, receiving the depth data obtained by the ToF sensor using the demodulated signal to demodulate the modulated optical signal reflected by the object in the field of view, wherein the modulated optical signal has The illuminating element of the illumination range corresponding to the field of view of the ToF sensor is modulated by using the modulation signal to modulate its own optical signal;
  • a combined depth map is obtained based on the combined depth data.
  • the data fusion processing is performed on all the received depth data to generate combined depth data, including:
  • the depth value obtained by the ToF sensor corresponding to the highest confidence value is used as the pixel point according to the confidence level for the pixel point in each ToF sensor corresponding to the pixel point.
  • the combined depth data is generated based on the determined depth value of each pixel.
  • the data fusion processing is performed on all the received depth data to generate combined depth data, including:
  • the combined depth data is generated based on the determined depth value of each pixel.
  • the depth camera further includes a color camera, and a field of view of the color camera covers a field of view superimposed by the at least two ToF sensors, the method further includes:
  • the received color data is aligned with the combined depth data, and a color image in which the depth information is blended is output.
  • the depth camera provided by the embodiment of the present application includes a processor, at least one light emitting component, and at least two ToF sensors, which can meet the requirements of a large field of view range depth map application.
  • the light-emitting elements in the depth camera use the same modulation signal to modulate and transmit the optical signal, and the ToF sensor demodulates the modulated optical signal reflected by the received object using the same demodulated signal corresponding to the modulated signal to generate depth data. It can avoid the problem of mutual interference due to the unsynchronized modulation and demodulation.
  • the processor performs data fusion processing on all the received depth data to obtain a combined depth map.
  • FIG. 1 is a schematic structural diagram of a depth camera according to an embodiment of the present application.
  • FIG. 2 is a schematic diagram of a layout of a ToF sensor according to an embodiment of the present application.
  • FIG. 3 is a schematic diagram of another arrangement manner of a ToF sensor according to an embodiment of the present application.
  • FIG. 4 is a schematic diagram of another arrangement manner of a ToF sensor according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of another arrangement manner of a ToF sensor according to an embodiment of the present application.
  • FIG. 6 is another schematic structural diagram of a depth camera according to an embodiment of the present application.
  • FIG. 7 is another schematic structural diagram of a depth camera according to an embodiment of the present application.
  • FIG. 8 is a flowchart of an implementation of a method for obtaining a combined depth map according to an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a position of a real depth camera and a virtual depth camera in the embodiment of the present application.
  • FIG. 10 is a schematic diagram of a projection relationship of a spatial point P in a real depth camera and a virtual depth camera according to an embodiment of the present application.
  • FIG. 11 is another schematic diagram of the projection relationship of the spatial point P in the real depth camera and the virtual depth camera in the embodiment of the present application.
  • the depth camera includes: a processor, at least one illuminating component, and at least two time-of-flight ranging ToF sensors, the at least one a superimposed illumination range of the illuminating elements covering a superimposed field of view of the at least two ToF sensors, wherein
  • the processor is configured to generate a modulated signal and a demodulated signal, and output the modulated signal to each of the light emitting elements, output the demodulated signal to each ToF sensor; and receive depth data input by each ToF sensor And performing data fusion processing on all the received depth data to generate combined depth data; and obtaining a combined depth map according to the combined depth data;
  • Each of the light-emitting elements is configured to receive a modulation signal input by the processor; use the modulation signal to modulate its own optical signal, and emit a modulated light signal to an object within its own illumination range;
  • Each ToF sensor is configured to receive a demodulated signal input by the processor; receive a modulated optical signal reflected by an object within a range of its own field of view; and demodulate the received optical signal using the demodulated signal, Generating depth data; outputting the generated depth data to the processor.
  • the depth camera provided by the embodiment of the present application includes at least two ToF sensors, and the spatial arrangement of the ToF sensors allows them to face different fields of view, respectively, and each field of view region can be combined to form a larger field of view region. .
  • the number and arrangement of ToF sensors and illuminating elements can be determined according to the actual field of view range, as long as the superimposed illumination range of all illuminating elements in the depth camera can cover the superimposed field of view of all ToF sensors. Just fine.
  • the processor in the depth camera provided by the embodiment of the present application can obtain an external control signal through an input interface of the depth camera, and the external control signal can be input by an operator for setting and adjusting corresponding parameters of the depth camera, for example, for setting Or adjust the exposure time of the depth camera, etc.
  • the processor may generate a modulation signal and a demodulation signal according to the external control signal, and the modulation signal corresponds to the demodulation signal.
  • the processor sends a modulated signal to each of the light-emitting elements, and transmits the demodulated signal to each of the ToF sensors, that is, all of the light-emitting elements in the depth camera use the same modulated signal, and all of the ToF sensors in the depth camera
  • the demodulated signals used are the same to achieve modulation and demodulation synchronization.
  • the light-emitting element After receiving the modulated signal sent by the processor, the light-emitting element modulates its own optical signal and transmits the modulated optical signal.
  • An object within the illumination range of each of the light-emitting elements reflects the modulated light signal of the light-emitting element and is received by a ToF sensor having a field of view corresponding to the illumination range.
  • the ToF sensor After receiving the modulated optical signal reflected by the object in the field of view, the ToF sensor demodulates the received modulated optical signal using the demodulated signal to generate depth data. Generated by each ToF sensor The depth data is the depth value of each pixel point that the ToF sensor acquires in its field of view range region. Each ToF sensor outputs the generated depth data to the processor.
  • the processor After receiving the depth data input by each ToF sensor, the processor performs data fusion processing on all the received depth data, and selects an appropriate original point for mapping the data in the superimposed field of view of the ToF sensor to generate a combined depth. Data, and according to the combined depth data, a combined depth map is obtained.
  • the processor can output the obtained combined depth map through the output interface of the depth camera as an output of the depth camera.
  • the processor may also generate a control signal for the ToF sensor according to the received external control signal, and issue corresponding parameters for the ToF sensor to perform corresponding control on the ToF sensor.
  • the depth camera provided by the embodiment of the present application includes a processor, at least one light emitting component, and at least two ToF sensors, which can meet the application requirements of the large field of view range depth map.
  • the light-emitting elements in the depth camera use the same modulation signal to modulate and transmit the optical signal, and the ToF sensor demodulates the modulated optical signal reflected by the received object using the same demodulated signal corresponding to the modulated signal to generate depth data. It can avoid the problem of mutual interference due to the unsynchronized modulation and demodulation.
  • the processor performs data fusion processing on all the received depth data to obtain a combined depth map.
  • the at least two ToF sensors may be arranged in a sensor matrix, the sensor matrix comprising at least one row and at least one column;
  • the geometric center of the target surface of each ToF sensor in the row is in a straight line, and the distance between two adjacent ToF sensors in the row meets the preset first distance requirement.
  • the angle between the target faces of two adjacent ToF sensors in the row satisfies a preset angle requirement;
  • the geometric center of the target surface of each ToF sensor in the column is located on a straight line, and the distance between two adjacent ToF sensors in the column meets the preset second distance requirement.
  • the target faces of two adjacent ToF sensors in the column are coplanar or parallel to each other.
  • the spatial arrangement of the ToF sensor can be determined according to actual conditions.
  • the following is a few examples to illustrate the arrangement of at least two ToF sensors included in the depth camera.
  • the angle between the target faces of two adjacent ToF sensors in the above example 1 may be 0, that is, the target faces of two adjacent ToF sensors are coplanar or parallel to each other, as shown in FIG. 3 .
  • the specific arrangement manner of the ToF sensors in each row may be consistent with the above example 1.
  • the geometric center of the target surface of each ToF sensor in the column is in a straight line, and the distance between two adjacent ToF sensors in the column needs to meet the preset second distance requirement.
  • the target faces of two adjacent ToF sensors in the column are coplanar or parallel to each other, as shown in FIG. 4 .
  • the specific arrangement manner of the ToF sensors in each row may be consistent with the above example 2
  • each The specific arrangement of the ToF sensors in the column may be consistent with the specific arrangement of the ToF sensors in each column of the above example 3, as shown in FIG. 5.
  • the first distance requirement preset in the embodiment of the present application, the preset second distance requirement, and the preset angle requirement may be set and adjusted according to actual conditions, such as according to an actual field of view.
  • the requirements of the area are set and adjusted, and the embodiment of the present application does not limit this.
  • the at least two ToF sensors are arranged on a preset spherical surface, and the positions of different ToF sensors on the preset spherical surface are different, and the actual required range of the field of view can be satisfied. .
  • the ToF sensor is arranged in the depth camera to install the ToF sensor.
  • the light-emitting component needs to be installed according to the superimposed field of view of the ToF sensor, so that the emitted light of the light-emitting component can cover the superimposed view of the ToF sensor. Field range.
  • the specific arrangement of the ToF sensor is not limited to the above-exemplified ones. A person skilled in the art can deduce other different examples according to the examples exemplified above, and the embodiments of the present application will not be described again.
  • the processor in the depth camera may include a control chip to realize a plurality of ToF sensors sharing a demodulation signal, and a plurality of light-emitting elements sharing a modulation signal, and the control chip generates a modulation signal of the light-emitting element according to the ToF technology principle.
  • the emitted light of the light-emitting element is controlled by the modulation signal, and at the same time, a corresponding demodulated signal is generated based on the modulated signal, and the demodulated signal is transmitted to all ToF sensors for demodulation of the modulated optical signal.
  • the processor can use a chip to collect multiple depth data.
  • the chip can be the aforementioned control chip for generating the modulated signal and the demodulated signal, or it can be a separate independent chip.
  • the chip After acquiring the depth data of the plurality of ToF sensors, the chip performs data fusion processing on each of the depth data to obtain a combined depth map, which is the output of the depth camera in the embodiment of the present application.
  • the processor may include: at least one field programmable gate array FPGA and a ToF controller TFC corresponding to each ToF sensor, where
  • the FPGA is configured to receive depth data of each TFC input, and perform data fusion processing on all received depth data to generate combined depth data; and obtain a combined depth map according to the combined depth data;
  • the first TFC of the TFC included in the processor is configured to generate a modulated signal and a demodulated signal, and output the modulated signal to each of the light emitting elements, and output the demodulated signal to each ToF sensor;
  • Each TFC in the TFC included by the processor is configured to receive depth data of the ToF sensor input corresponding thereto, and output the received depth data to the FPGA.
  • the TFC is a ToF controller, an ASIC chip that controls the ToF system and processes the output of the ToF sensor.
  • each ToF sensor included in the depth camera has a corresponding TFC to parse its output.
  • the ToF sensor of the analog signal output needs to be lost. After the analog-to-digital conversion, it is sent to the TFC.
  • the demodulated signals of all ToF sensors are generated by one of the TFCs to achieve synchronization of the demodulation, and the remaining TFCs are only responsible for receiving the depth data of the corresponding ToF sensor inputs.
  • the depth camera illumination circuit drives all of the light-emitting elements to modulate the light signal to be transmitted with a modulation signal from the aforementioned TFC that generates the demodulated signal.
  • each of the light-emitting elements may be composed of a plurality of LEDs, and may also be composed of a plurality of laser emitters and light-learning light elements.
  • the depth camera uses at least one FPGA to collect the depth data outputted by each TFC, and performs data fusion processing on all the depth data to obtain a combined depth map, which is used as an output of the entire depth camera for external use.
  • the FPGA can also receive external control signals and transmit control signals to the various TFCs.
  • an FPGA is applied to facilitate collection and fusion calculation of multiple signals.
  • other processors that meet the application acquisition and computing power requirements can also be used instead.
  • the DDR in FIG. 6 is a double rate synchronous dynamic random access memory, which is called: Dual Data Rate SDRAM, and Flash is a flash memory.
  • the positions of the components are only for convenience of description and do not represent the position in the actual structure.
  • each ToF sensor uses a TFC to parse the depth data it outputs.
  • the functionality of the TFC can be implemented using an FPGA to save the hardware overhead of the TFC.
  • FIG. 7 the depth data outputted by each ToF sensor is directly collected by the FPGA and the data parsing processing is completed, and the like, which is completed by the TFC in the embodiment shown in FIG. All modulated, demodulated, and control signals are also generated by the FPGA.
  • the FPGA in this embodiment can also be replaced by other processors that meet the requirements of acquisition and computing resources in practical applications.
  • the depth camera may further include a color camera, and a field of view of the color camera covers a field of view superimposed by the at least two ToF sensors:
  • the color camera is configured to collect color data, and output the collected color data to the processor;
  • the processor is further configured to receive color data input by the color camera, and align the received color data with the combined depth data to obtain a color image that is combined with depth information, that is, an RGB-D image.
  • color data can be acquired, and the processor aligns the color data with the combined depth data to obtain a color image in which depth information is integrated, that is, an RGB-D image, in which each pixel is The depth value of the point is stored, and the grayscale and color information of the point is stored.
  • the embodiment of the present application further provides a combined depth map obtaining method, which is applied to a processor in a depth camera, the depth camera including a processor, at least one light emitting component, and at least two flight times
  • the ranging ToF sensor, the superimposed illumination range of the at least one illuminating element covers the superimposed field of view of the at least two ToF sensors.
  • the method can include the following steps:
  • S110 generate a modulated signal and a demodulated signal
  • S120 output the modulated signal to each of the light emitting elements, and output the demodulated signal to each ToF sensor;
  • S130 For each ToF sensor, receive the depth data obtained by the ToF sensor after demodulating the modulated optical signal reflected by the object in the field of view range by using the demodulation signal;
  • the modulated optical signal is a light-emitting element having an illumination range corresponding to a field of view of the ToF sensor, modulated by the modulated signal and modulated by its own optical signal;
  • S140 Perform data fusion processing on all received depth data to generate combined depth data.
  • the technical solution provided by the embodiment of the present application is applied to a processor in a depth camera, where the depth camera includes at least two ToF sensors, and the spatial arrangement of the ToF sensors allows them to face different fields of view, respectively, and each field of view The regions can be combined to form a larger field of view range.
  • the number and arrangement of ToF sensors and illuminating elements can be determined according to the actual field of view range, as long as the superimposed illumination range of all illuminating elements in the depth camera can cover the superimposed field of view of all ToF sensors. Just fine.
  • the processor can generate a modulated signal and a demodulated signal, the modulated signal and the demodulated signal corresponding.
  • the processor transmits a modulated signal to each of the light emitting elements, and transmits the demodulated signal to each of the ToF sensors. That is to say, all of the light-emitting elements in the depth camera use the same modulation signal, and all of the ToF sensors in the depth camera use the same demodulation signal to achieve modulation and demodulation synchronization.
  • the light-emitting element After receiving the modulated signal sent by the processor, the light-emitting element modulates its own optical signal and transmits the modulated optical signal.
  • An object within the illumination range of each of the light-emitting elements reflects the modulated light signal of the light-emitting element and is received by a ToF sensor having a field of view corresponding to the illumination range.
  • the ToF sensor After receiving the modulated optical signal reflected by the object in the field of view, the ToF sensor demodulates the received modulated optical signal using the demodulated signal to generate depth data.
  • the depth data generated by each ToF sensor is the depth value of each pixel point that the ToF sensor acquires in its field of view range region.
  • Each ToF sensor outputs the generated depth data to the processor.
  • the processor After receiving the depth data input by each ToF sensor, the processor performs data fusion processing on all the received depth data, and selects an appropriate original point for mapping the data in the superimposed field of view of the ToF sensor to generate a combined depth. Data, and according to the combined depth data, a combined depth map is obtained.
  • the processor can output the obtained combined depth map through the output interface of the depth camera as an output of the depth camera.
  • the light-emitting elements in the depth camera use the same modulation signal to modulate and transmit the optical signal, and the ToF sensor demodulates the modulated optical signal reflected by the received object using the same demodulated signal corresponding to the modulated signal to generate depth data. It can avoid the problem of mutual interference due to the unsynchronized modulation and demodulation.
  • the processor performs data fusion processing on all the received depth data to obtain a combined depth map.
  • step S140 performs data fusion processing on all the received depth data to generate combined depth data, which may include the following steps:
  • Step 1 determining each field of view overlapping area of the at least two ToF sensors
  • Step 2 for each pixel in each field overlap region, according to the confidence level of the ToF sensor corresponding to the pixel point corresponding to the pixel point, the depth value obtained by the ToF sensor corresponding to the highest confidence value is taken as The depth value of the pixel;
  • Step 3 Generate combined depth data according to the determined depth value of each pixel.
  • Confidence and depth values are one-to-one correspondence, that is, each pixel of the depth data output by the ToF sensor has a corresponding depth value, and also has a confidence level.
  • the ToF sensor corresponds to a depth value corresponding to each original pixel point, and for each target pixel in the overlap region of the field of view, the depth of the ToF sensor corresponding to the highest value of the corresponding plurality of original pixel points is obtained.
  • the value is the depth value of the target pixel.
  • the field of view of the ToF sensor A and the ToF sensor B has an overlap area O.
  • both the depth value acquired by the ToF sensor A and the depth value acquired by the ToF sensor B exist.
  • the confidence value corresponding to the depth value acquired by the ToF sensor A is higher than the confidence level corresponding to the depth value acquired by the ToF sensor B, and the depth value acquired by the ToF sensor A is used as the depth value of the pixel point.
  • step S140 all the received depth data are subjected to data fusion processing to generate combined depth data.
  • the embodiment of the present application provides a specific calculation method.
  • a depth camera containing at least two ToF sensors is considered to be composed of a depth camera composed of a plurality of single ToF sensors, and the description of the depth camera in this embodiment refers to a depth camera of a single ToF sensor.
  • the prerequisites for implementing this calculation method are as follows:
  • the x-axis of the virtual depth camera coordinate system needs to be on the optical center of the real depth camera.
  • two real-depth cameras at different positions that meet the above preconditions are used, and the virtual depth camera is defined at the midpoint of the optical centerline of the two real-depth cameras, and according to actual needs.
  • the depth data of the two real depth cameras are respectively mapped to the virtual depth camera for data fusion, and the data exceeding the field of view of the virtual depth camera is discarded during the fusion process, and the data of the overlapping field of view area is selected according to the confidence degree, that is, If both real depth cameras have points mapped to the same point in the virtual depth camera, then the point with the greater confidence is retained.
  • the imaging plane of the real depth camera shown in the figure has an angle ⁇ with the imaging surface of the virtual depth camera.
  • the value of ⁇ can be 0 degrees, or it can be positive or negative, and the clips of two real depth cameras
  • the angle ⁇ can take different values, and the specific value can be set according to the actual situation.
  • the correspondence between two or more real depth cameras and virtual depth cameras can be extended on the basis of FIG. I will not repeat them here.
  • the following begins to derive the mapping from a real depth camera to a virtual depth camera.
  • the following derivation does not consider the effects of camera lens distortion.
  • Or is the lens optical center of a real depth camera
  • Ov is the lens optical center of the virtual depth camera
  • the connection between Or and Ov is parallel to the imaging plane of the virtual depth camera, and the connection is denoted as B.
  • a Cartesian coordinate system is established according to the front projection model imaged by the camera.
  • the x-axis of the two coordinate systems is coplanar
  • the y-axis is parallel to each other
  • the angle between the x-axes of the two coordinate systems is ⁇ .
  • the focal length of the real depth camera is f1
  • the focal length of the virtual depth camera is f2.
  • the projection point on the real depth camera coordinate system is Pr, the coordinates are (x1, y1), and the corresponding depth value is d1;
  • the projection point on the virtual camera coordinate system is Pv, and the coordinates are (x2, Y2), the corresponding depth value is d2, and the P point is equal to the y-axis coordinate of the two coordinate systems, and is denoted as yp.
  • the projection of the spatial point P on the plane of the x-axis of the two coordinate system is Po, from Po to the line connecting Or and Ov, denoted by dr and dv, and the angle between dr and B is denoted by ⁇ , dv and B The angle is denoted by ⁇ .
  • B, ⁇ , f1, and f2 are constant during the calculation.
  • k2 and k3 can be uniquely determined for a certain pixel point (x1, y1).
  • the coefficient (k1, k2, k3, k4) can be uniquely determined, as shown in the following equation:
  • the coefficient group corresponding to each point (x1, y1) in each real depth camera coordinate system can be obtained according to each real depth camera parameter and the virtual depth camera parameter (k1, K2, k3, k4, ⁇ ) are stored in the table, and the subsequent mapping calculation can obtain the coefficient substitution calculation by looking up the table, and the calculation resource consumption required for calculating the coefficient is omitted.
  • the subsequent mapping calculation can obtain the coefficient substitution calculation by looking up the table, and the calculation resource consumption required for calculating the coefficient is omitted.
  • the data fusion processing process for multiple depth data can be as follows:
  • the real depth camera starts initialization, first determine the position and focal length of the virtual depth camera, and then calculate the distance and inclination of each real depth camera relative to the virtual depth camera according to the parameters of the virtual depth camera, and then according to the determined parameters according to the foregoing
  • the theoretical derivation results calculate the mapping coefficients (k1, k2, k3, k4, ⁇ ) for each pixel of each real depth camera and generate a lookup table, and finally collect each pixel in the depth data obtained for each real depth camera.
  • the point is obtained by searching the table to obtain the mapping function coefficient to calculate the mapping target point position and depth value, and obtain the combined depth map.
  • the point is discarded, and if there are multiple points simultaneously mapped to the same target point, the maximum value of the confidence is retained. Depth value.
  • an algorithm such as target recognition, tracking, etc. can be performed thereon.
  • a combined depth map of a large field of view is used to track and locate a person; or a combined depth map of a large field of view is applied to a somatosensory game for recognition of a target action in a game.
  • step S140 performs data fusion processing on all the received depth data to generate combined depth data, which may include the following steps:
  • a first step determining each field of view overlap region of the at least two ToF sensors
  • a second step determining, for each pixel point in each field overlap region, a depth value of the pixel point according to a size of a depth value of the pixel point corresponding to the pixel point corresponding to the pixel point;
  • the third step generating combined depth data according to the determined depth value of each pixel.
  • the depth value of the pixel point can be determined according to the magnitude of these depth values.
  • the field of view of the ToF sensor A and the ToF sensor B has an overlap area O.
  • the depth value T1 acquired by the ToF sensor A and the depth acquired by the ToF sensor B there is both the depth value T1 acquired by the ToF sensor A and the depth acquired by the ToF sensor B.
  • the value T2, if T1>T2, the depth value T2 acquired by the ToF sensor B can be used as the depth value of the pixel point. It is also possible to take the average of T1 and T2 as the depth value of the pixel.
  • the depth camera may further include a color camera, the field of view of the color camera covering the field of view superimposed by the at least two ToF sensors, and the method may further include the following steps:
  • Step 1 receiving color data input by the color camera
  • Step 2 Align the received color data with the combined depth data, and output a color image in which the depth information is combined.
  • color data can be acquired, and the processor aligns the color data with the combined depth data to obtain a color image in which depth information is integrated, that is, an RGB-D image, in which each pixel is The depth value of the point is stored, and the grayscale and color information of the point is stored.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Measurement Of Optical Distance (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Optical Radar Systems And Details Thereof (AREA)

Abstract

一种组合深度图获得方法及深度相机，所述方法应用于深度相机中的处理器，所述深度相机包括所述处理器、至少一个发光元件和至少两个飞行时间测距ToF传感器，所述至少一个发光元件的叠加照射范围覆盖所述至少两个ToF传感器的叠加视场范围。深度相机中的发光元件使用相同的调制信号对光信号进行调制后发射，ToF传感器使用相同的与调制信号对应的解调信号对接收到的物体反射回的调制光信号进行解调，生成深度数据，处理器对接收到的所有深度数据进行数据融合处理，获得组合深度图。可以满足大视场范围深度图的应用需求，避免因调制解调不同步而相互干扰的问题。

Description

一种组合深度图获得方法及深度相机
本申请要求于2015年10月15日提交中国专利局、申请号为201510665910.6发明名称为“一种组合深度图获得方法及深度相机”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及视频监控技术领域，特别涉及一种组合深度图获得方法及深度相机。
背景技术
深度相机，与普通相机一样具有一定的分辨率，但其各个像素点对应存储的有该像素点对应物体到相机的距离，可以称之为深度，即深度相机输出的是深度图，图上每一个像素点的值都表示该像素点对应物体到相机的距离。
使用ToF(Time of flight，飞行时间测距)传感器的深度相机，是利用ToF技术进行深度图的输出。ToF技术是通过发射和接收经过调制的光信号，解析发射和接收光信号的时间差来测量反射光信号的物体与发射、接收端的距离。ToF传感器是ToF技术实际应用中的光信号接收和解析元件，其配合调制光发光元件来实现深度的测量。一般情况下，ToF传感器的输出经过一定数学变换后可以得到深度图。
目前业内使用的大都是使用单个ToF传感器的深度相机。单ToF传感器的深度相机能够测量深度的视场范围一般较小，可能无法满足某些需要大的视场范围深度图的应用的需求。为满足需要大的视场范围深度图的应用的需求，通常需要在同一个场景中使用多个单ToF传感器的深度相机，但这样又存在各个单ToF传感器的深度相机之间因调制解调不同步而互相干扰的问题。
发明内容
本申请实施例的目的在于提供一种组合深度图获得方法及深度相机，以解决多个单ToF传感器的深度相机之间互相干扰的问题。技术方案如下：
第一方面，本申请提供了一种深度相机，包括：一个处理器、至少一个 发光元件和至少两个飞行时间测距ToF传感器，所述至少一个发光元件的叠加照射范围覆盖所述至少两个ToF传感器的叠加视场范围，其中，
所述处理器，用于生成调制信号和解调信号，并将所述调制信号输出给每个发光元件，将所述解调信号输出给每个ToF传感器；接收每个ToF传感器输入的深度数据；将接收到的所有深度数据进行数据融合处理，生成组合深度数据；根据所述组合深度数据，获得组合深度图；
每个发光元件，用于接收所述处理器输入的调制信号；使用所述调制信号对自身的光信号进行调制，并向自身的照射范围内的物体发射调制光信号；
每个ToF传感器，用于接收所述处理器输入的解调信号；接收自身的视场范围内的物体反射回的调制光信号；使用所述解调信号对接收到的光信号进行解调，生成深度数据；将生成的深度数据输出给所述处理器。
在本申请的一种具体实施方式中，所述至少两个ToF传感器以传感器矩阵的方式进行排布，所述传感器矩阵包含至少一行和至少一列；
对所述传感器矩阵的每行，该行中每个ToF传感器的靶面的几何中心均位于一条直线上，该行中相邻两个ToF传感器之间的距离符合预设的第一距离要求，该行中相邻两个ToF传感器的靶面之间的夹角满足预设的夹角要求；
对所述传感器矩阵的每列，该列中每个ToF传感器的靶面的几何中心均位于一条直线上，该列中相邻两个ToF传感器之间的距离符合预设的第二距离要求，该列中相邻两个ToF传感器的靶面共面或互相平行。
在本申请的一种具体实施方式中，所述传感器矩阵的每行中相邻两个ToF传感器的靶面共面或互相平行。
在本申请的一种具体实施方式中，所述至少两个ToF传感器排布在预设球面上，不同ToF传感器在所述预设球面上的位置不同。
在本申请的一种具体实施方式中，所述处理器包括：至少一个现场可编程门阵列FPGA和与每个ToF传感器对应的ToF控制器TFC，其中，
所述FPGA，用于接收每个TFC输入的深度数据，并将接收到的所有深度数据进行数据融合处理，生成组合深度数据；根据所述组合深度数据，获得 组合深度图；
所述处理器包含的TFC中的第一TFC，用于生成调制信号和解调信号，并将所述调制信号输出给每个发光元件，将所述解调信号输出给每个ToF传感器；
所述处理器包含的TFC中的每个TFC，用于接收自身对应的ToF传感器输入的深度数据，并将接收到的深度数据输出给所述FPGA。
在本申请的一种具体实施方式中，还包括：彩色摄像机，所述彩色摄像机的视场范围覆盖所述至少两个ToF传感器叠加的视场范围；
所述彩色摄像机，用于采集颜色数据，并将采集到的颜色数据输出给所述处理器；
所述处理器，还用于接收所述彩色摄像机输入的颜色数据；将接收到的颜色数据与所述组合深度数据进行对准，获得融合了深度信息的彩色图像。
第二方面，本申请提供了一种组合深度图获得方法，应用于深度相机中的处理器，所述深度相机包括所述处理器、至少一个发光元件和至少两个飞行时间测距ToF传感器，所述至少一个发光元件的叠加照射范围覆盖所述至少两个ToF传感器的叠加视场范围，所述方法包括：
生成调制信号和解调信号；
将所述调制信号输出给每个发光元件，并将所述解调信号输出给每个ToF传感器；
针对每个ToF传感器，接收该ToF传感器采用所述解调信号对其视场范围内的物体反射回的调制光信号进行解调后，获得的深度数据，其中，所述调制光信号为具有与该ToF传感器的视场范围对应的照射范围的发光元件使用所述调制信号对自身的光信号进行调制后发射的；
将接收到的所有深度数据进行数据融合处理，生成组合深度数据；
根据所述组合深度数据，获得组合深度图。
在本申请的一种具体实施方式中，所述将接收到的所有深度数据进行数据融合处理，生成组合深度数据，包括：
确定所述至少两个ToF传感器的每个视场重叠区域；
针对每个视场重叠区域中的每个像素点，根据该像素点对应的每个ToF传感器中针对该像素点的置信度，将置信度最高值对应的ToF传感器获取的深度值作为该像素点的深度值；
根据确定的每个像素点的深度值，生成组合深度数据。
在本申请的一种具体实施方式中，所述将接收到的所有深度数据进行数据融合处理，生成组合深度数据，包括：
确定所述至少两个ToF传感器的每个视场重叠区域；
针对每个视场重叠区域中的每个像素点，根据该像素点对应的每个ToF传感器中针对该像素点的深度值的大小，确定该像素点的深度值；
根据确定的每个像素点的深度值，生成组合深度数据。
在本申请的一种具体实施方式中，所述深度相机还包括彩色摄像机，所述彩色摄像机的视场范围覆盖所述至少两个ToF传感器叠加的视场范围，所述方法还包括：
接收所述彩色摄像机输入的颜色数据；
将接收到的颜色数据与所述组合深度数据进行对准，输出融合了深度信息的彩色图像。
本申请实施例所提供的深度相机，包含一个处理器、至少一个发光元件和至少两个ToF传感器，可以满足大的视场范围深度图的应用的需求。深度相机中的发光元件使用相同的调制信号对光信号进行调制后发射，ToF传感器使用相同的与调制信号对应的解调信号对接收到的物体反射回的调制光信号进行解调，生成深度数据，可以避免因调制解调不同步而相互干扰的问题。处理器对接收到的所有深度数据进行数据融合处理，获得组合深度图。
附图说明
为了更清楚地说明本申请实施例和现有技术的技术方案，下面对实施例和现有技术中所需要使用的附图作简单地介绍，显而易见地，下面描述中的 附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请实施例中深度相机的一种结构示意图；
图2为本申请实施例中ToF传感器的一种排布方式示意图；
图3为本申请实施例中ToF传感器的另一种排布方式示意图；
图4为本申请实施例中ToF传感器的另一种排布方式示意图；
图5为本申请实施例中ToF传感器的另一种排布方式示意图；
图6为本申请实施例中深度相机的另一种结构示意图；
图7为本申请实施例中深度相机的另一种结构示意图。
图8为本申请实施例中组合深度图获得方法的一种实施流程图；
图9为本申请实施例中真实深度相机与虚拟深度相机位置示意图；
图10为本申请实施例中空间点P在真实深度相机和虚拟深度相机中投影关系的一种示意图；
图11为本申请实施例中空间点P在真实深度相机和虚拟深度相机中投影关系的另一种示意图。
具体实施方式
为了使本领域技术人员更好地理解本申请实施例中的技术方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
参见图1所示，为本申请实施例所提供的一种深度相机的结构示意图，该深度相机包括：一个处理器、至少一个发光元件和至少两个飞行时间测距ToF传感器，所述至少一个发光元件的叠加照射范围覆盖所述至少两个ToF传感器的叠加视场范围，其中，
所述处理器，用于生成调制信号和解调信号，并将所述调制信号输出给每个发光元件，将所述解调信号输出给每个ToF传感器；接收每个ToF传感器输入的深度数据；将接收到的所有深度数据进行数据融合处理，生成组合深度数据；根据所述组合深度数据，获得组合深度图；
每个发光元件，用于接收所述处理器输入的调制信号；使用所述调制信号对自身的光信号进行调制，并向自身的照射范围内的物体发射调制光信号；
每个ToF传感器，用于接收所述处理器输入的解调信号；接收自身的视场范围内的物体反射回的调制光信号；使用所述解调信号对接收到的光信号进行解调，生成深度数据；将生成的深度数据输出给所述处理器。
本申请实施例所提供的深度相机，包含至少两个ToF传感器，ToF传感器的空间排布使得他们可以分别面向不同的视场范围区域，各视场范围区域可以组合形成较大的视场范围区域。在实际应用中，可以根据实际的视场范围的需求确定ToF传感器和发光元件的数量及排布方式，只要使得该深度相机中所有发光元件的叠加照射范围能够覆盖所有ToF传感器的叠加视场范围即可。
本申请实施例所提供的深度相机中的处理器，可以通过该深度相机的输入接口获得外部控制信号，外部控制信号可以由操作人员为设置和调整深度相机的相应参数而输入，比如用于设置或调整深度相机的曝光时间等。处理器可以根据该外部控制信号生成调制信号和解调信号，调制信号和解调信号相对应。
处理器将调制信号发送给每一个发光元件，将解调信号发送给每一个ToF传感器，也就是说，该深度相机中的所有发光元件使用的调制信号相同，该深度相机中的所有的ToF传感器使用的解调信号相同，以实现调制、解调同步。
发光元件接收到处理器发送的调制信号后，对自身的光信号进行调制，并将调制光信号发射出去。每个发光元件的照射范围内的物体会将该发光元件的调制光信号反射回来，由具有与该照射范围相对应的视场范围的ToF传感器接收。
ToF传感器接收到其视场范围内的物体反射回的调制光信号后，使用解调信号对接收到的调制光信号进行解调，生成深度数据。每个ToF传感器生成的 深度数据为该ToF传感器在其视场范围区域中获取的每个像素点的深度值。每个ToF传感器将生成的深度数据输出给处理器。
处理器接收到每个ToF传感器输入的深度数据后，会将接收到的所有深度数据进行数据融合处理，对于ToF传感器的叠加视场范围内的数据选取合适的原始点进行映射，从而生成组合深度数据，并根据组合深度数据，获得组合深度图。处理器可以通过该深度相机的输出接口将获得的组合深度图输出，作为该深度相机的输出。
在实际应用中，处理器还可以根据接收到的外部控制信号生成针对ToF传感器的控制信号，为ToF传感器下发相应参数，以对ToF传感器实行相应的控制。
本申请实施例所提供的深度相机，包含一个处理器、至少一个发光元件和至少两个ToF传感器，可以满足大视场范围深度图的应用需求。深度相机中的发光元件使用相同的调制信号对光信号进行调制后发射，ToF传感器使用相同的与调制信号对应的解调信号对接收到的物体反射回的调制光信号进行解调，生成深度数据，可以避免因调制解调不同步而相互干扰的问题。处理器对接收到的所有深度数据进行数据融合处理，获得组合深度图。
在本申请的一个实施例中，所述至少两个ToF传感器可以以传感器矩阵的方式进行排布，所述传感器矩阵包含至少一行和至少一列；
对所述传感器矩阵的每行，该行中每个ToF传感器的靶面的几何中心均位于一条直线上，该行中相邻两个ToF传感器之间的距离符合预设的第一距离要求，该行中相邻两个ToF传感器的靶面之间的夹角满足预设的夹角要求；
对所述传感器矩阵的每列，该列中每个ToF传感器的靶面的几何中心均位于一条直线上，该列中相邻两个ToF传感器之间的距离符合预设的第二距离要求，该列中相邻两个ToF传感器的靶面共面或互相平行。
在实际应用中，ToF传感器的空间排布方式可以根据实际情况进行确定。下面例举几个实例对该深度相机包含的至少两个ToF传感器的排布方式进行说明。
实例1：
当该深度相机中的多个ToF传感器以传感器矩阵的方式进行排布，且该传感器矩阵只包含一行时，多个ToF传感器排布在同一条直线上，所有ToF传感器的靶面的几何中心均位于该直线上，且相邻两个ToF传感器之间的距离需要符合预设的第一距离要求，相邻两个ToF传感器的靶面之间的夹角需要满足预设的夹角要求，具体参见图2所示。
实例2：
上述实例1中相邻两个ToF传感器的靶面之间的夹角可以为0，即相邻两个ToF传感器的靶面共面或互相平行，具体参见图3所示。
实例3：
当该深度相机中的多个ToF传感器以传感器矩阵的方式进行排布，且该传感器矩阵包含多行和多列时，每行中的ToF传感器的具体排布方式可以与上述实例1一致。对于传感器矩阵中的每列，该列中每个ToF传感器的靶面的几何中心均位于一条直线上，该列中相邻两个ToF传感器之间的距离需要符合预设的第二距离要求，该列中相邻两个ToF传感器的靶面共面或互相平行，具体参见图4所示。
实例4：
当该深度相机中的多个ToF传感器以传感器矩阵的方式进行排布，且该传感器矩阵包含多行和多列时，每行中的ToF传感器的具体排布方式可以与上述实例2一致，每列中的ToF传感器的具体排布方式可以与上述实例3中每列中的ToF传感器的具体排布方式一致，具体参见图5所示。
需要说明的是，本申请实施例中预设的第一距离要求、预设的第二距离要求、预设的夹角要求等均可以根据实际情况进行设置和调整，如根据实际的视场范围区域的需求进行设置和调整，本申请实施例对此不做限制。
在本申请的一个实施例中，所述至少两个ToF传感器排布在预设球面上，不同ToF传感器在所述预设球面上的位置不同，能够满足实际所需的视场范围区域即可。
需要说明的是，本领域技术人员可以根据上述例举的本申请实施例所提 供的深度相机中的ToF传感器的排布方式安装ToF传感器，在安装ToF传感器过程中，还需要根据ToF传感器的叠加视场范围安装发光元件，使得发光元件的发射光能够覆盖ToF传感器的叠加视场范围。当然，ToF传感器的具体排布方式不限于上述例举的几种。本领域技术人员根据上述例举的几个实例可以推导出其他不同的实例，对此本申请实施例不再赘述。
在实际应用中，深度相机中的处理器可以包含一块控制芯片，以实现多个ToF传感器共用解调信号、多个发光元件共用调制信号，该控制芯片按照ToF技术原理生成发光元件的调制信号，通过调制信号控制发光元件的发射光，同时，根据调制信号产生相应的解调信号，并将解调信号传送给所有ToF传感器用作对调制光信号的解调。
对于多ToF传感器的输出，处理器可以使用一块芯片进行多个深度数据的采集，该芯片可以是前述用于产生调制信号和解调信号的控制芯片，还可以是一块另外的独立芯片。该芯片采集到多个ToF传感器的深度数据后，对每个深度数据进行数据融合处理，获得组合深度图，即为本申请实施例中深度相机的输出。
参见图6所示，在本申请的一个实施例中，所述处理器可以包括：至少一个现场可编程门阵列FPGA和与每个ToF传感器对应的ToF控制器TFC，其中，
所述FPGA，用于接收每个TFC输入的深度数据，并将接收到的所有深度数据进行数据融合处理，生成组合深度数据；根据所述组合深度数据，获得组合深度图；
所述处理器包含的TFC中的第一TFC，用于生成调制信号和解调信号，并将所述调制信号输出给每个发光元件，将所述解调信号输出给每个ToF传感器；
所述处理器包含的TFC中的每个TFC，用于接收自身对应的ToF传感器输入的深度数据，并将接收到的深度数据输出给所述FPGA。
TFC为ToF控制器，是一种ASIC芯片，用于控制ToF系统和处理ToF传感器的输出。
在本申请实施例中，该深度相机包含的每个ToF传感器都有对应的TFC解析其输出。根据ToF传感器类型的不同，模拟信号输出的ToF传感器需要将输 出进行模数转换后再送入TFC。所有的ToF传感器的解调信号都由其中某个TFC产生，以实现解调的同步，其余的TFC只负责接收相应的ToF传感器输入的深度数据。
该深度相机光照电路驱动所有发光元件用来自于前述产生解调信号的TFC的调制信号对要发射的光信号进行调制。在实际应用中，每个发光元件可以由多个LED组成，还可以由多个激光发射器和光学习光元件组成。
在本申请实施例中，该深度相机使用至少一个FPGA对各路TFC输出的深度数据进行采集，并对所有深度数据进行数据融合处理，获得组合深度图，作为整个深度相机的输出供外部使用。同时，FPGA还可以接收外部的控制信号，并将控制信号传输给各个TFC。
在本申请实施例中，应用FPGA，便于多路信号的采集和融合计算。在实际应用中，也可以使用满足应用采集和计算能力需求的其他处理器代替。需要说明的是，图6中DDR为双倍速率同步动态随机存储器，全称为：Dual Data Rate SDRAM，Flash为闪存，各个部件的位置仅是为了示意方便，并不代表实际结构中的位置。
在上述实施例中，每个ToF传感器都使用了一个TFC对其输出的深度数据进行解析。在另一个实施例中，TFC的功能可以使用FPGA实现，以节省TFC的硬件开销。参见图7所示，每个ToF传感器输出的深度数据都直接由FPGA进行采集并完成数据解析处理等一系列图6所示实施例中由TFC完成的工作。所有的调制信号、解调信号和控制信号也都由FPGA产生。当然，该实施例中的FPGA在实际应用中也可以由满足采集和计算资源要求的其他处理器代替。
在本申请的一个实施例中，该深度相机中还可以包括彩色摄像机，所述彩色摄像机的视场范围覆盖所述至少两个ToF传感器叠加的视场范围：
所述彩色摄像机，用于采集颜色数据，并将采集到的颜色数据输出给所述处理器；
所述处理器，还用于接收所述彩色摄像机输入的颜色数据；将接收到的颜色数据与所述组合深度数据进行对准，获得融合了深度信息的彩色图像，即RGB-D图像。
在深度相机中安装彩色摄像机，可以采集到颜色数据，处理器将颜色数据与组合深度数据进行对准，从而获得融合了深度信息的彩色图像，即RGB-D图像，其中，每个像素点既存储了该点的深度值，又存储了该点的灰度和颜色信息。
相应于上述的深度相机实施例，本申请实施例还提供了一种组合深度图获得方法，应用于深度相机中的处理器，该深度相机包括处理器、至少一个发光元件和至少两个飞行时间测距ToF传感器，至少一个发光元件的叠加照射范围覆盖至少两个ToF传感器的叠加视场范围。
参见图8所示，该方法可以包括以下步骤：
S110：生成调制信号和解调信号；
S120：将所述调制信号输出给每个发光元件，并将所述解调信号输出给每个ToF传感器；
S130：针对每个ToF传感器，接收该ToF传感器采用所述解调信号对其视场范围内的物体反射回的调制光信号进行解调后，获得的深度数据；
其中，所述调制光信号为具有与该ToF传感器的视场范围对应的照射范围的发光元件使用所述调制信号对自身的光信号进行调制后发射的；
S140：将接收到的所有深度数据进行数据融合处理，生成组合深度数据；
S150：根据所述组合深度数据，获得组合深度图。
为便于说明，将上述几个步骤结合起来进行说明。
本申请实施例所提供的技术方案应用于深度相机中的处理器，该深度相机包含至少两个ToF传感器，ToF传感器的空间排布使得他们可以分别面向不同的视场范围区域，各视场范围区域可以组合形成较大的视场范围区域。在实际应用中，可以根据实际的视场范围的需求确定ToF传感器和发光元件的数量及排布方式，只要使得该深度相机中所有发光元件的叠加照射范围能够覆盖所有ToF传感器的叠加视场范围即可。
处理器可以生成调制信号和解调信号，调制信号和解调信号相对应。处理器将调制信号发送给每一个发光元件，将解调信号发送给每一个ToF传感器， 也就是说，该深度相机中的所有发光元件使用的调制信号相同，该深度相机中的所有的ToF传感器使用的解调信号相同，以实现调制、解调同步。
发光元件接收到处理器发送的调制信号后，对自身的光信号进行调制，并将调制光信号发射出去。每个发光元件的照射范围内的物体会将该发光元件的调制光信号反射回来，由具有与该照射范围相对应的视场范围的ToF传感器接收。
ToF传感器接收到其视场范围内的物体反射回的调制光信号后，使用解调信号对接收到的调制光信号进行解调，生成深度数据。每个ToF传感器生成的深度数据为该ToF传感器在其视场范围区域中获取的每个像素点的深度值。每个ToF传感器将生成的深度数据输出给处理器。
处理器接收到每个ToF传感器输入的深度数据后，会将接收到的所有深度数据进行数据融合处理，对于ToF传感器的叠加视场范围内的数据选取合适的原始点进行映射，从而生成组合深度数据，并根据组合深度数据，获得组合深度图。处理器可以通过该深度相机的输出接口将获得的组合深度图输出，作为该深度相机的输出。
应用本申请实施例所提供的技术方案，可以满足大视场范围深度图的应用需求。深度相机中的发光元件使用相同的调制信号对光信号进行调制后发射，ToF传感器使用相同的与调制信号对应的解调信号对接收到的物体反射回的调制光信号进行解调，生成深度数据，可以避免因调制解调不同步而相互干扰的问题。处理器对接收到的所有深度数据进行数据融合处理，获得组合深度图。
在本申请的一种具体实施方式中，步骤S140将接收到的所有深度数据进行数据融合处理，生成组合深度数据，可以包括以下步骤：
步骤一：确定所述至少两个ToF传感器的每个视场重叠区域；
步骤二：针对每个视场重叠区域中的每个像素点，根据该像素点对应的每个ToF传感器中针对该像素点的置信度，将置信度最高值对应的ToF传感器获取的深度值作为该像素点的深度值；
步骤三：根据确定的每个像素点的深度值，生成组合深度数据。
为便于描述，将上述三个步骤结合起来进行说明。
ToF传感器输出的深度值与真实值之间可能存在误差，通常用置信度来表示深度值与真实值之间误差的大小，一般来说，置信度越大，误差越小。置信度与深度值是一一对应的，即ToF传感器输出的深度数据中每一像素点都有对应的深度值，也有置信度。
当深度相机中包含的多个ToF传感器的视场范围有重叠时，针对视场重叠区域的每个目标像素点，存在多个深度值。ToF传感器针对每个原始像素点的深度值对应有置信度，对于视场重叠区域中的每个目标像素点，将其对应的多个原始像素点中置信度最高值对应的ToF传感器获取的深度值作为该目标像素点的深度值。
比如，ToF传感器A和ToF传感器B的视场范围有重叠区域O，对于重叠区域O中的每个像素点，既存在ToF传感器A获取到的深度值，又存在ToF传感器B获取到的深度值。在该像素点，ToF传感器A获取到的深度值对应的置信度高于ToF传感器B获取到的深度值对应的置信度，则将ToF传感器A获取到的深度值作为该像素点的深度值。
对于其他非视场重叠区域中的每个像素点均有唯一一个深度值。根据确定的每个像素点的深度值，可以生成组合深度数据。
对于步骤S140将接收到的所有深度数据进行数据融合处理，生成组合深度数据，本申请实施例提供一种具体的计算方法。
将一个包含至少两个ToF传感器的深度相机看作为由多个单ToF传感器的深度相机组成，在本实施例中对于深度相机的描述均指的是单ToF传感器的深度相机。实施该计算方法的前提条件如下：
第一个：深度相机在安装时需保证所有深度相机的光心位于一条直线上，这些深度相机为真实深度相机，且真实深度相机坐标系的y轴互相平行；
第二个：虚拟深度相机坐标系的x轴需位于真实深度相机的光心连线上。
在图9中，使用了两个位于不同位置的符合上述前提条件的真实深度相机，将虚拟深度相机定义在两个真实深度相机光心连线的中点上，并根据实际需 求定义虚拟深度相机的焦距和视场范围。将两个真实深度相机的深度数据分别映射到虚拟深度相机上进行数据融合，融合过程中对于超出虚拟深度相机视场范围的数据进行丢弃，对重叠视场区域的数据根据置信度进行选择，即如果两个真实深度相机中都有点映射到了虚拟深度相机中的同一点，则保留置信度较大的那个点。图中所示的真实深度相机的成像面同虚拟深度相机的成像面之间存在夹角θ，θ的取值可以为0度，也可以为正值或负值，两个真实深度相机的夹角θ可以取不同的值，具体取值可以根据实际情况进行设定。在图9所示基础上可以扩展出两个以上真实深度相机与虚拟深度相机的对应关系。在此不再赘述。
以下开始推导从真实深度相机到虚拟深度相机的映射关系。以下推导不考虑相机镜头畸变的影响。
如图10所示，Or为某个真实深度相机的镜头光心，Ov为虚拟深度相机的镜头光心，Or与Ov的连线与虚拟深度相机的成像平面平行，该连线记为B。在Or处和Ov处分别按照相机成像的前投影模型建立直角坐标系，两坐标系的x轴共面，y轴互相平行，两坐标系x轴之间的夹角为θ。真实深度相机的焦距为f1，虚拟深度相机的焦距为f2。
空间中某一点P，在真实深度相机坐标系上的投影点为Pr，坐标为(x1,y1)，对应深度值为d1；在虚拟相机坐标系上的投影点为Pv，坐标为(x2,y2)，对应深度值为d2，P点相对于两个坐标系的y轴向坐标相等，记为yp。空间点P在两坐标系的x轴所在平面上的投影为Po，从Po分别作到Or和Ov的连线，记为dr和dv，dr与B的夹角记为α，dv与B的夹角记为β。以上各参数中，B，θ，f1，f2在计算过程中为常量。
依据三角关系有：
Figure PCTCN2016093546-appb-000001
可以解出：
Figure PCTCN2016093546-appb-000002
Figure PCTCN2016093546-appb-000003
Figure PCTCN2016093546-appb-000004
根据余弦定理有：
dv2＝dr2+B2-2*B*dr*cosα
d22＝dv2+yp2
可以解出：
Figure PCTCN2016093546-appb-000005
令
Figure PCTCN2016093546-appb-000006
则
Figure PCTCN2016093546-appb-000007
可以看出，对于某一确定像素点位置(x1,y1)，k1的值唯一确定。再依据三角关系有：
Figure PCTCN2016093546-appb-000008
Figure PCTCN2016093546-appb-000009
Figure PCTCN2016093546-appb-000010
令
Figure PCTCN2016093546-appb-000011
则有：
Figure PCTCN2016093546-appb-000012
可以看出，对于某确定像素点(x1,y1)，k2和k3可以唯一确定。
再依据三角关系有：
Figure PCTCN2016093546-appb-000013
Figure PCTCN2016093546-appb-000014
令
Figure PCTCN2016093546-appb-000015
可以看出，对于某确定像素点(x1,y1)，k4可以唯一确定。
综上，可以总结如下：对于在真实深度相机坐标系下的某一具有深度值d1的点(x1,y1)，其在虚拟深度相机坐标系下对应的映射点为(x2,y2)，深度值为d2。有如下关系：
Figure PCTCN2016093546-appb-000016
其中，对于每个固定像素点(x1,y1)，系数(k1,k2,k3,k4)可以唯一确定，如下式所示：
Figure PCTCN2016093546-appb-000017
因此，在深度相机的启动初始化过程中，即可根据每个真实深度相机参数和虚拟深度相机参数求出对于每个真实深度相机坐标系中每一点(x1,y1)对应的系数组(k1,k2,k3,k4,α)并存储在表中，之后的映射计算即可通过查表获得系数代入计算，省去计算系数所需的计算资源消耗。此外，由于
Figure PCTCN2016093546-appb-000018
可以得出：
Figure PCTCN2016093546-appb-000019
当B足够小时可以认为d2＝d1，以上映射公式可进一步简化。
以上推导过程是基于Or位于Ov左边时进行的，当Or位于Ov右边时，即如图11所示结构，同理可得映射函数：
Figure PCTCN2016093546-appb-000020
映射系数：
Figure PCTCN2016093546-appb-000021
基于以上多深度相机结构和真实深度相机到虚拟深度相机的映射关系，对于多个深度数据的数据融合处理过程可以如下：
在真实深度相机启动初始化时，首先确定虚拟深度相机的位置和焦距，再根据虚拟深度相机的参数计算各个真实深度相机相对于虚拟深度相机的距离和倾角，之后可以根据确定下来的这些参数按照前述理论推导结果计算对于每个真实深度相机的每个像素点的映射系数(k1,k2,k3,k4,α)并生成查找表，最后对于每个真实深度相机采集获得的深度数据中每个像素点通过查表获得映射函数系数计算映射目标点位置和深度值，获得组合深度图。在映射的过程中，如果映射目标点的位置位于虚拟深度相机视场范围外，则丢弃该点，如果存在多个点同时被映射到同一个目标点的情况，则保留置信度最大值对应的深度值。
对于获得的具有大视场范围的组合深度图，可以在其上执行目标识别，跟踪等算法。例如在室内环境中，利用大视场范围的组合深度图对人员进行跟踪、定位；或者将大视场范围的组合深度图应用在体感游戏上，用于游戏中对于目标动作的识别。
在本申请的另一个具体实施方式中，步骤S140将接收到的所有深度数据进行数据融合处理，生成组合深度数据，可以包括以下步骤：
第一个步骤：确定所述至少两个ToF传感器的每个视场重叠区域；
第二个步骤：针对每个视场重叠区域中的每个像素点，根据该像素点对应的每个ToF传感器中针对该像素点的深度值的大小，确定该像素点的深度值；
第三个步骤：根据确定的每个像素点的深度值，生成组合深度数据。
为便于描述，将上述三个步骤结合起来进行说明。
当深度相机中包含的多个ToF传感器的视场范围有重叠时，针对视场重叠区域的每个像素点，存在多个深度值。可以根据这些深度值的大小，确定该像素点的深度值。
比如，ToF传感器A和ToF传感器B的视场范围有重叠区域O，对于重叠区域O中的每个像素点，既存在ToF传感器A获取到的深度值T1，又存在ToF传感器B获取到的深度值T2，如果T1>T2，则可以将ToF传感器B获取到的深度值T2作为该像素点的深度值。还可以将取T1和T2的平均值作为该像素点的深度值。
对于其他非视场重叠区域中的每个像素点均有唯一一个深度值。根据确定的每个像素点的深度值，可以生成组合深度数据。
在本申请的一个实施例中，深度相机还可以包括彩色摄像机，所述彩色摄像机的视场范围覆盖所述至少两个ToF传感器叠加的视场范围，该方法还可以包括以下步骤：
步骤一：接收所述彩色摄像机输入的颜色数据；
步骤二：将接收到的颜色数据与所述组合深度数据进行对准，输出融合了深度信息的彩色图像。
在深度相机中安装彩色摄像机，可以采集到颜色数据，处理器将颜色数据与组合深度数据进行对准，从而获得融合了深度信息的彩色图像，即RGB-D图像，其中，每个像素点既存储了该点的深度值，又存储了该点的灰度和颜色信息。
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
本说明书中的各个实施例均采用相关的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于装置实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
本领域普通技术人员可以理解实现上述方法实施方式中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，所述的程序可以存储于计算机可读取存储介质中，这里所称得的存储介质，如：ROM/RAM、磁碟、光盘等。
以上所述仅为本申请的较佳实施例而已，并非用于限定本申请的保护范围。凡在本申请的精神和原则之内所作的任何修改、等同替换、改进等，均包含在本申请的保护范围内。

Claims (10)

  1. 一种深度相机，其特征在于，包括：一个处理器、至少一个发光元件和至少两个飞行时间测距ToF传感器，所述至少一个发光元件的叠加照射范围覆盖所述至少两个ToF传感器的叠加视场范围，其中，
    所述处理器，用于生成调制信号和解调信号，并将所述调制信号输出给每个发光元件，将所述解调信号输出给每个ToF传感器；接收每个ToF传感器输入的深度数据；将接收到的所有深度数据进行数据融合处理，生成组合深度数据；根据所述组合深度数据，获得组合深度图；
    每个发光元件，用于接收所述处理器输入的调制信号；使用所述调制信号对自身的光信号进行调制，并向自身的照射范围内的物体发射调制光信号；
    每个ToF传感器，用于接收所述处理器输入的解调信号；接收自身的视场范围内的物体反射回的调制光信号；使用所述解调信号对接收到的光信号进行解调，生成深度数据；将生成的深度数据输出给所述处理器。
  2. 根据权利要求1所述的深度相机，其特征在于，所述至少两个ToF传感器以传感器矩阵的方式进行排布，所述传感器矩阵包含至少一行和至少一列；
    对所述传感器矩阵的每行，该行中每个ToF传感器的靶面的几何中心均位于一条直线上，该行中相邻两个ToF传感器之间的距离符合预设的第一距离要求，该行中相邻两个ToF传感器的靶面之间的夹角满足预设的夹角要求；
    对所述传感器矩阵的每列，该列中每个ToF传感器的靶面的几何中心均位于一条直线上，该列中相邻两个ToF传感器之间的距离符合预设的第二距离要求，该列中相邻两个ToF传感器的靶面共面或互相平行。
  3. 根据权利要求2所述的深度相机，其特征在于，所述传感器矩阵的每行中相邻两个ToF传感器的靶面共面或互相平行。
  4. 根据权利要求1所述的深度相机，其特征在于，所述至少两个ToF传感器排布在预设球面上，不同ToF传感器在所述预设球面上的位置不同。
  5. 根据权利要求1所述的深度相机，其特征在于，所述处理器包括：至少一个现场可编程门阵列FPGA和与每个ToF传感器对应的ToF控制器TFC，其 中，
    所述FPGA，用于接收每个TFC输入的深度数据，并将接收到的所有深度数据进行数据融合处理，生成组合深度数据；根据所述组合深度数据，获得组合深度图；
    所述处理器包含的TFC中的第一TFC，用于生成调制信号和解调信号，并将所述调制信号输出给每个发光元件，将所述解调信号输出给每个ToF传感器；
    所述处理器包含的TFC中的每个TFC，用于接收自身对应的ToF传感器输入的深度数据，并将接收到的深度数据输出给所述FPGA。
  6. 根据权利要求1至5任一项所述的深度相机，其特征在于，还包括：彩色摄像机，所述彩色摄像机的视场范围覆盖所述至少两个ToF传感器叠加的视场范围：
    所述彩色摄像机，用于采集颜色数据，并将采集到的颜色数据输出给所述处理器；
    所述处理器，还用于接收所述彩色摄像机输入的颜色数据；将接收到的颜色数据与所述组合深度数据进行对准，获得融合了深度信息的彩色图像。
  7. 一种组合深度图获得方法，其特征在于，应用于深度相机中的处理器，所述深度相机包括所述处理器、至少一个发光元件和至少两个飞行时间测距ToF传感器，所述至少一个发光元件的叠加照射范围覆盖所述至少两个ToF传感器的叠加视场范围，所述方法包括：
    生成调制信号和解调信号；
    将所述调制信号输出给每个发光元件，并将所述解调信号输出给每个ToF传感器；
    针对每个ToF传感器，接收该ToF传感器采用所述解调信号对其视场范围内的物体反射回的调制光信号进行解调后，获得的深度数据，其中，所述调制光信号为具有与该ToF传感器的视场范围对应的照射范围的发光元件使用所述调制信号对自身的光信号进行调制后发射的；
    将接收到的所有深度数据进行数据融合处理，生成组合深度数据；
    根据所述组合深度数据，获得组合深度图。
  8. 根据权利要求7所述的方法，其特征在于，所述将接收到的所有深度数据进行数据融合处理，生成组合深度数据，包括：
    确定所述至少两个ToF传感器的每个视场重叠区域；
    针对每个视场重叠区域中的每个像素点，根据该像素点对应的每个ToF传感器中针对该像素点的置信度，将置信度最高值对应的ToF传感器获取的深度值作为该像素点的深度值；
    根据确定的每个像素点的深度值，生成组合深度数据。
  9. 根据权利要求7所述的方法，其特征在于，所述将接收到的所有深度数据进行数据融合处理，生成组合深度数据，包括：
    确定所述至少两个ToF传感器的每个视场重叠区域；
    针对每个视场重叠区域中的每个像素点，根据该像素点对应的每个ToF传感器中针对该像素点的深度值的大小，确定该像素点的深度值；
    根据确定的每个像素点的深度值，生成组合深度数据。
  10. 根据权利要求7至9任一项所述的方法，其特征在于，所述深度相机还包括彩色摄像机，所述彩色摄像机的视场范围覆盖所述至少两个ToF传感器叠加的视场范围，所述方法还包括：
    接收所述彩色摄像机输入的颜色数据；
    将接收到的颜色数据与所述组合深度数据进行对准，输出融合了深度信息的彩色图像。
PCT/CN2016/093546 2015-10-15 2016-08-05 一种组合深度图获得方法及深度相机 Ceased WO2017063435A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US15/767,607 US10713804B2 (en) 2015-10-15 2016-08-05 Method for obtaining combined depth image, and depth camera
EP16854817.0A EP3364643B1 (en) 2015-10-15 2016-08-05 Method for obtaining combined depth image, and depth camera

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510665910.6 2015-10-15
CN201510665910.6A CN106612387B (zh) 2015-10-15 2015-10-15 一种组合深度图获得方法及深度相机

Publications (1)

Publication Number Publication Date
WO2017063435A1 true WO2017063435A1 (zh) 2017-04-20

Family

ID=58517039

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/093546 Ceased WO2017063435A1 (zh) 2015-10-15 2016-08-05 一种组合深度图获得方法及深度相机

Country Status (4)

Country Link
US (1) US10713804B2 (zh)
EP (1) EP3364643B1 (zh)
CN (1) CN106612387B (zh)
WO (1) WO2017063435A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110945380A (zh) * 2017-07-18 2020-03-31 Lg伊诺特有限公司 ToF模块以及使用ToF模块的对象识别装置
CN111741284A (zh) * 2019-03-25 2020-10-02 华为技术有限公司 图像处理的装置和方法
CN114972468A (zh) * 2022-05-26 2022-08-30 杭州海康机器人技术有限公司 一种深度图像获取方法、装置及电子设备
US12096134B2 (en) 2019-03-25 2024-09-17 Huawei Technologies Co., Ltd. Big aperture blurring method based on dual cameras and TOF

Families Citing this family (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB201511551D0 (en) 2015-07-01 2015-08-12 St Microelectronics Res & Dev Photonics device
CN107063188A (zh) * 2017-05-19 2017-08-18 深圳奥比中光科技有限公司 大视角3d视觉系统
US10785400B2 (en) 2017-10-09 2020-09-22 Stmicroelectronics (Research & Development) Limited Multiple fields of view time of flight sensor
CN108279421B (zh) * 2018-01-28 2021-09-28 深圳新亮智能技术有限公司 带有高分辨率彩色图像的飞行时间相机
CN110278679B (zh) * 2018-03-13 2021-05-04 鸿海精密工业股份有限公司 感测模块及其电子装置
CN110276338A (zh) * 2018-03-13 2019-09-24 鸿海精密工业股份有限公司 感测模块及其电子装置
CN108564614B (zh) * 2018-04-03 2020-09-18 Oppo广东移动通信有限公司 深度获取方法和装置、计算机可读存储介质和计算机设备
CN110349196B (zh) * 2018-04-03 2024-03-29 联发科技股份有限公司 深度融合的方法和装置
CN110363051A (zh) * 2018-04-10 2019-10-22 鸿海精密工业股份有限公司 感测模块及其电子装置
US10999524B1 (en) 2018-04-12 2021-05-04 Amazon Technologies, Inc. Temporal high dynamic range imaging using time-of-flight cameras
CN108965579A (zh) * 2018-06-05 2018-12-07 Oppo广东移动通信有限公司 基于tof摄像头实现测距的方法及其装置、终端和存储介质
CN108897000B (zh) * 2018-06-19 2022-02-08 北京旷视科技有限公司 一种串扰抑制方法、装置、电子设备和计算机可读介质
US10674063B2 (en) * 2018-06-20 2020-06-02 Amazon Technologies, Inc. Synchronizing time-of-flight cameras
US10708484B2 (en) 2018-06-20 2020-07-07 Amazon Technologies, Inc. Detecting interference between time-of-flight cameras using modified image sensor arrays
CN108921888B (zh) * 2018-07-03 2022-07-19 京东方科技集团股份有限公司 一种深度图像检测方法、系统及计算机可读存储介质
CN108848320B (zh) * 2018-07-06 2021-01-26 京东方科技集团股份有限公司 深度检测系统及其曝光时间调整方法
US10681338B1 (en) 2018-07-24 2020-06-09 Amazon Technologies, Inc. Detecting interference in depth images captured using overlapping depth cameras
CN109031331A (zh) * 2018-08-07 2018-12-18 上海炬佑智能科技有限公司 飞行时间测距传感芯片及传感装置、电子设备
CN109164456B (zh) * 2018-09-21 2020-09-29 维沃移动通信有限公司 深度摄像头模组、移动终端及摄像头模组互扰处理方法
US11055866B2 (en) 2018-10-29 2021-07-06 Samsung Electronics Co., Ltd System and method for disparity estimation using cameras with different fields of view
CN109343079A (zh) * 2018-10-30 2019-02-15 合肥泰禾光电科技股份有限公司 一种测距避障方法及避障装置
CN109454669A (zh) * 2018-11-14 2019-03-12 深圳蓝胖子机器人有限公司 机器人实时感知系统及具有其的感知机器人
CN109544616B (zh) * 2018-12-11 2021-02-26 维沃移动通信有限公司 一种深度信息确定方法和终端
EP3671277A1 (en) * 2018-12-21 2020-06-24 Infineon Technologies AG 3d imaging apparatus and method
CN109541631B (zh) * 2019-01-07 2024-09-10 杭州蓝芯科技有限公司 一种基于光飞行时间的大视场面阵探测雷达
CN109963138A (zh) * 2019-02-15 2019-07-02 深圳奥比中光科技有限公司 一种深度相机及图像获取方法
KR102068929B1 (ko) * 2019-05-23 2020-01-21 박배억 3차원 카메라 시스템
WO2020243901A1 (en) 2019-06-04 2020-12-10 Texas Instruments Incorporated An optical time of flight sensor for navigation systems in robotic applications
DE102019215099B4 (de) * 2019-10-01 2022-09-29 Audi Ag Verfahren zum Bereitstellen einer aktuellen lokalen Umgebungszustandskarte für ein Kraftfahrzeug sowie Kraftfahrzeug zum Durchführen eines derartigen Verfahrens
WO2021114036A1 (zh) * 2019-12-09 2021-06-17 南昌欧菲生物识别技术有限公司 Tof相机及电子设备
KR102184210B1 (ko) * 2019-12-26 2020-11-27 박배억 3차원 카메라 시스템
CN111458693A (zh) * 2020-02-01 2020-07-28 上海鲲游光电科技有限公司 直接测距tof分区探测方法及其系统和电子设备
EP4189463A1 (en) * 2020-12-18 2023-06-07 Google LLC Steerable camera array for head-mounted display devices
CN112950517B (zh) * 2021-02-25 2023-11-03 浙江光珀智能科技有限公司 深度相机高动态范围深度图与灰度图的融合方法及装置
CN117055541A (zh) * 2022-05-06 2023-11-14 追觅创新科技(苏州)有限公司 自移动设备的控制方法、设备及存储介质
CN117670967A (zh) * 2022-08-26 2024-03-08 深圳市光鉴科技有限公司 一种快速深度重建方法、系统、设备及介质
TWI894668B (zh) * 2022-11-01 2025-08-21 鈺立微電子股份有限公司 飛時測距系統以及飛時測距方法
US12573066B2 (en) 2022-11-07 2026-03-10 Google Llc Depth estimation using a single near-infrared camera and dot illuminator
CN116365374B (zh) * 2023-03-23 2023-10-27 深圳市欧亚特电器设备有限公司 一种可远程控制的智能型配电柜及其控制方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030076484A1 (en) * 2000-11-09 2003-04-24 Canesta, Inc. Systems for CMOS-compatible three-dimensional image sensing using quantum efficiency modulation
CN101866056A (zh) * 2010-05-28 2010-10-20 中国科学院合肥物质科学研究院 基于led阵列共透镜tof深度测量的三维成像方法和系统
CN102741651A (zh) * 2010-02-01 2012-10-17 微软公司 用于飞行时间测距系统的多个同步光源
CN103026256A (zh) * 2010-07-06 2013-04-03 梅卡雷斯系统有限责任公司 具有补偿光源的光电子测量装置
CN104838284A (zh) * 2012-11-08 2015-08-12 蓝泰科尼克有限公司 用于至少两个ToF相机的拍摄方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5979500B2 (ja) * 2011-04-07 2016-08-24 パナソニックＩｐマネジメント株式会社 立体撮像装置
JP5924978B2 (ja) * 2012-02-28 2016-05-25 キヤノン株式会社 画像処理装置および画像処理方法
CN108345006B (zh) * 2012-09-10 2022-04-08 广稹阿马斯公司 捕捉运动场景的设备、装置和系统
KR102163728B1 (ko) * 2013-12-05 2020-10-08 삼성전자주식회사 거리영상 측정용 카메라 및 이를 이용한 거리영상 측정방법
EP4016981A3 (en) * 2013-12-24 2022-09-21 Sony Depthsensing Solutions A time-of-flight camera system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030076484A1 (en) * 2000-11-09 2003-04-24 Canesta, Inc. Systems for CMOS-compatible three-dimensional image sensing using quantum efficiency modulation
CN102741651A (zh) * 2010-02-01 2012-10-17 微软公司 用于飞行时间测距系统的多个同步光源
CN101866056A (zh) * 2010-05-28 2010-10-20 中国科学院合肥物质科学研究院 基于led阵列共透镜tof深度测量的三维成像方法和系统
CN103026256A (zh) * 2010-07-06 2013-04-03 梅卡雷斯系统有限责任公司 具有补偿光源的光电子测量装置
CN104838284A (zh) * 2012-11-08 2015-08-12 蓝泰科尼克有限公司 用于至少两个ToF相机的拍摄方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3364643A4 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110945380A (zh) * 2017-07-18 2020-03-31 Lg伊诺特有限公司 ToF模块以及使用ToF模块的对象识别装置
US11782161B2 (en) 2017-07-18 2023-10-10 Lg Innotek Co., Ltd. ToF module and object recognition device using ToF module
CN110945380B (zh) * 2017-07-18 2023-10-24 Lg伊诺特有限公司 ToF模块以及使用ToF模块的对象识别装置
CN111741284A (zh) * 2019-03-25 2020-10-02 华为技术有限公司 图像处理的装置和方法
US12096134B2 (en) 2019-03-25 2024-09-17 Huawei Technologies Co., Ltd. Big aperture blurring method based on dual cameras and TOF
CN114972468A (zh) * 2022-05-26 2022-08-30 杭州海康机器人技术有限公司 一种深度图像获取方法、装置及电子设备

Also Published As

Publication number Publication date
CN106612387B (zh) 2019-05-21
EP3364643A1 (en) 2018-08-22
EP3364643B1 (en) 2021-09-08
US20180300891A1 (en) 2018-10-18
US10713804B2 (en) 2020-07-14
CN106612387A (zh) 2017-05-03
EP3364643A4 (en) 2019-04-03

Similar Documents

Publication Publication Date Title
WO2017063435A1 (zh) 一种组合深度图获得方法及深度相机
US11310479B2 (en) Non-uniform spatial resource allocation for depth mapping
CN116194866B (zh) 使用6dof姿态信息对准来自分离相机的图像
US10430956B2 (en) Time-of-flight (TOF) capturing apparatus and image processing method of reducing distortion of depth caused by multiple reflection
CN106548489B (zh) 一种深度图像与彩色图像的配准方法、三维图像采集装置
TWI624170B (zh) 影像掃描系統及其方法
Staranowicz et al. Practical and accurate calibration of RGB-D cameras using spheres
JP7145432B2 (ja) 投影システム、画像処理装置および投影方法
WO2018201585A1 (zh) 用于3d成像的激光阵列
US11195259B2 (en) Apparatus and method for dynamic multi-camera rectification using depth camera
US20160105645A1 (en) Identification device, method, and computer program product
WO2018216341A1 (ja) 情報処理装置、情報処理方法、及びプログラム
US10705217B2 (en) Controlling multiple imaging sensors
WO2017169723A1 (ja) 画像処理装置および方法
WO2018216342A1 (ja) 情報処理装置、情報処理方法、及びプログラム
TWI567473B (zh) 投影對準技術
US11391843B2 (en) Using time-of-flight techniques for stereoscopic image processing
US10735665B2 (en) Method and system for head mounted display infrared emitter brightness optimization based on image saturation
WO2019047841A1 (zh) 一种解决3d视觉设备之间相互干扰的方法及装置
CN107111862A (zh) 低功率dma标记
JP2017125764A (ja) 物体検出装置、及び物体検出装置を備えた画像表示装置
US20250227200A1 (en) Sensory Configurations for Enabling Autonomous Navigation
TWI831495B (zh) 定位系統及定位方法
TWI549502B (zh) 影像相位補正方法
US20250045953A1 (en) Association matching method and association matching system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16854817

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 15767607

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2016854817

Country of ref document: EP