WO2020103427A1 - 对象检测方法、相关设备及计算机存储介质 - Google Patents

对象检测方法、相关设备及计算机存储介质

Info

Publication number
WO2020103427A1
WO2020103427A1 PCT/CN2019/090696 CN2019090696W WO2020103427A1 WO 2020103427 A1 WO2020103427 A1 WO 2020103427A1 CN 2019090696 W CN2019090696 W CN 2019090696W WO 2020103427 A1 WO2020103427 A1 WO 2020103427A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
pixel
dimensional coordinates
coordinate system
detected
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/090696
Other languages
English (en)
French (fr)
Inventor
方民权
李跃亮
李怀洲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP19886206.2A priority Critical patent/EP3876141B1/en
Publication of WO2020103427A1 publication Critical patent/WO2020103427A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/251—Fusion techniques of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • the present invention relates to the field of computer technology, and in particular, to object detection methods, related devices, and computer storage media.
  • the safe driving of vehicles mainly depends on the perception and detection of the surrounding environment of driving.
  • the lidar can accurately restore and construct a three-dimensional point cloud map of the scene, thereby achieving object detection and recognition in the scene.
  • lidar is easily affected by weather conditions and has a limited detection range. This will reduce the accuracy of object detection. For example, in poor weather conditions such as foggy and rainy days, the detection range of the lidar sharply decreases, which affects the accuracy of object detection. At the same time, the point cloud data obtained when using lidar for object detection is very sparse, it is difficult to achieve object detection based on sparse point cloud data.
  • Embodiments of the present invention disclose object detection methods, related equipment, and computer storage media, which can solve the problems of low accuracy of object detection and difficulty in object detection in the prior art.
  • an embodiment of the present invention discloses an object detection method.
  • the method includes: a computing device determines an object area of a binocular image, and acquires two-dimensional m pixel pairs from the object area of the binocular image Coordinates, and further determine the three-dimensional coordinates of the object to be detected in the camera coordinate system according to the two-dimensional coordinates of the m pixel pairs.
  • the camera coordinate system refers to a coordinate system formed by using any one of the binocular cameras as the origin.
  • the computing device converts the three-dimensional coordinates of the object to be detected in the camera coordinate system into data under the coordinate system where the laser point cloud data of the object to be detected is merged with the laser point cloud data into merged point cloud data, where Laser point cloud data is data obtained by scanning the object to be detected using lidar.
  • the computing device determines the detection object based on the merged point cloud data.
  • the binocular image in the present invention includes a first image and a second image collected by a binocular camera, each pair of m pixel pairs includes two pixels, one pixel is from the first image The other pixel is from the second image, and the pixel of each pixel pair corresponds to the same feature of the object area in the first image and the second image.
  • these two pixels are mapped pixel pairs of the object area of the binocular image, and any one pixel is the mapped pixel of the other.
  • the computing device may determine the three-dimensional coordinates of the object to be detected in the image coordinate system according to the two-dimensional coordinates of the m pixel point pairs, where the to-be-detected
  • the three-dimensional coordinates of the object in the image coordinate system include the two-dimensional coordinates and depth values of the object to be detected in the coordinate system established according to the first image.
  • the computing device converts the three-dimensional coordinates of the object to be detected in the image coordinate system into the camera coordinate system according to the focal length of the binocular camera and the offset of the origin in the image coordinate system relative to the origin of the camera coordinate system Three-dimensional coordinates.
  • the computing device calculates m based on the two-dimensional coordinates of m pixel pairs, the baseline and the focal length of the binocular camera
  • the depth value of each pixel pair in each pixel pair, the two-dimensional coordinates of the pixel pair include the two-dimensional coordinates of the pixel pair in one pixel in the first image and the pixel pair in the second image The two-dimensional coordinates of a pixel.
  • the computing device determines the first pixel according to the depth value of each pixel pair in the m pixel pair, wherein the depth value of the first pixel does not exceed the first preset value, and the first preset value may specifically be
  • the system customizes the settings, such as the experience value set by the user, or the value obtained according to some series of statistical data.
  • the three-dimensional coordinates of the object to be detected in the image coordinate system include the three-dimensional coordinates of the first pixel in the image coordinate system.
  • the depth value of the first pixel specifically does not exceed the average depth value and the second preset value
  • the average depth value is the average value obtained from the depth values of each pixel pair in the m pixel pairs.
  • the second preset value may specifically be a numerical value set by the system.
  • the second preset value may be a depth standard deviation, and the depth standard deviation is specifically for each pixel pair in the m pixel pair The standard deviation of the depth value.
  • the computing device interpolates the first pixel point according to a preset interpolation algorithm to obtain the first interpolation pixel Point, the three-dimensional coordinates of the first interpolation pixel point in the image coordinate system are included in the three-dimensional coordinates of the object to be detected in the image coordinate system, that is, the three-dimensional coordinates of the object to be detected in the image coordinate system include the first interpolation pixel The three-dimensional coordinates of the point in image coordinates.
  • the merged point cloud data further includes second interpolation The three-dimensional coordinates of the pixel point in the coordinate system where the laser point cloud data is located, wherein the second interpolated pixel point is specifically the coordinates of the laser point cloud data and the converted laser point cloud data of the object to be detected by the computing device using a preset interpolation algorithm Pixels obtained by interpolating data under the system.
  • the binocular image is an image acquired by the binocular camera using the distortion parameters of the binocular image and the external parameter matrix The image obtained after processing.
  • an embodiment of the present invention provides a computing device, and the apparatus includes a functional module or unit for performing the method described in the first aspect or any possible implementation manner of the first aspect above.
  • an embodiment of the present invention provides a computing device, including: a processor, a memory, a communication interface, and a bus; a processor, a communication interface, and a memory communicate with each other through a bus; a communication interface is used to receive and send data; and a memory , Used to store instructions; a processor, used to call instructions in the memory, to execute the method described in the first aspect or any possible implementation manner of the first aspect.
  • a computer non-transitory storage medium in a fourth aspect, storing program code for object detection.
  • the program code includes instructions for performing the method described in the first aspect or any possible implementation manner of the first aspect.
  • a chip product is provided to perform the method in the first aspect or any possible implementation manner of the first aspect.
  • FIGS 1A-1B are schematic diagrams of two possible network frameworks provided by embodiments of the present invention.
  • FIG. 2 is a schematic flowchart of an object detection method provided by an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of a scene taken by a binocular camera provided by an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a corrected binocular image provided by an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of an object area of an object to be detected in a binocular image provided by an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of pixel pairs in an object area of a binocular image provided by an embodiment of the present invention.
  • FIG. 7A is a schematic diagram of a depth value calculation provided by an embodiment of the present invention.
  • 7B is a schematic diagram of feature pixel points in different coordinate systems provided by an embodiment of the present invention.
  • FIG. 8 is a schematic diagram of a scene regarding a camera coordinate system and a lidar coordinate system provided by an embodiment of the present invention.
  • FIG. 9 is a schematic diagram of merging point cloud data provided by an embodiment of the present invention.
  • FIG. 10 is a schematic diagram of merged point cloud data clustering provided by an embodiment of the present invention.
  • FIG. 11 is a schematic structural diagram of an object detection device according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of a computing device according to an embodiment of the present invention.
  • the present invention provides an object detection method, a network framework to which the method is applicable, and related equipment.
  • FIG. 1A is a schematic diagram of a possible network framework provided by an embodiment of the present invention.
  • the network framework includes a vehicle 100, a binocular camera unit 102, a laser radar 103, and a computing device 104. among them,
  • the binocular camera unit 102 is used to collect binocular images.
  • the binocular camera unit may include two camera units, such as a left-eye camera unit and a right-eye camera unit.
  • the image collected by the left-eye camera unit may be referred to as a left-eye image
  • the image collected by the right-eye camera unit may be referred to as a right-eye image.
  • any camera unit included in the binocular camera unit may specifically be any one of the following: a driving recorder, a camera, a camera device, a camera, or other devices with camera functions.
  • the binocular camera unit may be specifically deployed in any one or more of the following devices: vehicle 100, computing device 104, or other devices (such as mobile phones, computers, etc.).
  • the illustration here takes the binocular camera unit deployed in the vehicle 100 as an example, and does not constitute a limitation.
  • the lidar 103 is used to scan and detect the object to be detected to obtain laser point cloud data of the object to be detected.
  • the laser point cloud data refers to point cloud data obtained by scanning the object to be detected using lidar.
  • the point cloud data is recorded in the form of points, and each point includes a three-dimensional coordinate.
  • the three-dimensional coordinate is used to describe the point.
  • there can also be depth information, etc. which is not limited in the present invention.
  • the computing device 104 is used to obtain a binocular image, based on which the visual point cloud data of the object to be detected in the image is obtained, that is, the three-dimensional coordinates of the pixel points of the object to be detected in the image are converted to data. Furthermore, the computing device 104 may be based on the visual point cloud data, and optionally may also combine the laser point cloud data of the object to be detected to realize detection and recognition of the object to be detected. How to realize the detection and recognition of the object to be detected is described in detail below in the present invention.
  • the computing device 104 may be deployed on the vehicle 100 for use, or may be deployed separately (specifically described in detail in the example of FIG. 1B below), and the present invention is not limited.
  • the illustration here shows an example where the computing device 104 is deployed in the vehicle 100.
  • the computing device includes but is not limited to any of the following: mobile phones, tablet computers (table personal computers), personal digital assistants (personal digital assistants, PDAs), mobile internet devices (mobile internet devices (MIDs), wearable devices (wearable) device, vehicle-mounted equipment and other equipment that supports and network communication, etc.
  • the applicable scenes of the present invention include but are not limited to the field of automatic driving, obstacle detection, and any other scenes that require object detection.
  • a sensor is installed (or deployed) in the vehicle 100 for collecting sensor data associated with the sensor.
  • the sensor includes but is not limited to lidar, global positioning system (GPS), etc.
  • the vehicle 100 may call the lidar sensor to scan the object to be detected to obtain laser point cloud data of the object to be detected.
  • the vehicle 100 may send the collected laser point cloud data of the object to be detected to the computing device 104, so that the computing device 104 can implement detection and recognition of the object to be detected based on the laser point cloud data.
  • the object to be detected is set by the system, for example, it may include but not limited to vehicles, people, puppies, trees, buildings, cups or other objects.
  • the binocular camera unit is also deployed in the vehicle 100.
  • the binocular camera unit reference may be made to the relevant description in the foregoing embodiment, and details are not described here.
  • the deployment mode and deployment location are not limited by the present invention.
  • the binocular camera unit 102 may be specifically deployed outside the vehicle 100.
  • the binocular camera unit 102 is deployed outside the sunroof of the vehicle.
  • the installation angle of the binocular camera unit can also be customized according to the user's actual needs or user preferences, which is not limited in the present invention.
  • the vehicle 100 may include, but is not limited to, a truck, truck, train, car, motorcycle, off-road vehicle, agricultural vehicle, or other vehicles or equipment.
  • any two of the vehicle 100, the binocular camera unit 102, and the computing device 104 may use wired communication technology or wireless communication technology to implement network communication between the devices.
  • the wired communication technology may refer to communication between two devices through a network cable or an optical fiber.
  • the wireless communication technology includes but is not limited to global mobile communication system (global system for mobile communications, GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), Wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long-term evolution (long term evolution, LTE), wireless local area network (wireless local area) networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), close range Wireless communication technology (near field communication, NFC) and infrared technology (infrared, IR), etc.
  • GSM global system for mobile communications
  • GPRS general packet radio service
  • code division multiple access code division multiple access
  • CDMA Code division multiple access
  • Wideband code division multiple access Wideband code division multiple access
  • WCDMA Wideband code division multiple access
  • FIG. 1B is a schematic diagram of another possible network framework provided by an embodiment of the present invention.
  • the network framework includes a binocular camera unit 202 and a computing device 204. among them,
  • the binocular camera unit 202 is deployed in a local terminal device 206, which includes but is not limited to vehicles, mobile phones, tablet computers (table personal computers), personal digital assistants (personal digital assistants, PDAs), and mobile internet devices (mobile internet) device, MID), wearable devices (wearable devices) and other devices that support and network communication.
  • a local terminal device 206 includes but is not limited to vehicles, mobile phones, tablet computers (table personal computers), personal digital assistants (personal digital assistants, PDAs), and mobile internet devices (mobile internet) device, MID), wearable devices (wearable devices) and other devices that support and network communication.
  • the computing device 204 is deployed on the cloud network side, for example, illustrated in a cloud server.
  • the computing device 204 communicates with the terminal device where the binocular camera unit 202 is deployed through the network to obtain the binocular image collected by the binocular camera unit 202, and then realizes the detection and recognition of the object to be detected in the image. After the computing device completes the recognition of the object to be detected, the recognition result of the object to be detected can be obtained.
  • the computing device may send the recognition result of the object to be detected to the terminal device through the network, so that the terminal device can learn the recognition result of the object to be detected.
  • the relevant description in the embodiment described in FIG. 1A please refer to the relevant description in the embodiment described in FIG. 1A for details, and no further description is provided here.
  • FIG. 2 is a schematic flowchart of an object detection method according to an embodiment of the present invention.
  • the object detection method shown in FIG. 2 includes the following implementation steps:
  • Step S201 The computing device acquires a binocular image, which is an image including the first image and the second image collected by the binocular camera.
  • the computing device can collect the binocular image by calling the binocular camera installed in its own device or by calling the binocular camera in other devices.
  • the binocular image includes the first image and the second image.
  • the first image is a left-eye image
  • the second image is a right-eye image.
  • the first image is a right-eye image
  • the second image is a left-eye image.
  • the left-eye image refers to an image captured by a camera located on the left side of the binocular camera (that is, the left-eye camera).
  • the right-eye image refers to an image obtained by using a camera located on the right side of the binocular camera (that is, the right-eye camera).
  • FIG. 3 for a schematic diagram of a scene of a binocular camera.
  • the binocular images obtained by the binocular camera capturing the same object are also different.
  • the left-eye camera A captures the left-eye image A
  • the right-eye camera B captures the right-eye image B.
  • the binocular image may be a pre-processed image.
  • the computing device may perform preprocessing on the binocular images collected by the binocular camera, and the preprocessing includes but is not limited to de-distortion processing, translation processing, rotation processing, or other processing for correcting images.
  • the computing device uses the distortion parameters of the binocular camera to dedistort the acquired binocular image.
  • the computing device can also use the external parameter matrix (external parameters) of the binocular camera to correct the binocular image, so that the corrected epipolar lines of the binocular image are collinear.
  • the external parameter matrix represents the relative posture between the left-eye camera and the right-eye camera, which is specifically used to adjust the influence on the binocular image due to the difference in relative posture.
  • the corrected binocular image includes the first image and the second image.
  • the three epipolar lines on the first image are I 0 , I 1 and I 2 .
  • the three epipolar lines on the second image are I ′ 0 , I ′ 1 and I ′ 2 .
  • I 0 and I ′ 0 are on the same straight line (that is, I 0 and I ′ 0 are collinear)
  • I 1 and I ′ 1 are on the same straight line
  • I 2 and I ′ 2 are on the same straight line Straight line.
  • Step S202 The computing device detects and recognizes the object to be detected in the binocular image using a preset object detection algorithm to obtain the object area in the binocular image.
  • the object area is an image area occupied by the object to be detected in the binocular image.
  • the computing device may use the first object detection algorithm to detect and identify the object to be detected in the first image, so as to obtain the image area occupied by the object to be detected in the first image, that is, the object area.
  • the computing device may use the second object detection algorithm to detect and identify the object to be detected in the second image to obtain the image area occupied by the object to be detected in the second image, that is, the object area.
  • the first object detection algorithm and the second object detection algorithm are both object detection algorithms set by the system, which are used to detect and identify the object to be detected. They may be the same or different, and the invention is not limited.
  • the object detection algorithm includes but is not limited to any one or more of the following combinations: convolutional neural networks (convolutional neural networks, CNN), region-based convolutional neural networks (region-based convolutional neural networks, RCNN), A glimpse of neural networks (you only look at once, YOLO), object detection based on a single deep neural network (single shot multibox detector (SSD), deep feed forward neural network (deep feed forward, DFF), and recurrent neural network (RNN ), Or other algorithms for object detection, etc.
  • CNN convolutional neural networks
  • region-based convolutional neural networks region-based convolutional neural networks
  • RCNN region-based convolutional neural networks
  • a glimpse of neural networks you only look at once
  • YOLO object detection based on a single deep neural network
  • SSD single shot multibox
  • the object detection algorithm is an SSD algorithm.
  • the computing device can take the binocular images including the first image and the second image as the input of the SSD, and use the SSD to identify the image area where the image to be detected in the first image is the first image The object area in.
  • the image area where the image to be detected in the second image identified by the SSD is the object area in the second image.
  • the object area may also be marked with a preset frame for display to the user.
  • the preset frame can be customized by the system, for example, according to user needs or actual preferences, and its shape is not limited.
  • the preset frame can be any of the following: rectangle, diamond, circle, etc. .
  • the present invention does not limit the number of objects to be detected.
  • the computing device uses an object detection algorithm to recognize that the first image in the binocular image includes three objects to be detected, which are objects to be detected 1-3, respectively.
  • a rectangular frame outlines the object areas where the three objects to be detected are respectively: area A, area B, and area C.
  • the object area of the object 1 to be detected in the first image is area A
  • the object area of the object 2 to be detected in the first image is area B
  • the object area of the object 3 to be detected in the first image is area C
  • the computing device recognizes that the second image in the binocular image includes three objects to be detected. As shown in the rectangular frame, the object areas where the three objects to be detected are located are: Area A 'and Area B. 'And area C'.
  • n is a positive integer.
  • the computing device may also match the respective object areas of the n to-be-detected objects in the first image and the respective object areas of the n to-be-detected objects in the second image to obtain n object areas of the binocular image Correct.
  • each pair of object areas includes an object area where the same object to be detected is located in the first image and an object area where the object to be detected is located in the second image.
  • Each object area pair can be identified with the same label information, which is used to identify the object area where the same object to be detected is located in the binocular image. It may specifically be the name, number, letter symbol, etc. of the object to be detected, which is not limited in the present invention.
  • FIG. 5 includes three pairs of object regions. Specifically, the area A and the area A 'form an object area pair, that is, image areas where the object to be detected 1 is located in a binocular image, respectively. The area B and the area B 'form an object area pair, that is, image areas where the object to be detected 2 is located in a binocular image, respectively. The area C and the area C 'form a pair of object areas, that is, image areas where the objects 3 to be detected are respectively located in binocular images.
  • the computing device may perform the following step S203: perform feature extraction on each object region under each object region pair in the n object region pairs to obtain a double The characteristic pixel points in the target area of the target image.
  • n 1 (a pair of an object to be detected and an object region) is taken as an example to explain related content.
  • Step S203 The computing device performs feature extraction on the object area of the binocular image to obtain respective characteristic pixel points in the object area of the binocular image. That is, feature pixel points in the object area of the first image and feature pixel points in the object area of the second image are obtained.
  • the computing device may use the first feature extraction algorithm to perform feature extraction on the object area of the first image to obtain feature pixel points in the object area of the first image.
  • the computing device uses the second feature extraction algorithm to perform feature extraction on the object area of the second image to obtain feature pixel points in the object area of the second image.
  • the number of characteristic pixel points is not limited, and it may specifically be one or more. In practical applications, there are usually two or more feature pixels in the object area.
  • the first feature extraction algorithm and the second feature extraction algorithm are both set by the system, and are used for feature extraction of the image. They may be the same or different, and the invention is not limited thereto.
  • the feature extraction algorithm includes, but is not limited to, any one or more of the following combinations: Oriented brief feature (ORB) extraction algorithm, direction gradient histogram feature (histogram of oriented gradient (HOG) extraction algorithm, local two Value pattern feature (local binary pattern, LBP) extraction algorithm and Haar feature extraction algorithm, or other algorithms for image feature extraction, etc.
  • Step S204 The computing device determines m pairs of pixel points according to the characteristic pixel points in the object area of the binocular image.
  • the m pixel point pairs are pixel point pairs obtained by sampling the object area where the object to be detected is located, and each pixel point pair includes two pixel points, one pixel point from the first image and the other from the second image, and each The pixel points of the pixel point pair respectively correspond to the same feature of the object area in the first image and the second image. That is, each pixel pair includes the reference pixel in the object area of the first image and the mapped pixel in the object area of the second image corresponding to the reference pixel.
  • the reference pixel and the mapped pixel are both Feature pixels in the object area.
  • the computing device may further determine m pairs of pixel points, each pixel point pair including the same position corresponding to the first image and the second image (binocular image)
  • the characteristic pixels where they are located That is, each pixel includes a reference pixel in the object area of the first image and a mapped pixel corresponding to the reference pixel in the object area of the second image, the reference pixel and the mapped pixel correspond to each other, They have the same image features, and they are all feature pixels obtained after feature extraction.
  • the computing device may use a feature pixel included in the object area of any image in the binocular image as a reference pixel, and find the reference pixel corresponding to the mapped pixel in the object area of another image, and then The reference pixel and the mapped pixel form a pixel pair.
  • the computing device may use any characteristic pixel point in the object area of the first image as a reference pixel point, and find / calculate the mapped pixel point corresponding to the reference pixel point from the object area of the second image.
  • the computing device may use the above pixel point pair determination principle to determine m pixel point pairs from the object area of the binocular image, where m is a positive integer.
  • the computing device may use a preset distance algorithm and use the reference pixel as a reference to find the pixel with the smallest distance from the reference pixel in the object area of the second image as the mapped pixel corresponding to the reference pixel
  • the preset distance algorithms include but are not limited to Hamming distance, Euclidean distance, and cosine distance algorithms. Taking the Hamming distance algorithm as an example, the computing device may use the Hamming distance algorithm to calculate the Hamming code corresponding to each feature pixel in the reference pixel and the object area of the second image, and then calculate the Hamming code corresponding to each feature pixel The distance between the reference pixel and the Hamming code is selected from each feature pixel as the mapped pixel corresponding to the reference pixel.
  • FIG. 6 a schematic diagram of pixel pairs in the object area of the binocular image is shown.
  • the pixel point pair 1 includes a reference pixel point 1 and a mapped pixel point 1 '(illustrated as 1 and 1', respectively), wherein the mapped pixel point 1 'is specifically the reference pixel point 1 located in the object area of the first image Corresponding to the pixel located in the object area of the second image, or the reference pixel 1 is specifically the mapped pixel 1 'located in the object area of the second image corresponding to the pixel located in the object area of the first image.
  • Pixel point pair 2 includes reference pixel point 2 and mapped pixel point 2 '
  • pixel point pair 3 includes reference pixel point 3 and mapped pixel point 3', for pixel point pair 2 and pixel point pair 3, please refer to pixel point pair 1
  • the two pixel points included in each pixel point pair correspond to each other in position, which will not be repeated here.
  • Step S205 The computing device calculates the depth value of each pixel pair in the m pixel pairs according to the two-dimensional coordinates of the m pixel pairs.
  • the two-dimensional coordinates of the pixel point pair include the two-dimensional coordinates of the pixel point pair of one pixel point in the first image and the two-dimensional coordinate of the pixel point pair of another pixel point in the second image, which is the invention.
  • the computing device calculates the depth value of the pixel point pair according to the two-dimensional coordinates in the binocular image of the characteristic pixel point in each pixel point pair, which can be specifically the depth value of any characteristic pixel point in the pixel point pair
  • the characteristic pixel point may specifically be a reference pixel point or a mapped pixel point.
  • the computing device may separately obtain the two-dimensional coordinates of the reference pixel point in the pixel pair in the first image and the two-dimensional coordinates of the mapped pixel point in the second image. Further, the depth value of the pixel point pair is calculated according to the acquired two-dimensional coordinates of the reference pixel point in the pixel point pair and the two-dimensional coordinates of the mapped pixel point.
  • the corrected epipolar lines are collinear, for the two-dimensional coordinates of the reference pixel and the mapped pixel included in each pixel pair, their vertical coordinates are the same, horizontal The coordinates are different.
  • binocular camera center point of each and are O l O r as shown in FIG.
  • the distance B between O l and Or is called the baseline distance.
  • the computing device can use the following formula (1) to calculate and obtain the depth value d of the pixel point pair (specifically, the depth value of the characteristic pixel point in the pixel point pair), which indicates that the characteristic pixel point corresponds to the real world The distance between the P point and the binocular camera.
  • d represents the depth value.
  • f is the focal length of the binocular camera.
  • B represents the distance between the center of the left-eye camera and the center of the right-eye camera in the binocular camera, which can also be referred to as the baseline distance, specifically, O l O r as shown in FIG. 7A.
  • the area coordinate system is the The coordinate system established in the object area is not the image coordinate system established in the binocular image.
  • the computing device before calculating the depth value of the characteristic pixel point, the computing device also needs to convert the two-dimensional coordinates in the area coordinate system to the two-dimensional coordinates in the image coordinate system. It is convenient for the computing device to directly calculate the depth value of the characteristic pixel point according to the two-dimensional coordinates of the characteristic pixel point located in the image coordinate system in the pixel point alignment, that is, the characteristic pixel point corresponds to the two-dimensional coordinate in the binocular image.
  • FIG. 7B a schematic diagram of two-dimensional coordinates where a certain pixel in the pixel pair is located in the area coordinate system and the image coordinate system, respectively.
  • FIG. 7B shows the two-dimensional coordinates of the reference pixel point under the first image (ie, the image coordinate system XOY) and the object area (ie, the area coordinate system X ′) located in the first image 2D coordinates under O'Y ').
  • FIG. 7B specifically shows the two-dimensional coordinates of the mapped pixel located under the second image (image coordinate system) and the two located under the object area of the second image (region coordinate system). Dimensional coordinates.
  • the two-dimensional coordinates of the feature pixel point under the area coordinate system X'O'Y 'are (x', y ').
  • the two-dimensional coordinates of the origin O '(0, 0) in the area coordinate system under the image coordinate system XOY are (x 0 , y 0 ).
  • the characteristic pixel point is located in the two-dimensional coordinate of the image coordinate system XOY, that is, the two-dimensional coordinate corresponding to the characteristic pixel point in the binocular image is (x, y).
  • x x 0 + x '
  • y y 0 + y'.
  • Step S206 The computing device determines the visual point cloud data of the object to be detected according to the depth value of each pixel pair in the m pixel pairs.
  • the computing device After the computing device obtains the depth value of each pixel pair in the m pixel pairs, it can combine the two-dimensional coordinates of the pixel pair to obtain the three-dimensional coordinates of the feature pixels included in the m pixel pairs.
  • the computing device may obtain the value according to the depth value of the reference pixel in the pixel pair and the two-dimensional coordinates of the reference pixel in the first image The three-dimensional coordinates of the reference pixel in the pixel pair.
  • the computing device may obtain the two-dimensional coordinates in the second image corresponding to the mapped pixel according to the depth value of the mapped pixel in the pixel pair The three-dimensional coordinates of the mapped pixel in the pixel pair.
  • the depth value of the reference pixel is d
  • the two-dimensional coordinates of the reference pixel in the first image are (x 1 , y 1 )
  • the three-dimensional coordinates of the reference pixel point are (x 1 , y 1 , d).
  • the computing device may determine the visual point cloud data of the object to be detected according to the three-dimensional coordinates of the m pixel points to the characteristic pixel points included in each. Specifically, there are the following embodiments.
  • the computing device may directly use the three-dimensional coordinates of the m pixel points to the feature pixel points included in each as the visual point cloud data of the object to be detected.
  • the visual point cloud data and the laser point cloud data are point cloud data in the same coordinate system. Therefore, when the coordinate system where the three-dimensional coordinates of the characteristic pixel points included in the m pixels are different from the coordinate system corresponding to the laser point cloud data, the three-dimensional coordinates of the m pixels need to be included in the three-dimensional The coordinates are converted into corresponding three-dimensional coordinates in the same coordinate system as the laser point cloud data, and then the converted three-dimensional coordinates are used as the visual point cloud data of the object to be detected.
  • the three-dimensional coordinates of the reference pixel in any pixel pair are (x 1 , y 1 , d), which specifically represents the The reference pixel corresponds to the three-dimensional coordinates located in the first image (that is, the image coordinate system XOY). Because the image coordinate system XOY where the three-dimensional coordinates of the reference pixel point are located is different from the lidar coordinate system where the laser point cloud data corresponds, the computing device needs to convert the three-dimensional coordinates of the reference pixel point to the three-dimensional coordinates under the lidar coordinate system. That is, it is converted into three-dimensional coordinates in the same coordinate system as the laser point cloud data.
  • the lidar coordinate system is specifically the reference coordinate system when collecting laser point cloud data.
  • FIG. 8 a schematic diagram of a scene of a camera coordinate system and a lidar coordinate system.
  • the camera coordinate system is a coordinate system established in space with any camera in the binocular camera as the center.
  • the figure shows the camera coordinate system O "-X" Y "Z” created with the center of the left-eye camera as the origin.
  • the computing device may use the internal parameter matrix (also called internal parameter) of the binocular camera to calculate the three-dimensional coordinates (x 2 , y) of the reference pixel corresponding to the camera coordinate system O ”-X” Y ”Z” 2 , z 2 ), specifically the following formula (2) shows the calculation formula of (x 2 , y 2 , z 2 ).
  • c y is the offset of the origin O in the image coordinate system relative to the origin O ”of the imaging coordinate system in the Y” direction.
  • the computing device may convert the three-dimensional coordinates (x 2 , y 2 , z 2 ) of the reference pixel points corresponding to the camera coordinate system O ”-X” Y ”Z” into the lidar coordinate system O ”′-X Three-dimensional coordinates (x 3 , y 3 , z 3 ) under "'Y"' Z "'. That is, the computing device is located in the three-dimensional coordinate reference pixels of the first image (image coordinate system) in the (x 1, y 1, d ), is converted to the same three-dimensional coordinates in the coordinate data and the laser radar (x 3, y 3 , z 3 ).
  • the computing device may determine the relative attitude between the camera coordinate system O ”-X” Y ”Z” and the lidar coordinate system O ”'-X”' Y ”'Z”' , Calculate the transformation matrix T O ”O” 'from the camera coordinate system to the lidar coordinate system.
  • the computing device can use the following formula (3) to convert the three-dimensional coordinates (x 2 , y 2 , z 2 ) corresponding to the reference pixel point in the camera coordinate system into the three-dimensional coordinates (x 3 , in the lidar coordinate system) y 3 , z 3 ).
  • the computing device may filter the feature pixels included in each of the m pixels to obtain filtered pixels. Further, the computing device may determine the first pixel corresponding to the visual point cloud data according to the filtered pixels, and then use the three-dimensional coordinates of the first pixel as the visual point cloud data of the object to be detected.
  • the computing device may retain the feature pixel points that satisfy the preset condition, and use it as a filtering pixel point to exclude feature pixel points that do not satisfy the preset condition.
  • the preset condition is set by the system self-definition, and is used to filter characteristic pixel points.
  • the preset condition may specifically be that the depth value of the feature pixel does not exceed the first preset value, and the first preset value is set by the system self-defined, for example, an experience value set according to user experience, or according to a series of data Statistics, etc.
  • the first preset value may specifically be a sum of an average depth and a second preset value, and the average depth value specifically refers to an average of depth values obtained for each pixel pair in the m pixel pair value.
  • the second preset value can be specifically set by the system.
  • the second preset value may specifically be a preset standard deviation, and the preset standard deviation specifically refers to the standard deviation obtained for the depth value of each pixel pair in the m pixel pair.
  • the computing device may use the filtered pixel as the first pixel.
  • the computing device may use a first preset interpolation algorithm to interpolate the filtered pixels to obtain one or more first interpolated pixels.
  • the computing device may use the filtered pixel point and the first interpolated pixel point as the first pixel point.
  • the three-dimensional coordinates of the first pixel point are used as the visual point cloud data of the object to be detected.
  • the number of the first pixel points is not limited, and may be multiple here.
  • the first preset quantity is a positive integer set by the system, for example, set according to user experience or actual requirements.
  • the first preset interpolation algorithm is specifically an algorithm set by the system, which is used for pixel interpolation.
  • the computing device may calculate the three-dimensional coordinates of the interpolated pixel points according to the respective three-dimensional coordinates of the two filtered pixel points, and thereby obtain the interpolated pixel points.
  • the interpolation algorithm includes but is not limited to any one of the following: lagrange (lagrange) interpolation algorithm, Newton (newton) interpolation algorithm, piecewise linear interpolation algorithm, nonlinear interpolation algorithm, piecewise cubic Hermite interpolation Algorithms and spline interpolation algorithms.
  • the computing device may convert the three-dimensional coordinates of the first pixel point into the same coordinate system as the laser point cloud data (that is, the lidar coordinate system ) The three-dimensional coordinates of the second pixel. Furthermore, the converted three-dimensional coordinates of the second pixel point are used as visual point cloud data of the object to be detected.
  • the relevant explanation in the foregoing first embodiment and details are not repeated here.
  • the computing device may use a second preset interpolation algorithm to interpolate the second pixel to obtain one or more second interpolated pixels. Since the three-dimensional coordinates of the second pixel point are three-dimensional coordinates in the lidar coordinate system, the three-dimensional coordinates of the second interpolation pixel point are also three-dimensional coordinates in the lidar coordinate system. Therefore, the computing device may directly use the three-dimensional coordinates of the second interpolated pixel point and the three-dimensional coordinates of the second pixel point as visual point cloud data of the object to be detected.
  • the second preset quantity is also set by the system self-defined, and it may be the same as or different from the first preset quantity above, and the invention is not limited thereto.
  • the second preset interpolation algorithm may also be an algorithm that is customized by the system and used for pixel interpolation. For details, reference may be made to the foregoing related explanation about the first preset interpolation algorithm, which is not repeated here.
  • the first preset interpolation algorithm and the second preset interpolation algorithm may be the same or different, and the invention is not limited thereto.
  • the number of pixels included in the visual point cloud data is usually greater than or equal to a third preset number, which can be obtained through a series of statistical data statistics , Such as 200, etc.
  • Step S207 The computing device determines the object to be detected based on the visual point cloud data of the object to be detected and the laser point cloud data of the object to be detected.
  • the laser point cloud data is data obtained by scanning the object to be detected through the lidar.
  • the visual point cloud data and the laser point cloud data are both three-dimensional coordinate data (also referred to as three-dimensional data for short) in the same coordinate system.
  • the computing device may merge the visual point cloud data and laser point cloud data of the object to be detected to generate visual-laser point cloud data (also referred to as merged point cloud data). Then, a preset point cloud segmentation algorithm is used to detect and identify the object to be detected in the generated merged point cloud data to obtain the object to be detected.
  • the laser point cloud data refers to data obtained by scanning the object to be detected in advance using lidar.
  • the data is based on data collected in the lidar coordinate system.
  • the preset point cloud segmentation algorithm is customized by the system and is used for clustering and identification of point cloud data.
  • the point cloud segmentation algorithm includes, but is not limited to, European point cloud segmentation, angle cosine point cloud segmentation, K-Means point cloud segmentation, or other algorithms for clustering and identifying point cloud data.
  • the computing device After acquiring the visual point cloud data and laser point cloud data of the object to be detected, the computing device generates merged point cloud data (that is, the visual-laser point cloud data above) according to the visual point cloud data and the laser point cloud data.
  • the number of the objects to be detected is not limited, and it may be one or more.
  • FIG. 9 for a schematic diagram of a possible point cloud data merger. As shown in Fig. 9, solid point data represents visual point cloud data, and hollow point data represents laser point cloud data.
  • the computing device uses the point cloud segmentation algorithm to cluster the merged point cloud data to obtain the respective cluster point cloud data of each object to be detected. Then, the computing device performs contour feature extraction on the cluster point cloud data of each object to be detected to identify and obtain the object to be detected corresponding to the cluster point cloud data. For example, taking a point cloud segmentation algorithm as an Euclidean point cloud segmentation algorithm as an example, the computing device may cluster point cloud data whose Euclidean distance is less than or equal to a preset distance as clustered point cloud data of an object to be detected. Then, contour extraction of the cluster point cloud data of the object to be detected is performed to obtain the contour of the object to be detected depicted by the number of cluster point clouds, and then the object to be detected is obtained.
  • the object to be detected may specifically include but not limited to any one of the following: vehicles, people, puppies, trees, buildings, cups, and the like.
  • FIG. 10 specifically shows the cluster point cloud data of a vehicle.
  • the computing device may measure the rectangular frame corresponding to the cluster point cloud data, for example, the height of the matrix frame If the width and the width meet the external dimensions of the preset vehicle, it can be determined that the object to be detected corresponding to the cluster point cloud data is the vehicle.
  • the external dimensions of the preset vehicle can be customized by the system, for example, according to user experience.
  • the present invention proposes an object detection solution based on a mixture of visual point cloud data and laser point cloud data , Can improve the accuracy of object detection.
  • FIG. 11 is a schematic structural diagram of an object detection device according to an embodiment of the present invention.
  • the object detection device 600 includes a communication module 602 and a processing module 604.
  • the processing module 604 can be used to control and manage the operation of the object detection device 600.
  • the processing module 604 is used to perform steps S201-S207 in FIG. 2 and / or to perform other contents of the technology described in the text.
  • the communication module 602 is used to communicate with other modules or devices.
  • the communication module 602 communicates with other devices to obtain binocular images collected by the binocular cameras of other devices.
  • the object detection device 600 may further include a storage module 606.
  • the storage module 606 is used to store program codes and data of the load balancing device 600, for example, program codes for object detection.
  • the processing module 604 is used to call the program code in the storage module 606 to implement some or all of the implementation steps in the embodiment described in FIG. 2 above, and / or to perform other content steps of the technology described in the text.
  • the processing module 604 may be a processor or a controller, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), or an application-specific integrated circuit (Application-Specific Integrated Circuit (ASIC), Field Programmable Gate Array (Field Programmable Gate Array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in conjunction with the disclosure of the present invention.
  • the processor may also be a combination of computing functions, for example, including one or more microprocessor combinations, a combination of DSP and microprocessor, and so on.
  • the communication module 602 may be a communication interface, a transceiver, a transceiver circuit, etc., where the communication interface is a general term and may include one or more interfaces, such as an interface between the communication module and the processing module, a load balancing device and the user equipment Interface etc.
  • the storage module 606 may be a memory, or other services or modules for providing storage functions.
  • the object detection device involved in this embodiment of the present invention may be the object detection device (also referred to as a computing device) shown in FIG. 7.
  • the computing device 700 includes one or more processors 701, a communication interface 702, and a memory 703.
  • the processor 701, the communication interface 702, and the memory 703 may be connected through a bus or other means.
  • the bus 704 is connected as an example. among them:
  • the processor 701 may be composed of one or more general-purpose processors, such as a central processing unit (Central Processing Unit, CPU).
  • the processor 701 may be used to run any one or more of the following functional module programs in the relevant program code: a communication module, a processing module, and a storage module. That is to say, the processor 701 executing the program code can realize any one or more functions of the functional modules such as the communication module and the processing module.
  • a communication module such as a communication module
  • processing module such as a communication module
  • storage module such as the communication module and the processing module. That is to say, the processor 701 executing the program code can realize any one or more functions of the functional modules such as the communication module and the processing module.
  • the communication module and the processing module please refer to the relevant explanations in the foregoing embodiments.
  • the communication interface 702 may be a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or use a wireless local area network interface) for communicating with other modules / devices.
  • a wired interface such as an Ethernet interface
  • a wireless interface such as a cellular network interface or use a wireless local area network interface
  • the communication interface 702 in the embodiment of the present invention may be specifically used to receive binocular images sent by other devices.
  • the memory 703 may include volatile memory (Volatile Memory), such as random access memory (Random Access Memory, RAM); the memory may also include non-volatile memory (Non-Volatile Memory), such as read-only memory (Read-Only Memory, ROM), flash memory (Flash), hard disk (HDD), or solid-state drive (SSD); memory 703 may also include a combination of the above types of memory.
  • volatile memory such as random access memory (Random Access Memory, RAM
  • non-Volatile Memory such as read-only memory (Read-Only Memory, ROM), flash memory (Flash), hard disk (HDD), or solid-state drive (SSD)
  • memory 703 may also include a combination of the above types of memory.
  • the memory 703 may be used to store a set of program codes, so that the processor 701 calls the program codes stored in the memory 703 to implement the functions of the communication module and / or the processing module involved in the embodiments of the present invention.
  • FIG. 10 or FIG. 11 is only one possible implementation manner of the embodiments of the present application.
  • the load balancing device may further include more or fewer components, which is not limited herein.
  • Embodiments of the present invention also provide a computer non-transitory storage medium.
  • the computer non-transitory storage medium stores instructions, and when it runs on a processor, the method flow shown in FIG. 2 can be implemented.
  • An embodiment of the present invention also provides a computer program product.
  • the computer program product runs on a processor, the method flow shown in FIG. 2 can be implemented.
  • the steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present invention may be implemented by hardware, or by a processor executing software instructions.
  • the software instructions can be composed of corresponding software modules, which can be stored in random access memory (Random Access Memory, RAM), flash memory, read-only memory (Read Only Memory, ROM), and erasable programmable read-only memory ( Erasable Programmable ROM (EPROM), Electrically Erasable Programmable Read Only Memory (Electrically EPROM, EEPROM), registers, hard disk, removable hard disk, CD-ROM or any other form of storage medium known in the art.
  • An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium.
  • the storage medium may also be a component of the processor.
  • the processor and the storage medium may be located in the ASIC.
  • the ASIC may be located in the computing device.
  • the processor and the storage medium may also exist as discrete components in the computing device.
  • a person of ordinary skill in the art may understand that all or part of the process in the method of the above embodiments may be completed by instructing relevant hardware through a computer program, and the program may be stored in a computer-readable storage medium. When executed, it may include the processes of the foregoing method embodiments.
  • the foregoing storage media include various media that can store program codes, such as ROM, RAM, magnetic disks, or optical disks.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

本发明实施例公开了对象检测方法，包括：计算设备从双目图像的对象区域中获取m个像素点对的二维坐标，双目图像包括利用双目摄像头采集的第一图像和第二图像，每个像素点对的一个像素点来自第一图像，另一个来自第二图像，每个像素点对的像素点分别对应第一图像和第二图像中的对象区域的相同特征；根据m个像素点对的二维坐标，确定待检测对象在摄像坐标系下的三维坐标，并转换为待检测对象的激光点云数据所在坐标系下的数据，以和激光点云数据合并为合并点云数据，进而通过合并点云数据确定所述检测对象。通过实施本发明实施例，能够解决现有技术中存在的对象检测的准确度不高、或者对象检测难以实现等问题。

Description

对象检测方法、相关设备及计算机存储介质 技术领域
本发明涉及计算机技术领域，尤其涉及对象检测方法、相关设备及计算机存储介质。
背景技术
在无人驾驶领域，车辆的安全行驶主要依赖于对行车周围环境的感知和检测。目前，由于激光雷达具有较高地分辨率，能够较为准确地还原并构建场景的三维点云图，进而实现场景中的对象检测和识别。
然而在实践中发现，激光雷达很容易受天气状况的影响，且探测距离有限。这将降低对象检测的准确度。例如，在雾天、雨天等状况不佳的天气环境下，激光雷达的探测距离急剧减小，影响对象检测的准确度。同时，利用激光雷达进行对象检测时获得的点云数据很稀疏，很难依据稀疏的点云数据实现对象检测。
发明内容
本发明实施例公开了对象检测方法、相关设备及计算机存储介质，能够解决现有技术中存在的对象检测的精确度不高以及难以实现对象检测等问题。
第一方面，本发明实施例公开提供了一种对象检测方法，所述方法包括：计算设备确定双目图像的对象区域，从该双目图像的对象区域中获取m个像素点对的二维坐标，进而根据该m个像素点对的二维坐标确定待检测对象在摄像坐标系下的三维坐标。该摄像坐标系指的是以双目摄像头中的任一摄像头为原点形成的坐标系。进一步地，计算设备将待检测对象在摄像坐标系下的三维坐标转换为待检测对象的激光点云数据所在坐标系下的数据，并和激光点云数据合并为合并点云数据，其中，该激光点云数据为利用激光雷达扫描待检测对象获得的数据。最后，计算设备根据该合并点云数据确定出检测对象。可选地，本发明中的双目图像包括利用双目摄像头采集的第一图像和第二图像，m个像素点对中每个像素点对包括两个像素点，一个像素点来自第一图像，另一个像素点来自第二图像，且每个像素点对的像素点分别对应第一图像和第二图像中的对象区域的相同特征。换句话说，这两个像素点为双目图像的对象区域的映射像素点对，任一个像素点为另一个的映射像素点。
通过实施本发明实施例，能够解决现有技术中存在的对象检测的精确度不高、或者难以实现对象检测等问题。
结合第一方面，在第一方面的第一种可能的实施方式中，计算设备可根据m个像素点对的二维坐标，确定待检测对象在图像坐标系下的三维坐标，其中，待检测对象在图像坐标系下的三维坐标包括待检测对象在根据第一图像建立的坐标系的二维坐标和深度值。进一步地，计算设备根据双目摄像头的焦距和图像坐标系下原点相对于摄像坐标系的原点的偏移量，将待检测对象在图像坐标系下的三维坐标转换为待检测对象在摄像坐标系下的三维坐标。
结合第一方面的第一种可能的实施方式，在第一方面的第二种可能的实施方式中， 计算设备根据m个像素点对的二维坐标、双目摄像头的基线和焦距，计算m个像素点对中每个像素点对的深度值，该像素点对的二维坐标包括像素点对位于第一图像中的一个像素点的二维坐标和像素点对位于第二图像中的另一个像素点的二维坐标。计算设备根据m个像素点对中每个像素点对的深度值，确定第一像素点，其中，第一像素点的深度值不超过第一预设值，该第一预设值具体可为系统自定义设置的，例如用户设置的经验值，或者根据一些列统计数据统计获得的数值等。其中，该待检测对象在图像坐标系下的三维坐标包括该第一像素点在图像坐标系下的三维坐标。
结合第一方面的第一种或第二种可能的实施方式，在第一方面的第三种可能的实施方式中，第一像素点的深度值具体不超过平均深度值与第二预设值之和，该平均深度值为对m个像素点对中每个像素点对的深度值求取的平均值。该第二预设值具体可为系统自定义设置的数值，可选地，该第二预设值可为深度标准差，该深度标准差具体为对m个像素点对中每个像素点对的深度值求取的标准差。
结合第一方面的第二种或第三种可能的实施方式，在第一方面的第四种可能的实施方式中，计算设备根据预设插值算法对第一像素点插值，获得第一插值像素点，该第一插值像素点在图像坐标系下的三维坐标包括在待检测对象在图像坐标系下的三维坐标之内，即是待检测对象在图像坐标系下的三维坐标包括第一插值像素点在图像坐标下的三维坐标。
结合第一方面，或第一方面的第一种至第四种中任一种可能的实施方式，在第一方面的第五种可能的实施方式中，该合并点云数据还包括第二插值像素点在激光点云数据所在坐标系下的三维坐标，其中，该第二插值像素点具体为计算设备采用预设插值算法对激光点云数据以及转换的待检测对象的激光点云数据所在坐标系下的数据插值，获得的像素点。
结合第一方面，或第一方面的第一种至第五种中的任一种可能的实施方式中，双目图像为利用双目图像的畸变参数和外参矩阵对双目摄像头采集的图像进行处理后获得的图像。
第二方面，本发明实施例提供了一种计算设备，所述装置包括用于执行如上第一方面或第一方面的任意可能的实施方式中所描述的方法的功能模块或单元。
第三方面，本发明实施例提供了一种计算设备，包括：处理器，存储器，通信接口和总线；处理器、通信接口、存储器通过总线相互通信；通信接口，用于接收和发送数据；存储器，用于存储指令；处理器，用于调用存储器中的指令，执行上述第一方面或第一方面的任意可能的实施方式中所描述的方法。
第四方面，提供了一种计算机非瞬态(non-transitory)存储介质，所述计算机非瞬态存储介质存储了用于对象检测的程序代码。所述程序代码包括用于执行上述第一方面或第一方面的任意可能的实施方式中所描述的方法的指令。
第五方面，提供了一种芯片产品，以执行上述第一方面或第一方面的任意可能的实施方式中的方法。
本发明在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍。
图1A-图1B是本发明实施例提供的两种可能的网络框架示意图。
图2是本发明实施例提供的一种对象检测方法的流程示意图。
图3是本发明实施例提供的一种双目摄像头拍摄的场景示意图。
图4是本发明实施例提供的一种矫正后的双目图像的示意图。
图5是本发明实施例提供的一种双目图像中待检测对象的对象区域的示意图。
图6是本发明实施例提供的一种双目图像的对象区域中像素点对的示意图。
图7A是本发明实施例提供的一种深度值计算的示意图。
图7B是本发明实施例提供的一种不同坐标系下的特征像素点的示意图。
图8是本发明实施例提供的一种关于摄像坐标系和激光雷达坐标系的场景示意图。
图9是本发明实施例提供的一种合并点云数据的示意图。
图10是本发明实施例提供的一种合并点云数据聚类的示意图。
图11是本发明实施例提供的一种对象检测装置的结构示意图。
图12是本发明实施例提供的一种计算设备的结构示意图。
具体实施方式
下面结合附图，对本发明的实施例进行描述。
为解决现有技术中存在的对象检测的准确度不高、或者对象检测难以实现等问题，本发明提出一种对象检测方法、所述方法适用的网络框架以及相关设备。
首先，参见图1A是本发明实施例提供的一种可能的网络框架示意图。如图1，该网络框架中包括车辆100、双目摄像单元102、激光雷达103以及计算设备104。其中，
双目摄像单元102用于采集双目图像，该双目摄像单元可包括两个摄像单元，例如左目摄像单元和右目摄像单元。其中，左目摄像单元采集的图像可称为左目图像，右目摄像单元采集的图像可称为右目图像。在实际应用中，双目摄像单元中包括的任一摄像单元具体可为以下中的任一项：行车记录仪、摄像头、摄像装置、相机或者其他具有摄像功能的设备。该双目摄像单元具体可部署在以下设备中的任一个或多个中：车辆100、计算设备104或者其他设备(例如手机、电脑等)。这里图示以双目摄像单元部署在车辆100中为例，并不构成限定。激光雷达103用于对待检测对象进行扫描和检测，以获得待检测对象的激光点云数据。其中，该激光点云数据是指利用激光雷达扫描待检测对象获得的点云数据，该点云数据以点的形式记录，每个点包括有一个三维坐标，该三维坐标除了用于描述该点的几何位置之外，还可有深度信息等等，本发明对此不做限定。
计算设备104用于获取双目图像，基于该双目图像获得图像中待检测对象的视觉点云数据，即将图像中待检测对象的像素点的三维坐标转换为和激光点云数据相同坐标下的数据。进而，计算设备104可基于该视觉点云数据，可选地还可结合待检测对象的激光点云数据实现待检测对象的检测和识别。关于如何实现待检测对象的检测和识别，具体在本发明下文详述。
在实际应用中，计算设备104可部署在车辆100上使用，也可单独部署(具体在如 下图1B实例中详述)，本发明不做限定。这里图示以计算设备104部署在车辆100中为例示出。该计算设备包括但不限于以下中的任一项：手机、平板电脑(table personal computer)、个人数字助理(personal digital assistant，PDA)、移动上网装置(mobile internet device，MID)、可穿戴式设备(wearable device)、车载设备以及其他支持和网络通信的设备等。需要说明的是，本发明适用的场景包括但不限于自动驾驶领域、障碍物检测领域以及其他任何需要进行对象检测的场景。
车辆100中安装(或部署)有传感器，用于采集该传感器关联的传感数据。该传感器包括但不限于激光雷达、全球定位系统(global positioning system，GPS)等等。以传感器为激光雷达传感器为例，车辆100可调用激光雷达传感器对待检测对象进行扫描，以获得该待检测对象的激光点云数据。可选地，车辆100可将采集的待检测对象的激光点云数据发送给计算设备104，便于计算设备104基于该激光点云数据实现待检测对象的检测和识别。关于激光点云数据具体可参见前述实施例中的相关阐述，这里不再赘述。该待检测对象为系统自定义设置的，例如其可包括但不限于车辆、人、小狗、树木、建筑物、杯子或者其他对象等。
可选地，车辆100中还部署有该双目摄像单元，关于双目摄像单元具体可参见前述实施例中的相关阐述，这里不再赘述。关于车辆100中部署的各个设备或元件，其部署方式和部署位置，本发明并不做限定。如图1A中，以双目摄像单元102为例，该双目摄像单元102具体可部署在车辆100外部，图示中双目摄像单元102部署在车辆天窗的外部。可选地，双目摄像单元的安装角度也可根据用户实际需求或用户喜好自定义设置，本发明不做限定。
在实际应用中，车辆100可包括但不限于卡车、货车、火车、小轿车、摩托车、越野车、农用车、或者其他交通工具或设备。
本发明实施例中，车辆100、双目摄像单元102以及计算设备104中的任意两个设备可采用有线通信技术或者无线通信技术实现设备间的网络通信。其中，该有线通信技术可以是指两个设备之间通过网线或光纤等方式通信。该无线通信技术包括但不限于全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)、无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)、蓝牙(bluetooth，BT)、全球导航卫星系统(global navigation satellite system，GNSS)、调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)以及红外技术(infrared，IR)等等。
请参见图1B，是本发明实施例提供的另一种可能的网络框架示意图。该网络框架中包括双目摄像单元202以及计算设备204。其中，
双目摄像单元202部署在本地的终端设备206中，该终端设备包括但不限于车辆、手机、平板电脑(table personal computer)、个人数字助理(personal digital assistant，PDA)、移动上网装置(mobile internet device，MID)、可穿戴式设备(wearable device) 以及其他支持和网络通信的设备。
计算设备204部署在云端网络侧，例如图示部署在云端服务器中。计算设备204通过网络与部署双目摄像单元202的终端设备相互通信，以获取双目摄像单元202采集的双目图像，进而实现图像中待检测对象的检测和识别。在计算设备完成待检测对象的识别后，可获得待检测对象的识别结果。可选地，计算设备可通过网络将该待检测对象识别结果发送给终端设备，便于终端设备获知该待检测对象的识别结果。关于本发明实施例中未描述或未示出的内容，可具体参见前述图1A所述实施例中的相关阐述，这里不再赘述。
基于前述实施例，下面阐述本发明涉及的对象检测方法。请参见图2，是本发明实施例提供的一种对象检测方法的流程示意图。如图2所示的对象检测方法包括如下实施步骤：
步骤S201、计算设备获取双目图像，该双目图像为通过双目摄像头采集的包括第一图像和第二图像在内的图像。
本发明中，计算设备可通过调用自身设备安装的双目摄像头，或者调用其他设备中的双目摄像头采集双目图像，该双目图像包括第一图像和第二图像。当第一图像为左目图像时，第二图像为右目图像。当第一图像为右目图像时，第二图像为左目图像。该左目图像是指利用双目摄像头中位于左侧的摄像头(即左目摄像头)拍摄获得的图像。右目图像是指利用双目摄像头中位于右侧的摄像头(即右目摄像头)拍摄获得的图像。
参见图3示出一种双目摄像头摄像的场景示意图。如图3中，由于双目摄像头各自的拍摄视角不同，因此双目摄像头拍摄同一物体获得的双目图像也存在差异。如图所示，左目摄像头A拍摄获得左目图像A，右目摄像头B拍摄获得右目图像B。
可选地，该双目图像可为经过预处理后的图像。具体的，计算设备可对双目摄像头采集的双目图像进行预处理，该预处理包括但不限于去畸变处理、平移处理、旋转处理或者其他用于矫正图像的处理等。例如，计算设备利用双目摄像头的畸变参数对采集的双目图像进行去畸变处理。可选地，计算设备还可利用双目摄像头的外参矩阵(外参数)对双目图像进行矫正，使得矫正后的双目图像的对极线共线。其中，外参矩阵表示左目摄像头和右目摄像头之间的相对姿势，其具体用于调整由于相对姿势的差异对双目图像带来的影响。
参见如图4示出一种矫正后的双目图像的示意图。如图4所示，矫正后的双目图像中包括第一图像和第二图像。第一图像上的三条对极线分别为I 0、I 1和I 2。第二图像上的三条对极线分别为I' 0、I′ 1和I' 2。其中，如图所示I 0和I' 0处于同一条直线上(即I 0和I' 0共线)、I 1和I′ 1处于同一条直线上、I 2和I' 2处于同一条直线上。
步骤S202、计算设备采用预设的对象检测算法对双目图像中的待检测对象进行检测和识别，以获得双目图像中的对象区域。该对象区域为待检测对象在双目图像中所占的图像区域。
具体的，计算设备可采用第一对象检测算法对第一图像中的待检测对象进行检测和识别，以获得待检测对象在第一图像中所占的图像区域，即对象区域。相应地，计算设备可采用第二对象检测算法对第二图像中的待检测对象进行检测和识别，以获得待检测 对象在第二图像中所占的图像区域，即对象区域。
其中，第一对象检测算法和第二对象检测算法均为系统自定义设置的对象检测算法，用于检测和识别待检测对象。它们可以相同，也可不相同，本发明不做限定。该对象检测算法包括但不限于以下中的任一项或多项的组合：卷积神经网络(convolutional neural networks，CNN)、基于区域的卷积神经网络(region-based convolutional neural networks，RCNN)、一瞥神经网络(you only look once，YOLO)、基于单个深层神经网络的对象检测(single shot multibox detector，SSD)、深度前馈神经网络(deep feed forward，DFF)以及递归神经网络(recurrent network，RNN)、或其他用于对象检测的算法等。
举例来说，假设对象检测算法为SSD算法。计算设备在进行对象区域识别时，可将包括第一图像和第二图像在内的双目图像分别作为SSD的输入，利用SSD识别第一图像中待检测图像所在的图像区域即为第一图像中的对象区域。相应地，利用SSD识别的第二图像中待检测图像所在的图像区域即为第二图像中的对象区域。可选地，在计算设备利用对象检测算法识别出图像中待检测对象所在的对象区域后，还可利用预设框标出该对象区域，以展示给用户查看。该预设框可为系统自定义设置的，例如根据用户需求或实际爱好设置的，其形状并不做限定，例如该预设框可为以下中的任一项：矩形、菱形以及圆等等。
其中，本发明针对待检测对象的数量并不做限定。当该待检测对象的数量有多个时，则该待检测对象所在的对象区域同样有多个。请参见图5示出一种双目图像中待检测对象所在的对象区域的示意图。如图5中，计算设备利用对象检测算法识别出双目图像中的第一图像中包括3个待检测对象，分别为待检测对象1-3。如图用矩形框出了3个待检测对象各自所在的对象区域，分别为：区域A、区域B以及区域C。具体的，待检测对象1位于第一图像中的对象区域为区域A，待检测对象2位于第一图像中的对象区域为区域B，待检测对象3位于第一图像中的对象区域为区域C。同样地，计算设备识别双目图像中的第二图像中包括3个待检测对象，如图用矩形框示出了3个待检测对象各自所在的对象区域，分别为：区域A’、区域B’以及区域C’。
可选地，在计算设备识别到双目图像中待检测对象所在的对象区域后，若待检测对象的数量有n个时，则双目图像中待检测对象所在的对象区域同样有n个，n为正整数。进一步地，计算设备还可对第一图像中的n个待检测对象各自的对象区域和第二图像中的n个待检测对象各自的对象区域进行匹配，以获得双目图像的n个对象区域对。其中，每个对象区域对中包括同一待检测对象在第一图像中所在的对象区域和该待检测对象在第二图像中所在的对象区域。每个对象区域对可用同一标签信息标识，该标签信息用于标识同一待检测对象位于双目图像中的对象区域。其具体可为待检测对象的名称、编号以及字母符号等等，本发明并不做限定。
例如引用上述图5的例子，图5中包括有3个对象区域对。具体的，区域A和区域A’组成一个对象区域对，即是待检测对象1分别位于双目图像中的图像区域。区域B和区域B’组成一个对象区域对，即是待检测对象2分别位于双目图像中的图像区域。区域C和区域C’组成一个对象区域对，即是待检测对象3分别位于双目图像中的图像区域。
在计算设备匹配获得双目图像的n个对象区域对后，计算设备可执行如下S203的步 骤：对n个对象区域对中每个对象区域对下的每个对象区域进行特征提取，以获得双目图像的对象区域中各自的特征像素点。本发明下文以n＝1(一个待检测对象和一个对象区域对)为例进行相关内容的阐述。
步骤S203、计算设备对双目图像的对象区域进行特征提取，以获得双目图像的对象区域中各自的特征像素点。即获得第一图像的对象区域中的特征像素点和第二图像的对象区域中的特征像素点。
具体的，计算设备可采用第一特征提取算法对第一图像的对象区域进行特征提取，以获得该第一图像的对象区域中的特征像素点。同样地，计算设备采用第二特征提取算法对第二图像的对象区域进行特征提取，以获得该第二图像的对象区域中的特征像素点。该特征像素点的数量并不做限定，其具体可为一个或多个。在实际应用中，对象区域中的特征像素点通常为两个及以上。
其中，第一特征提取算法和第二特征提取算法均为系统自定义设置的，用于图像的特征提取，它们可以相同，也可不相同，本发明并不做限定。该特征提取算法包括但不限于以下中的任一项或多项的组合：面向简要特征(oriented brief，ORB)提取算法、方向梯度直方图特征(histogram of oriented gradient，HOG)提取算法、局部二值模式特征(local binary pattern，LBP)提取算法以及Haar特征提取算法、或者其他用于图像特征提取的算法等。
步骤S204、计算设备根据双目图像的对象区域中的特征像素点，确定m个像素点对。该m个像素点对为对待检测对象所在的对象区域采样获得的像素点对，每个像素点对包括两个像素点，一个像素点来自第一图像，另一个来自第二图像，且每个像素点对的像素点分别对应第一图像和第二图像中的对象区域的相同特征。即是，每个像素点对中包括第一图像的对象区域中的参考像素点和第二图像的对象区域中与该参考像素点对应的映射像素点，该参考像素点和映射像素点均为对象区域中的特征像素点。
计算设备在获得双目图像的对象区域各自的特征像素点后，可进一步确定出m个像素点对，每个像素点对包括同一位置对应在第一图像和第二图像(双目图像)中各自所在的特征像素点。也即是，每个像素点包括第一图像的对象区域中的参考像素点和第二图像的对象区域中与该参考像素点对应的映射像素点，该参考像素点和映射像素点相互对应，具有相同的图像特征，且均为特征提取后获得的特征像素点。
具体的，计算设备可以双目图像中的任一图像的对象区域中所包括的一特征像素点为参考像素点，寻找该参考像素点对应在另一图像的对象区域中的映射像素点，进而将该参考像素点和该映射像素点组成一个像素点对。例如，计算设备可将第一图像的对象区域中的任一特征像素点作为参考像素点，从第二图像的对象区域中寻找/计算出与该参考像素点对应位置所在的映射像素点。相应地，计算设备可采用上述像素点对确定原理，从双目图像的对象区域中确定出m个像素点对，m为正整数。具体实现中，计算设备可采用预设距离算法，以参考像素点为基准，从第二图像的对象区域中查找出与该参考像素点距离最小的像素点作为该参考像素点对应的映射像素点，所述预设距离算法包括但不限于汉明距离、欧式距离以及余弦距离算法等等。以汉明距离算法为例，计算设备可采用汉明距离算法，计算参考像素点和第二图像的对象区域中各特征像素点对应的汉明编码，然后计算各特征像素点对应的汉明编码与参考像素点的汉明编码之间的距离大小， 从各特征像素点中选取出距离最小的一个特征像素点作为该参考像素点对应的映射像素点。示例性地，如图6示出双目图像的对象区域中像素点对的示意图。如图6示出3个像素点对，具体包括像素点对1、像素点对2以及像素点对3。其中，像素点对1包括参考像素点1以及映射像素点1’(图示分别为1和1’)，其中，映射像素点1’具体为位于第一图像的对象区域中的参考像素点1对应在第二图像的对象区域中所在的像素点，或者参考像素点1具体为位于第二图像的对象区域中的映射像素点1’对应在第一图像的对象区域中所在的像素点。换句话说，参考像素点1以及映射像素点1’相互对应。像素点对2包括参考像素点2和映射像素点2’，像素点对3包括参考像素点3和映射像素点3’，关于像素点对2和像素点对3具体可参见前述像素点对1的相关阐述，每个像素点对中包括的两个像素点在位置上相互一一对应，这里不再赘述。
步骤S205、计算设备根据m个像素点对的二维坐标，计算m个像素点对中每个像素点对的深度值。该像素点对的二维坐标包括像素点对位于第一图像中的一个像素点的二维坐标和像素点对位于第二图像中的另一像素点的二维坐标，也即是本发明上文所述的参考像素点的二维坐标和映射像素点的二维坐标。计算设备根据每个像素点对中的特征像素点各自对应在双目图像中的二维坐标，计算该像素点对的深度值，具体可为该像素点对中任一特征像素点的深度值，该特征像素点具体可为参考像素点或映射像素点。具体的，计算设备可分别获取像素点对中的参考像素点位于第一图像中的二维坐标、映射像素点位于第二图像中的二维坐标。进而根据获取的该像素点对中参考像素点的二维坐标和映射像素点的二维坐标，计算该像素点对的深度值。
在实际应用中，由于矫正后的双目图像的对极线共线，因此针对每个像素对中包括的参考像素点和映射像素点各自的二维坐标而言，它们的纵坐标相同，横坐标不同。例如图7A所示，双目摄像头各自的中心点分别为如图所示的O l和O r。O l与O r之间的距离B称为基线距离。参考像素点P l的二维坐标为(x l，y l)，映射像素点P r的二维坐标为(x r，y r)，其中y l＝y r。相应地，计算设备可采用如下公式(1)计算获得该像素点对的深度值d(具体可为像素点对中特征像素点的深度值)，该深度值表示特征像素点对应在真实世界中的P点和双目摄像头之间的距离。
Figure PCTCN2019090696-appb-000001
其中，d表示深度值。f表示双目摄像头的焦距。B表示双目摄像头中左目摄像头的中心和右目摄像头的中心之间的距离，也可称基线距离，具体如图7A所示的O l O r。可选地，若像素点对中的特征像素点(具体可为参考像素点或映射像素点)的二维坐标为位于区域坐标系下的二维坐标，该区域坐标系为在双目图像的对象区域中建立的坐标系，并非在双目图像中建立的图像坐标系。相应地，在计算特征像素点的深度值之前，计算设备还需将区域坐标系下的二维坐标转换为图像坐标系下的二维坐标。便于计算设备直接依据像素点对中特征像素点位于图像坐标系下的二维坐标，即特征像素点对应位于双目图像中的二维坐标，来计算该特征像素点的深度值。
举例来说，如图7B示出了像素点对中某特征像素点分别位于区域坐标系和图像坐标系下的二维坐标的示意图。当该特征像素点为参考像素点时，图7B示出了参考像素点位于第一图像(即图像坐标系XOY)下的二维坐标以及位于第一图像的对象区域(即区域 坐标系X'O'Y')下的二维坐标。当该特征像素点为映射像素点时，图7B具体示出了映射像素点位于第二图像(图像坐标系)下的二维坐标以及位于第二图像的对象区域(区域坐标系)下的二维坐标。
如图7B中，特征像素点位于区域坐标系X'O'Y'下的二维坐标为(x’，y’)。该区域坐标系中原点O'(0，0)位于图像坐标系XOY下的二维坐标为(x 0，y 0)。相应地，该特征像素点位于图像坐标系XOY下的二维坐标，也即是该特征像素点对应在双目图像中的二维坐标为(x，y)。其中，x＝x 0+x'，y＝y 0+y'。
步骤S206、计算设备根据m个像素点对中每个像素点对的深度值，确定待检测对象的视觉点云数据。
计算设备获得m个像素点对中每个像素点对的深度值后，可结合该像素点对的二维坐标，从而获得该m个像素点对中各自所包括的特征像素点的三维坐标。当像素点对中包括的特征像素点为参考像素点时，计算设备可根据像素点对中的参考像素点的深度值，以及该参考像素点对应在第一图像中的二维坐标，获得该像素点对中参考像素点的三维坐标。当像素点对中包括的特征像素点为映射像素点时，计算设备可根据像素点对中的映射像素点的深度值，并结合该映射像素点对应在第二图像中的二维坐标，获得该像素点对中映射像素点的三维坐标。
举例来说，以一像素点对中的参考像素点为例，该参考像素点的深度值为d、该参考像素点位于第一图像中的二维坐标为(x 1，y 1)，则该参考像素点的三维坐标具体为(x 1，y 1，d)。
进一步地，计算设备可根据m个像素点对各自所包括的特征像素点的三维坐标，确定待检测对象的视觉点云数据。具体存在以下几种实施方式。
第一种，在不考虑计算精度以及计算量的情况下，计算设备可直接将m个像素点对各自所包括的特征像素点的三维坐标，作为待检测对象的视觉点云数据。
具体的，为保证对象检测的准确性，需保证视觉点云数据和激光点云数据均为同一坐标系下的点云数据。因此，当m个像素点对各自所包括的特征像素点的三维坐标所在的坐标系和激光点云数据对应的坐标系不同时，需将m个像素点对各自所包括的特征像素点的三维坐标转换为与激光点云数据相同坐标系下对应的三维坐标，进而将转换后的三维坐标作为待检测对象的视觉点云数据。
举例来说，以任一像素点对中的特征像素点为参考像素点为例，该任一像素点对中参考像素点的三维坐标为(x 1，y 1，d)，其具体表示该参考像素点对应位于第一图像中(即图像坐标系XOY)的三维坐标。由于该参考像素点的三维坐标所在的图像坐标系XOY和激光点云数据对应所在的激光雷达坐标系不同，因此计算设备需将参考像素点的三维坐标转换为激光雷达坐标系下的三维坐标，即是转换为与激光点云数据相同坐标系下的三维坐标。该激光雷达坐标系具体为激光点云数据采集时的参考坐标系。
请参见图8示出摄像坐标系和激光雷达坐标系的场景示意图。该摄像坐标系为以双目摄像头中的任一摄像头为中心，在空间中建立的坐标系。这里图中示出以左目摄像头的中心为原点，创建的摄像坐标系O”-X”Y”Z”。具体实现时，计算设备可利用双目摄像头的内参矩阵(也可称为内参数)计算该参考像素点对应在摄像坐标系O”-X”Y”Z”下的三维坐标(x 2，y 2，z 2)，具体如下公式(2)示出(x 2，y 2，z 2)的计算公式。
Figure PCTCN2019090696-appb-000002
方向上的偏移量。c y为图像坐标系中的原点O相对于摄像坐标系的原点O”在Y”方向上的偏移量。
进一步地，计算设备可将参考像素点对应在摄像坐标系O”-X”Y”Z”下的三维坐标(x 2，y 2，z 2)，转换为激光雷达坐标系O”'-X”'Y”'Z”'下的三维坐标(x 3，y 3，z 3)。也就是，计算设备将该参考像素点位于第一图像(图像坐标系)中的三维坐标(x 1，y 1，d)，转换为与激光雷达数据相同坐标下的三维坐标(x 3，y 3，z 3)。
例如，参见图8所示的场景图，计算设备可根据摄像坐标系O”-X”Y”Z”和激光雷达坐标系O”'-X”'Y”'Z”'之间的相对姿态，计算摄像坐标系到激光雷达坐标系的变换矩阵T O”O”'。相应地，计算设备可采用如下公式(3)，将参考像素点对应在摄像坐标系下的三维坐标(x 2，y 2，z 2)转换为激光雷达坐标系下的三维坐标(x 3，y 3，z 3)。
Figure PCTCN2019090696-appb-000003
第二种，在考虑计算精度和计算量的情况下，计算设备可对m个像素点对各自所包括的特征像素点进行筛选，以获得筛选像素点。进一步地，计算设备可根据筛选像素点，确定能作为视觉点云数据对应的第一像素点，进而将第一像素点的三维坐标作为待检测对象的视觉点云数据。
具体的，计算设备可将满足预设条件的特征像素点保留，并作为筛选像素点，将不满足预设条件的特征像素点剔除。该预设条件为系统自定义设置的，用于筛选特征像素点。例如，该预设条件具体可为特征像素点的深度值不超过第一预设值，该第一预设值为系统自定义设置的，例如根据用户经验设置的经验值，或者根据一些列数据统计获得的数值等。可选地，该第一预设值具体可为平均深度和第二预设值之和，该平均深度值具体是指对m个像素点对中每个像素点对的深度值求取的平均值。第二预设值具体可为系统自定义设置的。可选地，该第二预设值具体可为预设标准差，该预设标准差具体是指对m个像素点对中每个像素点对的深度值求取的标准差。
进一步地，计算设备可将筛选像素点作为第一像素点。可选地，当筛选像素点的数量较少，例如小于或等于第一预设数量。为保证对象检测的准确性，计算设备可采用第一预设插值算法对筛选像素点进行插值，以获得一个或多个第一插值像素点。相应地，计算设备可将筛选像素点以及第一插值像素点一同作为第一像素点。进而将第一像素点的三维坐标作为待检测对象的视觉点云数据。该第一像素点的数量并不做限定，这里可为多个。
其中，第一预设数量为系统自定义设置的正整数，例如根据用户经验或实际需求设置的等。第一预设插值算法具体为系统自定义设置的算法，用于像素点的插值。例如，计算设备可根据两个筛选像素点各自的三维坐标，采用插值算法计算获得插值像素点的三维坐标，从而获得该插值像素点。该插值算法包括但不限于以下中的任一项：拉格朗日(lagrange)插值算法、牛顿(newton)插值算法、分段线性插值算法、非线性插值算法、分段三次埃尔米特插值算法以及样条插值算法等。
可选地，当第一像素点的三维坐标不为激光雷达坐标系下的三维坐标，计算设备可第一像素点的三维坐标转换为与激光点云数据相同坐标系下(即激光雷达坐标系)第二像素点的三维坐标。进而将转换后的第二像素点的三维坐标作为待检测对象的视觉点云数据。关于三维坐标的转换具体可参见前述第一种实施例中的相关阐述，这里不再赘述。
可选地，计算设备在获得激光雷达坐标系下第二像素点的三维坐标后，如果第二像素点的数量较少，例如数量小于或等于第二预设数量。则计算设备可采用第二预设插值算法对第二像素点进行插值，以获得一个或多个第二插值像素点。由于第二像素点的三维坐标为激光雷达坐标系下的三维坐标，则第二插值像素点的三维坐标同样为激光雷达坐标系下的三维坐标。因此，计算设备可直接将第二插值像素点的三维坐标和第二像素点的三维坐标一同作为待检测对象的视觉点云数据。
其中，第二预设数量同样为系统自定义设置的，其与上文的第一预设数量可以相同，也可不同，本发明并不做限定。第二预设插值算法也可为系统自定义设置的算法，用于像素点的插值，其具体可参见前述关于第一预设插值算法的相关阐述，这里不做赘述。第一预设插值算法和第二预设插值算法可以相同，也可不相同，本发明并不做限定。
在实际应用中，为保证对象检测的准确性，视觉点云数据中包括的像素点的数量通常大于或等于第三预设数量，该第三预设数量可为经过一系列统计数据统计获得的，例如200等。
步骤S207、计算设备根据待检测对象的视觉点云数据和待检测对象的激光点云数据，确定待检测对象。其中，激光点云数据为通过激光雷达扫描待检测对象获得的数据。该视觉点云数据和激光点云数据均为同一坐标系下的三维坐标数据(也可简称为三维数据)。
计算设备可将待检测对象的视觉点云数据和激光点云数据合并生成视觉-激光点云数据(也可称为合并点云数据)。进而采用预设点云分割算法对生成的合并点云数据中的待检测对象进行检测和识别，以获得该待检测对象。
其中，激光点云数据是指预先利用激光雷达对待检测对象进行扫描获得的数据该数据是基于激光雷达坐标系下采集的数据，具体可参见前述图8所示。预设点云分割算法为系统自定义设置的，用于点云数据的聚类和识别。该点云分割算法包括但不限于欧式点云分割、夹角余弦点云分割、K-Means点云分割、或者其他用于实现点云数据聚类和识别的算法。
具体的，计算设备获取待检测对象的视觉点云数据和激光点云数据后，依据该视觉点云数据和激光点云数据生成合并点云数据(即上文的视觉-激光点云数据)。该待检测对象的数量并不做限定，其可为一个或多个。请参见图9示出一种可能的合并点云数据的示意图。如图9中，实心点数据表示视觉点云数据，空心点数据表示激光点云数据。
进一步地，计算设备利用点云分割算法对合并点云数据进行聚类，以获得每个待检 测对象各自的聚类点云数据。然后，计算设备对每个待检测对象的聚类点云数据进行轮廓特征提取，以识别获得该聚类点云数据对应的待检测对象。例如，以点云分割算法为欧式点云分割算法为例，计算设备可将欧式距离小于或等于预设距离的点云数据聚类在一起，作为一个待检测对象的聚类点云数据。进而对该待检测对象的聚类点云数据进行轮廓提取，以获得该聚类点云数所描绘的待检测对象的轮廓，进而获知该待检测对象。该待检测对象具体可包括但不限于以下中的任一项：车辆、人、小狗、树木、建筑物以及杯子等。
请参见图10示出一种合并点云数据聚类的示意图。图10中，具体示出一个车辆的聚类点云数据，计算设备在获得车辆的聚类点云数据后，可对聚类点云数据对应所在的矩形框进行测量，例如该矩阵框的高度和宽度满足预设车辆的外形尺寸，则可确定该聚类点云数据对应的待检测对象为车辆。该预设车辆的外形尺寸具体可为系统自定义设置的，例如根据用户经验设置的等。
通过实施本发明实施例，能够解决现有技术中存在的对象检测精确度不高、或者难以实现对象检测等问题，本发明提出一种基于视觉点云数据和激光点云数据混合的对象检测方案，能提高对象检测的精确度。
结合上文图1-图10所述实施例中的相关描述，下面介绍本发明适用的相关装置。请参见图11是本发明实施例提供的一种对象检测装置的结构示意图。该对象检测装置600包括通信模块602和处理模块604。其中，处理模块604可用于对对象检测装置600的动作进行控制和管理。例如，处理模块604用于执行图2中的步骤S201-步骤S207，和/或用于执行文本所描述的技术的其他内容。通信模块602用于与其他模块或设备进行通信，例如，通信模块602和其他设备通信，以获得其他设备的双目摄像头采集的双目图像等。
可选地，该对象检测装置600还可包括存储模块606。该存储模块606用于存储负载均衡装置600的程序代码和数据，例如存储用于对象检测的程序代码。处理模块604用于调用该存储模块606中的程序代码以实现如上图2所述实施例中的部分或全部实施步骤，和/或用于执行文本所描述的技术的其他内容步骤。
其中，处理模块604可以是处理器或控制器，例如可以是中央处理器(Central Processing Unit，CPU)，通用处理器，数字信号处理器(Digital Signal Processor，DSP)，专用集成电路(Application-Specific Integrated Circuit，ASIC)，现场可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、晶体管逻辑器件、硬件部件或者其任意组合。其可以实现或执行结合本发明公开内容所描述的各种示例性的逻辑方框，模块和电路。所述处理器也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，DSP和微处理器的组合等等。通信模块602可以是通信接口、收发器、收发电路等，其中，通信接口是统称，可以包括一个或多个接口，例如通信模块与处理模块之间的接口、负载均衡装置与用户设备之间的接口等。存储模块606可以是存储器，或者其他用于提供存储功能的服务或模块。
当处理模块604为处理器，通信模块602为通信接口，存储模块606为存储器时，本发明实施例所涉及对象检测装置可以为图7所示的对象检测装置(也可称为计算设备)。
请参见图11所示，计算设备700包括一个或多个处理器701、通信接口702和存储 器703，处理器701、通信接口702和存储器703可通过总线或者其它方式连接，本发明实施例以通过总线704连接为例。其中：
处理器701可以由一个或者多个通用处理器构成，例如中央处理器(Central Processing Unit，CPU)。处理器701可用于运行相关的程序代码中以下任一项或多项功能模块的程序：通信模块、处理模块以及存储模块等。也就是说，处理器701执行程序代码可以实现通信模块以及处理模块等功能模块中的任一项或多项的功能。其中，关于所述通信模块以及处理模块具体可参见前述实施例中的相关阐述。
通信接口702可以为有线接口(例如以太网接口)或无线接口(例如蜂窝网络接口或使用无线局域网接口)，用于与其他模块/设备进行通信。例如，本发明实施例中通信接口702具体可用于接收其他设备发送的双目图像等。
存储器703可以包括易失性存储器(Volatile Memory)，例如随机存取存储器(Random Access Memory，RAM)；存储器也可以包括非易失性存储器(Non-Volatile Memory)，例如只读存储器(Read-Only Memory，ROM)、快闪存储器(Flash Memory)、硬盘(Hard Disk Drive，HDD)或固态硬盘(Solid-State Drive，SSD)；存储器703还可以包括上述种类的存储器的组合。存储器703可用于存储一组程序代码，以便于处理器701调用存储器703中存储的程序代码以实现本发明实施例中涉及的通信模块和/或处理模块的功能。
需要说明的，图10或图11仅仅是本申请实施例的一种可能的实现方式，实际应用中，负载均衡装置还可以包括更多或更少的部件，这里不作限制。关于本发明实施例中未示出或未描述的内容，可参见前述图2所述实施例中的相关阐述，这里不再赘述。
本发明实施例还提供一种计算机非瞬态存储介质，所述计算机非瞬态存储介质中存储有指令，当其在处理器上运行时，图2所示的方法流程得以实现。
本发明实施例还提供一种计算机程序产品，当所述计算机程序产品在处理器上运行时，图2所示的方法流程得以实现。
结合本发明实施例公开内容所描述的方法或者算法的步骤可以硬件的方式来实现，也可以是由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(Random Access Memory，RAM)、闪存、只读存储器(Read Only Memory，ROM)、可擦除可编程只读存储器(Erasable Programmable ROM，EPROM)、电可擦可编程只读存储器(Electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、只读光盘(CD-ROM)或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于计算设备中。当然，处理器和存储介质也可以作为分立组件存在于计算设备中。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。

Claims (13)

  1. 一种对象检测方法，其特征在于，所述方法包括：
    计算设备从双目图像的对象区域中获取m个像素点对的二维坐标，所述双目图像的对象区域为待检测对象在所述双目图像中所占的图像区域，所述双目图像包括利用双目摄像头采集的第一图像和第二图像，每个像素点对的一个像素点来自所述第一图像，另一个来自所述第二图像，且所述每个像素点对的像素点分别对应所述第一图像和第二图像中的所述对象区域的相同特征；
    根据m个像素点对的二维坐标，确定所述待检测对象在摄像坐标系下的三维坐标，其中，所述摄像坐标系指的是以所述双目摄像头中的任一摄像头为原点形成的坐标系；
    将所述待检测对象在摄像坐标系下的三维坐标转换为所述待检测对象的激光点云数据所在坐标系下的数据，并与所述激光点云数据合并为合并点云数据，所述激光点云数据为利用激光雷达扫描所述待检测对象获得的数据；
    通过所述合并点云数据确定所述检测对象。
  2. 根据权利要求1所述的方法，其特征在于，所述根据m个像素点对的二维坐标，确定所述待检测对象在摄像坐标系下的三维坐标包括：
    根据所述m个像素点对的二维坐标，确定所述待检测对象在图像坐标系下的三维坐标，其中，所述待检测对象在图像坐标系下的三维坐标包括待检测对象在根据所述第一图像建立的坐标系的二维坐标以及深度值；
    根据所述双目摄像头的焦距以及所述图像坐标系的原点相对于摄像坐标系的原点的偏移量，将所述待检测对象在所述图像坐标系下的三维坐标转换为所述待检测对象在所述摄像坐标系下的三维坐标。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述m个像素点对的二维坐标，确定所述待检测对象在图像坐标系下的三维坐标，包括：
    根据所述m个像素点对的二维坐标、所述双目摄像头的基线以及焦距，计算所述m个像素点对中的每个像素点对的深度值，所述像素点对的二维坐标包括所述像素点对位于所述第一图像中的一个像素点的二维坐标和所述像素点对位于所述第二图像中的另一个像素点的二维坐标；
    根据所述m个像素点对中的每个像素点对的深度值，确定第一像素点，其中，所述第一像素点的深度值不超过第一预设值，所述待检测对象在图像坐标系下的三维坐标包括所述第一像素点在图像坐标系下的三维坐标。
  4. 根据权利要求2或3所述的方法，其特征在于，所述第一像素点的深度值不超过第一预设值，包括：
    所述第一像素点的深度值不超过平均深度值与第二预设值之和，其中，所述平均深度值是对所述m个像素点对中每个像素点对的深度值求取的平均值。
  5. 根据权利要求3或4所述的方法，其特征在于，所述方法还包括：
    根据预设插值算法对所述第一像素点插值，获得插值像素点；其中，所述待检测对象在图像坐标系下的三维坐标包括所述插值像素点在图像坐标系下的三维坐标。
  6. 根据权利要求1-5中任一项所述的方法，其特征在于，所述双目图像为利用所述双目图像的畸变参数和外参矩阵对所述双目摄像头采集的图像进行处理后获得的图像。
  7. 一种计算设备，其特征在于，所述计算设备包括通信模块和处理模块，其中，
    所述通信模块，用于从双目图像的对象区域中获取m个像素点对的二维坐标，所述双目图像的对象区域为待检测对象在所述双目图像中所占的图像区域，所述双目图像包括利用双目摄像头采集的第一图像和第二图像，每个像素点对的一个像素点来自所述第一图像，另一个来自所述第二图像，且所述每个像素点对的像素点分别对应所述第一图像和第二图像中的所述对象区域的相同特征；
    所述处理模块，用于根据m个像素点对的二维坐标，确定所述待检测对象在摄像坐标系下的三维坐标，其中，所述摄像坐标系指的是以所述双目摄像头中的任一摄像头为原点形成的坐标系；
    所述处理模块，还用于将所述待检测对象在摄像坐标系下的三维坐标转换为所述待检测对象的激光点云数据所在坐标系下的数据，并与所述激光点云数据合并为合并点云数据，所述激光点云数据为利用激光雷达扫描所述待检测对象获得的数据；
    所述处理模块，还用于通过所述合并点云数据确定所述检测对象。
  8. 根据权利要求7所述的计算设备，其特征在于，
    所述处理模块，具体用于根据所述m个像素点对的二维坐标，确定所述待检测对象在图像坐标系下的三维坐标，其中，所述待检测对象在图像坐标系下的三维坐标包括待检测对象在根据所述第一图像建立的坐标系的二维坐标以及深度值；
    所述处理模块，还用于根据所述双目摄像头的焦距以及所述图像坐标系的原点相对于摄像坐标系的原点的偏移量，将所述待检测对象在所述图像坐标系下的三维坐标转换为所述待检测对象在所述摄像坐标系下的三维坐标。
  9. 根据权利要求8所述的计算设备，其特征在于，
    所述处理模块，具体用于根据所述m个像素点对的二维坐标、所述双目摄像头的基线以及焦距，计算所述m个像素点对中的每个像素点对的深度值，所述像素点对的二维坐标包括所述像素点对位于所述第一图像中的一个像素点的二维坐标和所述像素点对位于所述第二图像中的另一个像素点的二维坐标；
    所述处理模块，还用于根据所述m个像素点对中的每个像素点对的深度值，确定第一像素点，其中，所述第一像素点的深度值不超过第一预设值，所述待检测对象在图像坐标系下的三维坐标包括所述第一像素点在图像坐标系下的三维坐标。
  10. 根据权利要求8或9所述的计算设备，其特征在于，所述第一像素点的深度值 不超过第一预设值具体为：所述第一像素点的深度值不超过平均深度值与第二预设值之和，其中，所述平均深度值是对所述m个像素点对中每个像素点对的深度值求取的平均值。
  11. 根据权利要求9或10所述的计算设备，其特征在于，
    所述处理模块，还用于根据预设插值算法对所述第一像素点插值，获得插值像素点；其中，所述待检测对象在图像坐标系下的三维坐标包括所述插值像素点在图像坐标系下的三维坐标。
  12. 根据权利要求7-11中任一项所述的计算设备，其特征在于，所述双目图像为利用所述双目图像的畸变参数和外参矩阵对所述双目摄像头采集的图像进行处理后获得的图像。
  13. 一种计算设备，其特征在于，包括处理器，存储器，通信接口和总线；所述处理器、所述通信接口和所述存储器通过总线相互通信；所述通信接口，用于接收和发送数据；所述存储器，用于存储指令；所述处理器，用于调用所述存储器中的指令，执行如上权利要求1-6中任一项所述方法。
PCT/CN2019/090696 2018-11-23 2019-06-11 对象检测方法、相关设备及计算机存储介质 Ceased WO2020103427A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP19886206.2A EP3876141B1 (en) 2018-11-23 2019-06-11 Object detection method, related device and computer storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811409485.4 2018-11-23
CN201811409485.4A CN109614889B (zh) 2018-11-23 2018-11-23 对象检测方法、相关设备及计算机存储介质

Publications (1)

Publication Number Publication Date
WO2020103427A1 true WO2020103427A1 (zh) 2020-05-28

Family

ID=66003845

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/090696 Ceased WO2020103427A1 (zh) 2018-11-23 2019-06-11 对象检测方法、相关设备及计算机存储介质

Country Status (3)

Country Link
EP (1) EP3876141B1 (zh)
CN (1) CN109614889B (zh)
WO (1) WO2020103427A1 (zh)

Cited By (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111666876A (zh) * 2020-06-05 2020-09-15 北京百度网讯科技有限公司 用于检测障碍物的方法、装置、电子设备和路侧设备
CN111723778A (zh) * 2020-07-07 2020-09-29 厦门大学 基于MobileNet-SSD的车辆测距系统及方法
CN111814659A (zh) * 2020-07-07 2020-10-23 杭州海康威视数字技术股份有限公司 一种活体检测方法、和系统
CN112070838A (zh) * 2020-09-07 2020-12-11 洛伦兹（北京）科技有限公司 基于二维-三维融合特征的对象识别与定位方法及装置
CN112102404A (zh) * 2020-08-14 2020-12-18 青岛小鸟看看科技有限公司 物体检测追踪方法、装置及头戴显示设备
CN112101209A (zh) * 2020-09-15 2020-12-18 北京百度网讯科技有限公司 用于路侧计算设备的确定世界坐标点云的方法和装置
CN112184828A (zh) * 2020-08-21 2021-01-05 北京百度网讯科技有限公司 激光雷达与摄像头的外参标定方法、装置及自动驾驶车辆
CN112287824A (zh) * 2020-10-28 2021-01-29 杭州海康威视数字技术股份有限公司 基于双目视觉的三维目标检测方法、装置及系统
CN112330721A (zh) * 2020-11-11 2021-02-05 北京市商汤科技开发有限公司 三维坐标的恢复方法及装置、电子设备和存储介质
CN112668418A (zh) * 2020-12-17 2021-04-16 洛阳语音云创新研究院 猪只怀孕检测方法、装置、设备及存储介质
CN112927298A (zh) * 2021-02-25 2021-06-08 上海钛米机器人股份有限公司 目标对象的定位方法及装置、电子设备、存储介质
CN113763562A (zh) * 2021-08-31 2021-12-07 哈尔滨工业大学（威海） 基于双目视觉的立面特征检测及立面特征处理方法
CN114049557A (zh) * 2021-11-10 2022-02-15 中国天楹股份有限公司 一种基于深度学习的垃圾分选机器人视觉识别方法
CN114219830A (zh) * 2021-11-09 2022-03-22 浙江大华技术股份有限公司 一种目标跟踪方法、终端及计算机可读存储介质
CN114648694A (zh) * 2022-03-01 2022-06-21 无锡雪浪数制科技有限公司 一种基于深度相机与机器视觉的海缆排缆间隙识别方法
CN114821987A (zh) * 2021-01-18 2022-07-29 漳州立达信光电子科技有限公司 提醒方法、装置及终端设备
CN114898044A (zh) * 2022-05-19 2022-08-12 同方威视技术股份有限公司 检测对象成像方法、装置、设备及介质
CN114994634A (zh) * 2022-05-18 2022-09-02 盐城中科高通量计算研究院有限公司 一种巡逻车激光雷达探坑算法
CN115175105A (zh) * 2022-07-20 2022-10-11 苏州微创畅行机器人有限公司 定位方法、装置及自动摆位装置
CN115187565A (zh) * 2022-07-20 2022-10-14 东南大学 一种水下桥墩病害识别定位方法、装置、电子设备及存储介质
CN115239620A (zh) * 2022-06-10 2022-10-25 重庆大学 一种基于点云的钢筋网片焊接点自动识别方法及系统
WO2022227678A1 (zh) * 2021-04-29 2022-11-03 达闼机器人股份有限公司 三维目标检测方法、抓取方法、装置及电子设备
CN115482203A (zh) * 2022-08-25 2022-12-16 上海大学 一种基于多维机器视觉的医废垃圾桶残留物检测方法
CN116309666A (zh) * 2023-03-13 2023-06-23 江苏润模汽车检测装备有限公司 一种基于3d视觉的钢筋导程测量方法
CN116524017A (zh) * 2023-03-13 2023-08-01 明创慧远科技集团有限公司 一种用于矿山井下检测识别定位系统
CN116563817A (zh) * 2023-04-14 2023-08-08 禾多科技（北京）有限公司 障碍物信息生成方法、装置、电子设备和计算机可读介质
CN116793228A (zh) * 2023-07-18 2023-09-22 福州大学 基于双目视觉的盾构隧道管片错台精准测量方法
CN116819487A (zh) * 2023-06-28 2023-09-29 上海双微导航技术有限公司 点云滤波方法及计算机设备
CN116894829A (zh) * 2023-07-21 2023-10-17 盛景智能科技(嘉兴)有限公司 焊缝缺陷检测的方法、装置、计算机设备及存储介质
CN117274573A (zh) * 2023-09-26 2023-12-22 西南交通大学 基于二维图像和三维点云的飞机表面孔洞定位测量方法
WO2024060792A1 (zh) * 2022-09-22 2024-03-28 中车资阳机车有限公司 一种分体式集装箱吊具的锁孔定位系统及方法
CN119245609A (zh) * 2024-12-05 2025-01-03 湖南大学 一种装配式桥梁的线形管理方法、装置及介质
CN119879721A (zh) * 2024-10-30 2025-04-25 浙江理工大学 一种基于双目视觉的回转体尺寸测量方法及系统
CN121032883A (zh) * 2025-03-27 2025-11-28 徐工消防安全装备有限公司 一种登高工作平台载荷估算与预警方法
CN121415036A (zh) * 2025-10-17 2026-01-27 南开大学 一种高大空间内火灾检测定位嵌入式系统及其方法

Families Citing this family (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109614889B (zh) * 2018-11-23 2020-09-18 华为技术有限公司 对象检测方法、相关设备及计算机存储介质
CN110110793B (zh) * 2019-05-10 2021-10-26 中山大学 基于双流卷积神经网络的双目图像快速目标检测方法
CN110349216A (zh) * 2019-07-18 2019-10-18 合肥泰禾光电科技股份有限公司 货箱位置检测方法及装置
CN112241987B (zh) * 2019-07-19 2023-11-14 杭州海康威视数字技术股份有限公司 确定防区的系统、方法、装置及存储介质
CN110374045B (zh) * 2019-07-29 2021-09-28 哈尔滨工业大学 一种智能化除冰方法
CN110658495B (zh) * 2019-08-22 2023-08-11 北京天睿视迅科技有限公司 一种位置检测方法及装置
WO2021095269A1 (ja) * 2019-11-15 2021-05-20 住友電気工業株式会社 情報生成装置、情報生成方法、及びコンピュータプログラム
CN111191582B (zh) * 2019-12-27 2022-11-01 深圳市越疆科技有限公司 三维目标检测方法、检测装置、终端设备及计算机可读存储介质
CN111462096A (zh) * 2020-04-03 2020-07-28 浙江商汤科技开发有限公司 三维目标检测方法及装置
CN111582240B (zh) * 2020-05-29 2023-08-08 上海依图网络科技有限公司 一种对象数量的识别方法、装置、设备和介质
CN111724432B (zh) * 2020-06-04 2023-08-22 杭州飞步科技有限公司 物体三维检测方法和装置
CN111950426A (zh) * 2020-08-06 2020-11-17 东软睿驰汽车技术（沈阳）有限公司 目标检测方法、装置及运载工具
CN112037280A (zh) * 2020-08-17 2020-12-04 北京声智科技有限公司 物体距离测量方法及装置
CN112740225B (zh) * 2020-09-30 2022-05-13 华为技术有限公司 一种路面要素确定方法及装置
CN113552125B (zh) * 2021-07-19 2023-05-16 清华大学 一种高温风洞环境下材料的烧蚀速率测试装置及方法
CN113850815B (zh) * 2021-11-29 2022-03-08 季华实验室 工件点云获取方法、装置、电子设备及存储介质
CN114488103A (zh) * 2021-12-31 2022-05-13 云鲸智能(深圳)有限公司 测距系统、测距方法、机器人、设备及存储介质
CN114580537B (zh) * 2022-03-04 2025-06-10 北京亮道智能汽车技术有限公司 一种点云数据处理方法及装置
CN115035492B (zh) * 2022-06-21 2024-01-23 苏州浪潮智能科技有限公司 车辆识别方法、装置、设备和存储介质
CN116012297B (zh) * 2022-12-02 2023-08-22 广东机电职业技术学院 基于太赫兹的铁轨表面损伤检测方法、系统和存储介质
CN115861741B (zh) * 2023-03-01 2023-10-17 小米汽车科技有限公司 目标标定方法、装置、电子设备、存储介质及车辆

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101532320B1 (ko) * 2014-04-18 2015-07-22 국방과학연구소 자율주행 무인차량에 탑재된 쌍안카메라를 이용한 동적물체 검출방법
CN108122280A (zh) * 2017-12-20 2018-06-05 北京搜狐新媒体信息技术有限公司 一种三维点云的重建方法及装置
CN108427438A (zh) * 2018-04-11 2018-08-21 北京木业邦科技有限公司 飞行环境检测方法、装置、电子设备及存储介质
CN108594244A (zh) * 2018-04-28 2018-09-28 吉林大学 基于立体视觉和激光雷达的障碍物识别迁移学习方法
CN108828606A (zh) * 2018-03-22 2018-11-16 中国科学院西安光学精密机械研究所 一种基于激光雷达和双目可见光相机联合测量方法
CN109614889A (zh) * 2018-11-23 2019-04-12 华为技术有限公司 对象检测方法、相关设备及计算机存储介质

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10582121B2 (en) * 2016-01-12 2020-03-03 Mitsubishi Electric Research Laboratories, Inc. System and method for fusing outputs of sensors having different resolutions
US11025882B2 (en) * 2016-04-25 2021-06-01 HypeVR Live action volumetric video compression/decompression and playback

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101532320B1 (ko) * 2014-04-18 2015-07-22 국방과학연구소 자율주행 무인차량에 탑재된 쌍안카메라를 이용한 동적물체 검출방법
CN108122280A (zh) * 2017-12-20 2018-06-05 北京搜狐新媒体信息技术有限公司 一种三维点云的重建方法及装置
CN108828606A (zh) * 2018-03-22 2018-11-16 中国科学院西安光学精密机械研究所 一种基于激光雷达和双目可见光相机联合测量方法
CN108427438A (zh) * 2018-04-11 2018-08-21 北京木业邦科技有限公司 飞行环境检测方法、装置、电子设备及存储介质
CN108594244A (zh) * 2018-04-28 2018-09-28 吉林大学 基于立体视觉和激光雷达的障碍物识别迁移学习方法
CN109614889A (zh) * 2018-11-23 2019-04-12 华为技术有限公司 对象检测方法、相关设备及计算机存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3876141A4 *

Cited By (51)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111666876A (zh) * 2020-06-05 2020-09-15 北京百度网讯科技有限公司 用于检测障碍物的方法、装置、电子设备和路侧设备
CN111666876B (zh) * 2020-06-05 2023-06-09 阿波罗智联(北京)科技有限公司 用于检测障碍物的方法、装置、电子设备和路侧设备
CN111723778A (zh) * 2020-07-07 2020-09-29 厦门大学 基于MobileNet-SSD的车辆测距系统及方法
CN111814659A (zh) * 2020-07-07 2020-10-23 杭州海康威视数字技术股份有限公司 一种活体检测方法、和系统
CN111814659B (zh) * 2020-07-07 2024-03-29 杭州海康威视数字技术股份有限公司 一种活体检测方法、和系统
CN111723778B (zh) * 2020-07-07 2022-07-19 厦门大学 基于MobileNet-SSD的车辆测距系统及方法
CN112102404A (zh) * 2020-08-14 2020-12-18 青岛小鸟看看科技有限公司 物体检测追踪方法、装置及头戴显示设备
CN112102404B (zh) * 2020-08-14 2024-04-30 青岛小鸟看看科技有限公司 物体检测追踪方法、装置及头戴显示设备
CN112184828B (zh) * 2020-08-21 2023-12-05 阿波罗智联(北京)科技有限公司 激光雷达与摄像头的外参标定方法、装置及自动驾驶车辆
CN112184828A (zh) * 2020-08-21 2021-01-05 北京百度网讯科技有限公司 激光雷达与摄像头的外参标定方法、装置及自动驾驶车辆
CN112070838A (zh) * 2020-09-07 2020-12-11 洛伦兹（北京）科技有限公司 基于二维-三维融合特征的对象识别与定位方法及装置
CN112070838B (zh) * 2020-09-07 2024-02-02 洛伦兹（北京）科技有限公司 基于二维-三维融合特征的对象识别与定位方法及装置
CN112101209B (zh) * 2020-09-15 2024-04-09 阿波罗智联(北京)科技有限公司 用于路侧计算设备的确定世界坐标点云的方法和装置
CN112101209A (zh) * 2020-09-15 2020-12-18 北京百度网讯科技有限公司 用于路侧计算设备的确定世界坐标点云的方法和装置
CN112287824A (zh) * 2020-10-28 2021-01-29 杭州海康威视数字技术股份有限公司 基于双目视觉的三维目标检测方法、装置及系统
CN112330721A (zh) * 2020-11-11 2021-02-05 北京市商汤科技开发有限公司 三维坐标的恢复方法及装置、电子设备和存储介质
CN112330721B (zh) * 2020-11-11 2023-02-17 北京市商汤科技开发有限公司 三维坐标的恢复方法及装置、电子设备和存储介质
CN112668418A (zh) * 2020-12-17 2021-04-16 洛阳语音云创新研究院 猪只怀孕检测方法、装置、设备及存储介质
CN114821987B (zh) * 2021-01-18 2024-04-30 漳州立达信光电子科技有限公司 提醒方法、装置及终端设备
CN114821987A (zh) * 2021-01-18 2022-07-29 漳州立达信光电子科技有限公司 提醒方法、装置及终端设备
CN112927298B (zh) * 2021-02-25 2024-06-11 上海钛米机器人股份有限公司 目标对象的定位方法及装置、电子设备、存储介质
CN112927298A (zh) * 2021-02-25 2021-06-08 上海钛米机器人股份有限公司 目标对象的定位方法及装置、电子设备、存储介质
WO2022227678A1 (zh) * 2021-04-29 2022-11-03 达闼机器人股份有限公司 三维目标检测方法、抓取方法、装置及电子设备
CN113763562A (zh) * 2021-08-31 2021-12-07 哈尔滨工业大学（威海） 基于双目视觉的立面特征检测及立面特征处理方法
CN113763562B (zh) * 2021-08-31 2023-08-29 哈尔滨工业大学（威海） 基于双目视觉的立面特征检测及立面特征处理方法
CN114219830A (zh) * 2021-11-09 2022-03-22 浙江大华技术股份有限公司 一种目标跟踪方法、终端及计算机可读存储介质
CN114049557A (zh) * 2021-11-10 2022-02-15 中国天楹股份有限公司 一种基于深度学习的垃圾分选机器人视觉识别方法
CN114648694A (zh) * 2022-03-01 2022-06-21 无锡雪浪数制科技有限公司 一种基于深度相机与机器视觉的海缆排缆间隙识别方法
CN114994634A (zh) * 2022-05-18 2022-09-02 盐城中科高通量计算研究院有限公司 一种巡逻车激光雷达探坑算法
CN114994634B (zh) * 2022-05-18 2024-05-28 盐城中科高通量计算研究院有限公司 一种巡逻车激光雷达探坑算法
CN114898044A (zh) * 2022-05-19 2022-08-12 同方威视技术股份有限公司 检测对象成像方法、装置、设备及介质
CN114898044B (zh) * 2022-05-19 2024-01-23 同方威视技术股份有限公司 检测对象成像方法、装置、设备及介质
CN115239620A (zh) * 2022-06-10 2022-10-25 重庆大学 一种基于点云的钢筋网片焊接点自动识别方法及系统
CN115187565A (zh) * 2022-07-20 2022-10-14 东南大学 一种水下桥墩病害识别定位方法、装置、电子设备及存储介质
CN115175105A (zh) * 2022-07-20 2022-10-11 苏州微创畅行机器人有限公司 定位方法、装置及自动摆位装置
CN115482203A (zh) * 2022-08-25 2022-12-16 上海大学 一种基于多维机器视觉的医废垃圾桶残留物检测方法
CN115482203B (zh) * 2022-08-25 2026-07-31 上海大学 一种基于多维机器视觉的医废垃圾桶残留物检测方法
WO2024060792A1 (zh) * 2022-09-22 2024-03-28 中车资阳机车有限公司 一种分体式集装箱吊具的锁孔定位系统及方法
CN116524017B (zh) * 2023-03-13 2023-09-19 明创慧远科技集团有限公司 一种用于矿山井下检测识别定位系统
CN116524017A (zh) * 2023-03-13 2023-08-01 明创慧远科技集团有限公司 一种用于矿山井下检测识别定位系统
CN116309666A (zh) * 2023-03-13 2023-06-23 江苏润模汽车检测装备有限公司 一种基于3d视觉的钢筋导程测量方法
CN116563817B (zh) * 2023-04-14 2024-02-20 禾多科技（北京）有限公司 障碍物信息生成方法、装置、电子设备和计算机可读介质
CN116563817A (zh) * 2023-04-14 2023-08-08 禾多科技（北京）有限公司 障碍物信息生成方法、装置、电子设备和计算机可读介质
CN116819487A (zh) * 2023-06-28 2023-09-29 上海双微导航技术有限公司 点云滤波方法及计算机设备
CN116793228A (zh) * 2023-07-18 2023-09-22 福州大学 基于双目视觉的盾构隧道管片错台精准测量方法
CN116894829A (zh) * 2023-07-21 2023-10-17 盛景智能科技(嘉兴)有限公司 焊缝缺陷检测的方法、装置、计算机设备及存储介质
CN117274573A (zh) * 2023-09-26 2023-12-22 西南交通大学 基于二维图像和三维点云的飞机表面孔洞定位测量方法
CN119879721A (zh) * 2024-10-30 2025-04-25 浙江理工大学 一种基于双目视觉的回转体尺寸测量方法及系统
CN119245609A (zh) * 2024-12-05 2025-01-03 湖南大学 一种装配式桥梁的线形管理方法、装置及介质
CN121032883A (zh) * 2025-03-27 2025-11-28 徐工消防安全装备有限公司 一种登高工作平台载荷估算与预警方法
CN121415036A (zh) * 2025-10-17 2026-01-27 南开大学 一种高大空间内火灾检测定位嵌入式系统及其方法

Also Published As

Publication number Publication date
CN109614889A (zh) 2019-04-12
CN109614889B (zh) 2020-09-18
EP3876141A1 (en) 2021-09-08
EP3876141A4 (en) 2022-01-05
EP3876141B1 (en) 2025-11-19

Similar Documents

Publication Publication Date Title
CN109614889B (zh) 对象检测方法、相关设备及计算机存储介质
US11205284B2 (en) Vehicle-mounted camera pose estimation method, apparatus, and system, and electronic device
CN110163930B (zh) 车道线生成方法、装置、设备、系统及可读存储介质
US11474247B2 (en) Methods and systems for color point cloud generation
CN110176032B (zh) 一种三维重建方法及装置
CN114419165B (zh) 相机外参校正方法、装置、电子设备和存储介质
CN107636679B (zh) 一种障碍物检测方法及装置
CN107636680B (zh) 一种障碍物检测方法及装置
JP7325332B2 (ja) 三次元点群データ生成方法、および、三次元点群データ生成装置
CN111950426A (zh) 目标检测方法、装置及运载工具
WO2020102944A1 (zh) 点云处理方法、设备及存储介质
CN109474780B (zh) 一种用于图像处理的方法和装置
CN111837158A (zh) 图像处理方法、装置、拍摄装置和可移动平台
CN115147328A (zh) 三维目标检测方法及装置
CN111047634B (zh) 场景深度的确定方法、装置、设备及存储介质
US11044452B2 (en) Imaging system and method
CN111738033B (zh) 基于平面分割的车辆行驶信息确定方法及装置、车载终端
CN106485753A (zh) 用于无人驾驶汽车的摄像机标定的方法和装置
CN116385994A (zh) 一种三维道路线提取方法及相关设备
CN111950428A (zh) 目标障碍物识别方法、装置及运载工具
CN117115242B (zh) 标志点的识别方法、计算机存储介质和终端设备
CN114611635B (zh) 一种对象的识别方法、装置、存储介质及电子装置
CN110895821A (zh) 图像处理装置、存储有图像处理程序的存储介质及驾驶支援系统
CN112668505A (zh) 基于路侧相机的外参的三维感知信息获取方法和路侧设备
CN109658451B (zh) 一种深度感知方法，装置和深度感知设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19886206

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019886206

Country of ref document: EP

Effective date: 20210531

WWG Wipo information: grant in national office

Ref document number: 2019886206

Country of ref document: EP