CN103473757B - Method for tracing object in disparity map and system - Google Patents

Abstract

A kind of method for tracing object and system are provided, and the method comprises: first detected object of detection in the first disparity map and the position of acquisition the first detected object; The position of the first detected object is converted to the first detection relative position of the coordinate of the first predetermined reference point based in the first disparity map; Pre-motion orientation based on the first detection relative position and the first detected object and predetermined movement velocity, estimate the position of estimating of estimating object in the second disparity map corresponding to first of the first detected object; Second detected object of detection in the second disparity map and the position of acquisition the second detected object; The position of the second detected object is converted to the second detection relative position of the coordinate of the second predetermined reference point based in the second disparity map; Detect relative position based on the first second of position and the second detected object of estimating of estimating object, determine the tracking and matching relation between the first detected object and the second detected object.

Description

Method for tracing object in disparity map and system

Technical field

The present invention relates to image processing field, and more specifically, relate to a kind of object in disparity map and followTrack method and system.

Background technology

Now, stereo-picture technology is an important branch of computer vision, and it can be under multiple conditionObtain neatly the Stereo Vision of scenery, especially binocular vision information, relatively monocular imageHaving incomparable advantage, is the forward position research direction of image processing and computer vision field. SpecificallyGround, for example, by steric information, carry out object (, the car on road, people etc.) such as disparity map and followTrack, also traffic monitoring, auxiliary driving, automobile detection and tracking, alarm drive system, robot are lookedFeel that there is very important practical valency in the fields such as navigation, industrial products detection, medical diagnosis, virtual realityValue and vast potential for future development.

The general principle of stereoscopic vision is to observe same scenery from two (or multiple) viewpoints, to obtainPerceptual image under different visual angles, by the position deviation between principle of triangulation computed image pixel (depending onPoor (disparity)) obtain the 3 D stereo information of scenery. The third dimension of this process and human visionKnow that process is similar. Conventionally, for example with two of left and right video camera come in measurement environment any point or one rightThe 3 D stereo information of elephant, thus disparity map obtained.

Disparity map is for example, to (image and right video camera that, left video camera is taken are taken with left and right imageImage) corresponding points abscissa difference form image. Certainly, in this area, there are many its other partyMethod obtains disparity map, is not repeated herein. Due to the range information that disparity map has comprised scene, therefore baseCarry out a series of application, particularly carry out image tracing in disparity map, be always in binocular vision researchFor active field.

In prior art, propose to utilize disparity map to carry out the certain methods of tracing object.

For example,, in disclosed U.S. Patent application US2011228100A1(topic on September 22nd, 2011For " ObjectTrackingDeviceandMethodofControllingOperationofthe Same ")In a kind of object tracking technique is disclosed. Three of its two imaging devices shootings of installing from for example vehicleIn dimensional object image, generate the disparity map of the parallax of the each pixel that shows this object, determine detection range,To eliminate other before tracing object, for example pedestrian on depth direction from the disparity map generatingObject then detects tracing object, for example pedestrian in definite detection range, like this, can preventThe people by bike who is positioned at before tracing object, for example pedestrian detected. But this technology is only consideredThe detection range of how to confirm tracing object on depth direction (be the distance between object and imaging deviceFrom scope), but this technology is not considered to take, the left and right vehicle wheel of disparity map is turned or vehicle pitchesTime impact on tracing object.

In addition, the US Patent No. 7974442B2(announcing on July 5th, 2011 is entitled as " VehicularVisionSystem ") in disclose a kind of for identify and classifying vehicle near the car of object (target)Vision system. This system comprises the sensing producing in order to generate near the image of the depth map of scene vehicleDevice array, processes this depth map and presenting in advance near the destination object that may occur itself and vehicleTemplate is compared, by the template presenting in advance and this depth map are matched to produce object listing, and shouldSystem is processed this object listing and is produced target sizes and classification assessment, then, and when this car of this gtoal settingWhen mobile, follow the tracks of this target, and the position of definite target, classification and speed. This patent is only used in advanceThe template that first presents to compare with depth map, the left and right vehicle wheel of also not considering to take disparity map turn orImpact on tracing object when vehicle pitches.

But taking in the Vehicle Driving Cycle process of disparity map, always may produce some problems affects and looksThe poor shooting of figure and the tracking of object. For example,, at the turn inside diameter of loading binocular camera or due to road surfaceUneven and produce while jolting, though this object with respect to the invariant position of this vehicle, captured looksThe picture position of the object showing in poor figure is also possible, and great changes will take place. In this case, existingThe technology proposing in technology can not be followed the tracks of this object more accurately.

Therefore, need to a kind ofly carry out the method and system of tracing object exactly based on disparity map.

Summary of the invention

In order to address the above problem, according to an aspect of the present invention, provide a kind of right in disparity mapImage tracing method, comprises the following steps: the first detecting step, detect in the first disparity map one orMultiple the first detected objects and obtain the position of each the first detected object; The first switch process, by eachThe position of the first detected object is converted to coordinate each of the first predetermined reference point based in the first disparity mapIndividual first detects relative position; Estimate step, based on each first detection relative position and each the first inspectionSurvey pre-motion orientation and the predetermined movement velocity of object, estimate in the second disparity map corresponding to each theEach of one detected object first estimated the position of estimating of object; The second detecting step, detects and looks secondOne or more the second detected objects in poor figure and obtain the position of each the second detected object; Second turnsChange step rapid, the position of each the second detected object is converted to based on the predetermined ginseng of second in the second disparity mapEach the second detection relative position of the coordinate of examination point; Determining step, first estimates object based on eachEstimate second of position and described each the second detected object and detect relative position, determine with each theOne tracking and matching of estimating between each first detected object and each the second detected object that object is corresponding is closedSystem.

Preferably, the coordinate of described the first predetermined reference point can be included in road in the first disparity mapAt least one in abscissa and the ordinate of end point, and the coordinate of described the second predetermined reference point comprisesAt least one in abscissa and the ordinate of the end point of the same road in the second disparity map.

Preferably, described the first detecting step can also obtain the size of each the first detected object, wherein,The size of each the first detected object is as each the first size of estimating object, and the second detecting step alsoObtain the size of each the second detected object.

Preferably, described determining step can based on each first estimate object estimate position and size withAnd the second detection relative position and size of each the second detected object, determine each the first detected objectAnd the tracking and matching relation between each second detected object.

Preferably, described determining step can comprise the following steps: first estimate the pre-of object based on eachEstimate second of position and size and described each the second detected object and detect relative position and size, forEach first estimates object, calculates each first and estimates object and each the second detected object on horizontal planeOverlapping area; If estimate one of object and second detected object and weigh on described horizontal plane for one firstFolded area maximum, determines with this and first estimates the first detected object that object is corresponding and described second and examineSurveying one of object mates.

Preferably, described determining step can also comprise the following steps: if one first estimate object withTwo above the second detected objects overlapping area on described horizontal plane is all maximum, calculates that this is first pre-Estimate above the second detected object of object and two overlapping area in depth plane; If this first estimate rightResemble and one of described two above second detected objects overlapping area maximum in depth plane, determine withThis first is estimated one of the first detected object that object is corresponding and described two above second detected objects and isJoin.

Preferably, described determining step can also comprise the following steps: calculate each described the of coupling notOne estimate object estimate second of position and each the second detected object not mating detect relative position itBetween minimum range; If the minimum range of this calculating is less than predetermined threshold, determines with this and first estimateThe first detected object that object is corresponding mates with second detected object with minimum range.

Preferably, described predetermined threshold can be based on estimating with described first the first detection that object is correspondingThe maximum movement speed of object is determined.

Preferably, described determining step can also comprise the following steps: if first detected object is notMate with any the second detected object, determine that this first detected object disappears in the second disparity map, withAnd if second detected object do not mate with any the first detected object, determine this second detect rightResemble in the second disparity map is new object.

Preferably, in the method, definite each first detected object and each the second detected object itBetween tracking and matching relation after, first-phase that can be based on each the first detected object to detection position andThe second-phase of each the second detected object mating with it, to detection position, is proofreaied and correct each the first detected objectPre-motion orientation and predetermined movement velocity, predetermined as each the second detected object mating with itThe direction of motion and predetermined movement velocity.

According to a further aspect in the invention, provide a kind of object-tracking systems in disparity map, comprising:The first checkout gear, detects one or more the first detected objects in the first disparity map and obtains eachThe position of the first detected object; The first conversion equipment, is converted to base by the position of each the first detected objectEach the first detection relative position of the coordinate of the first predetermined reference point in the first disparity map; Estimate dressPut pre-motion orientation based on each the first detection relative position and each the first detected object and predeterminedMovement velocity, estimate in the second disparity map corresponding to each the first detected object each first estimate rightElephant estimate position; Second detection device, detects one or more the second detections in the second disparity mapObject and obtain the position of each the second detected object; The second conversion equipment, by each the second detected objectPosition be converted to the second predetermined reference point based in the second disparity map coordinate each second detect phaseTo position; Determining device, based on each first estimate object estimate position and described each second inspectionSurvey second of object and detect relative position, determine with each and first estimate corresponding each of object first and examineSurvey the tracking and matching relation between object and each the second detected object.

By technical scheme of the present disclosure, can carry out tracing object more accurately based on disparity map.

Brief description of the drawings

Only by example, the preferred embodiments of the present invention are described referring now to accompanying drawing, of the present disclosure above-mentioned withAnd other object, Characteristics and advantages will become more obvious, in the accompanying drawings:

Fig. 1 exemplarily illustrates the schematic diagram of putting into practice example vehicle system of the present invention;

Fig. 2 is exemplarily illustrated in turn inside diameter or produces while jolting and clap because road surface is unevenTake the photograph the schematic diagram of the variation of the picture position of the object showing in the disparity map obtaining;

Fig. 3 be exemplarily illustrate according to an embodiment of the invention in disparity map to image tracingThe flow chart of method;

Fig. 4 exemplarily illustrates that the object in disparity map according to another embodiment of the invention followsThe block diagram of track system;

Fig. 5 (a)-5 (c) exemplarily shows respectively binocular camera respectively at t-1 frame, t frame,T+1 frame is taken the disparity map and the gray-scale map that obtain;

Fig. 6 (a)-6 (d) be exemplarily show in t-1 frame, the subject area detecting is converted to based onThe schematic diagram of the subject area of road end point;

Fig. 7 exemplarily illustrates and estimates according to the subject area based on road end point in t-1 frameThe subject area based on road end point in t frame;

Fig. 8 (a)-(d) be exemplarily show in t frame, the subject area detecting is converted to based onThe schematic diagram of the subject area of road end point;

Fig. 9 (a) and (b) be to be exemplarily illustrated in to determine in each object and t frame detecting in t-1 frameThe schematic diagram of two methods that the tracking and matching between each object detecting is used while relation;

Figure 10 (a)-(c) is exemplarily illustrated in to determine in each object and t frame detecting in t-1 frameThe schematic diagram of the other method when tracking and matching between each object detecting is related to;

Figure 11 (a)-(b) is exemplarily illustrated in to determine in each object and t frame detecting in t-1 frameThe signal in the region of the object of the coupling obtaining after the tracking and matching relation between each object detectingFigure;

Figure 12 exemplarily illustrates the preset movement distance of proofreading and correct each object detecting in t-1 frame, doesFor the preset movement distance of each object of detecting in the t frame mating with it; And

Figure 13 (a)-(e) exemplarily illustrates that the preset distance utilizing after proofreading and correct determines and follow t+1 frameThe schematic diagram of track matching relationship.

Detailed description of the invention

Preferred embodiment of the present disclosure is described below with reference to accompanying drawings in more detail. Although show in accompanying drawingPreferred embodiment of the present disclosure, but should be appreciated that, can realize the disclosure and not with various formsThe embodiment that should be set forth here limits. On the contrary, providing these embodiments is in order to make the disclosureMore thorough and complete, and those skilled in the art can know these public affairs after having read the disclosureOther embodiments of not describing in opening also within the scope of the present disclosure.

Fig. 1 exemplarily illustrates the schematic diagram of putting into practice example vehicle system of the present invention.

In Fig. 1, for example, on vehicle, binocular camera is being installed, to take in Vehicle Driving Cycle process(example as shown in Figure 1) left image and right image. From captured left image and right image, generateDisparity map. At this, the mode that generates disparity map from captured left image and right image is this area skillArt personnel are known, therefore do not repeat. Thereby the interval scheduled time is (for example, in the situations of 25 frame/secondsUnder, 1/25 second, interval) carry out this shooting, make two or more disparity map input processings that generateIn device, determine that the object such as pedestrian or vehicle on road for example, in (1/25 second) phase this scheduled timeBetween tracking and matching relation, determine the object such as pedestrian or vehicle on road this scheduled time (for example1/25 second) whether still can trace into afterwards.

Certainly, on vehicle, applying method for tracing object of the present invention and system is only an example of example,Based on actual demand, for example traffic monitoring, robot visual guidance, industrial products detection, medical diagnosis,Virtual reality etc. can also be applied method for tracing object of the present invention and system to reach on other objectsTo the object to image tracing and location.

Fig. 2 is exemplarily illustrated in turn inside diameter or produces while jolting and clap because road surface is unevenTake the photograph the schematic diagram of the variation of the picture position of the object showing in the disparity map obtaining.

As what set forth, in reality, may there is such a case: turn at vehicle in background technologyIt is curved or produce the object that shows in the captured disparity map obtaining while jolting because road surface is unevenCan there is great variety in picture position. As shown in Figure 2, the top of Fig. 2 shows the disparity map of present frame,Suppose to follow the tracks of to as if the part that circle enclosed of upper left-hand. In the time that vehicle bends to right, see figure2 bottom, can see, the object of tracking has been offset certain distance left. In this case,The technology that proposes the degree of depth of utilizing object in prior art can not be followed the tracks of this object exactly.

Embodiment of the present disclosure also can realize good object tracking effect in these cases.

Fig. 3 be exemplarily illustrate according to an embodiment of the invention in disparity map to image tracingThe flow chart of method 300.

This method for tracing object 300 comprises: the first detecting step (S301), detects in the first disparity mapOne or more the first detected objects and obtain the position of each the first detected object; The first switch process(S302), the position of each the first detected object is converted to based on the predetermined ginseng of first in the first disparity mapEach the first detection relative position of the coordinate of examination point; Estimate step (S303), based on each the first detectionPre-motion orientation and the predetermined movement velocity of relative position and each the first detected object, estimate secondIn disparity map corresponding to each first position of estimating of estimating object of each the first detected object; The second inspectionSurvey step (S304), detect one or more the second detected objects in the second disparity map and obtain eachThe position of the second detected object; The second switch process (S305), turns the position of each the second detected objectBe changed to each the second detection relative position of the coordinate of the second predetermined reference point based in the second disparity map;Determining step (S306), based on each first estimate object estimate position and described each second detectSecond of object detects relative position, determines with each and first estimates first detection of corresponding each of objectTracking and matching relation between object and each the second detected object.

In one embodiment, described determining step can also comprise the following steps: calculate do not mate everyPhase is detected with second of each the second detected object not mating in the individual described first position of estimating of estimating objectTo the minimum range between position; If the minimum range of this calculating is less than predetermined threshold, determines and be somebody's turn to doFirst to estimate the first detected object that object is corresponding be to mate with second detected object with minimum range.

For example, conventionally suppose tracing object for example and vehicle relative capable that binocular camera is for example installedThe speed of sailing is for example 60km/h(to the maximum, in the track that indicates 60-120km/h), in the scheduled timeFor example, in interval (1/25 second), be multiplied by the time interval by maximal rate, can obtain tracing object at thisThe ultimate range of advancing with respect to vehicle is during this time about 6 meters. In this case, can be by above-mentioned predetermined thresholdValue is set to 6 meters. So, if the distance of estimating between object and detected object is greater than 6 meters, pre-The probability of estimating object and detected object and be the tracing object mating is just minimum, can think that both do not mate. And if the distance of estimating between object and detected object is less than 6 meters, can think that both areJoin.

Certainly, above-mentioned predetermined threshold is not limited to by maximal phase, travel speed be arranged, and it can be alsoArrange by empirical value, also can arrange by statistics, etc., the disclosure is not limitIn this.

The second predetermined reference point in the first predetermined reference point and the second disparity map in above-mentioned the first disparity mapCan be interrelated, and can indicate Same Physical implication. For example, this first predetermined reference point and secondPredetermined reference point can be but be not limited to the end point of road, and it can also be that other have same physicalThe point of implication, the point of the such as sun etc.

In one embodiment, the coordinate of described the first predetermined reference point can be included in the first disparity mapAbscissa and the ordinate of end point of road at least one, and described the second predetermined reference pointCoordinate is included at least one in the abscissa of end point of the same road in the second disparity map and ordinateIndividual.

That is to say, while considering this first predetermined reference point or the second predetermined reference point, having more than is to consider itComplete exact position. For example, in the case of mainly considering to turn in the left and right of for example vehicle, thisThe abscissa of the position of a little predetermined reference point is even more important, and can only use these predetermined reference pointThe abscissa of position, and in like manner, in the case of mainly considering that the pitching of for example vehicle, these are pre-Determine the ordinate of position of reference point by even more important, and can only use the position of these predetermined reference pointOrdinate, in like manner, can both in the case of considering to turn and pitch etc. in the left and right of such as vehicleTo use abscissa and the ordinate of position of these predetermined reference point simultaneously.

In two predetermined ginsengs of taking for twice based on being separated by certain hour interval in the residing situation of disparity mapThe disparity map of twice shooting is converted to respectively the disparity map based on these two predetermined reference point by examination point, makesIn the time of definite object tracking relationship, can eliminate the left and right of vehicle and turn and/or pitch to looking of takingThe bias effect that poor figure causes.

In one embodiment, described the first detecting step can also obtain the large of each the first detected objectLittle, wherein, the size of each the first detected object is as each the first size of estimating object, and secondDetecting step also obtains the size of each the second detected object.

For example, such size can comprise wide, high, the degree of depth of object, indicates three-dimensional size.

In one embodiment, described determining step can be based on each the first position of estimating of estimating objectDetect relative position and size with second of size and each the second detected object, determine each firstTracking and matching relation between detected object and each the second detected object.

If using the size of object also as determining a condition of object tracking and matching relation, can be more accurateReally determine whether object mates.

In one embodiment, described determining step can comprise the following steps: first estimate based on eachSecond of position and size and described each second detected object of estimating of object detects relative position and largeLittle, estimate object for each first, calculate each first and estimate object and each the second detected object existsOverlapping area on horizontal plane; If one first is estimated one of object and second detected object at described waterOverlapping area maximum in plane, determines with this and first estimates the first detected object and the institute that object is correspondingStating one of second detected object mates.

Calculating each first, to estimate object and each the second detected object overlapping area on horizontal plane passableMean width and the degree of depth of only considering each object, and do not consider the height of object.

In one embodiment, described determining step can also comprise the following steps: if one first pre-Estimate above the second detected object of object and two overlapping area on described horizontal plane all maximum, calculateThis first estimates above the second detected object of object and two overlapping area in depth plane; If this is years oldOne estimates one of object and described two above second detected objects overlapping area maximum in depth plane,Determine with this and first estimate the first detected object that object is corresponding and described two the second detected objects aboveOne of mate.

Calculating each first, to estimate object and each the second detected object overlapping area in depth plane passableMean width and the height of only considering each object, and do not consider the degree of depth of object.

In one embodiment, described determining step can also comprise the following steps: calculate do not mate everyPhase is detected with second of each the second detected object not mating in the individual described first position of estimating of estimating objectTo the minimum range between position; If the minimum range of this calculating is less than predetermined threshold, determines and be somebody's turn to doFirst to estimate the first detected object that object is corresponding be to mate with second detected object with minimum range.

Now, can the overlapping area of calculated level face and depth plane determine object matching step itAfter still do not find in the situation of match objects, carry out following steps.

For example, conventionally suppose tracing object for example and vehicle relative capable that binocular camera is for example installedThe speed of sailing is for example 60km/h(to the maximum, in the track that indicates 60-120km/h), in the scheduled timeFor example, in interval (1/25 second), be multiplied by the time interval by maximal rate, can obtain tracing object at thisThe ultimate range of advancing with respect to vehicle is during this time about 6 meters. In this case, can be by above-mentioned predetermined thresholdValue is set to 6 meters. So, if the distance of estimating between object and detected object is greater than 6 meters, pre-The probability of estimating object and detected object and be the tracing object mating is just minimum, can think that both do not mate. And if the distance of estimating between object and detected object is less than 6 meters, can think that both areJoin.

That is to say, in one embodiment, described predetermined threshold can be based on described first estimateThe maximum movement speed of the first detected object that object is corresponding is determined.

At this, it is pointed out that movement velocity, the direction of motion, the movement of the object of mentioning hereinDistance etc. is all for example, relative motion speed with respect to the object for taking disparity map (binocular camera)Degree, the direction of motion, displacement.

In one embodiment, described determining step can also comprise the following steps: if one first inspectionSurvey object and do not mate with any the second detected object, determine that this first detected object is in the second disparity mapDisappear, and if second detected object mate with any the first detected object, definite thisTwo detected objects are new objects in the second disparity map.

Like this, can draw, which object matching in the second disparity map is existing in the first disparity mapObject, in the second disparity map, traced into existing object in the first disparity map; Look secondWhich object in poor figure does not mate existing object in the first disparity map, and these are to liking secondEmerging object in disparity map; In the first disparity map, existing which object does not have in the second disparity mapCoupling, these objects have disappeared in the second disparity map.

In one embodiment, in the method, in definite each first detected object and each the second inspectionAfter surveying the tracking and matching relation between object, first-phase that can be based on each the first detected object is to inspectionLocation put and the second-phase of each the second detected object of mating with it to detection position, proofread and correct each firstThe pre-motion orientation of detected object and predetermined movement velocity are right as each second detection of mating with itThe pre-motion orientation of elephant and predetermined movement velocity.

Like this, by each object in cicada the first disparity map and each object in the second disparity mapMatching relationship, the object that can learn each coupling during this predetermined time interval with respect to for exampleThe distance that the vehicle of binocular camera is installed and move, the motion that can proofread and correct the object of each couplingDirection and movement velocity, to used the direction of motion and the fortune of this correction during next predetermined time intervalMoving speed is calculated the displacement of each object with respect to this vehicle, to estimate between next scheduled timeEvery after each object with respect to the position of vehicle. So, can be circularly in next scheduled timeInterim continues to follow the tracks of each object.

So, according to embodiments of the invention, can carry out image tracing exactly.

Fig. 4 exemplarily illustrates that the object in disparity map according to another embodiment of the invention followsThe block diagram of track system 400.

This object-tracking systems 400 comprises: the first checkout gear (401), detects in the first disparity mapOne or more the first detected objects and obtain the position of each the first detected object; The first conversion equipment(402), the position of each the first detected object is converted to based on the predetermined ginseng of first in the first disparity mapEach the first detection relative position of the coordinate of examination point; Estimating device (403), based on each the first detectionPre-motion orientation and the predetermined movement velocity of relative position and each the first detected object, estimate secondIn disparity map corresponding to each first position of estimating of estimating object of each the first detected object; The second inspectionSurvey device (404), detect one or more the second detected objects in the second disparity map and obtain eachThe position of the second detected object; The second conversion equipment (405), turns the position of each the second detected objectBe changed to each the second detection relative position of the coordinate of the second predetermined reference point based in the second disparity map;Determining device (406), based on each first estimate object estimate position and described each second detectSecond of object detects relative position, determines with each and first estimates first detection of corresponding each of objectTracking and matching relation between object and each the second detected object.

Certainly, this object-tracking systems 400 can also comprise and each of above-mentioned method for tracing object 300Other steps are installed one to one, are not repeated herein.

So, according to embodiments of the invention, can carry out image tracing exactly.

Illustrate in greater detail each step of method for tracing object of the present invention below with reference to an object lessonSuddenly.

Fig. 5 (a)-5 (c) exemplarily shows respectively binocular camera respectively at t-1 frame, t frame,T+1 frame is taken (figure left side) disparity map and (the figure right side) gray-scale map obtaining.

The binocular camera of supposing to be arranged on vehicle for example, is clapped continuously with predetermined time interval (1/25 second)Three frames are taken the photograph, i.e. t-1 frame, t frame, t+1 frame. Certainly, also can be by the different time intervalsTake, the present invention does not limit this predetermined time interval, and this predetermined time interval also can change,But in the disclosure, for simplified characterization, suppose that this predetermined time interval fixes, therefore utilize objectSpeed of related movement, the relative movement distance that direction can calculate object, in next timeIn interval, estimate the position of object, as described in detail later.

Next by detailed description how between t-1 frame, t frame, t+1 frame coupling and follow the tracks of rightResemble.

Fig. 6 (a)-6 (d) be exemplarily show in t-1 frame, the subject area detecting is converted to based onThe schematic diagram of the subject area of road end point.

First, Fig. 6 (a) exemplarily shows the subject area detecting in t-1 frame, sees in Fig. 6 (a)Square frame mark. Identify respectively 11 objects of this detection with mark 1-10.

Herein, in disparity map, a kind of method in detected object region can be in same Applicant in 2011 yearsWhat the application number of submitting to Patent Office of the People's Republic of China November 18 was 201110369183.0 is entitled as " road objectDetection method and system " current unexposed patent application in find. At this, this patent application is quotedInvest this. Certainly, those skilled in the art also can use the detected object that may occur in other or futureMethod come in t-1 frame detected object.

Fig. 6 (b) exemplarily shows the concrete position in the region of these 11 objects that detect in t-1 framePut and size (x (pixel), y (pixel), z (mm), w (mm), h (mm), l (mm)) wherein x, y representativeAbscissa and the ordinate of the top left corner apex of this object in the image of taking, z represents the dark of this objectDegree, for example distance between binocular camera that this object and shooting are used, and w represents this objectWidth, the height that h represents this object, l represents the degree of depth of this object itself. So, this x, y, zCan represent the particular location of this object, and this w, h, l can represent the size of this object.

Note, in the present invention, can only follow the tracks of with the particular location of object, also can makeWith particular location and size both follow the tracks of (will specifically describe after a while), and in Fig. 6 (b), be only forBe convenient to describe and understand, show particular location and both example of size in the region of these objects,But the invention is not restricted to this.

Particularly, for example region of object 0 is (391,494,21455,495,721,609), also just anticipatesTaste, the position of the top left corner apex of this object 0 be (x=391 (pixel), y=494 (pixel),Z=21455 (mm)), and the size of this object is (w=495 (mm), h=721 (mm), l=609 (mm)).Certainly, in accompanying drawing of the present disclosure, only provided the position of top left corner apex of object as the position of object,Certainly, it may occur to persons skilled in the art that the central point that utilizes object, other points of summit, the lower right corner etc.Position be also possible as the position of object. In addition, object 1, object 2, object 3, object 4,Each particular location of object 5, object 6, object 7, object 8, object 9, object 10 and size asShown in Fig. 6 (b), do not describe one by one at this.

Fig. 6 (c) exemplarily shows the road end point in t-1 frame.

Just as discussed previously, be applied to predetermined reference point in method of the present invention not necessarily road disappearLosing point, and can be the point with other physical meaning, for example as a reference point etc. with the sun.

At this, obtain road end point method can same Applicant on December 09th, 2011 toTo be 201110409269.1 be entitled as the application number that Patent Office of the People's Republic of China submits to that " road based on parallax information dividesCut object detecting method " current unexposed patent application and deliver on June 8th, 2005ZhenchengHu, FranciscoLamosa, KeiichiUchimura are entitled as " U-V-disparity:anefficientalgorithmforstereovisionbasedsceneanalysis”（ISBN:0-7695-2327-7）In find, at this, these reference citations are invested to this. Certainly, those skilled in the art also can useThe road that other or the following method that obtains road end point that may occur obtain in t-1 frame disappearsAbscissa or the ordinate of point.

Fig. 6 (d) exemplarily shows the subject area detecting in t-1 frame is converted to based on this roadThe subject area of end point.

As described above, in the case of mainly considering to turn in the left and right of for example vehicle, these are predeterminedThe abscissa of the position of reference point is even more important, and can only use the position of these predetermined reference pointAbscissa, and in like manner, in the case of mainly considering the pitching of for example vehicle, these predetermined referenceThe ordinate of position of point is even more important, and can only use the vertical seat of the position of these predetermined reference pointMark, in like manner, in the case of considering to turn the left and right of such as vehicle and pitch, wait can the while bothUse abscissa and the ordinate of the position of these predetermined reference point.

Therefore, illustratively, at this because the left and right turning phenomenon of vehicle comparatively common and cause rightResemble offset distance larger, therefore in the disclosure, exemplarily only consider the abscissa of road end point. ?This, the horizontal stroke of the road end point that the patent application that the application number that utilization is quoted is 201110409269.1 obtainsCoordinate is x_v=450 (pixel). (certainly, also can pass through V-disparity map (along the V in image coordinate systemCoordinate direction obtains representing the y coordinate of road end point, and on V-disparity map, Road is 0 in parallax valueTime the value of y, be the ordinate of road end point. At this, the disclosure is only to obtain road end pointAbscissa set forth as an example, but the disclosure is not limited to this. )

From Fig. 6 (d), can find out by by the horizontal seat of each subject area of the detection obtaining in Fig. 6 (b)Mark x deducts the abscissa x of road end point_v=450 (pixel) obtain each based on this road end pointIndividual subject area (x ', y, z, w, h, l)=(x-x_v, y, z, w, h, l). Particularly, for example, for object 0, will(391,494,21455,495,721,609) deduct x_v=450 (pixel) obtain (59,494,21455,495,721,609)。

Certainly, in another embodiment, the invention is not restricted to only consider that the abscissa of road end point disappearsExcept the left and right skew of binocular camera, the ordinate that also can only consider road end point is eliminated binocular and is taken the photographThe abscissa of the skew up and down of camera or consideration road end point and ordinate are eliminated binocular and are taken the photographThe left and right skew of camera and up and down skew, those skilled in the art are also very clear is considering road end pointIn the abscissa of ordinate or consideration road end point and the situation of ordinate, how to obtain baseIn each subject area of this road end point, be not repeated herein.

By being converted to the subject area based on road end point, can eliminating binocular camera is installedThe impact on the position of detected object in captured image such as turn and/or pitch of the left and right of vehicle,Thereby can contribute to determine more accurately object tracking and matching relation.

Fig. 7 exemplarily illustrates and estimates according to the subject area based on road end point in t-1 frameThe subject area based on road end point in t frame.

If be scheduled to the direction of motion and the movement velocity of each object in t-1 frame, can be by theThe time interval of being separated by between t-1 frame and t frame is extrapolated each in x, y, z direction of each objectFrom displacement (Δ z) for Δ x, Δ y. Wherein, this Δ x, Δ y, the arbitrary value in Δ z, for example Δ x get on the occasion of,The direction of motion corresponding to object on x axle is the positive direction of x axle, and for example Δ x gets negative value corresponding to rightThe direction of motion resembling on x axle is the negative direction of x axle. The direction of motion of these each objects and motion speedDegree can preset, and also can calculate by known manner, predict etc., is not repeated herein. In addition,Only describe by the predetermined direction of motion and movement velocity and calculated displacement at this, but also canDirectly preset movement distance in an embodiment. In the disclosure, movement velocity and the direction of motion can with (getOn the occasion of or negative value) displacement uses interchangeably.

And, in the size of supposing the object based on road end point in t-1 frame and the t frame of estimatingThe size of the object based on road end point identical, i.e. w_p=w，h_p=h, and l_p=l。

In the t frame of therefore, estimating according to the subject area based on road end point in t-1 frameSubject area based on road end point be (x '_p,y_p,z_p,w_p,h_p,l_p)=（x’+Δx,y+Δy,z+Δz,w,h,l)。

In the disclosure, to calculate and clearly set forth principle of the present invention in order to simplify, in t-1 frame,Pre-motion orientation and the movement velocity of each object are initialized as to (0,0,0), suppose when initial eachIndividual static with respect to binocular camera to liking.

In the t frame of therefore, estimating according to the subject area based on road end point in t-1 frameSubject area based on road end point be (x '_p,y_p,z_p,w_p,h_p,l_p)=（x’+0,y+0,z+0,w，h,l），As can be seen from Figure 7, the subject area based on road end point in the t frame of estimating and Fig. 6's (d) is rightResemble region identical.

Fig. 8 (a)-(d) be exemplarily show in t frame, the subject area detecting is converted to based onThe schematic diagram of the subject area of road end point.

Fig. 8 (a) exemplarily illustrates the subject area detecting in t frame, sees the square frame mark in Fig. 8 (a)Know. Identify respectively 5 objects of this detection with mark A, B, C, D, E.

As previously mentioned, herein, in disparity map, a kind of method in detected object region can be in identical applicationWhat the application number that people submitted on November 18th, 2011 to Patent Office of the People's Republic of China was 201110369183.0 is entitled asIn the current unexposed patent application of " road method for checking object and system ", find. At this, this is specialProfit application is quoted and is invested this. Certainly, those skilled in the art also can use other or future to occurDetected object method come in t frame detected object.

Fig. 8 (b) exemplarily show these 5 objects that detect in t frame region particular location andSize (x (pixel), y (pixel), z (mm), w (mm), h (mm), l (mm)), wherein as previously mentioned, x,Abscissa and the ordinate of the top left corner apex that y represents this object in the image of taking, z represents this objectThe degree of depth, i.e. this object and take for example distance between binocular camera that uses, and w representative shouldThe width of object, the height that h represents this object, l represents the degree of depth of this object itself. So, this x, y,Z can represent the particular location of this object, and this w, h, l can represent the size of this object.

Note, in the present invention, can only follow the tracks of with the particular location of object, also can makeWith particular location and size both follow the tracks of, and in Fig. 8 (b), be only for convenience of description and understand,Show particular location and both examples of size in the region of these objects, but the invention is not restricted to this.

Fig. 8 (c) exemplarily shows the road end point in t frame.

Just as discussed previously, be applied to predetermined reference point in method of the present invention not necessarily road disappearLosing point, and can be the point with other physical meaning, for example as a reference point etc. with the sun.

As previously mentioned, obtaining the method for road end point can be in same Applicant December 09 in 2011What the application number of submitting to Patent Office of the People's Republic of China day was 201110409269.1 is entitled as " based on the road of parallax informationRoad partage detection method " current unexposed patent application and deliver on June 8th, 2005ZhenchengHu, FranciscoLamosa, KeiichiUchimura are entitled as " U-V-disparity:anefficientalgorithmforstereovisionbasedsceneanalysis”（ISBN:0-7695-2327-7）In find, at this, these reference citations are invested to this. Certainly, those skilled in the art also can useOther or the following method that obtains road end point that may occur obtain the road end point in t frameAbscissa or ordinate.

Fig. 8 (d) exemplarily shows the subject area detecting in t frame is converted to based on this roadThe subject area of end point.

As described above, in the case of mainly considering to turn in the left and right of for example vehicle, these are predeterminedThe abscissa of the position of reference point is even more important, and can only use the position of these predetermined reference pointAbscissa, and in like manner, in the case of mainly considering the pitching of for example vehicle, these predetermined referenceThe ordinate of position of point is even more important, and can only use the vertical seat of the position of these predetermined reference pointMark, in like manner, in the case of considering to turn the left and right of such as vehicle and pitch, wait can the while bothUse abscissa and the ordinate of the position of these predetermined reference point.

Therefore, illustratively, at this because the left and right turning phenomenon of vehicle comparatively common and cause rightResemble offset distance larger, therefore in the disclosure, first only consider the abscissa of road end point. At this, profitThe abscissa of the road end point obtaining with the patent application that the application number of quoting is 201110409269.1 isx_v=287 (pixel). (certainly, also can pass through V-disparity map (sits along the V in image coordinate systemMark direction (ordinate direction), scanning from top to bottom, the cumulative pixel with same disparity valueNumber, and the V-disparity map of setting up) obtain representing the y coordinate of road end point, i.e. V-parallaxOn figure, Road is the value of the y of 0 o'clock in parallax value, is the ordinate of road end point. At this,The disclosure is only set forth as an example with the abscissa that obtains road end point, but the disclosure is notBe limited to this. )

From Fig. 8 (d), can find out by by the horizontal seat of each subject area of the detection obtaining in Fig. 8 (b)Mark x deducts the abscissa x of road end point_v=287 (pixel) obtain each based on this road end pointIndividual subject area (x ', y, z, w, h, l)=(x-x_v, y, z, w, h, l). Particularly, for example, for object B, will(340,347,16248,2988,2227,5816) deduct x_v=287 (pixel) obtain (53,347,16248,2988,2227,5816)。

Certainly, in another embodiment, the invention is not restricted to only consider the abscissa of road end point, alsoCan only consider the ordinate of road end point or abscissa and the ordinate two of consideration road end pointPerson, those skilled in the art are also very clear to be considered the ordinate of road end point or is considering that road disappearsLose under the abscissa and the situation of ordinate of point, it is right how to obtain based on each of this road end pointResemble region, be not repeated herein.

Fig. 9 (a) and (b) be to be exemplarily illustrated in to determine in each object and t frame detecting in t-1 frameThe schematic diagram of two methods that the tracking and matching between each object detecting is used while relation.

First with reference to figure 9 (a), according to the t frame of the actual detection shown in Fig. 8 (d) based on road end pointSubject area (x ', y, z, w, h, l) (for example subject area 1), can obtain the cube of Fig. 9 (a) upper left,Wherein the coordinate of top left corner apex is (x ', y, z), and the cubical wide of this object is w(mm), Gao WeiH(mm), and the degree of depth be l(mm).

According to the subject area based on road end point of the t frame of estimating according to t-1 frame shown in Fig. 7(x’_p,y_p,z_p,w_p,h_p,l_p) (for example subject area B), can obtain the cube of Fig. 9 (a) bottom right, whereinThe coordinate of top left corner apex be (x '_p,y_p,z_p), and the cubical wide of this object is w_p(mm), Gao Weih_p(mm), and the degree of depth be l_p（mm）。

If do not consider object size, do not consider (w, h, l) and (w_p,h_p,l_p) and only considerThe position of object, only consider object top left corner apex position (x ', y, z) and (x '_p,y_p,z_p) situation under,The method can comprise the object that calculates each t frame of estimating based on t-1 frame position (x '_p,y_p,z_p)With the minimum range between the position of the actual object detecting in each t frame (x ', y, z); If this calculatingMinimum range be less than predetermined threshold, determine with t-1 frame in detect in this object of detecting and t frameThis to as if coupling.

That is to say, particularly, for example, pass throughCalculate baseA position of the object 1 of the t frame of estimating in t-1 frame (x '_p,y_p,z_p) with each t frame in actualDistance d between each object B, C, the E detecting, the position of A, D (x ', y, z). Then, obtainThese are apart from the minimum range d in d_min(for example, suppose the top left corner apex of object 1 and object BDistance d minimum), and by these distance in minimum range d_minCompare with predetermined threshold, if shouldMinimum range d_minBe less than predetermined threshold, determine this object 1 of t-1 frame and having of this t frameSmall distance d_minObject (for example for object B) match.

Certainly the mode of calculating the distance between two objects is not limited to this, also can pass through except the upper left cornerDistance between other points beyond summit is calculated, and is not repeated herein.

That is to say, only consider the position of object, also can carry out the method for tracing object of disclosure enforcement.

Fig. 9 (b) show the position of not only considering object top left corner apex (x ', y, z) and (x '_p,y_p,z_p), alsoConsider size (w, h, l) and (w of object_p,h_p,l_p) situation under calculate the distance of two objectsSchematic diagram.

With reference to figure 9 (b), according to right based on road end point of the t frame of the actual detection shown in Fig. 8 (d)Resemble region (x ', y, z, w, h, l) (for example subject area 1), can obtain the cube of Fig. 9 (b) upper left, itsThe coordinate of middle top left corner apex is (x ', y, z), and the cubical wide of this object is w(mm), height is h(mm), and the degree of depth be l(mm). The central point of the subject area of the t frame of this actual detection is c.

According to the subject area based on road end point of the t frame of estimating according to t-1 frame shown in Fig. 7(x’_p,y_p,z_p,w_p,h_p,l_p) (for example subject area B), can obtain the cube of Fig. 9 (b) bottom right, whereinThe coordinate of top left corner apex be (x '_p,y_p,z_p), and the cubical wide of this object is w_p(mm), Gao Weih_p(mm), and the degree of depth be l_p(mm). The central point of the subject area of this t frame of estimating is c_p。

For example,, in order to calculate c and c_pBetween distance, pass through formula

$d=\sqrt{{(x^{\′}+w/2-(x_p+w_p/2))}^2+{(y+h/2-(y_p+h_p/2))}^2+{(z+w/2-(z_p+l_p/2))}^2},$ MeterCalculate the Stereocenter of two solid object (for example object 1 respectively with each object B, C, E, A, D)Distance d between point. Then, obtain these apart from the minimum range d in d_min(for example, suppose object1 with the distance minimum of the central point of object B), and by the minimum range d in these distances_minWith predetermined thresholdValue is compared, if this minimum range d_minBe less than predetermined threshold, this object 1 of determining t-1 frame withThis t frame there is minimum range d_minObject (for example for object B) match.

Certainly the mode of calculating the distance between two objects is not limited to this, also can pass through except in solidDistance between other points beyond heart point is calculated, and is not repeated herein.

Determining all objects 1,2 of estimating ... 10 with detected object A, B ... coupling between EAfter relation, if one is estimated object and do not mate with any detected object, determine and estimate object with thisCorresponding detected object disappears in t frame, and if detected object not with any object of estimatingCoupling, determines that this detected object is new object in t frame.

Like this, can determine that more accurately two objects are in solid space than the method shown in Fig. 9 (a)Distance, further to determine tracking and matching relation.

Above, can obtain the object and the actual object detecting in t frame, estimated with the calculating of simplifyingBetween tracking and matching relation, as the actual inspection in the t-1 frame corresponding with the object of estimating in t frameSurvey the tracking and matching relation between the actual object detecting in object and t frame.

Below, introduce and use complicated a little calculating to obtain more accurately the actual detected object in t-1 frameAnd the another kind of method of the tracking and matching relation in t frame between the actual object detecting.

Figure 10 (a)-(c) is exemplarily illustrated in to determine in each object and t frame detecting in t-1 frameThe schematic diagram of the another kind of method when tracking and matching between each object detecting is related to.

Obtaining the subject area based on road end point of t frame of the actual detection as shown in Fig. 9 (b)(x ', y, z, w, h, l) (for example subject area 1), wherein the coordinate of top left corner apex is (x ', y, z), and shouldThe cubical wide of object is w(mm), height is h(mm), and the degree of depth is l(mm), and according toThe subject area based on road end point of the t frame that t-1 frame is estimated (x '_p,y_p,z_p,w_p,h_p,l_p) (exampleAs subject area B), wherein the coordinate of top left corner apex be (x '_p,y_p,z_p), and this object is cubicalWide is w_p(mm), height is h_p(mm), and the degree of depth be l_p（mm）。

For example, in order to determine the subject area based on road end point (object 1) of the actual t frame detectingWith the subject area based on road end point (for example object B) of the t frame of estimating according to t-1 frame itBetween tracking and matching relation, can pass through respectively following steps (can certainly be only through following steps itOne, the order of these steps does not also limit):

That estimates object based on each estimates position (for example x '_p,y_p,z_p) and size (for example w_p,h_p,l_p）And the relative position of described each detected object (for example x ', y, z) and size (for example w, h, l), pinFor example, to each object (, object 0,1,2 of estimating ... 10), calculate each estimate object and each inspectionSurvey object (for example object B, C, E, A, D) upper overlapping at U-disparity map (, horizontal plane)Area.

Wherein, U-disparity map means on horizontal plane, the appearance of each object of looking from top to bottom,As shown in Figure 10 (a), look from top to bottom as seen, on horizontal plane, only consider object wide w andw_pAnd the degree of depth l of object and l_p, and can't see, also not consider height h and the h of object_p。

Then, if one estimate object (for example object 1) and detected object (for example object B, C,E, A, D) one of (for example object B) overlapping area maximum on U-disparity map, determine with shouldEstimate for example, detected object (for example remaining object 1) in the t-1 frame that object (object 1) is correspondingFor example, mate with one of detected object in t frame (object B).

Particularly, with reference to Figure 10 (a), calculated example as object 1 with object B for example on U-disparity mapWhen overlapping area (as the area of dash area in Figure 10 (a)), can pass through formula S u=(x '+w – x '_p)×(z_p+l_p-z) calculate. Calculating respectively object 1 and object B, object 1 and object C, object1 owns with object D's with object A, object 1 with object E, object 1) overlapping on U-disparity mapArea after, determine area maximum overlapping on U-disparity map object (for example object B) doFor the object matching with object 1.

So, only utilize the calculating of the overlapping area on U-disparity map can determine whether to mate theseObject. This U-disparity map, owing to only considering the appearance of the object of looking from the top down, therefore can be fittedThe local error of eliminating being offset up and down of binocular camera and cause. And, before combining based on roadIn the situation of the error that the conversion of road end point has caused to have eliminated the skew of binocular camera left and right, this is realExecute example can eliminate simultaneously binocular camera up and down skew and left and right be offset the impact causing.

But sometimes, so overlapping area on U-disparity map not necessarily only has a maximum,If one estimate object and more than two detected object on described U-disparity map, overlapping area is allGreatly, can calculate this estimate object and more than two detected object X-Y-disparity map (being depth plane)Upper overlapping area. Wherein X-Y-disparity map means in depth plane, each of looking from front to backThe appearance of object, as shown in Figure 10 (b), as seen from looking from front to back, in depth plane, it is right only to considerWide w and the w of elephant_pAnd height h and the h of object_p, and can't see, also not consider the degree of depth l of objectAnd l_p。

If this estimates object and described more than two one of detected object overlapping face on X-Y-disparity mapLong-pending maximum, determine with this estimate in detected object in the t-1 frame that object is corresponding and t frame described inIt is more than two that one of detected object mates.

Particularly, with reference to Figure 10 (b), estimate object and detected object is overlapping on X-Y-disparity map in calculatingArea (as the area of dash area in Figure 10 (b)) time, can pass through formula S_x-y=(x’+w–x’_p)×(y_p+h_p-y) calculate. Estimate object and this more than two detected object at X-Y-parallax having calculated respectivelyOn figure, after overlapping area, determine that a detection of area maximum overlapping on X-Y-disparity map is rightResemble as with the object of estimating object and matching, can determine with this and estimate in the t-1 frame that object is correspondingDetected object mate with this detected object in t frame.

For example, after having passed through the calculating of the overlapping area on U-disparity map and X-Y disparity map, pass throughThe concrete numerical value of disclosed each subject area in the disclosure, can obtain object B and 1, object C and2 overlapping areas in U-disparity map and/or X-Y-disparity map are all greater than 0, and other overlapping areas are all 0,Therefore can obtain object 1 and mate with object B, object 2 mates with object C.

Certainly, the disclosure should be mentioned that and first calculates the overlapping area of U-disparity map, has 2 in overlapping areaWhen maximum area, just use the overlapping area of X-Y-disparity map, but the disclosure is not limited to this, also can be singleSolely carry out the tracking and matching of object by the overlapping area of X-Y-disparity map, even can also independent or groupClose and use V-disparity map (appearance of the object of looking from left to right) to carry out the tracking and matching of objectDeng.

If, by the meter of Figure 10 (a) and the U-disparity map of Figure 10 (b) and the overlapping area of X-Y-disparity mapAfter calculation, some of t frame estimate object (for example, object 3,4 ... 10) and t frame oneA little detected objects (for example, object A, E, D) are not still mated, and can calculate coupling notDescribed in each, estimate the narrow spacing between the position of estimating position and each detected object not mating of objectFrom; For example, if the minimum range of this calculating is less than predetermined threshold (before predetermined 6 meters), determine withThis detected object of estimating the t-1 frame that object is corresponding with the detected object of the t frame with minimum range isCoupling.

By the concrete numerical value of disclosed each subject area in the disclosure, can calculate object 3 HesThe distance having between the object E of minimum range is 2 meters, is less than 6 meters of predetermined thresholds, therefore object 3E mates with object.

The method of the distance between concrete two objects of calculating can be as before with reference to figure 9 (a) and 9 (b)Shown method, is not repeated herein. Certainly, the disclosure is also not limited to this.

The exemplary distance showing between the Stereocenter point that calculates two objects only in Figure 10 (c)Mode, this be only for make to tracking and matching relation determine more accurate, instead of restriction.

Finally, if one is estimated object and do not mate with any detected object, determine and estimate object with thisCorresponding detected object disappears in t frame, and if detected object not with any object of estimatingCoupling, determines that this detected object is new object in t frame.

Particularly, in this example, object 4,5 ... 10 all do not mate, illustrate these objects 4,5 ... 10 have disappeared in t frame. And in the object detecting, there are 2 detected objects and institute in t frameEstimate object and do not mate, illustrate and in t frame, occurred two new objects, can be respectively by itBe designated object 11 and object 12.

Figure 11 (a)-(b) is exemplarily illustrated in to determine in each object and t frame detecting in t-1 frameThe signal in the region of the object of the coupling obtaining after the tracking and matching relation between each object detectingFigure.

Figure 11 (a) shows through above-mentioned and calculates, the object B that detects in t frame, C, E,The object 0,1,2 that detects in A, D and t-1 frame ... tracking and matching relation between 10. CanSee, object B is mated with object 1, and object C mates with object 2, and object E mates with object 3, and11 and 12 of objects are emerging two objects in t frame, therefore use respectively new mark 11,12Represent.

Figure 11 (b) shows in the image of t frame the subject area of last definite coupling visuallySquare frame.

Like this, by use separately or in combination overlapping area on U-disparity map, X-Y-disparity map andDistance, can than the method for the only service range shown in Fig. 9 (a) and Fig. 9 (b) more accurately determine two rightThe tracking and matching relation of elephant.

So, according to these one or more embodiment, what can obtain more accurately estimating in t frame is rightResemble and reality detect object between tracking and matching relation, as corresponding with the object of estimating in t frameT-1 frame in actual detected object and t frame in tracking and matching between the actual object detecting closeSystem.

Figure 12 exemplarily illustrates that the pre-motion orientation of proofreading and correct each object detecting in t-1 frame is with pre-Determine movement velocity (or displacement), as being scheduled to of each object detecting in the t frame mating with itThe direction of motion and predetermined movement velocity (or displacement).

The tracking and matching between each detected object in each detected object and t frame in definite t-1 frameAfter relation, the position based on road end point based on each detected object in t-1 frame and mating with itT frame in the position based on road end point of each detected object, can proofread and correct in t-1 frame eachThe pre-motion orientation of detected object and predetermined movement velocity, as each inspection in the t frame mating with itSurvey pre-motion orientation and the predetermined movement velocity of object.

Certainly,, in the disclosure, in order to simplify calculating, the time interval between the each frame of hypothesis equatesIn situation, only represent the concept of the direction of motion and movement velocity by displacement.

Δx=x’_t–x’_t-1,Δy=y_t–y_t-1,Δz=z_t–z_t-1. To new object 11 and 12, (Δ z) for Δ x, Δ y=(0,0,0). Wherein x '_tRepresent the horizontal seat of the position based on road end point of the detected object in t frameMark, and x '_t-1Represent the abscissa of the position based on road end point of the detected object in t-1 frame, y_tRepresent the ordinate of the position based on road end point of the detected object in t frame, and y_t-1Represent t-1The ordinate of the position based on road end point of the detected object in frame, z_tRepresent the detection in t-1 frameThe depth coordinate of the position based on road end point of object, and z_t-1Represent detected object in t frameThe depth coordinate of the position based on road end point.

If calculated, (Δ z), can be by the time between t-1 frame and t frame for Δ x, Δ yInterval, infers the direction of motion and the movement velocity of each detected object in t frame. Wherein, this Δ x, Δ y,Arbitrary value in Δ z, for example Δ x get on the occasion of, the direction of motion corresponding to object on x axle be x axle justDirection, and for example Δ x gets negative value, corresponding to object, the direction of motion on x axle is the negative direction of x axle.

As shown in Figure 12, for example, after correction, the displacement that is designated 1 object in t frame is(3 ,-18 ,-1730), in t frame, the displacement of emerging new object 11 is (0,0,0).It should be noted that in the disclosure, (get on the occasion of or negative value) displacement can with movement velocity andThe direction of motion is used convertibly.

Utilize the displacement of each object of the t frame after proofreading and correct to come the by exemplarily introducing belowEach object detecting in t+1 frame carries out tracking and matching. Certainly, those skilled in the art are according in the disclosureThe detailed description of method of tracking and matching of the object to t-1 frame and t frame, can inferCarry out the mode of tracking and matching for each object detecting in t+1 frame, still, in order to make thisOpen clearer, below still this is described, but these descriptions are not restrictive.

Figure 13 (a)-(e) exemplarily illustrates that the displacement of utilizing after proofreading and correct determines and follow t+1 frameThe schematic diagram of track matching relationship.

Figure 13 (a) exemplarily show utilize the t frame after proofreading and correct object (for example object 1,2,3,The subject area based on road end point of displacement 11,12) and the object of t frame (x ', y, z, w,H, l), estimate the subject area based on road end point of estimating object in t+1 frame (x '_p,y_p,z_p,w_p,h_p,l_p). Wherein, x_p=x’+Δx,y_p=yt+Δy,z_p=z_t+Δz，w_p=w，h_p=h,l_p=l,(x_p,y_p,z_p,w_p,h_p,l_p)=（x’+Δx,y+Δy,z+Δz,w,h,l）。

Then, utilizing above-mentioned application number is 201110369183.0 " the road method for checking object that is entitled asAnd system " current unexposed patent application (or disclosed or following may appearance in other prior ariesMethod for checking object) detect the actual detection in t+1 frame object (for example object A, B, C,D, E, F, G, H) subject area (position and size), see shown in Figure 13 (b) and Figure 13 (c).

Then, utilizing above-mentioned application number is 201110409269.1 be entitled as " based on the road of parallax informationRoad partage detection method " current unexposed patent application (or disclosed or not in other prior ariesCarry out the road vanishing Point Detection Method method that may occur) detect the abscissa that obtains road end pointx_v＝122(pixel)。

Similarly, each object of the detection by t+1 frame (for example object A, B, C, D, E,F, G, H) the abscissa x of subject area all deduct the abscissa x of road end point_v=122(pixel)Obtain each subject area (x ', y, z, w, h, l)=(x-x of this road end point based on t+1 frame_v,y,Z, w, h, l), as shown in Figure 13 (d).

Then, by reference to the method shown in Fig. 9 (a) or Fig. 9 (b) or with reference to the side shown in Figure 10 (a)-10 (c)Method is determined between object 1,2,3,11,12 and object A, B, C, D, E, F, G, HTracking and matching relation. Be not repeated herein.

Thereby obtain the subject area of the coupling as shown in Figure 13 (e).

Therefore, can further proofread and correct each coupling of t+1 frame object displacement underIn the tracking and matching of one frame (t+2 frame). So move in circles, can Continuous Tracking object, withFor traffic monitoring, auxiliary driving, automobile detection and tracking, alarm drive system, robot vision are ledEach practical field such as boat, industrial products detection, medical diagnosis, virtual reality.

Each step of method and the step of some variation of having given an example of tracing object in the disclosure, are mainly set forthSuddenly, those skilled in the art can know, install one to one also with the step of these steps and variationShould, in the scope of the present disclosure, be not repeated herein.

Here all patents, patent application, article, other publications, document and the things of quoting forAll objects are quoted its full text of merging at this by this. If any publication, document or the things of any mergingAnd between the disclosure, there is any inconsistent of term definition or use or conflict, being as the criterion with the disclosure.

Certainly, those skilled in the art have been appreciated that the essential implication of each term according to the disclosure,Content of the present disclosure is also not only confined to the given narrow implication of concrete term, but can indicateImplication widely in spirit of the present disclosure and principle.

More than open each of method and system that carrys out tracing object exactly based on disparity map of having describedEmbodiment, still, foregoing description is exemplary, not exhaustive, and be also not limited to disclosedEach embodiment. In the case of not departing from the scope and spirit of illustrated each embodiment, for this skillThe many modifications and changes of those of ordinary skill in art field are all apparent. Art technology peopleMember should be appreciated that, in the scope of claims or its equivalent, can need and it based on designHe carries out various amendments, combination, sub-portfolio and change at factor.

Claims (9)

1. the method for tracing object in disparity map, comprises the following steps:

The first detecting step, detects one or more the first detected objects and acquisition in the first disparity mapThe position of each the first detected object;

The first switch process, is converted to the position of each the first detected object based in the first disparity mapEach the first detection relative position of the coordinate of the first predetermined reference point;

Estimate step, based on the predetermined motion of each the first detection relative position and each the first detected objectDirection and predetermined movement velocity, estimate in the second disparity map corresponding to each of each the first detected objectFirst estimates the position of estimating of object;

The second detecting step, detects one or more the second detected objects and acquisition in the second disparity mapThe position of each the second detected object;

The second switch process, is converted to the position of each the second detected object based in the second disparity mapEach the second detection relative position of the coordinate of the second predetermined reference point;

Determining step, based on each first estimate object estimate position and described each second detect rightSecond of elephant detects relative position, determines that first to estimate first detection of corresponding each of object right with eachResemble the tracking and matching relation between each second detected object,

Wherein, the coordinate of described the first predetermined reference point is included in the end point of the road in the first disparity mapAbscissa and ordinate at least one, and the coordinate of described the second predetermined reference point is included in secondAt least one in abscissa and the ordinate of the end point of the same road in disparity map.

2. method according to claim 1, wherein, described the first detecting step also obtain eachThe size of one detected object, wherein, the size of each the first detected object first is estimated object as eachSize, and described the second detecting step also obtains the size of each the second detected object,

Wherein, described determining step based on each first estimate object estimate position and size and eachSecond of the second detected object detects relative position and size, determines each first detected object and eachTracking and matching relation between the second detected object.

3. method according to claim 2, wherein, described determining step comprises the following steps:

Based on each first estimate object estimate position and size and described each the second detected objectSecond detects relative position and size, estimates object for each first, calculates each first and estimates objectWith each second detected object overlapping area on horizontal plane;

If one first is estimated one of object and second detected object overlapping area on described horizontal planeMaximum, determine with this first estimate the first detected object that object is corresponding and described the second detected object itThe one, coupling.

4. method according to claim 3, wherein, described determining step is further comprising the steps of:

If one first to estimate above the second detected objects of object and two overlapping on described horizontal planeArea is all maximum, calculates this and first estimates object and two above the second detected objects weight in depth planeFolded area;

If first to estimate one of object and described two above second detected objects overlapping in depth plane for thisArea maximum, determine with this and first estimate the first detected object that object is corresponding and described more than twoOne of second detected object mates.

5. according to the method described in claim 1 or 4, wherein, described determining step also comprises following stepRapid:

Calculate not coupling each described first estimate object estimate position with do not mate each second examineSurvey the second minimum range detecting between relative position of object;

If the minimum range of this calculating is less than predetermined threshold, determine that first to estimate object corresponding with thisThe first detected object mates with second detected object with minimum range.

6. method according to claim 5, wherein, described predetermined threshold is based on described firstEstimate that the maximum movement speed of the first detected object that object is corresponding determines.

7. method according to claim 5, wherein, described determining step is further comprising the steps of:

If first detected object does not mate with any the second detected object, determine this first detectionObject disappears in the second disparity map, and

If second detected object does not mate with any the first detected object, determine this second detectionObject is new object in the second disparity map.

8. method according to claim 1, also comprises:

Determining that the tracking and matching between each first detected object and each the second detected object is related to itAfter, each second detection that the first-phase based on each the first detected object mates to detection position with itThe second-phase of object, to detection position, is proofreaied and correct pre-motion orientation and the predetermined fortune of each the first detected objectMoving speed, as pre-motion orientation and the predetermined movement velocity of each the second detected object mating with it.

9. the object-tracking systems in disparity map, comprising:

The first checkout gear, detects one or more the first detected objects and acquisition in the first disparity mapThe position of each the first detected object;

The first conversion equipment, is converted to the position of each the first detected object based in the first disparity mapEach the first detection relative position of the coordinate of the first predetermined reference point;

Estimating device, based on the predetermined motion of each the first detection relative position and each the first detected objectDirection and predetermined movement velocity, estimate in the second disparity map corresponding to each of each the first detected objectFirst estimates the position of estimating of object;

Second detection device, detects one or more the second detected objects and acquisition in the second disparity mapThe position of each the second detected object;

The second conversion equipment, is converted to the position of each the second detected object based in the second disparity mapEach the second detection relative position of the coordinate of the second predetermined reference point;

Determining device, based on each first estimate object estimate position and described each second detect rightSecond of elephant detects relative position, determines that first to estimate first detection of corresponding each of object right with eachResemble the tracking and matching relation between each second detected object,

Wherein, the coordinate of described the first predetermined reference point is included in the end point of the road in the first disparity mapAbscissa and ordinate at least one, and the coordinate of described the second predetermined reference point is included in secondAt least one in abscissa and the ordinate of the end point of the same road in disparity map.