JPH0851564A - Target tracking device - Google Patents

Abstract

(57) [Abstract] [Object] The present invention effectively eliminates erroneous determination due to the influence of a target-like object, a change in luminance, and a different shape in a target-tracking device, thereby tracking the target more stably. . [Structure] By modeling each of the RY signal and the BY signal at the time of setting a target feature by the brightness, the feature can be captured without paying attention to a change in hue with respect to a change in brightness. Further, by directly modeling the R-Y signal and the B-Y signal, it is possible to capture the features of an achromatic subject because they are not captured by the hue in the same manner, and also model the allowable error. Since it is not necessary to rigorously adjust parameters, it is possible to stably perform target tracking with respect to changes in luminance.

Description

Detailed Description of the Invention

[0001]

[Table of Contents] The present invention will be described in the following order. Field of Industrial Application Conventional Technology Problems to be Solved by the Invention Means for Solving the Problems Action Example (1) First Example (FIGS. 1 to 8) (1-1) System Configuration (1-2) ) Target tracking processing (1-2-1) Overview (1-2-2) Processing in each part (1-2-2-1) Subject feature amount extraction processing (1-2-2-2) Subject position Detection Process (1-3) Effect (2) Second Example (FIG. 9) (2-1) Target Tracking Process (2-2) Effect (3) Third Example (FIGS. 10 to 13) (3- 1) Target tracking processing (3-2) Effects (4) Fourth embodiment (FIGS. 14 to 17) (4-1) System configuration (4-2) Target tracking processing (4-3) Effects (5) Fourth Fifth Embodiment (FIGS. 18 to 20) (5-1) System Configuration (5-2) Target Tracking Process and Frame Synthesis Process (5-3) Effects (6) Other Embodiments of the Invention Results

[0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a target tracking device.
For example, the present invention can be applied to the one in which the shooting direction of the television camera is automatically followed as the target moves.

[0003]

2. Description of the Related Art Conventionally, in this kind of target tracking device, automatic tracking of the image pickup direction is realized by detecting the movement of the target object from the image taken by the television camera. The block matching method is generally used for this motion detection. Here, the block matching is a method of detecting the block having the maximum correlation with the original block defined by the peripheral image of the target object or the image of the characteristic portion from the captured image, and each pixel of the original block. The position of the block where the sum of the absolute values of the differences between the pixels is minimized is detected as the position of the target.

[0004]

However, in the case of the target tracking device using the block matching method, the position of the target is determined depending on the information contained in the original block. Therefore, there is a problem that tracking becomes difficult when the target itself is deformed or the size of the target changes. Also, the calculation of the absolute value sum of the differences between the pixels in the block in the captured image and the original block requires a large amount of calculation, which complicates the circuit configuration and makes it difficult to implement in a microcomputer. .

Therefore, there has been proposed a target tracking device for tracking the target using the color information of the color image signal in addition to the shape information of the target (Japanese Patent Application No. 5-61060). This target tracking device stores a certain fixed color as a feature amount, and based on the color, integrates a color image in each of the horizontal direction and the vertical direction. The target is tracked by using the region in contact with the designated region as the target.

Similarly, the following target tracking device has been proposed for tracking a target object using color information (Japanese Patent Application No. 5-261678). This target tracking device models the color of the target from a color difference signal (that is, the RY signal and the BY signal) of the target with a linear function, and sets the luminance Y and the saturation S of the target to 1. A predictive coefficient obtained by the two means is used as a feature amount.

In this target tracking device, an evaluation function is created for each of the horizontal and vertical directions from the error between the data calculated from the prediction coefficient and the actual data, and the target object is determined based on this evaluation function. It is designed to be tracked. However, in the case of this target tracking device, since the color of the target is approximated by a linear function, a model error is likely to increase due to a hue shift due to a change in luminance, and it may be difficult to detect a feature amount.

Therefore, in the above-mentioned target tracking device, a device in which the means for modeling the luminance Y and the saturation S of the target with a linear function is replaced with means for modeling a quadratic function is compared with the above method. Therefore, it has been proposed as an apparatus capable of increasing the modeling accuracy of the target color and improving the tracking ability of the target. However, in this case, the color setting is too strict, and stable target identification may be difficult. Therefore, some error is expected for the model, but it is difficult to set the error amount, and if the error amount is set too large, there is a problem that the target object and the background color cannot be distinguished. Moreover, it was impossible to identify the achromatic target.

The present invention has been made in consideration of the above points, and proposes a target tracking device capable of tracking a target object with higher accuracy without being affected by a change in brightness accompanying the movement of the target object. It is what

[0010]

In order to solve such a problem, in the present invention, the color prediction function determining means is used to convert the first and second color difference signals included in the attention area of the input color image signal. Therefore, the hue of the target object is determined, and the first and second color difference signals are modeled by a color prediction function having luminance as a variable according to the hue. Further, the allowable error model determining means calculates an allowable error model for each of the first color prediction function corresponding to the first color difference signal and the second color prediction function corresponding to the second color difference signal. By comparing this tolerance model with the input color image signal, a region that fits the tolerance model is extracted. Then, the position where the correlation between the reference feature plane obtained as the pixel values of these regions and the feature plane obtained for the input color image signal input after the setting of the reference feature plane is maximized is determined as the position where the target exists. To do so.

[0011]

[Operation] When setting the characteristics of the target object, the first image of the target object is taken.
By modeling each of the color difference signals and the second color difference signal with a color prediction function having a luminance as a variable, it is possible to capture a feature without paying attention to a change in hue with respect to a change in luminance. Further, by directly modeling the first and second color difference signals, it is possible to detect the target object without being caught by the hue.
This makes it possible to capture the characteristics of an achromatic target object. Further, by setting an allowable error model for the first and second color difference signals, it is not necessary to strictly adjust the parameters when setting the characteristics of the target object, and stable tracking is possible even with a change in brightness. Operation can be realized.

[0012]

【Example】

(1) First Embodiment Here, each color signal obtained for a subject as a target, that is, an RY signal and a BY signal is modeled into a quadratic function represented by luminance Y, Further, a target tracking device will be described in which an allowable error range for this quadratic function model is determined as an allowable error model, so that the subject can be reliably identified on the screen.

(1-1) System Configuration FIG. 1 shows the system configuration of the target tracking device 1. By the way, in this embodiment, the pixel values (Y, R) located at the j-th position in the horizontal direction and the i-th position in the vertical direction of the image stored in the memory.
-Y, B-Y) respectively Y _ij, R _ij, to be expressed as B _ij.

The image pickup device 2 is used for picking up an image of a subject, which is a target, and is composed of a television camera or the like. The color image signal TV1 of the subject photographed by the image pickup device 2 is output to the Y / C separation and A / D conversion circuit 3. The Y / C separation and A / D conversion circuit 3 converts the input color image signal TV1 into luminance signals Y, RY signals and BY.
The signals are Y / C separated, and these signals are independently A / D converted and output to the image memory 4.

The image memory 4 is used to store the luminance signals Y, RY signals and BY signals of the images sequentially picked up and taken in by the image pickup device 2, and under the control of the memory control circuit 5. The image data is updated accordingly. The memory control circuit 5 controls the state of the image memory 4 according to the memory control signal MC.

When the memory control circuit 5 identifies that the writing of the image data in the image memory 4 is completed by the memory control signal MC, the interrupt signal INT is sent to the microcomputer unit (hereinafter referred to as MPU unit) 6. The image data is output and notified that the image data in the image memory 4 can be accessed.

When the memory control circuit 5 identifies from the write inhibit signal PRT that the MPU unit 6 has started processing of the newly fetched image data, the memory control signal MC sends the image data to the image memory 4. Write protection and data protection. On the other hand, when the memory control circuit 5 identifies from the write inhibit signal PRT that the MPU unit 6 has completed processing of the image data currently fetched, the memory control signal MC indicates that the image data in the image memory 4 has been processed. Remove protection.

The MPU unit 6 takes in the characteristic amount of the target object from the setting screen based on the image data read out from the image memory 4, and also the signal processing used for searching for the target object in all the screens imaged after the acquisition. It is a department. M
The setting process of the target in the PU unit 6 is performed when the setting start signal S _on output when the switch 8 provided in the horizontal / vertical drive device 7 is pressed is input, Is identified as a target object and the feature amount is extracted.

Then, the MPU unit 6 continues the operation of detecting an object that matches the feature amount from the imaged image captured from the image pickup device 2 until the new object is reset, and operates to track the object. Further, the MPU unit 6 outputs a motor drive signal MD to the horizontal / vertical drive unit 7 based on the moving direction and the moving amount of the target detected from the captured image so that the target is always located in the center of the image pickup screen. The imaging direction of the imaging device 2 is moved in the horizontal direction and the vertical direction.

(1-2) Target Tracking Process (1-2-1) Outline Next, a detailed processing procedure executed in the MPU unit 6 will be described. The outline of this target tracking processing is shown in FIG. The target tracking process includes the subject setting process shown in step SP2,
It can be roughly classified into the tracking processing of the subject shown in step SP3. The detailed processing procedure executed in steps SP2 and SP3 is shown in FIGS.

First, an outline of a subject setting processing procedure will be described with reference to FIG. When the microprocessor of the MPU unit 6 starts the internal processing from step SP11, it determines whether or not to end the series of processing operations in the next step SP11. If a positive result is obtained here, the MPU unit 6
The microprocessor moves to step SP12 to end the target tracking process.

On the other hand, when a negative result is obtained, the microprocessor moves to step SP13 to release the write-protection of the image memory 4 and fetches the image picked up by the image pickup device 2 into the image memory 4. Let Then, the microprocessor waits for the input of the interrupt signal INT from the memory control circuit 5 in the next step SP14.

When the capture of a new captured image is finished, the microprocessor of the MPU unit 6 will start step SP.
15, the write inhibit signal PRT is output to the memory control circuit 5 to instruct the memory control circuit 5 to retain the memory contents. After that, the microprocessor proceeds to the process of extracting the feature amount of the object which is the target in step SP16. In the processing of step 16, the microprocessor creates a quadratic function model for the color signal extracted from the subject, and additionally creates a tolerance model.

When the setting of the feature amount is completed, the microprocessor proceeds to the processing shown in step SP17 and proceeds to the determination processing of whether or not to start tracking the subject. While the negative result is obtained, the process returns from step SP17 to step SP11 and repeats a series of operations. When a positive result is obtained in step SP17, the microprocessor proceeds to steps SP18 and SP1 shown in FIG.
In 9 and SP20, an instruction is given to load a new captured image into the image memory 4.

When new image data is stored in the image memory 4 by these processes, the microprocessor extracts, in step SP21, an area having a feature amount substantially the same as the feature amount of the subject from the screen. Subsequently, the microprocessor moves to step SP22 and detects the position of the target object from these extracted areas.

After that, the microprocessor controls the peripheral equipment based on these positions in step SP23. Incidentally, in this embodiment, the peripheral device is the horizontal / vertical drive device 7. After that, the processing of the microprocessor returns to step SP17, and these processing is repeated until the end of the subject tracking is instructed.

(1-2-2) Processing in Each Part The outline of the target tracking processing has been described above. Of these, the object feature amount extraction processing SP16 and the object position detection processing S are performed.
A more detailed processing procedure executed in P22 will be described.

(1-2-2-1) Object Feature Extraction Processing FIG. 5 shows the detailed processing procedure of step SP16. First, the microprocessor starts the process from step SP16-1. The microprocessor determines the hue of the area D (for example, 5 × 5 blocks) centered on the center of the screen in step SP16-2 under the condition that the subject is located in the center of the screen.

At this time, the microprocessor obtains the average value (Y _ave , R _ave , B _ave ) for the combination of the pixel values (Y _ij , R _ij , B _ij ) in the area D, and the average color signal R
By calculating the ratio of _ave and R _ave , θ corresponding to the hue is determined. Further, in this step SP16-2, the microprocessor reads from the ROM the parameters necessary for approximating the RY signal and the BY signal by a quadratic function having the brightness as a variable.

The parameters read here are RY
In addition to the brightness values R _lo and R _{hi at which the} signal value becomes 0 and the brightness values B _lo and B _{hi at which} the BY signal value becomes 0, the R-Y signal and B-
Maximum values R _max , B _max and minimum value R that the Y signal value can take
_min and B _min . Here, the brightness values R _lo and R _hi are end points of the quadratic function model which is a color signal prediction function. Here, FIG. 6 shows a color signal prediction function F _r (Y) obtained for RY signals.
Is an example of. Incidentally, the microprocessor also determines the achromatic color in this step SP16-2, and determines the end points so that the quadratic function becomes as blunt as possible. The achromatic color is determined here when both the average values R _ave and B _ave are below a certain value.

When the processing of these steps SP16-2 is completed, the processing of the microprocessor will proceed to step SP16.
Move to -3. Here, the parameters obtained in the previous step SP16-2 and the pixel values (Y _ij , R _ij , B) of the area D are used.
_ij ) based on

[Equation 1] as well as

[Equation 2] The coefficients a ₀ and a ₁ of the quadratic function models F _r (Y) and F _b (Y) given by are obtained by least-squares approximation. Accordingly, the color prediction functions F _r (Y) and F of the RY signal and the BY signal are obtained.
Determine _b (Y).

In this way, the color prediction function F for the subject
If _r (Y) and F _b (Y) is set, the processing of the microprocessor proceeds to the processing of step SP16-4. In step SP16-4, the microprocessor creates an error table for the color signal tolerance function. FIG. 7 shows the relationship between the color signal tolerance function and the error table. Now, the microprocessor processes the RY signal and BY of the area D _ij.
Assuming that the signal contains an error, the RY signal and BY
The upper limit error tables H _r (Y) and H _b (Y) and the lower limit error tables L _r (Y) and L _b (Y) are calculated for each of the signals, and the allowable error range is determined. The brightness Y is 0-2
Since an integer value of 55 is taken, if the error table is created in advance, it is possible to reduce the time required to calculate the equations (1) and (2) each time.

If the error is, for example, 10%, the upper limit error table H _r (Y) and the lower limit error table L _r (Y) are determined as follows. For example, if the RY signal is a positive value, the upper limit error table H _r (Y) is set by multiplying the coefficient a ₀ of the equation (1) by 1.1, and the coefficient a ₀ of the equation (1) is set.
Lower limit error table L _r (Y) by multiplying ₀ by 0.9
Set. On the other hand, if R is a negative value, the coefficient a in equation (1)
By multiplying ₀ by 0.9, the upper limit error table H _r (Y)
Is set, and the coefficient a ₀ of the equation (1) is multiplied by 1.1 to set the lower limit error table L _r (Y). B-
The upper limit error table H _r (Y) and the lower limit error table L _r (Y) for the Y signal are calculated in the same manner.

When the calculation of the color signal prediction function and the error table for the subject is completed in this way, the microprocessor proceeds to the processing of step SP16-5 and proceeds to the feature amount extraction processing of the subject. First, the microprocessor calculates the following equation for each point (Y _ij , R _ij , B _ij ) in the area D.

(Equation 3) as well as

[Equation 4] The position (i, j) that satisfies the relationship of is extracted, and the feature amount C (i, j) is set for the extracted position (i, j).

Here, the feature quantity C (i, j) is expressed by the following equation.

(Equation 5) Or

(Equation 6) Etc. are possible. In this embodiment, the expression (6) is used as the feature amount C.
(I, j), the feature quantity C (i, j
For j), the value is 0.

When the feature quantity C (i, j) extraction processing is completed in this way, the microprocessor proceeds to step SP1.
Moving to 6-6, the luminance Y (i, j) at the position (i, j) satisfying the expressions (3) and (4) in the region D is set as the reference feature amount R.
Store as (i, j). The reference feature amount R (i, j) stored at this time becomes a reference block for block matching at the time of detecting the position of the subject, which will be described later. The above processing is executed in step SP16 in the subject setting processing.

(1-2-2-2) Object Position Detection Processing Next, the detailed processing procedure executed in step SP22 of the object tracking processing will be described with reference to FIG. In the previous step SP21, the microprocessor obtains the position (i, j) satisfying the expressions (3) and (4) for the entire screen by the same processing as the step SP16-5 in the previous section, and determines these positions (i, j). ), The characteristic plane PC (i, j) is set with the characteristic amount C (i, j) as luminance Y (i, j).

The microprocessor moved to step SP22-2, the feature plane PC (i, j) and the reference feature amount R (i,
j) the correlation value MS _ij

(Equation 7) It is calculated based on. In the case of this expression, the position having the highest correlation is obtained as the minimum correlation value MS _ij . In this way, the block matching is the process of shifting the reference feature amount R (i, j) within the feature plane PC (i, j) and calculating the correlation value MS _ij at each point. After the completion of this block matching, the microprocessor obtains the minimum value MS _min of the correlation value MS _ij and the position (x _min , y _min ) where this value is obtained, and proceeds to the next step SP22-3.

The microprocessor is a step SP22-
Moving to 3, in order to determine whether the image having the highest correlation is effective as the target subject, the correlation value MS
_ij is compared with a preset threshold value TH _MS . At this time, if the correlation value MS _ij is less than or equal to the threshold value TH _MS , the detected position (x _min , y _min ) is set as the object position, and the presence / absence flag is set to "1". On the other hand, when the correlation value MS _ij is larger than the threshold value TH _MS, it is determined that the detected position (x _min , y _min ) indicates the same color area other than the subject, and the presence / absence flag is set to "
Set to 0 ".

When the discrimination processing regarding the existence of the subject is completed, the microprocessor proceeds to step SP22-4.
Processing, the reference feature amount R (i, j) is obtained for an area centered on the position (x _min , y _min ) detected only when the presence / absence flag is “1”, and this value is calculated. The reference feature amount R (i, j) stored until now is stored instead of the reference feature amount R (i, j). As a result, it is possible to deal with a change in shape due to movement.
The microprocessor is based on the position (i, j) calculated by these processes and the judgment result of the presence / absence of the target.
The motor of the horizontal / vertical drive device 7 shown in FIG. 2 is controlled to move the shooting direction of the image pickup device 2 so that the target object is automatically tracked so that the target object is always within the screen.

(1-3) Effects According to the above configuration, each color signal of the subject, that is, R-
The Y signal and the BY signal are modeled into quadratic functions F _r (Y) and F _b (Y) represented by the luminance Y, respectively, and an allowable error range is allowed for the quadratic function model. Error models H _r (Y), L _r (Y) and H _b (Y), L _b (Y)
By setting the above, it is possible to realize a target tracking device that can reliably track an achromatic subject.

Further, according to the above configuration, it is possible to realize a target tracking device capable of absorbing the change of the characteristic amount C (i, j) due to the change of the illumination condition and accurately extracting the characteristic amount regardless of the illumination change. can do. Furthermore, in the block matching, the area on the feature plane PC (i, j) and the reference feature amount R
By performing the matching process with (i, j), the shape information can be used for identifying the subject, and a target tracking device that can avoid erroneous recognition with surrounding objects having similar color information can be realized.

(2) Second Embodiment In this embodiment, in the subject tracking process executed after the subject setting process is completed, the neighborhood of the region extracted from the screen based on the color signal prediction function and the tolerance model. The target tracking device that can realize a stable tracking operation even when the error evaluation is strict by re-evaluating the pixel values of, and expanding the area that should be determined to be the same color as the subject before starting the block matching processing. Will be described.

(2-1) Target Tracking Process In this embodiment, the object setting process is the same as the process procedure of FIG. 3, and therefore only the object tracking process including the process different from the first embodiment will be described. . FIG. 9 shows the tracking process of the subject in this embodiment. By the way, in FIG.
The same processing parts as those in FIGS. 4 and 8 are designated by the same reference numerals. Therefore, step SP20 and step SP22
The processing of step SP31, step SP32, and step SP33 inserted between -2 and -2 is processing unique to this embodiment. Hereafter, this step SP31-SP
The process of 33 will be mainly described.

Now, the microprocessor is step SP3.
1, the position (i, j) having a pixel value satisfying the expressions (3) and (4) is obtained for the entire screen, and the feature amount C (i, j) at the position (i, j) satisfying the expression 2 is obtained. Put "0" and 2
The feature quantity C (i, j) at the position (i, j) that does not satisfy the expression is
By setting "1", the feature plane PC (i, j) is set.

When the calculation process of the characteristic plane PC (i, j) having the entire screen as an area is completed in step SP31,
The microprocessor moves to the processing of step SP32,
The color is re-evaluated for the pixels in the vicinity of the area where the feature amount C (i, j) is determined to be "0". For example, the feature amount C (i, j-1)
Is "0" and the feature amount C (i, j) is "1", the color is re-evaluated. Here revaluation position (i, j) the allowable value TH to the pixel value R _ij and B _ij are preset in the following formula

[Equation 8]

[Equation 9] It is executed by checking whether or not the above conditions are satisfied.

Here, when the expressions (8) and (9) are both satisfied, the microprocessor reevaluates the feature quantity C (i, j) at the position (i, j) to "0", Further, the same re-evaluation process is continued. The above-described re-evaluation processing is re-evaluation from left to right pixels in the horizontal direction, but similarly, re-evaluation from right to left pixels, re-evaluation from top to bottom pixels in the vertical direction, and re-evaluation. The re-evaluation from the pixel to the pixel above is sequentially executed. After the processing in these four directions is completed, the expansion processing of the area regarded as the same color as the subject is completed, and the microprocessor proceeds to the processing of step SP33.

Now, moving to step SP33, the microprocessor determines the position (i, j) where the feature amount C (i, j) is "0".
Is assigned luminance Y _ij , and the feature amount C (i, j) is "
The feature plane PC (i, j) is set by assigning a fixed value TH _Y to the position (i, j) of 1 ″. The fixed value TH _Y is a value set in advance and is the same as the subject. It has a meaning to show that it is not a color, for example, the brightness Y is 8.
If it is a bit, a value such as "256" or "0" is set as the fixed value TH _Y. The subsequent processing is the same as that in the first embodiment, and the description thereof is omitted.

(2-2) Effect With the above configuration, the pixel values in the vicinity of the area extracted from the screen should be re-evaluated based on the color signal prediction function and the allowable error model, and the same color as the subject should be determined. By expanding the areas and then shifting to the block matching processing of these areas and the reference feature value R (i, j), the error evaluation becomes strict, so that the areas outside the allowable error range are also included in the subject. Since the existence can be confirmed, it is possible to realize a target tracking device capable of performing a stable tracking operation as compared with the related art.

(3) Third Embodiment In this embodiment, when a subject is detected from within the screen, the detection area is limited to a part of a specific area instead of the whole of the image pickup screen, so that an object having a color similar to that of the object is obtained. A target tracking device having improved object discrimination power with respect to will be described.

(3-1) Target Tracking Process FIG. 10 in which parts corresponding to those in FIG. 3 are assigned the same reference numerals shows a procedure for setting an object used in the target tracking device of this embodiment. Further, FIG. 11 in which parts corresponding to those in FIG. 4 are denoted by the same reference numerals shows a subject tracking processing procedure used in the target tracking apparatus of the present embodiment.

As can be seen from FIGS. 10 and 11, in the case of the processing procedure of this embodiment, after the steps SP4 and SP19, step S for cutting out a specific area from the entire screen.
P41 and SP42 are provided, and a step SP43 for resetting the position to cut out the specific area is provided after step SP22. There is.

However, the processing of steps SP16, SP17 and steps SP21, SP22 is different from the case of FIG. 3 in that only the specific area cut out from the entire screen is processed. Incidentally, in this embodiment, as shown in FIG. 12, it is assumed that the size of the entire image is 128 × 128 pixels, and the size to be cut out as the specific area is 64 × 64 pixels.

First, when the subject is set, the microprocessor of the MPU unit 6 interrupts the interrupt signal I at step SP4.
When NT is confirmed and it is confirmed that the image memory 4 can be accessed, the process proceeds to step SP41 and 64 × 64 centered on the center position (x, y) of the screen.
The cutout start position (x start,
y start) is calculated. After this, the microprocessor moves to step SP15 and prohibits rewriting of all pixel data written in the image memory 4.

Further, the microprocessor shifts to the feature amount extraction processing of the object for only the specific area A cut out in the next step SP16, and creates a color prediction function for specifying the color of the object and its allowable error table. Then, the characteristic amount C (i, j) is obtained. When the feature amount C (i, j) for the subject is obtained in this way, the microprocessor shifts from step SP17 to step SP18 and starts the subject tracking process.

Upon confirming that new image data has been taken in at steps SP18 and SP19, the microprocessor which has started the tracking processing confirms the cut-out start position (based on the designated position giving the specific area A by the processing shown in step SP42). x start, y start) is calculated.
Note that the specific area A immediately after the tracking process is started
The specified position of and the center position of the screen match.

When the specific area A is determined in this way, the microprocessor shifts to the process of extracting the area of the same color as the feature amount C (i, j) of the subject from the specific area A, and the specific area given by 64 × 64 pixels is specified. The characteristic plane PC (i, j) is obtained from the area A, and the position of the subject is detected. Now, when the position of the subject is detected in steps SP21 and SP22 in this way, the process moves to a process of detecting the cut-out position of the specific area A used in the next process in step SP43.

Here, when the subject is present in the specific area A, that is, when the valid / invalid flag is "1" and the detected position is valid, the microprocessor is centered on the obtained position. The position of the subject is substituted for the specified position to cut out the area. On the other hand, if the subject does not exist in the specific area A, the microprocessor

[Equation 10] As shown in, the cutout position of the specific area A is sequentially switched until the subject is found.

That is, when the valid / invalid flag is "0" and the detected position is invalid, the microprocessor processes the cutout start position (x start, in the 128 × 128 image).
y start) from the point (x ₀ , y ₀ ) to (x
₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ), and (x ₄ , y ₄ ). At this time, the microprocessor switches in order according to the number of times t when the subject is determined to be invalid. However, when the number of times t exceeds T5, the number of times t is returned to "0" and the search operation is restarted from the point (x ₀ , y ₀ ). In this way, the target tracking device switches the specific area A for detecting the subject until the effective position is detected.

Incidentally, in the case of this embodiment, step SP43
Even if the position of the subject was calculated during the process of
When the calculated position is a position that cannot continuously exist for a time, it is determined that the detected position is invalid and the process corresponding to the number of times t is continued. On the other hand, when the positions are continuously detected in the same area, the detected position is determined to be valid and used in the next image cutout. The reliability of the position of the subject detected by these determination processes can be improved.

(3-2) Effect According to the above configuration, in setting the subject and in the tracking process from the screen, the specific region A in which the subject is likely to exist until just before the region used for detecting the subject. The target tracking device can improve the discriminating power of the subject even when the color of the background and the like includes many colors similar to the subject, and can efficiently detect the subject from the image. You can

Also, when the subject is lost, the specific area A
The detection process of the subject can be continued while sequentially switching the region on the screen to be cut out, and if the detected position of the subject does not show time continuity, this is not recognized as a valid position. As a result, a target tracking device with high detection accuracy can be realized.

(4) Fourth Embodiment In the above-described target tracking processing, the target tracking device which mainly tracks an object moving in the direction of the image plane has been described, but in the present embodiment, the zoom lens control of the image pickup device 2 is performed. A target tracking device that tracks a subject moving in the perspective direction so that the size of the subject on the screen becomes substantially constant will be described.

(4-1) System Configuration The target tracking device 11 shown in FIG. 14 in which parts corresponding to those in FIG. 1 are assigned the same reference numerals controls the zoom lens 12 of the image pickup device 2 by the control of the MPU unit 6. It has a similar system configuration except that Incidentally, this embodiment does not include the horizontal / vertical drive device 7 for moving the photographing direction of the image pickup device 2 in the horizontal direction and the vertical direction, but a system in which the horizontal / vertical drive device 7 is added to the system of the embodiment. Can also be applied.

In the case of this embodiment, the microprocessor of the MPU unit 6 is a feature plane PC extracted from the picked-up image.
The area of the subject is calculated based on (i, j), and the zoom lens 12 is controlled based on the calculation result. Next, a processing procedure executed in the MPU unit 6 will be described.

(4-2) Target Tracking Process In the case of this embodiment, the object setting process is the same as the process procedure of FIG. 3 described in the first embodiment, so only the object tracking process procedure will be described. To do. The tracking processing procedure in this embodiment is shown in FIG. Incidentally, in the processing procedure in FIG. 15, the processing corresponding to that in FIG. 4 is denoted by the same reference numeral.

As can be seen from FIG. 15, this processing procedure is the same as that shown in FIG. 15 except that two processings are provided after step SP22 for detecting the position of the object from the feature plane PC (i, j) obtained for the image pickup screen. The processing procedure is similar to the processing procedure of No. 4. The two newly added processes are the one-dimensional feature amount derivation process of step SP51 and the area calculation process of step SP52.

Here, step SP51 is the previous step S
Object position and feature plane PC calculated in P22
This is a process for obtaining the horizontal length and the vertical length of the subject from (i, j), and step SP52 is a process for obtaining the area from the horizontal and vertical lengths.

Then, two steps SP51 and SP5
Specific processing contents executed in 2 will be described with reference to FIGS. 16 and 17. The microprocessor of the MPU unit 6 creates a color prediction function for the color of the subject set by the user by a series of processes up to step SP22, and then uses the color prediction function or the like to generate the feature plane PC.
In addition to (i, j), the presence / absence of a subject and the position (x, y) of the subject are calculated.

In this state, when the microprocessor shifts to the processing of step SP51, the signal sequence HOR obtained by making the feature plane PC (i, j) one-dimensional in the vertical direction.
j and a signal sequence VERi obtained by one-dimensionalizing in the horizontal direction are created. Here, an example of creating the signal train VERi is shown in FIG. Further, the signal train HORj can be similarly created.

By the way, although certain numerical values are written in the signal sequence VERi in FIG. 16, these numerical values represent the characteristic quantity C (k, j) at the k-th column on the characteristic plane PC (i, j). It is calculated as follows. First, for each point (k, j) of the feature amount C (k, j) in the k-th column, the distance L between the point determined to be the same color as the subject and the position (x, y) of the subject is L =
It is determined as | k−x |. Then, the minimum value L _min after the distance to the position of the same color as the subject obtained on these k columns is obtained.
Is calculated as the value of VERk.

In the case of FIG. 16, a medium-sized fruit is a subject, and the feature amount C (k, k,
Since the distance L to j) is 4, the value of VER0 is calculated as 4. However, if there is no same color as the subject in that row, the set value V
Substitute _MAX . These processes are performed by the signal train HO.
The processing in step SP51 is also executed for Rj.

When the signal trains VERi and HORj are obtained in the horizontal direction and the vertical direction, respectively, the processing of the microprocessor continues at step SP5.
The procedure moves to 2 to calculate the lengths L _HOL and L _VER of the subject in the horizontal direction and the vertical direction, and to move to the process of obtaining the area of the subject based on the product L _HOL × L _VER .

Each length L in the horizontal and vertical directions
How to obtain _HOL and _LVER will be described with reference to FIG. Incidentally, although FIG. 17 shows the distance L _HOL in the horizontal direction, the distance L _{VER in the} vertical direction can be calculated in the same manner. When the processing of step SP51 is finished, the microprocessor searches for the end points m and n separately for the position (x, y) of the previously detected subject on the right side and the left side of the coordinate value x.

For example, the end point m located on the right side of the coordinate value x
Is a signal train H that is _{equal to} or greater than a preset threshold value TH _HOL.
Obtained by detecting the position of OLm. Also, in the case of the end point n located on the left side of the coordinate value x, similarly, it is obtained by detecting the position of the signal train HOLm that is equal to or more than the preset threshold value TH _HOL . And 2 obtained in this way
The difference between the two positions is the horizontal distance L _HOL = m−n. Since the vertical distance can be calculated in the same way,
The area of the subject can be obtained from the two lengths.

When the subject area setting processing in these steps SP52 is completed, the microprocessor moves to step SP23 and compares the calculated area with the preset subject area to move the zoom lens 12 back and forth. Control the motor. By the way, as the area of the subject, the area when the subject tracking is set or the area designated by various switches is used.

In actual control, the direction in which the zoom lens 12 is moved (far side or close-up side) is determined depending on whether the difference dS between the calculated area and the set area is positive or negative, and | dS | is used to zoom. The speed at which the lens 12 is moved is determined. The variation of the area due to the movement of the zoom lens 12 affects the next area extraction processing, and the zoom lens 1 is moved until the area becomes a predetermined area.

(4-3) Effect According to the above configuration, a quadratic function F _r (Y) in which the RY signal and the BY signal of the subject are represented by the brightness Y, respectively.
And F _b (Y) and the allowable error range for this quadratic function model is defined as allowable error models H _r (Y) and L _r.
(Y) and H _b (Y), L _b (Y) as a feature plane PC
By detecting the shape of the subject from (i, j) and controlling the position of the zoom lens 12 based on this detection result, it is possible to perform zoom photography so that the area of the subject is always constant. A tracking device can be realized.

(5) Fifth Embodiment In this embodiment, a frame corresponding to the size of the subject is generated by using the area detection function of the subject described in the previous embodiment, and the frame is displayed so as to be superimposed on the subject on the screen. A target tracking device capable of performing will be described.

(5-1) System Configuration FIG. 18 in which parts corresponding to those in FIG. 1 are assigned the same reference numerals shows the system configuration of the target tracking device 21 described in this embodiment. The target tracking device 21 has a similar configuration except that it has a frame image synthesizing device 22 for displaying a frame W on the color image signal TV1 based on the position and area information POS of the subject detected by the MPU unit 6. Have Incidentally, in this embodiment, the output of the frame image synthesizing device 22 is displayed on the screen of the display device 23.

(5-2) Target Tracking Process and Frame Combining Process In this embodiment, the subject setting process executed by the MPU unit 6 is the same as that shown in FIG. Only the frame composition processing procedure will be described. The tracking processing procedure in this embodiment is shown in FIGS. 19 and 20. Incidentally, in the processing procedure in FIG.
The same reference numerals are given to the processing of the portions corresponding to 5.

As can be seen from FIG. 19, this processing procedure is the same as that of the fourth embodiment except for the processing after the area calculation processing shown in step SP52. Therefore, only steps SP61 and SP62 will be described. Now, step SP61 is a process of calculating the position data of the four corners of the frame W from the position of the subject detected up to the previous step, the presence or absence of the subject, and the length in each direction.

The contents of this step SP61 are shown in FIG. The microprocessor branches the process depending on the presence of the subject, and if there is no subject, step SP16.
From -2 to step SP16-3, the frame position is calculated. Here, the microprocessor calculates the end points x ₀ and x ₁ in the horizontal direction from the position x of the subject and the horizontal length L _{HOL as} follows.

[Equation 11] And the end points y ₀ and y ₁ in the vertical direction from the position y of the subject and the length L _VER in the vertical direction.

[Equation 12] Calculate as. As a result, frame data (x ₀ , y ₀ ), (x ₀ , y ₁ ), (x ₁ , y) of four corners of the subject is obtained.
₀ ) and (x ₁ , y ₁ ) are output.

On the other hand, if the subject is lost, the process moves from step SP16-2 to step SP16-4.
In order that the frame W is not displayed on the screen of the display device 23, the positions outside the display screen are output as the frame positions, or the positions of the four corners are output as the frame positions so that the frame W has one point. Has been done. Then, in step SP62,
By displaying the frames given at these frame positions on the screen, a frame W that always surrounds the periphery is displayed in an overlapping manner regardless of the size and position of the subject and the movement of the subject.

(5-3) Effects According to the above configuration, a quadratic function F _r (Y) in which the R-Y signal and the B-Y signal of the subject are represented by the luminance Y, respectively.
And F _b (Y) and the allowable error range for this quadratic function model is defined as allowable error models H _r (Y) and L _r.
(Y) and H _b (Y), L _b (Y) as a feature plane PC
By detecting the shape of the subject from (i, j) and generating a frame W having a size corresponding to the area and position of the subject based on the detection result, the frame W is always surrounded by the subject.
It is possible to realize a target tracking device having a function of being surrounded by.

(6) Other Embodiments In the above embodiment, the color signal prediction function is set to the luminance Y.
The allowable error models H _r (Y), L _r (Y) and H _b (which are modeled by the quadratic functions F _r (Y) and F _b (Y) represented by Y) and L _b (Y) are used to detect the position of the object using the feature plane PC (i, j), but the present invention is not limited to this, and the brightness near the position of the object is considered. The iris may be controlled so that the average value of is calculated every time and the average value of the brightness becomes constant. A target tracking device having an auto iris function may be realized by using such a method.

Further, a target tracking device having an autofocus function may be realized by detecting a frequency component of luminance near the position of the subject and using it for focus control.
Furthermore, the function of controlling the posture so that the detected position of the subject is always in the center of the screen may be applied to image stabilization. Further, in the above-mentioned embodiment, the case where the subject in the image is tracked has been described, but the present invention is not limited to this, and the position of the player's hand or face in the image is tracked so that the game machine or the computer is controlled. It is possible to realize a pointing device similar to a mouse or the like in etc.

In the above embodiment, the image pickup device 2
Although the case where the target is tracked based on the color image signal TV1 input from the above is described, the present invention is not limited to this, and an image signal reproduced by a video tape recorder, a computer, a game machine or the like is used as an input signal, The targets set in these image signals may be tracked.

Further, although the basic system of the target tracking device has been described in the above embodiments, the target tracking device of the present invention can be applied to a TV telephone system and a security system. Further, it can also be applied to a system such as a multimedia system for tracking a subject with a television camera.

Although the quadratic function is used as the color signal prediction function in the above-described embodiments, the present invention is not limited to this, and other functions such as a linear function, exponential function, logarithmic function, trigonometric function, elliptic function, etc. The function may be used to predict the color. Further, in the above-described embodiment, the method of expanding one-dimensionally when expanding the color region has been described, but the present invention is not limited to this and can be applied to a method of expanding two-dimensionally.

Further, in the above-described embodiment, the area of the object obtained by the area calculation processing is used to make the zoom lens 12
However, the present invention can be applied to camera control such as subject extraction type iris control / focus control. Further, the present invention is not limited to this, and may be applied to a toy that moves forward and backward in accordance with an increase / decrease in area, an image encoding device that performs adaptive quantization that allocates more information in an area than information in other areas in the screen, and the like. It can be applied.

Further, in the above embodiment, the case where the area is calculated based on the position and distance of the subject has been described, but the present invention is not limited to this, and the number of points having the same color as the subject in the area is calculated. You may ask for it by a thing. Further, in the above-described embodiment, the case where the frame W is generated in the frame image synthesizing unit 22 and is displayed on the screen has been described. However, the present invention is not limited to this, display of characters and filling of the entire area, etc. Can also be used for

Further, in the above-described embodiment, the frame is not displayed when there is no subject, but the present invention is not limited to this, and a frame W different from the type surrounding the subject may be displayed. Characters may also be displayed. Further, this processing can be applied to switching of display colors.

[0095]

As described above, according to the present invention, each of the first and second color difference signals is modeled by using the luminance when the feature of the target object is set, and the allowable error model defined for this model is used. By capturing the characteristics of the target object by using the target tracking apparatus, it is possible to realize a target tracking device that can stably track the target object without worrying about a change in hue due to a change in luminance.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of a target tracking device according to the present invention.

FIG. 2 is a flow chart showing an outline of a tracking processing procedure.

FIG. 3 is a flowchart showing a procedure for subject setting processing.

FIG. 4 is a flow chart showing a procedure for subject tracking processing.

FIG. 5 is a flow chart showing a procedure of subject feature amount extraction processing.

FIG. 6 is a schematic diagram for explaining a color signal prediction function.

FIG. 7 is a schematic diagram for explaining a tolerance table.

FIG. 8 is a flow chart showing the procedure of subject position detection processing.

FIG. 9 is a flow chart showing a procedure for subject tracking processing.

FIG. 10 is a flowchart showing a procedure for setting a subject.

FIG. 11 is a flow chart showing a procedure for subject tracking processing.

FIG. 12 is a schematic diagram showing a characteristic region cut out from the screen.

FIG. 13 is a schematic diagram showing a relationship between a specific region and a designated cutout position.

FIG. 14 is a block diagram showing an embodiment of a target tracking device according to the present invention.

FIG. 15 is a flow chart showing a procedure for subject tracking processing.

FIG. 16 is a schematic diagram illustrating a configuration example of a one-dimensional signal sequence.

FIG. 17 is a schematic diagram showing an example of distance calculation.

FIG. 18 is a block diagram showing an embodiment of the target tracking device according to the present invention.

FIG. 19 is a flow chart showing a subject tracking processing procedure and a frame creation processing procedure.

FIG. 20 is a flow chart showing a procedure for calculating the position of a display frame.

[Explanation of symbols]

1, 11, 21 ... Target tracking device, 2 ... Imaging device, 3
... Y / C separation and A / D conversion circuit, 4 ... Image memory,
5 ... Memory control circuit, 6 ... MPU section, 7 ... Horizontal / vertical drive device, 8 ... Switch, 12 ... Zoom lens,
22 ... Frame image synthesizing unit, 23 ... Display device.

Claims (8)

[Claims] 1. A hue of a target is judged from first and second color difference signals included in an attention area of an input color image signal, and each of the first and second color difference signals is determined according to the hue. , A color prediction function determining means for modeling with a color prediction function having luminance as a variable, a first color prediction function corresponding to the first color difference signal, and a second color prediction corresponding to the second color difference signal. A tolerance model determining means for calculating a tolerance model for each of the functions is compared with the tolerance model obtained for each of the input color image signal and the first and second color prediction functions.
A color information determination unit that extracts a region that matches the allowable error model, and a feature plane that calculates a reference feature plane that represents the target object based on the pixel values of the region extracted by the color information determination unit of the attention region. With respect to the input color image signal, the setting means and the input color image signal input after setting the reference feature plane are compared with the allowable error model to extract a region that fits the allowable error model for the entire screen. The target tracking device, further comprising: position determining means for determining the position where the correlation between the characteristic plane and the reference characteristic plane is maximum as the position where the target exists. 2. A region expanding unit that re-evaluates pixel values adjacent to a region extracted as a region conforming to the tolerance model by the color information determining unit and increases the region conforming to the tolerance model. The target tracking device according to claim 1, further comprising: 3. The position detecting means compares an input color image signal input after setting the reference feature plane with the allowable error model, and determines an area matching the allowable error model as a specific area of the entire screen. The target tracking apparatus according to claim 1, wherein the target plane is extracted only for the specific area and the characteristic plane for the input color image signal is obtained only for the specific area. 4. The target tracking device according to claim 1, further comprising a sending unit that sends the position of the target object detected by the position detecting unit as a position input to a peripheral device. 5. The target tracking device according to claim 1, further comprising control means for controlling peripheral devices based on the position of the target object detected by the position detection means. 6. A size detecting means for detecting the size of the target on the screen from the position of the target detected by the position detecting means and the distribution of pixel values on the characteristic plane. The target tracking device according to claim 1, further comprising: 7. A size corresponding to the size of the target object based on the position of the target object on the screen determined by the position detection means and the size of the target object determined by the size detection means. 7. The target tracking device according to claim 6, further comprising auxiliary image generating means for generating the auxiliary image of. 8. An image pickup device for picking up an image of a target, and a hue of the target is judged from first and second color difference signals included in a region of interest among input color image signals input from the image pickup device, According to the hue, the first and second
A color prediction function determining means for modeling each of the color difference signals with a color prediction function having luminance as a variable; a first color prediction function corresponding to the first color difference signal; and a second color difference signal corresponding to the second color difference signal. Tolerance error model determining means for calculating a tolerance error model for each of the second color prediction functions and the tolerance error models obtained for the input color image signal and the first and second color prediction functions are compared. Then
A color information determination unit that extracts a region that matches the allowable error model, and a feature plane that calculates a reference feature plane that represents the target object based on pixel values of regions extracted by the color information determination unit of the attention region. With respect to the input color image signal, the setting means and the input color image signal input after setting the reference feature plane are compared with the allowable error model to extract a region that fits the allowable error model for the entire screen. Position determining means for determining a position where the correlation between the characteristic plane and the reference characteristic plane is maximum as the position where the target exists, and the imaging device based on the detection result of the position determining means. A target tracking device for driving the image pickup device to direct the photographing direction of the image pickup device to the direction in which the target object is located.