JP2000230809A - Distance data interpolation method, color image layering method, and color image layering device - Google Patents

Abstract

(57) [Summary] [Problem] To provide a method for interpolating distance data in an area where distance measurement could not be performed in a distance image, and a method and apparatus for layering a color image using the distance data. To do. SOLUTION: At the same time as capturing a distance image by a range finder, a color image is captured by a CCD (10). The distance data interpolation unit 17 uses the captured color image information to perform weighting interpolation according to the difference in color between the target pixel and the reference pixel and the difference in pixel coordinate value, thereby converting the distance data of the target pixel to the distance between surrounding pixels. Obtained from the data by interpolation. The hierarchization processing means 18 hierarchizes the color image using the distance data, and the extracted image is synthesized with another image in the image synthesis unit 21.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a distance data interpolation method, a color image layering method, and a color image layering apparatus.

[0002]

2. Description of the Related Art As a range finder device for performing three-dimensional shape measurement based on triangulation of projection light and observation image,
For example, Literature: Yoshimi et al., "Range Finder System with Multiple Light Sources", Journal of the Robotics Society of Japan, Vol.9, No.7, pp.8
03-812, 1991. Light section method and literature: Sato, Iguchi, "Range image input by spatial coding", IEICE Transactions on '85 /3Vol.J68-D No3 Some are based on the pattern light projection method.

[0003] Further, as a conventional technique relating to hierarchization and componentization of an image, a chroma key technique (blue background) for extracting a person in front by using color information while setting the background to a solid blue color.
There is. Here, "hierarchization of images and componentization" means that a part in an image is extracted and separated separately from other parts, and can be processed independently.

[0004]

However, in the distance measurement by triangulation, since the light projection position is different from the light reception position of the reflected light, a part of the subject visible from the light reception position is not visible from the light projection position. It can be invisible. In other words, occlusion caused by the shape of the subject, etc.
In some cases, the distance cannot be measured for some regions of the subject. Such a problem also occurs when the reflectance of light on the surface of the subject is low or when the surface of the subject has a property of scattering light.

Further, according to the above-mentioned chroma key technique, for example, when a person (foreground) is extracted and separated from the background, if the background is a single color and is not a color different from the human body and clothes,
The person cannot be extracted.

In view of the above problems, the present invention provides a method for interpolating distance data in an area where distance measurement cannot be performed in a distance image, and also eliminates the need for a single color background as in a chroma key. It is another object of the present invention to provide a color image layering method and apparatus capable of extracting a foreground of a person or the like from a real image and layering a color image.

[0007]

According to the present invention, distance data is interpolated by weighted interpolation using information of a color image taken of the same object. Since the interpolation is performed in consideration of not only the coordinate values of the pixels but also the color information of the subject, it is possible to interpolate appropriate distance data in consideration of consistency with the color image.

In the present invention, a color image is hierarchized using the distance information of the distance image on which the above-described interpolation processing has been performed. Since the color image is hierarchized based on the depth information of the distance image, it is possible to extract a region such as a person from the photographed image without making the background a single color like a chroma key. Therefore, it is possible to reduce the restrictions accompanying the layering of the color image.

Further, since the distance image that has undergone the interpolation processing has no missing area of the distance data, all the pixels of the color image can be subjected to hierarchization.

In addition, since the distance data is interpolated in consideration of the color difference between the target pixel and the reference pixel,
Also in the interpolation area of the distance data, the boundary between the foreground and the background matches the boundary between the foreground and the background in the color image with high accuracy. Therefore, when the foreground of the color image is extracted separately from the background based on the distance data (depth information of the subject), the contour of the extracted foreground object becomes sharp, and the accuracy of the image hierarchical processing is improved.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In a first aspect of the present invention, distance data of a pixel belonging to an area where distance measurement has not been performed in a distance image is stored in a pixel located close to the pixel and whose distance has been measured. Interpolating by referring to the distance data of
A color image is also acquired for the shooting target of the distance image, and the color of the pixel in the color image corresponding to the target pixel to be interpolated and the color of the pixel in the color image corresponding to the reference pixel are used. A weighting coefficient is calculated, and the calculated weighting coefficient is applied to the distance data of the reference pixel to obtain distance data.

Since interpolation is performed in consideration of not only the coordinate values of the pixels but also the color information of the object, highly accurate distance data interpolation can be performed in consideration of consistency with a color image.

[0013] In a second aspect of the present invention, in the first aspect, the weighting coefficient is determined based on a color of a pixel in the color image corresponding to the target pixel and a color of a pixel in the color image corresponding to the reference pixel. The color difference information is calculated by a calculation formula including the parameter as a parameter.

By adaptively changing the weight coefficient by reflecting the color difference between adjacent pixels, highly accurate distance data interpolation can be performed in consideration of consistency with a color image.

According to a third aspect of the present invention, in the second aspect, the calculation formula for calculating the weighting coefficient is:
The information on the difference between the coordinate values of the target pixel and the reference pixel is included as a parameter.

Thus, high-precision distance data interpolation can be performed in consideration of the coordinate position difference information between the target pixel and the reference pixel and the color information difference information between the corresponding pixels in the color image.

According to a fourth aspect of the present invention, in any one of the first to third aspects, the plurality of pixels whose distance has been measured and which are in contact with the boundary line of the area where the distance measurement has not been performed are formed. Let it be a reference pixel.

By using a plurality of pixels (preferably all pixels) in contact with the boundary line as reference pixels, the accuracy of interpolation is improved.

According to a fifth aspect of the present invention, in any one of the first to third aspects, a plurality of pixels having a distance measured within a predetermined distance from the pixel of interest are set as the reference pixels. .

As a result, the number of reference pixels can be reduced, and the load on the interpolation processing can be reduced.

According to a sixth aspect of the present invention, in any one of the first to third aspects, it is in contact with a boundary line of a region where the distance measurement has not been performed and is within a certain distance from the pixel of interest. The plurality of pixels for which the distance measurement has been performed are defined as the reference pixels.

As a result, the number of reference pixels can be reduced, and the load on the interpolation processing is reduced.

According to a seventh aspect of the present invention, in the first aspect, when the color image is clustered and the distance data in the distance-deficient region is interpolated, the distribution of the distance data in the same cluster becomes smooth. And
At the cluster boundaries, the distance data is discontinuous,
Determine the distribution of distance data.

Generally, if the color differs in a color image, the distance also differs in a distance image. That is,
A cluster boundary detected as a result of dividing the color image into regions is highly likely to coincide with a boundary of distance data in the distance image. From this viewpoint, by performing data interpolation in consideration of the cluster boundary, the accuracy of the distance data interpolation is improved. Further, since the consistency between the distance image and the color image is high, the foreground / background can be accurately separated / extracted when layering the color image using the distance data.

According to an eighth aspect of the present invention, in the first aspect, data interpolation is performed in consideration of an “edge” obtained by performing an edge enhancement process on a color image.

Since the cluster boundary does not always coincide with the contour of the object, the position of the boundary is corrected using the edge information to improve the interpolation accuracy of the distance data.

In a ninth aspect of the present invention, in any one of the first to eighth aspects, the distance image and the color image are images optically captured from the same viewpoint.

As a result, consistency between the distance image and the color image is maintained, and layering of the color image using the distance information is facilitated.

According to a tenth aspect of the present invention, the first aspect to the first aspect
In any one of the eighth aspect, the distance image is equivalent to performing a viewpoint conversion process on a distance image obtained by imaging from a viewpoint different from that of a color image, and imaging the distance image from the same viewpoint as the viewpoint of the color image. This is a distance image converted into a simple image.

Even if the imaging viewpoint of the color image is different from the imaging viewpoint of the distance image due to the configuration of the imaging system, the image is captured at the same viewpoint by the viewpoint conversion process using the distance information of the distance image. Thus, the same distance image can be obtained, and thus, it is possible to layer a color image using the distance image.

According to an eleventh aspect of the present invention, a distance image and a color image are obtained for the same object, the color image is hierarchized using distance information of the distance image, and a part of the color image is replaced with another part. Is a method that can be processed independently in distinction from the above, before the layering of the color image,
It is determined whether there is an area for which distance measurement has not been performed in the distance image, and when there is the area, for a pixel in the area, the method according to any one of claims 1 to 7 A first step of interpolating distance data to eliminate a missing part of the distance data from the distance image; and, based on distance information of each pixel of the distance image having passed through the first step, all pixels of the color image. And a second step of performing a layering process as an object.

Since the color image is hierarchized using the distance information of the distance image subjected to the interpolation processing, it is possible to extract a region such as a person from the photographed image without making the background a single color as in a chroma key. it can. Further, since the distance image which has been subjected to the interpolation processing has no missing area of the distance data, all the pixels of the color image can be subjected to hierarchization. Further, since the distance data is interpolated in consideration of the color difference between the target pixel and the reference pixel, the boundary between the foreground and background in the distance data interpolation region is Matches boundaries and high precision. Therefore, when the foreground of the color image is extracted separately from the background based on the distance data (depth information of the subject), the contour of the extracted foreground object becomes sharp, and the accuracy of the image hierarchical processing is improved.

According to a twelfth aspect of the present invention, a part of a color image hierarchized by the color image hierarchization method of the eleventh aspect is extracted and combined with another image.

Thus, it is possible to perform image composition while eliminating the limitation on the colors of the foreground and the background as in the chroma key technique.

According to a thirteenth aspect of the present invention, there is provided distance data missing area extracting means for extracting an area where distance measurement has not been performed in a distance image obtained by a range finder, and an area where distance measurement has not been performed. Distance interpolation means for interpolating the distance data of the pixels belonging to the pixel by referring to the distance data of the pixels measured close to the pixel, the distance interpolation means comprising: When data is interpolated based on distance data of a reference pixel, a weighting coefficient is obtained using a color of the pixel of interest and a color of the reference pixel in a color image captured of a target of the distance image, and the weighting coefficient is obtained. Is applied to the distance data of the reference pixel to obtain interpolation data.

As a result, a novel distance data interpolation device capable of accurately interpolating distance image data while maintaining high consistency with a color image can be obtained.

In the fourteenth aspect of the present invention, the thirteenth aspect
In the aspect, the distance interpolating unit may calculate a weighting coefficient by reflecting information on a color difference between a pixel corresponding to each of the target pixel and the reference pixel in a color image captured of the target of the distance image. Then, the obtained weighting coefficient is applied to the distance data of the reference pixel to obtain interpolation data.

As a result, a novel distance data interpolation device capable of accurately interpolating distance image data while maintaining high consistency with a color image can be obtained.

According to a fifteenth aspect of the present invention, in the thirteenth aspect, the distance interpolation means corresponds to each of the target pixel and the reference pixel in a color image taken of the target of the distance image. A weighting coefficient is obtained by reflecting information on a color difference between pixels and a difference in pixel coordinate values between the pixel of interest and the reference pixel, and the weighting coefficient is applied to distance data of the reference pixel to perform interpolation. It was configured to obtain data.

As a result, a novel distance data interpolation device capable of accurately interpolating distance image data while maintaining high consistency with a color image can be obtained.

In a sixteenth aspect of the present invention, interpolation of distance data is performed in consideration of cluster boundaries of a color image, and in a seventeenth aspect, interpolation of distance data is performed in consideration of edges obtained by edge enhancement processing. Thus, the accuracy of data interpolation in the distance data interpolation device is improved.

In the eighteenth aspect of the present invention, the thirteenth aspect
To the distance data interpolating apparatus according to any one of the seventeenth to seventeenth aspects, and the distance information interpolating process performed by the distance data interpolating apparatus. And a layering device for performing a layering process on pixels.

As a result, a novel color image layering device capable of extracting an accurate image and the like using the distance information can be obtained.

Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings.

(Embodiment 1) FIG. 1 is a block diagram showing the overall configuration of a system for hierarchizing color images using distance images.

A range finder for imaging the subject 23 has a light source A (reference numeral 1), a light source B (reference numeral 2), a half mirror 3, a rotating mirror 4, a lens 5, Half mirrors 6, 7, CCDs (solid-state imaging devices) 8, 9, 10, light source control unit 10, rotation control unit 11, light source A image processing unit 12a and light source B image processing unit 12b, color camera signal It has a processing unit 13, a control unit 14, and a distance calculation unit 15.

The wavelengths of the light sources A and B are generally set in the infrared region, and the light wavelengths of the respective light sources are different. Lights output from the light sources A and B are combined by the half mirror 3. The three CCDs (solid-state imaging devices) 8, 9, and 10 receive the light split by the half mirrors 6 and 7, respectively, and thus have the same optical viewpoint.

The range finder includes a distance data loss area determination unit 16 and a distance data interpolation unit 16 as a special configuration for performing interpolation processing and layering of color images on a distance image obtained by imaging. And a hierarchical processing unit 18.

The image synthesizing device 21 outputs a part of the color image (for example, a foreground object) extracted by the hierarchical processing unit 18 from an image captured by the video camera 19 or from the image memory 20. Combines with an image to form a new image. In the present system, the synthesized image is stored in the image memory 22.

FIG. 2 shows a characteristic operation procedure in this system.

That is, distance image data and color image data are acquired by the range finder (step S).
T1). Subsequently, it is determined whether or not there is a missing area of the distance data in the distance image. If there is a missing area of the data, the distance data is obtained by interpolation (step ST).
2). In this case, the distance data is interpolated using the color information of the color image captured at the same time.

Next, a hierarchical process (image extraction and separation process) of a color image is performed using the distance information (depth information) of the distance image in which the loss has been filled through the interpolation process (step ST3). Next, the image extracted and separated by the hierarchization is combined with another image (step ST4) and stored in the image memory (step ST5).

The outline of the characteristic operation of the system shown in FIG. 1 has been described above. Hereinafter, each process will be specifically described with reference to FIGS.

FIG. 3 shows the distance data loss area determination section 16 and the distance data interpolation section 17 extracted from the configuration of FIG.

The missing area determining unit 16 for the distance data determines an area where the distance data is missing in the distance image (S1) based on the distance image (S1) and the evaluation value image (S2).
Extract. As the evaluation value image, for example, a light intensity image used when calculating the distance of the range finder can be used.

The area to be extracted when it is determined that the distance data is missing is, for example, an area in which the distance image data is extremely large, or an extremely small area, or
This is an area where the light intensity is equal to or less than a certain value or equal to or more than a certain value in the evaluation value image (light intensity image).

The distance data interpolation unit 17 receives as input a signal (S4) indicating an area extracted as a missing area of distance data, a distance image (S1), and a color image (S3) captured from the same viewpoint as the distance image. The distance data in the missing area of the distance data is calculated by a predetermined interpolation calculation.

The distance data is obtained by calculating a weighting coefficient in accordance with the color difference between the corresponding pixels and the pixel coordinate value, and applying the weighting coefficient to the distance data of the reference pixel (weighted interpolation). ).

FIG. 4 is a diagram for explaining a method of calculating distance data in a missing area of distance data by weighted interpolation. In FIG. 4, an area A is a missing area of the distance data, and an area B is a surrounding area having the distance data.

Pixels (x,
The distance data in y) belongs to the surrounding area B and refers to the distance data in the reference pixel (s, t) in contact with the area, and is determined by weighting interpolation of the following (Equation 1).

[0061]

(Equation 1)

In (Equation 1), the integration path C is all surrounding pixels that are in contact with the missing area of the distance data (shown by thick black lines in FIG. 4). The numerator of (Equation 1) is a normalization term for setting the sum of the weights to 1.

It should be noted in (Equation 1) that, in addition to the distance information rx, y between the target pixel and the reference pixel in the image space, the distance information rRGB between the target pixel and the reference pixel in the color space is a parameter. That is, the weighting coefficient is obtained. The obtained weighting coefficient is multiplied by the distance data of the reference pixel, and the results are added up (line integration) for all pixels around the distance data deficient area to obtain distance data by interpolation.

As described above, in the weighted interpolation of (Equation 1), the distance data at the target pixel is obtained by weighting according to the difference between the coordinate value between the target pixel and the reference pixel and the difference between the RGB pixel values. As a result, the distance data at the pixel of interest strongly influences the distance data at the reference pixel whose distance on the pixel plane is short (the difference in pixel coordinate values is small) and whose color is close (the difference in RGB pixel values is small). Interpolated to receive. As described above, according to the present embodiment, the color difference between the pixel of interest and the reference pixel in the missing area of the distance data and the pixel coordinate The distance data can be determined from the distance data of the surrounding pixels by weighted interpolation according to the difference.

As a result, the boundary between the foreground and the background of the distance image including the area where the distance data is newly obtained by interpolation matches the boundary between the foreground and the background in the color image with high accuracy. Therefore, when extracting a part of the color image (for example, foreground) based on the distance information of the distance image, there is a disadvantage that the background color is mixed around the extracted color image (foreground) and the boundary becomes unclear. Does not occur. That is, it is possible to improve the detection accuracy of the object outline when the color image is hierarchized using the distance image.

The formula for calculating the distance in the missing area of the distance data in the present embodiment is not limited to the formula shown in (Equation 1) but may be the formula shown in (Equation 2) below.

[0067]

(Equation 2)

In (Equation 2), the stabilizing term λ in (Equation 1) is omitted.

The formula for interpolation of distance data is not limited to (Equation 1) and (Equation 2).
The one shown in (Equation 4) may be used. In (Equation 3) and (Equation 4), the square of the distance in the image space and the square of the distance in the color space in (Equation 1) and (Equation 2) are the distance and the color in the image space, respectively. It is replaced by the distance in space. .

[0070]

(Equation 3)

[0071]

(Equation 4)

Further, the distance in the image space and the distance in the color space used in (Equation 3) and (Equation 4) are used as interpolation formulas for distance data in the present embodiment. (Equation 5), which is replaced by the sum of the absolute value of the difference between the coordinate values between the target pixel and the reference pixel and the sum of the absolute values of the color components between the target pixel and the reference pixel in the color space. The arithmetic expression shown in Expression 6) can be adopted.

[0073]

(Equation 5)

[0074]

(Equation 6)

Further, in the interpolation calculation of the distance data in the present embodiment, not all the pixels on the integration path of the line integration are set as reference pixels, but the reference pixels are thinned out every several pixels on the integration path to reduce the number of reference pixels. Decreasing it is also effective. In this case, there is an advantage that the calculation time can be reduced.

Further, the interpolation calculation of the distance data does not need to be limited to the line integration using the neighboring pixels in contact with the missing area of the distance data as an integration path. For example, a similar result can be obtained by performing an area referencing a pixel having distance data within a certain distance from a target pixel in an image. Further, a similar result can be obtained by line integration using an adjacent pixel that is within a certain distance from the target pixel in the image and is in contact with the missing area of the distance data as an integration path. Also, instead of using all the pixels in the integration region for the area as reference pixels, the reference pixels are thinned out so as to be every few pixels in the integration region,
Calculation time can also be reduced.

FIG. 5 summarizes the operation for obtaining the distance data described above. That is, a missing area of the distance data is extracted (step ST1). Next, a weight coefficient corresponding to the difference in color in the color image and the difference in the pixel coordinate value is added to the distance data of the reference pixel in contact with the missing area of the distance data. , And summing up all the reference pixels (step ST
2).

FIG. 6 is a block diagram showing an example of the internal configuration of the hierarchical processing unit 18 in the system shown in FIG.

As shown in the figure, the depth image (S5) and the color image of the same object are input in parallel to the hierarchical processing unit 18. The hierarchical processing unit 18 includes a comparing unit 51 that compares a distance image value of the input distance image (S5) with a predetermined threshold value and outputs a result of the determination, and an extraction / separation unit 52. .

If the result of the determination by the comparing means 51 is that the value of the distance image is smaller than the threshold value, the extracting / separating means 52 outputs the pixel value of the color image (S3) as the foreground image (S6). As the background image, a fixed pixel value (for example, black or blue) is output for convenience. Conversely, when the value of the distance image is equal to or greater than a certain threshold, the pixel value of the color image (S3) is output as a background image, and the foreground image (S6) is output with a certain pixel value for convenience. I do. By such processing, the foreground and the background can be separated, and only the foreground can be extracted.

As described above, since the distance image is interpolated, there is no missing area in the distance data, and the interpolation takes into account the color information of the color image. , With the boundary between the foreground and background in the color image with high precision. Therefore, hierarchization using the distance image can be accurately performed for all the pixels of the color image.

In the case where the hierarchization is performed using a distance image that is not subjected to interpolation processing, pixels for which distance measurement could not be performed in the distance image may be filled with a constant value, for example, by filling the maximum value of a handleable distance range. By filling the distance value of
Hierarchy using the distance image can be performed for all the pixels of the color image. As described above, according to the present embodiment, it is possible to extract a foreground of a person or the like from a photographed image without using a background as a single color by using a distance image, and perform layering of a color image. it can. In the present embodiment, two layers have been described for simplicity, but it is clear that the number of layers can be increased to three or more by the same threshold processing.

(Embodiment 2) FIG. 7 is a block diagram showing a configuration of a range finder device used for measuring a distance image in the present embodiment.

As shown, the range finder device includes a light source 30, a slit 31, a plurality of distance image capturing cameras 32a to 32c, and a color image capturing camera 3.
4, the slit driving unit 35, and the imaging timing control unit 3
6, an image processing unit 37, a distance calculating unit 38, and a distance integrating unit 39.

In the above-described embodiment, the optical viewpoints of the camera that captures the distance image and the camera that captures the color image match, but in the present embodiment, they do not match. Therefore, in the present embodiment, viewpoint conversion is performed to match the viewpoints of the distance image and the color image.

FIG. 8 is a block diagram showing a configuration of a data interpolation device according to the present embodiment. The configuration in FIG. 8 is substantially the same as the configuration in FIG. 3, but differs in the present embodiment in that a viewpoint conversion unit 50 is provided.

The viewpoint conversion unit 50 receives as input the distance image (S1) and the image (S4) indicating the result of determining the missing area of the distance data, and performs viewpoint conversion on each image to match the viewpoint of the color image. The viewpoint conversion is performed by the following equation (Equation 7).

In equation (7), the viewpoint conversion parameter A
L is determined at the time of correction. That is, an object whose three-dimensional coordinate values (X, Y, Z) are known is imaged by a color image camera, and the coordinate values (x, y) in the color image and the known three-dimensional coordinate values (X, Y, Z) are taken. ) Are measured, A to L are determined by least squares fitting, and correction is performed.

[0089]

(Equation 7)

As described above, according to the present embodiment, even when the viewpoints of the distance image and the color image are different, by matching the viewpoints of both, the information of the color image can be used.
The distance data at the target pixel can be determined from the distance data at surrounding pixels. As a result, the boundary between the foreground and the background in the distance image including the area subjected to the interpolation processing matches the boundary between the foreground and the background in the color image with high accuracy, and the color image hierarchy using the distance image is used. It is possible to improve the detection accuracy of the foreground object contour at the time of image conversion.

In this embodiment, the viewpoint transformation is described as the coordinate transformation. However, the primary transformation and the secondary transformation shown in the following (Equation 8) and (Equation 9) are replaced with the viewpoint transformation of the viewpoint transformation described above. Post-processing or pre-processing for distance calculation can also be used.

[0092]

(Equation 8)

[0093]

(Equation 9)

Thus, it is possible to correct the difference in the characteristics of the optical system.

(Embodiment 3) FIG. 9 is a block diagram showing an overall configuration of a system for layering a color image using a range image according to a modification of the embodiment 1. In FIG.

In FIG. 9, components that perform the same operations as in the first embodiment are denoted by the same reference numerals as those in FIG. 1, and description thereof is omitted.

In this embodiment, similarly to the above-described embodiments, the missing distance data is interpolated in consideration of the color information of the object captured by the color camera, and the boundary between the foreground and the background determined from the distance information. Is matched with the foreground / background boundary determined from the color information, and the boundary of the subject is matched between the color image and the distance image.

However, in the present embodiment, the pixel values of the image (pixel values in the three-dimensional feature space of R, G, and B in the case of a color image) are classified into a plurality of clusters, and the boundary of the distance image is Interpolate the distance data to match the boundaries of the cluster.

That is, it is assumed that the distances from the imaging camera will be different for regions of different colors, and that the distances from the imaging camera will be the same for portions within the region of the same color. In this way, the information on the boundaries of the clusters is used for the interpolation processing of the distance data.

The operation of the clustering unit 1701 and the distance data interpolation unit 1702 will be described below.

A clustering unit 1701 performs clustering (region division) of a color image. Clustering in a broad sense is a process of dividing a target image into partial images (clusters) having uniform features, thereby making it easier to grasp the overall structure of the image. In the present embodiment, as described above, clustering of pixel values in the RGB space is performed. The cluster in the present embodiment means a closed area divided based on color similarity in a color image.

The clustering method is based on a pixel value (for a color image, a pixel value in a three-dimensional space of R, G, and B).
May be classified into a plurality of clusters, and is not particularly limited.

In this embodiment, the k-mean method (k-mean algorithm) is used as a clustering technique. Hereinafter, the k-mean method will be described.

The k-mean method is an algorithm for classifying the entire data into a predetermined number of clusters. k-mean
The method performs clustering by the following four procedures. (1) Determination of the number of clusters and termination conditions. (2) Arrangement of initial clusters. (3) Arrangement of data. (4) Judgment of termination. (If not terminated, return to (3).) In the present embodiment, the number of clusters is given as a fixed value (for example, 10). Further, the termination condition is that the number of data arranged in each cluster converges or the number of repeated calculation of cluster update reaches a certain number (for example, 10).

For the arrangement of the initial clusters, the i-th cluster centroid ci is obtained by (Equation 10) from the average vector m and the standard deviation vector σ of all data.

[0106]

(Equation 10)

Here, n is the number of clusters, i = 1,
2,. . . , N.

The result of clustering by the clustering unit 1701 is output as a cluster image (S6), that is, a multi-valued image of the number of clusters.

FIG. 10 is a diagram showing the positional relationship between a cluster and a missing area of distance data. In FIG. 10, the number of clusters is set to “7” for simplicity. By the clustering process described above, the pixels in the image are classified into any one of cluster 1 (reference numeral 901) to cluster 7 (reference numeral 907).

In FIG. 10, areas 908a, 908b, and 908c indicated by oblique lines are areas where distance data is missing. As shown, the missing region 908a of the distance data belongs to cluster 2 (reference numeral 902), the missing region 908b of the distance data belongs to cluster 5 (reference numeral 905), and the missing region 908c of the distance data belongs to cluster 7 (reference numeral 905). 907).

Next, a method of determining (interpolating) distance values in the missing areas 908a to 908c will be described. In FIG. 9, a distance data interpolation unit 1702 is a distance calculation unit 1
5 and the evaluation value image (S2) output from the distance data loss area determination unit 16
And a cluster image (S6) output from the clustering unit 1701 to determine the distance value in the missing area of the distance data.

The distance value in the missing area of the distance data is:
The energy E (Equation 11) for the distance value distribution d (x, y) in the missing area is defined, and the energy E is determined as the distance value distribution that minimizes this.

[0113]

[Equation 11]

As can be seen from this equation, the energy E in the missing area of the distance data is represented by the (x, y) coordinates of the pixel, the distance d, the partial differentiation of the distance d by the variable x, and the distance d
Is represented by a function that uses a partial derivative of the variable y as a parameter. Specifically, each of the partial differential values of the distance d is squared and added, and the product is multiplied by a weighting factor w (x, y) to obtain an area for the area A, and this is multiplied by １ ／. Is also required.

Here, the integration area A is an area including a missing area of the distance data and an area of one pixel adjacent to the missing area of the data (belonging to the area from which the distance data has been acquired). . The reason why the area including the distance data is included is that the distance data of the area where the distance data is missing is interpolated with reference to the distance data.

The weighting function w (x, y) is set so as to take a large positive value for each pixel in one cluster, and the pixel at the cluster boundary (ie, the pixel of a different cluster) Is set to take a small value of zero or a positive value (the weight function w does not become negative). That is, a pixel at a boundary with another cluster and a pixel within a cluster are clearly distinguished from each other, and a large difference is caused in the weight coefficient.

As a result, the distribution of the distance values in the defective region minimizing (Equation 11) greatly changes at the cluster boundary, and changes smoothly within one cluster. That is, the determination of the distance data of the missing region of the data belonging to one cluster is performed with reference to the distance values of the pixels surrounding the missing region and belonging to the cluster. Therefore, within that cluster, the distance value changes gradually little by little with reference to the distance value of surrounding pixels.

On the other hand, the distance value of the pixel at the boundary in contact with a different cluster is significantly different from the distance value of the other pixels that showed a smooth distribution in the cluster. Will do.

As a result, the boundary is accurately formed in the distance data so as to match the boundary of the color, and the foreground / background contours match with high accuracy between the color image and the distance image. Become like

The distribution of distance values in the missing area that minimizes (Equation 11) can be obtained as follows. That is,
The condition that the energy value E takes an extreme value is the variation δE =
Assuming 0, then:

[0121]

(Equation 12)

Then, the distribution of distance values in the missing area that satisfies this condition is determined by a numerical calculation such as an iterative method, a finite element method, or a boundary element method.

(Embodiment 4) FIG. 11 shows Embodiment 3 of the present invention.
FIG. 2 is a block diagram showing an overall configuration of a system for layering a color image using a distance image according to the modification of FIG. In FIG. 11, the same operations as those in the third embodiment are denoted by the same reference numerals as those in FIG. 9, and the description is omitted.

The feature of the present embodiment is that the edge strength is detected in a color image, and the weight coefficient w (x, y) in the above (Equation 11) is corrected by the edge information. In the third embodiment, the distance data is interpolated by focusing on the boundaries of the clusters. However, since the cluster boundaries may not exactly coincide with the boundaries between the foreground and the background,
In order to improve the accuracy, the distance data is interpolated using the edge information obtained by the edge enhancement processing.

The operation of the edge strength detecting section 1703 and the distance data interpolating section 1704 will be described below.

The edge strength detector 1703 calculates the edge strength for each pixel of the color image. The edge strength is, for example, the norm of the luminance gradient of the color image (the square root of the sum of squares of the horizontal luminance gradient and the vertical luminance gradient), or R
It is calculated as the sum of the norms of the gradients of the respective components of GB. The edge strength preferably takes into account both the vertical and horizontal components as described above, but may be calculated from only the absolute value of the horizontal gradient.

The distance data interpolation unit 1704 includes a distance image (S1) from the distance calculation unit 15, an evaluation value image (S2) from the distance data loss area determination unit 16, a cluster image (S6) from the clustering unit 1701, and edge intensity. The distance value in the missing area of the distance data is determined from the edge strength (S7) by the detection unit.

The distance value in the missing area of the distance data is
The energy E (Equation 13) for the distance value distribution d (x, y) in the missing area is defined, and the energy E is determined as the distance value distribution that minimizes this.

[0129]

(Equation 13)

Here, the integration area A is a missing area of the distance data and surrounding pixels in contact with the missing area of the distance data. The weight function W (x, y) is obtained by correcting the weight function w (x, y) in (Equation 11) by the edge strength.

The correction of the weight based on the edge strength is performed so that the weight becomes smaller where the edge strength is large.
As a result, the distribution of distance values in the missing region that minimizes (Equation 13) changes greatly at the cluster boundary and changes smoothly within the cluster. In addition, the distance value around the defective area is smoothly connected to the distance value of the pixel of the same cluster according to the result of the clustering, and is discontinuous with the distance value of the pixel of a different cluster. Further, the distance value is determined to be largely changed in a pixel having a large edge strength in the defective area.

The condition under which the energy value E of (Expression 13) takes an extreme value is as follows from the variation δE = 0.

[0133]

[Equation 14]

The distribution of the distance values in the missing area satisfying (Equation 14) is determined by a numerical calculation such as an iterative method, a finite element method, or a boundary element method.

[0135]

As described above, according to the present invention,
The boundary between the foreground and the background in the distance image is made to coincide with the boundary between the foreground and the background in the color image, and the detection accuracy of the foreground object outline when layering the color image using the distance image can be improved. Also, it is possible to extract a foreground of a person or the like from a real image and perform layering of a color image without making the background a single color as in a chroma key, and the practical effect is large.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an overall configuration of a system for layering a color image using a distance image according to Embodiment 1 of the present invention;

FIG. 2 is a flowchart showing main operations of the system according to the first embodiment;

FIG. 3 is a block diagram illustrating a configuration of a distance data interpolation device according to the first embodiment.

FIG. 4 is a diagram illustrating an interpolation calculation of distance data according to the first embodiment;

FIG. 5 is a flowchart showing a procedure of distance data interpolation calculation processing in the first embodiment.

FIG. 6 is a block diagram showing a configuration of an image hierarchical device according to the first embodiment;

FIG. 7 is a block diagram showing a configuration of a range finder according to a second embodiment of the present invention.

FIG. 8 is a block diagram illustrating a configuration of a distance data interpolation device according to a second embodiment.

FIG. 9 is a block diagram showing a configuration of a range finder according to a third embodiment of the present invention.

FIG. 10 is a diagram for explaining a clustering and a missing area of distance data according to the third embodiment;

FIG. 11 is a block diagram showing a configuration of a range finder according to a fourth embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Light source A 2 Light source B 3 Half mirror 4 Rotating mirror 5 Lens 6,7 Half mirror 8,9,10 Solid-state image sensor 11 Rotation control part 12a, 12b Image processing part 13 Color camera signal processing part 14 Control part 15 Distance calculation part Reference Signs List 16 Missing area determination unit for distance data 17 Distance data interpolation unit 18 Hierarchical processing unit 19 Color image capturing camera 20 Image memory 21 Image synthesis device 22 Image memory

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Atsushi Morimura, Inventor 1006 Kazuma Kadoma, Osaka Prefecture, Japan Matsushita Electric Industrial Co., Ltd. F-term (reference) 2F065 AA22 AA45 AA61 BB05 CC16 DD00 EE05 FF04 FF09 HH12 JJ03 JJ16 JJ26 LL12 LL13 LL37 QQ14 QQ24 QQ27 QQ32 5B057 BA15 CA01 CB01 CE08 CE09 DA07 DB03 DB06 DC05 GA06 GA02

Claims (18)

[Claims] 1. A method of interpolating distance data of a pixel belonging to an area where distance measurement has not been performed in a distance image with reference to distance data of a pixel whose distance is measured near the pixel. A color image is also obtained for the shooting target of the distance image, and a color of a pixel in the color image corresponding to a target pixel to be interpolated and a color of a pixel in the color image corresponding to a reference pixel are obtained. A distance coefficient is calculated using the calculated weighting coefficient and the distance data of the reference pixel to obtain distance data. 2. The weighting coefficient is calculated by a calculation formula including, as a parameter, difference information between a color of a pixel in the color image corresponding to the target pixel and a color of a pixel in the color image corresponding to the reference pixel. 2. The distance data interpolation method according to claim 1, wherein: 3. The distance according to claim 2, wherein the calculation formula for calculating the weighting coefficient includes, as a parameter, difference information between the coordinate value of the pixel of interest and the coordinate value of the reference pixel. Data interpolation method. 4. The reference pixel according to claim 1, wherein a plurality of pixels whose distance has been measured and which are in contact with a boundary of an area where the distance measurement has not been performed are set as the reference pixels. The interpolation method of the distance data described in. 5. It is within a certain distance from the pixel of interest.
4. The distance data interpolation method according to claim 1, wherein a plurality of pixels for which distance measurement has been performed are used as the reference pixels. 6. A plurality of pixels whose distance has been measured and which are in contact with a boundary line of an area where the distance measurement has not been performed and which are within a certain distance from the target pixel are set as the reference pixels. The distance data interpolation method according to any one of claims 1 to 3. 7. The color image is divided into a plurality of clusters, cluster boundaries are detected, and information of the cluster boundaries is used to smoothly change the value of distance data where pixels in the same cluster are distributed. 2. The distance data according to claim 1, wherein a distribution of the distance data in an area where the distance measurement is not performed is determined so that a value of the distance data changes discontinuously at a cluster boundary. Interpolation method. 8. The color image is divided into a plurality of clusters, cluster boundaries are detected, edge strengths are extracted from the color image, and distance data in an area where distance measurement is not performed in the distance image. 2. The distance data interpolation method according to claim 1, wherein the distribution is determined so as to minimize a calculation formula including the cluster boundary and the edge strength as parameters. 9. The distance image and the color image,
9. The distance data interpolation method according to claim 1, wherein the images are optically captured from the same viewpoint. 10. The distance image is subjected to viewpoint conversion processing on a distance image obtained by imaging from a viewpoint different from a color image, and is converted into an image equivalent to that obtained by imaging from the same viewpoint as the viewpoint of the color image. The distance data interpolation method according to any one of claims 1 to 8, wherein the distance image is a captured distance image. 11. A distance image and a color image are obtained for the same object, the color image is hierarchized using distance information of the distance image, and a part in the color image can be processed independently by distinguishing it from other parts. And the method
Before the layering of the color image, it is determined whether there is an area in which the distance measurement has not been performed in the distance image, and if there is the area, the pixels in the area are determined. 8. A first step of interpolating distance data by the method according to any one of items 7 to eliminate a missing part of the distance data from the distance image, and distance information of each pixel of the distance image having passed the first step. A second step of performing a hierarchization process on all pixels of the color image based on the above method. 12. An image synthesizing method, wherein a part of a color image hierarchized by the color image hierarchization method according to claim 11 is extracted and synthesized with another image. 13. A distance data missing area extracting means for extracting an area for which distance measurement has not been performed in a distance image obtained by a range finder, and distance data for a pixel belonging to the area for which distance measurement has not been performed. A distance interpolation means for interpolating with reference to the distance data of the pixel whose distance measurement is located in the vicinity of the pixel,
The distance interpolation means, when interpolating the distance data of the target pixel based on the distance data of the reference pixel, uses the color of the target pixel and the color of the reference pixel in the color image captured for the target of the distance image. A distance data interpolating apparatus for obtaining interpolation data by obtaining a weighting coefficient by applying the weighting coefficient to the distance data of the reference pixel. 14. A weighting coefficient reflecting information on a color difference between a pixel corresponding to each of the pixel of interest and a reference pixel in a color image captured of a target of the distance image, 14. The distance data interpolation method according to claim 13, wherein the interpolation data is obtained by applying the weighting coefficient to the distance data of the reference pixel. 15. The distance interpolation means includes: information on a color difference between a pixel corresponding to each of the target pixel and the reference pixel in a color image captured for the target of the distance image; 14. The distance data interpolation according to claim 13, wherein a weighting coefficient is obtained by reflecting a difference in pixel coordinate value between pixels, and the weighting coefficient is applied to distance data of the reference pixel to obtain interpolation data. apparatus. 16. The distance interpolating means uses cluster boundary information obtained as a result of clustering a color image obtained for an object to be photographed of the distance image, and calculates a distance at a location where pixels in the same cluster are distributed. Determining a distribution of distance data in an area where distance measurement has not been performed in the distance image, such that a value of the data changes smoothly and a value of the distance data changes discontinuously at a cluster boundary. 14. The distance data interpolation device according to claim 13. 17. The distance interpolating unit uses information on a cluster boundary obtained as a result of clustering a color image obtained for a shooting target of the distance image and edge strength extracted from the color image, Determining a distribution of distance data in an area where distance measurement is not performed in the distance image so as to minimize a calculation formula including the information on the cluster boundary and the edge strength as parameters. Item 13. The distance data interpolation device according to Item 13. 18. A distance data interpolating device according to claim 13, and said distance image interpolating process performed by said distance data interpolating device. A layering device that performs a layering process on all pixels of a color image captured of the target.