JPH10200902A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a high definition image that approximates to a three-plate system and also to perform input image processing of optional color filter arrangement by selecting a template that resembles a neighborhood image most and creating a proximity image. SOLUTION: A proximity reading part 109 reads an image signal that corresponding to a prescribed rectangular area with each pixel position on a CCD as a center from an input image buffer 108 and transfers it to a least square approximated part 112, and the part 112 processes it. A band and color information recovering part 114 obtains a proximity correction approximated image that corrects the absence of a color component of a proximity image. When these processing about entire pixel positions is finished, plural correction pixel values about each pixel are acquired, and the correction pixel values are undergone weighted balancing through processing in a three-plate image adding part 115 and stored in an output image buffer 116. Thereby, a three color image that corrects the absence of color components and the degradation of an optical system of an electronic camera 100 is acquired.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for restoring an original image signal from an image signal in which at least one piece of information relating to spectral characteristics is missing in accordance with each pixel position.

[0002]

2. Description of the Related Art In an image pickup apparatus such as an electronic still camera using an electronic image pickup system, a single-plate type using an image pickup device in which color filters are arranged in a mosaic-like light-receiving surface, or G (green) for one sheet
The two-panel imaging system that performs imaging with two CCDs provided with only mosaic filters of R (red) and B (blue) and another mosaic filter is a three-panel imaging system using a spectral system and three imaging elements. It has the advantage that it can be constructed at a lower cost than an imaging system.
However, in these methods, at least one or more pieces of color information are lost at each pixel position of an image, and therefore, in the image processing apparatus, color information that is missing for one pixel is used as color information that is not missing for another pixel. It is necessary to perform interpolation processing for estimating.

[0003] However, in the single-plate system or the two-plate system, moiré peculiar to the arrangement of color filters occurs, and the manner in which the moiré occurs differs for each color filter. Therefore, in the simple linear interpolation processing, artifacts such as unevenness of an edge portion, a change in angle, and coloring are generated. Conventionally, various interpolation techniques have been proposed to reduce these artifacts. Among them, the most effective interpolation techniques include: 1. A method of estimating a local geometric structure of an image and performing interpolation according to the structure, in order to reduce unevenness artifacts at the edge portion. To reduce the coloring of edges, use an image sensor with a color filter arrangement that can generate a broadband luminance signal by a simple interpolation method, and assume that the color signal changes in the same way as the luminance signal. And a method of performing the interpolation of

In the following, 1 is described in JP-A-61-501.
Regarding JP-A No. 423 and JP-A No. 423, each of the methods will be described taking the method disclosed in Japanese Patent Application Laid-Open No. 4-502006 as an example.

JP-A-61-501423 discloses R,
A method of interpolating a G signal corresponding to a luminance component in an image sensor having a G and B color filter array is disclosed. In this method, in the vicinity of 3 × 3 pixels where the G signal is missing,
Judge which of the geometric features of the edges, streaks, and corners the neighborhood best fits, and select an interpolation method specialized for each geometric feature based on this judgment result. ing.

[0006] Japanese Patent Application Laid-Open No. 4-502096 discloses that
It discloses an interpolation method for a low sampling density color component in an image signal in which a luminance component is obtained at a high sampling density and other color components are obtained at a relatively low sampling density. In this method, the color components of the low sampling density are calculated using the luminance components of the high sampling density.
For this reason, it is assumed that a sample value of the high sampling density signal S1 always exists at the sampling position of the low sampling density signal S2. Then, the difference between the sample values of the signal S1 at the sampling position of the signal S2 is compared with a predetermined threshold value. If the difference is smaller than the threshold value, simple linear interpolation using only the sample values of the signal S2 is performed.

If it exceeds the threshold value, it is determined that there is an edge between the sampling positions of the signal S2, and the signals S1 and S2
Are assumed to have a linear relationship of S2 = aS1 + b, and the coefficients a and b are calculated as S1 at the sampling position of the signal S2.
And the value of S2.

[0008]

In any of the above-mentioned prior arts, since there is information that is missing even in a luminance component given at a relatively high sampling density, the luminance component is deteriorated as compared with a luminance component obtained by a three-plate system. You can only get the information you have. Also, other color components calculated based on the luminance component can obtain only information that is deteriorated as compared with the color components obtained by the three-plate system.

Further, none of the above-mentioned prior arts describes a method for processing an input image having an arbitrary color filter arrangement. In the interpolation of the luminance component, it is necessary to prepare a geometric feature prepared in advance and an interpolation method corresponding thereto, and it is necessary to change the design when the color filter arrangement changes.

Further, none of the above-mentioned prior arts describes a method for coping with a malfunction caused by the influence of noise. When processing luminance and other color components,
Since interpolation is performed in one process, it is easily affected by pulse noise. In particular, in the processing of the luminance component, since the geometric features are classified in a 3 × 3 local area, erroneous classification due to noise is likely to occur.

None of the above-mentioned prior arts describes recovery of information whose band is limited by the optical system of the imaging apparatus. That is, the luminance component and the color component of another color are processed within the band-limited range, and the information lost due to the band limitation cannot be recovered.

The present invention has been made in view of the above-described problems, and a first object of the present invention is to obtain a high-definition image similar to a three-plate system and to obtain an arbitrary color image. An object of the present invention is to provide an image processing device that can process an input image having a filter arrangement.

It is a second object of the present invention to provide an image processing apparatus capable of reducing deterioration of an image signal due to noise in addition to the first object.

A third object of the present invention is to provide an image processing apparatus capable of recovering information lost due to band limitation in addition to the first and second objects.

[0015]

In order to achieve the above first object, a first aspect of the present invention is to provide an image processing apparatus, comprising: an image signal which lacks information on at least one or more spectral characteristics according to a position of a pixel; In an image processing apparatus for restoring an image signal, a neighborhood image extraction unit for extracting a neighborhood image of a predetermined size including a designated pixel from an image signal in which the information is missing, and a neighborhood image extracted by the neighborhood image extraction unit A template holding unit for holding a template group based on a plurality of geometric patterns having the same size as the neighboring image; information on a plurality of spectral characteristics included in the neighboring image extracted by the neighboring image extracting unit; Performing a matching between the plurality of template groups held by, and selecting an optimal template most similar to the neighboring image, A neighborhood approximation image generating means for producing a neighborhood approximation image approximating a neighborhood image based on the optimal template, and restoring missing information in the specified pixel based on the neighborhood approximation image generated by the neighborhood approximation image generation means And a restoring means for performing the restoration.

In a second aspect based on the first aspect, the restoring means includes a plurality of the neighborhood approximate images obtained from the plurality of the neighborhood images including a designated pixel.
Restoration processing is performed by weighted averaging of information on at least one or more spectral characteristics.

In a third aspect based on the first or second aspect, the template holding means includes: a first template group based on a plurality of geometric patterns; and an imaging device for capturing the first template group. A second template group degraded in accordance with the band limiting characteristics of the optical system of the above, wherein the neighborhood approximate image generation means selects an optimal template similar to the neighborhood image from the second template group, A neighborhood approximation image before band limitation is generated based on the first template group corresponding to this.

That is, in the first invention, in recovering an original image signal from an image signal in which at least one piece of information relating to spectral characteristics is missing in accordance with the position of a pixel, first, the image in which the information is missing is used. A neighborhood image of a predetermined size including a designated pixel is extracted from the signal. Then, a template group based on a plurality of geometric patterns having the same size as the extracted neighboring image is held in the template holding unit. Next, matching is performed between a plurality of pieces of spectral characteristic information included in the extracted neighboring image and a plurality of template groups held by the template holding unit, and an optimal template most similar to the nearby image is selected. Then, a neighborhood approximate image that approximates the neighborhood image is generated based on the optimal template. Then, based on the generated neighborhood approximate image, the missing information in the specified pixel is restored.

According to a second aspect, in the first aspect, when performing the restoration process, at least one of the plurality of the neighborhood approximate images obtained from the plurality of the neighborhood images including the designated pixel is provided. Information about one or more spectral characteristics is weighted and averaged.

According to a third aspect of the present invention, in the first or second aspect of the present invention, when generating a neighborhood approximation image, the first template group based on a plurality of geometric patterns and the first template group And a second template group that is degraded according to the band limiting characteristic of the optical system of the imaging apparatus, and an optimal template similar to the neighboring image is selected from the second template group, and the corresponding template is selected. A neighborhood approximation image before band limitation is generated based on the first template group.

[0021]

Embodiments of the present invention will be described below in detail with reference to the drawings. First, a first embodiment of the present invention will be described.

FIG. 1 is a diagram showing the configuration of an image processing system to which the present embodiment is applied, FIG. 2 is a diagram showing the configuration of the least squares approximation unit 112 of FIG. 1 and its peripheral circuits, and FIG. FIG. 3 is an explanatory diagram related to a color component mask.

As shown in FIG. 1, the image processing system of this embodiment includes an electronic camera 100 and an image processing apparatus 101 connected to the electronic camera 100. The electronic camera 100 includes a lens 102 as an optical system, a crystal filter 103 for optically band limiting, and a single-chip RGB random array CCD 104 as an image sensor. The output of the single-chip RGB random array CCD 104 is A / A.
It is connected to the image buffer 107 via the D conversion unit 105. The electronic camera 100 is further provided with a filter arrangement information holding ROM 106 storing an RGB filter arrangement of the single-chip RGB random arrangement CCD 104.

The image processing apparatus 101 includes an input image buffer 108 connected to the output of the image buffer 107 described above.
, The neighborhood reading unit 109, the pre-deterioration template holding ROM 111a and the post-deterioration template holding ROM
1b, a template holding ROM 111, a least square approximation unit 112, a band / color information restoring unit 114, a three-plate image adding unit 115, and an output image buffer 116.

FIG. 2 is a diagram showing the configuration of the least squares approximation unit 112 and its peripheral circuits. As shown in FIG. 2, the least square approximation unit 112 includes a color component mask generation unit 112a, a least square coefficient calculation unit 112b, an approximation error accumulation unit 112c,
An approximate coefficient holding buffer 112d and a template determining unit 112e are provided. The least squares approximation unit 112 further includes a neighborhood image input terminal 1 connected to the neighborhood read unit 109.
12r, a filter information input terminal 112p connected to the filter arrangement information holding ROM 106, a template input terminal 112q connected to the post-deterioration template holding ROM 111b, and an approximate parameter output terminal 112s connected to the band / color information restoring unit 114. And

As shown in FIG. 1, the electronic camera 100
The lines are connected to the image processing apparatus 101, of which the output of the image buffer 107 is
8. Least squares approximation unit 112 via neighborhood read unit 109
It is connected to the.

On the other hand, the filter arrangement information holding ROM 106
Is a color component mask generation unit 112 in the least square approximation unit 112 via a filter information input terminal 112p as shown in FIG.
a. The least square coefficient calculating unit 112b receives the input from the neighborhood reading unit 109 via the nearby image input terminal 112r and the deteriorated template holding ROM 111b via the template input terminal 112q in addition to the color component mask generating unit 112a. Connected
It is connected to output the held information to the approximate coefficient holding buffer 112d and the approximate error accumulating unit 112c.

The approximation coefficient holding buffer 112d and the approximation error accumulating unit 112c are both a template determining unit 112e.
And the template determining unit 112e is connected to output the determined content to the band / color information restoring unit 114 through the approximate parameter output terminal 112s.

As shown in FIG. 1, the band / color information restoring unit 11
4 is connected to receive an input from the pre-deterioration template holding ROM 111a in addition to the least square approximation unit 112,
It is connected to output the restoration result to the output image buffer 116 via the three-plate image adding unit 115. Template holding ROM 111a before deterioration and template holding RO after deterioration
M111b is a part of the template holding ROM 111.

The pre-deterioration template holding ROM 111a of the template holding ROM 111 holds a template #i (i = 0 to N-1) which is a grayscale image of a predetermined size having predetermined N kinds of geometrical features, and is degraded. A template Φi (i = 0 to N−1), which is an image of each template of the pre-deterioration template holding ROM 111a whose band is limited by the optical system of the electronic camera 100, is held in the post-template holding ROM 111b.

The output image buffer 116 holds a three-color image in which color loss and band deterioration have been corrected in each pixel of the input image. In the image processing apparatus 101, a plurality of correction values are set for each pixel during processing. Is calculated. In order to calculate the weights based on the reliability of the correction values, the output image buffer 116 further holds a reliability value indicating the reliability of the pixel value at each pixel position.

The operation of the image processing system having the above configuration will be described below.

First, an object is imaged using an electronic camera 100, and an optical image whose band is limited through a lens 102 and a crystal filter 103 is converted into a single-chip RGB random array CCD.
104 is formed. The output of the single-chip RGB random array two-dimensional CCD 104 is directly transferred to the image buffer 107 via the A / D converter 105, and one color determined by the color filter arrangement in the single-chip RGB random array CCD 104 for each pixel position on the CCD. Images with only components are saved.

When the imaging is completed, the image in the image buffer 107 is transferred to the input image buffer 108. At the same time, the color filter arrangement information is also transferred to the color component mask generation unit 112a from the filter arrangement information holding ROM 106 that holds the color filter arrangement of the image sensor of the electronic camera 100. Further, the pixel values and the reliability values of the color image in the output image buffer 116 are all initialized to zero.

Next, a CCD is read by the neighborhood reading unit 109.
An image signal corresponding to a predetermined rectangular area (for example, 5 × 5 pixels) centered on each pixel position (x, y) above is read out from the input image buffer 108 as a neighboring image h (p, q). , The least squares coefficient calculation unit 112 of the least squares approximation unit 112
b. In the following description, a set of variables (p,
q) is a relative coordinate with respect to the center pixel (x, y) of the neighboring image. For example, if the size of the neighborhood is 5 × 5, [−2, −1, 0, 1, 2] × [−2] , -1,0,1,
2]. In some cases, colors R, G, and B correspond to integer values c = 0, 1, and 2, respectively.

When the neighborhood image is transferred from the neighborhood reading unit 109 to the least square coefficient calculation unit 112b, the color component mask generation unit 112a calculates 1 for each color component of the neighborhood image at the pixel position where the color component exists, and others. Calculate the mask mask (p, q, c) (c = 0,1,2) which takes 0 at the position of. FIG. 3 illustrates a color component mask corresponding to a specific color filter arrangement in a neighboring image.

On the other hand, the least-squares coefficient calculating unit 112b calculates the template Φ of the deteriorated template holding ROM 111b.
The template Φ0 (p, q) is read from i (p, q) (i = 0, 1, 2,..., N−1), and each color component mask generated by the color component mask from the transferred neighboring image h is used. Approximate error e0 with each generated color component c (c = 0, 1, 2)
Approximation coefficients α0 (c) and β0 (c) for minimizing (1), e0 (2) and e0 (3) are calculated for each color component.
The least square coefficient calculation unit 112b calculates the calculated approximate coefficient α
0 (c) and β0 (c) are sent to the approximation coefficient holding buffer 112d together with the neighboring image h to minimize the approximation error e0
(1), e0 (2), e0 (3) and the corresponding template index i = 0 are used as the approximation error accumulator 112.
Send to c.

The approximate error accumulating section 112c calculates a cumulative addition error e0 = e0 (1) + e0 (2) + e0 (3) from the transferred approximate errors e0 (1), e0 (2), and e0 (3). Is transferred to the template determining unit 112e.

The template determining unit 112e determines a template that minimizes the cumulative addition error e. The template determining unit 112e compares the value of the cumulative addition error e obtained for each template, and determines a template that takes the minimum value. select.
The template determining unit 112e includes a template index, a cumulative addition error e corresponding to the template, and an approximation coefficient α corresponding to the template.
A storage unit for storing (c) and β (c) is provided.
Therefore, the index i = 0 of the template Φ0 (p, q)
When the cumulative addition error e0 corresponding to the template is transferred to the template determining unit 112e, these are stored in the storage unit. At this time, the approximation coefficients α0 (c) and β0 (c) are transferred from the approximation coefficient holding buffer 112d and are stored in the storage unit.

Next, the least-square coefficient calculator 112b reads the second template Φ1 (p, q) and repeats the above steps to obtain the index i = 1 of the template Φ1 (p, q) and this template Is obtained. Template determination unit 112e
Compares the cumulative addition error e0 with the cumulative addition error e1,
The value with the smaller error is stored in the storage unit together with the corresponding index. As for the approximation coefficient, the approximation coefficient of the template that generates a small cumulative addition error is stored in the storage unit. Thereafter, the third template Φ2 (p, q), 4
.. Are sequentially read to obtain the cumulative addition errors e2, e3,..., And are compared with the cumulative addition errors stored in the sequential storage unit. The template that minimizes the cumulative addition error is finally determined by leaving the index and the approximation coefficient in the storage unit.

After all the templates have been processed by the least squares coefficient calculator 112b, the template determiner 112e
Is the index min_i giving the minimum value of the error min_e
And the approximation coefficients min_α (c) and min_β (c) and the approximate evaluation value A inversely proportional to the minimum error value min_e are transferred to the band / color information restoration unit 114. Here, for example, the approximate evaluation value A is calculated by A = 1 / (1 + min_e).

The band / color information restoring unit 114 stores the undegraded template Ψ min_i (p, q) corresponding to the transferred index min_i in the undegraded template holding ROM 111.
a_min (α), min_α (c), min_α (c)
Using β (c), the equation f (p, q, c) = min_α (c) Ψmin_i (p, q) + min_β
(c) By calculating (c = 0,1,2), a neighborhood corrected approximate image f (p, q, c) in which the missing color component of the neighborhood image h (p, q) is corrected is obtained. By using the template before deterioration, f (p, q, c)
In, the deterioration of the band due to the optical system of the electronic camera 100 is also corrected. Band / color information restoration unit 114
Transfers the calculated f (p, q, c) together with the approximate evaluation value A to the three-plate image adding unit 115.

During the processing, the output image buffer 116
Each pixel position holds an average value and a reliability value of a plurality of corrected pixel values obtained in the processing up to this point. The three-plate image adding unit 115 includes an output image buffer 116
, The corrected pixel value and the reliability value of each pixel in the region for the neighboring image are read as variables Sf (p, q, c) and Q (p, q), respectively. Then, a new corrected pixel value Sf _new (p, q, c) and a reliability are obtained from these and the neighborhood correction approximate image f (p, q, c) and the approximate evaluation value A transferred from the band / color information restoration unit 114. The value Q _new (p, q) is calculated by the equation Q _new (p, q) = Q (p, q) + A, Sf _new (p, q, c) = (Q (p, q) · Sf (p, q, c) + A ・ f
(p, q, c)) / Q _new (p, q) (c = 0,1,2), and Sf _new (p, q, c) and Q _new (p, q) are output image buffers The data is written in an area 116 corresponding to the neighboring image.

When these processes are completed for all pixel positions, for each pixel of the image in the input image buffer 108, a proximity correction approximate image is calculated for all the neighboring images including the pixel. As a result, a plurality of corrected pixel values are obtained for each pixel. These corrected pixel values are weighted and averaged by the processing in the three-plate image adding unit 115 and accumulated in the output image buffer 116. As a result, a three-color image is finally obtained in which the lack of color components and the deterioration due to the optical system of the electronic camera 100 are corrected.

The components of the embodiment can be variously modified and changed. For example, the single-chip RGB random array CCD 104 may be a CCD having various color filters, or may be a two-chip CCD having one G and another R and B.

The following configuration is also possible to correspond to an electronic camera having various optical systems. On the electronic camera 100 side, a filter arrangement information holding ROM 106
Holds the band-limiting information of the optical system such as the lens 102 and the crystal filter 103 in addition to the color filter arrangement information. On the image processing apparatus 101 side, each of the pre-deterioration templates in the pre-deterioration template holding ROM 111a is stored. A plurality of post-deterioration templates are prepared assuming deterioration due to band limitation of various optical systems. Then, the least-squares approximation unit 112 selects one degraded template based on the optical system information in the filter arrangement information holding ROM 106.

Similarly, it is also possible to adopt a configuration in which information about the subject set by the user of the camera is transferred to the image processing apparatus 101, and the least-square approximation unit 112 selects a template set accordingly. is there. The three-plate image addition unit 115 can also store a plurality of approximate values obtained for each pixel, and change the values by determining an approximate value by majority decision or the like.

The electronic camera 100 has a crystal filter 1
03 and the band is not limited, the template holding ROM 111 stores the pre-deterioration template holding RO.
M111a alone, and the least squares approximation unit 112 is connected to the pre-deterioration template holding ROM 111a. In this case, the operation of each part is the least square approximation unit 11
2 is the same except that Ψi is used instead of Φi as a template. As a result, the band / color information restoring unit 1
In 14, a neighborhood correction approximation image is obtained in which the lack of color components is corrected but no degradation of the band is performed, and the lack of color components of the image in the input image buffer 108 is corrected in the output image buffer 116. A three-color image is obtained.

Hereinafter, a second embodiment of the present invention will be described.
FIG. 4 is a diagram illustrating a configuration of an image processing system to which the second embodiment of the present invention is applied, and FIG. 5 is a flowchart illustrating a processing flow of the least squares approximation unit 112 in FIG.

Referring to FIG. 4, the image processing system includes an electronic camera 200 and an image processing device 201 connected to the electronic camera 200. Electronic camera 200
Has a lens 202 as an optical system, a crystal filter 203 for performing band limiting, and a single-chip Bayer array CCD 204 as an image sensor, as in the first embodiment. The output of the single-chip Bayer array CCD 204 is A / D converted. Part 205
Through the image buffer 207. Information on the color filter array of the single-plate Bayer array CCD 204 in the electronic camera 200 is obtained by calculation in the image processing apparatus 201 using the regularity of the Bayer array as described later.

The image processing apparatus 201 includes an input image buffer 208, a neighborhood read section 209, and a flatness evaluation section 21.
0, a template holding ROM 211 having a pre-deterioration template holding ROM 211a and a post-deterioration template holding ROM 211b, a least square approximation unit 212, a color information restoration unit 213, a band / color information restoration unit 214, and an interpolation coefficient calculation unit 215 , A linear operation unit 216, and an output image buffer 217.

Image buffer 207 in electronic camera 200
Is output from the input image buffer 2 in the image processing apparatus 201.
08, the flatness evaluation unit 210 via the neighborhood read unit 209
It is connected to the.

The least-squares approximation unit 212 includes a flatness evaluation unit 2
10 is connected to receive the output of the post-deterioration template holding ROM 211b in the template holding ROM 211, and is connected to output the processing result to the color information restoring unit 213 and the band / color information restoring unit 214. Band
The color information restoring unit 214 is connected to receive the output of the pre-deterioration template holding ROM 211a in the template holding ROM 211 in addition to the least squares approximating unit 212. The color information restoring unit 213 is connected to receive the output of the post-deterioration template holding ROM 211b in the template holding ROM 211 in addition to the least square approximation unit 212.

In the pre-deterioration template holding ROM 211a of the template holding ROM 211, similarly to the first embodiment, a template #i (i = 0 to N−1) which is a grayscale image of a predetermined size having N kinds of geometric features is stored. )
And the template Φi (i = 0 to N−1), which is an image in which each template of the pre-deterioration template holding ROM 211a is subjected to the band limitation by the optical system of the electronic camera 200, is stored in the post-deterioration template holding ROM 211b. I have.

The color information restoring section 213 and the band / color information restoring section 214 are both connected to the interpolation coefficient calculating section 215. The linear operation unit 216 is connected to receive the outputs of the interpolation coefficient calculation unit 215 and the flatness evaluation unit 210, and is connected to output the restoration result to the output image buffer 217.

The operation of the image processing system having the above configuration will be described below.

In the second embodiment, similarly to the first embodiment, an image is taken by the electronic camera 200, the band is deteriorated by the optical system, and an image having only one color component at each pixel position is stored in the image buffer 207. Is held. When the imaging is completed, the image in the image buffer 207 is transferred to the input image buffer 208. Unlike the first embodiment, the color filter array of the image sensor in the electronic camera 200 uses the regularity of the Bayer array instead of being transferred from the electronic camera 200 to the image processing apparatus 201, and then uses the regularity of the Bayer array in the image processing apparatus 201 later. Is calculated.

Next, the following processing is performed for each pixel position (x, y) of the image in the input image buffer 208.

A predetermined rectangular area (for example, 5 × 5 pixels) at the pixel position (x, y) is read as a nearby image h (p, q) by the nearby reading unit 209.

In the following description, a set of variables (p, q)
Is a relative coordinate with respect to the center pixel (x, y) of this neighborhood image. For example, if the neighborhood size is 5 × 5, [−
2, -1,0,1,2] × [-2, -1,0,1,2]
, And the colors R, G, and B may be associated with integer values c = 0, 1, and 2, respectively.

The read neighboring image is sent to the flatness evaluation section 2
Sent to 10. The flatness evaluation unit 210 calculates the variance of the pixel value for the position where the pixel value exists for each color component, and if the sum of the variances for all the color components is equal to or more than a predetermined threshold value, the vicinity of the least square approximation unit 212 Transfer images.
Further, the flatness evaluation section 210 transfers the neighboring image to the linear operation section 216 irrespective of this evaluation.

When the neighboring image is transferred, the least square coefficient calculating unit 212 performs processing according to the flow shown in FIG. 5, and similarly to the first embodiment, the index min_i of the degraded template that best approximates the neighboring image and Approximation coefficient min
_Α (c) and min_β (c) (c = 0, 1, 2) are calculated.

Each step (S) in the flowchart shown in FIG. 5 performs the following processing.

S1: The coordinates (x, y) of the center pixel of the neighboring image are obtained, and based on this value and the regularity of the Bayer array, the color of the color component c with respect to the position (p, q) in the neighboring image is obtained. An array variable mask (p, q, c) representing a color component mask that takes 1 if a filter exists and 0 if not exists is calculated. FIG.
The function rem (a, b) returns the remainder of dividing a by b, and the function ab
s (a) gives the absolute value of a. Also, the variable N is initialized to the total number of templates, and the values of the array variables min_α (c) and min_β (c) for the variables min_i and c = 0, 1, 2 are initialized to 0. The array variable h (p, q) is initialized with the pixel values of the neighboring image. The variable i representing the index of the template element is 0
Is initialized to The variable min_e representing the minimum fitting error of the template is initialized to the maximum number that can be handled by the image processing device 201.

S2: The i-th template Φi (p, q) is read from the degraded template holding ROM 211b,
Assign to the array variable Φ (p, q). A variable c representing a color and a variable e representing an error are initialized to zero.

S3: Variables h (p, q), Φ (p, q), mask (p, q)
α, β that minimizes an error function err (c, α, β), which is a function of a color variable c and real numbers α and β, defined from (q, c). The error function err (c, α, β) is a measure of the error between the image αΦ + β obtained by processing the template image Φ with the gain α and the bias β at the position where the color component c exists, and the neighboring image.

(Equation 1)

Represents the sum of squares of a (p, q) in the neighboring image. Obtaining α and β that minimizes the error function err (c, α, β) is equivalent to adjusting the gain α and bias β of the template Φ so as to best approximate each color component of the neighboring image. The value of the minimized error function is added to the variable e, and the obtained gain α and bias β are substituted into elements α (c) and β (c) corresponding to the color c of the array variable. Thereafter, the variable c representing the color is incremented.

S4: It is determined whether or not the process S3 for all the color components RGB has been completed based on whether c representing the color has exceeded 2, that is, B, and if so, branch to S5. If not, continue to S3. I do.

S5: Upon reaching S5, the variable e becomes i
The error when the second degradation template element is applied to the neighboring image is shown. The variable min_e represents the minimum value of the fitting error for all degraded template elements less than i. If e is smaller than min_e, S6
To update the information on the template with the minimum fitting error. If it is larger, the process branches to S7.

S6: min_e is set to e, min_i is set to i,
For c = 0, 1, and 2, update min_α (c) and min_β (c) to α (c) and β (c).

S7: The index i is incremented.

S8: If the index i does not match N, branch to S2, otherwise, branch to S9.

Outputs min_i, mi of least squares approximation unit 212
n_α (c) and min_β (c) are transferred to the color information restoration unit 213 and the band / color information restoration unit 214. Color information restoration unit 2
In step 13, from the template holding ROM 211b after deterioration,
The band / color information restoration unit 214 reads templates Φ min_i (p, q) and Ψ min_i (p, q) corresponding to the index min_i from the pre-deterioration template holding ROM 211a, respectively, and approximates the coefficients min_α (c) and min_β (c ) Using the formula f (p, q, c) = min_α (c) Ψmin_i (p, q) + min_β
(c), g (p, q, c) = min_α (c) Φ min_i (p, q) + min_β
(c) (c = 0,1,2) is calculated, and a neighborhood corrected approximate image f (p, q, c) is obtained by correcting missing color components and band degradation of the neighborhood image h (p, q). , And a neighborhood corrected approximate image g (p,
q, c). Then, these are calculated by the interpolation coefficient calculation unit 215.
Transfer to

The interpolation coefficient calculator 215 estimates the color pixel value before deterioration from f (p, q, c) and g (p, q, c) using the pixel values in the neighboring image. Suitable information is calculated in the form of interpolation coefficients. Therefore, the neighborhood correction approximate image g
The input image buffer 2 which samples (p, q, c) according to the color filter arrangement of the electronic camera 200 in the vicinity.
The images g ′ (p, q) and f (p,
q, c) is examined, and the coefficients required to recover f from g 'by linear operation are calculated. Here, the interpolation coefficient calculation unit 215 calculates coefficients according to the following principle, for example.

First, each pixel of the color c of f is calculated as a linear sum of each pixel of the three-color image g based on the idea of the autoregressive model.

(Equation 2)

Consider approximation as follows. The autoregressive model is a model in which the pixel value of a certain pixel is represented by the sum of a weighted linear sum of pixel values around the pixel and a noise term. Here, the noise term is ignored. In this equation, the integer variable set (u, v) moves in a range D narrower than the range that the variable set (p, q) can take.

H (u, v) is an integer value representing the type of the color filter at the position (u, v) in the neighborhood as R = 0, G = 1, B = 2. This equation, around each position (p, q),
This indicates that the value of f at the position (p, q) is estimated using the pixel values of g sampled through the color filters arranged in the same manner as those around the position (0, 0) (center pixel).

In order to minimize the square error of the approximation by this equation, the equation to be satisfied by w ^c _uv is as follows for an integer variable set (k, l) that moves in the same range as the range D in which (u, v) moves.

(Equation 3)

Is obtained. In this equation, the second term on the left side and the second term on the right side represent spatial correlation information. All quantities other than w ^c _{uv on the} right and left sides can be calculated from f and g, and the entire equation can be solved by matrix operation. At the center of the neighborhood, (p, q) = (0,0) and g ′ (u, v) =
Since g (u, v, H (u, v)), the RGB value of the central pixel before deterioration of the neighboring image can be obtained from only the pixel value in the neighboring pixel using the coefficient w ^c _uv .

Based on this principle, the interpolation coefficient calculator 21
In 5, all colors c are obtained from f (p, q, c) and g (p, q, c).
For = 0, 1, 2 according to equation (2), the interpolation coefficient w ^c _uv
Is calculated and transferred to the linear operation unit 216.

The linear operation section 216 includes the flatness evaluation section 210
, The image transfer is not performed to the least squares approximation unit 212, and thus the transfer of the coefficient from the interpolation coefficient calculation unit 215 is not performed, the variable mask (p, q,
c) The flatness evaluation section 210 is calculated in the same manner as
, The positions corresponding to the respective color components of the neighboring image which have been transferred in advance are obtained, and the average value at those positions is used as the estimated value of the color component at the central pixel of the neighboring image, and written to the corresponding pixel position of the output image buffer 216. .

On the other hand, when the coefficients are transferred from the interpolation coefficient calculation unit 215, the pixels in the neighboring image are weighted and added by the transferred coefficients, and the output image buffer 21
7 is written to the corresponding pixel position.

When these processes are completed for all the pixel positions of the image in the input image buffer 208, the loss of the color component and the deterioration of the band at the relevant pixel position are determined in accordance with the geometrical features in the vicinity. This means that linear interpolation has been performed using the interpolation coefficient to be optimally corrected, and a three-color image in which the band degradation and the lack of color components due to the optical system of the electronic camera 200 have been corrected is obtained in the output image buffer 217.

In this configuration, various modifications are possible as in the first embodiment. It goes without saying that the image pickup device is not limited to the single-plate Bayer array CCD 204. There are various calculation methods of the interpolation coefficient calculation unit 215.
For example, a simple method of using the ratio of each image value to the image obtained from the color information restoring unit 213 and the image obtained from the band / color information restoring unit 214 or the ratio of the sum of squares of the pixel values in the whole neighboring image as an interpolation coefficient. It may be something.

In the second embodiment, similarly to the first embodiment, corrected images for a plurality of neighborhoods including each pixel position can be obtained. In order to effectively utilize this, the operation of the band / color information restoring unit 214 and the color information restoring unit 213 can be changed as follows.

The band / color information restoring unit 214 and the color information restoring unit 213 each store therein an image buffer having the same size (for example, 5 × 5 pixels) as the neighboring image. All pixel values are initialized to 0 when the image is transferred from 200.
Pixel position (x, y) of image in input image buffer 208
During the processing for, the image buffer in the band / color information restoration unit 214 stores the immediately preceding pixel position, for example, (x
-1, y), the band / color information restoring unit 2
14 to the interpolation coefficient calculation unit 215
ev (p, q, c) is retained.

In the process for the pixel position (x, y), the least square approximation unit 212 outputs min_i, min_α (c), min
When _β (c) is transferred, the band / color information restoring unit 214
Reads out f_prev (p, q, c) from the internal image buffer and reads the template {min_i corresponding to the index min_i from the pre-deterioration template holding ROM 211a.
(p, q) is read out, and approximation coefficients min_α (c), min_β
f (p, q, c) = (min_α (c) Ψmin_i (p, q) + min_β using (c)
(c) + f_prev (p + 1, q, c)) / 2 (when p is less than 2) f (p, q, c) = min_α (c) Ψmin_i (p, q) + min_β
(c) (when p is 2) obtains a three-color image f (p, q, c) in which the lack of color components and the deterioration of the band have been corrected for the neighboring image, and the interpolation coefficient calculation unit 21
5 is output. At the same time, f (p, q, c) is replaced by f_prev (p, q, c)
As an internal buffer.

Similarly, during the processing for the pixel position (x, y) of the image in the input image buffer 208, the immediately preceding pixel position (x−1) is stored in the image buffer in the color information restoring unit 213. , Y) in the color information restoration unit 2
14 to the interpolation coefficient calculation unit 215
ev (p, q, c) is retained. In the processing for the pixel position (x, y), the least square approximation unit 212 outputs min_i,
When min_α (c) and min_β (c) are transferred, the color information restoring unit 214 outputs g_prev (p, q, c) from the internal image buffer.
Is read, and the pre-deterioration template holding ROM 211 is read.
template Φ corresponding to index min_i from a
read min_i (p, q) and calculate approximation coefficients min_α (c), mi
Using n_β (c), the equation g (p, q, c) = (min_α (c) Φ min_i (p, q) + min_β
(c) + g_prev (p + 1, q, c)) / 2 (when p is less than 2) g (p, q, c) = min_α (c) Φ min_i (p, q) + min_β
(c) (when p is 2) obtains a three-color image g (p, q, c) in which the lack of color components has been corrected for the neighboring image, and outputs it to the interpolation coefficient calculation unit 215. At the same time, g (p, q, c) is held in the internal buffer as g_prev (p, q, c).

With this operation, the band / color information restoring section 21
4 and the color information restoring unit 213 generate the processing result of each round using the previous processing result, so that the reliability of the processing result is increased and the accuracy of interpolation is improved.

This embodiment can easily cope with the case where the electronic camera 200 does not have the crystal filter 203 and the band is not limited. In this case, the template holding R
Since the OM 211 includes only the pre-deterioration template holding ROM 211a, the least square approximation unit 212 receives the output of the pre-deterioration template holding ROM 211a and outputs the processing result only to the band / color information restoration unit 214.
At the same time, the color information restoration unit 213 is removed, and the interpolation coefficient calculation unit 215 is configured to receive signals only from the band / color information restoration unit 214. In this case, the operation of each part is the same except that the least squares approximation unit 212 uses Ψi instead of Φi as a template. The operation of the band / color information restoring unit 214 is the same, but as a result, a neighborhood corrected approximate image f (p, q, c) is obtained in which the loss of the color component is corrected but the band is not degraded.

The interpolation coefficient calculator 215 sets g (p, q, c) to the same image as f (p, q, c) and proceeds with the calculation. As a result, the interpolation coefficient calculating unit 215 determines the neighborhood image h in the single-plate state.
An interpolation coefficient suitable for correcting the missing color component of the center pixel h (0,0) of (p, q) is obtained, and the linear operation unit 216 performs interpolation using the coefficient. , A three-color image in which the lack of color components of the image in the input image buffer 208 has been corrected is obtained.

In this modification, if the band / color information restoring unit 214 is subjected to the same modification as the modification in which the corrected images for the plurality of neighborhoods including the respective pixel positions are effectively used, the band is limited. Even in the case where there is no data, the band / color information restoring unit 214 generates the processing result of each time by utilizing the previous processing result, thereby improving the accuracy of interpolation.

Hereinafter, a third embodiment of the present invention will be described.
FIG. 6 is a diagram showing a configuration of an image processing system to which the third embodiment of the present invention is applied. The electronic camera 300 is
This is the same configuration as the embodiment. Information on the color filter array of the single-plate Bayer array CCD 304 in the electronic camera 300 is obtained by calculation in the image processing apparatus 301 using the regularity of the Bayer array, as in the second embodiment.

The image processing apparatus 301 includes an input image buffer 308, a neighborhood reading unit 309, a template selecting unit 310, a template holding ROM 311 having a pre-deterioration template holding ROM 311a and a post-deterioration template holding ROM 311b, and a least square approximation unit. 312
A post-deterioration template processing unit 313, a band / color information restoring unit 314, a three-plate image addition unit 315a, a single-plate image addition unit 315b, a single-plate image buffer 316, an output image buffer 317, and a difference calculation unit. 318, a difference image buffer 319, and a regularity evaluation unit 320.

The image buffer 307 in the electronic camera 300
Is connected to the input image buffer 308 in the image processing apparatus 301 as in the second embodiment, but the input image buffer 308 is also connected to the difference calculation unit 318 in addition to the neighborhood read unit 309.

The neighborhood read unit 309 is connected to receive the output of the difference image buffer 319 in addition to the input image buffer 308, and reads out the read information from the regularity evaluation unit 3.
20 to be output. Regularity evaluation unit 320
Is a template selection unit 310 and a least squares approximation unit 312
It is connected to the. The template selection unit 310 includes a template holding ROM 311 in addition to the regularity evaluation unit 320.
Are connected to receive the outputs of the post-deterioration template holding ROM 311b and the pre-deterioration template holding ROM 311a, and select the selection result by the least square approximation unit 312,
Color information restoring section 314 and post-deterioration template processing section 31
3 to be output.

In the template holding ROM 311, as in the first and second embodiments described above, the template #i (i) which is a grayscale image of a predetermined size having N kinds of geometric features is stored in the pre-deterioration template holding ROM 311 a.
= 0 to N−1), and the template holding R after deterioration
The OM 311b includes a pre-deterioration template holding ROM 31.
1a is a template Φi (i = 0 to N−
1) is held.

The difference from the first and second embodiments is that the geometrical features of each template element are classified according to their structural complexity, and it is not a single integer value i for reading out each element. The point is that two sets of numbers (s, i) are required. s represents the complexity of the structure, and is the number of connected components when each template is binarized based on the average value. For example, s = 2 for a template containing only one edge pattern, and s = 3 for a template containing only one line segment pattern. Also,
i is used to access individual elements of the class of template with the same complexity s.

The internal configuration and operation of the least squares approximation unit 312 are the same as those in the second embodiment. The least square approximation unit 312 is connected to output the processing result to the band / color information restoration unit 314 and the post-deterioration template processing unit 313.

The band / color information restoring section 314 is connected to receive the outputs of the least squares approximating section 312 and the template selecting section 310, and outputs the restoration result to the three-plate image adding section 31.
It is connected to output to the output image buffer 317 via 5a.

The post-deterioration template processing unit 313 is connected to receive the outputs of the least square approximation unit 312 and the template selection unit 310, and outputs the processing result to the single image buffer 316 via the single image addition unit 315b. It is connected. The single-chip image buffer 316 is connected to output information stored in the difference calculation unit 318, and the difference calculation unit 318 is connected to the difference image buffer 319 to output the processing result.

The single-plate image buffer 316 and the output image buffer 317 hold a three-color image having the same size as the image in the input image buffer 308, and also hold a counter for each pixel. This counter is provided to hold the number of times writing has been performed at each pixel position of each buffer.

As in the first embodiment, since a plurality of pixel values are calculated at each pixel position in each buffer, the three-plate image addition unit 315a and the single-plate image addition unit 315 are used here.
The processing of averaging them is performed in b. This process requires the number of times of writing.

Hereinafter, the operation of the image processing system having the above-described configuration will be described. In the third embodiment, similarly to the first and second embodiments, an image is taken by the electronic camera 300, and an image having only one color component at each pixel position due to band degradation in the optical system is stored in the image buffer 307. Is held. When the imaging is completed, the image in the image buffer 307 is transferred to the input image buffer 308, and the pixel values and the counters of the color image in the output image buffer 317 and the single image buffer 316 are all initialized to zero.

Next, similar processing is repeated a plurality of times to perform correction. The first processing is performed on the image in the input image buffer 308. As a result of this processing, the input image is approximated using a low-complexity template, and at the same time, an image in which the lack of color components and the band degradation have been corrected is generated in the output image buffer 317. Similarly, the missing color component of the approximated input image is corrected using a template with low complexity, and an image sampled by the Bayer array is generated in the single-chip image buffer 316.

Since the image in the single-chip image buffer 316 is obtained by approximating the input image with a low-complexity template, only the rough geometrical characteristics of the input image are reflected, and an approximation error occurs. This error is stored in the difference image buffer 319 as an error image.

After the second time, processing is performed on the error image in the difference image buffer 319. The error image is approximated and degraded using a template with higher complexity than the previous iteration, and the processing result is added to the output image buffer 317. As the difference image, an error image of the input image and an approximate image obtained by adding all of the stepwise approximations up to and including the first time are output, and the error image is processed again. The details of this processing will be described below.

First, the input image buffer 308
The following processing is performed for each pixel position (x, y) of the image in the image.

A predetermined rectangular area (for example, 5 × 5 pixels) at the pixel position (x, y) is read by the neighborhood reading section 309 as a neighborhood image h (p, q). In the following description, the set of variables (p, q) is a relative coordinate with respect to the center pixel (x, y) of the neighboring image. For example, if the size of the neighborhood is 5 × 5, [−2, −1] , 0,1,2] × [−
2, -1, 0, 1, 2]. In some cases, colors R, G, and B correspond to integer values c = 0, 1, and 2, respectively.

The read neighboring image is sent to the regularity evaluation section 32.
Sent to 0. In the first processing, the neighboring image is transferred to the least squares approximation unit 312 without any processing.
At the same time, the closest value of the possible complexity of the template held in the template holding ROM 311 is output to the template selecting unit 310.

The template selection section 310 sets the value output from the regularity evaluation section 320 to an internal variable s representing the complexity, and stores the internal variable template holding ROM 311
And mediate between other components. Template selection part 3
When there is a request to read the i-th template from the components connected to the
Reference numeral 10 generates a read signal (s, i) using the internal variable s. From the unit 314, the pre-deterioration template holding ROM 311a is accessed.

The least squares approximation unit 312 calculates the neighborhood image h (p,
When q) is transferred, processing is performed according to the flow shown in FIG. 5, and similarly to the first and second embodiments, the index min_i of the degraded template that best approximates the neighboring image and the approximation coefficients min_α (c), min _Β (c) and the post-deterioration template processing unit 313 and the band / color information restoration unit 3
Transfer to 14.

The processing in the band / color information restoring unit 314 is the same as that in the second embodiment except that access to the template holding ROM 311 is performed via the template selecting unit 310. The band / color information restoring unit 314 transfers the neighborhood corrected approximate image f (p, q, c) to the three-plate image adding unit 315a.

The post-deterioration template processing unit 313 firstly, like the color information restoring unit 213 in the second embodiment, corrects only the lack of color components in the neighboring image h (p, q) to obtain a three-color image g. Calculate (p, q, c). Next, g (p, q, c) is finally sampled in accordance with the color filter arrangement of the electronic camera 300 in the vicinity to generate an image g ′ (p, q), which is then sent to the single-chip image adder 315b. Forward.

During the processing, the output image buffer 317
A plurality of approximate pixel values are obtained for each pixel position in the single-chip image buffer 316, and the average approximate pixel values are held in both buffers, and the number of approximate pixel values obtained so far is counted. Is to be held. This number is equal to the number of times writing has been performed on the pixel position so far.

The three-plate image adding unit 315a reads the average approximate pixel value Sf (p, q, c) and the counter Zf (p, q) from the area corresponding to the neighboring image in the image output buffer 317, and reads the new average value. The approximate pixel value Sf _new (p, q, c) is calculated by the following formula: Sf _new (p, q, c) = (Zf (p, q) · Sf (p, q, c) + f
(p, q, c)) / (Zf (p, q) +1) (c = 0,1,2) and the value of the new counter Zf _new (p, q) is given by the formula Zf _new (p, q) = Zf (p, q) +1, and Sf _new (p, q, c) and Zf
_new (p, q) is written back to the area of the output image buffer 317 corresponding to the neighboring image.

On the other hand, in the single-plate image adding unit 315b, the average approximate pixel value Sg '(p, q) of each pixel in the region corresponding to the neighboring image and the counter Z from the single-plate image buffer 316.
g ′ (p, q) is read out and a new average approximate pixel value Sg ′ is read out.
_new (p, q) is calculated as follows: Sg ′ _new (p, q) = (Zg ′ (p, q) · Sg ′ (p, q) +
g '(p, q)) / (Zg' (p, q) by +1) and new counter Zg _'new (p, q values of) the _{Zg' new (p, q)} = Zg '(p, q ) +1 and Sg ′ _new (p, q) and Zg ′ _new
(p, q) is written back to the area corresponding to the neighboring image in the single-chip image buffer 316.

When the above processing is completed for each pixel position of the image in the input image buffer 308, the image in the input image buffer 308 is approximated to the single-plate image buffer 316 by a simple template, and the band degradation correction is performed. The correction of only the lack of the color component is performed without performing the above operation, and the obtained image is further sampled by the color filter arrangement of the single-chip Bayer array CCD 304 to obtain an image in a single-plate state. Similarly, the output image buffer 317 is approximated by a simple template, and a color image that has been corrected for both missing color restoration and band degradation can be obtained.

In the first process, finally, the difference calculation unit 31
8 is a single-plate image buffer 316 and an input image buffer 308
And a difference image representing the approximation error is generated in the difference image buffer 319.

In the second and subsequent processes, the neighborhood reading unit 3
09 switches the subsequent input to the difference image buffer 309, and the processing up to this point is repeated for the difference image. In this repetition, only the regularity evaluation unit 320 has a different operation from the first time.

The regularity evaluation section 320 first calculates the magnitude of the variance of the transferred approximated image, and if it is equal to or smaller than a predetermined threshold, determines that the approximation error is sufficiently small. Do not transfer. If the value is equal to or larger than the predetermined threshold value, a Laplacian filter is applied to the neighboring image in which the maximum value is normalized to 1 and the minimum value is normalized to 0, and the sum of squares is calculated in the obtained image. If the sum of squares is larger than a predetermined threshold, the neighboring image is determined to be noise, and no transfer is performed.

In other cases, the complexity s of the neighboring image is calculated, the complexity s is transferred to the template selecting unit 310, and the neighboring image is transferred to the least square approximation unit 312. When the neighboring image has not been transferred to the least squares approximation unit 312 at all the pixel positions of the difference image, the repetition processing for the difference image ends.

During this time, the fine structure that could not be captured by the previous approximation is calculated as a difference image, the difference image is corrected for deterioration by approximation using a template having the complexity corresponding to the fine structure, and the result is stored in the output image buffer 317. Stored. As a result, in the output image buffer 317, a three-color image is obtained in which the deterioration due to the optical system of the electronic camera 300 and the lack of color are corrected.

In the third embodiment as well, various changes can be considered for the components. For example, the definition of the complexity s includes various candidates other than those described above, such as the intensity of the differential image. Various noise judgment criteria of the regularity evaluation unit 320 can be considered. Further, since the processing of each time is independent, it is also possible to reduce the size of the neighboring image read from the neighboring reading unit 309 according to the number of repetitions. This makes it possible to reduce the types of templates having high complexity.

In order to speed up the processing, it is possible to perform the processing in a plurality of stages in one stage. In this case, the three-plate image addition unit 315a, the single-plate image addition unit 315b, the single-plate image buffer 316, and the difference image buffer 319 become unnecessary, and the output image buffer 317 does not need to hold a counter. The band / color information restoration unit 314 directly outputs the restoration result to the output image buffer 317, the post-deterioration template processing unit 313 directly outputs the processing result to the difference calculation unit 318, and the difference calculation unit 318 directly outputs the processing result to the regularity evaluation unit 320. I do. The operation is also changed as follows.

For each pixel position (x, y) of the image in the input image buffer 308, the neighboring image h (p, q) is read by the neighboring reading unit 309 and sent to the regularity evaluation unit 320. The regularity evaluation unit 320 is a template holding ROM
The lowest value of the possible complexity of the template held in 311 is output to template selecting section 310, and neighboring image h (p, q) is transferred to least squares approximating section 312.

The least-squares approximation unit 312 calculates an index and an approximation coefficient of one of the post-deterioration templates selected by the template selection unit 310 that best approximates the neighboring image h (p, q), and The data is transferred to the processing unit 313 and the band / color information restoring unit 314.

The band / color information restoring unit 314 obtains a neighborhood correction approximate image f (p, q, c) obtained by correcting the lack of color components and the deterioration of the band with respect to the neighborhood image, and obtains the value f (p, 0,0, c)
Is added to the corresponding pixel position of the output image buffer 317.

On the other hand, the post-deterioration template processing unit 313
Calculates a three-color image g (p, q, c) obtained by correcting only missing color components with respect to the neighboring image h (p, q), and further calculates g (p, q, c) in the neighboring image. An image g '(p, q) is obtained by sampling according to the color filter arrangement of the electronic camera 300.
g ′ (p, q) is transferred to the difference calculator 318, and the neighborhood image h
The difference from (p, q) is calculated as a neighborhood difference image and transferred to the regularity evaluation unit 320.

The regularity evaluation section 320 is the same as the second embodiment.
The complexity of the transferred neighborhood difference image is checked in the same manner as in the processing in the third processing, and if the neighborhood difference image is not regarded as noise, the calculated complexity is used as the template selection unit 31.
0, and transfers the neighborhood difference image to the least squares approximation unit 312 again.
Transfer to

As a result, the post-deterioration template is applied to the neighboring image at each pixel position in ascending order of complexity, and the result of the application corrects the loss of the color component of the central pixel value of the nearby image and the deterioration of the band and outputs the corrected image. Output to the image buffer 317.

The electronic camera 300 has a crystal filter 3
03 and the band is not limited, the template holding ROM 311 stores the pre-deterioration template holding RO.
The configuration is changed so that the template selection unit 310 receives only the input from the pre-deterioration template holding ROM 311a and the regularity evaluation unit 320 only. In this case, the operation of each part is different from that of the related art in that the template selection unit 310 receives the template read request from any of the constituent elements, and
11a, the least square approximation unit 212 uses Ψi instead of Φi as a template, and the band / color information restoring unit 314 operates in the same manner. All that is required is to output a neighborhood correction approximate image f (p, q, c) in which no degradation is performed. As a result, as the second and subsequent iterations progress, the more complex geometric features of the input image are reflected, and the image in which the color component has been corrected is retained. A three-color image is obtained in which the lack of color components of the image in the image buffer 308 has been corrected.

Hereinafter, a fourth embodiment of the present invention will be described. FIG. 7 is a diagram showing the configuration of an image processing system to which the fourth embodiment of the present invention is applied, and FIG. 8 is a diagram showing the configuration of the least squares approximation unit 412 shown in FIG. 7 and its peripheral circuits.

In FIG. 7, the electronic camera 400 has the same configuration as that of the second embodiment except that the single-chip Bayer array CCD in FIG. 4 is replaced with a single-chip RGB random array CCD 404. Single plate R in electronic camera 400
Information on the color filter array of the GB random array CCD 404 is stored in advance in the image processing device 4 as shown in FIG.
01 is held in the color component mask generation unit 412a.

The image processing device 401 includes an input image buffer 408, a neighborhood read unit 409, a template storage ROM 411a before deterioration and a template storage RO after deterioration.
M411b and a template holding ROM 411 having
, Least square approximation unit 412, band / color information restoration unit 41
4, an image adding unit 415, and a three-plate image buffer 417a
And an approximate image buffer 417 including a single-plate image buffer 417b, a residual calculation unit 418, and a regularity determination unit 41.
9, a residual adder 420, and an output image buffer 421.

As shown in FIG. 8, the least square approximation unit 412 includes a color component mask generation unit 412a, a least square coefficient calculation unit 412b, an approximation error accumulation unit 412c, an approximation coefficient holding buffer 412d, and a template determination unit 412e. And The least square approximation unit 412 further includes a neighborhood image input terminal 412 connected to the neighborhood reading unit 409.
r, a template input terminal 412p connected to the deteriorated template holding ROM 411b, and an approximate parameter output terminal 412 connected to the band / color information restoring unit 414.
s, and a neighborhood approximate image output terminal 412 t connected to the image addition unit 415.

In FIG. 7, an image buffer 407 in the electronic camera 400 is connected to an input image buffer 408 in the image processing device 401. The input image buffer 408 is connected to the neighborhood reading unit 409 and the residual calculation unit 418. The neighborhood reading unit 409 is connected to the least square coefficient calculation unit 412b in the least square approximation unit 412 via the neighborhood image input terminal 412r shown in FIG.

In FIG. 8, the least square coefficient calculating section 412
b is connected to a color component mask generation unit 412a and a degraded template holding ROM 411b via a template input terminal 412p in addition to the neighborhood reading unit 409, and outputs a processing result to an approximation coefficient holding buffer 412d and an approximation error accumulating unit 412c. Connected to

The template determining section 412e is connected to receive the outputs of the approximate error accumulating section 412c, the approximate coefficient holding buffer 412d and the color component mask generating section 412a, and outputs the determined content to the neighborhood correction approximate image output terminal 41.
It is connected to the image addition unit 415 via 2t and to the band / color information restoration unit 414 via the approximate parameter output terminal 412s.

The template holding ROM 411 is the same as the second embodiment.
a and a template holding ROM 411b after deterioration. The template Ψi (i = 0 to N−1) which is a grayscale image of a predetermined size having predetermined N kinds of geometric features is stored in the pre-deterioration template storage ROM 411 a, and stored in the band / color information restoration unit 414. It is connected.
The template Φi (i = 0 to N−1), which is an image in which each template of the template storage ROM 411a before deterioration has been subjected to band deterioration by the optical system of the electronic camera 400, is stored in the template storage ROM 411b after deterioration.

As shown in FIG. 7, the band / color information restoring unit 4
Reference numeral 14 is connected to receive the outputs of the least square approximation unit 412 and the pre-deterioration template holding ROM 411a, and is connected to output the restoration result to the image addition unit 415. The image adding unit 415 is connected to independently process the outputs from the band / color information restoring unit 414 and the least squares approximating unit 412, and outputs the respective processing results to the approximate image buffer 417.
Three image buffer 417a and single image buffer 4
17b.

The three-plate image buffer 417a and the single-plate image buffer 417b hold an image having the same number of pixels as the image in the input image buffer 408, and also hold a counter for each pixel. This counter is provided to hold the number of times writing has been performed at each pixel position of each buffer. Similar to the first embodiment, since a plurality of pixel values are calculated at each pixel position in each buffer, here, the image adding unit 415 performs a process of averaging them. This process requires the number of times of writing.

The single-chip image buffer 417b is connected to output the stored contents to the residual adding section 420 via the residual calculating section 418 and the regularity determining section 419. on the other hand,
The three-plate image buffer 417a is connected to directly output the stored contents to the residual adding unit 420. Residual adder 4
Reference numeral 20 is connected to output the processing result to the output image buffer 421.

Hereinafter, the operation of the image processing system having the above configuration will be described.

As in the first to third embodiments, the subject image picked up by the electronic camera 400 is band-limited by the lens 402 and the quartz filter 403, and becomes a degraded optical image on the single-chip RGB random array CCD 404. It is formed. The single-chip RGB random array CCD 404 acquires any one of RGB color information of the deteriorated image for each pixel, and transfers the obtained image to the image buffer 407 as a digital image signal via the A / D converter 405. When the imaging is completed, the image information in the image buffer 407 is transferred to the input image buffer 408 in the image processing device 401,
Further, the pixel values and counters of the three-plate image buffer 417a and the single-plate image buffer 417b are all initialized to 0.
The following processing is performed for each pixel position (x, y) of the image information in 8.

A predetermined rectangular area (for example, 5 × 5 pixels) centered on the processing pixel position is read by the neighborhood reading section 409 as a neighborhood image h (p, q), and the least square approximation section 41
2 is transferred to the least squares coefficient calculation unit 412b. In the following description, the set of variables (p, q) is a relative coordinate with respect to the center pixel (x, y) of the neighboring image. For example, if the size of the neighborhood is 5 × 5, [−2, −1] , 0,1,
2] × [−2, −1, 0, 1, 2]. In some cases, the colors R, G, and B are respectively set to integer values c =
Description will be made in correspondence with 0, 1, and 2.

The least squares coefficient calculating section 412b sequentially reads out templates Φi (p, q) (i = 0 to N-1) stored in advance in the degraded template holding ROM 411b, and transfers the transferred neighborhood image h (p , q) for each color component c (c =
0, 1, 2)

(Equation 4)

Approximate coefficients α (c) and β (c) for minimizing the following are calculated.

Here, mask (p, q, c) is the coordinate (p, q, c) in the neighboring image on the single-chip RGB random array CCD 404.
If the color filter corresponding to the color component c exists at the position corresponding to q), the mask takes 1; otherwise, it takes 0, and is calculated by the color component mask generation unit 412a. FIG. 3 illustrates a color component mask corresponding to a specific color filter arrangement in a neighboring image.

The approximation error e (c) is calculated by the approximation error accumulator 412c.
And the approximation error accumulating unit 412c outputs the approximation error e
(c) is cumulatively added for all color components, and transferred to the template determining unit 412e as a cumulative approximate error e. The approximate coefficients α (c) and β (c) (c = 0, 1, 2) are sequentially output to the approximate coefficient holding buffer 412d and held there.

The template determination unit 412e holds the minimum value min_e of the cumulative approximation error with respect to the previous template Φi ′ (p, q) (i ′ <i) and the index min_i of the template that has given the minimum value. . The template determination unit 412e stores the error e transferred from the approximation error accumulation unit 412c as the minimum value
e, and if the new error is smaller,
And min_i is updated to i.

The template determining unit 412e also calculates the approximation coefficients for the template that gives the minimum value of the error by min_α (c) and min_β (c) (c = 0, 1, 2).
These values are updated to the approximate coefficients α (c) and β (c) read from the approximate coefficient holding buffer 412d at the same time as the update of min_e.

After all the degraded templates Φi have been processed, the approximation coefficients min_α (c), min_β (c) (c = 0, 1, 2) and the index min_i stored in the template determination unit 412e are assigned to the band. -Output to the color information restoration unit 414. At the same time, the template determination unit 412e first calculates the expression g (p, q, c) = min_α (c) Φ min_i (p, q) + min_β
An image g (p, q, c) in which the missing color component of the neighboring image h (p, q) is corrected by (c) is calculated.

Next, using the mask (p, q, c) calculated by the color component mask generation unit 412a, the color image g (p, q, c) is changed according to the color filter arrangement of the electronic camera 400 in the neighboring image. G ′ (p, q)
And transfers it to the image addition unit 415.

The band / color information restoring unit 414 determines the pre-deterioration template Ψ corresponding to the transferred index min_i.
min_i is read from the pre-deterioration template holding ROM 411a, and the transferred approximation coefficients min_α (c), min_β
(c) By using (c = 0, 1, 2), the missing color component of the neighboring image h (p, q) is corrected, and the vicinity correction approximate image f (p, q, c) by f (p, q, c) = min_α (c)
。 Calculated by min_i (p, q) + min_β (c) (c = 0,1,2) and transferred to the image adding unit 415.

During the processing, the three-plate image buffer 417
a and a single-plate image buffer 417b, a plurality of approximate pixel values are obtained for each pixel position. Both buffers hold their average approximate pixel values, and the number of approximate pixel values obtained so far is It is held in the counter. This is equal to the number of times writing has been performed on the pixel position so far.

In the image adding section 415, when f (p, q, c) is transferred from the band / color information restoring section 414, the average approximation of each pixel in the area corresponding to the neighboring image is performed from the three-plate image buffer 417a. The pixel value Sf (p, q, c) and the counter Zf (p,
q) and read a new average approximate pixel value Sf _new (p, q, c)
Is calculated by the following formula: Sf _new (p, q, c) = (Zf (p, q) · Sf (p, q, c) + f
(p, q, c)) / (Zf (p, q) +1) (c = 0,1,2), and the value of the new counter Zf _new (p, q) is calculated by the equation Zf _new (p, q). = Zf (p, q) +1, and Sf _new (p, q, c) and Zf _new (p, q)
q) is written back to the area for the neighboring image in the three-plate image buffer 417a.

On the other hand, when g ′ (p, q) is transferred from the template determining unit 412 e to the image adding unit 415, the image adding unit 415 sends each pixel in the area corresponding to the neighboring image from the single-plate image buffer 417 b. Average approximate pixel value Sg ′ (p,
q) and the counter Zg ′ (p, q) are read, and a new average approximate pixel value Sg ′ _new (p, q) is calculated by the following formula: Sg ′ _new (p, q) = (Zg ′ (p, q) · Sg ′ (P, q) +
g ′ (p, q)) / (Zg ′ (p, q) +1), the value of the new counter Zg ′ _new (p, q) is changed to Zg ′ _new (p, q) = Zg (p, q) +1 Sg ′ _new (p, q) and Zg ′
_new (p, q) is written back to the area corresponding to the neighboring image in the single-chip image buffer 417b.

When the above processing is completed for each pixel position of the image in the input image buffer 408, the image in the input image buffer 408 is approximated by the template in the single-plate image buffer 417b to correct the missing color component. After that, an approximate image in a single-plate state obtained again by sampling the color filters of the single-chip RGB random array CCD 404 is obtained. Similarly, the three-plate image buffer 417a is approximated by a template, and a color image corrected for missing color components and band degradation is obtained.

Further, the following processing is performed on the image in the approximate image buffer 417.

[0161] The residual operation unit 418
For each pixel position (x, y) of the image in 08, a neighboring image h (p, q) of a predetermined size centered on the pixel position is read. At the same time, an image corresponding to the area of the neighboring image is read out from the single-chip image buffer 417b, a difference from h (p, q) is calculated, and the regularity determining unit 41 calculates the difference as a residual image d (p, q).
9 is output.

The regularity judging section 419 determines whether or not the input residual image d (p, q) is an image having a regular structure.
Evaluation is made based on the same criteria as in the embodiment. First, d (p, q) is normalized to a maximum value 1 and a minimum value 0, and a Laplacian filter is applied. The result is sum of squares of the whole image and is compared with a predetermined threshold. If the difference is smaller than the threshold value, it is determined that the pixel has a regular structure, and the value d (0,0) of the central pixel of the residual image is output to the residual adding unit 420. If the value exceeds the threshold, nothing is transferred.

The residual adder 420 is a three-plate image buffer 4
17a to the position (x, y) in the input image buffer 408
Pixel value Sf (0,0, c) at the position corresponding to (c = 0,1,
Read out 2). When the value d (0,0) is transferred from the regularity determining unit 419, pixel values d '(0,0, c) for three colors are generated from d (0,0), and Sf ( 0,0, c) and each color c = 0,1
Add every two. Here, the single-chip RGB random array C corresponding to the position (x, y) in the input image buffer 408
Assuming that the color of the color filter existing at the position on the CD 404 is c ', d' (0,0, c) is calculated from d (0,0) by the following equation.

D ′ (0,0, c) = d (0,0) (when c = c ′)

(Equation 5)

When this processing is completed for all positions (x, y), the missing color components and the band degradation are corrected within a range in which the template in the template holding ROM 411 can correspond to the geometric feature in the vicinity at each pixel position. In addition, since the unsupported fine structure is reflected in the luminance component, a three-color image in which the deterioration and color loss due to the optical system of the electronic camera 400 are corrected is obtained in the output image buffer 421.

If the electronic camera 400 does not have the crystal filter 403 and the band is not limited, the template holding ROM 411 stores the pre-deterioration template holding ROM 411.
a, the least-squares coefficient calculation unit 412 b in the least-squares approximation unit 412 stores the pre-deterioration template holding ROM 41.
The configuration is changed so as to receive an input from 1a. In this case, the feature of the operation of each part is that the least squares approximation unit 4
12, the operation of the band / color information restoring unit 414 is the same as that of using ｉi instead of Φi as a template, but as a result, the missing correction of the color component is corrected, but the deterioration of the band is not corrected. The only point is that the image f (p, q, c) is obtained. As a result, the input image buffer 408
The output image buffer 421 obtains a three-color image in which only the lack of the color components of the image in the image is corrected.

In order to speed up the processing, a configuration in which the approximate image buffer 417 is not provided is also possible. In this case, since the least square approximation unit 412 directly outputs to the residual calculation unit 418 and the band / color information restoration unit 414 directly outputs to the residual addition unit 420, the image addition unit 415 is also unnecessary. The operation is as follows. For each pixel position (x, y) of the input image buffer 408, the residual calculation unit 418 and the image g ′ (p, q) in the single-plate state transferred from the least square approximation unit 412 and the pixel position ( x,
The difference between the neighboring images h (p, q) centered on y) is calculated and transferred to the regularity determining unit 419, and the difference between f (p, q, c) calculated by the band / color information restoring unit 414 is calculated. Central pixel value f (0,0, c)
Is transferred to the residual adder 420, and the residual adder 420 replaces f (0,0, c) transferred from the band / color information restorer 414 instead of Sf (0,0, c). Using d '(0,
0, c) is different from the fourth embodiment described above. As a result, although the accuracy of the correction is reduced, the storage capacity is small, and higher-speed processing can be performed.

Hereinafter, a fifth embodiment of the present invention will be described. FIG. 9 is a diagram showing a configuration of an image processing system to which the fifth embodiment of the present invention is applied, and includes an electronic camera 500 and an image processing device 501.

The electronic camera 500 of the present embodiment has the same configuration as the electronic camera 200 of the second embodiment except that the crystal filter 203 provided for band limitation is not provided. The output of the image buffer 507 in the electronic camera 500 is connected to the input image buffer 508 in the image processing device 501 as in the third embodiment.
The input image buffer 508 is connected to the neighborhood reading unit 509 and the difference calculation unit 518.

The image processing device 501 includes an input image buffer 508, a neighborhood read unit 509, a template selection unit 510, a template holding ROM 511, a least square approximation unit 512, a single-plate image generation unit 513, and a color information restoration unit. Unit 514, three-plate image addition unit 515a, single-plate image addition unit 515b, single-plate image buffer 516, output image buffer 517, difference calculation unit 518, difference image buffer 519, and regularity evaluation unit 520. , A template generation unit 521.

In FIG. 9, the neighborhood read unit 509 is connected to receive the outputs of the input image buffer 508 and the difference image buffer 519, and is connected to output the read information to the regularity evaluation unit 520. The regularity evaluation unit 520 is connected to the template generation unit 521, the three-plate image addition unit 515a, and the least square approximation unit 512. The least square approximation unit 512 has the same configuration as the configuration in the second embodiment, is connected to receive the outputs of the template selection unit 510 and the regularity evaluation unit 520, and outputs the processing result to the single-plate image generation unit 513 and the color information. It is connected to output to the restoration unit 514. Template selection section 510
Is connected to receive the output of the template holding ROM 511 and the template generation unit 521, and is connected to output the selection result to the single-plate image generation unit 513 and the color information restoration unit 514.

Since the input image has no band deterioration due to the crystal filter, the template holding ROM 511 does not hold the templates before and after the band limit deterioration unlike the previous embodiments. Instead, a grayscale image of a predetermined size having predetermined N types of geometric features is stored as a template Φi (i = 0 to N−1). Here, band degradation due to the lens 502 is usually ignored because it is small.

The color information restoring unit 514 includes the three-plate image adding unit 51
It is connected to an output image buffer 517 via 5a.
The veneer image generation unit 513 is connected to the veneer image buffer 516 via the veneer image addition unit 515b. The single-chip image buffer 516 is connected to output the stored contents to the difference calculation unit 518, and the difference calculation unit 518 is connected to the difference image buffer 51.
9 is connected to output the processing result.

The single-plate image buffer 516 and the output image buffer 517 hold images having the same number of pixels as the images in the input image buffer 508, but also hold a counter for each pixel. This counter is provided to hold the number of times data has been written to each pixel position in each buffer. Since a plurality of pixel values are calculated at each pixel position of each buffer as in the third embodiment,
Finally, a process for averaging them is performed. This process requires the number of times of writing.

Hereinafter, the operation of the image processing system having the above configuration will be described.

An image is taken by the electronic camera 500, and an image including only the band degradation due to the lens 502 and having only one color component at each pixel position is held in the image buffer 507 as in the previous embodiments. When the imaging is completed, the image in the image buffer 507 is
08, the pixel values of the color image in the output image buffer 517 and the single image buffer 516, and the counter are all initialized to zero. Next, a correction process is performed in two stages. In the first processing, the input image buffer 508
The following processing is performed for each pixel position (x, y) of the image in the image.

A predetermined rectangular area (for example, 7 × 7 pixels) at the pixel position (x, y) is read as a neighboring image h (p, q) by the neighboring reading unit 509. In the following description, the set of variables (p, q) is a relative coordinate with respect to the center pixel (x, y) in the neighboring image. For example, if the size of the neighborhood is 7 × 7, [-3,- 2, -1,0,1,2,
3] × [−3, −2, −1, 0, 1, 2, 3]. In some cases, colors R, G, and B correspond to integer values c = 0, 1, 2, respectively. Will be explained.

The read neighboring image is sent to the regularity evaluation section 52.
0, where it is transferred to the least squares approximation unit 512 without any processing. Template generator 52
No. 1 is not transferred in the first processing.

When a nearby image is transferred, the template generating section 521 regards the image as a grayscale image, smoothes the image while maintaining the edge, creates a new template, and transfers the template to the template selecting section 510. However, nothing is performed in the first processing.

The template selecting section 510 returns the template when the template is transferred from the template generating section 521 when there is a read request for the i-th template from another connected component. In the case where nothing has been transferred, it has a function of reading the i-th template Φi from the template holding ROM 511. In the first process, since nothing is transferred from the template generation unit 521, the template holding ROM 511 is accessed.

When the neighborhood image is transferred, the least squares count calculation unit 512 performs the processing according to the flow shown in FIG. 5, and similarly to the previous embodiments, the index of the template min_i that best approximates the neighborhood image and Approximation coefficient
Calculate min_α (c) and min_β (c) and transfer them to the single-plate image generation unit 513 and the color information restoration unit 514.

In the color information restoring unit 514, in order to restore missing color components for each pixel, the following expression is used: g (p, q, c) = min_α (c) Φ min_i (p, q) + min_β
(c) According to (c = 0,1,2), a three-color image g (p, q, c) in which the missing color component of the neighboring image h (p, q) is approximately filled is generated, and a three-plate image Adder 5
15a.

The single-plate image generation unit 513 converts the three-color image g (p, q, c) in which the missing color components of the neighboring image h (p, q) are approximately filled in, similarly to the color information restoration unit 514. Calculate, but also g
An image g ′ (p, q) obtained by sampling (p, q, c) according to the color filter arrangement of the electronic camera 500 in the vicinity image
Generate Then, g ′ (p, q) is added to the single-plate image adding unit 51.
5b.

During the processing, the output image buffer 517
A plurality of approximate pixel values are obtained for each pixel position in the single-plate image buffer 516, and the average approximate pixel values are held in both buffers, and the number of approximate pixel values obtained so far (= The number of times of writing is held in the counter.

In the three-plate image adding unit 515a, the average approximate pixel value Sg (p, q, c) of each pixel in the area corresponding to the neighboring image from the image output buffer 517 and the counter Zg (p,
q) is read out, and a new average approximate pixel value Sg _new (p, q, c) is calculated by using the transferred g (p, q, c), and a calculation formula Sg _new (p, q, c) = (Zg (p, q) Sg (p, q, c) + g
(p, q, c)) / (Zg (p, q) +1) (c = 0,1,2) and a new counter Zg _new (p, q)
Is calculated by the equation Zg _new (p, q) = Zg (p, q) +1, and Sg _new (p, q, c) and Zg
_new (p, q) is written back to the area of the image output buffer 517 corresponding to the neighboring image.

On the other hand, the single-panel image addition unit 515b calculates the average approximate pixel value Sg 'from the region corresponding to the neighboring image in the single-panel image buffer 516, as in the three-panel image addition unit 515a.
(p, q) and the counter Zg '(p, q) are read out, and a new average approximate pixel value Sg' is obtained using the transferred g '(p, q).
_new (p, q) is calculated by Sg ′ _new (p, q) = (Zg ′ (p, q) · Sg ′ (p, q) +
g ′ (p, q)) / (Zg ′ (p, q) +1) and a new counter Zg ′ _new (p,
q) is calculated by Zg ′ _new (p, q) = Zg ′ (p, q) +1, and Sg ′ _new (p, q) and Zg ′ _new
(p, q) is written back to the area of the single-chip image buffer 516 corresponding to the neighboring image.

When the above processing is completed for each pixel position of the image in the input image buffer 508, the image in the input image buffer 508 is approximated to the single-plate image buffer 516 by the template in the template holding ROM 511.
After compensating for the lack of color components, the single-layer Bayer array C
An image sampled by the color filter arrangement of the CD 504 is obtained. Similarly, in the output image buffer 517, a color image approximated by the template in the template holding ROM 511 and a missing color component is corrected is obtained.
At the end of the first processing, the difference calculation unit 518 calculates the difference between the single-plate image buffer 516 and the input image buffer 508,
The difference image representing the approximation error is output to the difference image buffer 519.

In the second processing, the input image buffer 50
The following processing is performed for each pixel position (x, y) of No. 8.

A rectangular area having the same size as the first time around the pixel position (x, y) is read again as a neighboring image h (p, q) by the neighboring reading unit 509. at the same time,
The area corresponding to the neighboring image in the difference image buffer 519 is read as the difference neighboring image d (p, q). The two images are transferred to the regularity evaluation unit 520.

The regularity evaluation section 520 examines the regularity of the difference neighboring image d (p, q) unlike the first processing. As a criterion for evaluation, a Laplacian filter is applied to an image obtained by normalizing the image to a maximum value of 1 and a minimum value of 0, and a value obtained by taking the sum of squares of the result with respect to the entire image is used as a noise measure. To determine a noise measure L1 for d (p, q), and to determine how regular an image composed of white noise is, each pixel value has a maximum value of 1, a minimum value of 0, and an average value of 0.5. Generated by a random number that produces white noise
Using the noise scale L0 calculated in advance for an image of the same size as (p, q), the relative noise scale L is calculated by the equation L =
It is calculated by (1-L1 / L0).

The relative noise scale L is d in the range of 0 to 1.
Represents the regularity of (p, q). If L is equal to or less than a predetermined threshold,
d (p, q) does nothing because it is noise. If it is equal to or greater than the predetermined threshold, d (p, q)
And h (p, q) to the least squares approximation unit 512,
Further, the relative noise measure L is transferred to the three-plate image adding unit 515a.

Unlike the first processing, the template generating section 521 regards the transferred d (p, q) as a grayscale image, smoothes it while keeping the edge, and creates a new template Φ ′ (p, q ) And a template selection unit 510
Transfer to

The template selection unit 510 sets the transferred template Φ ′ (p, q) as a new template set, and thereafter, when there is a template read request from another component, Φ ′ (p, q) ) return it.

The least square coefficient calculating section 512 calculates the neighborhood image h
When (p, q) is transferred, the same processing as the first time is performed, and the index min_i of the template that best approximates the neighboring image h (p, q) and the approximation coefficients min_α (c), min_β (c)
Is calculated and transferred to only the color information restoring unit 514. However, since the template read out by the least square coefficient calculating unit 512 is limited to one of Φ ′ (p, q) by the template selecting unit 510, min_α (c) and min_β (c) are
This means an optimum approximation coefficient when the neighborhood image h (p, q) is approximated by the template Φ ′ (p, q).

The processing in the color information restoring unit 514 differs from the first processing in that the center pixel h (0,
Only the missing color component in step (0) is recovered. Specifically, the equation g (0,0, c) = min_α (c) Φ ′ (p, q) + min_β (c)
According to (c = 0,1,2), a recovered pixel value g (0,0, c) (c = 0,1,2) of a color component corresponding to h (0,0) is generated, and The data is transferred to the image addition unit 515a.

The neighboring image h (p, p) in the output image buffer 517
For each pixel in the area corresponding to q), the pixel values Sg (p, q, c) (c = 3) of the three colors obtained by approximation by the template in the template holding ROM 511 in the first processing are obtained.
0, 1, 2) are held. Three-plate image addition unit 515
In a, the pixel value Sg (0,0, c) at the position corresponding to the position of the center pixel of the neighboring image h (p, q) in the image output buffer 517
(C = 0, 1, 2), and using the relative noise scale L transferred from the regularity evaluation unit 520, the previous approximation result Sg (0,0, c) and the second approximation result g ( Take the weighted sum of 0,0, c). Formula Sg _new (0,0, c) = (1−L) · Sg (0,0, c) + L · g
(0,0, c) (c = 0,1,2) gives a new value Sg _new (0,0, c) (c =
(0, 1, 2) is calculated and written back to the image output buffer 517. When the second processing is completed for each pixel position in the input image buffer, a three-color image in which the lack of color components has been recovered is obtained in the output image buffer 517.

Hereinafter, a sixth embodiment of the present invention will be described.
10 to 12 are diagrams for explaining the sixth embodiment, FIG. 10 is a diagram showing a configuration of an image processing system to which the present embodiment is applied, and FIG. 11 is a composite least square approximation unit in FIG. FIG. 12 is a detailed diagram of FIG. 612, and FIG. 12 is an explanatory diagram of the operation of the combined least squares approximation unit 612 in FIG.

In FIG. 10, the image processing system according to the present embodiment includes an electronic camera 600 and an image processing device 601, and the electronic camera 600 has the same configuration as the electronic camera 400 according to the fourth embodiment. An image buffer 607 in the electronic camera 600 is connected to an input image buffer 608 in the image processing device 601. Electronic camera 600
The information regarding the color filter arrangement of the single-chip RGB random array filter CCD 604 in the color component mask generation unit 61 in the image processing device 601 is previously stored in the same manner as in the fourth embodiment.
2a.

The image processing apparatus 601 includes an input image buffer 608, a neighborhood reading unit 609, a pre-deterioration template holding ROM 611a and a post-deterioration template holding RO.
Template holding ROM 611 including M611b
, A combined least squares approximation unit 612, a band / color information restoration unit 614, a three-plate image addition unit 615, and an output image buffer 616.

The composite least squares approximation unit 6 shown in FIG.
12 is a color component mask generation unit 61 as shown in FIG.
2a, a least squares coefficient calculating unit 612b, an approximate error accumulating unit 612c, an approximate coefficient holding buffer 612d, a template determining unit 612e, and a nearby image buffer 612f.
, A template accumulating section 612g, a template synthesizing section 612h, a least square coefficient calculating section 612i, an approximate error accumulating section 612j, and an approximate coefficient holding buffer 612k. The composite least squares approximation unit 612 further includes a neighborhood image input terminal 612r and a template input terminal 61
2p and an approximate parameter output terminal 612s.

The input image buffer 608 is connected to the neighborhood read section 609, and the neighborhood read section 609 is connected to the neighborhood image buffer 612f in the composite least squares approximation section 612 via the neighborhood image input terminal 612r. The combined least squares approximation unit 612 is connected to the post-deterioration template holding ROM 611b via the template input terminal 612p, and is connected to the band / color information restoration unit 614 via the approximation parameter output terminal 612s.

The structure of the template holding ROM 611,
Template holding ROM 611, band / color information restoring unit 6
14. Three-plate image adder 615 and output image buffer 6
The connection relation between 16 is the same as that of the first embodiment.

In this embodiment, similar to the first embodiment,
Since multiple correction values are calculated for each pixel during processing,
These are weighted and averaged based on the reliability of each correction value. Therefore, the output image buffer 616 holds a confidence value required for calculating a weighted average in addition to image data at each pixel position.

The neighborhood image buffer 612f in the composite least squares approximation section 612 is connected to receive the output of the neighborhood read section 609 via the neighborhood image input terminal 612r, and stores the stored information into the least square coefficient calculation section 612b and 6
12i for output. The least square coefficient calculation unit 612b includes a color component mask generation unit 612a and a template input terminal 61 in addition to the neighborhood image buffer 612f.
It is connected to receive the output of the post-degradation template holding ROM 611b via 2p, and is connected to output the processing result to the approximation coefficient holding buffer 612d and the approximation error accumulating unit 612c.

The template accumulation unit 612g is connected to receive the outputs of the approximation coefficient holding buffer 612d and the approximation error accumulation unit 612c, and is connected to output the accumulation result to the template synthesis unit 612h. The template synthesizing unit 612h outputs the synthesis result to the least square coefficient calculating unit 61.
2i and a template determination unit 612e.

The least square coefficient calculating section 612i includes a color component mask generating section 612 in addition to the template synthesizing section 612h.
a and the output of the neighboring image buffer 612f, and the processing result is stored in the approximate coefficient holding buffer 612k.
And to the approximation error accumulator 612j. The approximation coefficient holding buffer 612k and the approximation error accumulating unit 612j are both connected to the template determining unit 612e, and the template determining unit 612e is connected to the band / color information restoring unit 614 via the approximate parameter output terminal 612s.

The operation of the image processing system having the above configuration will be described below.

The subject image picked up by the electronic camera 600 is band-limited by the lens 602 and the crystal filter 603 as in the first to fourth embodiments.
Single-chip RGB random array CCD6 as a deteriorated optical image
04 is formed. Single-chip RGB random array CCD6
Reference numeral 04 obtains any one of RGB color information of the deteriorated image for each pixel, and transfers the obtained image to the image buffer 607 as a digital image signal via the A / D converter 605. When the imaging is completed, the image information in the image buffer 607 is stored in the input image buffer 608 in the image processing device 601.
And the pixel values and the reliability values of the output image buffer 616 are all initialized to zero.

After that, the following processing is performed for each pixel position (x, y) of the image information in the input image buffer 608.

[0210] A predetermined rectangular area (for example, 5 x 5) centered on the processing pixel position (x, y) is detected by the neighborhood read unit 609.
The pixel is read as the neighboring image h (p, q) and transferred to the composite least squares approximation unit 612. In the following description,
The set of variables (p, q) is the central pixel (x,
y), for example, when the size of the neighborhood is 5
If it is × 5, [−2, −1, 0, 1, 2] × [−2, −
[1, 0, 1, 2], and the colors R, G, and B may be associated with integer values c = 0, 1, and 2, respectively.

The neighborhood image h (p, q) read by the neighborhood reading unit 609 is transferred to the neighborhood image buffer 612f in the composite least squares approximation unit 612. Nearby image buffer 6
When h (p, q) is transferred to 12f, on the other hand, veneer RG
A variable mask that takes 1 if a color filter corresponding to the color component c exists at a position corresponding to the coordinates (p, q) in the neighboring image on the B random array CCD 604, and takes 0 otherwise.
(p, q, c) is calculated by the color component mask generation unit 612a and is stored therein. FIG. 3 illustrates a color component mask corresponding to a specific color filter arrangement in a neighboring image.

The least square coefficient calculating section 612b sequentially reads out templates Φi (p, q) (i = 0 to N-1) stored in advance in the degraded template holding ROM 611b, and transfers the templates Φi (p, q) (i = 0 to N-1) from the neighboring image buffer 612f. For each color component c (c = 0,1,2) of the neighboring image h (p, q), an approximation error is sequentially obtained using the variable mask (p, q, c).

(Equation 6)

Approximate coefficients α (c) and β (c) for minimizing are calculated.

The approximation error e (c) is calculated by the approximation error accumulator 612c.
And the approximation error accumulating unit 612c outputs the approximation error e
(c) is cumulatively added for all color components, and is transferred to the template accumulating unit 612g as a cumulative approximation error e. The approximate coefficients α (c) and β (c) (c = 0, 1, 2) are sequentially output to the approximate coefficient holding buffer 612d and held there. The template accumulation unit 612g holds a table T of N rows.

FIGS. 12A and 12B are diagrams for explaining the operation of the composite least squares approximation unit. FIG. 12A shows an arbitrary neighboring image.
(B) shows N templates from 0 to N-1, (c) is a table T, (d) is a sorted table T, (e) is a set V, (f) ) Is a variable U, and (g) is a composite template.

As shown in FIG. 12C, in the table T, each template Φi ′ (p, q) (i ′ <i)
, The cumulative approximation error and the approximation coefficient corresponding to Φi ′ (p, q) are registered in the (i ′ + 1) th row. FIG. 12C illustrates a case where the neighborhood image h (p, q) shown in FIG. 12A is approximated by the template Φi (p, q) in FIG. 12B.

When the cumulative approximation error e is transferred from the approximation error accumulator 612c, the template accumulator 612g stores the approximation coefficient α held in the approximation coefficient holding buffer 612d.
(c) and β (c) (c = 0, 1, 2) are read out and written together with i and e in the (i + 1) th row of the table T.

The least squares coefficient calculator 612b calculates all Φ
When the processing for i (i = 0 to N−1) is completed, the template synthesizing unit 612h configures a new template set in the following procedure using the table T held in the template accumulating unit 612g.

First, N internal variables mark (i) (i = 0 to
N-1) and the counter total_mark are all initialized to 0. Next, the rows of the table T are rearranged in ascending order of the cumulative approximation error as shown in FIG. 12D, and the template index corresponding to the row with the smallest cumulative approximation error is k ′.
And In FIG. 12D, k 'is 3. At the same time, the approximation coefficients corresponding to k ′ are αm (c), βm
(c) (c = 0, 1, 2). Further, mark (k ') is set to 1 and total_mark is set to 1.

Thereafter, the respective rows are read in the order of the rearranged rows, and when the cumulative approximation error is smaller than a predetermined threshold value, the approximate coefficient value of the row is read into variables α (c) and β (c). The absolute values eα and eβ of the differences from αm (c) and βm (c), respectively, are taken. Only when αm (c) and α (c) have the same sign and eα and eβ are smaller than a predetermined threshold, mark (k) is set to 1 using the template index k of the row and total_mark is incremented by 1. When the total_mark reaches the predetermined upper limit M 'or when this processing is completed for all rows, the repetition ends.

When this process is completed, a set V of template indexes k whose mark (k) is 1 is formed as shown in FIG. Integer set V
Are the indices of the template in which the approximation error is similar to the template k ′ in which the cumulative approximation error is the smallest and the approximation error is sufficiently small.

Subsequently, all subsets having a predetermined number of elements are created from V, and variables U (j) (j = 0 to M−1) having each subset as a value as shown in FIG. ). In FIG. 12F, the number of elements of each subset is two. Here, M is the total number of subsets that satisfy the condition. Further, a synthesis template Φ ′ for the variable U (j)
j (p, q) is constructed according to the following equation.

[0223]

(Equation 7)

In this equation, k takes all the integer values included in the integer set U (j). The function max is all k
Calculate the maximum value of the argument for Positive real number λ (j, k) and positive real number μ (j, k) are used to normalize λ (j, k) Φk (p, q) + μ (j, k) to maximum value 1 and minimum value 0. It is a normalized variable.

FIG. 12 (g) shows a composite template Φ'j
(p, q). variable

(Equation 8)

Is stored in the template synthesizing section 612h together with Φ'j (p, q). Here, j = 0 to M−1.

When the processing of the template synthesizing unit 612h is completed, the least squares coefficient calculating unit 612i sequentially converts the template Φ'j (p, q) (j =
0-M-1) and read out the least square coefficient calculating unit 612b.
Similarly, the approximation error is sequentially obtained for each color component c (c = 0, 1, 2) of the neighboring image h (p, q) transferred from the neighboring image buffer 612f.

(Equation 9)

Approximation coefficients α ′ (c) and β ′ (c)
Is calculated, and the approximation error e ′ is added to the approximation error accumulating unit 612j.
(C) is output. The approximation error accumulator 612j accumulates the approximation error e '(c) for all color components and transfers the accumulated approximation error e' = e '(0) + e' (1) + e '(2) to the template determiner 612e. I do. Approximation coefficients α '(c), β'
(c) (c = 0, 1, 2) is the approximate coefficient holding buffer 612
k are sequentially output and held.

The template determining unit 612e has the minimum value min_e ′ of the error with respect to the template Φ′j ′ (p, q) (j ′ <j) and the index given the minimum value.
min_j is held. Template determination unit 612
e is the error e ′ transferred from the approximation error accumulator 612d.
Is compared with the minimum value min_e ′ of the error that holds the inside, and if the new error is smaller, min_e ′ is replaced with e ′,
Update j to j.

The template determining unit 612e also calculates the approximation coefficients related to the template that gives the minimum value of the error by min_α ′ (c) and min_β ′ (c) (c = 0,
1, 2), and at the same time as the update of min_e ', these values are updated to the approximate coefficients α' (c) and β '(c) read from the approximate coefficient holding buffer 612k.

After all the templates Φ′j (p, q) are processed by the least squares coefficient calculating section 612i, the template determining section 612e uses the index min_j to which the minimum value of the error min_e ′ is given to the template synthesizing section 612h. Internal variables of

(Equation 10)

At the same time, the minimum value of the approximation error error min_e '
A ′ = 1 / (1 + min_
e '). And

[Equation 11]

And A 'are transferred to the band / color information restoring section 614.

The band / color information restoring unit 614 stores the pre-deterioration template Ψk (p, q) corresponding to all the integers k included in the transferred integer set U (min_j) in the pre-deterioration template holding ROM 611a. Coefficients read and transferred from

(Equation 12)

Using the following equation, a neighborhood corrected approximate image f (p, q, c) that approximates the image before deterioration of the neighborhood image is obtained from the following equation.

[0236]

(Equation 13)

Then, f (p, q, c) is transferred to the three-plate image adder 615 together with the approximate evaluation value A '.

During the processing, the output image buffer 616
In each pixel position, the average value and the reliability value of a plurality of corrected pixel values obtained by the processing so far are held.
The three-plate image adding unit 615 outputs the average value Sf of the corrected pixel values of the area corresponding to the neighboring image from the output image buffer 616.
(p, q, c) and the reliability value Q (p, q, c) are read out, and the neighborhood correction approximate image f (p, q, c) and the approximate evaluation value transferred from the band / color information restoring unit 614 are read out. From A ′, the average value Sf _new (p, q, c) and the confidence value Q _new (p, q) of the new corrected pixel value are calculated by the equation Q _new (p, q) = Q (p, q) + A ′, Sf _new (p, q, c) = (Q (p, q) · Sf (p, q, c) + A ′ ·
f (p, q, c)) / Q _new (p, q) (c = 0,1,2), and Sf _new (p, q, c) and Q _new (p, q) are output images The data is written to an area of the buffer 616 corresponding to the neighboring image.

When these processes are completed for all the pixel positions, for each pixel of the image in the input image buffer 608, a neighborhood correction approximate image for all the neighborhood images including the pixel is calculated, and for each pixel obtained as a result. Since a plurality of corrected pixel values are weighted and averaged by the processing in the three-plate image adding unit 615 and accumulated in the output image buffer 616, the three colors in which the deterioration and the lack of color due to the optical system of the electronic camera 600 are finally corrected are corrected. A color image is obtained.

If the accuracy of the correction is low, but the memory capacity is reduced and the processing is not speeded up, the image adder 615 may be deleted and the reliability value may not be held in the output image buffer 616. In this case, the band / color information restoring unit 6
14 outputs directly to the output image buffer 616. The effect is that the band / color information restoring unit 614 corrects the center pixel correction value f (p, q, c) of the band / color information corrected f (p, q, c).
(0,0, c) Only the output image buffer 6 (c = 0,1,2)
16 to be output.

The electronic camera 600 has a crystal filter 6.
03 and the band is not limited, the template holding ROM 611 stores the pre-deterioration template holding RO.
M611a only, and the combined least squares approximation unit 612 is connected to the pre-deterioration template holding ROM 611a.

In this case, the operation of each part is the same in the composite least squares approximation unit 612 except that Ψi is used instead of Φi as a template. As a result, the band / color information restoring unit 614 obtains a neighborhood correction approximate image in which the loss of the color component is corrected but the band is not degraded, and the output image buffer 616 stores the image in the input image buffer 608. A three-color image in which the missing color components are corrected is obtained.

In this modified example, it can be easily supposed that the same modification can be attained by performing the same modification as in the modification for increasing the processing speed.

Note that the specific embodiments described above include inventions having the following configurations.

(1) At least one pixel depends on the position of a pixel.
In an image processing apparatus for restoring an original image signal from an image signal in which information relating to one or more spectral characteristics is missing, a nearby image of a predetermined size including a designated pixel is extracted from the image signal in which the information is missing. A nearby image extracting unit; a template holding unit that holds a template group based on a plurality of geometric patterns having the same size as the nearby image extracted by the nearby image extracting unit; and a nearby image extracted by the nearby image extracting unit. Is selected between the information on the plurality of spectral characteristics included in the template and the plurality of template groups held by the template holding unit to select an optimal template most similar to the nearby image. A neighborhood approximation image generating means for producing a neighborhood approximation image approximating Based on the near approximation image generated by the generating means, the image processing apparatus characterized by comprising a, and restoring means for restoring the missing information in the designated pixel.

(2) The restoration means weights and averages information on at least one or more spectral characteristics of a plurality of neighborhood approximate images obtained from a plurality of neighborhood images including designated pixels. The image processing apparatus according to the configuration (1), wherein the restoration processing is performed by:

(3) The template holding means includes a first template group based on a plurality of geometric patterns, and a first template group degraded according to a band limiting characteristic of an optical system of the imaging apparatus. And the near-approximated image generating means selects an optimum template similar to the nearby image from the second template group, and performs band limiting based on the corresponding first template group. The image processing device according to the configuration (1) or (2), which generates the previous neighborhood approximate image.

(4) The restoring means calculates difference information between the near image and the near approximation image, extracts only structural components from the difference information based on a predetermined evaluation function, and obtains contrast information of the structural components. The image processing apparatus according to the configuration (2) or (3), wherein the missing information in the specified pixel is restored by modulating a luminance component of the neighborhood approximate image using the image processing.

(5) The restoring means calculates difference information between the neighboring image and the neighboring approximate image, extracts only structural components from the difference information based on a predetermined evaluation function, and Is the third based on the structural component
And generating a neighborhood re-approximation image approximating the neighborhood image by using the third template group, wherein the restoration unit is configured to generate the specified pixel based on the neighborhood approximation image and the neighborhood re-approximation image. (2) The image processing apparatus according to (2), wherein the missing information is restored.

(6) The template group held in the template holding means is classified for each different resolution, and the restoring means calculates difference information between the near image and the near approximate image, and calculates the difference information. Extracting only the structural component from the information based on a predetermined evaluation function, the neighborhood approximate image generation unit performs matching between the structural component and a template group corresponding to the resolution of the structural component in the template group. Selecting an optimal template, generating a differential neighborhood approximate image that approximates the difference information from the optimal template, and the restoring unit is configured to determine the missing information in the designated pixel based on the difference neighborhood approximate image and the neighborhood approximate image. And a new structural component is extracted from the difference information between the difference neighborhood approximate image and the structural component. The image processing apparatus according to configuration (2), wherein the restoration processing is repeated until the component becomes equal to or less than a predetermined threshold.

(7) The second template group held in the template holding means is classified for each different resolution, and the restoring means calculates difference information between the neighboring image and the neighboring approximate image. Extracting only the structural component from the difference information based on a predetermined evaluation function, the neighborhood approximation image generating means compares the structural component with the template group corresponding to the resolution of the structural component in the second template group. And selecting an optimal template by performing matching between them, generating a differential neighborhood approximate image that approximates the difference information from the optimal template, and the restoring unit is configured to execute the specified neighborhood based on the difference neighborhood approximate image and the neighborhood approximate image. The missing information in the restored pixel is restored, and a new structural component is extracted from the difference information between the approximate difference image and the structural component. The image processing apparatus according to configuration (3), wherein the restoration processing is repeated until the structural component becomes equal to or smaller than a predetermined threshold.

(8) The restoration means calculates spatial correlation information from the neighborhood approximation image, and applies an auto-regression model to the specified pixel based on the spatial correlation information to calculate the spatial correlation information at the specified pixel. The image processing apparatus according to any one of the configurations (1), (2), and (3), which restores missing information.

(9) The neighborhood approximate image generation means performs matching between the neighborhood image and the plurality of template groups, and combines a plurality of templates with small matching errors to generate a single optimal template. The image processing device according to the configuration (1) or (2), wherein

(10) The neighborhood-approximate-image generating means includes:
The image according to the configuration (3), wherein matching is performed between the neighboring image and the second template group, and a plurality of templates having small matching errors are combined to generate a single optimal template. Processing equipment.

(11) The neighborhood approximation image generation means:
A configuration (1) wherein the template group held in the template holding means is multiplied by a constant α, and a constant β is added and deformed to select an optimal template most similar to the neighboring image. (2), (4)-
(6) The image processing device according to (8) or (9).

(12) The neighborhood-approximate-image generation means includes:
The second template group held in the template holding unit is multiplied by a constant α, and a constant β is added and deformed to select an optimal template most similar to the neighboring image ( 3), (7),
The image processing device according to (10).

The correspondence between the configurations (1) to (12) and each of the above embodiments is as follows.

A. Configuration (1) includes at least first to sixth
In the first embodiment, the neighborhood image extracting unit includes the input image buffer 108 and the neighborhood reading unit 109.
Corresponding to The template holding means corresponds to the template holding ROM 111. The neighborhood approximation image generation unit corresponds to the least square approximation unit 112 and the band / color information restoration unit 114. The restoration unit corresponds to the three-plate image adding unit 115 and the output image buffer 116.

In the second embodiment, the neighboring image extracting means corresponds to the input image buffer 208 and the neighboring reading section 209. Template holding means is template holding ROM
This corresponds to 211. The neighborhood approximation image generation unit corresponds to the least square approximation unit 212, the band / color information restoration unit 214, and the color information restoration unit 213. The restoration means is an interpolation coefficient calculating unit 21
5, corresponding to the linear operation unit 216 and the output image buffer 217.

In the third embodiment, the neighborhood image extracting means corresponds to the input image buffer 308 and the neighborhood read unit 309. Template holding means is template holding ROM
311. The neighborhood approximation image generation unit corresponds to the template selection unit 310, the least square approximation unit 312, the band / color information restoration unit 314, and the post-deterioration template processing unit 313. The restoration means includes a three-plate image addition unit 315a, an output image buffer 317, a single-plate image addition unit 315b, a single-plate image buffer 316, a difference calculation unit 318, and a difference image buffer 3
19 and the regularity evaluation unit 320.

In the fourth embodiment, the neighboring image extracting means corresponds to the input image buffer 408 and the neighboring reading section 409. Template holding means is template holding ROM
411. The neighborhood approximation image generation unit corresponds to the least square approximation unit 412 and the band / color information restoration unit 414.
The restoration means includes an image addition unit 415 and a three-plate image buffer 417.
a, the single-plate image buffer 417b, the residual calculation unit 418, the regularity determination unit 419, the residual addition unit 420, and the output image buffer 421.

In the fifth embodiment, the neighborhood image extracting means corresponds to the input image buffer 508 and the neighborhood reading section 509. Template holding means is template holding ROM
511. The neighborhood approximation image generation unit corresponds to the template selection unit 510, the template generation unit 521, the least square approximation unit 512, the color information restoration unit 514, and the single-plate image generation unit 513. The restoration means is a three-plate image addition unit 515
a, single-chip image adder 515b, single-chip image buffer 51
6, the difference calculation unit 518, the difference image buffer 519, the regularity evaluation unit 520, and the output image buffer 517.

In the sixth embodiment, the neighboring image extracting means corresponds to the input image buffer 608 and the neighboring reading section 609. Template holding means is template holding ROM
611. The neighborhood approximation image generation unit corresponds to the composite least square approximation unit 612 and the band / color information restoration unit 614. The restoration means corresponds to the three-plate image addition unit 615 and the output image buffer 616.

For each neighborhood of the image in the single-plate state, the neighborhood approximation image generation means determines one template that is the best approximation in consideration of all the plurality of spectral components, and the restoration means restores the interpolation based on the template. As a result, compared to the case where the luminance component and the color component are handled separately, it is possible to detect minute geometric features included in the vicinity without being affected by moiré.

Thus, the problem P1 to be solved by the invention described below is solved. Also, since it is not necessary to obtain a specific color component at a high sampling density, P
2 is also resolved.

B. Configuration (2) includes at least first to sixth
In the first embodiment, the restoration unit corresponds to the three-plate image addition unit 115 and the output image buffer 116 in the first embodiment.

In the second embodiment, the restoration means comprises an interpolation coefficient calculation section 215, a linear operation section 216, and an output image buffer 2
17 corresponds.

In the third embodiment, the restoration means includes a three-plate image addition unit 315a, an output image buffer 317, a single-plate image addition unit 315b, a single-plate image buffer 316, and a difference calculation unit 31.
8, corresponding to the difference image buffer 319 and the regularity evaluation unit 320.

In the fourth embodiment, the restoration means comprises an image adding section 415, a three-plate image buffer 417a, a single-plate image buffer 417b, a residual calculating section 418, a regularity determining section 419, a residual adding section 420, and an output image buffer 421. Corresponding to

In the fifth embodiment, the restoration means includes a three-plate image addition unit 515a, a single-plate image addition unit 515b, a single-plate image buffer 516, a difference calculation unit 518, and a difference image buffer 51.
9, the regularity evaluation unit 520 and the output image buffer 517.

In the sixth embodiment, the restoration means corresponds to the three-plate image addition unit 615 and the output image buffer 616.

For each pixel of the input image, the neighborhood approximation image generation means generates a neighborhood approximation image for each of a plurality of neighborhoods including the pixel, and the restoration means calculates the approximate value of the pixel in the plurality of neighborhood approximation images. As a result of calculating the final approximate value of the pixel by weighted averaging, the approximation error in each neighborhood is corrected.

As a result, the problem P3 to be solved by the invention described below is solved.

C. At least the first, second, third, fourth and sixth embodiments correspond to the configuration (3). In the first embodiment, the template holding means is a template holding ROM.
111. The neighborhood approximation image generation unit corresponds to the least square approximation unit 112 and the band / color information restoration unit 114.

In the second embodiment, the template holding means corresponds to the template holding ROM 211. The neighborhood approximation image generation unit corresponds to the least square approximation unit 212, the band / color information restoration unit 214, and the color information restoration unit 213.

In the third embodiment, the template holding means corresponds to the template holding ROM 311. The neighborhood approximation image generation unit corresponds to the template selection unit 310, the least square approximation unit 312, the band / color information restoration unit 314, and the post-deterioration template processing unit 313.

In the fourth embodiment, the template holding means corresponds to the template holding ROM 411. The neighborhood approximation image generation unit corresponds to the least square approximation unit 412 and the band / color information restoration unit 414.

In the sixth embodiment, the template holding means corresponds to the template holding ROM 611. The neighborhood approximation image generation unit corresponds to the composite least square approximation unit 612 and the band / color information restoration unit 614.

The template holding means stores a first template group based on a plurality of geometric patterns and a second template group obtained by degrading the first template group in accordance with the band limiting characteristic of the optical system of the imaging apparatus. The neighborhood approximation image generating means selects the most similar optimal template from the second template group with respect to the neighborhood of each pixel of the input image, and based on the corresponding template of the first template group, performs the As a result, an image in which the band degradation of the optical system is corrected is obtained.

This solves the problem P5 to be solved by the invention described below.

D. At least the fourth embodiment corresponds to the configuration (4).

The restoration means corresponds to the image adder 415, the three-plate image buffer 417a, the single-plate image buffer 417b, the residual calculator 418, the regularity determiner 419, the residual adder 420, and the output image buffer 421.

The neighborhood approximation image generation means generates a neighborhood approximation image for each neighborhood of each pixel of the input image, and the restoration means calculates difference information between the neighborhood and the neighborhood approximation image. Is extracted, and the luminance component of the nearby approximation image is modulated using the contrast information of the structural component, so that even if the neighborhood cannot be completely approximated by the template, the component having the regular structure in the approximation error is the luminance component of the output image. Is reflected in

[0284] This solves the problem P7 to be solved by the invention described below.

E. At least the fifth embodiment corresponds to the configuration (5).

The neighborhood approximation image generation means includes a template selection unit 510, a template generation unit 521, a least square approximation unit 512, a color information restoration unit 514, and a single-plate image generation unit 513.
Corresponding to The restoration means includes a three-plate image addition unit 515a, a single-plate image addition unit 515b, a single-plate image buffer 516, a difference calculation unit 518, a difference image buffer 519, and a regularity evaluation unit 5.
20 and the output image buffer 517.

For the neighborhood of each pixel of the input image, the neighborhood approximation image generation means first generates a neighborhood approximation image, and the restoration means calculates a difference image which is an error between the neighborhood and the neighborhood approximation image.
A neighborhood approximation image generation unit calculates a structural component from the difference image based on a predetermined evaluation function to generate a third template group. Next, the neighborhood approximation image generation means approximates the neighborhood again by the third template group, and generates a neighborhood re-approximation image. Finally, the restoration unit performs correction based on the neighborhood approximate image and the neighborhood re-approximation image. As a result, even when the neighborhood includes a regular structure that cannot be approximated by the template of the template holding unit, the structure is extracted from the difference image representing the approximation error and the approximation is performed again.

This solves the problem P7 to be solved by the invention described below.

F. At least the third embodiment corresponds to the configurations (6) and (7).

The template holding means corresponds to the template holding ROM 311. The neighborhood approximation image generation unit includes a template selection unit 310, a least square approximation unit 312,
Color information restoring section 314 and post-deterioration template processing section 31
Corresponds to 3. The restoration means corresponds to the three-plate image addition unit 315a, the output image buffer 317, the single-plate image addition unit 315b, the single-plate image buffer 316, the difference calculation unit 318, the difference image buffer 319, and the regularity evaluation unit 320.

The template holding means has a group of templates classified for different resolutions, and for the vicinity of each pixel of the input image, the neighborhood approximation image generating means converts the resolution of the template from a low-resolution template to a high-resolution template. Since the approximation error is reduced while increasing the approximation, the approximation process can be performed at high speed without preparing a template having a complicated structure.

Thus, the problem P4 to be solved by the invention described below is solved.

G. At least the second embodiment corresponds to the configuration (8).

The restoring means corresponds to the interpolation coefficient calculation section 215, the linear operation section 216, and the output image buffer 217.

A neighborhood approximation image generating means generates a neighborhood approximation image for each neighborhood of each pixel of the input image, and a restoration means calculates spatial correlation information from the neighborhood approximation image. By applying the auto-regression model to and restoring the missing information, the interpolation process is performed in a common procedure regardless of the template selected by the neighborhood approximation image generating means.

Thus, the problem P6 to be solved by the invention described below is solved.

H. At least the sixth embodiment corresponds to the configurations (9) and (10).

The neighborhood approximation image generating means corresponds to the synthesis least square approximation unit 612 and the band / color information restoration unit 614.

The neighborhood approximation image generation means combines a plurality of templates with a small approximation error with the neighborhood of each pixel of the input image to generate a single optimal template.

As a result, the problem P8 to be solved by the invention described below is solved.

I. At least the first to sixth embodiments correspond to the configurations (11) and (12).

In the first embodiment, the neighborhood approximation image generating means includes the least square approximation unit 112 and the band / color information restoration unit 114
Corresponding to

In the second embodiment, the neighborhood approximation image generation means corresponds to the least square approximation unit 212, the band / color information restoration unit 214, and the color information restoration unit 213.

In the third embodiment, the neighborhood approximation image generation means includes a template selection unit 310, a least square approximation unit 312,
It corresponds to the band / color information restoring unit 314 and the post-deterioration template processing unit 313.

In the fourth embodiment, the neighborhood approximation image generating means includes the least square approximation unit 412 and the band / color information restoration unit 414
Corresponding to

In the fifth embodiment, the neighborhood approximation image generation means includes a template selection unit 510, a template generation unit 52
1, corresponding to the least square approximation unit 512, the color information restoration unit 514, and the single-plate image generation unit 513.

[0307] In the sixth embodiment, the neighborhood approximation image generation means includes the synthesis least square approximation unit 612 and the band / color information restoration unit 6
14 corresponds to.

For the neighborhood of each pixel of the input image, the neighborhood approximation image generating means firstly multiplies the template by a constant α for each color component in the neighborhood, and approximates the color component by a modified template obtained by adding the constant β. The optimal coefficients α and β are determined. Next, an approximation error due to the deformed template when using the coefficients α and β that provide the optimum approximation for each color component is calculated as a single-color approximation error, and the approximation of the neighborhood using the template is performed based on the single-color approximation errors of all color components By evaluating the degree, it is possible to approximate a plurality of color components with one template.

[0309] This solves the problems P1 and P2 to be solved by the invention described below.

The problems to be solved by the inventions solved by the above constitutions (1) to (12) are as follows.

P1 In the prior art, there is missing information even in a luminance component provided at a relatively high sampling density, and only information that is deteriorated compared to the luminance component obtained by the three-plate system can be obtained. Also, other color components calculated based on the luminance component can obtain only information that is deteriorated as compared with the color components obtained by the three-plate system.

P2 The prior art does not describe a method for processing an input image having an arbitrary color filter arrangement. In the interpolation of the luminance component, it is necessary to prepare a geometric feature prepared in advance and an interpolation method corresponding thereto, and it is necessary to change the design when the color filter arrangement changes.

P3 The prior art does not describe a method for coping with a malfunction caused by the influence of noise. When processing luminance components and other color components, interpolation is performed in a single process, so that it is easily affected by pulse noise. In particular, in the processing of the luminance component, since the geometric features are classified in a 3 × 3 local area, erroneous classification due to noise is likely to occur.

P4 In the prior art, when the size of a region for determining a geometric feature is increased in order to reduce the influence of noise, the determination cannot be performed by high-speed processing.

P5 The prior art does not describe the recovery of information whose band is limited by the optical system of the imaging device. The luminance component and the color components of other colors are processed within the band-limited range, and band recovery cannot be handled.

P6 In the prior art, a plurality of interpolation methods are prepared according to geometrical features, and no description is given for a single method.
When the luminance component is interpolated based on geometric features, interpolation methods must be prepared for the number of geometric features.
For high-precision processing, it is necessary to increase the number of geometric features, but the interpolation method must be considered accordingly.
Processing becomes complicated.

P7 In the prior art, there is no description about a geometric feature prepared in advance and a countermeasure against a case where it does not apply to the corresponding interpolation method. When a luminance component is classified according to a geometric feature, since the classification is forcibly classified into any of the geometric features, an error occurs in a region that does not apply, and high-precision processing cannot be performed. .

P8 In the prior art, high-precision processing cannot be performed unless a geometric feature prepared in advance and an interpolation method corresponding thereto are applied.

[0319]

According to the first aspect of the present invention, it is possible to obtain a higher definition image than before and to process an input image having an arbitrary color filter arrangement.

According to the second aspect of the present invention, in addition to the effects of the first aspect of the present invention, it is possible to reduce deterioration of the image signal due to noise.

According to the invention of claim 3, in addition to the effect of the invention of claim 1 or 2, information lost due to band limitation can be recovered.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an image processing system to which a first embodiment of the present invention is applied.

FIG. 2 is a diagram illustrating a configuration of a least squares approximation unit illustrated in FIG. 1 and peripheral circuits thereof;

FIG. 3 is an explanatory diagram of a mask generated by a color component mask generation unit.

FIG. 4 is a diagram illustrating a configuration of an image processing system to which a second embodiment of the present invention is applied.

FIG. 5 is a flowchart illustrating a processing flow of a least squares approximation unit 112 in FIG. 4;

FIG. 6 is a diagram showing a configuration of an image processing system to which a third embodiment of the present invention is applied.

FIG. 7 is a diagram illustrating a configuration of an image processing system to which a fourth embodiment of the present invention is applied.

8 is a diagram showing a configuration of a least squares approximation unit shown in FIG. 7 and its peripheral circuits.

FIG. 9 is a diagram illustrating a configuration of an image processing system to which a fifth embodiment of the present invention is applied.

FIG. 10 is a diagram illustrating a configuration of an image processing system to which a sixth embodiment of the present invention is applied.

11 is a diagram showing a configuration of a composite least squares approximation unit shown in FIG. 10 and its peripheral circuits.

FIG. 12 is a diagram for explaining the operation of a composite least squares approximation unit.

[Explanation of symbols]

100: electronic camera, 101: image processing device, 102:
Lens: 103: quartz filter, 104: single-chip RGB random array CCD, 105: A / D converter, 106: ROM for holding filter arrangement information, 107: image buffer, 1
08: input image buffer, 109: neighborhood read unit, 1
11: template holding ROM, 112: least square approximation unit, 114: band / color information restoration unit, 115: three-plate image addition unit, 116: output image buffer

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] January 19, 1998

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] 0037

[Correction method] Change

[Correction contents]

On the other hand, the least-squares coefficient calculating unit 112b calculates the template Φ of the deteriorated template holding ROM 111b.
The template Φ0 (p, q) is read from i (p, q) (i = 0, 1, 2,..., N−1), and each color component c generated by the color component mask from the transferred neighboring image h (C = 0,1,
Error e0 (1), e0 (2), e0 (3)
The approximation coefficients respectively minimize α0 (c), β0 (c ),
General expression e (c) = approximation error = ‖mask (p, q, c) · (α (c) Φi (p, q) + β (c)
Calculate for each color component using −h (p, q) ‖ . The least squares coefficient calculation unit 112b calculates the calculated approximate coefficients α0 (c), β0
(C) together with the neighboring image h and the approximate coefficient holding buffer 112
d and minimized approximation errors e0 (1), e0
(2), e0 (3) and the corresponding template index i = 0 are sent to the approximation error accumulator 112c.

[Procedure amendment 2]

[Document name to be amended] Statement

[Correction target item name] 0040

[Correction method] Change

[Correction contents]

Next, the least-square coefficient calculator 112b reads the second template Φ1 (p, q) and repeats the above steps to obtain the index i = 1 of the template Φ1 (p, q) and this template Is obtained. Template determination unit 112e
Compares the cumulative addition error e0 with the cumulative addition error e1,
The value with the smaller error is stored in the storage unit together with the corresponding index. As for the approximation coefficient, the approximation coefficient of the template that generates a small cumulative addition error is stored in the storage unit. Thereafter, the third template Φ2 (p, q), the fourth template Φ3 (p, q),.
The cumulative addition errors e2, e3,... Are obtained and compared with the values of the cumulative addition errors stored in the sequential storage unit, and the smaller one is left in the storage unit together with the index and the approximation coefficient. A template that minimizes the addition error is determined.

Claims (3)

[Claims] 1. An image processing apparatus for recovering an original image signal from an image signal in which at least one or more information on spectral characteristics is missing according to a position of a pixel, wherein the information is specified from an image signal in which the information is missing. Image extracting means for extracting a nearby image of a predetermined size containing the extracted pixels, and a template holding means for holding a template group based on a plurality of geometric patterns of the same size as the nearby image extracted by the nearby image extracting means And performing matching between information on a plurality of spectral characteristics included in the nearby image extracted by the nearby image extracting unit and a plurality of template groups held by the template holding unit to be most similar to the nearby image. Select the optimal template and generate a neighborhood approximation image that approximates the neighborhood image based on this optimal template. An image comprising: a neighborhood approximation image generating unit that forms the image; and a restoration unit that restores missing information in the designated pixel based on the neighborhood approximation image generated by the neighborhood approximation image generation unit. Processing equipment. 2. The method according to claim 1, wherein the restoration unit performs weighted averaging on information on the at least one or more spectral characteristics of a plurality of the neighboring approximate images obtained from the plurality of the neighboring images including a designated pixel. 2. The image processing apparatus according to claim 1, wherein a restoration process is performed. 3. The template holding means includes: a first template group based on a plurality of geometric patterns; and a second template group degraded according to a band limiting characteristic of an optical system of the imaging apparatus. The neighborhood approximate image generation means selects an optimal template similar to the neighborhood image from the second template group, and performs band limiting before the band limitation based on the first template group corresponding thereto. The image processing apparatus according to claim 1, wherein a neighborhood approximation image is generated.