JPH09214988A - Still image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image pickup device from which a still image with high quality is obtained at a low cost without a new complicated device configuration by picking up the same object with plural different magnification factors, converting the images into electric signals and applying image processing to the signals. SOLUTION: A microscope section 314 is provided with plural objective lenses 301 selected by a revolver 302, and a camera box or the like containing an RGB Bayer arrangement CCD 305. The RGB Bayer arrangement CCD 305 is placed at a focal position of the eyepiece lens 303 and converts images picked up with plural magnification factors as to the same object through the use of the objective lens 301 into electric signals, which are stored in the image memory 311 via the system bus 309 or the like in the main body 315. The images with plural magnification factors stored in the image memory 311 are composited so as to generate a still image with higher resolution than that from the image at a highest magnification factor among plural different magnification factors.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a still image pickup device, and more particularly, in converting an image of a subject optically formed into digital image information, high quality and high color reproducibility exceeding the resolution of an image pickup device are required. Still image pickup device.

[0002]

2. Description of the Related Art Various methods have been conventionally developed in order to obtain a high-quality still image exceeding the resolution of an image pickup device. (1 / N pixel) to vibrate (JP-A-58-111580), (2) to optically shift the image forming position by half a pixel (JP-A-61-270973), and (3) a plurality of Image sensors arranged with their half-pixel positions displaced, (4) Two image sensors are arranged at displaced positions, the deviations of the images captured by each are determined by correlation, and samplings that are displaced from each other are combined. To obtain a high-resolution image ("High-definition image input method using multiple cameras with different pixel apertures", Technical Report of the Institute of Television Engineers of Japan Vol.17, No.29, PP13-18, 1993), etc. Are known.

Further, when an image is picked up using an image pickup device having a mosaic filter, as a method of restoring missing color information and improving color reproducibility, (5) missing color components are values of pixels of the same surrounding components. It is known to interpolate from, or (6) investigate the correlation between color components and estimate the missing color component from other color components.

[0004]

However, in the methods described in (1) and (2) above, a uniform sampling interval shift is created, so that the synthesizing process is facilitated, but the sampling interval shift is reduced. It requires a new dedicated device with a complicated structure to create. Also, (3)
Like the method described in (1) and (2), the method described in (1) is easy to combine, but it is extremely difficult to accurately arrange the image pickup elements so that a half pixel can be displaced between the image pickup elements. It's difficult.

In the method described in (4), since the deviation amount of the sampling interval is not set in advance, the device configuration is simple and there is no problem in the placement accuracy, but conversely, the calculation for finding the corresponding points becomes complicated and the synthesizing process is performed. The time will increase significantly.

The methods described in (5) and (6) are
Since the missing color component value is obtained from one image, the resolution is lowered by the interpolation process, or inconvenient coloring occurs when the assumption of interpolation or correlation does not hold.

The still image pickup device of the present invention has been made in view of the above problems, and its first purpose is to:
It is an object of the present invention to provide a still image capturing apparatus that can obtain a high quality still image that exceeds the resolution of an image sensor at low cost without newly adding a complicated apparatus configuration.

A second object of the present invention is to provide a still image pickup device which can obtain a high quality still image which exceeds the resolution of the image pickup device by a simple combining process.

A third object of the present invention is to provide a still image pickup device capable of restoring a missing color without newly using a complicated device structure.

[0010]

In order to achieve the above-mentioned first object, a still image pickup device of the present invention comprises an image-forming variable magnification means for forming an image of the same subject at a plurality of different variable magnifications. Imaging means for converting a plurality of scaled images of the same subject, which have been scaled by the imaging scaling means, into electric signals, respectively, and the plurality of scaled images converted into electric signals by the imaging means. And a plurality of scaled images stored in the image storage means are combined to have a higher resolution than the scaled image obtained at the highest magnification of the plurality of different scales. And an image synthesizing unit for generating an image.

In order to achieve the above-mentioned second object, the image synthesizing means combines the scaled image obtained at the highest magnification among the plurality of scaled images with another scaled image. Magnification control means for controlling the image-magnification variability means so that the relative magnification between them becomes a predetermined rational number such that the sampling points of both variegated images do not completely match, and this magnification-variation control means. Associating means for associating the pixel values of the plurality of scaled images obtained under the control of (1) with the pixel values of the image of higher resolution than the scaled image obtained at the highest magnification, and the associating means. And an image restoration unit that restores an image having a higher resolution than the scaled image obtained at the highest magnification, based on the association by. Further, in order to achieve the above-mentioned third object, the image pickup means has a color mosaic filter, and the predetermined rational number is a fraction of an integer.
It is.

That is, the still image pickup device of the present invention is
A plurality of scaled images of the same subject obtained by scaling by the image-forming scaling unit that images the same subject at a plurality of different scaling factors are converted into electric signals by the image capturing unit,
The plurality of scaled images each converted into an electric signal are stored in the image storage means. Then, the stored plurality of scaled images are combined, and an image having a higher resolution than the scaled image obtained at the highest magnification among the plurality of different scales is generated by the image combining means.

At the time of image synthesis, the relative magnification between the scaled image obtained at the highest magnification of the plurality of scaled images and the other scaled images is determined by the sampling points of these scaled images. Are controlled by the scaling control means so that the predetermined rational numbers do not completely match, and the plurality of scaling images obtained under the control of the scaling control means are controlled. The pixel value and the pixel value of the image having a higher resolution than the scaled image obtained at the highest magnification are associated by the associating means, and based on the association by the associating means, at the highest magnification. An image having a resolution higher than that of the obtained scaled image is restored by the image restoration means.

Further, a color mosaic filter is added to the image pickup means, and the scaling ratio is controlled so that the predetermined rational number becomes 1 / integral.

[0015]

DETAILED DESCRIPTION OF THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings.

First, the outline of this embodiment will be described. First
As an outline of the above, the principle of resolution enhancement by combining images with different scaling factors will be described with reference to FIGS. For simplicity, the image sensor is considered as a one-dimensional line sensor, but the same principle can be easily extended to a two-dimensional area sensor.

With the image sensor and the object to be photographed fixed, the same subject can be magnified at various magnifications (× 1.0, × 0.75, ×).
It is assumed that a plurality of magnified images E, F, and G as shown in FIG. Here, the range common to all the magnified images is called the standard angle of view. On each of the scaled images, the standard angle of view has a different size according to the scale of magnification as shown by the dotted line frame in FIG. 1A, but corresponds to the standard angle of view of each of the scaled images. The range is called a partial standard image, and the largest partial standard image is called a standard image. Also,
The magnification of each variable-magnification image with respect to the magnification that gives the standard image is referred to as the variable magnification (relative magnification). FIG. 1A shows a standard image E and
A scaled image F with a scale factor of 3/4 (0.75) and a scale factor of 3 /
5 shows a scaled image G of 5 (0.6).

These images can be obtained, for example, as shown in FIG. 1B, by vertically scanning an object to be photographed by an image pickup device in which photosensors are regularly arranged in a line.
At this time, a portion corresponding to the optical sensor is called an opening p. Further, the openings p are arranged side by side at a constant interval, and this interval is defined as an opening interval (repeating unit) q. At this time, p / (p + q) is called an aperture ratio. Furthermore, it is assumed that the sensitivity of the device is constant (rectangular aperture) within the aperture.

Considering the horizontal line Z on the standard image E shown in FIG. 1A, the linear region corresponding to the horizontal line Z is linearly illuminated as shown in FIG. When the image is taken by the sensor, the imaging output E ′ as shown in FIG.
F'and G'are obtained. In the figure, the horizontal axis represents the horizontal spatial coordinates (not the coordinates on the element), the curve represents the actual luminance distribution in the linear region, and the dotted line represents the sensitivity of the image sensor. The part where the sensitivity is not 0 corresponds to the aperture. In this description, the sensitivity pattern for the spatial coordinates, which is determined by the arrangement of the linear photosensors and the photographing magnification, is called a sampling pattern. Therefore, if the luminance distribution of the object to be photographed and the sampling pattern are integrated, a pixel value corresponding to each aperture in the pattern can be obtained.

The arrow in FIG. 1C represents the center position of the aperture in each sampling pattern, and the size represents the pixel value obtained by sampling. The sampling patterns of the scaled images F and G are scaled by the reciprocal of the scaling ratio with respect to the sampling pattern of the standard image E. The sampling pattern of the standard image E is called a standard sampling pattern, and the sampling pattern scale-multiplied corresponding to the scaling factors of the scaled images F and G is called a scaling sampling pattern.

Therefore, assuming that the aperture is represented by the δ function, combining these different sampling patterns produces a combining result as shown in FIG. 1 (d). From the figure, in the shaded area only, a sampling pattern that is denser than the sampling pattern of the standard image E obtained by shooting at the highest magnification among the plurality of magnifications described above can be obtained. An image with a higher resolution than the scaled image obtained at the highest magnification (magnification of the standard image x 1.0) out of the different magnifications (x 1.0, x 0.75, x 0.6) is generated. , A high-quality still image that exceeds the resolution of the image sensor can be obtained.

Here, if the aperture interval becomes 1 / n, the band obtained by the sampling theorem becomes n times. Based on this theorem, most of the conventional high-resolution techniques newly use a dedicated device to create a uniform sampling deviation, and move the generated imaging plane in parallel to uniformly reduce the sampling interval and to achieve uniform sampling. Was increasing the bandwidth. However, high resolution can also be achieved by estimating and restoring the pixel values obtained by a finer sampling pattern from the pixel values obtained by a certain sampling pattern. In this case, it does not matter if the sampling intervals are non-uniform. Absent. The method described in (4) as the conventional technique is
Although such a non-uniform sampling interval is created using a plurality of cameras, it requires complicated processing such as obtaining a correlation of sampling between images by correlation.

Therefore, in the first outline, by using a simple mechanism of zooming, it is possible to realize high resolution of an image without newly requiring a complicated device configuration and even if the deviation of sampling intervals is not uniform. There is.

The operation of the image pickup apparatus in this case is as follows. That is, when the subject and the angle of view are selected, the image-forming variable-magnification means changes the size of the subject image at a plurality of magnifications so that the standard angle of view becomes equal to or larger than the designated angle of view. The photographing means performs photographing every time the image-magnification varying means sets the magnification, and the obtained plurality of magnified images of the same subject are stored in the image storage means. The synthesizing unit synthesizes the plurality of scaled images stored in the image storage unit to generate an image having a higher resolution than the scaled image obtained at the highest magnification of the plurality of different scales.

The second outline of this embodiment will be described below.

The resolution can be increased by the above-described scaling by calculating from the magnification where the sampling point comes as a result of the scaling. However, if the regularity of the change of the sampling points due to such scaling is used. As described below, a higher resolution can be realized.

How the sampling points change due to the magnification change depends on how the photographing magnification is selected. First, FIG. 2 shows the case where a rational number is selected as the relative magnification.
This will be described with reference to (a) to FIG. 2 (d). FIG. 2A shows the luminance distribution of the photographing target, and FIGS. 2B and 2C.
Are sampling patterns of a standard image and a variable-magnification image with a variable magnification of 2/3 (= 0.666), respectively, obtained by an image sensor with an aperture ratio of 2/3.

In FIG. 2D, the aperture ratio is 1 and the aperture spacing is as shown in FIG.
The sampling pattern of 1/6 of the aperture interval of (b) is shown. As in FIG. 1, the vertical axis represents the pixel value. Here, it is assumed that the edge of the aperture of the image sensor is exactly at the position corresponding to the optical axis. Hereinafter, FIG.
The sampling patterns shown in (c) and (d) will be referred to as A, B, and S, respectively. In addition, a in FIG.
1, a2, a3, b1, b2 of FIG. 2 (c), FIG. 2 (d)
S1 to s18 represent pixel values obtained by sampling the imaging target having the brightness distribution shown in FIG. 2A at each sampling interval, and the arrow in the brightness direction represents the size and the center position of each pixel. ing.

Here, paying attention to the end of the opening of each sampling pattern, (1) the left end T1 of the opening of the fourth pixel of the sampling pattern A and the left end T2 of the opening of the third pixel of the sampling pattern B are completely. On the right side from this position, the pattern of the deviation of the sampling intervals of A and B from the optical axis to T1 or T2 is repeated.

(2) The ends of the openings of the sampling patterns A and B always coincide with the ends of the openings of the sampling pattern S.

Therefore, by utilizing the properties of (1) and (2), the value of each pixel of the sampling patterns A and B can be expressed by the sum of the values of the pixels of the sampling pattern S, for example, as shown in FIG. A1 in) is s1 to s in FIG.
4 can be expressed by the formula (s1 + s2 + s3 + s4) / 4. If this is generalized, due to the periodicity of (1), the pixel values of the sampling patterns A and B in this cycle can both be represented by the pixel values of the sampling pattern S included in the cycle. This is expressed in the form of a matrix in FIG. 2 (e). FIG.
The * mark in (e) represents multiplication.

Generally, the scaling ratio of the sampling pattern A of the standard image and the sampling pattern B of the scaled image is 1:
When n / m (n and m are positive integers and n <m), the scale relationship of the sampling intervals is 1: m / n. That is,
After sampling m pixels for A and n pixels for B, the sampling positions of both images overlap, and the interval up to that point becomes the unit of the cycle. Furthermore, the sampling patterns A and B are p /
When obtained with an element having an aperture ratio of (p + q) (p and q are positive integers), “W = n (p + q) / {np, nq, mp,
, the same property as (1) and (2) holds, and an image with an aperture ratio of 1 and an aperture interval of “1 / W” at the same magnification as that for obtaining the sampling pattern A is obtained. If the sampling pattern by the element is selected as S, S
It is possible to combine the pixel value obtained by the above with the pixel values of A and B, and the relationship between the two is

[Equation 1]

Is represented by Where Δa = W × p
/ (P + q), Δb = Δa × m / n, s _{1 to mW} are pixel values of S for one period, and a _i and b _j are i and j pixel values of A and B, respectively. In the example of FIG. 2, W = 6, Δa
= 4 and Δb = 6.

The S thus determined is called a common sampling pattern, the aperture interval of the common sampling pattern S is called a common sampling interval, and the pixel value obtained from the common sampling pattern S is called a common sampling value.
Also, the relational expression between the common sampling values and the pixel values of the scaled image expressed in a matrix in the form of FIG. 2E corresponds to the common sampling expression, and the number of common sampling values appearing in this common sampling expression corresponds to the common sampling period. The matrix is called a common sampling matrix. The common sampling matrix is
It shows which common sampling value is used to represent each pixel value of the standard image and the scaled image, and the element is only 0 or 1.

In this way, when the scaling factor is a rational number, the relation between the pixel value of the scaled image and the pixel value obtained by more dense sampling is represented by the equations (1a) and (1b). Since this can be derived in a simple form, the resolution can be increased by solving this relational expression in reverse and obtaining s from a and b. According to the common sampling, the pixel values obtained by different samplings from a plurality of scaled images can be made to coincide with the same common sampling value, so that the amount of information can be substantially increased. Further, since the expression by the common sampling value has a relationship within the cycle of the deviation of the sampling interval, it becomes possible to divide the image into blocks in each cycle unit and process each independently, so that the processing efficiency can be improved.

The common sampling relational expression exemplified in the equation (1) regarding the partial areas of a plurality of scaled images can be generalized as follows.

That is, the relative magnification (magnification) of a partial standard image cut out from a plurality of variable-magnification images with respect to the standard image is set to {1, S1, S2, ..., Sn-1}, which is common to the common sampling period. When the common sampling relational expression of each sampling is obtained based on the sampling expression matrix, g _i = H _i f (i = 1 ... n) (2) can be derived. Here, g _i is an image value of each scaled image, and f is a common sampling value. A matrix that increases the amount of information by combining all of these relationships

[Equation 2]

Then, f can be obtained by solving

The operation of the image pickup apparatus in this case is as follows. That is, the scaling factor determining means selects a scaling factor of a predetermined rational number, and drives the image forming scaling means and the imaging means in accordance with the scaling factor to store the image of the specified scaling factor in the image storage means. Next, the image cutout unit reads out a partial image corresponding to the area of the standard image from the scaled image in the image storage means.
The associating unit associates the plurality of cut-out partial scaled images with the sampling expression matrix determined by the scaling factor of the scaling factor determination unit, and derives a common sampling relational expression as exemplified in Expression (3). . The image restoration means calculates and estimates the common sampling value based on this relational expression. For this purpose, it is desirable to use an estimation method as described in the third to seventh outlines to be described later, but especially when aperture correction is possible, the method described in the above-mentioned prior art (4) is used. May be.

By making the scaling ratio a rational number as described above, it is possible to efficiently and accurately obtain pixel values in block units at common sampling intervals smaller than any of the sampling intervals of the obtained image.

By the way, in the case where the scaling factor is a rational number, the center positions (centers of scaling, corresponding to the optical axis) of the partial standard images (scaled images) are the same, or even if there is a position shift, the optical axis is changed. It is premised that certain conditions such as fixed position are met. However, when the present apparatus is applied to an electronic camera with camera shake, such a condition does not hold. Therefore, first, it is determined whether or not a specific condition is satisfied, and if not satisfied, the scaling factor is set to an irrational number and sampling processing is performed as described in an eighth outline described later.

The operation of the image pickup apparatus in this case is as follows. That is, when the subject and the angle of view are selected, the image-forming variable-magnification means changes the size of the subject image at a plurality of magnifications so that the standard angle of view becomes equal to or larger than the designated angle of view. The photographing means performs photographing every time the image-magnification varying means sets the magnification, and the obtained plurality of magnified images of the same subject are stored in the image storage means.
The synthesizing unit distinguishes the restoration processing for the scaled image according to the situation. That is, in some cases, efficient sampling processing is performed by utilizing the regularity of the shift between the scaled images, and when it is not possible due to the positional shift, for example, as shown in the above-mentioned prior art (4). In addition, a known uniform sampling process from random sampling is applied to estimate and restore an image obtained when images are taken at a closer sampling interval in a region within the standard angle of view.

The third outline of this embodiment will be described below. In the third outline, a color mosaic filter is added to the photographing means, and a scaling factor of 1 / integral ratio (integer zoom-out rate) is adopted as the scaling factor. In particular, if there is only a discontinuous scaling mechanism, a scaling factor of an integer is easier to realize than a general rational number. However, when values can be obtained for all pixels of the image pickup means, the sampling points of another image do not come between the sampling points of one image between the scaled images, so that the sampling positions do not have a complementary relationship. , I cannot get new information. On the other hand, when the sampling is not performed on a pixel-by-pixel basis and the values relating to the primary color components RGB are only partially obtained, as in the case of an image pickup unit to which a color mosaic filter is added, a scaling factor of an integer fraction is obtained. However, the sampling positions are complemented by the scaling.

FIG. 3 is a diagram for explaining how missing pixels are estimated in the sampling relationship between the scaled images when the image pickup device (CCD) is arranged in the Bayer array. Here, in the RGB Bayer array CCD, the missing pixel for the G component of the image having an unknown value at the standard magnification is estimated by the variable-magnification image with the variable magnification of 0.5.

Image G in image (a) at standard magnification
Focusing on the component, the G component is the pixel G2, G4, G5, G
7, G10, G12, G13, G15 can only be obtained,
G1 and G3 are missing pixels. Here, in the image (a) at the standard magnification, for example, the pixel G3,
Focusing on G4, G7, and G8, a composite of these corresponds to the g2 component of the scaled image (b) with a scaling ratio of 0.5, and the relationship of G3 + G4 + G7 + G8 = g2 is established. Solving this for the missing pixels G3 and G8,
G3 + G8 = g2-G7-G8. This means that the missing pixels G3, G8 in the standard image can be estimated with higher accuracy by combining the scaled images, i.e. by adding constraints, as shown in FIG.
(C) shows how information about the missing pixel of the G component in the standard image (a) is supplemented by the g2 and g3 components of the scaled image.

The fourth outline of this embodiment will be described below.

As explained in the second outline, increasing the resolution with respect to the scaling of rational numbers means solving equation (3). However, the equation (3) is generally over- or under-conditioned, and f cannot be uniquely determined. There are several methods to uniquely determine f in such a case, but as is well known here, some constraint is added to the equation (3),

(Equation 3)

We will choose a technique that minimizes In the equation (4), Q and R are discrete expressions of the constraint condition, and γ is a parameter indicating the strength of the constraint. This method is very effective for images that are statistically characterized by some property, and by selecting Q and R well for general images and very special images, Can respond optimally. The given constraint is estimated based on the least square method under the constraint condition that reflects the property of the subject as the most general purpose. For example, the brightness component of the grayscale image that does not change abruptly above a certain rate, such as the expression (18) described later, or the color difference component such as the expression (14) described later spatially very slowly. Only things that change can be mentioned.
The solution to this problem is

(Equation 4)

However, the strength of the constraint can be controlled by adjusting the magnitude of the constant γ. In equation (5),
The matrix for G is called the restoration matrix.

Therefore, the image restoration means is given the discretization matrix Q for the given constraint, and the restoration means calculates the equation (5) using Q to estimate the common sampling value.

The fifth outline of the present embodiment will be described below.

In the equation (5) in the fourth outline, depending on how to select the value of the parameter γ representing the strength of the constraint,
Common sampling values may not be restored well.
For example, assuming that the smoothness of the restored image is a constraint condition as shown in equation (18) described later, if γ is too large, the image quality of the restored image deteriorates. Therefore, here, the parameter γ that determines the constraint condition is updated by the scaled image and the high-resolution image after image combination. That is, by using a part of the plurality of scaled images obtained by the image restoration means to perform a restoration process by selecting a value of γ and comparing the obtained result with the results of the remaining scaled images. , So that we can choose an appropriate γ that allows a reconstruction that fits well in all scaled images.

First, a scaled image is obtained with _n scale factors {S ₁ ... S _n }, and among them, {S _p1 , S _p2 , ... S _pm (p1 ... p
For m, f is restored using m (<n) appropriate natural numbers of images that do not overlap. From this f, the remaining {S _q1 ,
About S _q2 , ... S _qn-m }

(Equation 5)

[Mathematical formula-see original document] is calculated, and γ that minimizes this is determined.

The sixth outline of the present embodiment will be described below. In the sixth outline, the image-magnification scaling means scales only in one direction parallel to the sensor array of the image-capturing means, and the image restoration means partially converts high-resolution pixel values obtained at a closer sampling interval. Restore without error. That is, in the fourth outline, the least-squares approximation is directly applied to the formula (3), but when the relational expression is one-dimensional, the common sampling value may be partially obtained without an error. I am using this.

For example, FIG. 4A shows the sampling relationship between a standard image with an aperture ratio of 1 and a 3/4 × zoom-out (magnification) image, where the pixel value of the standard image is g and the zoom-out is Let h for the image and f for the common sampling value, then the common sampling representation is

(Equation 6)

Is as follows. Here, ff4 = h1-g1, f
f56 = g2-ff4, ff78 = h2-ff56, f
Using the new variable f9 = g3-ff78, this relationship becomes

(Equation 7)

It can be written that f4 and f9 have only one 1 in the row, so it can be determined directly from ff4 and ff9 without error. Since the calculation between the pixel values of the scaled image is addition / subtraction between rows of the common sampling matrix, if addition / subtraction is performed as in this example, and finally there is only one 1 in each row, which sampling value is You can see if it can be uniquely calculated without error.

Such a modification is an operation of adding and subtracting only (basic modification) so as not to output values other than 0 and 1.
This can be achieved by repeating the above so that the number of 1's becomes the smallest. This procedure can be realized by, for example, the flow of FIG. 9 of the second embodiment described later, but has the effect of removing the redundancy included in the common sampling matrix. Simplifying equation (3) in this way, the common sampling expression relational expression

(Equation 8)

I will call it. {P1, p2, ... pm} indicates the position where the common sampling value is completely obtained, and v ₁
... v _m , u ₁ ... u _n are pixel value vectors g of the scaled image
_It is a value created by addition and subtraction of the elements of _i (i = 1 ... N), and the former is the value of f _p1 ... F _pm . f _q1 ... F _qn are common sampling values that cannot be obtained directly, and are associated with u through Hs. Then, an appropriate inverse estimation method may be used for this relational expression. For example, if the method described in the fourth outline is applied after this, when f _p1 ... F _pm are known, the constraint condition of equation (5) is

[Equation 9]

Since Qs, Rs and Hs in the equation (9) are respectively set as Q, R, H in the equation (5), the remaining unknowns f _q1 ... F _qn can be estimated.

Utilizing this fact, the image restoration means (3)
When an equation is given, the above-described basic deformation is performed according to a flow shown in FIG. 9 described later, or the deformation process is executed in advance and the common sampling value {f _p1 ... _pm } will be requested first. After that, the undetermined value is restored by using the estimation method and the like described in the fourth outline.

The seventh outline of the present embodiment will be described below.

The transformation procedure excluding the redundancy of the sampling expression matrix described in the sixth outline can be calculated in advance by the flow of FIG. 9, which will be described later, if the combination of scaling factors to be used for synthesis is determined in advance. The number of 1s in each row of the transformation result is
It represents how many corresponding sampled values are associated with at least the number of other sampled values.
It is considered as a measure of local resolution, and the smaller the better (1
If so, uniquely obtained). Therefore, for each common sampling value, the reciprocal of this number is referred to as a resolution pattern.

FIG. 4B shows the scaling factors {1, 3 respectively.
/ 4} and (1, 3/5} are calculation examples of the resolution pattern, and the white portion 801 is 1 (uniquely determined) and the gray portion 802 is the common sampling value of 1/3 unit of the pixel width. Indicates 1/2, and the black portion 803 indicates 1/3 (resolution does not increase at all) In this way, when the combination of the scaling factors is changed, the cycle and phase of the high resolution position are variously changed. A resolution pattern can be created.

Utilizing this fact, here, what kind of resolution pattern occurs for various combinations of scaling factors is stored in advance in the resolution pattern storage unit. Then, when capturing an image, the frequency analysis means performs a frequency analysis of the standard image to identify a part in the image containing a lot of local high frequency components, and the scaling factor determination means sets the resolution pattern as close as possible to the resolution pattern. A set of scaling factors having a pattern is selected from the resolution pattern storage means.

The eighth outline of this embodiment will be described below.

In the eighth outline, an irrational number is selected as the scaling factor. When an irrational number is selected as the scaling factor, the two sampling intervals are shifted indefinitely and the periodicity as in the case of rational numbers is not seen. Moreover, the magnitude of the shift is not random but is a series of slightly different patterns.

In this case, the image pickup means (1) performs image pickup at an appropriate magnification and timing by the image formation magnification varying means, or (2) the magnification of the image prestored in the image storage means is unreasonable. Only images that have a numerical relationship are imaged. The obtained plurality of images are stored in the image storage means. In the case of (2), the image synthesizing means limits the scaling ratio to an irrational number when the images used for the synthesizing are read from the image storage means. , And the overlap of sampling intervals is reduced as much as possible, and a high-resolution image is obtained by using a known high-resolution method such as a sampling insertion method.

The ninth outline of this embodiment will be described below. In the ninth outline, the image synthesizing means has a detecting means for detecting a positional deviation amount between a plurality of scaled images (a central position corresponding to the optical axis between the scaled images), and the image synthesizing means is a center of the standard image. The image is restored with reference to the position, considering the shift amount of the sampling interval due to the position shift of each scaled image.

The first embodiment of the present invention will be described below. The first embodiment is one in which the present invention is applied to a microscope, and has a form in which a high resolution function is added to an existing microscope. FIG. 5 is a configuration diagram of the first embodiment, and FIG. 6 is a configuration diagram of an image combining unit in the configuration of the first embodiment.

In FIG. 5, the image pickup device is a microscope unit 314.
The microscope unit 314 includes an objective lens group 301, a revolver 302 that holds the objective lens unit 301, and an eyepiece lens 303, and a revolver / focus drive unit 304 and an eyepiece lens 303 that control rotation and focus of the revolver 302. A camera box holding a CCD 305 of an RGB Bayer array as a color mosaic filter is installed at the focal position of. RGB Bayer array C
The signal line from the CD 305 is connected to the system bus 309 via the color separation sample hold circuit 307 of the main body 315 and the A / D converter 308.

On the contrary, the control signal to the CCD 305 of the RGB Bayer array is sent from the system bus 309 to the CCD driver 3
It is input to the RGB Bayer array CCD 305 via 06. In addition to the above, the system bus 309 also includes a control unit 310 for controlling the operation of the entire apparatus, an image holding RAM 311 for storing a color image, an image synthesizing unit 312, and an external terminal 313.
Is connected.

FIG. 6 is a detailed block diagram of the image synthesizing unit 312 shown in FIG. The image synthesizing unit 312 is an image cutting unit 4
01, a common sampling expression configuration unit 402, an image restoration unit 403, and an image composition buffer (RAM) 404. The common sampling expression construction unit 402 includes an image block decomposition unit 402a and a scaling pattern table (ROM) 40.
2b, and the image restoration unit 403 is a partial restoration calculation unit 40.
3a and an image block synthesis unit 403b. In FIG. 6, the solid line represents the flow of image data, and the dotted line represents the flow of other information.

The operation of the above configuration will be described below.

First, a sampler is set by a photographer, and the angle of view of a high resolution image to be finally obtained is set by appropriately rotating the revolver 302 and selecting the objective lens 301.

After the setting of the sample and the setting of the angle of view are completed, the control unit 310 stores the initially selected magnification as a standard magnification and the magnification change (when using another objective lens connected to the revolver 302). The ratio of the objective lens magnification to the standard magnification) is calculated, and the objective lens magnification is stored only when the variable magnification becomes a fraction of an integer. In this embodiment, it is assumed that n objective lens magnifications have been stored. The scaling pattern table 402b of the image composition unit 312 stores the block size to be used by the image block decomposition unit 402a and the restoration matrix to be used by the partial restoration calculation unit 403a for each combination of scaling factors.

The control unit 310 calculates a set of magnifications from the stored set of objective lens magnifications, and uses this as an index to access the magnification change pattern table 402b of the image composition unit 312 through the system bus 309 to obtain the corresponding block size. Is loaded into the internal buffer of the image block decomposition unit 402a, and the restoration matrix is loaded into the internal buffer of the partial restoration calculation unit 403a.

After that, the control unit 310 controls the revolver / focus drive unit 304 to select only one of the objective lenses 301 having the stored objective lens magnification, and each time, it focuses on the image pickup surface of the CCD 305 in the RGB Bayer array. And the CCD Bayer array CCD 305 is driven to image the scaled object. The electric signals obtained by the CCD 305 of the RGB Bayer array are color separation sample hold 307, A / D
After passing through the conversion unit 308, the RGB primary color images are decomposed and stored in the image holding RAM 311.

When the selection of the objective lens 301 is completed and all the scaled images of a predetermined scale factor are stored in the image holding RAM 311, the image cutout unit 401 determines the area position in each scaled image corresponding to the standard angle of view. The calculation is performed and the portion is read out to the image compositing buffer 404. The read position is calculated from the fact that all the images in the image holding RAM 311 have a common center and a variable magnification relationship.

Before proceeding to the next processing, the block size and the method of calculating the restoration matrix will be described. The block size and the restoration matrix are calculated in advance and the scaling pattern table 40
Stored in 2b. FIG. 3 exemplifies a sampling relationship between a standard image cut out and a 0.5 × zoom-out image in a Bayer array CCD. The common sampling interval coincides with the pixel interval of the standard image because the scaling ratio is selected as an integer. Since the Bayer array is a repetition of a 2 × 2 pattern as shown in FIG. 3, for each combination of scaling factors {S1, S2, ... {1
/ Si} (least common multiple of i = 1 ... N) .times.2 The sampling relationship within a square block is repeated over the entire image. The length of the side of this square is the block size, and the size is Si times in each variable-magnification image.

In the example of FIG. 3, the least common multiple of {1, 2} =
2, 2 × 2 = 4, the block size is 4 in the unit of the common sampling interval, and in each variable-magnification image, the common sampling interval is the same as the pixel width of the standard image, and thus the size is {4, 2}. Further, in the description of this embodiment, since it is a color image and is two-dimensional, a new description is necessary regarding the common sampling expression and the restoration matrix.

For two-dimensional sampling, as seen in the subscripts shown in FIG. 3, the row direction of each block is a vertical vector, and the vertical vectors from each row are connected in the column direction from top to bottom. By making the two-dimensional pixel value one-dimensional, the same discussion as in the above-mentioned sixth outline can be made. In the combination of the scaling factors {1, 1/2} shown in FIG. 3, the common sampling matrix is as follows for the R and G components, for example.

[0084]

(Equation 10)

Here, fR1 ... fR16 etc. are the restored values of each primary color component. In this embodiment, processing is performed based on the constrained least squares method, but the constrained condition is slightly complicated because color images are taken into consideration. Here, the color difference components (RY, BY) of a general natural image are used as a constraint that they do not have high spatial frequency components. Specifically, from the primary color values (R, G, B) to the luminance / color difference values (Y, RY, BY)

[Equation 11]

And the smoothness of each component was collectively expressed as a constraint condition.

(Equation 12)

Is used after being discretized. x and y are spatial coordinates,
wY, wRY, wBY are constraint weights, and wY << w
RY and wBY. Since this type of constraint is not applied individually to each primary color component image, it is necessary to represent the primary color component image to be restored by one vector when discretizing. In that case, the sampling relation is

(Equation 13)

And the discretization of the integral of equation (14) is

[Equation 14]

Can be written as Then, the matrices H, Q, and R of the equations (15) and (16) can be substituted into the equation (3) to obtain the restoration matrix of the equation (4) in advance.

As described above, the block size and the restoration matrix obtained in advance are controlled by the control unit 310 before the variable-magnification image is picked up by the image block decomposition unit 402a and the partial restoration calculation unit 4.
It is stored in 03a. After cutting out all the scaled images, restoration processing is performed in block units. The image block decomposition unit 402a reads, from the partial image of each variable-magnification primary color component, a block having a block size (block size of the standard image × magnification) determined by the corresponding magnification from the image combining buffer 404, and (11), (12),
It is transferred to the internal buffer of the partial restoration calculation unit 403a as a vector of one column as shown in the equation (15). The partial restoration calculation unit 403a multiplies the transferred vector by the restoration matrix to obtain RGB primary color restored images at once.

The restored image of the obtained vector is transferred to the image block synthesizing unit 403b for each of the three primary colors, and here, each primary color component synthetic image of the image synthesizing buffer 404 is
It is written as a two-dimensional block at a position corresponding to the block being processed. When this process is completed for all blocks, the three primary color restored image having the same angle of view as the standard image is obtained in the image synthesis buffer 404. This image is output to the image holding RAM 311 or the external terminal 313 via the system bus 309.

The second embodiment of the present invention will be described below.
In the second embodiment, the present invention is applied to a scanner, and image synthesis processing is performed by the host computer 613. FIG. 7 is a block diagram of the second embodiment, and FIG. 8 is a block diagram of the image combining unit shown in FIG.

In FIG. 7, a scanner body 612 includes a cylindrical lens 601, a zoom mechanism 602,
A zoom driver 603, a CCD line sensor 604, a line sensor moving mechanism 606, and a CCD / moving mechanism driver 605 are provided, and these are connected to a control unit 609 via a system bus 608. The output of the CCD line sensor 604 is connected to the system bus 608 via the A / D converter 607. A host computer 613 is connected to the system bus 608, and an image holding RAM 610 and an image composition unit 611 are provided inside the host computer 613.

The image composition unit 611 is composed of a plurality of blocks as shown in FIG. It is roughly divided into an image cutout unit 701, a common sampling expression configuration unit 704, an image restoration unit 705, a wavelet decomposition unit 702, a scaling factor determination unit 703, and an image synthesis buffer (RAM) 706. The common sampling expression configuration unit 704 has an image block decomposition unit 704a and a scaling pattern table 704b, and the image restoration unit 705 has a partial restoration calculation unit 705a and an image block synthesis unit 705b.

In this embodiment, the position of the common sampling value and the calculation procedure can be calculated accurately according to the flow of FIG. 9 if the combination of the scaling factors is determined, and the partial restoration calculation unit 705a can directly calculate the common value without error. A method of estimating the remaining unknown sampling values after first calculating the sampling values is adopted. Therefore, the scaling pattern table 704b uses the combination of scaling factors as an index, and the common sampling value positions {p1, p2, ... pm} shown in equation (9) that are directly obtained without the need for estimation.
And a corresponding variable v1 ... Vm calculation method and a restoration matrix in which Hs and Qs are H and Q in the equation (5) are stored in advance. The scaling pattern table 704b further includes
The resolution pattern defined in the seventh outline calculated using the 0 flow is also stored.

The operation of the above configuration will be described below. In FIG. 8, the solid line represents the flow of image data and the dotted line represents the flow of other information. In this embodiment, first, the host computer 6 that is an interface with the user
The scanner body 6 specifies the imaging range of the imaging target from 13
12 control units 609. The control unit 609 first sets the zoom mechanism 602 to a zoom magnification that covers the imaging range, and stores this zoom magnification as the standard magnification. Then, the CCD line sensor 604 is driven to capture a standard image having a desired angle of view line by line. The signal obtained by the A / D conversion unit 607 is once stored in the image synthesis buffer 706.
And then transferred to the wavelet decomposition unit 702.

The wavelet decomposition approximates a signal by a linear sum of a waveform called a wavelet, which has a narrow spread and is scaled and scaled, and the weight of a waveform having a large scale is a magnitude of a local low frequency component. Can be regarded as representing the magnitude of the local high frequency component. A Haar function, a spline function, or the like is used for the wavelet. The wavelet decomposition unit 702
As a result of wavelet decomposition of a signal (horizontal image) for each line by a spline wavelet,
The result of the highest frequency level having an average value equal to or larger than a certain threshold value is quantized into a local high frequency component, which is transferred to the internal buffer of the scaling factor determination unit 703.

In the scaling pattern table 704b of the common sampling expression constructing unit 704, the resolution pattern defined in the above-mentioned seventh outline according to the combination of scaling factors used for synthesis is calculated in advance by the flow of FIG. 10 and stored. ing. The scaling factor determination unit 703 accesses the scaling pattern table 704b, and calculates the inner product of the transferred pattern of the existing position of the high frequency component and the resolution pattern stored in the scaling pattern table 704. And
The combination of scaling factors corresponding to the resolution pattern giving the largest inner product value is selected, and the optimal combination of scaling factors for each line is stored in its own internal buffer.

The required combination of scaling factors differs for each line, but if scanning is performed in such a manner that the scaling factor is changed each time each line is read, the scanning will be very slow. Therefore, in this embodiment, for all lines in the imaging range, the union of combinations of scaling factors required for each line is taken, and the entire imaging region is scanned with each scaling factor that is an element of the union. Therefore, after scanning all the lines of the standard image, the scaling factor determination unit 70
3 calculates from the combination information of the scaling factors stored in its own internal buffer, the number of times the scanning is performed by changing the scaling factor and the number of the line which requires each scaling factor.

Thereafter, the control unit 609 controls the zoom driver 6
The zoom mechanism 602 is driven at a magnification specified by the magnification change determination unit 703 via 03. The zoom mechanism 602 is a CCD
While focusing on the line sensor 604, the target image is scaled at a designated magnification only in the direction in which the CCD line sensor 604 is arranged. The scaled target image becomes a one-dimensional image signal through the CCD line sensor 604 and the A / D conversion unit 607, and the control unit 609 controls the line sensor moving mechanism 606 via the CCD / moving mechanism driver 605 to display the designated area. Image scanning RAM by scanning
At 610, a scaled two-dimensional image of a predetermined area is obtained.

After scanning with all the necessary scaling factors, the image cropping unit 701 calculates the area of the image of another scaling factor corresponding to the angle of view of the image having the standard scaling factor, and that part is buffered for image synthesis. Read to 706. The read position is calculated from the fact that all the images in the image holding RAM 610 for each line have a variable power relationship with the midpoint of the line being common. A combination of scaling factors for each line of the obtained scaling partial image is already held in the internal buffer of the scaling factor determination unit 703. Scale factor determination unit 703
Is a block size for each line and the above-mentioned restoration matrix, and the pixel position and calculation procedure that are directly obtained without error are obtained by subtracting the scaling pattern table 704b, and the block size is obtained in the image block decomposition unit 704a and the restoration matrix and directly obtained pixels. The position and the calculation procedure are transferred to the partial restoration calculation unit 705a.

When the block size is designated, the image block disassembling unit 704a divides one line into a plurality of blocks by a block having a size of the designated block size × the scaling ratio for each scaled image, and the partial restoration computing unit 705a Send to internal buffer. The partial restoration calculation unit 705a executes the calculation procedure of the pixel value of the pixel position directly obtained, which is transferred from the scaling factor determination unit, obtains the common sampling value directly obtained first, and estimates the remaining values using the restoration matrix. To do. The calculated partial image is written in the corresponding block area of the combined image by the image block combining unit 705b. When this process is completed for all blocks, the image composition buffer 70
In 6, a high-resolution image with the same angle of view as the standard image is obtained.
This image is stored in the image holding RAM through the system bus 608.
Written in 610.

FIG. 9 is a flow chart showing a process of removing the redundancy of the common sampling matrix. The algorithm continues the basic transformation of subtracting one row vector from another row vector to reduce the number of 1's in the row vector until no further transformation is possible. The transformation procedure is
It is stored by sequentially saving the operation of subtracting the j-th row vector from the i-th row vector in the calculation stack as a set of (i, j).

Step s0: The common sampling matrix is assigned to the array H. The computational stack is initially emptied. H
All row number pairs (i, j) of are stacked on the tuple stack.

Step s1: For all the row vector number pairs (i, j) of the pair stack H, the one whose inner product of the corresponding row vectors Li and Lj of H is 0 is removed from the pair stack.

Step s2: The counter holding the number of times of basic deformation is set to 0.

Step s3: Check whether the set stack is empty.

Step s4: One pair is taken out from the pair stack.

Step s5: The fetched pair (i, j)
For the row vectors Li and Lj of H corresponding to, the difference vector d = Li-Lj is created.

Step s6: It is checked whether or not the minimum value of the elements of the difference vector d is negative. If not negative, the process branches to step s7, and if negative, the process proceeds to step s8.

Step s7: Replace the i-th row vector of H with the difference vector d. The set of row numbers (i, j) is stacked on the calculation stack and the counter is incremented by 1.

Step s8: The sign of the difference vector d is inverted (d is replaced with -d).

Step s9: Check whether the minimum value of the difference vector d is negative, and if not negative, branch to step s10.
If negative, proceed to step s3.

Step s10: Replace the jth row vector of H with the difference vector d. The set of row numbers (i, j) is stacked on the calculation stack and the counter is incremented by 1.

Step s11: Check whether the counter is positive, and if positive, go to step s1. If not positive, go to step s.
Branch to 12.

Step s12: Examine each row of H to find all rows having a number of 1's, and set row numbers r1 to rm.
To memorize. For r1 to rm, the column number Pi (i = 1 ... M) having 1 of the corresponding row vector is obtained, paired with the row number and stacked on the set stack, and the corresponding row is removed from H. Reverse the computational stack.

By this process, the redundancy of the common sampling matrix is removed, and the matrix Hs representing the relationship only for the common sampling values that cannot be directly obtained and the second element of the pair of the pair stack are directly connected to each other. The position of the common sampling value to be obtained is obtained. When the pixel values of the scaled image are actually given as column vectors, the process of subtraction of the rows obtained in the calculation stack is executed in order from the top, and the first obtained vector of the pair of the stack stack of the obtained vectors is executed. If the value of the position specified by the element is taken, it is the common sampling value of the position specified by the second element of the pair of the tuple stack. The remaining common sampling value can be estimated by multiplying the vector consisting of the remaining values of the obtained vector by the restoration matrix calculated from Hs.

FIG. 10 is a flow for obtaining a resolution pattern from a simplified common sampling matrix.

Step S13: The common sampling positions {p1 ... pm} obtained directly with Hs are given.

Step S14: An m-th order square matrix C in which only the i-th row and the pi-th column are 1 and the others are 0 is created.

Step S15: For the matrix formed by stacking Hs and C in the row direction, the sum of each row is normalized to 1 and H '
To memorize.

Step S16: The sum of H'for each column is calculated.

As a result, a pattern having a value which is 1 at the common sampling position directly obtained and which is inversely proportional to the number of common sampling values which are not independent even if simplified at other positions is calculated.

The third embodiment of the present invention will be described below. The third embodiment is an example in which a normal electronic camera is equipped with a high resolution function by zooming. FIG. 11 is a block diagram of the third embodiment, and FIG. 12 is a block diagram of the image synthesizing section shown in FIG.

In FIG. 11, the image pickup apparatus has a single structure,
A single lens CCD area sensor 1003, a CCD driver 1004, and an image holding RAM 1009 are connected to a system bus 1006 via a zoom lens driving / control unit 1002, an A / D conversion unit 1005, and a zoom lens 1001. The system bus 1006 has a positional deviation detection mechanism 1
010 and the image composition unit 1008 are also connected.

The image composition section 1008 has a block structure as shown in FIG.
1, a common sampling expression configuration unit 1103, an image restoration unit 1104, a scaling factor determination unit 1102, a random sampling synthesis unit 1105, and an image synthesis buffer 1106. The common sampling expression construction unit 1103 includes a block size calculation unit 1103a and a sampling calculation unit 110.
3b, an image block decomposition unit 1103c, and the image restoration unit 1104 includes a restoration parameter setting unit 1104b,
Partial decompression residual part 1104a, image block composition part 110
4c. Further, in FIG. 12, the solid line represents the flow of image data, and the dotted line represents the flow of other information.

The operation of the above configuration will be described below. In the present embodiment, it is desirable that the image pickup device is originally fixed completely, but this embodiment corresponds to a case where the device is not fixed sufficiently stably. In this case, since it takes time to change the magnification, a shift occurs between the magnified images. Therefore, calculate the amount of displacement for each scaled image,
Create a common sampling matrix that takes into account the parallel movement of the corresponding sampling intervals. However, the sampling matrix may not be well constructed depending on the shift amount. For example, a magnification ratio {1, n / m obtained with an imager having an aperture ratio of 1
(N and m are integers and n <m)},
If the latter is translated by p / q (p and q are integers) pixels with respect to the former, whether or not the effect of the translation can be treated depends on the condition that the edges of the apertures of both samplings coincide: i = mj / n + mp / nq (i and j are integers) (17) It depends on whether there is a solution. here,
i and j indicate the number of pixels of each sampling up to the position where the openings match.

As shown in FIG. 13, scaling factors {1, 3 /
4}, when the center position of the latter is displaced by 1/4 pixel from the center position of the former (standard image), the above equation is i = (4/3) j + ( 1 /
3) and the integer solution (i, j) = {(-1, -1),
(3, 2)} is obtained.

This indicates that, as shown in FIG. 13, the normal sampling relationship in the case where there is no shift is established outside the range of (-1, 3) in the pixel unit of the standard image. On the other hand, within this range, the positional relationship of the shift becomes special. In the description below, the former is a special block 120.
1 and the latter divided into a normal block size is called a normal block 1202. Such an integer solution is mp
It is always possible if / q is a multiple of the common sampling interval.

On the other hand, as shown in FIG. 14, the shift is 2 /
When the number of pixels is 3, the equation (17) is i = (4/3) j +
(8/9), and since there is no integer solution, the above block division cannot be performed. In this case, the second
Processing that makes good use of the regularity described in the outline of will not be possible.

Therefore, in this embodiment, for each of the scaling factors used for synthesis, when the amount of parallel translation of sampling due to displacement can be sufficiently approximated to a multiple of the common sampling interval when there is no displacement of sampling. Just image. In this case, a periodicity can be surely created in the deviation of sampling, and the block processing can be performed as described in the second outline above. Instead, since it takes a long time to capture an image of each magnification, the value of the magnification used for composition is set to a predetermined small number by using the method described in the above-mentioned fifth outline, and it is necessary to limit the deviation. A large number of variable-magnification images for checking are captured, and the restoration parameters are adjusted using these to improve the restoration accuracy.

When the image of the scaling ratio used for the composition does not satisfy the condition, the composition method is switched from the high resolution method exclusively for zooming to the commonly used sampling composition method. In that case, the scaling factor is selected as an irrational number to eliminate the periodicity of the sampling shift, and the resolution is increased by a known restoration method.

The operation of the image pickup apparatus is as follows. First, when the subject and the angle of view are selected, the control unit 1007 controls the single-chip CCD area sensor 1003 at the selected magnification to capture one standard image and records it in the image holding RAM 1009. Next, the scaling factor determination unit 1102 selects one of a plurality of combinations of scaling factors for synthesis which are determined in advance, and the control unit 1 along with the common sampling interval.
Tell 007.

The control section 1007 controls the zoom lens 1 through the zoom drive / control section 1002 so that the zoom ratio becomes the selected one.
001 is controlled, and at the same time, the displacement detection mechanism 1010 detects the displacement of the imaging device. Position shift detection mechanism 101
0 is realized by using a gyro, for example. Control unit 1007
Calculates whether the detected positional deviation is sufficiently close to a specified multiple of the common sampling interval, and if it is close, drives the single-chip CCD area sensor 1003 through the CCD driver 1004 to capture an image and perform A / D conversion. An image signal is stored in the image holding RAM 1009 through the unit 1005. The positional deviation amount is transferred to the internal buffer of the scaling factor determination unit 1102 in units of the common sampling interval.

The scaling factor determination unit 1102 informs the control unit 1007 of another combination of scaling factors when an appropriate position shift does not occur within a certain period of time and imaging is not performed, or the synthesis method is switched to an irrational number. Decide whether to select a scaling factor. When the imaging is performed within the fixed time, the scaling factor determining unit 1102 notifies the control unit 1007 of some scaling factors for checking that are determined for the selected combination of scaling factors. The positional deviation amount is detected by the positional deviation detection mechanism 1010,
Although calculated and transmitted to the scaling factor determination unit 1102, the control unit 1007 captures an image regardless of the information, and an image having an arbitrary positional deviation and a scaling factor for checking is stored in the image holding RAM 1009. When the imaging is completed for all the specified scaling factors for checking, the image cropping unit 1
Reference numeral 101 reads each scaled image from the image holding RAM 1009, extracts a partial image corresponding to the standard image, and stores it in the image synthesis buffer 1106. From this point, processing is performed in block units in blocks of a size corresponding to the sampling deviation period.
A special block 1201 as shown in FIG. 3 is formed, and a normal block 1202 is formed in the periphery.

The block size calculation unit 1103a receives the image buffer 110 sent from the scaling ratio determination unit 1102.
Based on the scaling ratio and position shift of the image for composition in 6 (1
Solve equation (7) and calculate the appropriate block division method.
Based on this division, the sampling calculation unit 1103b
Computes the common sampling matrix. When the calculation up to this point is completed, the image block decomposition unit 1103c reads out an image for each block from each scaled image in the image synthesis buffer 1106 according to the calculated block division, and the one-dimensional vector g as shown in equation (5). Is formed and transferred to the partial restoration calculation unit 1104a. The sampling matrix H of the corresponding block is transferred from the sampling calculator 1103b.

When this step is completed, the image cutout unit 1
Reference numeral 101 reads a check variable-magnification partial image corresponding to the standard image of the check variable-magnification image into the image combining buffer 1106.

At this point in time, the necessary data are available, so the restoration operation is performed. First, the restoration parameter determination unit 1104b
, The initial values {γ1, γ2, ... γn} stored in advance are transmitted to the partial restoration calculation unit 1104a as the parameter γ used in the equation (5). Partial restoration calculation unit 1104a
Calculates a common sampling value fi (i = 1 ... n) from the given .gamma.i, H and g according to the equation (5). In the equation (5), Q is a smoothness condition.

(Equation 15)

A discretized version of is used. Y represents the pixel value of the restored image. The obtained result f is sent to the restoration parameter setting unit 1104b.

The restoration parameter determination unit 1104b creates, from each fi, a sampling image having a corresponding magnification and positional deviation with respect to each read check variable-magnification partial image, and determines an error from the check variable-magnification partial image. Compare. The calculated error amounts are weighted and added together for the images of all the check magnifications, and are used as the error evaluation value ei of the restoration by the selected parameter γi. The restoration parameter determination unit 1104b selects the parameter value γi that gives the smallest value among the obtained ei, and calculates the swing width w of the parameter from ei. And centering around γi ±
A value is randomly selected within the range of w and the parameter {γ1
..., γn} is newly created.

This parameter is again restored by the restoration calculation unit 1104.
Then, the process described above is repeated. The preset swing width w becomes smaller as the error value becomes smaller. When ei falls below a certain threshold, the iteration ends,
The restored common sampling value is the image block synthesis unit 1.
It is written in the corresponding position of the composite image in the image composition buffer 1106 by 104c. When this process is completed for all blocks, the image synthesis buffer 1106
A high-resolution image with the same angle of view as the standard image can be obtained. This image is stored in the image holding RAM through the system bus 1006.
Written in 1009.

On the other hand, when the scaling factor determination unit 1102 determines the switching of the combining method, the processing in block units is not performed. In this case, the scaling factor determining unit 1102 randomly selects an irrational scaling factor and sends it to the control unit 1007, and the scaled image is stored in the image holding RAM 1009 together with the displacement information by the displacement detection mechanism 1010. The image cutout unit 1101 reads out an area corresponding to the standard image from each scaled image and stores it in the image synthesis buffer 1106.
The random sampling synthesizing means 1105 uses a known sampling reconstruction method based on the scaling factor and the position shift amount.
An image obtained at a sampling interval smaller than that of a regular standard image is reconstructed and written in the image holding RAM 1009.

The specific embodiments described above include inventions having the following configurations, and "corresponding embodiments of the invention" and "effects" of each configuration are as follows.

(Structure 1) Image-magnification scaling means for forming an image of the same subject at a plurality of different magnifications, and a plurality of scaled images of the same subject obtained by scaling by the image-magnification scaling means. Image pickup means for converting each to an electric signal, image storage means for storing the plurality of scaled images each converted to an electric signal by the image pickup means, and the plurality of scaled images stored in the image storage means An image synthesizing unit for synthesizing to generate an image having a higher resolution than a scaled image obtained at the highest scale factor among the plurality of different scale factors, the still image capturing apparatus.

(Corresponding Embodiment of the Invention) As the embodiment relating to the present invention, the above-mentioned first, second and third embodiments are provided.
The embodiment corresponds to. An optical path insertion type objective lens 301 in FIG. 5, a cylindrical lens 601 in FIG. 7, and a zoom lens 1001 in FIG. 11 correspond to the image-forming variable power unit in the configuration. The image pickup means in the configuration are the CCD Bayer array CCD 305 of FIG. 5, the CCD line sensor 604 of FIG.
The single-plate CCD area sensor 1003 in FIG. 11 corresponds, but it is also possible to use another complementary color filter array or a three-plate image pickup system. It is desirable to use the image holding RAM 311 of FIG. 5, the image holding RAM 610 of FIG. 7, and the image holding RAM 1009 of FIG. 11 as the image storage means in the configuration, but a storage device independent of the hard disk or the memory card imaging device is used. It is also possible. The image composition means in the configuration corresponds to the image composition section 312 of FIG. 6, the image composition section 611 of FIG. 8, and the image composition section 1008 of FIG.
The image synthesizing means may be built in the image pickup apparatus main body, or a host computer independent of the apparatus may be used.

(Effect) An image having a resolution higher than that of a scaled image obtained at the highest magnification among a plurality of different scales, that is, an image obtained at a sampling interval closer than any of the obtained scaled images is calculated. By doing so, it is possible to obtain an image with a resolution that exceeds that of the image sensor, and thus it is possible to improve the resolution. Further, since a complicated structure is not newly required and the only mechanism required is the variable power mechanism, the present apparatus can be constructed at low cost. (Structure 2) The image synthesizing means determines the relative magnification between the scaled image obtained at the highest magnification of the plurality of scaled images and the other scaled images, and the sampling of both scaled images. Magnification control means for controlling the image-magnification scaling means so that the points have a predetermined rational number such that the points do not completely match, and a plurality of scaled images obtained under the control of the scaling magnification control means. And a pixel value of the image having a higher resolution than the scaled image obtained at the highest magnification, and a correspondence means for associating the pixel value of the image with the highest magnification. The still image pickup device according to the configuration (1), further comprising: an image restoration unit that restores an image having a higher resolution than the scaled image.

(Corresponding Embodiment of the Invention) The above-described first, second and third embodiments correspond to the embodiment of the present invention. The image composition means in the configuration corresponds to the image composition section 312 of FIG. 6, the image composition section 611 of FIG. 8, and the image composition section 1008 of FIG. 12, and the scaling control means is the control section 310 of FIG. Scaling pattern table 402b of FIG. 8, scaling factor determination unit 703 of FIG. 8, scaling factor determination unit 110 of FIG.
2 corresponds. As the associating means, the common sampling expression configuring unit 402 in FIG. 6, the common sampling expression configuring unit 704 in FIG. 8, and the common sampling expression configuring unit 11 in FIG.
03 corresponds, but may include an image clipping unit (401, 701, 1101) described separately in the embodiment. The image restoration means corresponds to the image restoration unit 403 in FIG. 6, the image restoration unit 705 in FIG. 8, and the image restoration unit 1104 in FIG. In each of these embodiments, the image is processed on a block-by-block basis, but it is not always necessary to divide the image into blocks to process when block distortion is a concern. Also, the effects of the aperture are all on the image restoration section (403, 7).
05, 1104), but the image clipping unit (401, 701, 1101) may perform aperture correction in advance.

(Effect) By making the scaling ratio a rational number, it is possible to efficiently and accurately obtain pixel values in block units at common sampling intervals smaller than any of the samplings of the obtained images. As a result, a high-quality still image that exceeds the resolution of the image sensor can be obtained by a simple combining process.

(Structure 3) The still image pickup device according to the structure (2), characterized in that the image pickup means has a color mosaic filter, and the predetermined rational number is a fraction of an integer. (Embodiment) The above-described first embodiment corresponds to the embodiment of the present invention. The color mosaic filter in the configuration corresponds to the RGB Bayer array added to the CCD 305 of the RGB Bayer array shown in FIG. 3, but various arrays of primary color and complementary color type mosaic filters may be used, which are shown in the second embodiment. Alternatively, the CCD line sensor 604 may be applied with a stripe filter added.

(Effect) When information cannot be obtained on a pixel-by-pixel basis by the mosaic filter, in other cases, it is possible to restore the missing color component by using a scaling factor of 1 / integer, which is not very useful, and thus simple scaling is possible. Color reproducibility can be improved by operation.

(Structure 4) The image restoration means uses the least-squares method for the pixel value of an image having a higher resolution than the scaled image obtained at the highest magnification under the constraint condition that reflects the nature of the subject. Configuration characterized by estimation based on (2)
Alternatively, the still image capturing device according to (3).

(Corresponding Embodiment of the Invention) The above-described first, second and third embodiments correspond to the embodiment of the present invention. The image restoration means in the configuration is the partial restoration calculation unit 403a in the image restoration unit 403 in FIG. 6, the partial restoration calculation unit 705a in the image restoration unit 705 in FIG. 8, and the partial restoration calculation in the image restoration unit 1104 in FIG. The section 1104a corresponds to this. "Characteristic of subject" includes "statistical characteristic of subject" and "constraint condition reflecting statistical characteristic"
As for, the expression (18), which assumes that the fluctuation of the luminance is spatially smooth to some extent, and the expression (14), which assumes that the spatial fluctuation of the color difference component is much smoother than that of the luminance component, correspond to Other objects may be used as long as they statistically reflect the statistical properties of the subject.

(Effect) By using the least squares method with a constraint condition to solve the equation (4), it is possible to explicitly specify the statistical characteristics that the estimated common sampling value should have, and especially to restore the color component. In this case, the constraint included in the natural image can be fully utilized. Further, when the adaptive processing as described in the fifth outline is not performed, the matrix required for the calculation of the equation (5) can be calculated in advance, so that the high speed processing can be performed.

(Structure 5) The still image capturing apparatus described in Structure (4) is characterized in that the parameter for determining the constraint condition is updated by the scaled image and the high resolution image after synthesis.

(Corresponding Embodiment of the Invention) The above-mentioned third embodiment corresponds to the embodiment of the present invention. The constraint condition in the configuration is (1) in the third embodiment.
8) corresponds, and its strength corresponds to the scalar value γ in the formula, but it is applied to the first embodiment, and the constraint condition (14) is applied.
Expression, strength, and a set of weighting factors in the expression (wY, wR-Y, wB
It is also possible to use a vector value such as -Y).

(Effect) Generally, when the scaled image used for restoration is increased, the common sampling value used for deriving the relational expression of common sampling increases. That is, the size of the block to be restored becomes large. On the contrary, if the number of scaled images is reduced, the accuracy of restoration is reduced. According to this configuration, the strength of the constraint parameter can be selected so that it fits the data well, so even if a small scaled image is used for the restoration process, the restoration accuracy does not drop, and the block size is kept small and more accurate. It is possible to achieve high resolution and high color reproduction.

(Structure 6) The image-forming variable magnification means performs the variable magnification only in one direction parallel to the sensor array of the image pickup means, and the image restoration means uses the variable-magnification image obtained at the highest magnification. The still image pickup device according to the configuration (2), wherein the pixel values of a high-resolution image are partially restored without error.

(Corresponding Embodiment of the Invention) The above-described second embodiment corresponds to the embodiment of the present invention. The image restoration means in the configuration is the image restoration unit 705 of FIG.
The CCD line sensor 60 of FIG. 7 corresponds to the image pickup means.
4 corresponds.

(Effect) Since the portion where the common sampling value is directly obtained is obtained more accurately than the simple least squares method, the restoration error is smaller than in the case of estimation and more accurate high precision and color reproduction can be performed. . Since it is possible to calculate in advance which pixel value can be directly obtained without error, the scale of the restoration calculation becomes small and the efficiency becomes high.

(Structure 7) The synthesizing means stores, with respect to various combinations of scaling factors, a resolution pattern storage means for storing a pattern of pixel positions restored from the corresponding scaled image without error by the image restoring means, and an object. Frequency analysis means for analyzing the frequency characteristics of the, and the scaling control means refers to the resolution pattern storage means to generate a pattern most similar to the distribution of the high frequency components of the subject image obtained by the frequency analysis part. The still image pickup device according to the configuration (6), characterized in that a combination of scaling factors to be given is selected.

(Corresponding Embodiment of the Invention) The above-described second embodiment corresponds to the embodiment of the present invention. The resolution pattern storage means in the configuration corresponds to the scaling pattern table 704b in FIG. The wavelet analysis unit 702 of FIG. 8 corresponds to the frequency analysis means,
A Fourier transform, an edge detection operator, a high-pass filter, or the like may be used as long as a local high-frequency component can be extracted. The scaling factor control unit corresponds to the scaling factor determination unit 703 in FIG. 8, and the image restoration unit corresponds to the image restoration unit 7 in FIG.
05 corresponds.

(Effect) It is possible to avoid the phenomenon that the target high-frequency part comes to a part where the resolution does not rise at all depending on the combination of the scaling factors, and it is possible to prevent the difference in resolution from being increased between images.

(Structure 8) The image synthesizing means includes a scaled image obtained at the highest magnification among the plurality of scaled images,
The still image pickup device according to the configuration (1), which uses only an image whose relative magnification with respect to other scaled images becomes an irrational number.

(Corresponding Embodiment of the Invention) The above-described third embodiment corresponds to the embodiment of the present invention. The random sampling composition unit 1105 in FIG. 12 corresponds to the image composition means in the configuration. (Effect) When the holding of the image pickup device is not stable and the method of utilizing the regularity as described in the second outline cannot be used, the overlapping of sampling intervals is prevented and the most efficient image high resolution is obtained. Enable

(Structure 9) The image synthesizing means has a deviation detecting means for detecting a positional deviation amount between a plurality of scaled images, and corrects the detected positional deviation to perform composition. The still image capturing device according to any one of (1) to (8).

(Corresponding Embodiment of the Invention) The above-described third embodiment corresponds to the embodiment of the present invention. The random sampling composition unit 1105 in FIG. 12 corresponds to the image composition means in the configuration. (Effect) The holding stability of the imaging device is poor, and
When the method of utilizing the regularity as outlined in 7 cannot be used, it is possible to prevent duplication of the sampling intervals and most efficiently achieve the high resolution.

(Structure 10) The still image pickup device according to any one of structures (1) to (9), characterized in that a zoom lens is used as the image-forming variable power unit.

(Corresponding Embodiment of the Invention) The above-mentioned third embodiment corresponds to the embodiment of the present invention. The zoom lens 1001 in FIG. 11 corresponds to the zoom lens in the configuration.

(Effect) The present apparatus can be realized at the lowest cost by using the usual zoom lens and zoom control mechanism as the image-forming variable power unit.

Since the synthesizing means and the image storing means do not have to be provided on the side of the image pickup device, it is possible to realize high resolution and high color reproduction in a system in which an ordinary electronic camera, video camera or the like is connected to a computer.

(Structure 11) The still image pickup device according to any one of structures (1) to (9), characterized in that a mechanism capable of inserting a lens group into an optical path is used as the image-magnification varying means.

(Corresponding Embodiment of the Invention) The above-described first embodiment corresponds to the embodiment of the present invention. The lens group corresponds to the objective lens 301 held by the revolver 302 shown in FIG. 5, and the insertion mechanism corresponds to the revolver and the focus drive unit 304. However, the lens group may be one, and the insertion operation is performed by a human. May be.

(Effect) Since it is not necessary to continuously change the scaling ratio for high resolution, even a device such as a microscope equipped with only a discrete scaling unit can improve image quality at a very low cost. realizable. In addition, the discrete scaling mechanism captures images at a higher speed than when a continuous scaling mechanism is used.
In addition to high accuracy, manual zooming is also possible.

(Structure 12) The still image according to any one of structures (1) to (9), wherein the image-forming variable-magnification means uses different magnifications in the vertical and horizontal directions with respect to the imaging means. Imaging device.

(Corresponding Embodiment of the Invention) The above-mentioned second embodiment corresponds to the embodiment of the present invention. The resolution pattern storage means in the configuration corresponds to the scaling pattern table 704b in FIG. The wavelet analysis unit 702 of FIG. 8 corresponds to the frequency analysis means,
A Fourier transform, an edge detection operator, a high-pass filter, or the like may be used as long as it can extract a local high-frequency component. The scaling factor control unit corresponds to the scaling factor determination unit 703 of FIG. 8, and the image restoration unit corresponds to the image restoration unit 705 of FIG.

(Effect) When the resolution is required to be increased in only one direction, the resolution can be freely changed in a certain direction like a scanner, and when a line sensor is used, the vertical and horizontal scaling factors are changed. By not making them equal, the resolution can be increased only in the desired direction. By limiting the scaling in only one direction, the scale of the restoration calculation becomes small and the restoration accuracy can be improved, as described in the sixth outline.

[0177]

According to the first aspect of the present invention, a high quality still image exceeding the resolution of the image pickup device can be obtained at low cost without newly adding a complicated device configuration.

According to the invention described in claim 2, in addition to the effect of the invention described in claim 1, a high quality still image exceeding the resolution of the image pickup device can be obtained by a simple combining process. Become.

According to the third aspect of the invention, in addition to the effects of the first and second aspects of the invention, the missing color can be restored without newly using a complicated device configuration. Like

[Brief description of drawings]

FIG. 1 is a diagram for explaining a method of generating a high-resolution image exceeding the resolution of an image sensor from a plurality of scaled images.

FIG. 2 is a diagram for explaining a method in the case of achieving high resolution by utilizing the regularity of changes in sampling points due to scaling.

FIG. 3 is an explanatory diagram of a sampling relationship between scaled images when a CCD with a Bayer array is used.

FIG. 4 is an explanatory diagram of a change in resolution pattern due to a combination of scaling factors.

FIG. 5 is a configuration diagram of the first embodiment of the present invention.

FIG. 6 is a detailed configuration diagram of an image combining unit shown in FIG.

FIG. 7 is a configuration diagram of a second embodiment of the present invention.

8 is a configuration diagram of an image composition unit shown in FIG. 7. FIG.

FIG. 9 is a flowchart showing a process of removing redundancy of a common sampling matrix.

FIG. 10 is a flowchart for obtaining a resolution pattern from a simplified common sampling matrix.

FIG. 11 is a configuration diagram of a third embodiment of the present invention.

12 is a configuration diagram of an image composition unit shown in FIG.

FIG. 13 is a diagram for explaining the influence of the deviation of the sampling interval due to the position deviation (when the effect of the deviation can be absorbed).

FIG. 14 is a diagram for explaining the influence of the deviation of the sampling interval due to the position deviation (when the effect of the deviation cannot be absorbed).

[Explanation of symbols]

301 ... Objective lens, 302 ... Revolver, 303 ... Eyepiece lens, 304 ... Revolver / focus drive unit, 305
... RGB Bayer array CCD, 306 ... CCD driver, 307 ... Color separation S / H, 308 ... A / D converter, 3
09 ... System bus, 310 ... Control section, 311 ... Image holding RAM, 312 ... Image combining section, 313 ... External terminal, 3
14 ... Microscope section, 315 ... Main body.

Claims (3)

[Claims] 1. An image-forming variable magnification means for forming an image of the same subject at a plurality of different magnification-change ratios, and a plurality of variable-magnification images of the same object obtained by changing the magnification by the image-forming magnification change means. Image pickup means for converting into a signal, image storage means for storing the plurality of scaled images each converted into an electric signal by the image pickup means, and the plurality of scaled images stored in the image storage means And a image synthesizing unit for generating an image having a higher resolution than the scaled image obtained at the highest scale factor among the plurality of different scale factors, and a still image capturing apparatus. 2. The image synthesizing means calculates a relative magnification between a magnified image obtained at the highest magnification among the plurality of magnified images and another magnified image of these magnified images. Magnification control means for controlling the image-magnification variability means so that the sampling points do not have a predetermined rational number, and a plurality of magnification variability values obtained under the control of the variability control means. An associating unit that associates the pixel value of the image with the pixel value of the image having a higher resolution than the scaled image obtained at the highest magnification, and based on the association by the associating unit, at the highest magnification. The still image capturing apparatus according to claim 1, further comprising: an image restoration unit that restores an image having a higher resolution than the obtained scaled image. 3. The still image pickup device according to claim 2, wherein the image pickup means has a color mosaic filter, and the predetermined rational number is a fraction of an integer.