JP2020009099A - Image processing device, image processing method, and program - Google Patents

Abstract

An image processing apparatus, an image processing method, and a program capable of efficiently acquiring a high-resolution image with a small burden are provided. A first obtaining means for obtaining a first image; a determining means for determining a region to be increased in resolution in the first image based on the first image; a second acquisition means for acquiring a second image having a narrower angle of view than the first image based on the determined area; and generating means for generating an image. [Selection drawing] Fig. 2

Description

  The present invention relates to a technique for acquiring a high-resolution image.

  When magnifying a part to be viewed in detail from an image captured at a wide angle, simply enlarging the area results in a blurred image because a subject in a distant view has no resolution. On the other hand, for example, if a narrow-angle image zoomed to the telephoto side is captured at the same position as the reference wide-angle image, and a high-resolution image is generated using the narrow-angle image at the corresponding position with reference to the wide-angle image, Also, even when the image is enlarged, details of the subject and the like can be seen. In addition, for example, in Patent Document 1, a user desires by photographing a wide-angle image and a narrow-angle image and combining the narrow-angle image with the wide-angle image by using the consistency of each pixel of the aligned images. A technique for generating a resolution image is disclosed.

JP 2006-229802 A

  As described above, when a high-resolution image is generated using a wide-angle image and a narrow-angle image, when processing is performed on the entire image, it is necessary to capture many narrow-angle images, and the necessary number of images must be captured. Therefore, a large burden is imposed on the user. Also, in an image processing device such as an imaging device, it takes an enormous amount of time to process these many images. Also in the technique of Patent Document 1 described above, since the matching is determined in pixel units for all pixels of the narrow-angle image used to generate a high-resolution image, the processing load is reduced by the number of shot images. To increase.

  Therefore, an object of the present invention is to enable a high-resolution image to be efficiently acquired with a small burden.

  An image processing apparatus according to an aspect of the present invention includes a first obtaining unit that obtains a first image, and a determining unit that determines an area to be increased in resolution in the first image based on the first image. A second acquisition unit configured to acquire a second image by imaging an area having a smaller angle of view than the first image based on the area determined to have a higher resolution; and the first image. And generating means for generating a high-resolution image using the second image.

  According to the present invention, a high-resolution image can be efficiently acquired with a small burden.

FIG. 2 is a block diagram illustrating a configuration and functions of the imaging device according to the first embodiment. 5 is an operation flowchart of the imaging device according to the first embodiment. FIG. 9 is a diagram illustrating a process of determining a region to be increased in resolution. It is a flowchart of a high-resolution area | region determination process. FIG. 9 is a diagram illustrating a relationship between an evaluation value and a priority. FIG. 7 is a diagram illustrating a narrow-angle image acquisition process. It is a flowchart of a high-resolution image generation process. FIG. 9 is a diagram illustrating a high-resolution image generation process. It is a block diagram showing the function of the imaging device of a 2nd embodiment. It is an operation | movement flowchart of the imaging device of 2nd Embodiment. It is a figure explaining the relation between an evaluation value and a priority in a 2nd embodiment. FIG. 11 is a diagram illustrating a process of selecting a narrow-angle image according to the second embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.
<First embodiment>
In the first embodiment, a region to be increased in resolution is determined based on a wide-angle image acquired by shooting, and a narrow-angle image captured with a narrower angle of view than when a wide-angle image is shot is determined based on the determined region. An example in which a high-resolution image is generated by acquiring a high-resolution image will be described.
FIG. 1A is a diagram illustrating a schematic hardware configuration of an imaging apparatus as an application example of the image processing apparatus according to the present embodiment, and FIG. 1B is a diagram illustrating a wide-angle image of the imaging apparatus according to the present embodiment. FIG. 3 is a functional block diagram illustrating functions from photographing to generation of a high-resolution image.
First, the configuration of the imaging apparatus according to the present embodiment will be described with reference to FIG.

  In FIG. 1A, the optical system 101 includes a lens group including a zoom lens, a focus lens, and a shift lens, an aperture adjustment mechanism, a shutter mechanism, and the like. The optical system 101 adjusts the magnification, focus position, light amount, and the like of a subject image that reaches the imaging unit 102. The imaging unit 102 includes a photoelectric conversion element such as a CCD or a CMOS sensor that photoelectrically converts a subject image formed on an imaging surface through the optical system 101 and converts the image into an electric signal. The A / D conversion unit 103 converts the analog imaging signal input from the imaging unit 102 into digital image data.

The image processing unit 104 performs, in addition to the normal image signal processing, the determination processing of the high-resolution area according to the present embodiment, the calculation of the focal length appropriate for acquiring the narrow-angle image used for generating the high-resolution image, And a process of generating a high-resolution image. The details of the processing for determining the high-resolution area, the processing for calculating the appropriate focal length, and the processing for generating the high-resolution image will be described later.
The optical system control unit 105 performs drive control for the lens group of the optical system 101, drive control for the aperture adjustment mechanism, drive control for the shutter mechanism, and the like. Although details will be described later, especially in the case of the present embodiment, the optical system control unit 105 controls the zoom position (focal length) of the optical system 101 with respect to the zoom lens and controls the shift lens under the control of the system control unit 106. It performs shift drive control and the like.

A system control unit 106 is a control function unit that controls and controls the operation of the entire apparatus according to the present embodiment. Based on an instruction transmitted from the operation unit 107, drive control of the optical system 101 and the imaging unit 102, image processing Control of predetermined image processing in the unit 104 is performed.
The display unit 108 includes a liquid crystal display or an organic EL (Electro Luminescence) display, and displays an image generated by the imaging unit 102, an image read from the recording unit 109, a user interface screen such as various setting menus, and the like. I do.
The recording unit 109 has a function of recording an image, and may include an information recording medium using a package containing a rotating recording body such as a memory card or a magneto-optical disk on which a semiconductor memory is mounted. The recording medium may be detachable.
The image processing unit 104, the optical system control unit 105, the system control unit 106, the display unit 108, and the recording unit 109 are connected by a bus, and exchange image data and various information between them via the bus. Is performed.

Next, with reference to FIG. 1B, each functional block from the acquisition of a wide-angle image to the generation of a high-resolution image in the imaging device of the present embodiment will be described.
The functional block illustrated in FIG. 1B includes a wide-angle image acquisition unit 111, an area determination unit 112, a narrow-angle image acquisition unit 113, and an image generation unit 114. In the present embodiment, the functions of the wide-angle image acquisition unit 111 and the narrow-angle image acquisition unit 113 are realized, for example, by the system control unit 106 controlling the optical system 101 via the optical system control unit 105. The functions of the area determination unit 112 and the image generation unit 114 are executed by the image processing unit 104, for example.

  The wide-angle image acquisition unit 111 acquires a wide-angle image used as a reference image in the image processing according to the present embodiment. In the imaging device according to the present embodiment, the function of the wide-angle image acquisition unit 111 is realized by the system control unit 106, for example. For example, as a function of the wide-angle image acquisition unit 111, the system control unit 106 controls the zoom lens of the optical system 101 via the optical system control unit 105 to adjust the focal length to be short so that wide-angle shooting is performed. I do. The focal length at this time is set according to, for example, an instruction from the user via the operation unit 107. Then, in this state, the image data captured by the imaging unit 102 and output from the A / D conversion unit 103 is sent to the image processing unit 104 as wide-angle image information.

  The area determination unit 112 executes a process of determining an area to be increased in resolution based on information of the wide-angle image acquired by the wide-angle image acquisition unit 111. In the imaging device according to the present embodiment, the function of the region determination unit 112 is realized by, for example, the image processing unit 104. The details of the process for determining the high resolution area will be described later. Similarly, the details will be described later, but the number of regions determined to increase the resolution is not limited to one, but is often plural. The area information indicating the high-resolution area determined by the area determining unit 112 of the image processing unit 104 is sent to the system control unit 106.

  The narrow-angle image acquisition unit 113 captures a narrow-angle image based on the area information determined by the area determination unit 112. In the imaging device according to the present embodiment, the function of the narrow-angle image acquisition unit 113 is realized by the system control unit 106, for example. For example, the system control unit 106 first calculates an appropriate focal length for photographing a narrow-angle image used for high resolution based on the area information from the area determination unit 112. Then, the system control unit 106 controls the zoom lens of the optical system 101 via the optical system control unit 105 to control the zoom lens of the narrow-angle image to be used for high resolution so as to have the focal length. Execute, that is, execute a narrow-angle image acquisition process. Details of the narrow-angle image acquisition processing will be described later. When a plurality of regions are determined as the regions to be increased in resolution by the region determining unit 112, the narrow-angle image acquiring unit 113 acquires a plurality of narrow-angle images corresponding to the plurality of regions. In the narrow-angle image acquisition process, image data captured by the imaging unit 102 and output from the A / D conversion unit 103 is sent to the image processing unit 104 as information on a narrow-angle image.

  The image generation unit 114 refers to the wide-angle image acquired by the wide-angle image acquisition unit 111, and generates and outputs a high-resolution image using the narrow-angle image acquired by the narrow-angle image acquisition unit 113. In the imaging device according to the present embodiment, the function of the image generation unit 114 is realized by the image processing unit 104, for example. Details of the high-resolution image generation processing will be described later. The data of the high-resolution image generated by the image generation unit 114 of the image processing unit 104 is then recorded, for example, in the recording unit 109 or displayed on the display unit 108 under the control of the system control unit 106.

FIG. 2 is a flowchart illustrating a flow of a processing procedure from the above-described wide-angle image acquisition processing to the high-resolution image generation processing in the imaging apparatus according to the present embodiment. In the following description, steps S201 to S204 of each process are abbreviated as S201 to S204, and the same applies to other flowcharts described later.
Hereinafter, the details of each process from the wide-angle image acquisition process to the high-resolution image generation process in the imaging apparatus of the present embodiment will be described with reference to the functional blocks of FIG. 1B and the flowchart of FIG.

  In step S201, the wide-angle image acquisition unit 111 acquires a scene to be photographed by photographing a wide-angle image used as a reference image in image processing according to the present embodiment, and determines data of the acquired wide-angle image in area determination. To the unit 112. FIG. 3A is a diagram illustrating an example of the wide-angle image 301 of the scene captured by the wide-angle image acquisition unit 111 in step S201.

Next, in step S <b> 202, the region determination unit 112 determines a region for which the resolution is to be increased based on the wide-angle image acquired by the wide-angle image acquisition unit 111.
Here, when determining an area to be increased in resolution using information of a scene captured as a wide-angle image, the area determination unit 112 first sets the wide-angle image 301 to a plurality of block areas (hereinafter simply referred to as blocks). ). The block size and the number of divisions when the wide-angle image 301 is divided into blocks can be arbitrarily set. FIG. 3B shows a state in which the wide-angle image 301 in FIG. 3A is divided into a plurality of M × N blocks, for example. FIG. 3 is a diagram showing an example of a wide-angle image 311. Then, the area determination unit 112 obtains a priority, which will be described later, for each block of the wide-angle image 311 and determines an area to be increased in resolution based on the priority for each block. FIG. 3C shows a wide-angle image 321 divided into blocks similar to the wide-angle image 311 in FIG. 3B, in which an area 322 and an area 323 are determined as an example of an area to be increased in resolution. Is shown.

FIG. 4 is a flowchart showing details from the block division performed in S202 of FIG. 2 to the process of determining a high-resolution area based on the priority of each block.
First, in S401, the area determination unit 112 divides the wide-angle image 311 into a plurality of M × N blocks, for example, as shown in FIG.
Next, in S402, the area determination unit 112 obtains an evaluation value corresponding to the feature of the image for each block, and determines a priority when determining a high-resolution area based on the evaluation value for each block. calculate. In the case of the present embodiment, it is assumed that contrast information, saliency information, reliability information of a recognition result of a specific subject, and the like are used as evaluation values according to the characteristics of the image of each block.

  Although details will be described later, in the present embodiment, the region determination unit 112 acquires information representing edge intensity characteristics as contrast information for each block. The edge strength characteristic is used to determine the presence or absence of a texture in the subject area, and the area determination unit 112 determines that an area with a higher edge strength is an area with more texture. Further, in the present embodiment, the saliency information is information indicating a degree of visually conspicuousness. The region determination unit 112 acquires, for example, difference information between the feature of the subject region in the image and the features of the subject regions in the vicinity thereof as the saliency information. The difference information as the saliency information is used to determine whether or not the area is a visually conspicuous area, and the area determination unit 112 determines that the greater the saliency information (difference information), the more visually the area is prominent. In addition, as the reliability information of the recognition result, information representing the reliability of the recognition result in the detection and recognition of the specific subject is obtained. In the present embodiment, an example is described in which an empty area is detected and recognized as a specific subject, for example. The area determination unit 112 determines the recognition result when the subject area detected from the image is recognized as a sky area. Get reliability information.

  FIG. 5A is a diagram showing a graph 501 used when the area determination unit 112 obtains a priority according to contrast information. The graph 501 has a characteristic in which the higher the value of the contrast information, the higher the priority value between the lower limit and the upper limit. A lower limit of the priority is set for contrast information lower than the lower limit, and an upper limit of priority is set for contrast information higher than the upper limit. FIG. 5B is a diagram showing a graph 502 used when the area determination unit 112 obtains a priority according to saliency information. The graph 502 has such a characteristic that, from the lower limit value to the upper limit value, the priority value increases as the value of the saliency information increases. A lower limit value of priority is set for saliency information equal to or less than the lower limit value, and an upper limit value of priority is set for saliency information equal to or more than the upper limit value. FIG. 5C shows a graph 503 used when the area determination unit 112 obtains a priority according to the reliability information of the recognition result when the sky area is detected and recognized, for example. FIG. The graph 503 has a characteristic that, from the lower limit value to the upper limit value, the priority value decreases as the value of the reliability information of the recognition result of the sky region increases. An upper limit of the priority is set for reliability information equal to or lower than the lower limit, and a lower limit of priority is set for reliability information equal to or higher than the upper limit. Note that FIG. 5C shows an example in which a sky region with little change in image features is targeted. Conversely, another subject (for example, a person's face, etc.) with large changes in image features is to be recognized. In the case of, the characteristic may be such that the higher the value of the reliability information, the higher the value of the priority. FIG. 5D is a table 540 showing weights respectively given to the contrast information, the saliency information, and the reliability information of the recognition result of the sky area. The region determining unit 112 assigns the weights shown in Table 540 to the contrast information, the saliency information, and the reliability information of the recognition result of the sky region.

Hereinafter, the process of setting the priorities in the region determining unit 112 according to the contrast information, the saliency information, the reliability information of the recognition result of the specific subject, and the like will be described in detail.
When setting the priority based on the contrast information, the area determination unit 112 detects the edge strength by performing band-pass filter processing on the wide-angle image 311, and uses the equation (1) to detect the edge strength of each block. The integrated value Fr of the intensity is calculated as contrast information. In the expression, (i, j) is a coordinate in the block, fr (i, j) is an edge intensity value at each coordinate in the target block, and Fr is an integral value of an edge intensity of each coordinate in the target block ( Contrast information).

  Here, in the case of a subject region having a large edge strength and a large amount of texture, the integral value of the edge strength increases, and the value of the contrast information increases. When there is a subject having many textures in the wide-angle image 311, it is preferable that the subject region is a high-resolution image. For this reason, the area determination unit 112 sets a high priority for a subject area having a large number of textures in the wide-angle image 311, that is, an area having a high contrast information value by using the graph 501 shown in FIG. I do. On the other hand, in the case of a subject area having a small edge strength and a small texture, the integrated value of the edge strength is small and the value of the contrast information is small. Further, when there is a subject having a small texture in the wide-angle image 311, it may be considered that the subject region is not necessarily required to be a high-resolution image. For this reason, the area determination unit 112 sets a low priority for a subject area having a small texture in the wide-angle image 311, that is, an area having a low contrast information value by using the graph 501 shown in FIG. I do.

When the priority is set based on the saliency information of the subject, the area determination unit 112 calculates, as the saliency information, a difference between the feature amount of the block of interest and the feature amounts of blocks around the block. In the present embodiment, it is assumed that the feature amount of a block includes at least one of a luminance value, a hue value, and an edge strength in the block. The area determination unit 112 calculates an absolute value (SAD: Sum of Absolute Difference) of the difference between the feature amounts of each block with respect to these feature amounts according to the following equation (2). In the expression, fv _A (i, j) is a feature amount at each coordinate in the target block, fv _B (i, j) is a feature amount at each coordinate in the peripheral block, and Fv is an absolute value of a difference between the feature amounts. It is.

  Here, when the difference between the feature amounts of the target block and its surrounding blocks is large, the correlation between those blocks is considered to be low. When the correlation between the blocks is low, the subject area including the blocks is considered to be a visually prominent subject area, and it is desirable to generate a high-resolution image of the subject area. For this reason, the region determination unit 112 uses the graph 502 shown in FIG. 5B to calculate the saliency, which is the absolute value of the difference between the feature amounts of the subject region having a low correlation between the blocks, ie, the target block and the peripheral blocks. A high priority is set for an area having a high information value. On the other hand, when the difference between the feature amounts of the target block and the peripheral blocks is small, it is considered that the correlation between the blocks is high. In this case, the subject area including these blocks is not considered to be a visually prominent subject area, and the subject area need not be a high-resolution image. Therefore, based on the graph 502 of FIG. 5B, the region determination unit 112 sets a low priority to a subject region having a high correlation between blocks, that is, a region having a low value of saliency information. In the above description, an example of the method of calculating the correlation using the SAD, which is the absolute difference value, has been described. However, the present invention is not limited to this, and another method may be used.

  When setting the priority based on the reliability information of the recognition result of the specific subject, the area determination unit 112 first calculates the luminance, the hue, and the edge amount as the feature amount in each block. Further, the region determination unit 112 determines, for each feature amount, whether or not the region to be recognized matches a range considered as a specific subject region based on the feature amounts of the blocks. Is calculated in accordance with. Then, the area determination unit 112 multiplies, for example, the reliability information calculated for each of the feature amounts, and acquires the information after the multiplication as final reliability information. In the case of the present embodiment, a sky area is taken as an example of a specific subject area to be recognized. In addition, as a technique for recognizing a sky area or the like, a technique described in, for example, JP-A-2015-32966 can be used. Here, the sky area generally has high luminance in many cases, has a hue in a predetermined hue range such as blue in the sky or white in clouds, and has a small edge amount in many cases. Therefore, when the sky area is a recognition target, a value that is higher as the luminance value is higher, the hue is within a predetermined hue range, and the edge amount is smaller is set as the reliability information of the recognition result. Then, it is considered that the necessity of the sky region is not a high-resolution image. Therefore, based on the graph 503 shown in FIG. 5C, the region determining unit 112 determines, for a region having a high degree of matching with the range regarded as a sky region, that is, a region having high reliability information of the recognition result of the sky region. , Set a lower priority. On the other hand, based on the graph 503 in FIG. 5C, the area determination unit 112 sets a high degree for an area having a low degree of matching with a range regarded as a sky area, that is, an area having low reliability information of the recognition result of the sky area. Set the priority.

As described above, in S402 of FIG. 4, the region determination unit 112 obtains the contrast information, the saliency information, and the reliability information of the recognition result as evaluation values when setting priorities, and for each of these evaluation values. Calculate the priority.
Next, in S403, the area determination unit 112 calculates the priority W (M, N) for each block by weighting the priority calculated for each evaluation value in S402 using the following equation (3). . Note wherein n represents the number of the obtained evaluation value, a _n is the priority set for each evaluation value. Further, α _n in the expression is a weight set for the priority for each evaluation value, and for example, the weight shown in a table 504 in FIG. 5D is used. Since the contrast information indicating the presence / absence of the texture of the subject is important in determining the area to be increased in resolution, the weight for the contrast information is set to the largest value in the table 504 in FIG. ing. The weight may be specified by the user.

  Next, in S404, the region determination unit 112 determines that the value of the priority W (M, N) is, for example, a predetermined threshold value from among the priorities W (M, N) determined for each block by Expression (3) in S403. The block having the higher priority W (M, N) is selected. Then, the region determining unit 112 determines a region corresponding to the selected block as a region to be photographed with a narrow-angle image to increase the resolution. For example, as shown in the example of FIG. 3C, in the wide-angle image 321, for example, the region 322 and the region 323 are determined as the regions to be increased in resolution.

  As described above, the area determination unit 112 captures a narrow-angle image based on a priority set according to an evaluation value such as contrast information, saliency information, and reliability information of a recognition result of a specific subject. To determine the area to be increased in resolution. As a specific example, a description will be given of a subject region in which the texture of the subject is large and the degree of saliency is high. For example, a subject region with many textures is considered to be a region that is effective in capturing a narrow-angle image and increasing the resolution. In addition, the subject region having a high degree of saliency is a conspicuous region, is likely to be noticed by the user, and is considered to be a region that is effective in capturing a narrow-angle image and increasing the resolution. Therefore, the region determination unit 112 sets a high priority for a subject region having a large texture of the subject or a subject region with a high degree of saliency, and determines the region to be a high resolution region. On the other hand, in the case of an area having a small texture and a low saliency, the difference between the wide-angle image and the narrow-angle image is small, and it is considered that the area is difficult for the user to see. For this reason, the area determination unit 112 sets a low priority for an area having a small texture and a low saliency, so that the area is not determined as a high resolution area.

The description returns to the flowchart of FIG. After S202, the process of the imaging device proceeds to S203.
In step S203, the narrow-angle image acquisition unit 113 captures a narrow-angle image based on the region to be increased in resolution determined in step S202 as described above. In the present embodiment, when shooting a narrow-angle image, the focal length of the zoom lens of the optical system 101 is adjusted to reduce the shooting angle of view, and then the shift lens is shifted to perform the shift operation. Shooting is performed by moving the shooting position.

  FIGS. 6A to 6H are diagrams illustrating a state in which a narrow-angle image of a region to be increased in resolution is captured with reference to a wide-angle image. FIG. 6A is a diagram showing a wide-angle image 601 similar to that shown in FIG. FIG. 6B is a diagram illustrating an example of a moving order (moving order indicated by an arrow in the drawing) when the narrow angle of view is moved by shift driving of the lens after narrowing the angle by zooming up in S203. It is. Note that FIG. 6B shows a wide-angle image 611 divided into blocks in the same manner as the example of FIG. 3B, and as shown by arrows in the figure, the moving order is an order according to the arrangement of each block. However, the order of movement is not limited to this example. 6C shows a wide-angle image 621 similar to that shown in FIG. 3C and regions 622 and 623 determined to be high-resolution within the wide-angle image 621, and a region 622 to be increased in resolution, for example. FIG. 6 is a diagram showing a narrow-angle image 631 which is captured at a narrow-angle shooting angle of view when performing the operation. Note that, in FIG. 6C, a feature point 624 in the wide-angle image 621 and a feature point 632 in the narrow-angle image 631 will be described later. When generating a high-resolution image corresponding to the region 622 in FIG. 6C, for example, a narrow-angle image 641 shown in FIG. 6D, a narrow-angle image 643 shown in FIG. The narrow-angle image 644 shown in FIG. When generating a high-resolution image corresponding to the area 623 in FIG. 6C, for example, a narrow-angle image 642 shown in FIG. 6E and a narrow-angle image 645 shown in FIG. You. Note that the order of arrangement in each of FIGS. 6D to 6H depends on the order in which narrow-angle images are acquired.

  As described above, the narrow-angle image acquisition unit 113 controls the zoom lens of the optical system 101 to narrow the shooting angle of view, and then changes the narrow angle of view to the wide-angle image in FIG. The shift lens is driven to shift so as to move in a moving order covering the photographing range corresponding to 611. Then, the narrow-angle image acquisition unit 113 captures an image in which the region to be increased in resolution determined by the region determination unit 112 is included in the shooting angle of view while moving the narrow-angle shooting angle of view. To do it. Here, it is determined whether or not the region to be increased in resolution is included in the narrow-angle image, for example, by determining the feature point of the region to be increased in resolution of the wide-angle image 621 in FIG. This is performed by comparing the similarity with a feature point (for example, feature point 632) of the narrow-angle image 631. As the similarity, the feature point 624 in the area 622 of the wide-angle image 621 to be increased in resolution and the feature point 632 in the narrow-angle image 631 have a similarity when the corresponding feature information has a predetermined ratio or more. Is determined to be high. The characteristic points for determining whether or not the region to be increased in resolution is in the narrow-angle image include a signal detected by a band-pass filter, an edge detector, or the like in each of the wide-angle image 621 and the narrow-angle image 631. And a region having a large edge strength can be used.

Specifically, the narrow-angle image acquisition unit 113 calculates the degree of correlation between the feature point 624 of the region to be increased in resolution in the wide-angle image 611 and the feature point 632 in the narrow-angle image 631. The presence or absence of the correspondence between the images is detected. As the degree of correlation, for example, a correlation value R _SAD calculated by equation (4) is used. In the expression, f (i, j) is a pixel value of each coordinate in the wide-angle image, and g (i, j) is a pixel value of each coordinate in the narrow-angle image.

Then, the narrow-angle image acquisition unit 113 determines that a set of feature points for which the calculated correlation value R _SAD is equal to or smaller than the threshold is a corresponding point, and determines that the similarity is high. In the example of FIG. 6C, the feature point 624 of the region of the wide-angle image 621 whose resolution is to be increased and the feature point 632 in the narrow-angle image 631 have a high correlation, and the similarity is high. Is determined. When such a high similarity feature point is included in a predetermined ratio or more, the narrow-angle image acquisition unit 113 determines that the narrow-angle image includes an area to be increased in resolution, and determines that narrow-angle image. Obtain and record. The narrow-angle images 641 to 645 shown in FIGS. 6D to 6H are recorded narrow-angle images.

  In the above-described example, the narrow-angle image corresponding to the region where the resolution is increased in the wide-angle image is realized by the imaging method in which the shift lens is shift-driven, but the narrow-angle image corresponding to the region where the resolution is increased by another method is realized. Shooting of a corner image may be realized. For example, based on the amount of camera shake when so-called hand-held shooting is being performed, the moving direction and moving amount of the shooting angle of view at the time of a narrow angle are calculated, and based on the moving method and the moving amount, an area for higher resolution is calculated. A corresponding narrow-angle image may be captured. In this case, for example, the narrow-angle image acquisition unit 113 determines the shooting angle of view at the narrow angle based on the focal length of the optical system 101 and the amount of camera shake detected by a camera shake detection sensor (not shown in FIG. 1). The movement direction and the movement amount of the are calculated. Further, the narrow-angle image acquisition unit 113 determines whether the range captured by the narrow-angle shooting angle of view is within the range where the wide-angle image is captured, based on the method and amount of movement of the narrow-angle shooting angle of view. Calculate which range corresponds to which range. Then, the narrow-angle image acquisition unit 113 captures a narrow-angle image when the shooting range in which the narrow-angle shooting angle of view has moved due to camera shake becomes a range corresponding to an area for increasing the resolution within the shooting range of the wide-angle image. Execute Also in this example, the similarity determination based on the comparison of the feature points as described above may be performed to determine whether or not the area of the wide-angle image to be increased in resolution is shown in the narrow-angle image. When hand-held shooting is performed, image quality may be degraded due to camera shake, and the amount of movement of the optical image on the imaging surface of the imaging unit 102 due to camera shake is, for example, the distance to the subject, the focal length of the optical system 101, and the like. , And the amount of camera shake. For this reason, when capturing a narrow-angle image, the narrow-angle image acquisition unit 113 performs capturing at a shutter speed that minimizes the effect on image quality due to camera shake. In general, if the image is taken at a speed higher than the shutter speed Tv calculated by the following equation (5), the image quality deterioration due to camera shake becomes almost invisible, so the narrow-angle image acquisition unit 113 calculates the value calculated by the equation (5). The reference value is set, and a narrow-angle image is shot at a faster shutter speed. Note that f in the equation is the focal length of the optical system 101.

  Tv = 1 / f Equation (5)

As described above, after the narrow-angle image corresponding to the area whose resolution is to be increased is obtained in S203, the process of the imaging apparatus proceeds to S204 in FIG.
In S204, the image generation unit 114 refers to the wide-angle image acquired by the wide-angle image acquisition unit 111, and generates a high-resolution image using the narrow-angle image acquired by the narrow-angle image acquisition unit 113. The process of generating the high-resolution image will be described with reference to FIGS. 7, 8A to 8C. FIG. 7 is a flowchart showing the details of the process of S204. FIG. 8A shows a wide-angle image 801 similar to that of FIG. 3A acquired by the wide-angle image acquisition unit 111, and FIG. 8B shows a region where high resolution is obtained by the narrow-angle image acquisition unit 113. FIG. 13 is a diagram showing one of the narrow-angle images 811 acquired corresponding to FIG. FIG. 8C is a diagram used to explain the correspondence between the wide-angle image 821 obtained by enlarging the wide-angle image 801 in FIG. 8A and the narrow-angle image 831 acquired by the narrow-angle image acquisition unit 113. Note that FIG. 8C shows only a part of the enlarged wide-angle image, which is actually an enlarged wide-angle image showing the same pattern as the wide-angle image 801 in FIG. 8A. .

  First, in S701 of the flowchart in FIG. 7, the image generation unit 114 determines the size of the subject in the narrow-angle image 811 shown in FIG. 8B and the size of the subject in the wide-angle image 801 shown in FIG. The wide-angle image is enlarged as shown in FIG. A wide-angle image 821 in FIG. 8C is an image in which only a part of the wide-angle image obtained by enlarging the wide-angle image 801 in FIG. 8A is extracted and the size of the subject in the wide-angle image 801 is a narrow-angle image. 831 so as to match the size of the subject.

Next, in step S702, the image generation unit 114 performs a process of extracting a feature point in the image from the wide-angle image 821 and the narrow-angle image 831 that are enlarged as illustrated in FIG. It is assumed that the feature point extraction uses the same method as that described in S203 described above.
Next, in step S703, the image generation unit 114 calculates the degree of correlation between the feature points extracted from the enlarged wide-angle image 821 and the narrow-angle image 831 to obtain a correspondence having a correspondence between the images. Search for points. In calculating the degree of correlation, the above-described equation (4) is used. Then, as a result of the corresponding point search processing, the feature point 822 and the feature point 832 in the enlarged wide-angle image 831 of FIG. 8C, and similarly, the feature point 823 and the narrow-angle image 831 in the wide-angle image 821 are obtained. It is assumed that the feature points 833 within are searched as corresponding points each having a corresponding relationship.

  Next, in step S704, the image generation unit 114 performs positioning between the wide-angle image 801 and the narrow-angle image 811 based on the corresponding points between the images searched in step S703, and generates the wide-angle image 801 and the narrow-angle image 811. Combine to generate a high-resolution image. As a process for generating a high-resolution image by image synthesis, for example, a method described in JP-A-2007-164258 can be used. In this case, the image generation unit 114 calculates a projective transformation formula using a combination of four or more corresponding feature points of both images to be combined, and transforms the narrow-angle image using the projective transform formula. Performs positioning for the wide-angle image. Then, the image generation unit 114 generates a high-resolution image by synthesizing the narrow-angle image, which has been subjected to the transformation and alignment by the projection conversion formula as described above, with a wide-angle image. When a high-resolution image is generated, if a corresponding narrow-angle image is not acquired in spite of an area to be increased in resolution, the image is generated using the wide-angle image as it is. The image generator 114 outputs the high-resolution image generated by the above-described processing.

In the above description, an example in which a high-resolution image is generated by combining a narrow-angle image with a wide-angle image has been described. However, without combining with the wide-angle image, only the narrow-angle image is referred to by referring to the wide-angle image. A high-resolution image may be generated by combining them so that they are connected. In this case, the image generation unit 114 calculates the positional relationship between the plurality of narrow-angle images based on the correspondence between the wide-angle image and the narrow-angle image, and connects the narrow-angle images based on the positional relationship. To generate a high-resolution image.
In addition, without performing the processing of combining the wide-angle image and the narrow-angle image and the combining processing of joining the narrow-angle images as described above, the correspondence relationship between the wide-angle image and the narrow-angle image is maintained, and an instruction to enlarge the wide-angle image is issued. , A narrow-angle image corresponding to the enlargement instruction may be displayed.

Further, in the present embodiment, when generating a high-resolution image, an example is described in which an image is generated using a wide-angle image when there is no corresponding narrow-angle image, but without using a wide-angle image, A narrow-angle image acquisition process may be performed, and a high-resolution image may be generated using the acquired narrow-angle image.
Further, in the present embodiment, when generating a high-resolution image, the corresponding feature points of the wide-angle image and the narrow-angle image are calculated and aligned. However, the feature points are not used, and an arbitrary range around the pixel of interest is used. A template may be used to calculate a corresponding area in block units and perform positioning.

  As described above, in the imaging apparatus according to the first embodiment, a region to be increased in resolution is determined based on a captured wide-angle image, a narrow-angle image corresponding to the region is acquired, and a high-resolution image is generated. ing. Thus, according to the imaging apparatus of the first embodiment, a narrow-angle image corresponding to a region to be increased in resolution in a wide-angle image can be efficiently acquired, and a burden on a user is reduced, and a high-resolution image can be acquired. it can.

<Second embodiment>
In the second embodiment, a region to be increased in resolution is determined based on a captured wide-angle image and a narrow-angle image, and a narrow-angle image used to generate a high-resolution image of the determined region is selected. An example of generating a high-resolution image will be described. The configuration of the imaging apparatus according to the second embodiment is the same as that of FIG.
FIG. 9 is a diagram showing functional blocks from capturing a wide-angle image and a narrow-angle image to generating a high-resolution image in the imaging device according to the second embodiment.

  As shown in FIG. 9, the functional blocks in the second embodiment include a wide-angle image acquisition unit 901, a narrow-angle image acquisition unit 902, an area determination unit 903, and an image generation unit 904. In the second embodiment, as in the case of the first embodiment, the functions of the wide-angle image acquisition unit 901 and the narrow-angle image acquisition unit 902 are performed by, for example, the system control unit 106 via the optical system control unit 105 and the optical system 101. Is realized by controlling Further, the functions of the region determination unit 903 and the image generation unit 904 are executed by the image processing unit 104, for example.

The wide-angle image acquisition unit 901 acquires a wide-angle image used as a reference image, as in the case of the first embodiment.
The narrow-angle image acquisition unit 902 acquires a plurality of narrow-angle images within the range of the angle of view when the wide-angle image acquisition unit 901 acquires a wide-angle image.
The region determination unit 903 executes a process of determining a region to be increased in resolution based on information on the wide-angle image acquired by the wide-angle image acquisition unit 901 and the narrow-angle image acquired by the narrow-angle image acquisition unit 902.
The image generation unit 904 selects a region to be increased in resolution from among the plurality of narrow-angle images acquired by the narrow-angle image acquisition unit 902 based on the information on the region to be increased in resolution determined by the region determination unit 903. Select a narrow-angle image corresponding to. Then, the image generation unit 904 generates and outputs a high-resolution image with reference to the wide-angle image acquired by the wide-angle image acquisition unit 901 using the selected narrow-angle image.

  FIG. 10 is a flowchart illustrating a flow of a processing procedure from acquisition of a wide-angle image and a narrow-angle image to generation of a high-resolution image in the imaging apparatus according to the second embodiment. Hereinafter, details of each processing from the acquisition processing of the wide-angle image and the narrow-angle image to the high-resolution image generation processing in the imaging device of the second embodiment will be described with reference to the functional blocks of FIG. 9 and the flowchart of FIG. .

In step S1001, the wide-angle image acquisition unit 901 acquires a wide-angle image in the same manner as in step S201 in FIG. The data of the wide-angle image is sent to the area determination unit 903 via the narrow-angle image acquisition unit 902.
Next, in step S1002, the narrow-angle image acquisition unit 902 captures a narrow-angle image used to generate a high-resolution image for a scene to be captured, and sends the data of the narrow-angle image to the region determination unit 903. . In the second embodiment, as described in the first embodiment, as a method for photographing a narrow-angle image, a photographing method in which a narrow-angle photographing field of view is moved by shift driving of a shift lens, a hand-held method, and the like. A shooting method based on the amount of camera shake at the time of shooting can be used.

  Next, in step S <b> 1003, the region determination unit 903 determines a region for which resolution is to be increased based on the wide-angle image acquired by the wide-angle image acquisition unit 901 and the narrow-angle image acquired by the narrow-angle image acquisition unit 902. I do. The region determination unit 903 according to the second embodiment determines an edge when comparing the same subject region in the wide-angle image and the narrow-angle image in addition to the information used to set the priority in S202 of the first embodiment. The priority is set using the difference in the intensity, and the area to be increased in resolution is determined. That is, in the second embodiment, in addition to the evaluation values of the contrast information, the saliency information, and the reliability information of the recognition result of the specific subject used in the first embodiment, a difference value of the edge strength is used as the evaluation value. Priority is set.

Hereinafter, the process of setting the priority using the difference value of the edge strength as the evaluation value will be described with reference to FIGS. 11 (a) and 11 (b). The setting of the priority according to each evaluation value such as contrast information, saliency information, and reliability information of the recognition result of a specific subject is the same as in the first embodiment, and a description thereof will be omitted.
FIG. 11A is a diagram illustrating a graph 1101 used when the area determination unit 903 sets a priority according to a difference value of edge strength. The graph 1101 has such a characteristic that, from the lower limit value to the upper limit value, the priority value increases as the difference value of the edge strength increases. A lower limit value of the priority is set for the difference value of the edge strength equal to or smaller than the lower limit value, and an upper limit value of the priority is set for the difference value of the edge strength equal to or larger than the upper limit value. FIG. 11B is a table 1102 in which the values of the weights for the contrast information, the saliency information, and the reliability information of the recognition result of the sky region are further added with the value of the weight given to the difference in the edge strength. In the second embodiment, similarly to the example of the first embodiment, each priority is a value expressed in a range of 0 to 1, and a priority value is assigned to each evaluation value for the result. .

The region determining unit 903 of the second embodiment calculates the difference value of the edge strength between the wide-angle image and the narrow-angle image. However, since the size of the subject shown in those images is different, first, the wide-angle image Is enlarged so that it is the same size as the subject. Next, the region determination unit 903 extracts a feature point from each of the enlarged wide-angle image and the narrow-angle image, obtains a correspondence relationship between the extracted feature points, and performs alignment. As the alignment processing, a method similar to S702 to S704 of the first embodiment is used. Further, the region determining unit 903 detects the edge strength by performing band-pass filtering on each of the aligned images, and uses the following equation (6) to determine the edge strength between the two images. Is calculated (absolute value). In the expression, e _A (i, j) is the edge strength of each coordinate of the enlarged wide-angle image, and e _B (i, j) is the edge strength of each coordinate of the narrow-angle image.

  Then, using the graph 1101 in FIG. 11A, the area determination unit 903 sets a priority according to the difference value E (absolute value) of the edge strength calculated by the equation (6). That is, the region determining unit 903 sets a higher priority as the edge strength difference value E increases, and conversely, sets a lower priority as the edge strength difference value E decreases.

Also in the second embodiment, as in the first embodiment described above, the area determination unit 903 weights the priority of each evaluation value in each block using the following equation (7), and Priority W (M, N) is calculated. Note that n in the expression is the number of acquired evaluation values, and b _n is the priority for each evaluation value. Further, α _n in the expression is a weight set for the priority for each evaluation value, and for example, the weight shown in the table 1102 in FIG. 11B is used. In the case of the second embodiment, since the difference value between the contrast information and the edge strength is important when determining the area to be increased in resolution, the weight for the contrast information and the edge strength difference is set to a large value in Table 1102. Have been. The weight may be specified by the user and determined.

  In addition, the region determining unit 903 of the second embodiment determines a region to be increased in resolution based on the priority set based on each evaluation value to which the above-described edge strength difference value is added. As specific examples, two evaluation values of contrast information and a difference value of edge strength will be described as an example, and a priority set based on these evaluation values will be described. The contrast information is an integral value of the edge strength as described above, and is an evaluation value indicating whether or not the texture of the subject area is large. The subject area having many textures is an area that is effective in increasing the resolution as described in the first embodiment. When the difference value of the edge strength in the second embodiment is large, a high-resolution image can be generated by using a narrow-angle image. Therefore, when the texture of the subject area is large and the difference value of the edge strength is large, the area determination unit 903 sets the priority to be high. On the other hand, when the texture of the subject region is small and the difference value of the edge strength is small, it is considered that a high-resolution image is not so effective, so the region determining unit 903 sets the priority to low. Set.

The description returns to the flowchart of FIG.
The image generation unit 904 selects a narrow-angle image to be used when generating a high-resolution image, based on the information on the area determined in S1003 as described above.
FIGS. 12A to 12J are diagrams illustrating a state in which a narrow-angle image to be used for generating a high-resolution image is selected based on a wide-angle image and a narrow-angle image captured and acquired, respectively. is there. FIG. 12A is a diagram illustrating a wide-angle image 1201 obtained by the wide-angle image obtaining unit 901. FIG. 12B is a diagram illustrating an example of a region determined to have a higher resolution in the wide-angle image 1211 by the region determination unit 903. The region 1212 and the region 1213 are determined to have a higher resolution. Let it be an area. Each of the images illustrated in FIGS. 12C to 12J is a diagram illustrating a part of the narrow-angle images 1221 to 1228 among the plurality of narrow-angle images acquired by the narrow-angle image acquiring unit 902. is there.

  The image generation unit 904 determines that the value of the priority W (M, N) among the priorities W (M, N) determined for each block by the above-described equation (7) is equal to or more than a predetermined threshold, for example. A block having a high priority W (M, N) is selected. Then, the image generation unit 904 selects a narrow-angle image including the selected block as an image used for generating a high-resolution image. Specifically, of the narrow-angle images 1221 to 1228 in FIGS. 12C to 12J, the images are selected in accordance with the regions 1212 and 1213 determined to have a higher resolution in FIG. 12B. Are narrow-angle images 1222, 1223, 1224, 1225, 1227, and 1228. It should be noted that the narrow-angle images 1221 and 1226 do not include a region for which the resolution is to be increased, and thus these narrow-angle images are not selected and are not used when generating a high-resolution image.

  In step S1005, the image generation unit 904 generates a high-resolution image using the narrow-angle image selected in step S1004 with reference to the wide-angle image acquired by the wide-angle image acquisition unit 111. The process for generating a high-resolution image is the same as that in the first embodiment, and a description thereof will not be repeated.

  As described above, in the second embodiment, not only a wide-angle image but also a narrow-angle image is acquired in advance, and a narrow-angle image corresponding to a region determined to have higher resolution is selected, and the selected narrow-angle image is obtained. Generate a high-resolution image based on the corner image. Thus, according to the imaging apparatus of the first embodiment, a narrow-angle image corresponding to a region to be increased in resolution in a wide-angle image can be efficiently acquired, and a burden on a user is reduced, and a high-resolution image can be acquired. it can.

Note that the image generation unit 904 may delete the narrow-angle image not used for the high-resolution image, or generate the high-resolution image using all the narrow-angle images without selecting the narrow-angle image. May be.
In addition, when a high-resolution image is generated, if a narrow-angle image corresponding to a region to be increased in resolution is not obtained, the image may be generated using the wide-angle image as it is.

  Each function in the functional block of each embodiment described above may be realized only by a hardware configuration, or may be realized by a software configuration by a CPU or the like executing a program. Further, a part may be realized by a hardware configuration and the rest may be realized by a software configuration. The program for this software configuration is not limited to the case where it is prepared in advance, but may be obtained from a recording medium such as an external memory (not shown) or obtained via a network (not shown).

  In the above-described embodiment, a digital camera is assumed as an example of the imaging device. However, various devices capable of capturing an image, such as a monitoring camera, an industrial camera, a vehicle-mounted camera, and a medical camera, such as a smartphone and a tablet terminal, are used. Is also applicable.

  The program for realizing one or more functions in the signal processing according to the present invention can be supplied to a system or an apparatus via a network or a storage medium, and is read and executed by one or more processors of a computer of the system or the apparatus. It can be realized by doing. Further, it can be realized by a circuit (for example, an ASIC) that realizes one or more functions.

  Each of the above-described embodiments is merely an example of an embodiment for carrying out the present invention, and the technical scope of the present invention should not be interpreted in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features.

  101: optical system, 102: imaging unit, 104: development processing unit, 105: optical system control unit, 106: system control unit, 109: recording unit, 111: wide-angle image acquisition unit, 112: area determination unit, 113: narrow Corner image acquisition unit 114: Image generation unit

Claims (22)

First acquisition means for acquiring a first image;
Determining means for determining, based on the first image, a region to be increased in resolution in the first image;
A second acquisition unit configured to acquire a second image by imaging an area having a smaller angle of view than the first image based on the area determined to have a higher resolution;
Generating means for referring to the first image and generating a high-resolution image by using the second image;
An image processing apparatus comprising: First acquisition means for acquiring a first image;
Second acquisition means for acquiring a second image having a smaller angle of view than the first image;
Determining means for determining, based on the first image and the second image, an area in the first image to be increased in resolution;
From among the plurality of second images acquired by the second acquisition means, a second image corresponding to the area determined to be increased in resolution is selected, and the second image is selected using the selected second image. Generating means for generating a high-resolution image;
An image processing apparatus comprising:   The image processing apparatus according to claim 2, wherein the generation unit deletes the unselected second image from the plurality of second images.   4. The image processing apparatus according to claim 1, wherein the determining unit obtains information on contrast from the first image, and determines the area to be increased in resolution based on the information on contrast. 5. The image processing apparatus according to any one of the preceding claims.   5. The image processing apparatus according to claim 4, wherein the determination unit determines the higher-resolution area in the first image as the higher-resolution area by giving priority to the higher-resolution area. 6.   The image processing apparatus according to claim 4, wherein the determination unit acquires an integrated value of edge strength for each region as the contrast information.   The said determination means acquires the information of the saliency which represents the degree of visual prominence from the said 1st image, and determines the area | region which raises the resolution based on the information of the saliency. Item 4. The image processing device according to any one of Items 1 to 3.   The image processing apparatus according to claim 7, wherein the determination unit determines the region to be made higher in resolution by giving priority to the region with the higher saliency in the first image.   9. The method according to claim 7, wherein the determining unit obtains, as the saliency, a correlation based on a difference between a feature of a subject area in the first image and a feature of a subject area around the subject area. 10. Image processing device.   The image processing apparatus according to claim 9, wherein the determining unit determines an area having a higher resolution in the first image such that an area having a lower correlation is prioritized.   4. The method according to claim 1, wherein the determining unit obtains reliability of a result of recognizing a specific region from the first image, and determines the region to be increased in resolution based on the reliability. 5. The image processing device according to any one of claims 1 to 4.   The image processing apparatus according to claim 11, wherein the determination unit does not determine an area having a higher degree of reliability in the first image as an area for which the resolution is to be increased so as to be prioritized.   The image processing apparatus according to claim 12, wherein the determination unit obtains, as the reliability of the result of recognition of the specific area, the reliability of the result of recognition of an empty area.   The determining means is configured to calculate a difference between an edge strength of a subject area in the first image and an edge strength of the same subject area in the second image as the subject area in the first image. The image processing apparatus according to claim 2, wherein the area for which the resolution is to be increased is determined on the basis of the following.   15. The image processing apparatus according to claim 14, wherein the determining unit determines the area to be increased in resolution by giving priority to an area having a larger difference in the edge strength in the first image. apparatus.   When the second image corresponding to the area whose resolution is to be increased has not been acquired, the generation unit generates an image of an area corresponding to the second image that has not been acquired among the first images. The image processing apparatus according to claim 1, wherein the high-resolution image is generated using the image processing apparatus.   The second acquisition unit shifts the shooting angle of view within the angle of view of the first image in a state where a shooting angle of view when capturing an image is set to the angle of view of the second image. The image processing apparatus according to any one of claims 1 to 16, wherein an image photographed by moving the image processing apparatus is acquired as the second image.   The second acquisition unit is configured to set a shooting angle of view of the second image by a shake of an imaging device that shoots an image in a state in which a shooting angle of view when capturing an image is set to the angle of view of the second image. The image processing apparatus according to claim 1, wherein an image captured when moving is acquired as the second image.   19. The image processing apparatus according to claim 18, wherein the second obtaining unit obtains the second image captured using a shutter speed calculated according to a focal length of the imaging device. . An image processing method executed by the image processing device,
A first acquisition step of acquiring a first image;
A determining step of determining, based on the first image, a region to be increased in resolution in the first image;
A second acquisition step of acquiring a second image by imaging an area having a smaller angle of view than the first image, based on the area determined to have a higher resolution;
A generation step of generating a high-resolution image by using the second image with reference to the first image;
An image processing method comprising: An image processing method executed by the image processing device,
A first acquisition step of acquiring a first image;
A second acquisition step of acquiring a second image having a smaller angle of view than the first image;
Determining a region in the first image to be increased in resolution based on the first image and the second image;
From among the plurality of second images acquired in the second acquisition step, a second image corresponding to the region determined to be increased in resolution is selected, and the second image is selected using the selected second image. A generation step of generating a high-resolution image;
An image processing method comprising:   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 19.

Images