WO2012147523A1 - Dispositif d'imagerie et procédé de génération d'image - Google Patents

Dispositif d'imagerie et procédé de génération d'image Download PDF

Info

Publication number
WO2012147523A1
WO2012147523A1 PCT/JP2012/059977 JP2012059977W WO2012147523A1 WO 2012147523 A1 WO2012147523 A1 WO 2012147523A1 JP 2012059977 W JP2012059977 W JP 2012059977W WO 2012147523 A1 WO2012147523 A1 WO 2012147523A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
addition
unit
pixel value
added
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2012/059977
Other languages
English (en)
Japanese (ja)
Inventor
愼一 今出
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Olympus Corp
Original Assignee
Olympus Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Olympus Corp filed Critical Olympus Corp
Publication of WO2012147523A1 publication Critical patent/WO2012147523A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio

Definitions

  • the present invention relates to an imaging device, an image generation method, and the like.
  • Some modern digital cameras and video cameras can be used by switching between still image shooting mode and movie shooting mode. For example, there is one that can shoot a still image with a higher resolution than a moving image by a user operating a button during moving image shooting.
  • the method of switching between the still image shooting mode and the moving image shooting mode has a problem that when a user notices a photo opportunity, a decisive moment is often already missed.
  • Patent Documents 1 and 2 disclose a technique for synthesizing a high-resolution image from a low-resolution image acquired by pixel shift.
  • this method requires imaging by pixel shift, which complicates the camera configuration.
  • an imaging device an image generation method, and the like that can acquire a high-resolution image from a low-resolution moving image with simple processing.
  • an image acquisition unit that acquires a captured image
  • an addition unit that is a unit for acquiring an added pixel value is set for each of a plurality of pixels of the captured image, and a pixel value included in the addition unit is set.
  • An addition image generation unit that obtains the addition pixel value by weighted addition, obtains an addition image based on the obtained addition pixel value, a compression processing unit that compresses the addition image, and a decompression that decompresses the compressed addition image A processing unit; an estimation calculation unit that estimates a pixel value of the captured image based on the expanded added image; and an image output unit that outputs a high-resolution image based on the pixel value estimated by the estimation calculation unit;
  • the addition image generating unit sequentially shifts the addition unit to obtain the first to n-th addition images (n is a natural number of 2 or more (including the value)) as the addition images.
  • the compression processing unit An average of 1st to nth added images is obtained as an average image, and the average image and the mth added image (m is a natural number equal to or less than n (including the value)) among the 1st to nth added images ) As the m-th difference image, the obtained average image and the m-th difference image are compressed, and the decompression processing unit calculates the compressed average image and the m-th difference image.
  • the present invention relates to an imaging apparatus that decompresses and obtains the first to n-th added images.
  • the addition units are sequentially pixel-shifted to obtain the first to nth addition images, and the average image of the first to nth addition images is obtained. Then, an mth difference image that is a difference between the average image and the mth addition image is obtained, and the average image and the mth difference image are compressed. Then, the compressed images are expanded to obtain first to n-th added images, and a captured image is estimated based on the first to n-th added images.
  • the addition image generation unit may sequentially set the addition unit horizontally by one pixel at a time to set the first to nth positions, and to set the first to nth positions.
  • the first to n-th added images may be acquired in step S1, and the addition unit of the m-th position and the (m + 1) -th position among the first to n-th positions may include a common pixel.
  • the compression processing unit may obtain an average value of the addition pixel values at the first to n-th positions as a pixel value of the average image;
  • a difference image generation unit that obtains a difference value between the pixel value and the added pixel value at the m-th position as a pixel value of the m-th difference image.
  • the pixel value of the average image can be obtained by averaging the pixel values of the first to n-th added images. Further, the pixel value of the mth difference image can be obtained from the difference value between the pixel value of the average image and the pixel value of the mth addition image.
  • a low-resolution moving image generation unit that generates a low-resolution moving image is included, the captured image is an image of an RGB Bayer array, and the addition unit is four adjacent pixels.
  • the RGB pixel values of the low-resolution moving image may be obtained by estimation based on the first to third addition images among the first to fourth addition images.
  • an RGB color image can be generated from the added image in which RGB is mixed by addition. Thereby, it is possible to easily obtain a color moving image without restoring the captured image.
  • the compression processing unit may reversibly compress the m-th difference image by entropy coding.
  • the difference image is a difference between the average image and the addition image, the entropy of the pixel value can be made smaller than that of the captured image. Therefore, the difference image can be compressed at a high compression rate by entropy encoding.
  • the estimation calculation unit obtains a pixel value of the RGB Bayer array of the captured image by estimation
  • the image output unit includes: A high-resolution moving image may be output based on an image obtained by demosaicing the estimated pixel values of the RGB Bayer array.
  • the image processing apparatus may further include a noise reduction processing unit that performs adaptive noise reduction processing that adaptively adjusts the moving average range according to the change width of the pixel value with respect to the mth difference image. Good.
  • the difference image is the difference between the average image and the added image
  • the amplitude of the pixel value is smaller than that of the captured image. Therefore, noise can be effectively reduced by performing adaptive noise reduction processing on the difference image.
  • the image acquisition unit acquires a captured image of the first frame as a reference captured image, and a difference between the captured image of the second frame before or after the first frame and the reference captured image.
  • the added image generation unit acquires the first to nth added images based on the reference captured image and the first to nth added images based on the differential captured image. May be.
  • the inter-frame difference image becomes a difference between the added images having high similarity.
  • generated from the difference image between frames can be improved rather than the average image produced
  • the estimation calculation unit Is a difference value between the added pixel value of the first position and the added pixel value of the second position, and the added pixel value of the first area excluding the overlapping area from the addition unit of the first position
  • a relational expression between the first intermediate pixel value that is and the second intermediate pixel value that is the addition pixel value of the second region excluding the overlap region from the addition unit of the second position, and the difference value The first intermediate pixel value is estimated using the relational expression, and the pixel value of each pixel included in the addition unit is obtained using the estimated first intermediate pixel value. May be.
  • the estimation calculation unit may include an intermediate included in the intermediate pixel value pattern when successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern.
  • a relational expression between pixel values is represented using the addition pixel value
  • the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values is compared with the addition pixel value, and similarity is evaluated, Based on the similarity evaluation result, an intermediate pixel value included in the intermediate pixel value pattern may be determined so that the similarity is the highest.
  • the intermediate pixel value can be obtained by estimation based on a plurality of added pixel values acquired by pixel shifting while the addition unit is superimposed.
  • a captured image is acquired, an addition unit, which is a unit for acquiring an added pixel value, is set for each of a plurality of pixels of the captured image, and a pixel value included in the added unit is weighted. Adding to obtain the added pixel value, obtaining an added image based on the obtained added pixel value, compressing the added image, decompressing the compressed added image, and based on the decompressed added image.
  • the addition units are sequentially shifted to the first to n-th added images (n is 2 or more).
  • the present invention relates to an image generation method for obtaining the first to n-th added images by decompression.
  • FIG. 1 is an explanatory diagram of a first data compression method.
  • FIG. 2 is an example of a Huffman code table.
  • FIG. 3 shows a first configuration example of the imaging apparatus.
  • FIG. 4 is a first code allocation example of fusion image data.
  • FIG. 5 shows a second code allocation example of the fused image data.
  • FIG. 6 shows a second configuration example of the imaging apparatus.
  • FIG. 7 shows an example of code assignment of fusion image data in the second configuration example of the imaging apparatus.
  • FIG. 8 shows a modified configuration example of the imaging apparatus.
  • FIG. 9 is an explanatory diagram of a low-resolution moving image generation method.
  • FIG. 10 is an explanatory diagram of a low-resolution moving image generation method.
  • FIG. 11 is an explanatory diagram of a low resolution moving image generation method.
  • FIG. 12 shows a third configuration example of the imaging apparatus.
  • FIG. 13 shows an example of an imaging process for simultaneously acquiring a high-resolution still image and a low-resolution still image.
  • FIG. 14 is a sequence example of an imaging process for simultaneously acquiring a high-resolution still image and a low-resolution still image.
  • FIG. 15 shows an example of data reconstruction before data compression.
  • FIG. 16 shows an example of a data unit set in data reconstruction.
  • FIG. 17 shows an example of the superimposed shift addition value in units of reconstruction data.
  • FIG. 18 is a configuration example of the imaging unit when adding four pixels of the same color.
  • FIG. 19 is an explanatory diagram of adaptive noise reduction processing.
  • FIG. 20A and FIG. 20B are explanatory diagrams of adaptive noise reduction processing.
  • FIG. 20A and FIG. 20B are explanatory diagrams of adaptive noise reduction processing.
  • FIG. 20A and FIG. 20B are explanatory diagrams of adaptive noise reduction processing.
  • FIG. 21 is an explanatory diagram of adaptive noise reduction processing.
  • FIG. 22 is an explanatory diagram of adaptive noise reduction processing.
  • FIG. 23 is an explanatory diagram of adaptive noise reduction processing.
  • FIG. 24 shows a fourth configuration example of the imaging apparatus.
  • FIG. 25 is a schematic diagram of processing performed by the fourth configuration example of the imaging apparatus.
  • FIG. 26 is a modified configuration example of the imaging unit in the fourth configuration example of the imaging apparatus.
  • 27A and 27B are explanatory diagrams of estimated pixel values and intermediate pixel values.
  • FIG. 28 is an explanatory diagram of restoration estimation processing.
  • FIG. 29 is an explanatory diagram of restoration estimation processing.
  • FIG. 30 is an explanatory diagram of restoration estimation processing.
  • Digital camera and video camera products have a digital camera that mainly shoots still images with a video shooting function, or a video camera that mainly shoots video with a still image shooting function. There is something. If such a device is used, there is the convenience that a still image and a moving image can be shot with a single device.
  • the captured image by pixel addition to obtain a plurality of low-resolution images A 1 ⁇ A 4 the average image of the plurality of low-resolution images A 1 ⁇ A 4 Find M. Then, the difference image D 1 ⁇ D 4 of the average image M and the low-resolution image A 1 ⁇ A 4 asked, compressing the difference image D 1 ⁇ D 4 for example by entropy coding.
  • the difference images D 1 to D 4 the pixel values are considered to be unevenly distributed near zero, so that the compression rate can be improved.
  • a technique for performing so-called super-resolution processing on a low-resolution image captured by pixel shift is conceivable.
  • addition reading is performed while sequentially shifting the position, and a high-definition image is temporarily assumed based on the plurality of position-shifted images. Then, the assumed image is degraded to generate a low resolution image, which is compared with the original low resolution image, and the high definition image is estimated so that the difference is minimized.
  • an ML (Maximum-Likelihood) method As this super-resolution processing, an ML (Maximum-Likelihood) method, a MAP (Maximum-A-Posterior) method, a POCS (Projection-Onto-Convex-Set) method, an IBP (Iterative Back-Projection) method, and the like are known.
  • ML Maximum-Likelihood
  • MAP Maximum-A-Posterior
  • POCS Projection-Onto-Convex-Set
  • IBP Iterative Back-Projection
  • Patent Document 1 For example, in Patent Document 1 described above, low-resolution images that have been pixel-shifted during moving image shooting are sequentially captured in time series, and a plurality of low-resolution images are combined to assume a high-resolution image.
  • a technique for performing the above-described super-resolution processing on a high-resolution image and estimating a high-resolution image with high likelihood is disclosed.
  • Patent Document 2 a plurality of pixel-shifted low-resolution images are picked up, temporary pixels constituting the high-resolution image to be obtained are set as sub-pixels, and the average value of the sub-pixels is taken as an image.
  • a technique for estimating the pixel value of the sub-pixel so as to match the pixel value of the low-resolution image that has been made is disclosed. In this method, initial values of a plurality of sub-pixels are set, pixel values of sub-pixels excluding sub-pixels to be calculated are subtracted from pixel values of the low-resolution image, and pixel values are sequentially obtained for adjacent pixels. Apply.
  • the added pixel values a (1) 00 to a (4) 01 of the low resolution images A 1 to A 4 are shifted by overlapping pixels, and adjacent added pixel values include common pixels.
  • the process of estimating a high-resolution image can be simplified by using the added pixel value shifted by the superimposed pixel.
  • the frame is, for example, a timing at which an image is captured by an image sensor or a timing at which one captured image is processed in image processing.
  • one image in the image data is also referred to as a frame as appropriate.
  • a captured image of an RGB Bayer array (hereinafter referred to as “high-definition frame image fx”) is acquired by reading all pixels in a frame fx.
  • An addition unit (a pixel group to be added) composed of four pixels is set in the high-definition frame image fx, and the pixel values of the addition unit are weighted and added. At this time, the addition unit is shifted horizontally or vertically while superimposing one pixel, and four pixel addition images A 1 to A 4 are generated.
  • V ij is the pixel value of the address (i, j) in the high-definition frame image.
  • the addition positions of the generated pixel addition images A 1 to A 4 are matched, and the average image M is generated by superposing the four pixel addition images and taking the addition average of the values at the same position.
  • the four-pixel added values of the pixel-added images A 1 to A 4 are respectively represented as ⁇ a (1) ij , a (2) (i + 1) j , a (3) (i + 1) (j + 1) , a (4) i ( j + 1) ⁇
  • the pixel value a M ij of the average image M is expressed by the following equation (2).
  • a M ij [a (1) ij + a (2) (i + 1) j + a (3) i (j + 1) + a (4) (i + 1) (j + 1) ] / 4 (2)
  • the difference (projection onto the direction vector (1, ⁇ 1)) between the average image M and the pixel-added images A 1 to A 4 is set as difference images D 1 to D 4 , respectively.
  • the difference images D 1 to D 4 and the average image M are combined to form fused image data F (M, D 1 to D 4 ).
  • the 4-pixel addition difference value constituting the difference images D 1 to D 4 is represented as ⁇ a D1 ij , a D2 ij , a D3 ij , a D4 ij ⁇ , it can be expressed as the following expression (3).
  • fused still images and moving fused image data F (M, D 1 ⁇ D 4) generates, may be recorded it, the recorded F (M, D 1 ⁇ D 4 )
  • “high-definition still image data” or “moving image data” can be appropriately generated.
  • the difference that is, projection onto the direction vector (1, ⁇ 1)
  • the present embodiment is not limited to this.
  • the principal component axis straight line with the largest variance
  • the axis orthogonal to the principal component axis is obtained. It goes without saying that the data can be further compressed by using the projected value.
  • Entropy coding is a compression technique in which the occurrence probability of a pixel value is obtained and a short code length is assigned from the one with the highest occurrence probability.
  • the shortest code length is assigned to the average value of the pixel value data of the average image M and the code length is increased as the value is farther from the average value. That is, since the pixel value data of the difference images D 1 to D 4 has an average of zero and is distributed with the occurrence probability of zero as a peak, the shortest code may be assigned to the zero value. It is also effective to perform nonlinear quantization on the pixel value data of the average image M and the difference images D 1 to D 4 according to the probability density.
  • the average value ⁇ M ij is obtained for each predetermined image area by the following equation (4), and the difference between the average value ⁇ M ij and each added average value ⁇ a M ij ⁇ .
  • the value ⁇ a ′ M ij ⁇ is obtained by the following equation (5).
  • the difference value ⁇ a ′ M ij ⁇ is used as the data of the average image M again, and Huffman coding may be applied.
  • the difference value ⁇ a ′ M ij ⁇ can form an occurrence distribution of values universally centered on zero.
  • a ′ M ij a M ij ⁇ M ij (5)
  • (h0, v0) is the start point of the area where the average value ⁇ M ij is calculated
  • (h, v) is the end point of the area where the average value ⁇ M ij is calculated.
  • the area for calculating the average value ⁇ M ij is the entire average image M.
  • the area for calculating the average value ⁇ M ij and the number of average values a M ij used for calculating the average value ⁇ M ij may be determined by evaluating the balance with the data compression rate.
  • the fixed value m AVR shown in the following equation (6) may be applied, and may be handled as look-ahead information without being treated as fluctuation data. In this case, the average value m AVR need not be recorded as data.
  • the following expression (6) is applied to the case of the weighted 4-pixel addition value defined by the above expression (1), and it should be noted that an appropriate value is appropriately set depending on the number of added pixels and the weighting method. is necessary.
  • m AVR v max ⁇ ⁇ 1+ (1 / r) + (1 / r) + (1 / r 2 ) ⁇ / 2 (6)
  • r is a weighting parameter
  • v max is a maximum value that defines the pixel value.
  • Second Data Compression Method Next, a second data compression method for further reducing the data amount of the above-described difference images D 1 to D 4 will be described.
  • an appropriate axis that minimizes the occurrence distribution variance is selected. If the value obtained by projecting the difference value on the axis is obtained, the data can be further compressed.
  • the axis with the smallest dispersion of the occurrence distribution is, for example, the axis (1, 1) orthogonal to the axis (1, ⁇ 1) along the occurrence distribution having a negative correlation.
  • the recording data may be the difference addition value ⁇ 1 , ⁇ 2 ⁇ instead of the 4-pixel addition difference value ⁇ a D3 ij , a D4 ij ⁇ .
  • the image data composed of the difference addition values ⁇ 1 , ⁇ 2 ⁇ is expressed as ⁇ 1 , ⁇ 2 ⁇
  • the fusion image data to be finally recorded is F (M, a D1 ij , a D2 ij , ⁇ 1 , ⁇ 2 ).
  • High-definition still image reproduction method In the first data compression method described above, fusion image data F (M, D 1 to D 4 ) is obtained. A method for reconstructing and estimating a high-definition still image (frame image fx in FIG. 1) from F (M, D 1 to D 4 ) will be described.
  • the obtained 4-pixel addition value ⁇ a (1) ij , a (2) (i + 1) j , a (3) (i + 1) (j + 1) , a (4) i (j + 1) ⁇ is one high definition
  • the fused image data F (M, a D1 ij , a D2 ij , ⁇ 1 , ⁇ 2 ) is obtained.
  • a method for restoring and estimating a high-definition still image from F (M, a D1 ij , a D2 ij , ⁇ 1 , ⁇ 2 ) will be described.
  • the addition unit is shifted four times to obtain the addition images A 1 to A 4 and the average image M and the difference images D 1 to D 4 are obtained.
  • the form is not limited to this.
  • the addition unit A 9 may be shifted nine times to obtain the added images A 1 to A 9 and the average image M and the difference images D 1 to D 9 may be obtained.
  • Imaging Device FIG. 3 shows a configuration example of an imaging device that performs the above-described data compression processing and high-definition frame image fx restoration processing.
  • the imaging apparatus includes an imaging unit 10 that performs imaging and data compression processing, and an image processing unit 20 that performs restoration processing of a high-definition image.
  • the image processing unit 20 may be built in the camera body or may be configured by an external information processing apparatus such as a PC.
  • the imaging unit 10 includes a lens 110, an imaging element 120 (imaging sensor), an addition image generation unit 130 (superimposition shift weighted addition image generation unit), a compression processing unit 140, and a compressed data recording unit 150 (fusion compression data). Recording unit), a low-resolution moving image generation unit 160 (standard moving image generation unit), a moving image data compression unit 170, a moving image data recording unit 180, and a monitor display unit 190.
  • the lens 110 forms an image of the subject 100.
  • the image sensor 120 captures the formed subject image.
  • An analog signal obtained by imaging is converted into a digital signal by an A / D converter (not shown).
  • the addition image generation unit 130 adds the pixel values of the captured image while shifting the pixels, and generates pixel addition images A 1 to A 4 from the captured image.
  • the compression processing unit 140 generates an average image M and difference images D 1 to D 4 from the pixel addition images A 1 to A 4, and performs a process of compressing the average image M and the difference images D 1 to D 4 .
  • the compression processing unit 140 includes a difference image generation unit 141, an average image generation unit 142, an entropy encoding unit 143, and a data compression unit 144.
  • the average image generation unit 142 generates an average image M from the pixel addition images A 1 to A 4 .
  • the difference image generation unit 141 generates difference images D 1 to D 4 from the pixel addition images A 1 to A 4 and the average image M.
  • the entropy encoding unit 143 compresses the difference images D 1 to D 4 by lossless compression such as the above-described entropy encoding.
  • the data compression unit 144 compresses the average image M by irreversible compression such as M-JPEG or JPEG-XR.
  • the compressed data recording unit 150 records the compressed image data.
  • the compressed data recording unit 150 includes a difference data recording unit 151 that records the compressed difference images D 1 to D 4 and an average image data recording unit 152 that records the compressed average image M.
  • the low-resolution moving image generation unit 160 generates a real-time low-resolution moving image by processing described later with reference to FIG.
  • the moving image data compression unit 170 compresses a low resolution moving image by using, for example, AVCHD (H.264), M-JPEG, JPEG-XR, or the like.
  • the moving image data recording unit 180 records a compressed low resolution moving image.
  • the monitor display unit 190 displays a real-time or reproduced low-resolution moving image.
  • the image processing unit 20 includes an expansion processing unit 205, an added image noise reduction processing unit 220, an estimation calculation unit 230 (high-definition image restoration estimation unit), a high-definition still image generation unit 240, a high-definition moving image generation unit 250, and a high-quality standard.
  • a moving image generation unit 260, a moving image compressed data decompression unit 270, a standard moving image generation unit 280, an image output unit 290, and an image selection unit 295 are included.
  • the decompression processing unit 205 decompresses the compressed data and reproduces the added images A 1 to A 4 .
  • the expansion processing unit 205 includes a compressed data expansion unit 200 and an addition image reproduction unit 210 (weighted addition image reproduction unit).
  • the compressed data decompression unit 200 performs a process of decompressing the compressed average image M and the difference images D 1 to D 4 .
  • the added image reproduction unit 210 performs a process of reproducing the added images A 1 to A 4 from the average image M and the difference images D 1 to D 4 .
  • the added image noise reduction processing unit 220 performs adaptive noise reduction processing, which will be described later with reference to FIG. 19 and the like, on the added images A 1 to A 4 .
  • the estimation calculation unit 230 restores the high-definition frame image fx by estimation based on the added images A 1 to A 4 .
  • the restored image is a Bayer array RAW image. This estimation calculation will be described later with reference to FIG.
  • the high-definition still image generation unit 240 performs demosaicing processing on the restored image with the Bayer array, and performs image processing such as gradation correction processing on the image to generate a high-definition still image. At this time, a still image at the timing selected by the image selection unit 295 is generated. The timing is selected by a user instruction. The user may select the timing by viewing the moving image displayed on the monitor display unit 190, or may select the timing by viewing the moving image output by the image output unit 290.
  • the high-definition moving image generation unit 250 performs demosaicing processing on the restored moving image with the Bayer array, and performs image processing such as gradation correction processing on the moving image to generate a high-definition moving image.
  • the high-quality standard video generation unit 260 downsamples the high-definition video and generates, for example, a video with the number of high-definition pixels as a high-quality standard video.
  • the standard moving image generation unit 280 generates a low resolution moving image by processing described later with reference to FIG. 9 and the like using the addition images A 1 to A 4 reproduced by the addition image reproduction unit 210, and the low resolution moving image is standardized. Output as a movie.
  • the standard moving image is, for example, a moving image having the number of high-definition pixels.
  • the moving image compression data expansion unit 270 may expand the compressed low resolution moving image from the moving image data recording unit 180, and the standard moving image generation unit 280 may output the expanded low resolution moving image as a standard moving image. .
  • the image output unit 290 outputs a high-definition still image, a high-definition moving image, a high-quality standard moving image, and a standard moving image to, for example, a display device or a printer.
  • FIG. 4 shows a first code allocation example.
  • 24 bits are assigned to the pixel value of the coordinates (i, j) of the fused image data F (M, D 1 to D 4 ).
  • 8 bits are assigned to the pixel value a M ij of the average image M
  • 4 bits are assigned to the pixel values a D1 ij to a D4 ij of the difference images D 1 to D 4 .
  • the difference pixel values ⁇ a D1 ij , a D2 ij , a D3 ij , a D4 ij ⁇ has a common average value a M ij .
  • the average value a M ij is canceled, the difference value between adjacent pixel values can be completely restored even if the average image is degraded.
  • the method of specifying and estimating the original pixel value relationship based on the difference value of the adjacent addition values is a restoration estimation process that is very convenient for the compression method of the present method.
  • FIG. 6 shows a second configuration example of the imaging device as a configuration example in the case of performing reversible compression using the average value m AVR described in the above equation (6).
  • the imaging device includes an imaging unit 10 and an image processing unit 20.
  • the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
  • the imaging unit 10 includes a lens 110, an image sensor 120, an addition image generation unit 130, a difference image generation unit 141, an average image generation unit 142, an average value calculation unit 145, a subtraction unit 146, a lossless compression encoding unit 147, and compressed data recording.
  • Unit 150 low-resolution moving image generation unit 160, moving image data compression unit 170, moving image data recording unit 180, and monitor display unit 190.
  • the components of the image processing unit 20 are the same as those of the image processing unit 20 in FIG.
  • the average value calculation unit 145 calculates the average value m AVR described in the above equation (6) based on the average image M.
  • the subtracting unit 146 subtracts the average value m AVR from the pixel value a M ij of the average image M, and outputs the pixel value md ij after the subtraction.
  • the lossless compression encoding unit 147 performs lossless compression on the data of the difference images D 1 to D 4 and the data of the pixel value md ij .
  • the compressed data recording unit 150 records the compressed data and the average value m AVR .
  • the compressed data decompression unit 200 decompresses the difference images D 1 to D 4 , the pixel value md ij, and the average value m AVR from the compressed data.
  • the addition image reproduction unit 210 reproduces the average image M from the pixel value md ij and the average value m AVR, and reproduces the addition images A 1 to A 4 from the average image M and the difference images D 1 to D 4 .
  • FIG. 7 shows an example of code allocation of the fused image data F (M, D 1 to D 4 ) in the second configuration example.
  • the average value m AVR is arranged at the head of the pixel values md ij, 8 bits each are assigned to the mean value m AVR and pixel values md ij.
  • 4 bits are assigned to each of the pixel values d ij of the difference images D 1 to D 4 .
  • the second configuration example even when the occurrence distribution probability of the pixel value md ij of the average image M is not biased, it is possible to reduce the entropy because the bias can be created by converting to the 4-pixel different color addition image. A compression effect in lossless compression can be expected. Further, by adopting the difference between the average value m AVR and the pixel value md ij as data, a stable data value can be obtained without greatly depending on the position (average value) of the occurrence probability distribution, and a compression effect is expected. it can. Also, since all are lossless compression, there is no data degradation before restoration estimation.
  • FIG. 8 shows a modified configuration example of the imaging apparatus when the added images A 1 to A 4 are compressed.
  • the imaging device includes an imaging unit 10 and an image processing unit 20.
  • the imaging unit 10 includes a lens 110, an image sensor 120, an added image generation unit 130, a data compression unit 148, a data recording unit 155, a low resolution moving image generation unit 160, a moving image data compression unit 170, and a monitor display unit 190.
  • the components of the image processing unit 20 are the same as those of the image processing unit 20 in FIG.
  • the data compression unit 148 compresses the added images A 1 to A 4 by irreversible compression such as M-JPEG or JPEG-XR.
  • the data recording unit 155 records the added images A 1 to A 4 compressed by the data compression unit 148 and the low resolution moving image compressed by the moving image data compression unit 170.
  • the imaging apparatus includes an image acquisition unit (for example, the imaging device 120), the addition image generation unit 130, the compression processing unit 140, the expansion processing unit 205, and the estimation calculation unit 230.
  • An image output unit 290 is included.
  • the image acquisition unit acquires a captured image fx (high-definition frame image).
  • the addition image generation unit 130 sets an addition unit, which is a unit for acquiring an addition pixel value (for example, a (1) ij ), for each of a plurality of pixels (for example, every four pixels) of the captured image, and is included in the addition unit.
  • the pixel values are weighted and added (the above formula (1)) to obtain an added pixel value, and added images A 1 to A 4 based on the obtained added pixel value are obtained.
  • the compression processing unit 140 compresses the added images A 1 to A 4 .
  • the decompression processing unit 205 decompresses the compressed added images A 1 to A 4 .
  • the estimation calculation unit 230 estimates the pixel value v ij of the captured image fx based on the expanded added images A 1 to A 4 .
  • the image output unit 290 outputs a high resolution image based on the estimated pixel value v ij .
  • the addition image generation unit 130 sequentially acquires the first to fourth (first to nth in a broad sense) addition images A 1 to A 4 by sequentially shifting the addition unit.
  • the compression processing unit 140 obtains the average of the first to fourth addition images A 1 to A 4 as the average image M (the above equation (2)), and the average image M and the m-th addition image A m (m Is a natural number less than or equal to n (including its value) as the m-th difference image D m (the above equation (3)), and the obtained average image M and difference image D m are compressed.
  • Decompression processing unit 205 decompresses the compressed average image M and the difference image D m by determining the added image A 1 ⁇ A 4.
  • the captured image fx can be efficiently compressed. That is, although the captured image fx and the four added images A 1 to A 4 have the same number of pixels, the entropy of the pixel value of the difference image D m can be made smaller than that of the captured image fx as described above. The rate can be improved. As a result, the captured image fx can be restored from the high-compression-rate image data to extract a high-definition still image at an arbitrary timing. Further, since the added images A 1 to A 4 are generated from one captured image fx, a restored image with a high temporal resolution with little blurring can be obtained even with moving objects.
  • the data can be reduced by being able to reduce the resolution at high speed immediately after the sensor, the data processing load in the subsequent stage can be reduced, and a high-speed image processing system for high-definition images without high-speed data processing. Can be built. In addition, it is possible to provide a technology that can be easily applied to existing systems.
  • the captured image fx can be reproduced from the compressed data with a simple process. That is, the added images A 1 to A 4 are image data obtained by superposition shift addition, and a restoration estimation process described later can be applied.
  • This restoration estimation process can simplify the process of estimating a high-resolution image from a low-resolution image as compared with Patent Documents 1 and 2 described above.
  • the addition image generation unit 130 sequentially shifts the addition unit one pixel at a time horizontally or vertically (i-axis direction or j-axis direction).
  • the position (for example, coordinates (0,0), (1,0), (1,1), (0,1)) is set, and the added images A 1 to A 4 are respectively displayed at the first to fourth positions. get.
  • the addition unit of the m-th position and the m + 1-th position (for example, (0, 0), (1, 0)) includes a common pixel (v 10 , v 11 ).
  • the compression processing unit 140 includes an average image generation unit 142 and a difference image generation unit 141.
  • the average image generation unit 142 calculates the average value a M ij of the added pixel values at the first to fourth positions as the pixel value of the average image.
  • the difference image generation unit 141 uses the pixel value a M ij of the average image and the difference value (a D1 ij ) of the added pixel value (for example, a (1) ij ) of the m-th position as the pixel of the m-th difference image. Calculate as a value.
  • the pixel value of the average image M can be obtained by averaging the addition pixel values of the addition images A 1 to A 4 .
  • the pixel values of the difference images D 1 to D 4 can be obtained from the difference value between the pixel value of the average image and the addition pixel values of the addition images A 1 to A 4 .
  • the compression processing unit 140 losslessly compresses the mth difference image Dm by entropy coding.
  • the captured image fx is an RGB Bayer array image.
  • the estimation calculation unit 230 obtains the pixel value v ij of the RGB Bayer array of the captured image fx by estimation.
  • the image output unit 290 outputs a high-resolution moving image based on an image obtained by demosaicing the estimated RGB Bayer array pixel values v ij .
  • the high-definition moving image generation unit 250 performs demosaicing processing, performs image processing such as gradation conversion on the image, generates a high-resolution moving image
  • the image output unit 290 outputs the high resolution moving image to a display device or the like.
  • a high-resolution moving image is a moving image having the same number of pixels as the number of pixels of the captured image.
  • the low resolution moving image generation unit 160 and the standard moving image generation unit 280 simply generate a low resolution moving image from the added images A 1 to A 4 . A method for generating this low-resolution moving image will be described in detail.
  • low-resolution moving image data is generated using three addition images A 1 to A 3 among the addition images A 1 to A 4 .
  • the added image A 1 color relationship of weighted using four pixel sum value near that matches the non-detection value a (1) 12, the estimated value of the undetected value a (1) 12 Perform approximate calculation.
  • This estimated value a (1) 12 is obtained as a 4-pixel addition value on the assumption that the weight of the R color pixel is set to the reference “1”.
  • non-detection value ⁇ a (1) 12 ⁇ 1 ⁇ R 22 + (1 / r) ⁇ G 12 + (1 / r) ⁇ G 23 + (1 / r 2 ) ⁇ B 13 (10)
  • ⁇ a (1) 22 ⁇ adjacent to the right of the undetected value ⁇ a (1) 12 ⁇ is expressed by the following expression (11).
  • a (1) 22 1 ⁇ R 22 + (1 / r) ⁇ G 32 + (1 / r) ⁇ G 23 + (1 / r 2 ) ⁇ B 33 (11)
  • a (1) 12 may be obtained as an interpolated value (average, etc.) of a (1) 02 and a (1) 22 adjacent to the left and right of a (1) 12 , A (1) 12 may be obtained by interpolation from the surrounding pixel addition values having the same weighting.
  • the estimated value of the undetected value a (3) 12 Perform approximate calculation.
  • the estimated value a (3) 12 is obtained as a 4-pixel addition value on the assumption that the weight of the B color pixel is set to the reference “1”.
  • ⁇ a (3) 13 ⁇ adjacent to the lower side of the undetected value ⁇ a (3) 12 ⁇ is expressed by the following expression (14).
  • a (3) 13 (1 / r 2 ) ⁇ R 24 + (1 / r) ⁇ G 14 + (1 / r) ⁇ G 23 + 1 ⁇ B 13 (14)
  • a (3) 12 may be obtained as an interpolated value (average, etc.) of a (3) 11 and a (3) 13 adjacent to the upper and lower sides of a (3) 12. It is also possible to obtain a (3) 12 by interpolation from surrounding pixel addition values with equal weighting.
  • G 12 and G 23 in a (1) 12 and a (3) 12 are the same pixel and are adjacent pixels, as shown in the following equation (16), if it is equal to the average value G 12/23 Handle with approximation.
  • G 12 ⁇ G 23 ⁇ G 12/23 (G 12 + G 23) / 2 (16)
  • ⁇ a (2) 12 ⁇ is expressed by the following equation (19) using the above equation (16).
  • a (2) 12 (1 / r) ⁇ R 22 + 1 ⁇ G 12 + (1 / r 2 ) ⁇ G 23 + (1 / r) ⁇ B 13 ⁇ (1 / r) ⁇ R 22 + [1 + (1 / r 2 )] ⁇ G 12/23 + (1 / r) ⁇ B 13 (19)
  • G 12/23 is derived as shown in the following equation (20).
  • R 22 and B 13 are obtained.
  • a frame image of a low-resolution moving image is configured with R 22 , G 12/23 and B 13 obtained in this way as pixel values having the same coordinates.
  • a low-resolution three-plate color image can be obtained by simple arithmetic processing, which is effective as a moving image generation process that requires real-time processing. Further, in this method, as compared with the conventional four-pixel addition method of the same color, a shift in the center of gravity position of the G color when the Bayer is formed does not occur, and higher image quality can be obtained.
  • the imaging apparatus includes the low-resolution moving image generation unit 160 that generates a low-resolution moving image.
  • the captured image fx is an RGB Bayer array image
  • the addition unit is four adjacent pixels (for example, four pixels of R, Gr, Gb, and B).
  • the addition image generation unit 130 acquires first to fourth addition images A 1 to A 4 in which RGB are mixed by weighted addition at the first to fourth positions.
  • the low-resolution moving image generation unit 160 performs RGB pixel values (for example, FIG. 11) of the low-resolution moving image based on the first to third A 1 to A 3 addition images.
  • R 22 , G 12/23 , B 13 are obtained by estimation.
  • a low-resolution three-plate color image can be obtained by simple arithmetic processing, which is effective as a moving image generation process that requires real-time processing such as live view, for example.
  • the RGB pixel value at the same position for example, a 12
  • the center of gravity of G color does not shift when the Bayer method is used, and higher image quality can be obtained as compared with the conventional 4-pixel addition method of the same color. Obtainable.
  • the fusion image data F (M, D 1 to D 4 ) is compressed. In this embodiment, however, after the difference images D 1 to D 4 are entropy-encoded. Data reconstruction may be performed and the reconstructed data may be compressed.
  • FIG. 12 shows a third configuration example of the imaging apparatus as a configuration example when data reconstruction is performed.
  • the imaging device includes an imaging unit 10 and an image processing unit 20.
  • the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
  • the imaging unit 10 includes a lens 110, an image sensor 120, a low resolution image generation unit 125 (a moving image low resolution Bayer image generation unit), a first addition image generation unit 130, a second addition image generation unit 135, and a difference.
  • the image processing unit 20 includes a compressed data decompression unit 200, an addition image reproduction unit 210, a low resolution image estimation calculation unit 215 (a low resolution Bayer image generation unit for moving images), an estimation calculation unit 230, and a high-definition still image generation unit 240. , A high-definition video generation unit 250, a high-quality standard video generation unit 260, a standard video generation unit 280, an image output unit 290, and an image selection unit 295.
  • the low-resolution image generation unit 125 newly generates a low-resolution image having a Bayer array with a 1 ⁇ 4 pixel number, for example, by adding four pixels of the same proximity and the same color.
  • the second addition image generation unit 135 performs addition processing described with reference to FIG. 1 on the low resolution image to generate addition images B 1 to B 4 .
  • the average image generation unit 142 generates an average image of the addition images A 1 to A 4 and an average image of the addition images B 1 to B 4 .
  • the difference image generation unit 141 generates a difference image between the addition images A 1 to A 4 and the average image, and a difference image between the addition images B 1 to B 4 and the average image.
  • the entropy encoding unit 143 performs entropy encoding on the difference image from the difference image generation unit 141.
  • the pixel data reconstruction unit 165 reconstructs the pixel data of the average image and the pixel data of the difference image subjected to entropy coding by a method described later with reference to FIG. 16, and outputs reconstruction data u ij .
  • the third addition image generation unit 175 generates an addition value ⁇ ij by performing addition processing described later with reference to FIG. 17 on the reconstructed pixel data.
  • the data compression unit 185 compresses the addition value ⁇ ij by irreversible compression such as M-JPEG or JPEG-XR.
  • the compressed data decompression unit 200 decompresses the added value ⁇ ij from the compressed data.
  • Adding image reproduction unit 210 reproduces the reconstructed data u ij from the extended summation value xi] ij, to reproduce the added image A 1 ⁇ A 4, B 1 ⁇ B 4 from the reconstructed data u ij.
  • the low-resolution image estimation calculation unit 215 performs estimation processing to be described later with reference to FIGS. 27A to 30 on the added images B 1 to B 4 to restore the Bayer-array low-resolution image.
  • the standard moving image generation unit 280 performs demosaicing on the Bayer array low resolution image to generate a low resolution moving image (standard moving image).
  • FIG. 13 conceptually shows an example of an imaging process for simultaneously acquiring a high resolution still image and a low resolution still image.
  • the imaging frame from the imaging device is high resolution (for example, 12 megapixels) and the frame rate is high speed (for example, 60 fps).
  • the even frames f 0 , f 2 ,... are converted from a 12-megapixel Bayer image into four 3-megapixel addition images by the addition processing described with reference to FIG.
  • the odd-numbered frames f 1 , f 3 ,... are converted into a 3 megapixel Bayer array image by, for example, general addition of four pixels of similar colors.
  • the converted image is converted into four 0.75 megapixel added images by the addition processing described in FIG.
  • the average image and the difference image are generated from these converted images, and the difference image is compressed by entropy coding.
  • the compressed difference image is subjected to pixel bit reconstruction together with the average image. Pixel addition is performed on the reconstructed pixel data, and normal image data compression processing is performed on the data after the addition.
  • pixel addition is performed on the reconstructed pixel data for the following reason. That is, as described above with reference to FIG. 12, an average image and a difference image are generated from the added images A 1 to A 4 and B 1 to B 4 , and the difference image is entropy encoded.
  • the average image data and the encoded difference image data are no longer pixel value sets having image characteristics, and are merely an array of bits with high randomness. Therefore, it cannot be recognized as data having a bias in a specific code. That is, it is safer to consider that the entropy is high.
  • the entropy is reduced by using the addition effect in the third addition image generation unit 175.
  • the effect of subsequent data compression is enhanced.
  • FIG. 14 shows a sequence example of the imaging process.
  • the high-resolution image of the even frame f 0 is exposed and read in the period T 0
  • the pre-encoding process (PE 0) and the normal image data compression process (compression 0) are performed in the period T 1 + T 2 . I do.
  • exposure and read-out of the high resolution image of the odd-numbered frames f 1 in the period T 1 are performed.
  • the normal image data compression processing of the even frame f 0 and odd frames f 1 consecutive is processed within the period 2T serially.
  • FIG. 15 shows an example of data reconstruction before data compression.
  • This example is a data reconstruction example in the case where the data before data compression is formed with 12 bits per pixel, but the data compression process can be processed only with 8 bits per pixel.
  • the pixel value bit string of the average image is represented by m 10 to m 16 and m 20 to m 26
  • the pixel value bit string of the difference image after entropy coding is represented by d 10 to d 14 and d 20 to d. 24 .
  • the bit strings d 10 to d 14 and d 20 to d 24 are arrayed in the upper bits of the reconstructed data, and the bit strings m 10 to m 16 and m 20 are arranged.
  • the upper bits in m 26 are arranged in the upper bits of the reconstructed data as much as possible. This is to prevent the pixel value after entropy coding and the pixel value of the average image from being deteriorated by data compression decoding as much as possible.
  • FIG. 16 shows an example of a data unit set in data reconstruction.
  • the difference value data ⁇ d k ⁇ of the difference image after entropy coding is added to the pixel value data ⁇ a M ij ⁇ (fixed bit width) of the average image, and the data is re-synthesized.
  • the pair ⁇ a M ij ⁇ + ⁇ d k ⁇ of the difference value data after entropy encoding and the pixel value data of the average image is reconstructed as a data unit ⁇ u ij ⁇ by regarding it as a virtual pixel value.
  • the average image pixel value ⁇ a M ij ⁇ is arranged on the lower bit side, and the difference value data ⁇ d k ⁇ is arranged on the upper bit side.
  • the decoding error of the data unit ⁇ u ij ⁇ can affect the average image pixel value having a fixed bit width. That is, it is possible to minimize the influence of the decoding error on the difference value data. This is because the difference value data is entropy-encoded data, so that if there is a decoding error, it is reproduced as a value with a large error, so that it is avoided.
  • the data ⁇ d k ⁇ after the entropy encoding is calculated from the code bit length N to the pixel value ⁇ a M ij ⁇ of the average image. to meet the code bit length obtained by subtracting the code bit length n M0 [n-n M0] , it may be suitably coupled configuration. Therefore, the data ⁇ d k ⁇ may be divided and formed across a plurality of reconstruction units ⁇ u ij ⁇ .
  • FIG. 17 shows an example of the superimposed shift addition value of the reconstruction data unit ⁇ u ij ⁇ . Since ⁇ u ij ⁇ , which is a virtual pixel, is a value obtained as a result of the average image bit addition and arbitrarily bit-separated, it is not possible to expect bias in the distribution of distribution or high correlation between neighboring pixels. Therefore, as shown in FIG. 17, again performs superimposition shift weighted addition processing on the reconstructed data ⁇ u ij ⁇ , to generate a sum value ⁇ ij ⁇ .
  • the added value ⁇ ij ⁇ is obtained by the following equation (21).
  • ⁇ ij [u ij + (1 / r) u (i + 1) j + (1 / r) u i (j + 1) + (1 / r 2 ) u (i + 1) (j + 1) ] / 4 (21)
  • FIG. 18 shows a configuration example of the imaging unit when adding four pixels of the same color.
  • the imaging unit illustrated in FIG. 18 includes a lens 110, an imaging element 120, added image generation units 301 to 304, difference image generation units 311 to 314, average image generation units 321 to 324, entropy encoding units 331 to 334, and Bayer image generation. Section 340, moving image data compression section 350, and compressed data recording section 150.
  • the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
  • the average image generation units 321 to 324 generate average images based on the added images of the respective colors.
  • the difference image generation units 311 to 314 generate difference images of the average image and the addition image of each color, respectively.
  • the entropy encoding units 331 to 334 perform entropy encoding of the difference images for each color.
  • the Bayer image generation unit 340 arranges the pixel values of the average image of the four colors in the Bayer array, and generates one Bayer array image.
  • the moving image data compression unit 350 irreversibly compresses moving image data based on the Bayer array image using, for example, AVCHD (H.246), M-JPEG, JPEG-XR, or the like.
  • ANR adaptive noise reduction
  • ANR refers to adaptive moving average filtering that performs pixel value averaging while adaptively varying the target range along the spatial axis.
  • ANR when the range of change of the sampling value within the target range is small, the range of the moving average is widened, and when the range of change of the sampling value within the target range is large, the range of the moving average is narrowed.
  • the average of the sampling values included in a predetermined pixel value width centered on the pixel value at the processing position (target position) is calculated, and is set as the pixel value at the processing position again.
  • the case where this method is applied has a large effect when the amplitude change width of the sampling value is small.
  • the pixel value a M ij of the average image and the pixel value (for example, a (1) ij ) of the added image have high pixel value correlation. Since these difference values a D1 ij are projections onto the vector (1, ⁇ 1), the occurrence probabilities of the difference values a D1 ij are distributed in a narrow range, and the amplitude of the sampling value of the difference image compared to the original captured image Change can be reduced. Therefore, if ANR is applied to this difference image, it is considered that noise in the difference image can be effectively reduced.
  • the 4-pixel addition value (sampling value) to be processed is a k
  • the 4-pixel addition value in the vicinity that falls within the range of width ⁇ ⁇ / 2 around a k is specified.
  • the 4-pixel addition values that fall within the range of width ⁇ ⁇ / 2 are a k ⁇ 2 , a k ⁇ 1 , and a k + 1 .
  • d1 represents a range of 4-pixel addition values that take a moving average in the left direction (negative direction of the k-axis) of the 4-pixel addition values ak to be processed.
  • d2 represents the range of the 4-pixel addition value that takes a moving average in the right direction (the positive direction of the k-axis) of the 4-pixel addition value ak to be processed. It is assumed that d1 and d2 satisfy the condition shown in the following formula (24).
  • d1 + d2 is the maximum, d1 and d2 are integers of 0 or more (inclusive), 0 ⁇ (24)
  • the difference images D 1 to D 4 are each two-dimensional array data.
  • the ANR is applied separately in the horizontal direction and the vertical direction.
  • an average value a ij ′ (a h ij + a v ij ) / 2) of the moving average values a h ij and a v ij is calculated, and the average value a ij ′ is calculated. Is the final moving average value.
  • the imaging device includes a noise reduction processing unit (added image noise reduction processing unit 220).
  • the noise reduction processing unit performs adaptive noise reduction processing for adaptively adjusting the moving average range d1 + d2 according to the change width of the pixel value, and performs the m-th difference image D. to m .
  • the range d1 + d2 is adjusted according to the number of pixel values that fall within the range of width ⁇ ⁇ / 2 from the pixel value ak to be processed, and the average value of the pixel values in the range d1 + d2 is calculated as the pixel value.
  • a Reset as k .
  • the average image and the difference image are generated directly from the captured image.
  • the difference between the captured images is obtained between frames, and the inter-frame difference is obtained.
  • An average image and a difference image may be generated from the images.
  • FIG. 24 shows a fourth configuration example of the imaging apparatus as a configuration example when generating an average image and a difference image from the inter-frame difference image.
  • This imaging apparatus includes an imaging unit 10 and an image processing unit 20.
  • the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
  • the imaging unit 10 includes a lens 110, a subtraction unit 115, an imaging element 120, an addition image generation unit 130, a difference image generation unit 141, an average image generation unit 142, an average value calculation unit 145, a subtraction unit 146, and a lossless compression encoding unit 147.
  • the components of the image processing unit 20 are the same as those of the image processing unit 20 in FIG.
  • the added image generation unit 130 obtains an added image from the reference image V t and the difference images V ′ t ⁇ 1 and V ′ t + 1 .
  • the subsequent processing is the same as the processing described above with reference to FIG.
  • the estimation calculation unit 230 restores the original captured image, first, the reference image V t and the difference images V ′ t ⁇ 1 and V ′ t + 1 are restored from the added image, and the captured image V t ⁇ 1 is obtained from these images.
  • V t ⁇ V ′ t ⁇ 1 , V t + 1 V t ⁇ V ′ t + 1 is obtained.
  • FIG. 25 schematically shows processing performed by the imaging apparatus.
  • RAW image data of frames f 0 , f 1 ,... Taken at a predetermined time interval (for example, 1/60 sec.) are represented as V 0 , V 1 ,.
  • the captured image of the reference frame f 3k + 1 is V 3k + 1
  • the captured images for generating the difference frames f 3k and f 3k + 2 are V 3k and V 3k + 2 .
  • k is an integer of 0 or more (including its value).
  • the reference captured image V 3k + 1 is high-definition RAW image data itself, and four 4-pixel addition images A (3k + 1) 1 to A (3k + 1) 4 are generated by performing superposition shift weighting addition on V 3k + 1 . Also, the difference between corresponding pixel values of the images V 3k and V 3k + 2 in the frames before and after the reference image and the reference image V 3k + 1 is calculated , and difference frame images V ′ 3k and V ′ 3k + 2 are generated. These difference frame images V ′ 3k and V ′ 3k + 2 are respectively subjected to superposition shift weighted addition, and four 4-pixel addition images A (3k) 1 to A (3k) 4 , A (3k + 2) 1 ⁇ A (3k + 2) 4 is generated.
  • an average image and a difference image are generated from the four-pixel addition image, and the average image and the difference image are subjected to compression processing (particularly lossless compression processing). Is recorded as fused image data.
  • the compression effect as moving image data can be increased compared to the above-described method in FIG. Note that entropy reduction in dissimilar portions cannot be expected with simple RAW data differences, and therefore, subsequent processing becomes effective when lossless compression is considered.
  • FIG. 26 shows a modified configuration example of the imaging unit in the fourth configuration example.
  • the imaging unit includes a lens 110, a subtraction unit 116, an imaging element 120, an addition image generation unit 130, a difference image generation unit 141, an average image generation unit 142, an average value calculation unit 145, a subtraction unit 146, and a lossless compression encoding unit 147.
  • the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
  • the inter-frame difference image of the added image is obtained instead of the inter-frame difference image of the captured image.
  • the added image generation unit 130 performs superposition shift weighted addition on the captured images V 3k , V 3k + 1 , and V 3k + 2 , and adds the added images A (3k) 1 to A (3k) 4 , A (3k + 1). 1 to A (3k + 1) 4 and A (3k + 2) 1 to A (3k + 2) 4 are generated.
  • the subtracting unit 116 adds the reference image addition images A (3k + 1) 1 to A (3k + 1) 4 and the addition images A (3k) 1 to A (3k) 4 and A (3k + 2) 1 to A (3k + 2).
  • the image acquisition unit e.g., image sensor 120
  • the captured image V 3k + 1 of the first frame f 3k + 1 obtained as a reference photographed image
  • the first frame f 3k + 1 The difference between the captured image V 3k of the previous or subsequent second frame (for example, the previous frame f 3k ) and the reference captured image V 3k + 1 is acquired as the differential captured image V ′ 3k .
  • the addition image generation unit 130 includes first to fourth (first to nth in a broad sense) addition images A (3k + 1) 1 to A (3k + 1) 4 based on the reference captured image V 3k + 1 , and the difference captured image V ′. obtaining first to n of the addition image a (3k) 1 ⁇ a ( 3k) 4 based on 3k.
  • the second frame may be a frame f 3k + 2 after the first frame f 3k + 1 .
  • the differential captured image V ′ 3k + 2 and the first to nth added images A (3k + 2) 1 to A (3k + 2) 4 are acquired.
  • the difference image and the added image may be obtained using both the frames before and after the first frame f 3k + 1 .
  • the inter-pixel correlation in the frame is increased by the pixel addition in the frame, and the inter-frame difference image can be obtained by subtraction of the added images.
  • the difference effect can be expected.
  • an average image generated from an inter-frame difference image is considered to have lower entropy than an average image generated from a normal addition image. Therefore, it is considered that the compression rate by entropy coding of the average image can be improved.
  • the estimation process performed by the above-described estimation calculation unit 230 will be described in detail with reference to FIG.
  • the addition pixel values ⁇ a 00 , a 10 , a 11 , a 01 ⁇ will be described as an example (i and j are integers greater than or equal to 0 (including their values)). It is the same. Further, the case where the addition unit is set for every 2 ⁇ 2 pixels will be described as an example, but the present invention is not limited to this, and may be, for example, every 3 ⁇ 3 pixels.
  • FIG. 27A and 27B are explanatory diagrams of the estimated pixel value and the intermediate pixel value. Summing the pixel values shown in FIG. 27 (A) ⁇ a 00, a 10, a 11, a 01 ⁇ is added pixel value of the added image A 1 ⁇ A 4 described in FIG. 1 ⁇ a (1) 00, a ( 2) Corresponds to 10 , a (3) 11 , a (4) 01 ⁇ . In the estimation process, final estimated pixel values v 00 to v 22 are estimated using this added pixel value.
  • the estimated pixel value v ij corresponds to the pixel value of the captured image fx described in FIG.
  • intermediate pixel values b 00 to b 21 are estimated from the added pixel values a 00 to a 11 .
  • Intermediate pixel value corresponds to 2-pixel sum values, for example, b 00 corresponds to the sum of the pixel values v 00 and v 01.
  • Final pixel values v 00 to v 22 are estimated from these intermediate pixel values b 00 to b 21 .
  • the intermediate pixel values b 00 to b 20 are estimated based on the added pixel values a 00 and a 10 in the first row in the horizontal direction.
  • the weighting factor r 2
  • the addition pixel values a 00 and a 10 are expressed by the following expression (25).
  • a 00 v 00 + (1/2) v 01 + (1/2) v 10 + (1/4) v 11
  • a 10 v 10 + (1/2 ) v 11 + (1/2) v 20 + (1/4) v 21 (25)
  • B 00 , b 10 , and b 20 are defined as shown in the following formula (26).
  • b 00 v 00 + (1 / r)
  • v 01 v 00 + (1/2) v 01
  • b 10 v 10 + (1 / r)
  • v 11 v 10 + (1/2) v 11
  • b 20 v 20 + (1 / r)
  • v 21 v 20 + (1/2) v 21 (26)
  • intermediate pixel values b 10 and b 20 can be obtained as a function of b 00 as shown in the following equation (29). In this way, a high-definition combination pattern of intermediate pixel values ⁇ b 00 , b 10 , b 20 ⁇ is obtained with b 00 as an unknown.
  • b 00 (unknown number)
  • b 10 2 (a 00 -b 00 )
  • the pattern ⁇ a 00 a 10 ⁇ of the added pixel value is compared with the pattern ⁇ b 00 , b 10 , b 20 ⁇ of the intermediate pixel value. Then, an unknown number b 00 that minimizes the error E is derived and set as the intermediate pixel value b 00 .
  • the addition unit (for example, a 00 ) set to the first position and the second position subsequent to the first position are set.
  • the addition unit (for example, a 10 ) is overlapped.
  • the estimation calculation unit 230 obtains a difference value ⁇ i 0 between the added pixel values a 00 and a 10 at the first and second positions.
  • the first intermediate pixel value b 00 is the addition of the first area (v 00 , v 01 ) obtained by removing the overlapping area (v 10 , v 11 ) from the addition unit a 00. It is a pixel value.
  • the second intermediate pixel value b 20 is an addition pixel value of the second region (v 20 , v 21 ) obtained by removing the overlap region (v 10 , v 11 ) from the addition unit a 10 .
  • a relational expression between the first and second intermediate pixel values b 00 and b 20 is expressed using a difference value ⁇ i 0 .
  • the first and second intermediate pixel values b 00 and b 20 are estimated using the relational expression. Using the estimated first intermediate pixel value b 00 , pixel values (v 00 , v 10 , v 11 , v 01 ) of each pixel included in the addition unit are obtained.
  • superimposing means having an area where the addition unit and the addition unit overlap.
  • the addition unit a 00 and the addition unit a 10 are two estimated pixels v 10. is to share the v 11.
  • the position of the addition unit is the position and coordinates of the addition unit in the captured image, or the position and coordinates of the addition unit on the estimated pixel value data (image data) in the estimation process.
  • the next position is a position shifted from the original position by a pixel, and is a position where the position and coordinates do not coincide with the original position.
  • continuous intermediate pixel values including the first and second intermediate pixel values are defined as intermediate pixel value patterns ( ⁇ b 00 , b 10 , b 20 ⁇ ).
  • the estimation calculation unit 230 represents the relational expression between the intermediate pixel values included in the intermediate pixel value pattern using the added pixel values a 00 and a 10 .
  • the similarity is evaluated by comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values with the added pixel value. Based on the similarity evaluation result, the intermediate pixel values b 00 , b 10 , and b 20 included in the intermediate pixel value pattern are determined so that the similarity is the highest.
  • the intermediate pixel value can be estimated based on a plurality of added pixel values acquired by pixel shifting while the addition unit is superimposed.
  • the intermediate pixel value pattern is a data string (a set of data) of intermediate pixel values in a range used for the estimation process.
  • the addition pixel value pattern is a data string of addition pixel values in a range used for the estimation process.
  • the estimation calculation unit 230 has an intermediate pixel value pattern ( ⁇ b 00 , b 10 , b 20 ⁇ ) represented by a relational expression between intermediate pixel values. And an evaluation function Ej representing an error between the pixel value (a 00 and a 10 ).
  • the intermediate pixel values b 00 , b 10 and b 20 included in the intermediate pixel value pattern are determined so that the value of the evaluation function Ej is minimized.
  • the value of the intermediate pixel value can be estimated by expressing the error by the evaluation function and obtaining the intermediate pixel value corresponding to the minimum value of the evaluation function.
  • the initial value of the intermediate pixel estimation can be set with a simple process by obtaining the unknown using the least square method. For example, searching for an image portion suitable for initial value setting (Patent Document 2) is unnecessary.
  • each pixel value (for example, v 00 , v 10 , v 01 , v 11 ) of the addition unit is weighted and added to the added pixel value (a 00 ). get.
  • the pixel value (v 00 , v 10 , v 01 , v 11 ) of each pixel of the addition unit is estimated.
  • each pixel value of the addition unit is weighted and added to obtain an added image, and the pixel value of the high resolution image can be estimated from the obtained added image.
  • the reproducibility of the high-frequency component of the subject can be improved. That is, when the pixel values of the addition unit are simply added, a rectangular window function is convoluted for imaging.
  • a window function containing more high frequency components than the rectangle is convoluted for imaging. Therefore, it is possible to acquire an added image that includes more high-frequency components of the subject, and to improve the reproducibility of the high-frequency components in the estimated image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

L'invention porte sur un dispositif d'imagerie qui contient une unité d'acquisition d'image, une unité de génération d'images sommes, une unité de traitement de compression, une unité de traitement de décompression, une unité de calcul d'estimation et une unité de sortie d'image. L'unité de génération d'images sommes décale séquentiellement par pixels l'unité d'addition dans une image capturée (fx), et acquiert des images sommes (A1-A4) au moyen d'une addition pondérée des valeurs de pixel contenues dans l'unité d'addition. L'unité de traitement de compression détermine la moyenne des images sommes (A1-A4) à titre d'image moyenne (M), détermine la différence entre l'image moyenne (M) et une image somme (Am) à titre d'image différence (Dm), et compresse l'image moyenne (M) et l'image différence (Dm) déterminées. L'unité de traitement de décompression décompresse l'image moyenne (M) et l'image différence (Dm) compressées afin de déterminer les images sommes (A1-A4). L'unité de calcul d'estimation estime les valeurs de pixel (vij) de l'image capturée (fx) sur la base des images sommes (A1-A4). L'unité de sortie d'image délivre une image à haute résolution qui est obtenue sur la base des valeurs de pixel (vij).
PCT/JP2012/059977 2011-04-27 2012-04-12 Dispositif d'imagerie et procédé de génération d'image Ceased WO2012147523A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011-099265 2011-04-27
JP2011099265A JP2012231377A (ja) 2011-04-27 2011-04-27 撮像装置及び画像生成方法

Publications (1)

Publication Number Publication Date
WO2012147523A1 true WO2012147523A1 (fr) 2012-11-01

Family

ID=47072045

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/059977 Ceased WO2012147523A1 (fr) 2011-04-27 2012-04-12 Dispositif d'imagerie et procédé de génération d'image

Country Status (2)

Country Link
JP (1) JP2012231377A (fr)
WO (1) WO2012147523A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2503656A (en) * 2012-06-28 2014-01-08 Canon Kk Compressing and decompressing plenoptic images
US10122988B2 (en) 2014-04-17 2018-11-06 Canon Kabushiki Kaisha Image encoding apparatus, image decoding apparatus, methods of controlling the same, and storage medium
WO2021200191A1 (fr) * 2020-03-31 2021-10-07 ソニーグループ株式会社 Dispositif et procédé de traitement d'image, et programme

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6450288B2 (ja) * 2015-09-18 2019-01-09 日本電信電話株式会社 画像符号化装置、画像復号装置、画像符号化方法及び画像復号方法
JP6214841B1 (ja) * 2016-06-20 2017-10-18 オリンパス株式会社 被検体内導入装置、送信方法及びプログラム
WO2017221468A1 (fr) * 2016-06-20 2017-12-28 オリンパス株式会社 Dispositif d'introduction, procédé de transmission, et programme
JP6467455B2 (ja) * 2017-04-18 2019-02-13 キヤノン株式会社 撮像装置、その制御方法、プログラム及び記録媒体
JPWO2021028754A1 (fr) * 2019-08-09 2021-02-18
KR102709415B1 (ko) 2020-04-29 2024-09-25 삼성전자주식회사 이미지 압축 방법, 인코더, 및 인코더를 포함하는 카메라 모듈

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009124621A (ja) * 2007-11-19 2009-06-04 Sanyo Electric Co Ltd 超解像処理装置及び方法並びに撮像装置
JP2010004396A (ja) * 2008-06-20 2010-01-07 Sanyo Electric Co Ltd 画像処理装置、画像処理方法及び撮像装置
JP2011151569A (ja) * 2010-01-21 2011-08-04 Olympus Corp 画像処理装置、撮像装置、プログラム及び画像処理方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009124621A (ja) * 2007-11-19 2009-06-04 Sanyo Electric Co Ltd 超解像処理装置及び方法並びに撮像装置
JP2010004396A (ja) * 2008-06-20 2010-01-07 Sanyo Electric Co Ltd 画像処理装置、画像処理方法及び撮像装置
JP2011151569A (ja) * 2010-01-21 2011-08-04 Olympus Corp 画像処理装置、撮像装置、プログラム及び画像処理方法

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2503656A (en) * 2012-06-28 2014-01-08 Canon Kk Compressing and decompressing plenoptic images
GB2503656B (en) * 2012-06-28 2014-10-15 Canon Kk Method and apparatus for compressing or decompressing light field images
US10122988B2 (en) 2014-04-17 2018-11-06 Canon Kabushiki Kaisha Image encoding apparatus, image decoding apparatus, methods of controlling the same, and storage medium
WO2021200191A1 (fr) * 2020-03-31 2021-10-07 ソニーグループ株式会社 Dispositif et procédé de traitement d'image, et programme
JPWO2021200191A1 (fr) * 2020-03-31 2021-10-07
US11770614B2 (en) 2020-03-31 2023-09-26 Sony Group Corporation Image processing device and method, and program
JP7643452B2 (ja) 2020-03-31 2025-03-11 ソニーグループ株式会社 画像処理装置および方法、並びにプログラム

Also Published As

Publication number Publication date
JP2012231377A (ja) 2012-11-22

Similar Documents

Publication Publication Date Title
WO2012147523A1 (fr) Dispositif d'imagerie et procédé de génération d'image
US9210341B2 (en) Image processing device, imaging device, information storage medium, and image processing method
US9094648B2 (en) Tone mapping for low-light video frame enhancement
US8315481B2 (en) Image transmitting apparatus, image receiving apparatus, image transmitting and receiving system, recording medium recording image transmitting program, and recording medium recording image receiving program
US9307212B2 (en) Tone mapping for low-light video frame enhancement
US8698906B2 (en) Image processing device, imaging device, information storage medium, and image processing method
JP4849130B2 (ja) 画像処理装置、および画像処理方法、並びにプログラム
US20130083220A1 (en) Image processing device, imaging device, information storage device, and image processing method
JP2009194896A (ja) 画像処理装置及び方法並びに撮像装置
WO2009145201A1 (fr) Dispositif et procédé de traitement d'image et dispositif d'imagerie
WO2013008517A1 (fr) Appareil de capture d'images et procédé de génération d'images
JP2009253667A (ja) 動画処理装置及び動画処理方法、動画処理プログラム
JP4810807B2 (ja) 動画像変換装置、動画像復元装置、および方法、並びにコンピュータ・プログラム
JP2012085214A (ja) データ圧縮方法およびデータ圧縮装置
JP5159461B2 (ja) 画像処理装置、画像処理方法及び撮像装置
JP4942563B2 (ja) 画像処理方法、画像処理装置、及びこの画像処理装置を備えた電子機器
WO2012147630A1 (fr) Dispositif d'imagerie et procédé de génération d'image
WO2011010431A1 (fr) Dispositif de traitement d’images, procédé de traitement d’images et dispositif de capture d’images
US20130155272A1 (en) Image processing device, imaging device, and image processing method
JP2012231379A (ja) 撮像装置及び画像生成方法
JP2012142676A (ja) 撮像装置及び画像生成方法
JP2013125999A (ja) 画像処理装置、撮像装置及び画像処理方法
JP2012099870A (ja) 撮像装置及び撮像方法
JP2012119852A (ja) 撮像装置及び撮像方法
JP2009260633A (ja) 画像圧縮装置、画像圧縮装置を備えた撮像記録装置、および、画像圧縮プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12777461

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12777461

Country of ref document: EP

Kind code of ref document: A1