JP2005123749A - Imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of obtaining a recording image with high image quality without the need for an exclusive processing sequence. <P>SOLUTION: A pixel line whose RAW data are missing by the adoption of a configuration of displaying a live view image for a period of main exposure operation is interpolated or replaced with RAW data. The RAW data of a G (Green) pixel at the left side of a pixel line for a recording image located above by two columns as shown in the arrow C are interpolated to the left side pixel of the pixel line shown in the arrow A, and the RAW data of a B (Blue) pixel at the right side of the same pixel line are interpolated to the right side pixel. The RAW data of a R (Red) pixel at the left side of a pixel line located above by two columns as shown in the arrow D are interpolated to the left side pixel of the pixel line shown in the arrow B, and the RAW data of a G (Green) pixel at the right side of the same pixel line are interpolated to the right side pixel. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an imaging apparatus having a function of displaying an image of a subject during an exposure operation for generating a pixel signal for recording.

  Conventionally, in a digital camera equipped with a display such as a liquid crystal display, there is a technique described in, for example, Japanese Patent Application Laid-Open No. 2004-151867 as a technique that enables a subject being exposed to be confirmed on the display.

In Patent Document 1, the pixel array surface of the image sensor is divided into two fields for every two horizontal lines, and image data obtained from a pixel column of one field is used for recording, and A technique for using image data obtained from a pixel row for display on a liquid crystal display or the like is disclosed.
JP 2002-135626 A

  However, in the technique of Patent Document 1, since an image signal is read out from approximately half the number of pixels of the image sensor, even if a subject is imaged using an image sensor with a high pixel count, this image capture is performed. It is difficult to obtain an image having an image quality suitable for the number of pixels of the element, and it is difficult to say that the performance of the image pickup element having a higher pixel is fully utilized for the image quality of the recorded image.

  Therefore, if the image data of the other field that is missing due to use for display is interpolated by a separate dedicated processing sequence in addition to the normal image processing performed by a general digital camera, the pixel of the image sensor Although an image with an image quality suitable for the number can be obtained, the provision of the dedicated processing sequence as described above will increase the cost of the digital camera.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an imaging apparatus capable of obtaining a high-quality recorded image without using a dedicated processing sequence.

  The invention according to claim 1 is provided with a plurality of types of pixels that receive different colors, and a plurality of pixel lines in which pixels are arranged in one direction are arranged in a crossing direction intersecting the one direction. An exposure operation instruction for instructing the imaging device to start an exposure operation for generating a recording pixel signal to be recorded in a predetermined recording unit; , Causing the image sensor to perform an exposure operation, and outputting RAW data at predetermined time intervals from a plurality of pixels located in some of the pixel lines arranged in the crossing direction. RAW data is acquired from another pixel other than the plurality of pixels at a predetermined timing after the imaging control unit and the exposure start instruction, and the plurality of pixels are paired based on the RAW data of the other pixels. An image processing unit that performs image processing including processing for interpolating RAW data to be performed, a first image generation unit that generates an RGB image from the post-interpolation RAW data generated by the image processing unit, and the first image generation An image pickup apparatus comprising: a recording control unit that records the RGB image generated by the unit in the predetermined recording unit. The RAW data refers to a group of image data before digital image processing is performed.

  According to a second aspect of the present invention, in the imaging apparatus according to the first aspect, the image processing unit uses RAW data of pixels located in the vicinity of the plurality of pixels as RAW data corresponding to the plurality of pixels. And performing the interpolation.

  According to a third aspect of the present invention, in the imaging device according to the second aspect, the imaging element is formed by Bayer arrangement of the plurality of types of pixels, and the image processing units are in contact with each other at one vertex. Four pixels are used as a block, and for each pixel belonging to a block including the plurality of pixels, alternative RAW data is interpolated or replaced using RAW data of a pixel belonging to a block adjacent to the block. Is.

  According to a fourth aspect of the present invention, in the imaging device according to any one of the first to third aspects, RAW data obtained from the plurality of pixels is converted into predetermined pixel data, and an RGB image based on the pixel data is converted. A second image generation unit to be generated and an image display unit to display the RGB image generated by the second image generation unit are provided.

  According to a fifth aspect of the present invention, in the imaging apparatus according to the fourth aspect, the imaging element is formed by Bayer arrangement of the plurality of types of pixels, and the second image generation unit is configured by the imaging element. The RGB image is generated based on RAW data obtained from pixels located on intersections of predetermined pixel lines set for each (2n + 1) columns in two orthogonal arrangement directions of the pixels included in It is what. (N is an integer of 1 or more)

  According to a sixth aspect of the present invention, in the imaging apparatus according to any one of the first to fifth aspects, the input operation unit is further exposed by the imaging element to obtain an image to be recorded in the predetermined recording unit. It has a function of instructing an exposure time of operation, and the image generation unit generates the image when an exposure time instructed by the input operation unit is longer than the predetermined time. It is.

  According to the first aspect of the present invention, when an exposure start instruction is performed by the input operation unit, the image sensor is caused to perform an exposure operation, and among the plurality of pixel lines configured in the crossing direction, RAW data is output at a predetermined time interval from a plurality of pixels located in some pixel lines, and RAW data is acquired from other pixels other than the plurality of pixels at a predetermined timing after the exposure start instruction, Since the RAW data corresponding to the plurality of pixels is interpolated based on the RAW data of the other pixels, a high-quality recorded image (recorded in a predetermined recording unit) is used without using the dedicated processing sequence as described above. RGB image) can be obtained.

  According to the second aspect of the present invention, the interpolation is performed by using the raw data of the pixels located in the vicinity of the plurality of pixels as the raw data corresponding to the plurality of pixels. Can be generated.

  According to a third aspect of the present invention, the imaging device is formed by Bayer arrangement of the plurality of types of pixels, and the four pixels that are in contact with each other at one vertex are used as a block, and the block includes the plurality of pixels. For each pixel that belongs, the alternative RAW data is interpolated or replaced using the RAW data of the pixel belonging to the block adjacent to the block, so that an image that is as faithful as possible to the subject image can be generated.

  According to a fourth aspect of the present invention, RAW data obtained from the plurality of pixels is converted into predetermined pixel data, an RGB image based on the pixel data is generated, and the generated RGB image is displayed. Accordingly, the subject image can be displayed on the image display unit while performing an exposure operation for generating the recording pixel signal.

  According to the invention described in claim 5, the image pickup element is formed by Bayer arrangement of the plurality of types of pixels, and each (2n + 1) columns in two orthogonal arrangement directions of the pixels provided in the image pickup element. Since the RGB image is generated based on the RAW data obtained from the pixels located on the intersection of the predetermined pixel lines set to, the array of the RAW data is also a Bayer array. As a result, it is possible to generate a beautiful RGB image with little or no distortion or false color of the image generated based on the RAW data.

  According to the sixth aspect of the present invention, the input operation unit has a function of indicating the exposure time of the exposure operation by the image sensor for generating the recording pixel signal, and the exposure time specified by the input operation unit. Since the image is generated when the time is longer than the predetermined time, when the exposure time instructed by the input operation unit is longer than the predetermined time, the RAW data is transferred from other pixels than the plurality of pixels. It is possible to record an RGB image based on a predetermined recording unit and display an RGB image based on RAW data obtained from a plurality of pixels on an image display unit.

  An embodiment of an imaging apparatus according to the present invention will be described.

  FIG. 1 is a front view of the imaging apparatus 1, and FIG. 2 is a rear view of the imaging apparatus 1.

  As shown in FIGS. 1 and 2, the imaging apparatus 1 includes a power button 2, an optical system 3, an LCD (Liquid Crystal Display) 4, an optical finder 5, a built-in flash 6, a mode setting switch 7, A quadruple switch 8, a shutter button 9, and a long-second monitoring switch 10 are provided.

  The power button 2 is used to switch on / off the power of the imaging apparatus 1.

  The optical system 3 includes a zoom lens, an unillustrated mechanical shutter, and the like, and forms an optical image of a subject on an imaging surface of an imaging element 11 (see FIG. 3) such as a CCD (Charge Coupled Device). is there.

  The LCD 4 displays a live view image and an image (recorded image) recorded on a memory card m (see FIG. 3), which will be described later, and reproduces and displays an image recorded on the memory card m.

  The live view image is an image picked up by the image pickup device 11 that is switched and displayed on the LCD 4 at a constant period (1/30 second) until the image of the subject is recorded. The state is displayed on the LCD 4 in substantially real time, and the photographer can check the state of the subject on the LCD 4.

  The optical viewfinder 5 enables optical observation of a range where a subject is photographed.

  The built-in flash 6 irradiates the subject with illumination light when the amount of exposure to the image sensor 11 is insufficient.

  The mode setting switch 7 is a “shooting mode” for shooting a subject image, a “playback mode” for playing back and displaying a shot image recorded on the memory card m on the LCD 4, and a “menu mode” for setting various functions. "Is a switch for setting the mode switching between the The mode setting switch 7 is composed of a three-contact slide switch that slides up and down. When the switch is set down, the image pickup apparatus 1 is set to the shooting mode. When the switch is set at the center, the playback mode is set. Is done.

  Although not described in detail, the quadruple switch 8 is a switch for performing movement of the zoom lens in the optical axis direction, exposure correction, frame advance of a recorded image to be reproduced on the LCD 4, and the like. Further, the imaging apparatus 1 of the present embodiment includes a program AE mode, an aperture priority AE mode, a shutter speed priority AE mode, a manual mode, and a bulb shooting mode as shooting mode types. These modes can be switched by operating the quadruple switch 8 with the mode set.

  The program AE mode is a mode in which a combination of an aperture value and a shutter speed is automatically set according to a preset AE (Auto Exposure) program. The aperture priority AE mode is an aperture value preset by a photographer. The shutter speed setting AE mode is a mode in which an image is taken by setting an aperture value at a shutter speed preset by the photographer, and the manual mode is a mode in which the photographer takes a picture. Is a mode in which imaging is performed with a shutter speed and an aperture value set in advance.

  The bulb shooting mode is a mode in which the photographer instructs the shutter button 9 to open and close the mechanical shutter (not shown), and the shutter button 9 is completely pressed down (hereinafter, this operation is fully pressed). The exposure operation for recording is started by the electronic shutter, and the mechanical shutter is closed by releasing the full press of the shutter button 9 and the exposure operation is ended. Therefore, in this bulb photography mode, the subject can be photographed by exposing the imaging device 1 to a relatively long time (for example, 10 seconds).

  The shutter button 9 is a button that is pressed in two stages (half-press and full-press), and is used to indicate the timing of exposure control. In each of the shooting modes, when the shutter button 9 is not operated, an optical image of the subject is captured every 1/30 (second), and a live view image is displayed on the LCD 4. In addition, in the above-described program AE mode, aperture priority AE mode, shutter speed priority AE mode, and manual mode, the imaging apparatus 1 is configured such that the shutter speed 9 By setting to an imaging standby state in which an aperture value and the like are set and being fully pressed, an optical image of a subject to be recorded on the memory card m is determined. The function of the shutter button 9 in the bulb shooting mode is as described above.

  Here, the imaging apparatus 1 of the present embodiment is configured to perform an exposure operation for generating a recording pixel signal when any one of the above-described photographing modes is set and the long-second monitoring mode described later is set. The state of the subject (more precisely, it is recorded in the memory card m) not only during the period until the start of the main exposure operation (hereinafter referred to as the main exposure operation) is instructed by the shutter button 9, but also during the main exposure operation. It is characterized in that the image state) is displayed on the LCD 4 in real time as a live view image.

  In the program AE mode and the aperture priority AE mode, the main exposure operation period (recording pixel signal generation period) corresponds to the shutter speed calculated by the imaging apparatus 1 after the shutter button 9 is fully pressed. In the shutter speed priority AE mode, after the shutter button 9 is fully pressed, this corresponds to the shutter open period corresponding to the shutter speed set in advance by the photographer. This corresponds to a period from when the shutter button 9 is fully pressed until a time preset by the photographer (exposure time) elapses. In the bulb shooting mode, the shutter button 9 is kept fully pressed. Equivalent to.

  The long-second monitoring switch 10 is used when the period of the main exposure operation is significantly longer than a preset display period (for example, 1/30 second) of the live view image when any of the above-described shooting modes is set. This is for setting a mode for displaying a live view image on the LCD 4 during the exposure operation period (long-second monitoring mode). Each time the button is pressed, setting / non-setting of the long-second monitoring mode is switched alternately. Note that this long-second monitoring mode is particularly effective when the bulb shooting mode is set. Therefore, in the following description, the case where the imaging apparatus 1 is set to the bulb shooting mode will be described. And

  FIG. 3 is a block configuration diagram showing a system of the imaging apparatus 1. In the figure, the same members as those shown in FIGS. 1 and 2 are denoted by the same reference numerals.

  The imaging apparatus 1 includes an optical system 3, an LCD 4, an imaging device 11, a timing generator 12, a signal processing unit 13, an A / D conversion unit 14, an image memory 15, and a VRAM (Video Random Access Memory) 16. The operation unit 17 and the control unit 18 are provided.

  The optical system 3 corresponds to the optical system 3 shown in FIG. 1 and includes a mechanical shutter as described above.

  The imaging device 11 has a plurality of photoelectric conversion elements, such as photodiodes, that are two-dimensionally arranged in a matrix. On the light receiving surface of each photoelectric conversion element, for example, R (red) and G (green) having different spectral characteristics, respectively. ), B (blue) color filter with Bayer arrangement in which color filters of 1: 2: 1 are arranged. Hereinafter, the photoelectric conversion element in which a color filter is provided is referred to as a pixel of the image sensor 11. The imaging device 11 converts the optical image of the subject formed by the optical system 3 into analog electrical signals (image signals) of R (red), G (green), and B (blue) color components. , B are output as image signals of respective colors.

  Although not described in detail, the image pickup device 11 is an interline type image pickup device including a light receiving unit made of a photodiode or the like, a vertical transfer unit, a horizontal transfer unit, and the like. The charge of each pixel is determined by a progressive transfer method. Is taken out. That is, the charges accumulated in each light receiving unit are transferred to the vertical transfer unit by a vertical synchronization signal, and the charges transferred to each vertical transfer unit are sequentially transferred from the pixels close to the horizontal transfer path by the horizontal synchronization signal to the horizontal transfer path. Is taken out as a pixel signal. The timing of starting and ending the imaging operation such as reading (horizontal synchronization and vertical synchronization) of the output signal of each pixel in the imaging device 11 and the exposure operation by the imaging device 11 is controlled by a timing generator 12 described later. The

  In the present embodiment, the method for generating a recorded image differs between when the imaging apparatus 1 is set to the long-second monitoring mode and when it is not set by the long-second monitoring switch 10.

  That is, when the long-second monitoring mode is not set, for example, when the bulb shooting mode is set, an instruction to start the main exposure operation (start generation of a recording pixel signal) is issued by starting full-pressing operation of the shutter button 9. The exposure operation is continued for almost all the pixels of the image sensor 11 without extracting the pixel signal until the full-pressing operation of the shutter button 9 is released. For this reason, nothing is displayed on the LCD 4. Thereafter, when the full-pressing operation of the shutter button 9 is released, the exposure operation is continued without taking out the pixel signal, and the main exposure operation by those pixels that have accumulated charges is finished, and the pixel signal is output from these pixels. Is extracted, and a recorded image is generated using the extracted pixel signal.

  On the other hand, when the long-second monitoring mode is set, the plurality of pixels of the image sensor 11 include a pixel for displaying a live view image (hereinafter referred to as LV display pixel) and a pixel for recording on the memory card m (hereinafter referred to as recording). Are used in advance.

  Specifically, in accordance with the ratio between the number of pixels of the image sensor 11 and the number of display pixels of the LCD 4, a plurality of pixels arranged in the vertical direction of the image sensor 11 corresponding to the vertical direction of the LCD 4 (arrow A direction in FIG. 2). Among the lines, pixels located on some pixel lines are used for LV display, and pixels located on other pixel lines are used for recorded images. For example, when the number of pixels in the vertical direction of the image sensor 11 is 2880 (pieces) and the number of display pixels in the vertical direction of the LCD 4 is 240 (pieces), as shown in FIG. The row of pixel lines is used for the live view image. In FIG. 4, only the two horizontal pixel lines of the image sensor 11 are illustrated.

  When an instruction to start the main exposure operation (start generation of a recording pixel signal) is issued by starting full-pressing operation of the shutter button 9, the exposure operation using the LV display pixels and the recording pixels is performed.

  Further, during the main exposure operation, pixel signals are extracted from the LV display pixels at predetermined time intervals, and live view images are sequentially generated using the extracted pixel signals. As a result, live view images are displayed on the LCD 4 at regular time intervals. In parallel with this, the recording pixel continuously performs the exposure operation without extracting the pixel signal. Thereafter, when the full exposure operation by the recording pixel is completed by releasing the full-pressing operation of the shutter button 9 and a pixel signal is extracted from the recording pixel, a recorded image is generated using the extracted pixel signal. A method for generating the recorded image will be described later.

  Based on the reference clock CLK0 transmitted from the control unit 18, the timing generator 12 controls the readout of the drive control signal for the image sensor 11, for example, the timing signal for integration start / end (exposure start / end), and the light reception signal of each pixel. A clock signal such as a signal (horizontal synchronization signal, vertical synchronization signal, etc.) is generated and output to the image sensor 11.

  The signal processing unit 13 performs predetermined analog signal processing on the analog pixel signal output from the image sensor 11. The signal processing unit 13 includes a CDS (correlated double sampling) circuit and an AGC (auto gain control) circuit, reduces noise of the pixel signal by the CDS circuit, and adjusts the level of the pixel signal by the AGC circuit.

  The A / D conversion unit 14 converts the analog R, G, and B pixel signals output from the signal processing unit 13 into digital pixel signals composed of a plurality of bits, respectively.

  The image memory 15 is a memory that temporarily stores the pixel signal output from the A / D conversion unit 14 and is used as a work area for performing processing described later on the image signal by the control unit 18. .

  The VRAM 16 is a buffer memory for pixel signals of an image reproduced and displayed on the LCD 4, and has a pixel signal recording capacity corresponding to the number of pixels of the LCD 4.

  The operation unit 17 includes a switch for detecting a release operation of the shutter button 9, a mode setting switch 7, a quadruple switch 8, a long-second monitoring switch 10, and the like.

  The control unit 18 is composed of a microcomputer having a built-in storage unit (including first and second image storage units 30 and 31 to be described later) including, for example, a ROM that stores a control program and a RAM that temporarily stores data. The drive of each member mentioned above is related and controlled.

  Here, the significance and the like of displaying the live view image on the LCD 4 during the main exposure operation will be described.

  When the long-second monitoring mode is set, when the exposure time is long (for example, when the shutter button 9 is fully pressed for a relatively long time in the bulb shooting mode), the image sensor 11 is performed after the start of the full-pressing operation of the shutter button 9. When all the pixel signals obtained by the exposure operation are used for recording in the memory card m, there is no pixel signal that can be used for live view image display, so the shutter button 9 is fully pressed. In FIG. 5, the state of the image recorded on the memory card m cannot be displayed on the LCD 4 as a live view image.

  Further, even when the above-described configuration for displaying the live view image on the LCD 4 is provided during the main exposure operation, the configuration in which an image showing the state of the subject at each time point is displayed as a live view image is as follows. The following problems occur.

  FIG. 5 shows a live view image displayed on the LCD 4 at each time point in the 12-second period for three fireworks continuously launched for 12 seconds after the full-pressing operation of the shutter button 9 is started. In FIG. 5, the state of the subject imaged 1/30 second before that is displayed as a live view image at each time point.

  In such a live view image display method, when the end of the main exposure operation is to be instructed at any timing within the 12 seconds, the fireworks locus is not displayed on the LCD 4 until the instructed timing. The person directly recognizes the state of the fireworks, predicts the timing when the desired state such as the state of the fireworks becomes the most vivid, for example, and determines the subject image to be captured at the prediction timing.

  However, in this case, it is unclear what image was obtained until a series of imaging operations of the imaging device 1 is completed, and the image recorded on the memory card m is significantly different from the desired image. There is.

  In view of the above points, in the present embodiment, the imaging apparatus 1 includes the long-second monitoring mode as described above, and when the long-second monitoring mode is set, a period until the shutter button 9 is fully pressed, That is, in addition to the period until the start of the main exposure operation is instructed, when the exposure time of the main exposure operation is significantly longer than the live view image display cycle (for example, 1/30 second), The state of the image recorded on the memory card m is also displayed on the LCD 4 in real time as a live view image (substantially equivalent to the period during which the pressed state continues).

  In order to realize such a configuration, as described above, a plurality of pixels of the image sensor 11 are divided into LV display pixels and recording pixels, and a period from when the main exposure operation ends to when a pixel signal is extracted. Reads out the pixel signal generated by the LV display pixel at a constant period (for example, 1/10 second), sequentially generates a live view image using the read pixel signal, and displays it on the LCD 4.

  Further, in the present embodiment, since the state of the image recorded on the memory card m is displayed on the LCD 4, as shown in FIG. 6, the period of the main exposure operation by the recording pixels when the long second monitoring mode is set, A live view image generated in the previous cycle (more precisely, an added image to be described later) and an image obtained in the current cycle are added to generate the current live view image. The live view image with the image to be superimposed is displayed.

  For example, as shown in FIG. 6, after 12 seconds from the start of the full-pressing operation of the shutter button 9, the image is superimposed as if the three fireworks launched with a time difference during the 12-second period were opened simultaneously. A live view image is generated, and the live view image is displayed on the LCD 4.

  Then, when the end of the main exposure operation by the recording pixel is instructed by releasing the full press operation of the shutter button 9, the imaging device 1 generates a recorded image using the pixel signal output from the recording pixel. At this time, a pixel signal is extracted from the LV display pixel for displaying a live view image. If the pixel signal is left as it is, an image is lost at a portion corresponding to the LV display pixel. 1, as will be described later, this missing portion is interpolated using a pixel signal obtained from a recording pixel.

  In order to realize the configuration as described above, as shown in FIG. 3, the control unit 18 functionally includes a first imaging control unit 19, a first recording control unit 20, an addition processing unit 21, and an aspect correction processing unit 22. , Second imaging control unit 23, second recording control unit 24, RAW interpolation unit 25, first image processing unit 26, second image processing unit 27, display control unit 28, image compression unit 29, first image storage unit 30 And a second image storage unit 31.

  When the long-second monitoring mode is set, the first imaging control unit 19 performs the vertical direction of the LCD 4 (FIG. 2) during the period from when the bulb photographing mode is set until the end of the main exposure operation is instructed by the shutter button 9. In the pixel arrangement direction corresponding to the arrow A direction), a plurality of pixel lines arranged at predetermined intervals are selected as LV display pixel lines, and the LV display pixels are exposed while performing an exposure operation. A clock output from the timing generator 12 to the image sensor 11 so that the pixel signal is extracted from the display pixel at a predetermined cycle (for example, 1/30 second), and the exposure operation is performed without extracting the pixel signal from the recording pixel. Controls the signal.

  For example, in FIG. 4A, among a plurality of vertical pixel lines in the image sensor 11, one pixel line is set as a pixel line for LV display for every 12 pixel lines. ) Shows the other pixel lines set as pixel lines for the recorded image, and the first imaging control unit 19 is the period until the end of the main exposure operation is instructed by the shutter button 9. The pixel signal is extracted from the pixels positioned on the LV display pixel line at a predetermined cycle (for example, 1/30 second), while the exposure operation is continued without extracting the pixel signal from the recorded image pixel line.

  FIG. 7 shows the time when 1 second has elapsed from the start timing of 3 fireworks fired continuously for 12 seconds from the start of the full-pressing operation of the shutter button 9 when the long-second monitoring mode is set, and 4 seconds. As an example, a time point that has passed, a time point that has passed 8 seconds, and a time point that has passed 12 seconds, an image (a) obtained from an LV display pixel at each time point, and an image ( b), an image (c) generated using the data after aspect correction, an image (d) generated using RAW data obtained from the recording pixels, and a recorded image recorded on the memory card m ( e).

  In the present embodiment, the image obtained when it is assumed that, for example, RAW data has been read from the recording pixels after one second has elapsed after the start of the main exposure operation by starting the full press operation of the shutter button 9 is shown in FIG. 7 (d-1), and the black line P in FIG. 7 (d-1) indicates that the pixel signal of the LV display pixel is used to display the live view image during the main exposure operation. This is a missing part.

  Note that, when the long-second monitoring mode is not set, the first imaging control unit 19 has a predetermined period from the LV display pixel during the period until the start of the main exposure operation is instructed by the shutter button 9. When a pixel signal is taken out (for example, 1/30 second) and the start of the main exposure operation is instructed by the shutter button 9, a recording pixel signal is generated by almost all pixels of the image sensor 11. However, no pixel signal is extracted. Accordingly, nothing is displayed on the LCD 4 during the main exposure operation.

  When the long recording mode is set, the first recording control unit 20 outputs RAW data (output from the LV display pixel through the A / D conversion unit 14 and the like during the period from the start to the end of the main exposure operation). The pixel signal corresponding to the black line P shown in FIG. 7D) is temporarily recorded in the first image storage unit 30. As shown in FIG. 7A, the RAW data obtained from the LV display pixels corresponding to the portion of the black line P recorded in the first image storage unit 30 is as shown in FIG. The image generated from the RAW data is recorded in the first image storage unit 30 in a form corresponding to the state in which the images are adjacent in the vertical direction.

  When the long-second monitoring mode is set, the addition processing unit 21 sequentially stores the RAW data every time RAW data is recorded in the first image storage unit 30 after the second period after the start of the main exposure operation. The addition process is performed, and the data subjected to the addition process is recorded in the first image storage unit 30.

  As shown in FIG. 7, for example, when 1 second has elapsed after the start of the main exposure operation, the addition processing unit 21 generates an image (a-1) including RAW data output from the LV display pixel at that time. Then, an addition image (b-1) is generated by adding the addition image composed of the RAW data that has been subjected to addition processing one cycle (for example, 1/30 second) before that time. Similarly, after the start of the main exposure operation, for example, at the time when 4 seconds have elapsed, when 8 seconds have elapsed, and when 12 seconds have elapsed, the addition processing unit 21 outputs the RAW data output from the LV display pixels at each time. Images (a-2), (a-3), and (a-4), and an addition image consisting of RAW data that has been added one cycle (for example, 1/30 second) before that time. The added images (b-2), (b-3), and (b-4) are generated.

  As described above, when an image composed of pixel signals obtained in the current cycle is represented as Pn (n = 1, 2,..., K−1, k), the exposure operation by the LV display pixel and the exposure operation are performed. The data Gn obtained by repeating the reading of the RAW data obtained n by n times becomes Gn = P1 + P2 +... + Pn.

  When the addition image generated by the addition processing unit 21 is displayed on the LCD 4 as shown in FIG. 7C, the aspect correction processing unit 22 corresponds the number of pixels of the data after the addition processing to the aspect of the LCD 4. The aspect-corrected data is recorded in the second image storage unit 31. As a result, when the long-second monitoring mode is set, the period of the main exposure operation (the period in which the shutter button 9 is fully pressed) and the live view image (moving object trajectory) in which the image of the subject at each time point is superimposed. Can be displayed on the LCD 4.

  When the end of the main exposure operation is instructed, the second imaging control unit 23 outputs a clock signal output from the timing generator 12 to the imaging device 11 so as to output a pixel signal from the recording pixel when the long second monitoring mode is set. When the long-second monitoring mode is not set, the clock signal output from the timing generator 12 to the image sensor 11 is controlled so that the pixel signal is output from almost all pixels.

  The second recording control unit 24 temporarily records the RAW data obtained through the A / D conversion unit 14 and the like under the control of the second imaging control unit 23 in the second image storage unit 31.

  The RAW interpolation unit 25 performs RAW interpolation processing using the RAW data recorded in the second image storage unit 31 by the second recording control unit 24. That is, for example, if the RAW interpolation unit 25 is finished when the main exposure operation has been completed for 12 seconds from the start, the RAW interpolation unit 25 determines that the RAW data missing for LV display (FIG. 7 (d- 4) is interpolated using the RAW data obtained through the processing of the A / D conversion unit 14 and the like from the recording pixels at that time. Details of the RAW interpolation processing will be described later.

  The first image processing unit 26, the aspect-corrected image output from the aspect correction processing unit 22, the RAW data after RAW interpolation output from the RAW interpolation unit 25 when the long-second monitoring mode is set, and the non-setting of the long-second monitoring mode At the time of setting, pixel data of each pixel (hereinafter referred to as RGB pixel data) is calculated from substantially all RAW data recorded in the second image storage unit 31. That is, for each pixel, the first image processing unit 26 determines, for example, four adjacent pixels (two G pixels and one R and B pixels) such that the pixel is located in the upper left corner. An RGB image is generated from the RAW data of these four pixels.

  The second image processing unit 27 performs R (R (based on white level corresponding to the light source), black level correction for correcting the black level to the reference black level for the RGB image processed by the first image processing unit 26. Red balance, G (green), B (blue) color component digital signal level conversion, white balance adjustment, R (red), G (green), B (blue) digital signal γ characteristics Γ correction is performed to correct.

  The display control unit 28 transfers pixel data of the image to the VRAM 16 in order to display the image output from the second image processing unit 27 on the LCD 4. Thus, when the long second monitoring mode is not set, the period until the main exposure operation is started, and when the long second monitoring mode is set, the main exposure operation period is added to the period until the main exposure operation is started. Among them, the state of the subject (the state of the image recorded on the memory card m in the long-second monitoring mode) can be displayed on the LCD 4 in real time as a live view image.

  The image compression unit 29 performs JPEG (Joint Picture Experts Group) such as two-dimensional DCT (Discrete Cosine Transform) conversion and Huffman coding on the pixel data of the recorded image subjected to the above-described various processes by the second image processing unit 27. Compressed image data is generated by performing a predetermined compression process according to the method, and an image file in which information (information such as a compression rate) relating to a captured image is added to the compressed image data is recorded in the memory card m.

  In the memory card m, image data is recorded in time series, and for each frame, the compressed image compressed by the JPEG method includes index information (frame number, exposure value, shutter speed, compression) about the captured image. Information such as rate, shooting date, flash on / off data at the time of shooting, and scene information).

  Next, imaging processing at the time of setting the long-second monitoring mode, which is a characteristic part of the imaging apparatus 1 of the present embodiment, will be described using a time chart shown in FIG.

  As shown in FIG. 8, when the long second monitoring mode is set, a live view image is generated and displayed by the pixel signal of the LV display pixel until the shutter button 9 is fully pressed at time T = T1. The control unit 28 displays the live view image on the LCD 4. Here, since it is assumed that the imaging device 1 is set to the bulb photographing mode, the mechanical shutter is kept open until the full-pressing operation of the shutter button 9 is released, and the LV display pixel. In addition, the recording pixel continuously performs the exposure operation until the release point.

  When the full-press operation of the shutter button 9 is started at time T = T1, the first imaging control unit 19 applies 1 to the LV display pixel from time T = T2, which is delayed for a predetermined time from the full-press start timing. While the pixel signal is output at a period of / 10 seconds, the exposure operation is continuously performed without outputting the pixel signal from the recording pixel.

  Then, after clearing the storage area of the first image storage unit 30, the first recording control unit 20 stores the pixel signal read from the LV display pixel at a period of 1/10 second in the first image storage unit 30. Let If the acquired pixel signal is obtained in the first cycle after the start of full-pressing of the shutter button 9, the addition processing by the addition processing unit 21 is not executed, and the aspect correction processing unit 22 and the first correction The two-image processing unit 27 performs various processes such as an aspect correction process. On the other hand, when the read pixel signal is obtained in the second and subsequent cycles, the addition processing unit 21 is output from the LV display pixel at each time point and is recorded in the first image storage unit 30. After performing the addition process for adding the added image and the addition image generated one cycle before that time, the aspect correction processing unit 22 and the second image processing unit 27 perform various processes such as an aspect correction process. .

  When the processing such as the aspect correction processing is completed, the display control unit 28 displays the processed image on the LCD 4 and the above-described processing is repeated until the full-press operation of the shutter button 9 is released. As a result, a live view image on which images captured at each time point are superimposed is displayed on the LCD 4.

  In FIG. 8, since the display cycle of the live view image is 1/30 second, until the pixel signal is acquired from the LV display pixel in the next cycle (1/10 second), each cycle is displayed. It shows that the image composed of the obtained pixel signals is displayed on the LCD 4 three times each. For example, the pixel signal “output 1” obtained from the LV display pixel in the first cycle, that is, the “accumulation 1” period after the shutter button 9 is fully pressed is designated as “image 1”. Displayed on the LCD 4 three times. In addition, the pixel signal “output 2” obtained from the LV display pixel in the second and subsequent cycles, for example, the “accumulation 2” period, which is the second cycle, is “image 2” configured by this pixel signal. And the above-mentioned “image 1” are added, and this added image “image 1 + 2” is displayed on the LCD 4 three times. FIG. 8 shows that the pixel signal readout operation (charge transfer to the vertical transfer path, etc.) of the LV display pixel during the full-pressing operation of the shutter button 9 was periodically performed 50 times. .

  Then, when the full-pressing operation of the shutter button 9 is released at time T = T3, the mechanical shutter is closed, and after the start of the full-pressing operation of the shutter button 9 at time T = T4 delayed by a predetermined time from the release timing, The exposure operation by the recording pixels that have been continuously performed is completed, and the second imaging control unit 23 reads out the pixel signal during the period of “accumulation α” from time T = T2 to time T = T4. The second recording control unit 20 outputs the pixel signal “output α” generated by the exposure operation by the recording pixels continuously performed to the recording pixel, and the second recording control unit 20 outputs the pixel signal “output α” to the first pixel. The image is stored in the two-image storage unit 31.

  Thereafter, the second image processing unit 27 performs processing such as black level correction on the pixel signal “output α” stored in the second image storage unit 31, and then the image compression unit 29 performs compression processing. The compressed image is recorded on the memory card m. The display control unit 28 displays an image obtained by performing various processes such as an aspect correction process on the added image “1+... +49” generated immediately before the full pressing operation of the shutter button 9 is released for a predetermined time. To display.

  Next, RAW interpolation processing by the above-described RAW interpolation unit 25, which is a characteristic part of the present embodiment, will be described with reference to FIG.

  4A shows an array of RAW data output from all the pixels of the image sensor 11 via the A / D converter 14 and the like. FIG. 4B shows the LV during the main exposure operation. The pixels used for display and recorded images are shown, and FIG. 8C shows a method for interpolating the missing RAW data by configuring the live view image to be displayed during the main exposure operation. .

  As shown in FIG. 4B, the image composed of the RAW data obtained from the pixels for the recorded image has RAW data missing from the LV display pixel lines at a predetermined interval in the vertical direction. As described above, in this embodiment, the missing pixel data is interpolated using the RAW data obtained from the pixels for the recorded image.

  At that time, when interpolation processing is performed in the state of RAW data and each RAW data array after interpolation is a Bayer array, an interpolation process is performed to obtain an image of the same size as when the long-second monitoring mode is not set In addition, the processing after the second image processing unit 27 can be made common to when the long second monitoring mode is not set. Therefore, as shown in FIG. 4C, for the pixel line indicated by the arrow A, for example, a G (green) pixel signal is input to the left pixel and a B (blue) RAW data is input to the right pixel. For example, for the pixel line indicated by arrow B, it is preferable to interpolate R (red) RAW data for the left pixel and G (green) RAW data for the right pixel.

  Therefore, in this embodiment, the RAW data of the missing pixel is interpolated using the RAW data of the pixel located on the same type of pixel line as the pixel line of the pixel, and the array of the RAW data after the interpolation is the Bayer array. It is trying to become.

  In addition, interpolation is performed using the same type of RAW data closest to the interpolation destination pixel so that an unnatural image such as the shape of the subject is not generated is generated.

  Specifically, in the present embodiment, with respect to the pixel line indicated by the arrow A, the left pixel has the same arrangement of the missing pixels as indicated by the arrow C, and the missing pixel line RAW data of the left G (green) pixel in the nearest pixel line (pixel line located at the upper side of the second column in FIG. 4) is interpolated, and the pixel line ( The RAW data of the right B (blue) pixel in the pixel line located at the upper side of the second column) is interpolated. As for the pixel line indicated by the arrow B, the pixel on the left side has the same pixel array as the missing pixel line as indicated by the arrow D and is closest to the missing pixel line (two columns). RAW data of the left R (red) pixel in the upper pixel line) is interpolated and the right pixel of the right G (green) in the pixel line (pixel line located two columns above) is interpolated. The raw data of the pixel is interpolated.

  As a result, the processing after the second image processing unit 27 can be made common with that when the long second monitoring mode is not set, and the occurrence of distortion, false color, or the like can be prevented or suppressed in the subject image.

  In addition, in the period until the start of the main exposure operation is instructed by the shutter button 9, the live view image is generated using the RAW data obtained from the LV display pixels. At the time of setting, the live view image can be reliably displayed on the LCD 4 in real time during the main exposure operation.

  Then, the RAW data is selected so that the array of each RAW data composing the live view image and the array of the RAW data after interpolation become a Bayer array, and processing such as image processing by the second image processing unit 27 is performed. Since the above-described interpolation processing is performed in the state of the raw data before the second image processing unit 27, the processing after the second image processing unit 27 can be made common with the time when the long second monitoring mode is not set. The interpolation processing is not complicated as compared with the configuration in which the pixel data after the image processing according to 27 is used to interpolate the pixel data to the missing part.

  The present invention is not limited to the above embodiment, and the following embodiments (1) to (11) can also be employed.

  (1) In the above embodiment, the RAW data of the missing pixel is interpolated using the RAW data of the pixel of the pixel line located above the second column of the pixel. You may make it interpolate using the raw data of the pixel of the pixel line located 2 columns below a pixel line to be processed, or you may perform an interpolation process as follows.

  9 to 11 are diagrams for explaining modifications of the missing image interpolation processing method.

  As shown in FIG. 9, assuming that pixels are missing as in the above embodiment (FIG. 4), for the pixel line indicated by arrow A, the left pixel includes two columns as indicated by arrows C and E. Of the pixel lines located on the upper side and the lower side of the second column, the average value of the RAW data of each G (green) pixel located on the left side is calculated, and this average value is interpolated as the RAW data of the pixel, In the right pixel, as shown by arrows C and E, the average value of the RAW data of each B (blue) pixel located on the right side among the pixel lines located on the upper side and the lower side of the second column is shown. The average value is calculated and interpolated as RAW data of the pixel.

  As for the pixel line indicated by the arrow B, the left pixel includes R (red) positioned on the left side of the pixel lines positioned on the upper side and the lower side of the second column as indicated by arrows D and F. The average value of the RAW data of each pixel is calculated, and this average value is interpolated as the RAW data of the pixel. As shown by arrows D and F, the right side pixel has two columns above and two columns below. The average value of the RAW data of each G (green) pixel located on the right side among the pixel lines located at is calculated, and this average value is interpolated as the RAW data of the pixel.

  This also makes it possible to share the processing after the second image processing unit 27 with when the long-second monitoring mode is not set, and further prevent the subject image from being distorted, false color, etc. Can be suppressed.

  In FIG. 10A, as shown by the oblique lines, four adjacent pixels are made one block, and processing is performed in units of blocks.

  As shown in FIG. 10A, the RAW data of the pixels belonging to the blocks BR1 and BR2 including the pixels of the missing pixel lines (pixel lines indicated by arrows A and B) are adjacent to the blocks BR1 and BR2. Setting is performed using the RAW data of the pixels of the blocks BR3 and BR4.

  Specifically, for the pixel line indicated by the arrow A, G (green) RAW data located in the upper left part of the block BR3 adjacent to the upper side of the block BR1 is displayed on the left pixel, and Interpolates B (blue) RAW data located in the upper right part of BR3. For the pixel line indicated by the arrow G, the RAW data of the left R (red) pixel is replaced with the R (red) RAW data located at the lower left of the block BR3, and the right G (green) pixel. Is replaced with G (green) RAW data located in the lower right part of the block BR3.

  Similarly, with respect to the pixel line indicated by arrow B, the left pixel has R (red) RAW data located in the upper left portion of the block BR4 adjacent to the upper side of the block BR2, and the right pixel has the block BR4. G (green) RAW data located in the upper right part is interpolated. For the pixel line indicated by the arrow H, the RAW data of the left G (green) pixel is replaced with the G (green) RAW data located in the lower left part of the block BR4, and the right B (blue) pixel. Are replaced with B (blue) RAW data located in the lower right part of the block BR4.

  As described above, since it is only necessary to interpolate or replace the RAW data of the same color pixel located in the vicinity of the missing pixel, a natural recorded image can be obtained by relatively simple processing. In this case, since the interpolation is performed in units of blocks, it is superior in reproducing an image such as a straight line as compared with the case of performing interpolation in units of pixel lines.

  In FIG. 10A, a pixel line to be processed and a pixel line below it constitute a block, and the RAW data of the pixels belonging to this block are converted to the pixels belonging to the block adjacent to the upper side of the block. Although interpolation or replacement is performed with RAW data, the present invention is not limited to this.

  For example, as shown in FIG. 10 (b), the processing target pixel line and the pixel line above it constitute blocks BR5 and BR7, and the RAW data of the pixels belonging to the blocks BR5 and BR7 are converted into the block BR5. Interpolation or replacement may be performed with RAW data of pixels belonging to the blocks BR6 and BR8 adjacent to the upper side of BR7. Alternatively, as shown in FIG. 10 (c), the processing target pixel line and the pixel line below it constitute blocks BR9 and BR11, and the RAW data of the pixels belonging to the blocks BR7 and BR11 are transferred to the block BR7, You may make it interpolate or substitute by the raw data of the pixel which belongs to the blocks BR10 and BR12 which adjoin below BR11.

  In FIG. 11, as shown by hatching, four adjacent pixels are made one block, and the processing is performed in units of blocks, as in FIG. 10, but the pixel line to be processed and the pixel line above it or A block is composed of the lower pixel lines, and RAW data of pixels belonging to this block is interpolated or replaced with average RAW data of pixels belonging to blocks adjacent to both sides of the block.

  As shown in FIG. 11A, for the arrow A, the pixel line indicated by the arrow I below forms one block BR13. With respect to the pixel line indicated by the arrow A, the average RAW data of each G (green) pixel located in the upper left part of each of the blocks BR14 and BR15 adjacent to the block BR13 on the upper and lower sides in the pixel on the left side. Interpolation is performed, and average RAW data of each B (blue) pixel located in the upper right part of each of the blocks BR14 and BR15 is interpolated to the right pixel.

  For the pixel line indicated by the arrow I, the RAW data of the left R (red) pixel is replaced with the R (red) RAW data located at the lower left of each of the blocks BR14 and BR15, and the right G (green) ) Pixel data is replaced with G (green) RAW data located in the lower right part of each of the blocks BR14 and BR15.

  In addition, as shown in FIG. 11B, for the arrow B, one pixel BR16 is constituted by the pixel line indicated by the arrow J above the arrow B. With respect to the pixel line indicated by the arrow B, the average RAW data of each G (green) pixel located in the upper left of each of the blocks BR17 and BR18 adjacent to the block BR16 on the upper side and the lower side in the pixel on the left side. Interpolation is performed, and average RAW data of each B (blue) pixel located in the upper right part of each of the blocks BR17 and BR18 is interpolated to the right pixel.

  For the pixel line indicated by arrow J, the RAW data of the left G (green) pixel is replaced with the G (green) RAW data located at the lower left of each of the blocks BR17 and BR18, and the right B (blue) ) Pixel data is replaced with B (blue) RAW data located at the lower right of each of the blocks BR17 and BR18.

  Even in this case, it is only necessary to interpolate or replace the average RAW data of the pixels of the same color located in the vicinity of the missing pixels, so that a natural recorded image can be obtained by a relatively simple process.

  (2) In the above embodiment, when generating a live view image, LV display pixels are selected in units of columns. However, the present invention is not limited to this, and as shown in FIG. Pixels separated by the interval (pixels displayed in black and white inversion) may be selected.

  At that time, LV display pixels are selected for each (2n + 1) (n is an integer of 1 or more) in the vertical direction and the horizontal direction so that the array of pixels selected as LV display pixels is a Bayer array. do it.

  For example, in FIG. 12, n = 2 and LV display pixels are selected every five columns in the vertical and horizontal directions (LV display pixels are shown in black). That is, in FIG. 12, when numbers are assigned from the top to the bottom in the vertical direction and from the left to the right in the horizontal direction, respectively, the fourth, ninth and fourteenth rows from the top in the vertical direction and from the left in the horizontal direction. A live view image is generated using RAW data of pixels located in the second and seventh columns.

  FIG. 13 shows a case in which 1 second has elapsed with respect to the three fireworks fired continuously for 12 seconds after the shutter button full-press operation is started when the long-second monitoring mode is set. As an example, a time point of 4 seconds, a time point of 8 seconds, a time point of 12 seconds, and an image (a) generated using RAW data of LV display pixels at each time point, and after the addition process FIG. 6 is a diagram showing an image (b) generated using RAW data of the image, an image (c) generated using RAW data of a recording pixel, and a recorded image (d) recorded on the memory card m. .

  As shown in FIG. 13B, an image obtained from the RAW data read from the recording image pixels after, for example, 1 second has elapsed after the shutter button 9 is fully pressed, is shown in FIG. 13C. The RAW data output from the LV display pixels corresponds to the black point P ′ in FIG. 13C, and if an image is generated using this RAW data, the RAW data output from the LV display pixel is shown in FIG. ).

  Then, when pixels for LV display are selected in this way, when the main exposure operation is completed, interpolation may be performed using the RAW data of the closest pixel of the same color. In FIG. 12, for example, for the G (green) pixel indicated by the arrow P, the average value of the RAW data obtained from the four G (green) pixels located around the shaded area is used as the RAW data for that pixel. That's fine.

  FIG. 12 shows the case where the pixels for the live view image are selected for every five columns in both the vertical direction and the horizontal direction. However, the present invention is not limited to this, and the LV display pixels are set every seven columns or every eleven columns. In addition, for example, LV display pixels are selected every 5 columns in the vertical direction and every 7 columns in the horizontal direction, and are selected as the pixels for live view image display in each direction. The intervals of the pixel lines to be performed may be different.

  (3) In the above embodiment, the LCD 4 is provided to display the image of the subject, and the live view image or the like is displayed on the LCD 4. However, the present invention is not limited to this, and an electronic viewfinder is provided instead of the optical viewfinder 5. In such a case, the live view image may be displayed on the electronic viewfinder.

  (4) In the above embodiment, the pixels located in one pixel line for every 12 columns are used for the live view image. However, the number of pixels of the image sensor 11 in the vertical direction and the LCD 4 are not limited to this ratio. Depending on the ratio to the number of display pixels, it can be set, for example, at a ratio of one for every six columns, or one for every 16 columns.

  (5) In the above embodiment, different pixel lines are selected for LV display alternately in the vertical direction, as shown in FIG. 4B, so that the array of LV display pixels is a Bayer array. In particular, when an imaging device with a high pixel count is used, as shown in FIG. 14, the pixel lines of the live view image are selected so that a predetermined number of the same type of pixel lines continue in the vertical direction in each cycle. When the raw data is read from the pixels located in these pixel lines and the raw data of the same color pixels are added to generate a live view image or a recorded image as one raw data, Insufficient exposure can be compensated.

  (6) In the above embodiment, the advantage when the imaging apparatus 1 is set to the long-time monitoring mode when the bulb shooting mode is set has been described. However, other shooting modes such as the manual mode are set. Even when the long-second monitoring mode is set, the plurality of pixels of the image sensor 11 are divided into LV display and recording as described above, and during the main exposure operation, the LV display pixels are divided. Alternatively, the pixel signal is read out to display the live view image, and when the main exposure operation is completed, the pixel signal is obtained from the recording pixel to generate the recorded image.

  (7) In the above embodiment, the progressive transfer type image sensor is adopted from the viewpoint of easy generation of the drive control signal for driving the image sensor, but the image pickup apparatus adopts the interlace transfer type image sensor. The present invention is also applicable.

  (8) In the above embodiment, the memory card m is used as a means for recording an image. However, the present invention is not limited to this, and the imaging apparatus 1 is connected to a device having a storage capacity such as a computer, a communication cable, or a communication network. If it is configured to be communicable via a computer, an image may be recorded on a device such as the computer.

  (9) The present invention can be applied to a so-called honeycomb CCD in which octagonal pixels are arranged in a checkered pattern as shown in FIG.

  That is, as shown in FIG. 15, after considering a row of adjacent RGB pixels as one pixel line (for example, R1, R2), a pixel line used for LV display is selected as in the above embodiment. You may make it do. Alternatively, the column of R and B pixels is regarded as one pixel line, and the column of G pixels is regarded as one pixel line, and a pixel line used for LV display is selected as in the above embodiment. Good. In this case, in order to obtain a beautiful live view image with less image distortion and false color, the arrangement of pixel signals taken out for LV display is perpendicular to the R and B pixel lines and the G pixel lines (FIG. 21). It is preferable to extract pixel signals alternately in the vertical direction).

  (10) In the above-described embodiment, a CCD in which primary color filters (R (red), G (green), and B (blue) color filters) are disposed on the light receiving surface of each photoelectric conversion element is employed as the image sensor 11. However, the present invention is not limited to this, and a CCD provided with complementary color filters (C (cyan), M (magenta), Y (yellow)) may be employed.

  (11) At the time of setting each shooting mode, using the technique in which a plurality of pixels of the image sensor 11 are divided into LV display pixels and recording pixels as described above, as in the above embodiment. Further, the following release time lag reduction mode can be provided.

  FIGS. 16A to 16G are timing charts showing the operations of the image sensor 11 when the release time lag reduction mode is set. FIG. 16A shows a transfer path (vertical transfer path and FIG. 16B shows the exposure operation by the LV display pixel, and FIG. 16C shows the output operation of the vertical synchronization signal (VD) applied to the image sensor 11. FIG. 16D shows the readout operation of the pixel signal generated by the exposure operation of the LV display pixel, FIG. 16E shows the live view image display operation, and FIG. Indicates the operation for instructing the end of the main exposure operation by the photographer, and FIG. 16G shows the recording operation in the release time lag reduction mode performed by the instruction operation. FIG. 16H shows the recording operation when the release time lag reduction mode is not set, in order to explain the difference between the recording operation when the release time lag reduction mode is not set and the recording operation when the release time lag reduction mode is set. It is a time chart which shows.

  As shown in FIG. 16, when the release time lag shortening mode is set, the charge remaining in the transfer path is transferred to the overflow drain (not shown) until the shutter button 9 instructs to start the main exposure operation. The cleaning operation to be swept away is performed at a period of 1/30 seconds, for example, and the exposure operation by the LV display pixel and the recording pixel is performed every time the cleaning operation is completed. When a vertical synchronization signal is applied to the image sensor 11, the pixel signal generated by the exposure operation by the LV display pixel is read out. An image generated from the read pixel signal is displayed almost in parallel with the pixel signal read operation in the next cycle. Note that pixel signals read from the recording pixels are discarded until a start instruction for the main exposure operation is issued.

  Then, when an instruction to start the main exposure operation by the shutter button 9 is performed during readout of the pixel signals from the LV display pixel and the recording pixel, the exposure operation by the recording pixel is continuously performed. By closing the shutter when a predetermined exposure condition is satisfied, the exposure operation by the recording pixels is finished.

  On the other hand, when the release time lag reduction mode is not set, if an instruction to start the main exposure operation is issued by the shutter button 9 while the pixel signal is being read from the LV display pixel and the recording pixel, the imaging device 1 transfers the image. After performing the path cleaning (removal of the read pixel signal), the exposure operation by almost all pixels is started, and for example, the mechanical shutter is closed at the shutter speed calculated while the shutter button 9 is half-pressed. The exposure operation is terminated.

  As described above, when the release time lag reduction mode is set, a recorded image is not generated using all the pixels as in the case where the release time lag reduction mode is not set, but some pixels (recording) are recorded. Therefore, the size of the recording image is smaller than that of the recording image obtained when the release time lag reduction mode is not set (about half in the vertical direction (vertical direction)).

  However, the charge accumulated in the recording pixels at the time when the start instruction of the main exposure operation is issued can be part of the pixel signal generated by the main exposure operation. In other words, it can be considered that the main exposure operation has already been performed before the timing at which the start instruction of the main exposure operation is performed. As a result, the mechanical shutter can be closed (the main exposure operation is terminated) at a timing closer to the timing at which the start instruction of the main exposure operation is performed compared to when the release time lag reduction mode is not set (T2 <T1). . Thereby, an image close to the subject image desired by the photographer can be generated as a recorded image.

  The imaging device 1 described above mainly includes the inventions shown in the following supplementary notes 1 to 9.

  [Supplementary Note 1] An image pickup device including a plurality of types of pixels that receive different colors, and a plurality of pixel lines in which pixels are arranged in one direction are arranged in a crossing direction that intersects the one direction; An input operation unit for instructing the imaging device to start an exposure operation for generating a recording pixel signal to be recorded in the recording unit, and when the exposure start instruction is given by the input operation unit, the imaging device An imaging control unit that performs an exposure operation and outputs RAW data at a predetermined time interval from a plurality of pixels located in a part of the plurality of pixel lines arranged in the crossing direction; RAW data is acquired from other pixels other than the plurality of pixels at a predetermined timing after the exposure start instruction, and the RAW corresponding to the plurality of pixels is based on the RAW data of the other pixels. An image processing unit for interpolating data, a first image generation unit for generating an RGB image from the interpolated RAW data generated by the image processing unit, and an RGB image generated by the first image generation unit An imaging apparatus comprising: a recording control unit that causes the predetermined recording unit to perform recording.

  [Supplementary note 2] The supplementary note 1, wherein the image processing unit performs the interpolation using RAW data of pixels located in the vicinity of the plurality of pixels as RAW data corresponding to the plurality of pixels. Imaging device.

  [Supplementary Note 3] The imaging element is formed by Bayer arrangement of the plurality of types of pixels, and the image processing unit uses four pixels that are in contact with each other at one vertex as a block and includes the plurality of pixels. The imaging apparatus according to appendix 2, wherein, for each pixel belonging, the alternative RAW data is interpolated or replaced using RAW data of a pixel belonging to a block adjacent to the block.

  [Supplementary Note 4] A RAW data obtained from the plurality of pixels is converted into predetermined pixel data, and an RGB image based on the pixel data is generated, and an RGB generated by the second image generation unit The imaging apparatus according to any one of supplementary notes 1 to 3, further comprising an image display unit that displays an image.

  [Supplementary Note 5] The image pickup element is formed by Bayer arrangement of the plurality of types of pixels, and the second image generation unit (2n + 1) in each of two orthogonal arrangement directions of the pixels included in the image pickup element. The imaging apparatus according to appendix 4, wherein the RGB image is generated based on RAW data obtained from pixels located on intersections of predetermined pixel lines set for each column. (N is an integer of 1 or more)

  [Supplementary Note 6] The input operation unit further has a function of instructing an exposure time of an exposure operation by the imaging device for obtaining an image to be recorded in the predetermined recording unit, and the image generation unit includes: The imaging apparatus according to any one of appendices 1 to 5, wherein the image is generated when an exposure time instructed by the input operation unit is longer than the predetermined time.

  [Supplementary Note 7] The second image generation unit sequentially adds RAW data obtained from the plurality of pixels every predetermined time to generate the RGB image, and the image display unit displays the RGB image. The imaging apparatus according to any one of appendices 4 to 6, wherein: According to the present invention, the RGB image is generated by sequentially adding the RAW data of the plurality of pixels obtained from the imaging element every predetermined time, and the image display unit displays the RGB image. Therefore, the state of the subject is superimposed and displayed on the image display unit during the exposure operation period for generating the recording pixel signal. Thus, the photographer can confirm the image recorded in the predetermined recording unit by visually recognizing the image displayed on the image display unit.

  [Appendix 8] The imaging control unit causes the plurality of pixels to output pixel signals at predetermined time intervals until the exposure start instruction is performed by the input operation unit, and when the exposure start instruction is performed, A first shooting mode in which an exposure operation performed by a plurality of pixels is continuously performed without outputting a pixel signal, and then the pixel signals are output to the plurality of pixels as recording pixel signals at a predetermined timing. When the exposure start instruction is issued, the generation of recording pixel signals is started for almost all pixels of the image sensor, and the pixel signals are output to the pixels as recording pixel signals at a predetermined timing. The imaging apparatus according to appendix 3 or 4, characterized by comprising: According to the present invention, in the second shooting mode, an image including a pixel signal used for display on the image display unit in the first shooting mode is generated, so that it is larger than that in the first shooting mode. A beautiful recorded image can be obtained. Further, in the first photographing mode, the exposure operation performed by the plurality of pixels is continuously performed without outputting the pixel signal, so that the exposure operation for generating the recording pixel signal is completed. The exposure operation can be terminated at a timing close to the instruction timing (timing at which an operation for determining a subject image is performed by the input operation unit). As a result, an image close to the subject image desired by the photographer can be obtained.

  [Supplementary Note 9] The supplementary notes 1 to 3, wherein the partial pixel lines that output pixel signals of an image to be displayed on the image display unit are alternately adjacent to different pixel lines in the intersecting direction. The imaging apparatus according to any one of 8. According to the present invention, the partial pixel lines that output the pixel signals of the image to be displayed on the image display unit are such that different pixel lines are alternately adjacent in the intersecting direction. A relatively beautiful image with few colors can be displayed on the image display unit.

It is a front view of the imaging device concerning the present invention. It is a rear view of the imaging device concerning the present invention. It is a block block diagram which shows the system of an imaging device. (A) shows an array of RAW data output from all pixels of the image sensor, and (b) is used for a live view image and a recorded image when the shutter button is fully pressed in the bulb photographing mode. (C) is a figure which shows the interpolation method of the raw data missing by having comprised so that a live view image might be displayed during exposure. It is a figure which shows an example of the conventional live view image. It is a figure which shows an example of the live view image displayed in the imaging device which concerns on this invention. For the three fireworks fired continuously for 12 seconds after the shutter button full-press operation is started, the image generated using the pixel signal of the recording pixel at each time point and the RAW after the addition processing An image generated using data, an image generated using RAW data after aspect correction, an image obtained from pixel signals output from pixels for live view images, and a recorded image recorded on a memory card FIG. It is a time chart which shows the imaging process in the long second monitoring mode of the imaging device which concerns on this invention. It is a figure which shows the other interpolation method of the missing RAW data. It is a figure which shows the other interpolation method of the missing RAW data similarly. It is a figure which shows the other interpolation method of the missing RAW data similarly. It is a figure which shows the other selection method of the pixel for live view images. When the selection method of FIG. 12 is adopted, when the long-second monitoring mode is set, when one second has elapsed for the three fireworks fired continuously for 12 seconds from the start of the main exposure operation, As an example, the time point after 2 seconds, the time point after 8 seconds, and the time point after 12 seconds are generated at each time point using an image generated using the pixel signal of the recording pixel and the RAW data after the addition process. 2 is a diagram showing an image generated using an RAW data after aspect correction, an image generated using a pixel signal of a recording pixel, and a recorded image recorded on a memory card. It is a figure which shows the other selection method of the pixel for live view images. It is a figure which shows the modification of an image pick-up element. It is a time chart which shows the exposure operation at the time of release time lag shortening mode setting and non-setting.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 4 LCD display part 7 Shutter button 10 Long second monitoring switch 11 Imaging element 18 Control part 19 First imaging control part 20 First recording control part 21 Addition processing part 22 Aspect correction processing part 23 Second imaging control part 24 2 recording control unit 25 RAW interpolation unit 26 first image processing unit 27 second image processing unit 28 display control unit 29 image compression unit 30 first image storage unit 31 second image storage unit

Claims (6)

An image sensor comprising a plurality of types of pixels each receiving a different color, and a plurality of pixel lines arranged in one direction and arranged in a crossing direction intersecting the one direction;
An input operation unit for instructing the image sensor to start an exposure operation for generating a recording pixel signal to be recorded in a predetermined recording unit;
When the exposure start instruction is given by the input operation unit, the image sensor is caused to perform an exposure operation, and among the plurality of pixel lines arranged in the intersecting direction, a plurality of pixels located in some pixel lines An imaging control unit for outputting RAW data at predetermined time intervals from the pixels;
Image processing for acquiring RAW data from other pixels other than the plurality of pixels at a predetermined timing after the exposure start instruction and interpolating RAW data corresponding to the plurality of pixels based on the RAW data of the other pixels And
A first image generation unit that generates an RGB image from the interpolated RAW data generated by the image processing unit;
An image pickup apparatus comprising: a recording control unit configured to record the RGB image generated by the first image generation unit in the predetermined recording unit.   The imaging apparatus according to claim 1, wherein the image processing unit performs the interpolation using RAW data of a pixel located in the vicinity of the plurality of pixels as RAW data corresponding to the plurality of pixels. .   The imaging element is formed by Bayer arrangement of the plurality of types of pixels, and the image processing unit uses four pixels that are in contact with each other at one vertex as a block, and each pixel belonging to the block including the plurality of pixels On the other hand, the imaging apparatus according to claim 2, wherein the alternative RAW data is interpolated or replaced using RAW data of pixels belonging to a block adjacent to the block.   RAW data obtained from the plurality of pixels is converted into predetermined pixel data, a second image generation unit that generates an RGB image based on the pixel data, and an RGB image generated by the second image generation unit is displayed. The imaging apparatus according to claim 1, further comprising an image display unit.   The image sensor is formed by Bayer array of the plurality of types of pixels, and the second image generation unit is arranged for each (2n + 1) columns in two orthogonal array directions of the pixels included in the image sensor. The imaging apparatus according to claim 4, wherein the RGB image is generated based on RAW data obtained from a pixel located on a set intersection of predetermined pixel lines. (N is an integer of 1 or more)   The input operation unit further has a function of instructing an exposure time of an exposure operation by the imaging device for obtaining an image to be recorded in the predetermined recording unit, and the image generation unit is configured to input the input operation unit. The imaging apparatus according to claim 1, wherein the image is generated when an exposure time indicated by the is longer than the predetermined time.

Images