JPH0411471A - motion detection device - Google Patents

Abstract

(57) [Summary] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

[Detailed Description of the Invention] (Industrial Application Field) The present invention can be used in an anti-shake device that compensates for movement caused by camera shake or vibration in an image being captured by a camera, an automatic tracking device that tracks a moving subject, etc. The present invention relates to a preferred motion detection device.

(Prior art) In recent years, the development of video equipment such as video cameras and electronic cameras has been remarkable, and in order to enable more reliable and comfortable shooting operations, image movement caused by camera shake, shaking, etc. can be corrected and shaken. The camera incorporates a motion compensation device that enables high-quality photography without any distortion.

Motion compensation methods for motion compensation devices include mechanical compensation that uses inertia to keep the axes of the lens and image sensor constant against rotation of the camera body, and optical compensation that uses optical members such as variable apex prisms. There are two methods: an image processing correction method that performs correction by moving the screen through image processing, and the like.

The mechanical correction method requires a special mechanical structure to support the lens and imaging system, while the optical correction method requires special optical members such as the variable apex prism mentioned above. In contrast, image processing correction methods do not require any special mechanical structure or optical components, and have the great advantage of being able to perform motion correction using only signal processing using electric circuits. expected to go.

(Problems to be Solved by the Invention) However, the motion correction device using the conventional image processing method described above has the following drawbacks compared to mechanical and optical devices.

In other words, when performing motion correction using an image processing method, the image has movement when it is captured by an image sensor or image pickup tube, and in post-processing, the image is shifted according to the amount of movement of the image. It was constructed in such a way that the movement of the image was stopped.

For this reason, the images obtained at the imaging stage are blurred due to movement, and even if the image movement is corrected in the subsequent processing, the resolution of the final image is low (and only poor quality images are obtained). I couldn't do it.

(Means for Solving Interval Points) The present invention was made for the purpose of solving the above-mentioned problems, and its features are as follows: Detecting the movement of an image and compensating for the movement of the image. A motion detection device comprising a motion detection means for detecting motion of an image, a correction means for correcting the motion of the image based on the output of the detection means, and a filtering process for compensating for deterioration in resolution due to the motion of the image. The present invention provides a motion detection device comprising filter means for performing the following steps.

(Function) This allows filtering to compensate for deterioration in resolution due to image movement, resulting in high resolution and
It is possible to obtain motion-corrected images without shaking or blur.

In particular, a high-quality motion-corrected image can be obtained by adaptively filtering using the motion vector of the image obtained from the motion amount detection device used for motion correction.

(Embodiments) Hereinafter, embodiments of the motion compensation device according to the present invention will be described in detail with reference to the respective figures.

FIG. 1 is a block diagram showing a first embodiment of a motion compensation device according to the present invention.

In the figure, 1 is a subject, 2 is an imaging lens, and 3 is an imaging element such as a CCD or an imaging tube that photoelectrically converts the subject image formed on the imaging surface by the photographic lens 2 and outputs an imaging signal. 4 is an amplifier that amplifies the image signal output from the image sensor 3 to a predetermined level, and 5 is an AD that converts the input analog image signal into a digital signal.
A converter, 6 is a frame memory for storing the image signal converted into a digital signal by the AD converter 5, and 7 is a motion amount detection circuit for determining a motion vector from the AD converted image signal. As a method for calculating the motion vector, methods such as the so-called representative point matching method and the gradient method can be used. Reference numeral 8 denotes a memory readout circuit for generating a readout address and executing readout in order to read out image signals from the frame memory 6.

9 is a parameter setting circuit for setting filtering parameters for preventing image deterioration, and IO is a filter.

11 is a DA converter that converts the digital image signal that has passed through the filter 10 into an analog image signal; 12 is a synchronization signal adding circuit that adds a synchronization signal to the image signal; and 13 is an output video signal.

The photographic lens 2 forms an image of the subject 1 on the imaging surface of the image sensor 3. The image on the image sensor 3 is captured by the lens 2) the image sensor 3,
Or, it is accompanied by movement due to the movement of the subject 2. The image signal output from the image sensor 3 is amplified by an amplifier 4, converted to a digital signal by an AD converter 5, and then
It is temporarily stored in the frame memory 6.

The digital image signal output from the AD converter 5 is also sent to the motion amount detection circuit 7. The motion vector data determined by the motion amount detection circuit 7 is sent to a memory readout circuit 8 and a parameter setting circuit 9.

The motion detection circuit 7 requires data from a screen one frame or one field before the current screen in order to calculate the amount of motion, and therefore needs to have a frame memory. This frame memory may have a common configuration with the frame memory 6, or may be provided separately.

The memory read circuit 8 generates an address for reading from the frame memory 6 multiplied by an offset based on the motion vector data. Thereby, the data read out from the frame memory 6 is moved almost in the opposite direction to the movement of the image, and the movement of the image is corrected for being read out. That is, image blur correction is performed on the memory.

The parameter setting circuit 9 determines parameters such as filter coefficients for filtering from the motion vector obtained from the motion amount detection circuit 7 and sends them to the filter 10. The filter 1O performs filtering on the motion-corrected image signal read out from the frame memory 6 to reduce blurring or deterioration in resolution caused by the motion of the image on the image sensor 3.

Further, the filter 10 has characteristics of a bypass filter and a bandpass filter, as will be described later.

The image signal output from the filter 10 is sent to the DA converter 11
is converted into an analog signal by synchronous signal addition circuit 1.
2 is combined with a synchronizing signal and output as a video signal 13.

Improving resolution through filtering will be explained below.

FIG. 2 is a plan view showing the movement of the image.

21 is an output screen. The output screen 21 has fixed coordinates and serves as a reference, and processes such as image stabilization are performed on this. For example, consider the screen of a monitor display.

22 is a previous image, and 23 is a current image. 24 is the previous image 22
This is the motion vector when moving toward the current image 23.

On the output screen 2I, the video signal 13 of FIG. 1 is displayed. For example, when the subject 2 is a stationary object, an image displayed due to camera shake or the like moves from a previous image 22 to a current image 23 when moving from one time to another.

In the motion correction method using image processing, a motion vector 24 is calculated in the motion amount detection circuit 7, and when reading data of the current image 23 from the frame memory 6, the motion vector 24 is
The motion correction is performed by shifting the data by 4 on the output screen 21 and adding an offset to the readout address so that the current image 23 almost overlaps the previous image 22.

Here, since the current image 23 is accompanied by movement, the value of each pixel is integrated in the direction of the motion vector 24. Therefore, the current image 23 shown in FIG. 2 actually only shows the positions of the centers of gravity of the sides of each figure.

FIG. 3 is a front view showing the movement of the optical system.

In the figure, 31 indicates one cell when the image sensor is a solid-state image sensor such as a CCD. For convenience of explanation, it is assumed that the direction in which the image moves matches the direction in which one of the cells of the image sensor 3 is arranged. The image sensor 3 photoelectrically converts the pattern of the optical image on the image sensor 3 during a predetermined exposure time. 32 is an optical image at the start of exposure, and 33 is an optical image at the end of exposure. As in the case of FIG. 2, the optical image 32 at the start of exposure moves at the end of exposure due to the movement of the surface image during exposure, and becomes the optical image 33 at the end of exposure.

The motion vector 24 shown in FIG. 3 is the optical image 3 at the start of exposure.
2 and the optical image 33 at the end of exposure. This is almost the same as shown in FIG. 2, but strictly speaking there are some differences. In other words, in Figure 2, 2
It is a motion vector of an image between approximately the middle time of each of the exposure times of the images. On the other hand, in FIG. 3, the motion vectors of the currently targeted image are shown from the start of exposure to the end of exposure. Therefore, when the amount of motion of the image changes rapidly, the values of the two motion vectors may differ. - For boat fishing, it is safe to assume that the two are almost the same, but what is determined by the motion amount detection circuit 7 is the motion vector 24 shown in FIG.
The motion vector 24 shown in the figure may be used after some correction.

FIG. 4 is a graph showing characteristics of a filter for filtering.

In the figure, 41 is a point spread function based on the movement of the image. The horizontal axis X, which is a spatial coordinate, is taken along the direction of movement. The length of the motion vector 24 is assumed to be a. At this time, the image formed by one point of the subject 1 moves during the exposure time and becomes approximately like a dotted line distribution function 41.

If this is h (x), then h (x)=Rect(x/a) −−−−−−(1
).

42 is a frequency characteristic indicating image deterioration due to movement.

If f is the frequency and the frequency characteristic 42 is expressed as H(f),
Since H(f) is the Fourier transform of the point spread function h (x), H(x) = (sin uaf) /xf ---(
2).

43 is the frequency characteristic of the filter, and when expressed as P(f), P (f)=1/H(f)=πf/(sin xaf)---(3). Such a filter is called an inverse filter.

That is, H(f) ・P (F)=1, and P(f
), the deterioration of the image resulting in H(f) is compensated for. By filtering using this filter in the filter 10 of FIG. 1, the resolution of the output video signal 13 becomes high (
Good image quality can be obtained. However, since the inverse filter becomes infinite at the frequency where H(f) becomes O, it can only be realized approximately. Further, the frequency range may be a range where the frequency spectrum of the image signal exists.

As is clear from FIG. 4 and equations (2) and (3), the magnitude a of the motion vector 24 is included as a parameter in the degradation characteristic H(f) and the compensation filter characteristic P(f). Furthermore, the X-axis and the f-axis are along the direction of the motion vector 24, and it can be seen that P (f) depends on the magnitude and direction of the motion vector 24.

Therefore, it is desirable that the filter 10 adaptively change the filter according to the motion vector 24.

There are two methods for implementing filtering processing in the filter 10. One is filtering on the frequency axis, in which the image signal read from the frame memory 6 is Fourier transformed using FFT (fast Fourier transform method), multiplied by an inverse filter P (f) and inverse Fourier transformed, and the filtered image is obtained. It's about getting a signal.

Another method is filtering on the time axis,
In this method, the inverse filter P (f) is subjected to inverse Fourier transform to obtain an impulse response, which is then convolved with the image signal from the frame memory 6 to obtain a filtering output.

When realizing an inverse filter by convolution on the time axis, the impulse response S (
You may start from x).

5(X) = to δ゛(X) ・・・・・・・・・ (4) Here, K is the proportionality constant, δ(x) is the delta function, δ゛(
X) is the derivative of the delta function.

Also, * is a symbol representing convolution, and sgn
(x) is .

Note that equation (4) is obtained by Fourier transforming equation (3) using a delta function.

FIG. 5 is a graph showing the impulse response of the inverse filter.

51 is the impulse response of the inverse filter, (
4) represents the formula.

Equation (4) continues indefinitely in the X-axis direction, so it must be terminated midway. For this reason, it is desirable to multiply the signal by a window function such as a Hamming window before using it for filtering processing.

In an inverse filter, the characteristic is to have a large gain in the frequency region where most of the image information is lost due to image deterioration, that is, around the frequency where the value of the frequency characteristic 42 is O, and in the high frequency region where there is originally little image information. This results in an output image with a poor SN ratio. Therefore, in the filter 10, a Wiener filter or the like may be used.

The frequency characteristics of Wiener filter R(f) can be expressed as follows.

I4■(f) ・・・・・・・・・・・・ (5) Here, ΦI, (f), Φ, (fl are noise,
The power spectrum of the image signal is shown, and [Yes] indicates a complex conjugate.

Here, Φ, (f), Φ, (f) are difficult to obtain accurately, so Φ, (f) are assumed to be white noise and are constant.
Φ and (f) are Gaussian. Or Φ. (f)/
An assumption is made that ゛Φ, (f) is constant over all frequencies, and is determined in advance.

In the Wiener filter, when the frequency of the signal component is sufficiently large compared to the noise component, the value becomes almost the same as that of an inverse filter, and conversely, the value approaches O when the noise component is larger than the signal component.

FIG. 6 is a graph showing the frequency characteristics of the Wiener filter. 61 is the frequency characteristic of the Wiener filter. When compared with the frequency characteristic 43 of the inverse filter, it is clear that the gain is small in the frequency region with a poor SN ratio.The Wiener filter also adaptively changes its characteristics according to the motion vector 24. It is desirable that this is the same as in the case of an inverse filter.

In the filter 10, the filter characteristics of the various other filters 1O generally have characteristics of a bypass filter or a bandpass filter.

FIG. 7 is a block diagram showing a second embodiment of the present invention.

The second embodiment shows an apparatus that is effective when there is a moving area and a still area in an image, and when the moving area is divided into a plurality of parts and each has a different motion vector.

71 is the output of the motion amount detection circuit 7, which is a motion vector for each small block or pixel in the screen. 72 is an area discrimination circuit, 73 is its output as an address offset signal, and 74
．． Reference numeral 75 denotes other outputs of the region discrimination circuit 72, which are a region signal and a motion vector within the region, respectively.

76 is a switch that sends the input signal to either of the two output lines. 77 is an output video signal that has been processed for each area.

The steps up to the stage of AD converting the human input image signal, storing it in the frame memory 6, and sending it to the motion amount detection circuit 7 are the same as in the first embodiment.

The motion amount detection circuit 7 converts the motion vector 71 into the area discrimination circuit 7.
Send to 2.

Based on this, the area discrimination circuit 72 divides the screen into a still area and a plurality of moving areas having motion vectors. The area discrimination circuit 72 selects a desired area from among the divided areas and sends the motion vector of that area to the memory reading circuit 8 as an address offset signal 73. Based on this, the memory read circuit 8 performs address offset when reading the image signal from the frame memory 6. This shifts the entire screen.

The area determination circuit 73 sends the area signal 7 to the parameter setting circuit 9.
4 and a motion vector 75 within the area, and the parameter setting circuit 9 sends a filter with different characteristics for each area to the filter 10.
Set to . The area signal 74 is also sent to a switch 76, and the switch 76 sends the image signal from the frame memory 6 to the DA converter 11 in the case of a still area, and to the filter 10 in the case of a moving area. The output of filter 10 is then sent to DA converter 11. That is, filtering processing is performed only in the case of a moving region.

Note that the switch 76 can also be used together with the filter 10 by applying a filter that transmits all frequency bands in the case of a static region as a special case.

The analog output signal of the DA converter 11 is added with a synchronizing signal by a synchronizing signal adding circuit 12 and sent out as an output video signal 77 subjected to processing for each region.

FIG. 8 is a plan view showing an image including a plurality of movements.

In the figure, 81 is an output screen, which corresponds to the screen of a monitor display.

8283 are the first images of the previous screen.

This is the second two images. 84 and 85 are the first on the current screen respectively.
This is the second image. Further, 86 is a background image, which in FIG. 8 is an image in which small squares are lined up.

When transitioning from the previous screen to the current screen, the first and second images 8283 of the previous screen are moving with respect to the output screen 81,
They become the first and second images 84 and 85 of the current screen, respectively. However, the direction and magnitude of movement of the two images are different. Furthermore, the background image 86 is not moving here, and is the same between the previous screen and the current screen.

FIG. 9 is a plan view showing the area determination results.

91.92 are the 1st. This is the second motion area. 9
3.94 are respectively the 1st. These are motion vectors within the second motion region 91,92. 95 is a stationary area.

The area discriminating circuit 72 in FIG. 7 performs area division and calculation of intra-area vectors as shown in FIG.

That is, the first. Second motion area 91,92 and stationary area 9
5 is divided, and a motion vector 93.94 within the region is determined for the first and second motion regions 91.92.

If the device shown in FIG. 7 is a tracking device, tracking of a specific image is performed. Second image 83 on the current screen
is being tracked, the region determining circuit 72 sends a motion vector 94 within the second motion region 92 to the memory reading circuit as an address offset signal 73. At this time, an offset is applied to the reading of the image signal from the frame memory 6 and the image is shifted, resulting in an output screen 81.
Above, the second image 84 on the current screen is the first image 83 on the previous screen.
displayed in the same position. Images in other areas are shifted.

The switch 76 switches the first. Only the image signals of the second moving area 91 and 92 are sent to the filter 10, and the image signals of the still area 95 are sent directly to the DA converter 11.

In the filter 10, the first. For the images in the second motion region 91.92, the motion vectors within the region 93.94, respectively.
A filter with different characteristics depending on the parameter setting circuit 9
is set and filtering processing is performed. The settings of the filter and parameters used in the filter 10 are the same as in the first embodiment.

(Effects of the Invention) As described above, according to the motion detection device of the present invention, the deterioration in image quality caused by image movement is compensated for by performing filtering processing, and the resolution of the output image of the motion correction device is increased. This has the effect of producing high-quality images.

Furthermore, the characteristics of the filter used for filtering processing are
In particular, by adaptively changing the motion vector based on the motion vector originally obtained for motion correction, the image quality can be made close to the optimum, and there is an effect that the cost does not increase.

[Brief explanation of the drawing]

FIG. 1 is a block diagram showing a first embodiment of a motion detection device according to the present invention, FIG. 2 is a diagram showing image movement, FIG. 3 is a diagram showing movement of an optical image, and FIG. 4 is a diagram showing filtering. FIG. 5 is a characteristic diagram showing the characteristics of the filter; FIG. 5 is a characteristic diagram showing the characteristics of the inverse filter; FIG. 6 is a characteristic diagram showing the characteristics of the Wiener filter; FIG. 7 is a block diagram showing the second embodiment of the present invention. 8 is a diagram showing an image including a plurality of movements, and FIG. 9 is a diagram showing the region discrimination results. 2nd Seki

Claims (2)

[Claims] (1) A motion detection device that detects the motion of an image and compensates for the motion of the image, comprising a motion detection means that detects the motion of the image, and a correction that corrects the motion of the image based on the output of the detection means. A motion detection device comprising: a means for detecting a motion; and a filter means for performing a filtering process to compensate for deterioration in resolution due to the movement of the image. (2) The motion detection device according to claim (1), wherein the characteristics of the filter section are configured to be adaptively determined according to the motion of the image.