JPH0730895A - Picture processor and its processing method - Google Patents

Abstract

PURPOSE: To provide an image processor which efficiently compresses a moving image, is inexpensive and has excellent reliability and its processing method. CONSTITUTION: This image processor has an FDCT transforming part 12 which performs DCT transformation of eight pixels * eight lines image information and a coefficient compensation circuit 20 that compensates for the frequency coefficient of a transformed signal. The circuit 20 is provided with frame memory 28 which accumulates a signal that represents a frequency space of one frame ahead and a deciding circuit 30 that corrects the frequency coefficient of the memory 28 so as to correspond to the movement of an moving image between frames based on signals from the memory 28 and the part 12. Thus, efficient compression is carried out by encoding inter-frame finite difference between the current frame and a preceding frame which is optimally corrected.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and a processing method for digitally processing image information,
In particular, it relates to a method for efficiently compressing a moving image.

[0002]

2. Description of the Related Art In recent years, research and development of multimedia systems that handle various data such as images and sounds in an integrated manner have been advanced, and along with this, accumulation and transmission of digitized still / moving images are indispensable. Has become.

However, when the image information is digitized,
The amount of data is very large compared to voice data and the like. For example, when a color moving image of 720 * 480 pixels is digitized, a high data rate of several 100 Mb / s is required, and there is a problem in terms of transmission speed and recording medium to execute this.

In order to solve such a problem, a technique for compressing image information has been developed. CCITT is the standard for videophones and video conferences, and CMTT / 2 is the standard for TV transmissions. On the other hand, for recording media, still images are JPEG (Joint).
Photographic Expert Group
p), the moving image is an MPEG (Moving Picture).
Re Expert Group) is being standardized internationally.

In JPEG, as a still image compression / encoding method, there is a method of performing intraframe compression based on two-dimensional DCT (discrete cosine transform). According to this method, as shown in FIG. 8, image information divided into blocks of 8 (pixels) * 8 (lines) is input to the FDCT conversion unit 100, where it is converted into a signal representing a frequency space. . Frequency coefficient (DC
The T coefficient) is quantized by the quantization unit 102, and then encoded by the encoding unit 104 by zigzag scanning, run length encoding, or the like. The data encoded in this way is
It is transmitted via a transmission line or stored in a storage medium. To reproduce the image data, that is, to decompress the compressed image data, the data compressed by the procedure reverse to the above may be sequentially decoded, dequantized, and inverse DCT-transformed.

On the other hand, some methods for compressing moving images have been developed. Similar to JPEG described above, a method of repeatedly compressing a still image of each frame, a method of utilizing the correlation in the time domain between frames and performing DCT conversion of the difference between frames (one without motion compensation in MPEG),
Or, in addition to the difference between frames, in addition to this,
There is a method of performing inter-frame prediction by performing motion compensation.

Here, inter-frame prediction by motion compensation means not only coding of inter-frame difference, but also obtaining a predictive value from a motion vector with respect to a reference frame and encoding the difference and the predictive value to obtain more data. It is intended to be compressed. That is, since the image signal is composed of a frame in which a series of moving objects are recorded, the position of the object in the frame is adjusted or moved to effectively compress the inter-frame difference in the continuous image signal. It is based on the idea that you can.

However, in order to perform inter-frame prediction by such motion compensation, today's technology requires very expensive hardware. This problem is less important in broadcast or distribution applications where the source video is compressed (encoded) at the central station at once and distributed to tens of thousands of receivers (decoders). On the other hand, compression and compression / expansion (recording and playback) become a major hardware problem when executed in an immediate application such as a multimedia personal computer or a home video device (VCR, canceller, etc.). .

Therefore, from a practical point of view, the above-described intraframe compression is used as the compression and coding method for moving images rather than the interframe prediction by motion compensation (JP
Iterative compression of a still image in EG) or DCT conversion (M
It is preferable to use one that does not perform motion compensation in PEG).

FIG. 9 shows a method of performing DCT conversion on a difference between frames for encoding. As shown in the figure, the image signal is input to the subtractor 120 in block units of 8 * 8 pixels, where the difference from the image signal of the block corresponding to the previous frame stored in the frame memory 120 is Be taken. The signal representing the difference from the subtractor 120 is DCT-transformed into a signal representing frequency space by the FDCT transform unit 124, and the transformed frequency coefficient is encoded via the quantizing unit 126 and the encoding unit 128. Further, the output of the quantization unit 126 is returned to the state before the DCT conversion via the inverse quantization unit 130 and the inverse FDCT conversion unit 132, and then added with the output of the frame memory 122 by the adder 134, and one frame before. Image data is stored in the frame memory 122. In this way, the encoding unit 12
8 outputs encoded image data.

[0011]

However, in the conventional compression of a moving image as shown in FIG. 9, only a simple inter-frame difference is performed for the pixel space. The difference according to the magnitude is uniformly expressed in the frequency coefficient by the DCT transform. For example,
When the 8 * 8 block difference is DCT-transformed, frequency coefficients corresponding to 64 pixels are distributed in the frequency space as shown in FIG. Then, the high frequency component represents a coefficient corresponding to a pixel level motion, and the low frequency component represents a coefficient corresponding to a block level motion.
Generally, when the motion of an object is small, that is, when the difference between frames is smaller than 1 pixel (sample / frame is smaller than 1), the DC between frames is reduced.
T has excellent compression efficiency. On the contrary, when the movement of the object increases, that is, when the inter-frame difference is 2 pixels or more (sample / frame is 2 or more), the difference is directly represented in the high frequency component. Therefore, a large number of bits are required to encode the difference value, and as a result, the compression efficiency is extremely deteriorated.

Therefore, when the difference between frames is used,
Compared to JPEG, which repeatedly performs in-frame compression of a still image, the movement in which compression is more efficient is limited to a very small and impractical range. This is a general reason for adopting the JPEG method rather than the inter-frame DCT in moving image compression.

An object of the present invention is to solve the above-mentioned problems of the prior art and to provide an inexpensive and highly reliable image processing apparatus which can efficiently compress a moving image and a processing method thereof. .

Another object of the present invention is to provide an image processing apparatus and a processing method thereof capable of data compression of input image information in frequency space. Another object of the present invention is to improve the prediction efficiency by giving each frequency coefficient an optimum prediction coefficient according to the movement of an object, and to perform efficient data compression while maintaining high image quality. An apparatus and a processing method thereof are provided.

[0015]

The present invention discloses a novel technique called so-called coefficient compensation, which improves the conventional JPEG system and inter-frame DCT system.

The basic idea is that even with a two-dimensional still image signal (Markov) represented by a single correlation coefficient in the pixel space, the correlation coefficient between frames in the Fourier transform (DCT) domain is a coefficient. However, in the conventional approximation of the inter-frame difference in the pixel region, the correlation coefficient between frames has a constant value, and in most cases, the prediction coefficient is constantly 1. (Set to 0), the prediction efficiency is improved by giving an optimum prediction coefficient to each frequency coefficient.

Therefore, in order to solve the above-mentioned problems of the prior art, the image processing apparatus according to the present invention inputs the image information of m pixels * n rows, and outputs the image information as the first image data representing the frequency space. Conversion means for converting into the following, and storage means for storing the second image data signal converted by the conversion means and representing the frequency space corresponding to the image information of the previous frame,
An optimizing means connected to the converting means and the storing means for correcting the second image data signal so as to correspond to the motion of the moving image between frames based on the first and second image data signals; It has an encoding means for encoding the difference value between the second image data signal corrected by the encoding means and the first image data signal.

In the image processing method according to the present invention, the input image information of m pixels * n rows represents the first frequency space.
Corresponding to the movement between the frames of the image information from the first image data and the second image data representing the frequency space of the image information of the previous frame stored in the frame memory. Selecting a prediction coefficient and modifying the second image data signal according to the selected prediction coefficient,
The difference value between the corrected image data signal and the first image data signal is encoded.

[0019]

According to the present invention, since input image information can be converted into data representing a frequency space and each frequency coefficient of this data can be compensated according to the motion of a moving image, the prediction efficiency is improved and the image quality is improved. Efficient data compression is possible without deterioration. Therefore, when compressing a moving image according to the present invention, it is possible to utilize the advantages of JPEG intra-frame compression and the advantages of MPEG inter-frame compression.

[0020]

EXAMPLES Examples of the present invention will be described below. As described above, the present invention discloses a new technique called so-called coefficient compensation, which allows the conventional JPEG to be used.
This is to improve the system and interframe DCT system.

The coefficient compensation is performed by giving an optimum prediction coefficient to each frequency coefficient.
Is obtained as follows.

If there is one frame and the next frame, the image signals on these are

[0023]

[Equation 1] Then, the variance of signals within a frame and between frames (assuming the average to be 0) is

[0024]

[Equation 2] It is represented by. Since the dispersion corresponds to the frequency = 0 part of the power spectral density, the equations (1) and (2) are calculated as Sin
Xij ⁽ⁿ⁾ and Xij will be expressed as tra and Sinter.
^{If (n + 1)} is the DCT coefficient,

[0025]

[Equation 3] The optimum prediction coefficient of the DCT coefficient between frames is

[0026]

[Equation 4]

This equation is statistically obtained when the stationary state is assumed, and Sinter (P) and Sintra are obtained.
(P) is the power spectral density in the frequency space of the intra-frame signal and the signal of the inter-frame difference, respectively.

Table 1 shows an example of the optimum prediction coefficient ρ for DCT transformation of 8 (pixels) * 8 (lines) according to this equation. Table 1 shows that the motion is 1 pixel vertically,
It is a prediction coefficient when the correlation coefficient between frames is 0.95 with two pixels in the horizontal direction.

[0029]

[Table 1]

In general, the value of the optimum prediction coefficient ρ is a function of the motion between frames and the correlation coefficient of the image signal within the frame. For this reason, it is desirable that the prediction coefficient ρ be based on an adaptive algorithm that is located near the optimum value regardless of the changing parameter. This one approach is shown in FIG. In the figure, the vertical and horizontal axes represent the vertical and horizontal movements of the pixel, respectively, and the prediction coefficient ρ in this motion vector space is represented by several non-overlapping regions,
For example, to separate into S0-S3. Each area is
Each of the prediction coefficients shown in Table 1 corresponds to S3 to S0 as the motion increases. However, the number of the areas is determined according to the application and is not limited to the number in FIG.

According to the above algorithm, the prediction coefficient used in the actual simulation (hereinafter, "prediction mask")
2) is shown in FIG. The coefficient values of the prediction masks S0 to S3 are 0, 1/4, 2 in order to facilitate the calculation process.
/ 4, 3/4 and 4/4 (the coefficient value increases as the density of the drawing increases). Each prediction mask S0-S
3 corresponds to FIG. 1, and the area of the coefficient value “0” increases as the motion increases. As can be seen from the figure, the relationship between the masks is such that each mask is included in another, that is, the prediction mask S0 is included in the prediction mask S1 and the prediction mask S1 is included in the prediction mask S2. , Which allows the optimization to be easily performed globally.

If the motion of the moving image is large, by selecting the prediction mask S0, the frequency coefficient of the high frequency component having large motion can be cut off. By doing so, the compression of the moving image can be performed by JPEG. In-frame compression will be approached. On the contrary, if the motion of the video is small,
By selecting the prediction mask S3, it is possible to pass the frequency coefficient of the low frequency component, and by doing so,
The compression of a moving image comes close to the compression by the difference between frames of MPEG.

Next, in the image processing apparatus of this embodiment,
FIG. 3 shows a configuration for performing the above coefficient compensation. In the figure, the part indicated by the dotted line shows the coefficient compensating circuit added to the part for executing the conventional JPEG.

The image processing apparatus 10 of this embodiment comprises an FDCT transform unit 12 for performing discrete cosine transform of input image information consisting of 8 (pixel) * 8 (line) blocks, and an FD.
Output of CT converter 12 and output 2 from coefficient compensation circuit 20
4 is subtracted, a quantizer 16 for quantizing the output of the subtractor 14, and an output of the quantizer 16 is MPEG.
And a bitstream unit 18 for encoding with a bitstream configuration compatible with. These configurations are
This is the same as the conventional technique that combines DCT conversion, zigzag scanning, or run length encoding in JPEG.

The image processing apparatus 10 further includes a coefficient compensating circuit 20, which dequantizes the output from the quantizing unit 16 and an output from the dequantizing unit 22. An adder 26 for adding the output 24, and an adder 26
, A decision circuit 30 for inputting the output of the frame memory 28 and the output of the FDCT conversion unit 12, and selecting an optimum prediction mask, and a decision circuit 30.
It has a multiplier 32 that multiplies the prediction mask selected by and the output of the frame memory 28 and supplies the output 24 to the subtractor 14. Further, information for identifying the prediction mask selected by the decision circuit 30, for example, 2-bit information is output to the MPEG user data register 34 if four prediction masks are selected for each block of one frame. Are encoded in an MPEG compatible bitstream and transmitted or stored with the image data signal.

Next, the operation will be described. The image information of 8 * 8 pixels is DCT-converted by the FDCT converter 12, and image data indicating 64 DCT coefficients (frequency coefficients) corresponding to the number of pixels is output. In the present embodiment, since the image signal is processed in the frequency space, that is, the coefficient space, the frame memory 28 stores the corresponding DCT coefficients of the previous frame converted by the FDCT conversion unit 12. The DCT coefficient of the currently input block and the corresponding DCT coefficient of the previous frame are input to the decision circuit 30.

The decision circuit 30 executes arithmetic processing according to the following equation to select the optimum prediction mask for the block.

[0038]

[Equation 5]

Xij is the output of the FDCT converter 12, Yij is the FDCT output of the previous frame, and Sk, i, j (k is 0, 1,
2, 3) are prediction masks.

In this way, the decision circuit 30 selects the prediction mask corresponding to the motion of the moving image, that is, the motion vector. As a result, when the motion is very large, the prediction mask S0 is selected and the motion is determined. If it is very small, the prediction mask S3 is selected. The prediction mask is then multiplied in multiplier 32 with the DCT coefficient from frame memory 28 to compensate for the frequency coefficient. In addition, 2-bit information for identifying the selected prediction mask is output to the user data register 34.

The compensated DCT coefficient is supplied to the subtractor 14 as the output 24, and the difference from the converted DCT coefficient of the input image is taken. The difference value is quantized and encoded and output from the bit stream unit 18 as an MPEG compatible bit configuration. The output of the quantization unit 16 is further dequantized and then added to the compensated DCT coefficient to be stored in the frame memory 28 as the DCT coefficient of the previous frame.

Next, another embodiment of the present invention will be described. In the above embodiment, four prediction masks S0 to S3 are selected for each block (the prediction mask is not changed for each frame), but this embodiment prepares 32 prediction masks calculated in advance. The optimum prediction mask is selected for each frame from the inside.

The 32 prediction masks prepared in advance are arranged in a linear order such that each of them is included in the other, similar to the one shown in FIG. If one frame is 72
If it consists of 0 * 480 pixels, the number of blocks is 540
Assuming that the number of prediction masks is 0, and 32 prediction masks are appropriately selected for each block, 5-bit information must be added together for each block and transmitted. For this reason, in the present embodiment, the optimum four prediction masks are selected from the 32 prediction masks for each frame, and as in the above embodiment, among the four prediction masks for each block. Select one from.

This algorithm can be implemented by the configuration shown in FIG. 4, and this may be performed by the decision circuit 30. First, we start with an arbitrary set of four prediction masks. Within the frame to be encoded,
The number of blocks selecting each prediction mask is counted, and the result is stored in the counters 40-43 as histogram information. At the end of the frame, the next set of prediction masks is determined by comparing these counting results with a predetermined threshold Ref0-Ref3.

As a result of making the thresholds Ref0-Ref3 equal (5400/4 each) and comparing with the count of the histogram, it is assumed that the selected ratio of the prediction masks S0-S3 is R0-R3. Here, the prediction mask S0
-Since the non-zero part of S3 is gradually expanded from S1 to S3 (there is a relationship in which masks are included), S1-S3 is fixed and S0 is changed (R0 is 1).
The prediction masks S1 to S3 do not change even if the next mask is selected depending on whether the direction is shifted from / 4 to + or −. Next, the prediction mask S0 is changed to S2.
Even if S3 is fixed and S1 is changed, S0, S2, S
3 does not change and only S1 can be changed. By sequentially performing this, the prediction masks S0 to S3 are changed. This is a feature that arises from the fact that the prediction masks have a relationship in which non-zero portions are sequentially included as described above.

Next, the result of the simulation according to this embodiment will be described. Here, four masks (the ones that are optimum for a certain specific sequence) are fixed from the beginning, that is, a case where the prediction mask is not changed for each frame and a case of adaptive control in which the prediction mask is changed for each frame. And compared.

FIG. 5 shows a case where four masks are mobile and calendar in a case of four fixed masks.
FIG. 6 shows how the adaptive control achieves an optimum state, showing how it was selected during a sequence called (hereinafter abbreviated as M & C). In both figures, the vertical axis represents the number of blocks when a 720 (pixel) * 480 (line) frame is divided into 8 * 8 blocks, and the horizontal axis represents the number of sequence frames.

Optimization is 25% for each of the four masks
Can be considered achieved when selected,
In the figure, the lowermost curve corresponds to the mask S0 of FIG. 2, and the uppermost curve corresponds to S3. Mask S0
Is selected in the non-predictive part of the frame (the part with large motion), and as can be seen in FIG. 5, towards the end of the sequence (where there is more motion in frames 120-140 of the sequence) To increase.

In the case where the mask is fixed, when the pixel becomes totally non-predictive, the predictive mask (filter) S0
The usage rate is about 100%, and the coefficient compensation is the conventional JP
Approaching EG processing. On the other hand, when there is no motion, the prediction efficiency increases and the prediction mask (filter) S3 is 100%.
Used, coefficient compensation approaches the motion-compensated MPEG (predicted) processing. In the case of adaptive control, the quality of the whole pixel is maintained against variations in the input image,
The usage rate of each mask is maintained near 25% so that the image quality is close to the optimum. In other words, this coefficient compensation can be regarded as being located between JPEG and motion-compensated MPEG.

FIG. 7 shows a noise simulation result when the above M & C sequence is coded. FIG. 7 shows JPEG, fixed case, adaptive case, and inter-frame JPEG, respectively. "JPEG"
Curve shows the result of intraframe compression by JPEG, and “interframe JPEG” is the GOP of 15 frames.
(Group of picture) frame D
The CT method is shown, which means that for one JPEG coded frame, the subsequent 14 frames are coded as predicted pixels. "Mask fixed case" shows the result of one fixed optimal prediction mask (which saves additional user information).
The "adaptive case" corresponds to the result of the four split prediction masks shown in FIG.

The first part of the simulation (frames 0 to 15) is the rate defined in the MPEG-CORE simulation (which was agreed on in MPEG2 by the MPEG committee of ISO, this simulation is also based on this). Buffer (rate-buf)
fer) It will be necessary to comment that it is not stable due to improper initialization of controls. The rest of the results are almost stable, and the coding mode increases as the unpredicted part of the image increases around frames 120-130 (also understood in SNR).
It is close to G.

The compressed image quality was evaluated by an internal video expert and provided a significant improvement over JPEG and interframe DCT compression. This technique can also be applied to motion compensated MPEG.

[0053]

As described in detail above, according to the present invention, the image information is processed in the frequency space instead of simply performing the inter-frame compression in the pixel space. Therefore, the appropriate compression corresponding to the motion of the moving image is performed. Is possible. Further, since the optimum prediction coefficient can be given to each frequency coefficient according to the motion of the moving image, the prediction efficiency of the moving image is also improved, and as a result, efficient data compression can be performed without degrading the moving image. .

[Brief description of drawings]

FIG. 1 is a diagram showing a separation state of prediction coefficients in a motion vector space.

FIG. 2 is a diagram showing an example of a prediction mask.

FIG. 3 is a block diagram showing a configuration for performing coefficient compensation in the image processing apparatus according to the embodiment of the present invention.

FIG. 4 is a block diagram showing a configuration for performing coefficient compensation in another embodiment.

FIG. 5 is a view showing a selection state of prediction masks fixed in a test sequence.

FIG. 6 is a diagram showing a selection state of a prediction mask in adaptive control.

FIG. 7 is a diagram showing a result of noise simulation by encoding.

FIG. 8 is a block diagram showing a conventional JPEG intra-frame compression method.

FIG. 9 is a block diagram showing a conventional compression method by DCT conversion of inter-frame difference of MPEG.

FIG. 10 is a diagram showing a distribution of frequency coefficients subjected to DCT conversion.

[Explanation of symbols]

 12 FDCT transform unit 16 Quantization unit 18 Bit stream unit 20 Coefficient compensation circuit 22 Inverse quantization unit 28 Frame memory 30 Decision circuit

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical indication H04N 7/133 Z

Claims (8)

[Claims] 1. A conversion unit for inputting image information of m pixels * n rows and converting the image information into first image data representing a frequency space, and image information of a previous frame converted by the conversion unit. Storage means for storing a second image data signal representing a frequency space corresponding to, and conversion means and storage means connected to the motion of a moving image between frames based on the first and second image data signals. Optimization means for modifying the second image data signal, and encoding for encoding a difference value between the second image data signal modified by the optimization means and the first image data signal. And an image processing apparatus. 2. The optimizing means according to claim 1, wherein the optimizing means is configured to perform the first prediction according to a prediction coefficient determined by a function of a motion between frames of the image information and a correlation coefficient of the image information in the frame. An image processing device, characterized in that the image data signal of No. 2 is modified. 3. The image processing apparatus according to claim 2, wherein the optimizing unit selects a prediction coefficient according to a motion of a moving image from a plurality of predetermined prediction coefficients. 4. The optimizing means according to claim 3, wherein when the motion of the moving image between frames is large,
An image processing apparatus, characterized in that a prediction coefficient for cutting off high frequency components of the image data signal is selected. 5. The optimizing means according to claim 3, wherein when the motion of the moving image between frames is small,
An image processing apparatus, wherein a prediction coefficient that transmits a low-frequency component of the image data signal is selected. 6. The optimizing means according to claim 3, wherein the optimizing means counts the prediction coefficient selected for each of the image information, and the prediction coefficients are averaged and selected from the counting result. An image processing apparatus, comprising: means for adjusting selection of prediction coefficients. 7. The image processing apparatus according to claim 1, wherein the conversion unit performs a two-dimensional discrete cosine conversion of the image information. 8. The input image information of m pixels * n rows is converted into first image data representing a frequency space, and the first image data and the image information of the previous frame stored in the frame memory are converted. From the second image data representing the frequency space, a prediction coefficient corresponding to the movement between the frames of the image information is selected, the second image data signal is modified according to the selected prediction coefficient, and the modified image An image processing method characterized by encoding a difference value between a data signal and the first image data signal.