JPH09200763A - Motion compensation method in video coding - Google Patents

Abstract

(57) Abstract: In a motion compensation process of moving picture coding, there is provided a method in which each block selects an optimum one from a plurality of prediction picture synthesizing methods. Four motion vectors are involved in each block, and each block selects an optimum one from a plurality of prediction image synthesizing methods using the four motion vectors. [Effect] Since the predictive image synthesizing method can be selected according to the local motion pattern of each block, the predictive characteristic can be improved.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion compensation method in moving picture coding.

[0002]

2. Description of the Related Art It is known that, in high-efficiency coding of moving images, motion compensation utilizing similarity between temporally adjacent frames has a great effect on information compression. Motion compensation systems, which are the mainstream of current image coding technology, are MPEG1 and MP2, which are international standards for moving image coding systems.
This is block matching with half-pixel accuracy that is also adopted in EG2. In this method, an image to be encoded is divided into a large number of blocks, and the motion vector of each block is calculated in the horizontal and vertical directions with a half length of the distance between adjacent pixels being the minimum unit. This processing is expressed as follows using mathematical expressions. The predicted image of the frame (current frame) to be encoded is P (x, y), and the reference image (P
R (x, y) is a decoded image of a frame that is temporally close to and has already been encoded. Also, x and y
Is an integer, and it is assumed that a pixel exists at a point whose coordinate value is an integer in P and R. At this time, the relationship between P and R is

[0003]

[Equation 1]

It is represented by However, assuming that the image is divided into n blocks, Bi represents a pixel included in the i-th block of the image, and (ui, vi) represents a motion vector of the i-th block.

In block matching with half-pixel accuracy, u
i and vi are calculated with the minimum unit of half the distance between pixels, that is, 1/2 in this case. Therefore, it is necessary to find the luminance value of a point where the coordinate value is not an integer and no pixel actually exists in the reference image (hereinafter, such a point is referred to as an interpolation point). In this case, bilinear interpolation using four peripheral pixels is often used. When this interpolation method is described by a mathematical expression, the brightness value R (x + α, at the interpolation point (x + α, y + β) of the reference image is defined with the decimal components of the coordinate value as α and β (0 ≦ α, β <1).
y + β) is

[0006]

[Equation 2]

[0007]

As mentioned above, block matching with half-pixel precision is widely used at present, but MPEG1 and M
An application that requires a higher information compression rate than PEG2 requires a more sophisticated motion compensation method. The disadvantage of block matching is that all pixels in the block must have the same motion vector. Therefore, in order to solve this problem, a motion compensation method has recently been proposed which allows adjacent pixels to have different motion vectors. The motion compensation based on the spatial transformation, which is an example of this method, will be briefly described below.

In motion compensation based on spatial transformation, the relationship between the predicted image P and the reference image R is

[0010]

(Equation 3)

It is represented by However, assuming that the image is divided into n small areas (patches), Pi represents a pixel included in the i-th patch of the image. Also, the conversion function f
i (x, y) and gi (x, y) represent the spatial correspondence between the image of the current frame and the reference image. At this time, Pi
The motion vector of the pixel (x, y) within is (x−fi (x,
y), y-gi (x, y)). by the way,
Block matching can also be interpreted as a special case of motion compensation based on spatial transformation, where the transformation function is constant. However, when the term motion compensation based on spatial transformation is used in this specification, block matching is not included therein.

The form of the conversion function is affine transformation

[0013]

(Equation 4)

Example using Nakaya (see Nakaya et al., "Fundamental Study of Motion Compensation Based on Triangular Patches", IEICE Technical Report, IE90-106, Hei 2-03), Bilinear transformation

[0015]

(Equation 5)

Example using GJ Sullivan and RL
Baker, "Motion compensation for video compression u
sing control grid interpolation ", Proc. ICASSP '91
, M9.1, pp.2713-2716, 1991-05). Here, aij and bij are motion parameters estimated for each patch. When performing actual image coding, a
Instead of directly transmitting ij and bij, the motion vector of the vertex of the patch is transmitted. For example, if an affine transformation is adopted as the transformation function and a triangular patch is used, the
The motion parameter aij can be calculated from the motion vector of each vertex. Therefore, the receiving side can configure the same conversion function as the transmitting side from the received motion vector of the vertex. On the other hand, when the bilinear transformation is used as the transformation function, the same processing can be realized by using a rectangular patch and transmitting the motion vectors of the four vertices of the patch. Hereinafter, the case where the affine transformation is used as the transformation function will be described, but this description can be applied to the case where the bilinear transformation is used as it is.

Even if the conversion function is fixed, various variations can be considered for the motion compensation based on the spatial conversion, and an example thereof is shown in FIG. In this example, the motion vector is constrained to change continuously at the boundary of the patch. In the following, it is considered that a predicted image of the original image 102 of the current frame is synthesized using the reference image 101.
For this purpose, first, the current frame is divided into a plurality of polygonal patches, and the patch-divided image 108 is obtained. The vertices of a patch are called grid points, and each grid point is shared by a plurality of patches. For example, the patch 109 has grid points 110, 1
11 and 112, and these lattice points also serve as the vertices of other patches. After dividing the image into a plurality of patches in this way, motion estimation is performed. In the example shown here, motion estimation is performed between each grid point and the reference image. As a result, each patch is transformed in the reference image 103 after the motion estimation. For example, patch 10
Reference numeral 9 corresponds to the deformed patch 104. This is because, as a result of the motion estimation, it is estimated that the lattice points 105, 106 and 107 have moved to 110, 111 and 112, respectively. The predicted image is synthesized by calculating the conversion function for each pixel in the patch and obtaining the brightness value of the corresponding point from the reference image according to equation 3. The process of applying a deforming operation to a part of one image and attaching it to another image in this way is called texture mapping or image warping. This is 3 as mentioned above
This can be realized by calculating the six motion parameters of Expression 4 from the motion vector of each vertex and calculating Expression 4 for each pixel.

The motion compensation based on the spatial transformation is characterized in that it can deal with motion patterns such as rotation and enlargement that cannot be dealt with by block matching. On the other hand, there is a problem that the calculation amount of processing is large, but as a method for simplifying this, a method disclosed in Japanese Patent Application No. 06-193971 has been devised.

Another drawback of the motion compensation system based on the spatial transformation shown in FIG. 1 is that it cannot deal with discontinuity of motion vectors. Generally, in a moving image, discontinuity of a motion vector occurs at a boundary between a moving object and a stationary object. However, in the method shown in FIG. 1, the continuity of the motion vector (the motion vector continuously changes not only in the patch but also at the boundary of the patch) is assumed, and therefore in the reproduced image after decoding. Problems such as unnatural distortion occur at the boundary between the moving object and the stationary object.

[0020]

Since a motion model based on a simple translation is used in block matching, it is not possible to sufficiently approximate the motion of an object in an image. On the other hand, since motion vector continuity is assumed in motion compensation based on spatial transformation, a problem arises that unnatural distortion occurs at the boundary between the moving object and the stationary object in the reproduced image.

[0021]

An image is divided into blocks, and a plurality of motion vectors are involved in each block. Then, these motion vectors are used so that the optimum motion compensation method can be selected from a plurality of candidates.

[0022]

BEST MODE FOR CARRYING OUT THE INVENTION In FIG. 2, an image is divided into n blocks in the horizontal direction and m blocks in the vertical direction (n and m are positive integers), and (n + 1) × is arranged at a position where the vertices of one or a plurality of blocks overlap. (M
It shows a state in which +1) grid points are arranged.
For example, 201 and 202 are blocks, 203, 204 and 2
Reference numerals 05 and 206 are grid points. Note that the grid points do not necessarily have to be present on the vertices of the block, but in the present specification, they are assumed to be present at the positions shown in FIG.

It is assumed that each grid point can have a motion vector. As a method of estimating the motion vector, a method of performing block matching with a block having a lattice point at the center can be considered. It is assumed that the predicted image in each block is composed using the motion vectors of the four lattice points located at the vertices of the block. For example, considering the block 201, grid points 203, 204, 2
The motion vectors of four grid points 05 and 206 are used. Thus, every block in the image can be made to involve four grid points.

There can be considered a plurality of methods of performing motion compensation using four lattice points. As one of them, there is a method in which one is selected from the four grid points and the motion vector of the grid point is used as the motion vector of the entire block. For example, in block 201 of FIG. 2, grid points 203, 204, 2
It is possible to select the grid point having the optimum motion vector from 05 and 206. Further, the average value of the motion vectors of the four lattice points may be used as the block motion vector. Furthermore, when the motion vector of one or two grid points is significantly different from the others, the maximum and minimum values are excluded from the horizontal and vertical components of the four motion vectors, and then the average is calculated. You can also think of a way to ask. The method of synthesizing this predicted image is the same as that of block matching, and good prediction characteristics can be obtained by simple calculation.
However, sufficient characteristics cannot be obtained when there are a plurality of objects that rotate, scale up or down, or move in different directions within the same block.

FIG. 3 shows an example in which an image is divided into a plurality of right triangles. In the example of block 301, grid points 303, 3
Triangle formed by 05 and 304 and grid point 30
It is divided into a triangle formed by 4, 305 and 306. In the example of block 302, grid points 307, 31
A triangle formed by 0 and 308 and a grid point 307,
It is divided into a triangle formed by 309 and 310. As a method of synthesizing a predicted image in the case of performing such division, first, a method of assigning one motion vector to each part from the four motion vectors involved in a block is conceivable. For example, block 301
In the example of, the motion vector of the grid point 303 is set for the triangle formed by the grid points 303, 305, 304,
The motion vector of the grid point 306 can be assigned to the triangle formed by the grid points 304, 305, and 306. This method is slightly more complicated than the above-described method of allocating one motion vector to the entire block, but the range of motion that can be handled is widened accordingly. For example, when the boundary between the moving object and the stationary object is located on the diagonal line connecting the grid points 304 and 305 in the block division of 301, it can be expected that a high prediction characteristic is exhibited. If one of the examples shown in block 301 and block 302 can be selected as the block division method, the range of motion patterns that can be handled can be further expanded. Obviously, there are other means than the method shown in FIG. 3 as a method of dividing the block into two parts. In the division other than the method of FIG. 3, the same processing can be realized by allocating one of the four motion vectors to each part.

As another predictive image combining method using the division of FIG. 3, a method using the affine transformation shown in Equation 4 can be considered. For example, in the example of the block 301, by using the motion vectors of the grid points 303, 305, 304, it is possible to synthesize a predicted image based on the affine transformation on the triangle formed by these grid points. In addition, the same processing is performed on the grid points 305,
It can also be applied to the triangle formed by 306 and 304. Here, the technique common to the motion compensation based on the spatial transformation described in “Prior Art” is used. This method is more complicated in processing than the method described above, but has an important feature that it can cope with rotation and enlargement / reduction of an object. However, since the motion vector is continuous within a block, good characteristics cannot be obtained when there are multiple objects that move differently within the block. Also, in this method as well, by making it possible to select two methods for block division,
The prediction characteristics can be further improved.

As a method of synthesizing a predicted image that is very similar to the example using the affine transformation, there is a method using the co-linear transformation shown in the equation (5). For example, in the case of block 201 in FIG. 2, 8 is represented by bij in Equation 5 by using four motion vectors of grid points 203, 204, 205, 206.
Individual parameters can be determined. In this way, the prediction image may be combined with the pixels in the patch using a technique common to the motion compensation based on the spatial conversion. The method using the co-linear transformation exhibits characteristics very similar to the method using the affine transformation, but the processing is a little complicated. However, depending on the motion pattern, it may show better characteristics than when the affine transformation is used, so it is convenient to include it in the options of the method of synthesizing the predicted image.

FIG. 4 is an example showing a method of synthesizing a predicted image in which a block is divided into four parts. Grid points 402, 4
A rectangular block 401 constituted by 03, 404, and 405 has four small blocks 406, 407, and 40.
It is divided into 8 and 409. Then, for example, the small block 4
In 06, the motion vector of the grid point 402 and the small block 40
7 is a motion vector of the grid point 404, a small block 408.
The block matching by four small blocks can be realized by allocating the motion vector of the grid point 403 to the sub-block and the motion vector of the grid point 405 to the small block 409. Since this method divides the block into four parts, it can be applied to the case where there are more objects that move differently as compared with the method described above. However, it cannot sufficiently cope with rotation and enlargement / reduction of an object. Obviously, there are other means than the method shown in FIG. 4 as a method of dividing the block into four parts. In the division other than the method of FIG. 4, the same processing can be realized by allocating one of the four motion vectors to each part.

As described above, each of the prediction image synthesizing methods in the block described here has its own advantages and disadvantages. A moving image is generally characterized in that the movement pattern greatly differs depending on the area within the image. For example, the motion pattern is completely different in the stationary area, the parallel moving area, the rotating area, and the boundary between the moving area and the stationary area. Therefore, even if the image is divided into blocks so that local correspondence can be made for each area, sufficient prediction characteristics can be obtained if there is only one method of combining predicted images (= conventional block matching, etc.). Can not. Therefore, it becomes possible to improve the prediction characteristic by making it possible to select the most suitable method for each block from among the above-described many methods of combining predicted images. If only the error of the prediction image is considered, it is advantageous to have a larger number of methods of synthesizing prediction images having different characteristics in order to improve the characteristics.

It is not always necessary to select only one predictive image composition method for each block. It is also possible to further improve the prediction characteristic by selecting two combining methods (for example, affine transformation and block 4-division) and taking the average value of the luminance values between the two as the predicted image in the block. It is possible. It is known that, by averaging different prediction images, the effects of noise removal and a low-pass filter are generally obtained, and the prediction characteristics are improved. MPE
In the standard moving image coding methods such as G1 and MPEG2, it is possible to perform an operation of averaging images predicted from two directions (forward and backward) (= bidirectional prediction). This is because. It is obvious that the invention described in this specification can be applied to this bidirectional prediction.

[0031]

According to the present invention, it is possible to improve the prediction characteristic in the motion compensation process of moving image coding.

[Brief description of drawings]

FIG. 1 is a diagram showing an example of motion compensation processing based on space conversion.

FIG. 2 is a diagram showing an example of arrangement of blocks and grid points.

FIG. 3 is a diagram showing an example of a method of performing motion compensation by dividing a block into two right triangles.

FIG. 4 is a diagram showing an example of a method of dividing a block into four small blocks to perform motion compensation.

[Explanation of symbols]

101: Reference image, 102: Original image of the current frame, 10
3. Reference image patches and grid points after motion estimation, 104,
109 ... patch, 105-107, 110-112, 2
03-206, 303-310, 402-405 ... Lattice points, 201, 202, 301, 302, 401 ... Blocks, 406-409 ... Small blocks.

Claims (22)

[Claims] 1. In moving image encoding, when motion compensation is performed by dividing an image into n (horizontal) and (m) vertical (m and n are positive integers) square or rectangular blocks, motion compensation is performed on all blocks. A motion characterized in that each of the four motion vectors is involved, and that each block can be selected from two or more candidates by a method of synthesizing a motion-compensated prediction image in the block using the four motion vectors. Compensation method. 2. The motion compensation method according to claim 1, wherein a method of synthesizing a motion compensation prediction image in a block can be selected from three or more candidate methods. 3. The motion compensation method according to claim 1, wherein a method of synthesizing a motion compensation prediction image in a block can be selected from four or more candidate methods. 4. The motion compensation method according to claim 1, wherein a method of synthesizing a motion compensation prediction image in a block can be selected from five or more candidate methods. 5. The motion compensation method according to claim 1, wherein a method of synthesizing a motion compensation prediction image in a block can be selected from eight or more candidate methods. 6. One of four motion vectors involved in a block is used as one of candidates for a method of synthesizing a motion-compensated prediction image in the block, and all pixels in the block have the motion vector. 6. The motion compensation method according to claim 1, further comprising: 7. A method of synthesizing a motion-compensated prediction image within a block includes four options of a method of using one of the motion vectors involved in the block as the motion vector of all the pixels within the block. The motion compensation method according to claim 3, wherein all of them are included. 8. As one of candidates for a method of synthesizing a motion-compensated predicted image in a block, an average value of four motion vectors involved in the block is used so that all pixels in the block follow the motion vector. 6. The motion compensation method according to claim 1, further comprising: 9. As one of the candidates for a method of synthesizing a motion-compensated prediction image in a block, the maximum and minimum of four horizontal components and four vertical components of four motion vectors involved in the block are selected. 6. The method according to claim 1, further comprising a method of obtaining a motion vector constituted by an average value of two components excluding the one and making all pixels in the block follow the motion vector. Motion compensation method. 10. A block is divided into four parts as one of candidates for a method of synthesizing a motion-compensated prediction image within a block, and one of four motion vectors associated with the block is associated with each of the parts. 6. The method according to claim 1, further comprising a method for causing pixels in each part to follow a corresponding motion vector. 11. A block is divided into two parts as one of candidates for a method of synthesizing a motion-compensated prediction image in a block, and one of four motion vectors involved in the block is associated with each of the parts. 6. The method according to claim 1, further comprising a method for causing pixels in each part to follow a corresponding motion vector. 12. A block is divided into four parts as one of candidates for a method of synthesizing a motion-compensated prediction image within a block, and one of the four motion vectors associated with the block is associated with each of the parts. 6. The method according to claim 1, further comprising a method for causing pixels in each part to follow a corresponding motion vector. 13. A method of synthesizing a motion-compensated predicted image in a block, wherein two of four motion vectors involved in the block are selected and each motion vector is divided by a diagonal line of the block. Motion compensation method according to any one of claims 1 to 5, characterized in that it comprises a method of associating two right-angled triangles with all pixels in each right-angled triangle according to the corresponding motion vector. . 14. As one of candidates for a method of synthesizing a motion-compensated prediction image within a block, the block is divided into two right-angled triangles divided by their diagonals, and each part is involved in the block. 6. A method for selecting three from four motion vectors and providing a motion vector within a right triangle by an affine transformation determined by the three motion vectors. The motion compensation method described in. 15. The motion compensation method according to claim 13, wherein both of the two methods of dividing a block by a diagonal line are provided as candidates for a method of synthesizing a motion compensation prediction image in a block. 16. A quadrature 3 by a co-linear transformation determined by four motion vectors involved in a block, as one of candidates for a method of synthesizing a motion-compensated prediction image in a block.
The motion compensation method according to claim 1, further comprising a method of providing a motion vector within a polygon. 17. A method of using an average of two predicted images in a block generated by two different combining methods as a predicted image in the block, as one of candidates for a method of combining motion-compensated predicted images in a block. The motion compensation method according to claim 1, further comprising: 18. A predictive image within a block generated by a method using one of the four motion vectors involved in the block to ensure that all pixels within the block follow that motion vector, and the block. A block generated by a method in which each pixel is divided into four parts, each of which is associated with one of the four motion vectors involved in the block, so that the pixels in each part follow the corresponding motion vector. 6. The motion compensation method according to claim 1, further comprising a method of determining an average of the predicted image in the block and the predicted image in the block as the predicted image in the block. 19. A prediction image in a block generated by a method of making all pixels in a block follow the motion vector by using an average value of four motion vectors involved in the block, and the prediction image in the block. Within a block generated by a method in which each of the four motion vectors involved in the block is associated with one of the four parts, and the pixel in each part follows the corresponding motion vector. 6. The motion compensation method according to claim 1, further comprising a method of determining an average of the predicted image and a predicted image in a block. 20. A motion vector constituted by an average value of two components excluding a maximum one and a minimum one of four horizontal and vertical components of four motion vectors involved in a block. And a prediction image in the block generated by a method in which all the pixels in the block follow the motion vector, and the block is divided into four parts, and four motion vectors related to the block A prediction image in a block generated by a method in which each of the pixels in each part follows a corresponding motion vector. The motion compensation method according to any one of claims 1 to 5, which is characterized. 21. A storage medium in which coded information generated by a moving picture coding system including the motion compensation method according to any one of claims 1 to 20 is recorded. 22. An image decoding apparatus for decoding coded information generated by a moving picture coding method including the motion compensation method according to any one of claims 1 to 20.