CN118075475A

CN118075475A - A video encoding and decoding method and related equipment

Info

Publication number: CN118075475A
Application number: CN202211484382.0A
Authority: CN
Inventors: 吕梁; 陈烛稼; 马霖峰
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-11-24
Filing date: 2022-11-24
Publication date: 2024-05-24

Abstract

The application provides a video coding and decoding method and related equipment, wherein the method can comprise the following steps: dividing an image frame in an image sequence of an original video to obtain a region to be processed, wherein the region to be processed comprises one or more image blocks; determining a reference area of a target image block in a reference frame, wherein the reference frame is a frame which needs to be referred when the image frame is coded, the target image block is one of one or more image blocks, and the area occupied by the reference area is in the area occupied by the area to be processed, which the target image block belongs to; the target image block is encoded according to the reference region. The application can avoid the need of carrying out full-quantity encoding and decoding on the reference frame during encoding and decoding, and reduce the calculation cost.

Description

Video encoding and decoding method and related equipment

Technical Field

The present application relates to the field of video encoding and decoding, and in particular, to a video encoding and decoding method and related devices.

Background

With the development of video encoding and decoding technology, services such as video color ring, video advertisement and the like are widely applied. The above-mentioned services put higher demands on the speed of encoding and decoding, and for this reason, the video codec standard h.264 proposes a parallel encoding and decoding method to accelerate the encoding and decoding of video. The video codec standard h.264 may divide each frame of image into one or more slices with finer granularity. Each slice is self-contained, meaning that the slice contains all the syntax structures required by the decoding end, and when video frames are decoded, the decoding is independent in units of slices. But this independence refers to slice independence in the spatial domain, and still depends on multiple slices in the reference frame during the encoding and decoding of the current frame.

In some application scenarios (such as video color ring superimposed advertising), the portion to be superimposed in the original video is only a small portion of the image frame. However, due to the dependency limit of the slice codec in the prior art on the reference frame, encoding and decoding of the full image of the reference frame are required, which results in a great computational overhead.

Disclosure of Invention

The embodiment of the application provides a video coding and decoding method and related equipment, which are used for obtaining a region to be processed by dividing an image frame of a video, wherein the coding of image blocks in a single region to be processed depends on a reference region at the same position in a reference frame, so that full coding and decoding are not needed during coding and decoding, and the calculation cost can be reduced.

In a first aspect, an embodiment of the present application provides a video encoding method, which may include: dividing an image frame in an image sequence of an original video to obtain a region to be processed, wherein the region to be processed comprises one or more image blocks;

Determining a reference area of a target image block in a reference frame, wherein the reference frame is a frame which needs to be referred when the image frame is coded, the target image block is one of the one or more image blocks, and the area occupied by the reference area is in the area occupied by the area to be processed to which the target image block belongs;

and encoding the target image block according to the reference area.

According to the method, the reference area of the target image block (the current coding block) in the reference frame can be determined, and the area occupied by the reference area is in the area occupied by the to-be-processed area to which the target image block belongs, so that the coding of the target image block in the to-be-processed area only depends on the area at the same position in the reference frame, full coding is not needed, and the calculation cost can be reduced.

With reference to the first aspect, in a possible implementation manner of the first aspect, the determining a reference area of the target image block in the reference frame includes:

Calculating the area occupied by the area to be treated;

Under the condition that the area indicated by the motion vector of the target image block is not in the area occupied by the area to be processed, carrying out constraint processing on the target image block to obtain a constraint motion vector of the target image block; the region indicated by the constraint motion vector is in a region occupied by a region to be processed to which the target image block belongs;

and determining a reference area of the target image block from the reference frame according to the constraint motion vector.

It can be seen that, when the region occupied by the reference region is not within the region occupied by the region to be processed, motion estimation of the target image block under constraint is required, so that the region occupied by the reference region is within the region occupied by the region to be processed, and it is ensured that only the region at the same position in the reference frame can be referred to when encoding the image block within the region to be processed.

With reference to the first aspect, in a possible implementation manner of the first aspect, the performing constraint processing on the target image block to obtain a constrained motion vector of the target image block includes:

respectively calculating pixel residual errors between the region indicated by the motion vector of the adjacent block and the target image block according to the motion vector of the adjacent block of the target image block;

And if the motion vector with the pixel residual error of 0 exists, if the region indicated by the motion vector of the adjacent block corresponding to the pixel residual error of 0 is not in the region occupied by the region to be processed, carrying out constraint processing on the target image block to obtain the motion vector of the target image block.

It will be appreciated that if the pixel residual is 0, the target image block is a skip macroblock. Therefore, in the case where the region indicated by the motion vector is not a region occupied by the region to be processed, the skip macroblock can be subjected to constraint processing as a normal image block.

With reference to the first aspect, in a possible implementation manner of the first aspect, the constraint processing includes: integer pixel motion estimation and sub-pixel motion estimation.

With reference to the first aspect, in a possible implementation manner of the first aspect, the performing constraint processing on the target image block to obtain a motion vector of the target image block includes:

Performing the whole pixel motion estimation on the target image block according to the motion vector corresponding to the adjacent block of the target image block to obtain a whole pixel motion searching end point;

and carrying out sub-pixel motion estimation on the target image block according to the whole-pixel motion searching end point to obtain a motion vector of the target image block.

It can be understood that the region indicated by the motion vector of the target image block is within the region occupied by the region to be processed to which the target image block belongs, and the reference region is ensured to be within the region occupied by the region to be processed to which the target image block belongs by whole-pixel motion estimation.

With reference to the first aspect, in a possible implementation manner of the first aspect, the performing, according to a motion vector corresponding to a neighboring block of the target image block, the whole pixel motion estimation on the target image block to obtain a whole pixel motion search endpoint includes:

if the region indicated by the motion vector of the adjacent block of the target image block exists in the occupied region of the region to be processed, calculating a first code rate distortion cost of the motion vector of the adjacent block;

And determining a whole pixel motion searching end point according to a motion vector corresponding to the minimum first code rate distortion cost in the first code rate distortion costs.

It can be seen that the whole pixel motion search starting point is the one of the motion vectors with the smallest coding cost, so that the reference region with the smallest coding cost can be determined.

With reference to the first aspect, in a possible implementation manner of the first aspect, the performing the whole pixel motion estimation on the target image block according to a motion vector corresponding to a neighboring block of the target image block to obtain a whole pixel motion search endpoint includes:

If the region indicated by the motion vector of the adjacent block of the target image block is not in the occupied region of the region to be processed, scaling the motion vector corresponding to the adjacent block of the target image block in equal proportion according to the boundary information of the region to be processed and the position information of the target image block to obtain a scaled motion vector, wherein the region indicated by the scaled motion vector is in the occupied region of the region to be processed;

Calculating a second code rate distortion cost of the scaled motion vector;

And determining a whole pixel motion searching end point according to a motion vector corresponding to the minimum second code rate distortion cost in the second code rate distortion costs.

It can be seen that the area indicated by the scaled motion vector is within the area occupied by the area to be processed, and the determined whole pixel motion search end point is the one with the minimum coding cost, so that the reference area of the target image block is ensured to be within the area occupied by the area to be processed.

With reference to the first aspect, in a possible implementation manner of the first aspect, the performing the sub-pixel motion estimation on the target image block according to the whole pixel motion search endpoint to obtain a motion vector of the target image block includes:

calculating a sub-pixel reference area according to the sub-pixel interpolation calculation relation;

And if the region indicated by the motion vector corresponding to the whole pixel motion searching end point is not in the region occupied by the sub-pixel reference region, taking the motion vector corresponding to the whole pixel motion searching end point as the motion vector of the target image block.

It can be seen that the sub-pixel reference area is a reference area with higher precision, and when the reference area is not located in the area occupied by the sub-pixel reference area, the sub-pixel interpolation dependency is indicated, so that the whole-pixel motion vector can be saved.

And if the region indicated by the motion vector corresponding to the whole pixel motion searching end point is in the region occupied by the sub-pixel reference region, sub-pixel motion estimation is carried out on the target image block, and the motion vector of the target image block is obtained.

It can be seen that the sub-pixel reference area is a reference area with higher precision, and when the reference area is in the area occupied by the sub-pixel reference area, it is indicated that there is no sub-pixel interpolation dependence, and sub-pixel estimation is required to ensure that the reference area is in the area to be processed.

With reference to the first aspect, in a possible implementation manner of the first aspect, the performing sub-pixel motion estimation on the target image block to obtain a motion vector of the target image block includes:

Performing sub-pixel motion estimation on the target image block according to the whole-pixel motion searching end point, and determining a sub-pixel motion vector, wherein the area pointed by the sub-pixel motion vector is in the area occupied by the sub-pixel reference area;

Calculating a third code rate distortion cost of the sub-pixel motion vector;

and taking the sub-pixel motion vector corresponding to the minimum third code rate distortion cost in the third code rate distortion cost as the motion vector of the target image block.

It can be understood that the region indicated by the motion vector of the target image block is within the region occupied by the region to be processed to which the target image block belongs, and the reference region with higher precision can be obtained through sub-pixel motion estimation, so that the reference region is ensured to be within the region occupied by the region to be processed to which the target image block belongs, and the sub-pixel cross-region dependency risk does not exist.

With reference to the first aspect, in a possible implementation manner of the first aspect, the calculating an area occupied by the area to be processed includes:

calculating the sequence number of a first image block and the sequence number of a last image block in the region to be processed;

Calculating the area information occupied by the area to be processed, wherein the area information comprises the number of lines of the area to be processed, and whether the first line and the last line of the area to be processed are full or not;

And calculating the area occupied by the area to be processed according to the serial number of the first image block, the serial number of the last image block and the area information.

In a second aspect, an embodiment of the present application provides a video decoding method, which may include: analyzing a code stream to obtain a reference area of a target image block in a reference frame, wherein the target image block is any one of one or more image blocks contained in an area to be processed, the area to be processed is obtained by dividing any image frame in an image sequence of an original video, the area to be processed is an area with superimposed information, the reference frame is a frame needing to be referred when the image frame is encoded, and the area occupied by the reference area is in the area occupied by the area to be processed of the target image block;

and predicting the target image block according to the reference area.

According to the method, the reference area of the target image block (the current decoding block) in the reference frame can be determined, and the area occupied by the reference area is within the area occupied by the to-be-processed area to which the target image block belongs, so that the decoding of the target image block in the to-be-processed area only depends on the area at the same position in the reference frame, full decoding is not needed, and the calculation cost can be reduced.

With reference to the second aspect, in a possible implementation manner of the second aspect, after the decoding the target image block according to the reference area, the method further includes:

and copying the code stream in the area which is not subjected to decoding processing, wherein the area which is not subjected to decoding processing is an area except for the area occupied by the area to be processed in the image frame.

In a third aspect, an embodiment of the present application provides an encoder including a partition unit, a prediction unit, and an entropy encoding unit, wherein:

the segmentation unit is used for carrying out segmentation processing on image frames in an image sequence of an original video to obtain a region to be processed, wherein the region to be processed comprises one or more image blocks;

The prediction unit is further configured to determine a reference area of a target image block in a reference frame, where the reference frame is a frame that needs to be referred to when the image frame is encoded, the target image block is one of the one or more image blocks, and an area occupied by the reference area is in an area occupied by an area to be processed to which the target image block belongs;

the entropy coding unit is used for coding the target image block according to the reference area.

With reference to the third aspect, in a possible implementation manner of the third aspect, the prediction unit is specifically configured to:

Calculating the area occupied by the area to be treated;

With reference to the third aspect, in a possible implementation manner of the third aspect, the constraint processing includes: integer pixel motion estimation and sub-pixel motion estimation.

With reference to the third aspect, in a possible implementation manner of the third aspect, the prediction unit is specifically configured to: performing the whole pixel motion estimation on the target image block according to the motion vector corresponding to the adjacent block of the target image block as a starting point to obtain a whole pixel motion searching end point;

With reference to the third aspect, in a possible implementation manner of the third aspect, the prediction unit is specifically configured to: if the region indicated by the motion vector of the adjacent block of the target image block exists in the occupied region of the region to be processed, calculating a first code rate distortion cost of the motion vector of the adjacent block;

With reference to the third aspect, in a possible implementation manner of the third aspect, the prediction unit is specifically configured to: if the region indicated by the motion vector of the adjacent block of the target image block is not in the occupied region of the region to be processed, scaling the motion vector corresponding to the adjacent block of the target image block in equal proportion according to the boundary information of the region to be processed and the position information of the target image block to obtain a scaled motion vector, wherein the region indicated by the scaled motion vector is in the occupied region of the region to be processed;

Calculating a second code rate distortion cost of the scaled motion vector;

With reference to the third aspect, in a possible implementation manner of the third aspect, the prediction unit is specifically configured to: calculating a sub-pixel reference area according to the sub-pixel interpolation calculation relation;

With reference to the third aspect, in a possible implementation manner of the third aspect, the prediction unit is specifically configured to: and if the region indicated by the motion vector corresponding to the whole pixel motion searching end point is in the region occupied by the sub-pixel reference region, sub-pixel motion estimation is carried out on the target image block, and the motion vector of the target image block is obtained.

Calculating a third code rate distortion cost of the sub-pixel motion vector;

With reference to the third aspect, in a possible implementation manner of the third aspect, the splitting unit is specifically configured to: calculating the sequence number of a first image block and the sequence number of a last image block in the region to be processed;

In a fourth aspect, embodiments of the present application provide a decoder comprising an entropy decoding unit and an inter prediction unit, wherein:

The entropy decoding unit is used for analyzing the code stream to obtain a reference area of a target image block in a reference frame, wherein the target image block is any one of one or more image blocks contained in an area to be processed, the area to be processed is obtained by dividing any image frame in an image sequence of an original video, the area to be processed is an area with superimposed information, the reference frame is a frame needing to be referred when the image frame is encoded, and the area occupied by the reference area is in the area occupied by the area to be processed of the target image block;

And the inter-frame prediction unit is used for predicting the target image block according to the reference area.

With reference to the fourth aspect, in a possible implementation manner of the fourth aspect, the inter prediction unit is further configured to:

In a fifth aspect, embodiments of the present application provide an encoder comprising a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the encoder to implement the method described in any of the preceding first aspects.

Optionally, the encoder further comprises a communication interface for receiving and/or transmitting data and/or for providing input and/or output to the processor.

The above embodiment is described taking a processor (or general-purpose processor) for executing a method by calling a computer specification as an example. In particular implementations, the processor may also be a special purpose processor in which the computer instructions are already preloaded in the processor. In the alternative, the processor may include both a special purpose processor and a general purpose processor.

In the alternative, the processor and memory may be integrated in one device, i.e., the processor and memory may be integrated.

In a fifth aspect, an embodiment of the present application provides a decoder, the encoder comprising a processor and a memory, the processor being configured to execute instructions stored in the memory, to cause the decoder to implement the method described in any of the preceding second aspects.

Optionally, the decoder further comprises a communication interface for receiving and/or transmitting data and/or for providing input and/or output to the processor.

In a sixth aspect, embodiments of the present application also provide a computer-readable storage medium having instructions stored therein that, when executed on at least one processor, implement the method described in any of the first aspects or the method described in any of the second aspects.

In a seventh aspect, the present application provides a computer program product comprising computer instructions which, when run on at least one processor, implement a method as described in any of the preceding first aspects.

Alternatively, the computer program product may be a software installation package or a mirror package, which may be downloaded and executed on a computing device in case the aforementioned method is required.

The advantages of the technical solutions provided in the second to seventh aspects of the present application may refer to the advantages of the technical solutions in the first aspect, and are not described herein.

Drawings

The drawings to which the present application is applied are described below.

Fig. 1A is a schematic diagram of a single frame according to slice region division provided in the present application;

FIG. 1B is a schematic diagram of spatial domain independence of slice according to the present application;

FIG. 2A is a schematic diagram of a multi-threaded encoding provided by the present application;

FIG. 2B is a schematic diagram of slice dependent reference frames during encoding according to an embodiment of the present application;

FIG. 2C is a schematic diagram of a multi-threaded decode provided by an embodiment of the present application;

FIG. 2D is a schematic diagram of slice dependent on a reference frame when decoding a frame according to an embodiment of the present application;

FIG. 3A is a schematic diagram of superimposing content on a video according to an embodiment of the present application;

FIG. 3B is a schematic diagram of a decoding error according to an embodiment of the present application;

FIG. 3C is a schematic diagram of a single slice decoding according to the present application for slices co-located only with reference frames;

FIG. 4A is a schematic diagram of a video system according to an embodiment of the present application;

FIG. 4B is a schematic diagram of an encoder according to an embodiment of the present application;

FIG. 4C is a schematic diagram of a decoder according to an embodiment of the present application;

Fig. 5 is a schematic view of a scene of locally overlapping information on video according to an embodiment of the present application;

fig. 6 is a schematic flow chart of a video encoding method according to an embodiment of the present application;

FIG. 7A is a schematic diagram of a different type of image block provided by an embodiment of the present application;

FIG. 7B is a schematic diagram of block-based motion estimation provided by an embodiment of the present application;

FIG. 7C is a schematic diagram of determining a matching block for an image block according to an embodiment of the present application;

FIG. 8A is a flow chart of calculating the range of a region to be processed according to an embodiment of the present application;

FIG. 8B is a schematic diagram of an image frame provided by an embodiment of the present application;

FIG. 8C is a schematic diagram of coordinates of an image block according to an embodiment of the present application;

FIG. 9A is a schematic diagram of a sub-pixel interpolation provided by an embodiment of the present application;

FIG. 9B is a schematic flow chart of constraint processing for a target image block according to an embodiment of the present application;

FIG. 10A is a schematic diagram of adjacent blocks of different sizes provided by an embodiment of the present application;

FIG. 10B is a schematic diagram of scaling a motion vector according to an embodiment of the present application;

FIG. 11A is a schematic diagram of a sub-pixel motion estimation process according to an embodiment of the present application;

FIG. 11B is a schematic diagram of a sub-pixel reference area according to an embodiment of the present application;

fig. 12 is a schematic flow chart of video decoding according to an embodiment of the present application;

Fig. 13 is a schematic view of a scene of superimposing information on a video according to an embodiment of the present application;

fig. 14 is a schematic structural diagram of a decoding device according to an embodiment of the present application;

Fig. 15 is a schematic structural diagram of a video decoding system according to an embodiment of the present application.

Detailed Description

Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The technical scheme related to the embodiment of the application not only can be applied to the existing video coding and decoding standards (such as H.264/advanced video coding (advanced video coding, AVC), H.265/high-efficiency video coding (HIGH EFFICIENCY video coding, HEVC), H.266 and other standards) but also can be applied to future video coding and decoding standards.

For ease of understanding, the following description of probabilities associated in part with embodiments of the present application is given by way of example for reference. The following is shown:

1. Video coding and decoding

Video codec generally refers to processing a video or sequence of images that form a video.

Video encoding (video encoding) refers to a process of compressing a video file into a code stream by a compression technique.

Video decoding (video decoding), which refers to a process of restoring a code stream into a reconstructed image according to a specific syntax rule and processing method.

In the field of video codec, the terms "picture", "frame" or "image" may be used as synonyms. The combination of the encoding portion and the decoding portion may also be referred to as encoding and decoding (encoding and decoding).

2. Image block (block)

In most coding frameworks, the video consists of a series of pictures, one picture being called a frame. The image is divided into at least one slice, each slice in turn being divided into image blocks (blocks). Video encoding or video decoding is performed in units of image blocks, and for example, encoding processing or decoding processing may be performed from left to right, from top to bottom, line by line, starting from the upper left corner position of an image.

The image block may be a Macroblock (MB) in the video codec standard h.264/AVC, a sub-block in the video codec standard h.264/AVC, or a Coding Unit (CU) in the High Efficiency Video Coding (HEVC) standard. The embodiment of the present application is not particularly limited thereto.

In the embodiment of the present application, an image block being subjected to encoding processing or decoding processing is referred to as a current image block (current block), and an image in which the current image block is located is referred to as a current frame (current image).

3. Strip/slice (slice)

One slice contains part or all of the data of a frame, in other words, a frame can be encoded into several slices. A structure containing multiple slices may be referred to as a slice group (slice group). In the H.264/AVC standard, a slice contains at least one macroblock and at most data of an entire frame of image. In different coding implementations, the number of slices made in the same frame is not necessarily the same.

In the H.264/AVC standard, the objective of slice design is primarily to prevent the spreading of bit errors. Because the decoding operations are independent from slice to slice, the data (e.g., predictive coding) referenced by a slice decoding process cannot exceed the boundaries of a slice.

4. Redundant slice

A redundant data volume for error repair.

5. Bi-directional predictive interpolation coding frame (bi-directional interpolated prediction frame)

Also known as bi-predictive coded frames, also known as B frames, exploit temporal redundancy information between the coded frames preceding and following the source image sequence to compress the amount of coded frame types. B frame records the difference between the frame and the front and back frames, and the decoding needs not only the previous buffer picture but also the picture after decoding, and the final picture is generated by the superposition of the front and back pictures and the frame data.

6. Forward predictive coding frame (PREDICTIVE FRAME)

Also known as inter-prediction encoded frames, also known as P frames, utilize redundant information in front of the source image sequence to compress the encoded frame type of the amount of transmitted data. The P-frame represents the difference between the current frame and the previous frame, and the difference defined by the current frame needs to be overlapped by the previously buffered picture during decoding to generate the final picture.

7. Sequence parameter set (SPS PARAMETER SET)

SPS is used to describe parameters of a continuously encoded video sequence in which a set of global parameters (including coding level, resolution, etc.) of the encoded video sequence are maintained. A so-called coded video sequence is a sequence of structures after pixel data of an original video is coded frame by frame.

8. Image parameter set (picture PARAMETER SET, PPS)

PPS is used to describe the parameters of a picture or pictures in a sequence, i.e. the parameters on which the encoded data of each frame depends, are stored in a picture parameter set.

A parameter set is an independent unit of data that is independent of other syntax elements outside the parameter set. One parameter set does not correspond to a particular image or sequence, the same sequence parameter set may be referenced by multiple image parameter sets, and similarly, the same image parameter set may be referenced by multiple images.

9. Reference frame (REFERENCE FRAME)

In video coding, in order to achieve compression, a new frame is formed by combining frames and motion vectors by buffering parts of the picture, these frames are called reference frames, which are frames to be referred to in decoding other frames,

10. Motion vector (motion vector)

The motion vector is an important parameter in the inter prediction process, and refers to the offset between the current encoded block and the reference region (matching block).

11. Motion estimation (motion estimation)

Motion estimation is a method of finding matching blocks of a coded block from a reference frame, the basic idea being to divide each frame of an image sequence into a number of macro blocks that do not overlap each other and consider that the displacement amounts of pixels within the macro blocks are the same, and then find the block most similar to the current block, i.e. the matching block, according to a certain matching criterion for each macro block within a given specific search range from the reference frame. The relative displacement of the matching block and the current block is the non-motion vector.

When video compression is carried out, the current block can be completely recovered by only storing the motion vector and residual data.

12. Motion compensation (motion compensation)

Motion compensation is a method of synthesizing a transform of a next frame using motion vectors of a current frame, a method of describing differences between adjacent frames (adjacent here means adjacent in coding terms, two frames are not necessarily adjacent in play order), specifically describing how each tile of a previous frame moves to a certain position in the current frame.

This approach is often used by video compression/video codecs to reduce spatial redundancy in video sequences. It may also be used to perform operations of de-interleaving (DEINTERLACING) and motion interpolation (motion interpolation).

13. Slice context sequence (slice context list): for receiving a list holding all slice data in a frame.

The above description of the probabilities may be used in the following embodiments.

In order to increase the speed of video encoding and decoding, the video encoding and decoding standard H.264/AVC provides a slice segmentation method, and each frame of image is divided according to a slice region. Referring to fig. 1A, fig. 1A is a schematic diagram of a single frame according to slice region division according to an embodiment of the present application. As can be seen in fig. 1A, an image may be partitioned into one or more slices, such as the single frame 100 shown in fig. 1A partitioned into slice0, slice1, slice2, slice3, and slice4. Each slice can be decoded independently of other slices in the same image. Meaning that intra-prediction and inter-prediction cannot cross slice boundaries.

Advantages of employing slice segmentation include one or more of the following:

first, the robustness is good, each image is divided into independent slices, errors are encountered during decoding, resynchronization can be achieved, and error code diffusion and transmission are limited.

Second, matching the maximum transmission unit (maximum transmission unit, MTU) size involves the concept of a network layer, where the size of each packet is limited when it is sent over the network. Thus, slice structure coding improves transmission efficiency and fault tolerance by providing decodable blocks available for transmission and supporting Arbitrary Slice Order (ASO), flexible Macroblock Order (FMO), and redundant slice methods. By keeping the number of bytes per slice approximately constant, smaller network packets and fixed-size network packets can be achieved.

Third, parallel processing, because each slice is independent, the encoding and decoding and loop filtering can be processed in parallel, so that the processing speed is increased and the processing efficiency is improved.

As can be seen from fig. 1A, h.264/AVC divides a picture into one or several slices with finer granularity. Each slice is self-contained, meaning that the slice contains all the syntax structures required by the decoding end, and is generally composed of two parts, one part being the overall information used to save the slice (e.g., the type of current slice, etc.), and the other part typically being a set of consecutive macroblock structures (or macroblock skip information). When decoding video, each image in the video is independently decoded in units of slices, and the independence refers to slice independence on a spatial domain, and all slices of a previous frame still need to be referred to in decoding.

Referring to fig. 1B, fig. 1B is a schematic diagram of spatial domain independence of slice according to an embodiment of the present application. Because the decoding operations are independent from slice to slice. Therefore, as can be seen from fig. 1B, if slice1 and slice3 are lost in the currently decoded frame 102 in the code stream transmission, slice2 can still be decoded successfully if the reference frame 101 is decoded correctly.

In the encoding stage, the encoder can divide the current encoding frame into a plurality of slice areas according to the image division technology, and allocate the slice areas to a plurality of threads for encoding simultaneously, and when all slice encoding in one frame is completed, the encoder outputs a complete encoding frame.

Referring to fig. 2A, fig. 2A is a schematic diagram of a multi-thread encoding according to an embodiment of the present application. As can be seen from fig. 2A, the current encoded frame 200 is divided into slice1, slice2, and slice3, slice1 in the current encoded frame 200 is allocated to thread 1, slice2 in the current encoded frame 200 is allocated to thread 2, and slice3 in the current encoded frame is allocated to thread 3, respectively. Thread 1, thread 2 and thread 3 perform encoding simultaneously, a complete encoded frame can be output in the encoding completion of slice1, slice2 and slice3 in the current encoded frame 200, and all slice data in the frame after the encoding completion can be saved in the slice context sequence (slice contaxt list).

Referring to fig. 2B, fig. 2B is a schematic diagram of slice dependent reference frames during encoding according to an embodiment of the present application. As can be seen from fig. 2B, during the encoding phase, the encoding of slice1 in the current encoded frame 200 requires reference to the pixel information of slice2 region in the reference frame 201; for the encoding of slice2 in the current encoded frame 200, reference is required to the pixel information of slice3 region in the reference frame 201, and for the encoding of slice3 in the current encoded frame 200, reference is required to the pixel information of slice1 region in the reference frame 201.

Referring to fig. 2C, fig. 2C is a schematic diagram illustrating a multi-thread decoding according to an embodiment of the present application. As can be seen from fig. 2C, in the decoding stage, when the decoder receives a single frame image cut into multiple slices, the slices are sequentially added into a queue, and after all slices in the single frame image are waiting for receiving, the main thread M is notified to start multi-thread decoding. Meanwhile, the main thread M notifies the plurality of decoding threads T to fetch the corresponding slice from slice context list for decoding. Finally, waiting until all threads complete the task and returning. I.e. multiple slices of a single frame can be decoded in parallel.

Referring to fig. 2D, fig. 2D is a schematic diagram of slice dependent on a reference frame when decoding a frame according to an embodiment of the present application. As can be seen from fig. 2D, in the decoding phase, the decoding of slice1 in the current decoded frame 202 requires reference to the pixel information of slice4 region in reference frame 203 and the pixel information of slice2 region in reference frame 204, rather than relying solely on the pixel information of slice1 regions in reference frame 203 and reference frame 204.

In current application practice, since the independence of slice is only that of spatial domain. I.e. a plurality of slices in the current image are independent. In the coding and decoding process, multiple slices depending on the reference frame are needed for coding and decoding the current frame. Therefore, in practical applications, the full decoding and full encoding of the slice are generally performed.

In some application scenarios, for example, when video color ring is superimposed with advertisement, the region to be superimposed occupies only a small part of the video frame.

Referring to fig. 3A, fig. 3A is a schematic diagram of superimposing content on a video according to an embodiment of the application. As can be seen in fig. 3A, the overlay area 301 is located at the bottom of the picture of the video 300, accounting for only a small portion of the picture of the video 300. However, due to the dependency limit of slice encoding and decoding on the reference frame in the prior art, decoding and encoding of the full image are required, and the calculation overhead may be large.

In summary, the slice provided by the current scheme is independent in the spatial domain, but since the video is a continuous image, generally all the video needs to be encoded and decoded by referring to the previous and next frames in the time dimension. Therefore, when slices are independent only in the spatial domain, decoding of slices requires slice information referencing the full frame ahead, meaning that reference frames require full decoding. If only a single slice is forcedly decoded, the quality of the subsequent image is affected under the condition that the reference frame is not correctly decoded. Fig. 3B is a schematic diagram of a decoding error provided by an embodiment of the present application, and it can be seen from fig. 3B that, during decoding, an image in a video 300 is divided into slice1, slice2, and slice3. Because slice2 is decoded in the event that the reference frame of the image in video 300 fails to decode correctly, decoding errors occur, resulting in some content in the slice region not being displayed.

When the slice on a certain frame in the video is decoded independently, the current scheme needs to decode all slices in the reference frame of the current frame to ensure that the slice is decoded correctly, which means that full decoding is needed, and extra performance overhead is caused. Taking 480P full-scale codec as an example, a single CPU core can handle decoding and encoding of about 3 h.264 channels, and taking a typical 2P20C server as an example, a maximum of 120 channels can be presumably supported. And services such as video color ring and the like need to support about 1000 paths of concurrency by 2P20C, so that the current scheme and mechanism cannot meet the requirement of high concurrency scenes.

In view of the above, the present application provides a video encoding and decoding method and related apparatus. The method and the device have the advantages that any frame in an image sequence of an original video is segmented in a time domain to obtain a plurality of areas to be processed (such as slices), so that the slices in a single frame can be completely separated, and meanwhile, in the whole video playing process, decoding of the single slice only depends on slices in the same position of the previous frames. Referring to fig. 3C, fig. 3C is a schematic diagram of a single slice decoding according to the present application. As can be seen from fig. 3C, decoding of slice1 in the current frame 303 refers to only slice1 in the reference frame 301 and slice1 in the reference frame 302; decoding of slice2 in the current frame 303 refers to only slice2 in the reference frame 301 and slice2 in the reference frame 302; decoding of slice3 in the current frame 303 refers to only slice3 in the reference frame 301 and slice3 in the reference frame 302; decoding of slice4 in the current frame 303 refers to only slice4 in the reference frame 301 and slice4 in the reference frame 302. Therefore, decoding slice1 in the current frame 303 does not require full decoding of the reference frame 301 and the reference frame 302, which can reduce computational overhead.

Fig. 4A is a schematic diagram of a frame of a video decoding system 4 according to an embodiment of the present application. As used herein, the term "decoder" generally refers to both an encoder and a decoder. In the application, the term "video coding" or "coding" may refer generally to video encoding or video coding. The encoder 410 and the decoder 420 of the video decoding system 4 are configured to perform segmentation processing on a current encoded frame according to the video processing method proposed by the present application, to obtain a to-be-processed area (e.g., slice), and determine matching blocks for image blocks (e.g., macro blocks) in the to-be-processed area, so that the positions of the matching blocks are within the range of the to-be-processed area. Therefore, the decoding of the to-be-processed region can only depend on the region at the same position in the reference frame, the reference frame does not need to be decoded in a full amount, and the calculation cost can be reduced.

As shown in fig. 4A, video coding system 4 includes a source device 41 and a destination device 42. Source device 41 generates encoded video data. Thus, the source device 41 may be referred to as a video encoding device. Destination device 42 may decode the encoded video data generated by source device 41. Thus, destination device 42 may be referred to as a video decoding device. Various implementations of source device 41, destination device 42, or both may include one or more processors and memory coupled to the one or more processors. The memory may include, but is not limited to RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures that can be accessed by a computer, as described in this disclosure.

Source device 41 and destination device 42 may include various devices including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, televisions, cameras, display devices, digital media players, video game consoles, vehicle mount computers, or the like.

Destination device 42 may receive encoded video data from source device 41 via link 43. Link 43 may comprise one or more media or devices capable of moving encoded video data from source device 41 to destination device 42. In one example, link 43 may comprise one or more communication media that enable source device 41 to transmit encoded video data directly to destination device 42 in real-time. In this example, source device 41 may modulate the encoded video data according to a communication standard, such as a wireless communication protocol, and may transmit the modulated video data to destination device 42. The one or more communication media may include wireless and/or wired communication media such as a Radio Frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the internet). The one or more communication media may include routers, switches, base stations, or other equipment that facilitate communication from source device 41 to destination device 42.

In another example, encoded data may be output from output interface 411 to storage device 44. Similarly, encoded data may be accessed from storage 44 through input interface 421. Storage device 44 may include any of a variety of distributed or locally accessed data storage media such as hard drives, blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

In another example, storage device 44 may correspond to a file server or another intermediate storage device that may hold the encoded video generated by source device 41. Destination device 42 may access stored video data from storage device 44 via streaming or download. The file server may be any type of server capable of storing encoded video data and transmitting the encoded video data to destination device 42. Example file servers include web servers (e.g., for websites), FTP servers, network Attached Storage (NAS) devices, or local disk drives. Destination device 42 may access the encoded video data over any standard data connection, including an internet connection. This may include a wireless channel (e.g., wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both suitable for accessing encoded video data stored on a file server. The transmission of encoded video data from storage device 44 may be a streaming transmission, a download transmission, or a combination of both.

The motion vector prediction techniques of this disclosure may be applied to video encoding and decoding to support a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, streaming video transmission (e.g., via the internet), encoding of video data for storage on a data storage medium, decoding of video data stored on a data storage medium, or other applications. In some examples, video coding system 4 may be used to support unidirectional or bidirectional video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

Video coding system 4 illustrated in fig. 4A is merely an example, and the techniques of this disclosure may be applicable to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, the data is retrieved from local memory, streamed over a network, and so forth. The video encoding device may encode and store data to the memory and/or the video decoding device may retrieve and decode data from the memory. In many examples, encoding and decoding are performed by devices that do not communicate with each other, but instead only encode data to memory and/or retrieve data from memory and decode data.

In the example of fig. 4A, source device 41 includes a video source 412, an encoder 410, and a video coding system 4. In some examples, video coding system 4 may include a regulator/demodulator (modem) and/or a transmitter. Video source 412 may include a video capture device (e.g., a video camera), a video archive containing previously captured video data, a video feed interface to receive video data from a video content provider, and/or a computer graphics system for generating video data, or a combination of such sources of video data.

Encoder 410 may encode video data from video source 412. In some examples, source device 41 transmits encoded video data directly to destination device 42 via video coding system 4. In other examples, the encoded video data may also be stored onto storage device 44 for later access by destination device 42 for decoding and/or playback.

In the example of fig. 4A, destination device 42 includes an input interface 421, a decoder 420, and a display device 422. In some examples, the input interface 421 includes a receiver and/or a modem. Input interface 421 may receive encoded video data via link 30 and/or from storage 44. The display device 422 may be integrated with the destination device 42 or may be external to the destination device 42. In general, display device 422 displays decoded video data. Display device 422 may include a variety of display devices, such as a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or other types of display devices.

Although not shown in fig. 4A, in some aspects, the encoder 410 and decoder 420 may each be integrated with an audio encoder and decoder, and may include an appropriate multiplexer-demultiplexer unit or other hardware and software to handle encoding of both audio and video in a common data stream or separate data streams. In some examples, the MUX-DEMUX units may conform to the ITU h.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP), if applicable.

Encoder 410 and decoder 420 may each be implemented as any of a variety of circuits, such as: one or more microprocessors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), discrete logic, hardware, or any combinations thereof. If the present application is implemented in part in software, the device may store instructions for the software in a suitable non-volatile computer-readable storage medium and the instructions may be executed in hardware using one or more processors to implement the techniques of the present application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered one or more processors. Each of encoder 410 and decoder 420 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder/decoder (codec) in the respective device.

This disclosure may generally refer to encoder 410 as "signaling" or "transmitting" certain information to another device, such as decoder 420. The term "signaling" or "transmitting" may generally refer to the transfer of syntax elements and/or other data used to decode compressed video data. This transfer may occur in real time or near real time. Alternatively, this communication may occur over a period of time, such as may occur when syntax elements are stored to a computer-readable storage medium in an encoded bitstream when encoded, which the decoding device may then retrieve at any time after the syntax elements are stored to such medium.

Encoder 410 and decoder 420 may operate in accordance with a video compression standard, such as High Efficiency Video Coding (HEVC), or an extension thereof, and may conform to the HEVC test model (HM). Or encoder 410 and decoder 420 may operate in accordance with other industry standards such as the ITU-T h.264, h.265 standards, or extensions of such standards. However, the techniques of this disclosure are not limited to any particular codec standard.

Fig. 4B is a schematic diagram of an encoder 410 according to an embodiment of the present application. Encoder 410 is used to output video to post-processing entity 413. Post-processing entity 413 represents an instance of a video entity that may process the encoded video data from encoder 410, such as a Media Aware Network Element (MANE), a stitching/editing device, or a video color ring forwarding platform. In some cases, post-processing entity 413 may be an instance of a network entity. In some video coding systems, post-processing entity 413 and encoder 410 may be parts of separate devices, while in other cases, the functionality described with respect to post-processing entity 413 may be performed by the same device that includes encoder 410. In one example, post-processing entity 413 is an example of storage device 44 of FIG. 4A.

Functional components of the encoder 410 may include a partition unit 4101, a prediction unit 4102, a residual generation unit 112, a transform unit 4103, a quantization unit 4104, and an entropy encoding unit 4105. The prediction unit 4102 includes a mode selection unit 4102a, an inter prediction unit 4102b, and an intra prediction unit 4102c. For image block reconstruction, the encoder 410 may further include an inverse quantization unit 4106, an inverse transform unit 4107, and a summing unit 111. Loop filter unit 4108 represents one or more loop filter units, such as a deblocking filter unit, an adaptive loop filter unit (ALF), and a Sample Adaptive Offset (SAO) filter unit. In other implementations, the loop filtering unit 4108 may be implemented as a post loop filter. In one example, encoder 410 may also include a storage unit 4109.

The storage unit 4109 may store video data to be encoded by components of the encoder 410. Video data stored in the storage unit 4109 may be obtained from the video source 412. Storage unit 4109 may be a reference image storage unit that stores reference video data for encoding video data by encoder 410 in intra, inter coding modes. The memory cell 4109 may be formed from any of a variety of memory cell devices, such as dynamic random access memory cells (DRAM) including Synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory cell devices. The memory unit 4109 may be provided by the same memory unit device or separate memory unit devices. In various examples, the storage unit 4109 may be on-chip with other components of the encoder 410, or off-chip with respect to those components.

Referring to fig. 4B, fig. 4B is a schematic diagram of an encoder according to an embodiment of the application. As shown in fig. 4B, the encoder 410 receives video data and stores the video data in a video data storage unit. The segmentation unit segments the video data into a number of regions to be processed, which contain a number of image blocks, and these image blocks may be further segmented into smaller blocks, e.g. image block segments based on a quadtree structure or a binary tree structure. The area to be processed may be a slice, tile or other larger unit. Encoder 410 generally illustrates the components that encode image blocks within a video slice to be encoded. The slice may be divided into a plurality of tiles (and possibly into a set of tiles called slices), for example, for the coding standard h.264/AVC, the slice may be divided into a plurality of macro blocks.

In a possible implementation manner, the segmentation unit is further used for calculating an area occupied by the area to be processed, and is specifically used for calculating the sequence number of the first image block and the sequence number of the last image block in the area to be processed; calculating the area information occupied by the area to be processed, wherein the area information comprises the number of lines of the area to be processed, and whether the first line and the last line of the area to be processed are full or not; and calculating the area occupied by the area to be processed according to the serial number of the first image block, the serial number of the last image block and the area information.

The mode selection unit 4102a in the prediction unit 4102 may select one of a plurality of possible coding modes for the current image block, such as one of a plurality of intra coding modes or one of a plurality of inter coding modes. The prediction unit 4102 may provide the resulting intra-frame, inter-coded blocks to a residual generation unit 112 to generate residual blocks, and to a summation unit 111 to reconstruct the encoded blocks used as reference pictures. In some embodiments, the mode selection unit 4102a may select a combination of intra prediction and inter prediction modes, where prediction is based on inter prediction information and intra prediction information, and the mode selection unit 4102a may also select a resolution (e.g., sub-pixel precision or integer pixel precision) of the run vector for the block in the case of inter prediction.

The intra prediction unit 4102c within the prediction unit 4102 may perform intra predictive encoding of the current image block with respect to one or more neighboring blocks in the same frame or slice as the current block to be encoded, removing spatial redundancy. The inter prediction unit 4102b in the prediction unit 4102 may perform inter predictive encoding of the current image block with respect to one or more matching blocks in one or more reference images to remove spatial redundancy.

Specifically, the inter prediction unit 4102b may be used to determine an inter prediction mode for encoding the current image block. For example, the inter prediction unit 4102b may use rate-distortion analysis to calculate rate-distortion values for various inter prediction modes in the set of candidate inter prediction modes and select the inter prediction mode with the best rate-distortion characteristics from among them. Rate-distortion analysis typically determines the amount of distortion (or error) between an encoded block and an original, unencoded block encoded to produce the encoded block, as described above, as well as the bit rate (that is, the number of bits) used to produce the encoded block. For example, the inter prediction unit 4102b may determine that an inter prediction mode having the smallest code rate distortion cost for encoding the current image block in the candidate inter mode set is an inter prediction mode for inter predicting the current image block.

The inter prediction unit 4102b is configured to predict motion information (e.g., motion vectors) of one or more sub-blocks in the current image block based on the determined inter prediction mode, and obtain a matching block generating the current image block using the motion information (e.g., motion vectors) of the one or more sub-blocks in the current image block. The inter prediction unit 4102b may locate the matching block to which the motion vector points in any reference frame in the reference image list, where the region occupied by the reference region to which the motion vector points is within the region occupied by the region to be processed to which the image block belongs. The inter prediction unit 4102b may generate a reference index containing the matching blocks in the reference image list and motion information (e.g., motion vector) indicating between the current image block and the matching blocks. Wherein the region to which the motion vector points is within the region to be processed. The inter prediction unit 4102b may output the reference index, the prediction method indicator, and the motion vector as motion information of the current encoded block. For use by decoder 420 in decoding image blocks of a slice.

In still another example, the inter prediction unit 4102b performs a motion compensation process using the motion information of each sub-block to generate a matching block of each sub-block, thereby obtaining a matching block of the current image block. It should be appreciated that the inter prediction unit 4102b herein performs motion estimation and motion compensation processes.

In the embodiment of the present application, the inter prediction unit 4102b may determine whether the motion vector of the current image block is out of range according to the region occupied by the region to be processed. That is, if the region indicated by the motion vector of the current image block is not within the region to be processed, the boundary crossing is described. And carrying out constraint processing on the out-of-range current image block to obtain a constraint motion vector of the current image block, so that the area pointed by the constraint motion vector is in the area occupied by the area to be processed. It is understood that the matching block is located within the region indicated by the motion vector of the current image block.

In one possible implementation, the inter prediction unit 4102b is specifically configured to: respectively calculating pixel residual errors between the region indicated by the motion vector of the adjacent block and the target image block according to the motion vector of the adjacent block of the target image block; and if the motion vector with the pixel residual error of 0 exists, if the region indicated by the motion vector of the adjacent block corresponding to the pixel residual error of 0 is not in the region occupied by the region to be processed, carrying out constraint processing on the target image block to obtain the motion vector of the target image block. Wherein the constraint processing includes: integer pixel motion estimation and sub-pixel motion estimation.

In one possible implementation, the inter prediction unit 4102b is specifically configured to: performing the whole pixel motion estimation on the target image block according to the motion vector corresponding to the adjacent block of the target image block to obtain a whole pixel motion searching end point; and carrying out sub-pixel motion estimation on the target image block according to the whole-pixel motion searching end point to obtain a motion vector of the target image block.

In one possible implementation, the inter prediction unit 4102b is specifically configured to: if the region indicated by the motion vector of the adjacent block of the target image block exists in the occupied region of the region to be processed, calculating a first code rate distortion cost of the motion vector of the adjacent block; and determining a whole pixel motion searching end point according to a motion vector corresponding to the minimum first code rate distortion cost in the first code rate distortion costs.

In one possible implementation, the inter prediction unit 4102b is specifically configured to: if the region indicated by the motion vector of the adjacent block of the target image block is not in the occupied region of the region to be processed, scaling the motion vector corresponding to the adjacent block of the target image block in equal proportion according to the boundary information of the region to be processed and the position information of the target image block to obtain a scaled motion vector, wherein the region indicated by the scaled motion vector is in the occupied region of the region to be processed; calculating a second code rate distortion cost of the scaled motion vector; and determining a whole pixel motion searching end point according to a motion vector corresponding to the minimum second code rate distortion cost in the second code rate distortion costs.

In one possible implementation, the inter prediction unit 4102b is specifically configured to: calculating a sub-pixel reference area according to the sub-pixel interpolation calculation relation; and if the region indicated by the motion vector corresponding to the whole pixel motion searching end point is not in the region occupied by the sub-pixel reference region, taking the motion vector corresponding to the whole pixel motion searching end point as the motion vector of the target image block.

In one possible implementation, the inter prediction unit 4102b is specifically configured to: and if the region indicated by the motion vector corresponding to the whole pixel motion searching end point is in the region occupied by the sub-pixel reference region, sub-pixel motion estimation is carried out on the target image block, and the motion vector of the target image block is obtained.

In one possible implementation, the inter prediction unit 4102b is specifically configured to: performing sub-pixel motion estimation on the target image block according to the whole-pixel motion searching end point, and determining a sub-pixel motion vector, wherein the area pointed by the sub-pixel motion vector is in the area occupied by the sub-pixel reference area; calculating a third code rate distortion cost of the sub-pixel motion vector; and taking the sub-pixel motion vector corresponding to the minimum third code rate distortion cost in the third code rate distortion cost as the motion vector of the target image block.

The intra prediction unit 4102c may perform intra prediction on the current image block. In particular, the intra prediction unit 4102c may determine an intra prediction mode used to encode the current block. For example, the intra prediction unit 4102c may calculate rate-distortion values of various intra prediction modes to be tested using rate-distortion analysis, and select an intra prediction mode having the best rate-distortion characteristics from among the modes to be tested. In any case, after selecting the intra prediction mode for the image block, the intra prediction unit 4102c may provide information indicating the selected intra prediction mode of the current image block to the entropy encoding unit 4105 so that the entropy encoding unit 4105 encodes the information indicating the selected intra prediction mode.

After the prediction unit 4102 generates a matching block of the current image block via inter prediction, intra prediction, the encoder 410 forms a residual image block by subtracting the above matching block from the current image block to be encoded. Residual generation unit 112 represents one or more components that perform this subtraction operation. The residual video data in the residual block described above is applied to the transform unit 4103. The transform unit 4103 transforms the residual video data into residual transform coefficients using a transform such as a Discrete Cosine Transform (DCT) or a conceptually similar transform. The transform unit 4103 may convert the residual video data from a pixel value domain to a transform domain, such as the frequency domain.

After quantization, the entropy encoding unit 4105 entropy encodes the quantized transform coefficients. For example, entropy encoding unit 4105 may perform Context Adaptive Variable Length Coding (CAVLC), context Adaptive Binary Arithmetic Coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability Interval Partitioning Entropy (PIPE) coding, or another entropy encoding method or technique. After entropy encoding by entropy encoding unit 4105, the encoded bitstream may be transmitted to decoder 420, or archived for later transmission or retrieval by decoder 420. The entropy encoding unit 4105 may also entropy encode syntax elements of the current image block to be encoded.

The inverse quantization unit 4106 and inverse transform unit 4107 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, e.g., for later use as a matching block for a reference image. The summing unit 111 adds the reconstructed residual block to the matching block generated by the inter prediction unit 4102b or the intra prediction unit 4102c to generate a reconstructed image block. The filtering unit 4108 may be adapted to reconstruct image blocks to reduce distortion, such as block artifacts (blockartifacts). Then, the reconstructed image block is stored as a matching block in the decoded image storage unit 4109, and can be used as a matching block by the inter prediction unit 4102b to perform inter prediction on a block in a subsequent video frame or image.

It should be appreciated that other structural variations of encoder 410 may be used to encode a video stream. For example, for some image blocks or image frames, the encoder 410 may directly quantize the residual signal without processing by the transform unit 4103, and correspondingly without processing by the inverse transform unit 4107; or for some image blocks or image frames, the encoder 410 does not generate residual data and accordingly does not need to be processed by the transform unit 4103, the quantization unit 4104, the inverse quantization unit 4106 and the inverse transform unit 4107; or the encoder 410 may directly store the reconstructed image block as a matching block without processing by the filtering unit 4108; or the quantization unit 4104 and the inverse quantization unit 4106 in the encoder 410 may be combined together. The filtering unit 4108 is optional, and in the case of lossless compression encoding, the transforming unit 4103, the quantizing unit 4104, the inverse quantizing unit 4106, and the inverse transforming unit 4107 are optional. It should be appreciated that the inter prediction unit and the intra prediction unit may be selectively enabled according to different application scenarios, and in this case, the inter prediction unit is enabled.

Fig. 4C is a block diagram of a decoder 420 according to an embodiment of the present application. In the example of fig. 4C, decoder 420 includes entropy decoding unit 4201, prediction processing unit 4202, inverse quantization unit 4203, inverse transform unit 4204, summing unit 211, filtering unit 4205, and stored unit 4206. The prediction processing unit 4202 may include an inter prediction unit 4202b and an intra prediction unit 4202a. In some examples, decoder 420 may perform a decoding process that is substantially reciprocal to the encoding process described with respect to encoder 410 from fig. 4B.

During decoding, decoder 420 receives an encoded video bitstream from encoder 410 that represents image blocks of an encoded video slice and associated syntax elements. The decoder 420 may receive video data from the network entity 423 and may optionally also store the video data in a video data storage unit (not shown). The video data storage unit may store video data, such as an encoded video bitstream, to be decoded by components of decoder 420. The video data stored in the video data storage unit may be obtained, for example, from the storage device 44, from a local video source such as a camera, via wired or wireless network communication of the video data, or by accessing a physical data storage medium. The video data storage unit may be as a stored unit (CPB) for storing encoded video data from an encoded video bitstream.

Thus, although a video data storage unit is not illustrated in fig. 4C, the video data storage unit and the storage 4206 may be the same storage unit or may be separately provided storage units. Video data storage unit and storage 4206 may be formed from any of a variety of storage unit devices, such as: dynamic random access memory cells (DRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory cell devices, including Synchronous DRAM (SDRAM). In various examples, the video data storage unit may be integrated on-chip with other components of decoder 420 or off-chip with respect to those components.

Network entity 423 may be, for example, a server, a MANE, a video editor/splicer, or other such device for implementing one or more of the techniques described above. Network entity 423 may or may not include an encoder, such as encoder 410. The network entity 423 may implement portions of the techniques described in this disclosure before the network entity 423 sends the encoded video bitstream to the decoder 420. In some video decoding systems, network entity 423 and decoder 420 may be part of separate devices, while in other cases the functionality described with respect to network entity 423 may be performed by the same device that includes decoder 420. In some cases, the network entity 423 may be an example of the storage 44 of fig. 4A.

The entropy decoding unit 4201 of the decoder 420 entropy decodes the bitstream to generate quantized coefficients and some syntax elements. Entropy decoding unit 4201 forwards the syntax elements to prediction processing unit 4202. The decoder 420 may receive syntax elements at the video slice level and/or the picture block level.

When a video slice is decoded as an intra-decoded (I) slice, the intra-prediction unit 4202a of the prediction processing unit 4202 may generate a matching block of image blocks of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or image. When a video slice is decoded as an inter-decoded (i.e., B or P) slice, the inter-prediction unit 4202B of the prediction processing unit 4202 may determine an inter-prediction mode for decoding a current image block of a current video slice based on the syntax element received from the entropy decoding unit 4201, and decode the current image block (e.g., perform inter-prediction) based on the determined inter-prediction mode. Specifically, the inter prediction unit 4202b may determine whether to predict the current image block of the current video slice using a new inter prediction mode, if the syntax element indicates that the current image block is predicted using the new inter prediction mode, predict motion information of the current image block or a sub-block of the current image block of the current video slice based on the new inter prediction mode (e.g., a new inter prediction mode specified by the syntax element or a default new inter prediction mode), and thereby acquire or generate a matching block of the current image block or the sub-block of the current image block using the motion information of the predicted current image block or the sub-block of the current image block through the motion compensation process.

The motion information herein may include reference picture information and motion vectors, wherein the reference picture information may include, but is not limited to, uni-directional/bi-directional prediction information, a reference picture list number, and a reference picture index corresponding to the reference picture list. For inter prediction, a matching block may be generated from one of the reference pictures within one of the reference picture lists.

In the embodiment of the application, the reference area occupies an area within the area occupied by the area to be processed where the image block is located. Therefore, in decoding video, it is not necessary to decode all regions in the reference frame of the current image frame. Only the region at the same position is decoded, full decoding is not needed, and the calculation cost can be reduced.

The decoder 420 may construct reference picture lists, i.e., list 0 and list 1, based on the reference pictures stored in the storage unit 4206. The reference frame index of the current image may be included in one or more of reference frame list 0 and list 1. In some examples, it may be that the encoder 410 signals whether to decode a particular block using a new inter prediction mode, or it may also signal whether to use a new inter prediction mode, and which new inter prediction mode to specifically decode a particular block. It should be understood that the inter prediction unit 4202b here performs a motion compensation process.

The dequantization unit 4203 dequantizes, i.e., dequantizes, the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 4201. The inverse quantization process may include: the quantization parameter calculated by the encoder 410 for each image block in the video slice is used to determine the degree of quantization that should be applied and likewise the degree of inverse quantization that should be applied. Inverse transform unit 4204 applies an inverse transform to the transform coefficients, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, in order to generate residual blocks in the pixel domain.

After the inter prediction unit 4202b generates a matching block for the current image block or a sub-block of the current image block, the decoder 420 obtains a reconstructed block, i.e., a decoded image block, by summing the residual block from the inverse transform unit 4204 with the corresponding matching block generated by the inter prediction unit 4202 b. The summing unit 211 represents a component that performs this summing operation. A loop filtering unit (in or after the decoding loop) may also be used to smooth out pixel transitions or otherwise improve video quality, if desired. The filtering unit 4205 may represent one or more loop filtering units, such as a deblocking filtering unit, an adaptive loop filtering unit (ALF), and a Sample Adaptive Offset (SAO) filtering unit.

Although filtering unit 4205 is shown as an in-loop filtering unit in fig. 4C, in other implementations filtering unit 4205 may be implemented as a post-loop filtering unit. In one example, the filtering unit 4205 is adapted to reconstruct a block to reduce block distortion, and the result is output as a decoded video stream. Also, the decoded image blocks in a given frame or image may be stored in the storage unit 4206, and the reference image for subsequent motion compensation may be stored via the storage unit 4206. The via storage unit 4206 may be part of a storage unit that may also store decoded video for later presentation on a display device (e.g., display device 422 of fig. 4A), or may be separate from such storage unit.

It should be appreciated that other structural variations of decoder 420 may be used to decode the encoded video bitstream. For example, the decoder 420 may generate an output video stream without processing by the filtering unit 4205; or for some image blocks or image frames, the entropy decoding unit 4201 of the decoder 420 does not decode quantized coefficients, and accordingly does not need to be processed by the inverse quantization unit 4203 and the inverse transform unit 4204. The loop filter unit is optional; and for the case of lossless compression, the inverse quantization unit 4203 and the inverse transform unit 4204 are optional. It should be appreciated that the inter prediction unit and the intra prediction unit may be selectively enabled according to different application scenarios, and in this case, the inter prediction unit is enabled.

Referring to fig. 5, fig. 5 is a schematic view of a scene of locally overlapping information on video according to an embodiment of the present application. As can be seen from fig. 5, any frame in the image sequence of the original video may be composed of a single slice. In order to increase the encoding and decoding speed, the preprocessing stage of the original video may be an encoding stage of the original video. The encoder may segment any frame in the image sequence of the original video, cut the current encoded frame into a plurality of slice regions, and perform an independent process for each slice. The cutting modes and the cutting number can be varied according to the requirements of the service scene, for example, 1 is larger and smaller, N pieces are uniform, and the like. For example, in the video color ring service, the superimposition processing is generally performed at the lowest part of the image, so that a one-size-one cutting mode can be used.

As can be seen from fig. 5, the current encoded frame 501 is a single slice image, and the current encoded frame 501 is cut into 6 equal-sized slice regions, including slice1, slice2, slice3, slice4, slice5, and slice6. Wherein each slice region contains a plurality of tiles. For the to-be-processed area (i.e. the current coding slice area), the encoder can check whether the motion vector of the image block is out of range in the inter-frame motion process according to the slice area, so as to realize the constrained image block motion estimation.

For example, the slice6 area of the current encoded frame 501 is an area to be processed, and in the motion estimation process, a matching block of an image block in the slice6 area of the current encoded frame 501 can be calculated according to the motion vector found in the motion estimation process. It will be appreciated that the matching block is an image block on the reference frame 502, and the reference frame may be cut in the same cutting manner, so that the reference frame 502 also contains 6 slice regions with the same size. Since there may be a plurality of motion vectors found, there may be a plurality of matching blocks of the image block on the reference frame 502. The encoder may select a matching block within the slice6 region of the reference frame 502 from a plurality of matching blocks as a matching block for the image block. The encoder may generate a reference index containing the matching blocks in the slice6 region of the reference frame 502, as well as motion information indicating between the current image block and the matching blocks.

After the preprocessing of the original video is completed, the content may be stored in a storage device, such as storage device 44 shown in fig. 4A.

The transcoding playing platform can read the video files subjected to multi-slice segmentation from the storage device, decode, superimpose text/pictures and encode the few slices according to service requirements, and copy the other slices. For example, as can be seen from fig. 5, if the content is superimposed on the lowest part of the video in the color ring service (for example, the slice6 area), since in the preprocessing stage, the matching block of the image block in the slice6 area of the current encoded frame is located in the slice6 area in the reference frame 502. Therefore, the transcoding and playing platform can perform independent processing on the slice6 area, perform decoding-overlapping content-transcoding on the slice6 area, perform copy processing on other slice areas (i.e. slice1 area, slice2 area, slice3 area, slice4 area and slice5 area) which do not need overlapping content, and then display the video of the overlapping content on the display device. Therefore, all slice areas on the reference frame do not need to be completely decoded, and the slice6 area on the current decoded frame can be correctly decoded, so that the aim of high concurrence and rapid local decoding is fulfilled.

Referring to fig. 6, fig. 6 is a flowchart of a video encoding method according to an embodiment of the application.

In the embodiment of the present application, an image block being encoded is referred to as a target image block, and an image in which the target image block is located is referred to as a current frame. Therefore, the video encoding method mentioned in the present application may be specifically a method for encoding a series of images included in a video. The method includes, but is not limited to, the steps of:

step S601, performing segmentation processing on an image frame in an image sequence of an original video to obtain a region to be processed.

It is understood that video includes a series of pictures (pictures), one picture being referred to as a frame. To increase the encoding and decoding speed, the image is divided into at least one slice, each slice being in turn divided into image blocks. Video encoding or video decoding is in units of image blocks. For example, the encoding process and decoding process may be performed from left to right, from top to bottom, line to line, starting from the upper left corner position of the image. The area to be processed may be a slice area, and the area to be processed may include one or more image blocks. The image block may be a macroblock in the video codec standard h.264, and the macroblock is divided into sub-blocks, or may be a Coding Unit (CU) in the HEVC standard. The embodiment of the present application is not particularly limited thereto.

Referring to fig. 7A, fig. 7A is a schematic diagram of a different type of image block according to an embodiment of the present application. As can be seen from fig. 7A, during the preprocessing stage, the encoder can divide the image frame 701 of the original video into N equal divisions or one-size-one divisions into the region to be processed 702. The specific division manner and the division number can be determined according to the service requirement, which is not particularly limited by the embodiment of the present application. While each of the regions to be processed 702 may be partitioned into image blocks 7021 for encoding, the image blocks 7021 may be arbitrarily sized blocks including, but not limited to, 16 x 16 pixels, 8 x 16 pixels, and 16 x 8 pixels, and 8 x 8 pixels, etc. The image block 7021 may be further divided into smaller blocks including, but not limited to, 4×8 pixels, 8×4 pixels, 4×4 pixels, and the like.

Two variables i8 and i4 are introduced to represent the type of image block, i8 acts on three types of 8×16 pixels and 16×8 pixels and 8×8 pixels, and i8 acts on three types of 4×8 pixels, 8×4 pixels and 4×4 pixels. The numbers of blocks on the 8×16 pixels and 16×8 pixels and 8×8 pixels partitions in fig. 7A represent the value of i8, indicating what number of blocks the block belongs to on the current image block. Similarly, the numbers of each block on the blocks of 4×8 pixels, 8×4 pixels, and 4×4 pixels represent the i4 value, indicating to which block the block belongs on the current image block.

In step S602, a reference area of the target image block in the reference frame is determined.

It will be appreciated that there is a strong temporal correlation between successive frames of images in the video. That is, a lot of redundant information is included between adjacent frames. Therefore, when video coding is performed, the time correlation between frames is often utilized to reduce the redundancy between frames, so as to achieve the purpose of compressing data. In the encoding stage, when the encoder encodes a current image block (i.e., a target image block in the embodiment of the present application) in a current frame, firstly, more than one reference frame is arbitrarily selected from the frames encoded by the video image, and a reference area of the target image block is determined from the reference frames. Currently, motion estimation techniques are mainly used to encode video to increase the compression ratio.

Referring to fig. 7B, fig. 7B is a schematic diagram illustrating block-based motion estimation according to an embodiment of the present application. As can be seen from fig. 7B, the basic idea of block-based motion estimation is to divide each frame in an image sequence into a number of mutually non-overlapping image blocks 7021, the size of these image blocks 7021 tend to be fixed, and the displacement amounts of all pixels within the image blocks 7021 are considered to be the same. Thus, for each image block 7021 in the current image frame 701, the block that best matches the current image block 7021, i.e., the matching block 7041, can be found according to a certain matching rule within a given search area 703 of the reference frame 704. The motion displacement is calculated from the relative positions of the matching block 7041 and the current image block 7021, and the obtained motion displacement is the motion vector of the current image block 7021. Assuming that the coordinates of the upper left corner of the current image block 7021 are (x 1, y 1) and the coordinates of the upper left corner of the matching block 7041 of the current image block 7021 are (x 2, y 2), the motion vector of the current image block 7021 may be (mv_x=x1-x 2, mv_y=y1-y 2).

Referring to fig. 7C, fig. 7C is a schematic diagram of determining a matching block for an image block according to an embodiment of the present application. As can be seen from fig. 7C, the image frame 701 is divided into three to-be-processed regions, a first to-be-processed region, a second to-be-processed region, and a third to-be-processed region, respectively.

Wherein the target image block 7011 is located in the second region to be processed. In the block-based motion estimation process, a plurality of matching blocks, namely a first matching block 7041, a second matching block 7042 and a third matching block 7043, can be found in the reference frame 704 for the target image block 7011 in the image frame 701 according to the motion vector found in the motion estimation process. Then, the encoder may determine the positions of the first, second, and third matching blocks 7041, 7042, and 7043 according to the position of the target image block 7011 and the first, second, and third motion vectors, respectively. Finally, the encoder may find a block within the same area as the target image block 7011 based on the above-described position of the matching block, and use a block within the same area as the target image block 7011 as the matching block of the target image block 7011.

As can be seen from fig. 7C, it can be determined from the position of the first matching block 7041 that the first matching block 7041 is within the range of the second to-be-processed region to which the target image block 7011 belongs, and thus, the encoder can save the first motion vector for subsequent decoding thereof.

It should be noted that, the area occupied by the matching block may be considered to be within the area occupied by the reference area mentioned in the embodiment of the present application.

In one possible implementation manner, in order to determine the reference area of the target image block, the encoder needs to calculate the area range occupied by the area to be processed, and in the case that the area indicated by the motion vector of the target image block is not in the area to be processed, the encoder performs constraint processing on the target image to obtain the constraint motion vector of the target image block. The region indicated by the constraint vector is in the region occupied by the region to be processed, so that the encoder can determine the reference region of the target image block from the reference frames corresponding to the image frames according to the constraint vector.

Referring to fig. 8A, fig. 8A is a flowchart illustrating a process of calculating a range of a region to be processed according to an embodiment of the application. The above-mentioned process includes, but is not limited to, the following steps:

Step S801: the sequence number (last_mb_n) of the last tile in the region to be processed is calculated.

Referring to fig. 8B, fig. 8B is a schematic diagram of an image frame according to an embodiment of the application. The image frame 801 shown in fig. 8B is any frame in the image sequence of the original video, and as can be seen from fig. 8B, the image frame 801 contains 60 image blocks, 10 image blocks in one row and 6 image blocks in one column. The image frame 801 is divided to obtain three areas to be processed, namely, a slice1 area, a slice2 area, and a slice3 area, where each slice area may include one or more image blocks. The width of image frame 801 may be represented by mb_width, and the height of image frame 801 may be represented by mb_height in units of the number of image blocks.

The first_mb_n is used for representing the sequence number of the first image block in the nth slice, and the last_mb_n is used for representing the sequence number of the last macro block in the nth slice, wherein n is an integer value less than or equal to x, and x is the number of the areas to be processed.

If the to-be-processed area (slice_n) where the target image block is located belongs to the last to-be-processed area (slice_last) in the image frame 801, the sequence number of the last image block of the to-be-processed area is denoted as last_mb_last. Since the number (x) of the areas to be processed in fig. 8B is 3, the sequence number may be denoted as last_mb_3, and the value thereof may be (mb_height) x_mb_width) -1.

If the to-be-processed area (slice_n) where the target image block is located does not belong to the last to-be-processed area (slice_last) in the image frame 801 and belongs to the first to-be-processed area (slice_l), the sequence number last_mb_l of the last image block can be obtained according to the sequence number first_mb_l (l+1) of the first image block in the first+1 to-be-processed area, where the sequence number last_mb_l of the last image block is first_mb_l (l+1) -1. Wherein, the value of l is smaller than x positive integer.

Step S802: and calculating the range information of the area to be processed.

The sequence number of the first image block and the sequence number of the last image block in the to-be-processed area can be determined according to the to-be-processed requirement of the target image block, so that the number of lines of the to-be-processed area of the target image block, and the information of whether the first line and the last line are full or not can be calculated.

If the sequence number (first_mb_n) of the first image block in the to-be-processed area (slice_n) cannot be divided by mb_width, it indicates that the image block in the first line in the to-be-processed area is not full, and therefore, setting flag_first=1; if the first tile in the region to be processed is divisible by mb_width, it indicates that the tile of the first line in the region to be processed is full, and therefore, flag_first=0 is set.

If the sequence number ((last_mb_x) +1) of the next image block of the last image block in the area to be processed cannot be divided by mb_width, it indicates that the image block of the last line in the area to be processed is not full, and therefore, a flag_end=1 is set; if the last tile in the region to be processed can be divided by mb_width, it indicates that the tile of the last line in the region to be processed is full, and therefore, flag_end=0 is set.

If the integer part of first_mb_n divided by mb_width is the same as the integer part of last_mb_n divided by mb_width, indicating that the area to be processed contains one line, the line size of the area to be processed is equal to 1.

If the integer portion of first_mb_n divided by mb_width is not the same as the integer portion of last_mb_n divided by mb_width, indicating that the region to be processed contains multiple lines, the line size is the integer portion of last_mb_x divided by mb_width minus the integer portion of first_mb_x divided by mb_width plus 1.

Therefore, the encoder can calculate the range of the area to be processed based on the above information, i.e., the sequence number first_mb_n of the first image block, the sequence number last_mb_n of the last image block, the line size, whether the first line is full of the line flag_first, and whether the last line is full of the line flag_last in the area to be processed (slice_n).

Referring to fig. 8C, fig. 8C is a schematic diagram of coordinates of an image block according to an embodiment of the application. As can be seen from fig. 8C, mb_width and mb_height are the width and length of the image frame 801. Since an image block is a basic unit of encoding and decoding, a coordinate system x-y in units of image blocks can be established with the upper left corner of the image frame 801 as the origin (0, 0). Thus, the basic unit of coordinates is an image block.

For any one of image blocks 8011 in image frame 801, the abscissa of image block 8011 may be the distance mb_x from the upper left corner of image block 8011 to the y-axis, and the ordinate of image block 8011 may be the distance mb_y from the upper left corner of image block 8011 to the x-axis. For the sub-block 8012, the image block to which the sub-block 8012 belongs is further divided to obtain the sub-block 8012, and after determining the type (say, 4×4) of the sub-block 8012 and the coordinates (mb_x ', mb_y') of the image block to which the sub-block 8012 belongs, the coordinates of the sub-block 8012 can be determined. In general, the coordinates of the image blocks in the image frame start from 0.

Therefore, the type (mb_type), motion vector (mv_x, mv_y), and coordinates (mb_x, mb_y) of the target image block can be determined according to fig. 7A, 7B, and 8C, respectively. The encoder may calculate a reference region of the target image block in the reference frame (i.e., a region indicated by the motion vector of the target image block) according to the type (mb_type), the motion vector (mv_x, mv_y), and the coordinates (mb_x, mb_y) of the target image block. Next, the encoder may determine whether the reference region is within the range of the region to be processed according to the calculated region to which the target image block is referenced in the reference frame and the calculated range of the region to be processed.

For example, assuming that the type (mb_type) of the target image block is i8=2 and i4=1, the coordinates (mb_x+4, mb_y+8) of the upper left corner of the target image block can be obtained. The upper left corner coordinates (mb_x+4+mv_x, mb_y+8+mv_y), the upper right corner coordinates (mb_x+4+mv_x+4, mb_y+8+mv_y), the lower left corner coordinates (mb_x+4+mv_x, mb_y+8+mv_y+8) and the lower right corner coordinates (mb_x+4+mv_x+4+mv_x+4, mb_y+8+mv_y+8) of the matching block can be calculated according to the motion vector (mv_x, mv_y) obtained in the motion estimation process and the type of the target image block.

It will be appreciated that as shown in fig. 7B, a plurality of motion vectors can be obtained in the motion estimation process, so that there are a plurality of matching blocks for the target image block. If the coordinates of the four corners of the matching block are in the area to be processed, the area where the matching block is located is indicated to be in the area to be processed where the target image block belongs. If the coordinates of the four corners of the matching block are not in the to-be-processed area, the area where the matching block is located is not in the to-be-processed area where the target image block belongs, namely the area indicated by the motion vector of the target image block is not in the to-be-processed area where the target image block belongs.

In one possible implementation, the encoder may perform constraint processing on the target image block to obtain a constrained motion vector of the target image block, where an area indicated by the constrained motion vector is in a hospital occupied by a to-be-processed area to which the target image block belongs. The encoder may then determine a reference region of the target image block from the reference frames corresponding to the image frames based on the constrained motion vector.

It will be appreciated that video is typically divided into a number of groups of pictures (GOP), which is a set of consecutive pictures. The first frame of each group is encoded without motion estimation, and such frames are referred to as intra-coded frames (INTRA FRAME) or I-frames. The other frames in each group use inter-coded frames (INTER FRAME) and the past frames are used to predict the current frame, referred to as forward predictive coded frames or P frames. Predicting the current frame using the past frame and the future frame is referred to as bi-predictive interpolation encoding frame or B frame. Thus, in the inter-frame motion estimation process, an image block finds similar regions between its reference frames, and motion vectors are used to indicate the coordinate offsets of the image block between its similar regions between its reference frames.

In some possible ways, the encoder may calculate a prediction motion vector according to the motion vectors of the neighboring blocks of the target image block, and the region to which the prediction motion vector points has no pixel residual with the target image block, so that the transform coefficients are quantized to 0, and the type of the target image block is skip macroblock.

It can be appreciated that the H.264/AVC codec standard uses skip macroblock types, taking advantage of temporal and spatial correlation to save the code stream. Therefore, if the motion vector of the neighboring encoded image block of the target image block is not the residual error between the region pointed in the target image block and the target image block, the pixel information of the target image block does not need to be encoded, and the data need to be restored only according to the motion vector of the surrounding encoded image blocks and the reference frame.

It will be appreciated that the prior art requires reference to the previous and subsequent frames for encoding and decoding in the time dimension, and thus there is time-domain dependence, which is affected for the following reasons:

1) skip macroblock predicts motion vector cross-region pixel dependence.

For the type of the target image block being a skip macro block, a motion vector can be obtained according to the neighboring image blocks of the skip macro block, and the reference area pointed by the obtained motion vector is not in the area to be processed to which the skip macro block belongs.

2) The image block motion vector depends across region pixels.

For the type of the target image block which is not a skip macro block, a motion search starting point can be obtained according to the adjacent image blocks of the target image block, and the reference area pointed in the subsequent search process is not in the to-be-processed area of the target image block.

3) The interpolation of sub-pixels at the boundary of the area to be processed depends.

In the encoding and decoding process, in order to improve the encoding and decoding precision, the precision of the motion vector can be improved to 1/4 pixel size, so a sub-pixel interpolation method is adopted to determine a finer reference area. But when the region to which the motion vector of the target image block points is at the boundary of the region to be processed, there may be a risk that the sub-pixels of the sub-pixel search reference depend on the pixels of the region not to be processed.

It should be noted that interpolation is an imaging method that can increase the effective pixels when calculating the pixels. The interpolation pixel is to calculate the formed actual pixel according to a certain motion mode to generate a new pixel point, and insert the new pixel point into a gap adjacent to the original pixel, thereby achieving the purposes of increasing the total pixel quantity and increasing the pixel density.

Referring to fig. 9A, fig. 9A is a schematic diagram illustrating sub-pixel interpolation according to an embodiment of the application. As can be seen from fig. 9A, 1/2 pixel cc is calculated by the whole pixel E, F, G, H, I, J; 1/2 pixel ff is calculated from the integer pixel A, C, G, M, Q, S; 1/2 pixel gg is calculated by 1/2 pixel dd, ee, ff, hh, ii, jj; 1/4 pixel a is calculated from integer pixel G and 1/2 pixel cc; 1/4 pixel c is calculated from integer pixel G and 1/2 pixel ff; 1/4 pixel d is calculated from 1/2 pixel cc and 1/2 pixel ff; the 1/4 pixel f is calculated by 1/2 pixel gg and integer pixel H; 1/4 pixel k is calculated from 1/2 pixel ff and 1/2 pixel kk; 1/4 pixel m is calculated from 1/2 pixel kk and 1/2 pixel hh.

It can be seen that if the region pointed to by the motion vector of the target image block is at the boundary of the region to be processed, for example, points to 1/2 pixel ff, since 1/2 pixel ff is calculated by the integer pixel A, C, G, M, Q, S and the integer pixel a and the integer pixel C are not in the region to be processed, there is a risk that the sub-pixels of the sub-pixel search reference depend on the pixels of the region not to be processed.

Based on the above analysis, in order to solve the time domain dependency problem, so that the range of the reference area of the target image block is within the range of the area to be processed to which the target image block belongs, constraint processing needs to be performed on the target image block.

Referring to fig. 9B, fig. 9B is a schematic flow chart of constraint processing for a target image block according to an embodiment of the present application. As can be seen from fig. 9B, the encoder needs to determine whether the target image block is a skip type macroblock, and the encoder can calculate pixel residuals between the region indicated by the motion vector of the neighboring block and the target image block according to the motion vector of the neighboring block, respectively, if the pixel residuals are 0 under a certain motion vector, it is indicated that the target image block is a skip macroblock.

The encoder may determine whether the reference region is within the region to be processed according to the calculated reference region indicated by the motion vector and the calculated region to be processed.

If the pixel residual is 0 and the region indicated by the motion vector of the adjacent block corresponding to the pixel residual is in the region to be processed, the target image block can be used as skip type coding.

If the pixel residual is 0 and the region indicated by the motion vector of the adjacent block is not in the region to be processed, the encoder performs constraint processing on the target image block which is not used as a skip type, so that the motion vector of the target image block can be obtained.

As can be seen from fig. 9B, the constraint processing includes: integer pixel motion estimation and sub-pixel motion estimation.

The encoder firstly carries out integral pixel motion estimation on the target image block according to the motion vector corresponding to the adjacent block of the target image block as a starting point, and obtains an integral pixel motion searching end point. Then, the encoder may perform sub-pixel motion estimation on the target image block according to the whole-pixel motion search end point, to obtain a motion vector of the target image block.

1) Integer pixel motion estimation

The purpose of the whole-pixel motion estimation is to find the best match for the target image block within the range of the region to be processed, so that the range of the reference region of the target image block is within the range of the region to be processed.

The encoder may determine, according to the motion vectors corresponding to the neighboring blocks of the target image block, whether the reference area pointed by the motion vectors is within the to-be-processed area to which the target image block belongs.

Referring to fig. 10A, fig. 10A is a schematic diagram of adjacent blocks of different sizes according to an embodiment of the present application.

As can be seen from fig. 10A, the adjacent blocks (block a, block B, and block C) of the target image block E are all 16×16 pixels, which are the same size as the target image block E. The motion vector for the target image block E may be predicted using neighboring blocks A, B and C of the same block size.

As can be seen from (B) in fig. 10A, the adjacent blocks (block a, block B, and block C) of the target image block E are the same size as the target image block E, block a is 8×4 pixels, block B is 4×8 pixels, and block C is 16×8 pixels. The motion vector of the target image block E may be predicted by adjacent blocks A, B and C having different block sizes, and the motion vector of the target image block E may be taken as the median of the motion vectors of the blocks a, B and C during processing.

If the reference area indicated by the motion vector of the adjacent block of the target image block is in the to-be-processed area of the target image block, calculating the code rate distortion cost, and selecting the motion vector with the minimum code rate distortion cost as the whole pixel motion searching end point. It can be understood that there may be a plurality of reference areas indicated by the motion vectors in the to-be-processed area to which the target image block belongs, so there are also a plurality of calculated code rate distortion costs.

If the region indicated by the motion vector of the adjacent block of the target image block is not in the range of the region to be processed, the encoder can perform equal-proportion scaling on the operation vector corresponding to the adjacent block of the target image block according to the range information of the region to be processed and the position information of the target image block to obtain a scaled operation vector, wherein the region indicated by the scaled motion vector is in the range of the region to be processed.

Referring to fig. 10B, fig. 10B is a schematic diagram illustrating scaling of a motion vector according to an embodiment of the present application.

As can be seen from (a) in fig. 10B, the motion vector 1001a is the motion vector of the adjacent block a of the target image block, and it can be seen that when the encoder translates the motion vector 1001a to the upper left corner of the target image block, the motion vector 1002a of the target image block is obtained.

Whereas the range of the reference region 1003a indicated by the motion vector 1002a is not within the range of the region to be processed, as can be seen from (B) of fig. 10B, the encoder may scale the motion vectors of the adjacent blocks a of the target image block in equal proportion according to the range information of the region to be processed and the position information of the target image block, so that the reference region 1003B indicated by the scaled motion vector 1002B is within the range of the region to be processed.

It will be appreciated that there are a plurality of adjacent blocks of the target image block, and fig. 10B is an example of an adjacent block a, and the adjacent block B and the adjacent block C may be processed according to the same processing manner for the adjacent block a, so that a second code encoding of a plurality of scaled motion vectors may be obtained. Then, the encoder can determine the motion searching end point of the whole pixel according to the motion vector corresponding to the minimum second code rate distortion cost in the second code rate distortion costs, and store the motion vector corresponding to the minimum second code rate distortion cost in the second code rate distortion costs. Meanwhile, the encoder calculates the code rate distortion cost under the condition that the motion vector is (0, 0), compares the code rate distortion cost with the second code rate distortion cost of the motion vector obtained after scaling, and stores the motion vector with the minimum code rate distortion cost as a whole pixel motion searching end point.

In the case that the region indicated by the motion vector of the target image block is within the range of the region to be processed, the encoder may perform sub-pixel motion estimation on the target image block according to the whole-pixel motion search end point, to obtain the motion vector of the target image block.

2) Sub-pixel motion estimation

The sub-pixel motion estimation is to optimize the result of the whole pixel motion estimation on the accuracy of 1/2 pixel and 1/4 pixel to obtain the sub-pixel motion vector with smaller code rate distortion cost, thereby improving the coding efficiency. The purpose of adopting sub-pixel motion estimation in the application is to calculate the available area of the area referenced by the target image block when the area is near the boundary of the area to be processed according to the sub-pixel interpolation calculation relation, so as to obtain a more matched reference area. Thus, the encoder can determine whether the reference region is within the region to be processed according to the region to be referenced by the calculated target image block and the calculated region to be processed.

The encoder may calculate the sub-pixel reference area according to the sub-pixel interpolation calculation relationship, please refer to fig. 11A, fig. 11A is a schematic diagram of a sub-pixel motion estimation process provided in an embodiment of the present application, and it can be seen from fig. 11A that the encoder first performs 1/2 pixel interpolation on 8 1/2 pixel positions around a position (D point) corresponding to an optimal whole-pixel motion vector obtained by whole-pixel motion estimation, then calculates a rate distortion cost of the 8 1/2 pixel points around the a point obtained by interpolation and a rate distortion cost of the a point, and finds a 1/2 pixel point with the minimum rate distortion cost as a position (E point) of the optimal 1/2 pixel precision motion vector. Then, 1/4 pixel interpolation is carried out at 8 1/4 pixel positions around the position (E point) corresponding to the optimal 1/2 pixel precision motion vector, then the code rate distortion cost of 8 1/4 pixel points around the E point obtained by interpolation and the code rate distortion cost of the E point are calculated, and the 1/4 pixel point with the minimum code rate distortion cost is found to be used as the position (F point) of the optimal 1/4 pixel precision motion vector. And then comparing the optimal 1/2 pixel precision motion vector with the code rate distortion cost of the optimal 1/4 pixel precision motion vector, and selecting the optimal sub-pixel motion vector with the minimum code rate distortion cost. Thus, the sub-pixel reference area can be calculated according to fig. 11A.

Referring to fig. 11B, fig. 11B is a schematic diagram of a sub-pixel reference area according to an embodiment of the present application, and it can be seen from fig. 11B that the sub-pixel reference area is far from the boundary of the area to be processed after improving the motion vector accuracy. Therefore, when the region pointed by the motion vector of the target image block is at the sub-pixel reference region boundary, the sub-pixels for which the sub-pixel search reference does not exist depend on the region boundary to be processed

The encoder may determine whether a reference area indicated by the integer pixel motion vector obtained in the integer pixel motion estimation process is within a sub-pixel reference area, and if the reference area indicated by the integer pixel motion endpoint obtained in the integer pixel motion estimation process is not within the sub-pixel reference area, does not perform sub-pixel motion estimation, and stores the integer pixel motion vector.

If the region indicated by the motion vector corresponding to the whole pixel motion search end point obtained in the whole pixel motion estimation process is in the sub-pixel reference region, sub-pixel motion estimation is performed on the target image block according to the method shown in fig. 11A. In the sub-pixel motion estimation process, the motion vector obtained in each step is judged, and whether the area indicated by the motion vector obtained in each step is in the sub-pixel reference area is judged. Then, a motion vector of the region indicated by the motion vector within the sub-pixel reference region is taken as a sub-pixel motion vector. Next, a third rate-distortion cost for the sub-pixel motion vector is calculated. And finally, taking the sub-pixel motion vector corresponding to the minimum code rate distortion cost in the third code rate distortion cost as the motion vector of the target image block, and storing the motion vector.

Step S603, encoding the target image block according to the reference area.

Specifically, after motion estimation is completed, the encoder may obtain an encoding mode (including a motion vector pointing to a reference area, indication information of a reference frame where the reference area is located, etc.) of the minimum rate distortion cost between frames of the target image block, and then calculate the encoding mode using the target image block as the minimum rate distortion cost of the intra image block. And finally, selecting a coding mode with the minimum code rate distortion cost in the intra-frame coding and the inter-frame coding to carry out coding. Finally, the encoder may encode the encoding mode (including motion vectors pointing to the reference region, indication of the reference frame in which the reference region is located, etc.) and the segmentation information (e.g., encoding the image frame) into the bitstream, and send the encoded bitstream to the decoding end (e.g., destination device 42 shown in fig. 4A).

It should be noted that, because the temporal independence between the to-be-processed regions may cause the reduction of the referenceable region of the target image block, when the inter-region motion occurs, the inter-frame motion is difficult to find a matching reference region in the current to-be-processed region, and the code rate distortion cost is often greater than that of the intra-frame mode. Therefore, the encoder can select the intra mode with the minimum rate distortion cost for encoding, thereby ensuring the quality of video.

It should be noted that, in the fixed code rate scenario, the encoder can have better allocated code rate through the 2pass coding strategy, so as to ensure the quality of video.

Referring to fig. 12, fig. 12 is a schematic flow chart of video decoding according to an embodiment of the present application, in which an image block being decoded is referred to as a target image block, and an image in which the target image block is located is referred to as a current frame. Therefore, the video decoding method mentioned in the present application may be a decoding method of a series of images included in a video. The method includes, but is not limited to, the steps of:

step S1201: analyzing the code stream to obtain a reference area of the target image block in the reference frame;

specifically, after receiving the code stream from the encoder, the decoder decodes the code stream to obtain the reference region of the target image block in the reference frame.

The target image block is any one of one or more image blocks contained in a region to be processed, the region to be processed is obtained by dividing any image frame in an image sequence of an original video, and the region to be processed is a region needing information superposition. The reference frame is a frame that needs to be referred to when encoding an image frame, and the range of the reference area is within the range of the area to be processed to which the target image block belongs.

Step S1202: and predicting the target image block according to the reference area.

Specifically, since the division process divides the image frame into a plurality of processing regions, a region in the reference frame referred to when encoding an image block in a region to be processed (region requiring superimposition information) is within the range of the region to be processed to which the target image block belongs. Therefore, in the inter prediction process, the decoder does not need to decode other regions in the reference frame, and can predict the target image block in the case of decoding the reference region in the reference frame.

The application scenario of the embodiment of the present application is described below.

Aiming at high concurrency real-time transcoding under the video color ring service, multi-slice segmentation preprocessing can be performed on an original video in advance, and slices obtained after segmentation can be completely isolated. When playing the video color ring in real time, the slice to be decoded is calculated according to the superposition position on the video color ring, and as the slice segmentation is completely isolated, a few slices can be independently processed, and the approximate processing steps can include decoding, superposition, encoding and network abstraction layer unit (network abstraction layer unit, NALU) encapsulation and transmission processing. While other slices remain unchanged, the real-time computing overhead can be reduced, and the performance is improved.

Taking a typical color ring+advertisement service as an example, the part of the superposition information is smaller than 10% of the image, and the calculation overhead is 1/10 of the full decoding and full encoding assuming that transcoding superposition operation is performed on a slice region of 1/10.

Referring to fig. 13, fig. 13 is a schematic view of a scene of superimposing information on a video according to an embodiment of the present application. As can be seen from fig. 13, the size of the original video is 480×640, and the portion to be superimposed with information is 1/10 of the original video, that is, the size of the area of the superimposed information is 480×64. It can be understood that, because the area requiring the superimposed information is 1/10 of the original video, when any image frame in the image sequence of the original video is encoded according to the video encoding method shown in fig. 6, in the process of dividing the image frame, the area requiring the superimposed information can be divided to obtain the area to be processed, such as the area slice1 and the area slice2. The segmentation information for the video is then saved into supplemental enhancement information (Supplemental Enhancement Information, SEI) and transmitted into the h.264 bitstream. The h.264 code stream includes a slice1 code stream and first field information (first_mb_in_slice=1), a slice2 code stream and second field information (first_mb_in_slice=x), video partition information, and the like.

As can be seen from fig. 13, in the transcoding stage of the video color ring, the decoder can obtain the segmentation information of the video by decoding the h.264 code stream (the SEI information in the video), then specifies the slice to be decoded (for example, slice2 shown in fig. 13) to obtain the original YUV data about slice2 according to the service requirement (for example, which part of the video needs the superimposition information), because of the slice segmentation process, the slice2 can be decoded according to the video decoding method shown in fig. 12 without fully decoding all the slices on the reference frame, and other slices (for example, slice1 shown in fig. 13) that do not need the superimposition information are transmitted in the form of a network abstraction layer unit (network abstraction layer unit, NALU) code stream in a buffer.

The encoder multiplexes encoding information such as SPS and PPS from the code stream, and superimposes YUV data obtained by decoding slice2 as an input to the encoder.

At this time, slice division is performed in the encoder (the number of slices is 1), and the encoded image data is YUV data after superposition, so as to ensure that the type of the encoded time frame is consistent with the type of the frame to which the slice belongs in the original code stream. Meanwhile, the second field information (for example, first_mb_in_slice) in the NALU of the output slice type is modified so as to be consistent with the second field information (for example, first_mb_in_slice) of the NALU to which slice1 belongs in the original code stream. If first_mb_in_slice=x in the original code stream, the encoder outputs the encoded NALU, filters the NALU of SPS, PPS and SEI types, and replaces the NALU of the slice type with the NALU of the corresponding slice type in the original code stream.

According to the prior 2P20C server with a single CPU core, about 3 paths of full decoding and full encoding are supported, 30 paths can be achieved by using slice, and the overall performance can be up to 1200 paths. Real-time high concurrency transcoding independent of proprietary hardware is achieved, and support of a single server can be greater than 1000 paths of transcoding.

It can be seen that, based on the slice time domain independent method, local encoding and decoding of slices can be achieved, local decoding of slices can be normally performed without completely decoding all slices of a reference frame, and superposition information (text, pictures and video) can be achieved without encoding the whole frame, so that the aim of high concurrence and rapid local encoding and decoding is achieved, improvement is concentrated on an off-line processing end and an on-line processing end, and modification of client equipment software is not needed.

The method of the embodiment of the present application is described above, and the apparatus of the embodiment of the present application is provided below.

Fig. 14 is a schematic structural diagram of a decoding device according to an embodiment of the present application, which is a schematic block diagram of an implementation manner of an encoding device or a decoding device (abbreviated as a decoding device 140) according to an embodiment of the present application. The decoding device 140 may include a processor 1410, a memory 1430, and a bus system 1450. The processor is connected with the memory through the bus system, the memory is used for storing instructions, and the processor is used for executing the instructions stored by the memory. The memory of the encoding device stores program codes, and the processor may call the program codes stored in the memory to perform various video encoding or decoding methods described in the present application, particularly, a video encoding or decoding method in various new inter prediction modes, and a method of predicting motion information in various new inter prediction modes. To avoid repetition, a detailed description is not provided herein.

In an embodiment of the present application, the processor 1410 may be a central processing unit (Central Processing Unit, abbreviated as "CPU"), and the processor 1410 may also be other general purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 1430 may include a Read Only Memory (ROM) device or a Random Access Memory (RAM) device. Any other suitable type of storage device may also be used as memory 1430. Memory 1430 may include code and data 1431 that are accessed by processor 1410 using bus system 1450. The memory 1430 may further include an operating system 1433 and application programs 1435, the application programs 1435 including at least one program that allows the processor 1410 to perform the video encoding or decoding methods described herein (and in particular the encoding or decoding methods described herein). For example, the applications 1435 may include applications 1 through N, which further include video encoding or decoding applications (simply referred to as video coding applications) that perform the video encoding or decoding methods described in this disclosure.

The bus system 1450 may include a power bus, a control bus, a status signal bus, and the like in addition to the data bus. For clarity of illustration, the various buses are labeled as bus system 1450 in the drawing.

Optionally, the decoding device 140 may also include one or more output devices, such as a display 1470. In one example, the display 1470 may be a tactile display incorporating a display with a tactile unit operable to sense touch input. A display 1470 may be coupled to processor 1410 via a bus system 1450.

Fig. 15 is a schematic structural diagram of a video coding system according to an embodiment of the present application, which is an explanatory diagram of an example of a video coding system 1500 including the encoder 410 of fig. 4B and/or the decoder 420 of fig. 4B according to an exemplary embodiment. Video coding system 1500 may implement a combination of various techniques of this disclosure. In the illustrated embodiment, video coding system 1500 may include an imaging device 1501, a video encoder 410, a video decoder 420 (and/or a video encoder implemented via logic 1507 of a processing unit 1506), an antenna 1502, one or more processors 1503, one or more memories 1504, and/or a display device 1505.

As shown, imaging device 1501, antenna 1502, processing unit 1506, logic 1507, video encoder 410, video decoder 420, processor 1503, memory 1504, and/or display device 1505 can communicate with each other. As discussed, although video coding system 1500 is depicted with video encoder 410 and video decoder 420, in different examples, video coding system 1500 may include only video encoder 410 or only video decoder 420.

In some examples, as shown, video coding system 1500 may include an antenna 1502. For example, antenna 1502 may be used to transmit or receive an encoded bitstream of video data. Additionally, in some examples, video coding system 1500 may include a display device 1505. Display device 1505 may be used to present video data. In some examples, as shown, logic 1507 may be implemented by processing unit 1506. The processing unit 1506 may include application-specific integrated circuit (ASIC) logic, a graphics processor, a general purpose processor, or the like. The video coding system 1500 may also include an optional processor 1503, which optional processor 1503 may similarly comprise application-specific integrated circuit (ASIC) logic, a graphics processor, a general purpose processor, or the like. In some examples, the logic 1507 may be implemented in hardware, such as video encoding dedicated hardware, and the processor 1503 may be implemented in general-purpose software, an operating system, and the like. In addition, the memory 1504 may be any type of memory, such as volatile memory (e.g., static random access memory (Static Random Access Memory, SRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.), and the like. In a non-limiting example, the storage 1504 may be implemented by an overspeed cache. In some examples, logic 1507 may access memory 1504 (e.g., to implement an image buffer). In other examples, logic 1507 and/or processing unit 1506 may include memory (e.g., buffers, etc.) for implementing image buffers, etc.

In some examples, video encoder 410 implemented by logic circuitry may include an image buffer (e.g., implemented by processing unit 1506 or memory 1504) and a graphics processing unit (e.g., implemented by processing unit 1506). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include a video encoder 410 implemented by logic 1507 to implement the various modules discussed with reference to fig. 4B and/or any other encoder system or subsystem described herein. Logic circuitry may be used to perform various operations discussed herein.

The video decoder 420 may be implemented in a similar manner by logic 1507 to implement the various modules discussed with reference to the decoder 420 of fig. 4C and/or any other decoder system or subsystem described herein. In some examples, the video decoder 420 implemented by logic circuitry may include an image buffer (implemented by the processing unit 1506 or the memory 1504) and a graphics processing unit (e.g., implemented by the processing unit 1506). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include a video decoder 420 implemented by logic 1507 to implement the various modules discussed with reference to fig. 4C and/or any other decoder system or subsystem described herein.

In some examples, antenna 1502 of video coding system 1500 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frame, indicators, index values, mode selection data, etc., discussed herein, such as data related to the encoded partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and/or data defining the encoded partitions). The video coding system 1500 may also include a video decoder 420 coupled to the antenna 1502 and for decoding the encoded bitstream. Display device 1505 is used to present video frames.

In the steps of the above method flow, the description order of the steps does not represent the execution order of the steps, and it is possible to execute in the above description order, and it is also possible to execute in the above description order.

Those of skill in the art will appreciate that the functions described in connection with the various illustrative logical blocks, modules, and algorithm steps described in connection with the disclosure herein may be implemented as hardware, software, firmware, or any combination thereof. If implemented in software, the functions described by the various illustrative logical blocks, modules, and steps may be stored on a computer readable medium or transmitted as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media, such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementing the techniques described in this disclosure. The computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood that the computer-readable storage medium and data storage medium do not include connections, carrier waves, signals, or other transitory media, but are actually directed to non-transitory tangible storage media. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The instructions may be executed by one or more processors, such as one or more Digital Signal Processors (DSPs), general purpose microprocessors, application Specific Integrated Circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the term "processor" as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functions described by the various illustrative logical blocks, modules, and steps described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combination codec. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.

The embodiment of the application also provides a computer readable storage medium. The computer readable storage medium includes instructions for implementing the video encoding method described above, such as the video encoding method in the embodiment of fig. 6 or 13. And is also used to implement the video decoding method described above, such as the video decoding method in the embodiment of fig. 12 or fig. 13.

The computer readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), etc.

In embodiments of the application, words such as "exemplary" or "such as" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.

Reference to "at least one" in embodiments of the application means one or more, and "a plurality" means two or more. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a. b, c, (a and b), (a and c), (b and c), or (a and b and c), wherein a, b, c may be single or plural. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: three cases of A alone, A and B together, and B alone, wherein A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship.

And, unless otherwise indicated, the use of ordinal numbers such as "first," "second," etc., by embodiments of the present application is used for distinguishing between multiple objects and is not used for limiting a sequence, timing, priority, or importance of the multiple objects. For example, the first container storage management device and the second container storage management device are provided for convenience of description, and are not intended to represent differences in device structures, deployment orders, importance levels, etc. of the first container storage management device and the first container storage management device.

Those of ordinary skill in the art will appreciate that the techniques of the present application may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an Integrated Circuit (IC), or a set of ICs (e.g., a chipset). The various components, modules, or units are described in this disclosure in order to emphasize functional aspects of the devices for performing the disclosed techniques, but do not necessarily require realization by different hardware units. Indeed, as described above, the various units may be combined in a codec hardware unit in combination with suitable software and/or firmware, or provided by an interoperable hardware unit (including one or more processors as described above).

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; these modifications or substitutions do not depart from the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video encoding method, characterized in that the method comprises:

Segmenting the image frames in the image sequence of the original video to obtain a region to be processed, wherein the region to be processed includes one or more image blocks;

Determine a reference area of a target image block in a reference frame, wherein the reference frame is a frame that needs to be referenced when encoding the image frame, the target image block is one of the one or more image blocks, and an area occupied by the reference area is within an area occupied by an area to be processed to which the target image block belongs;

The target image block is encoded according to the reference area.

2. The method according to claim 1, wherein determining a reference area of the target image block in the reference frame comprises:

Calculating the area occupied by the area to be processed;

In the case where the area indicated by the motion vector of the target image block is not within the area occupied by the area to be processed, constraining the target image block to obtain a constrained motion vector of the target image block; wherein the area indicated by the constrained motion vector is within the area occupied by the area to be processed to which the target image block belongs;

A reference area of the target image block is determined from the reference frame according to the constrained motion vector.

3. The method according to claim 2, wherein the step of performing constraint processing on the target image block to obtain a constrained motion vector of the target image block comprises:

According to the motion vectors of the neighboring blocks of the target image block, respectively calculating the pixel residuals between the regions indicated by the motion vectors of the neighboring blocks and the target image block;

In the case where there is a motion vector with a pixel residual of 0, if the area indicated by the motion vector of the adjacent block corresponding to the pixel residual of 0 is not within the area occupied by the area to be processed, constraint processing is performed on the target image block to obtain the motion vector of the target image block.

4. The method according to claim 2 or 3 is characterized in that the constraint processing includes: integer pixel motion estimation and sub-pixel motion estimation.

5. The method according to claim 4, characterized in that the step of performing constraint processing on the target image block to obtain the motion vector of the target image block comprises:

Performing integer pixel motion estimation on the target image block according to motion vectors corresponding to adjacent blocks of the target image block to obtain an integer pixel motion search end point;

The sub-pixel motion estimation is performed on the target image block according to the integer-pixel motion search end point to obtain a motion vector of the target image block.

6. The method according to claim 5, characterized in that the step of performing the integer pixel motion estimation on the target image block according to the motion vectors corresponding to the adjacent blocks of the target image block to obtain the integer pixel motion search end point comprises:

If there is an area indicated by a motion vector of an adjacent block of the target image block within the area occupied by the to-be-processed area, calculating a first rate-distortion cost of the motion vector of the adjacent block;

An integer-pixel motion search end point is determined according to a motion vector corresponding to a minimum first rate-distortion cost among the first rate-distortion costs.

7. The method according to claim 5, characterized in that the step of performing integer pixel motion estimation on the target image block according to the motion vectors corresponding to the adjacent blocks of the target image block to obtain the integer pixel motion search end point comprises:

If the area indicated by the motion vector of the adjacent block of the target image block does not exist within the area occupied by the to-be-processed area, then proportionally scale the motion vector corresponding to the adjacent block of the target image block according to the boundary information of the to-be-processed area and the position information of the target image block to obtain a scaled motion vector, wherein the area indicated by the scaled motion vector is within the area occupied by the to-be-processed area;

Calculating a second rate-distortion cost of the scaled motion vector;

An integer-pixel motion search end point is determined according to a motion vector corresponding to a minimum second rate-distortion cost among the second rate-distortion costs.

8. The method according to any one of claims 5 to 7, characterized in that the step of performing the sub-pixel motion estimation on the target image block according to the integer-pixel motion search end point to obtain the motion vector of the target image block comprises:

Calculate the sub-pixel reference area according to the sub-pixel interpolation calculation relationship;

If the area indicated by the motion vector corresponding to the integer-pixel motion search end point is not within the area occupied by the sub-pixel reference area, the motion vector corresponding to the integer-pixel motion search end point is used as the motion vector of the target image block.

9. The method according to claim 8, wherein the step of performing the sub-pixel motion estimation on the target image block according to the integer-pixel motion search end point to obtain the motion vector of the target image block comprises:

If the area indicated by the motion vector corresponding to the integer-pixel motion search end point is within the area occupied by the sub-pixel reference area, sub-pixel motion estimation is performed on the target image block to obtain the motion vector of the target image block.

10. The method according to claim 9, wherein performing sub-pixel motion estimation on the target image block to obtain the motion vector of the target image block comprises:

Perform sub-pixel motion estimation on the target image block according to the integer-pixel motion search end point to determine a sub-pixel motion vector, wherein the area pointed to by the sub-pixel motion vector is within the area occupied by the sub-pixel reference area;

Calculating a third rate-distortion cost of the sub-pixel motion vector;

The sub-pixel motion vector corresponding to the smallest third rate-distortion cost among the third rate-distortion costs is used as the motion vector of the target image block.

11. The method according to any one of claims 2 to 10, characterized in that the step of calculating the area occupied by the area to be processed comprises:

Calculate the sequence number of the first image block and the sequence number of the last image block in the area to be processed;

Calculate the area information occupied by the area to be processed, the area information includes the number of rows of the area to be processed, and whether the first row and the last row of the area to be processed are full;

The area occupied by the area to be processed is calculated according to the sequence number of the first image block, the sequence number of the last image block and the area information.

12. A video decoding method, characterized in that the method comprises:

Parsing the bitstream to obtain a reference area of a target image block in a reference frame, wherein the target image block is any one of one or more image blocks contained in the area to be processed, the area to be processed is obtained by segmenting any image frame in an image sequence of an original video, the area to be processed is an area of superimposed information, the reference frame is a frame that needs to be referenced when encoding the image frame, and the area occupied by the reference area is within the area occupied by the area to be processed to which the target image block belongs;

The target image block is predicted according to the reference area.

13. The method according to claim 12, characterized in that after decoding the target image block according to the reference area, it further comprises:

Copying is performed on the code stream in the area not subjected to decoding processing, wherein the area not subjected to decoding processing is an area in the image frame other than an area occupied by the area to be processed.

14. An encoder, characterized in that the encoder comprises a segmentation unit, a prediction unit and an entropy coding unit, wherein:

The segmentation unit is used to segment the image frames in the image sequence of the original video to obtain a region to be processed, wherein the region to be processed includes one or more image blocks;

The prediction unit is further used to determine a reference area of the target image block in a reference frame, wherein the reference frame is a frame that needs to be referenced when encoding the image frame, the target image block is one of the one or more image blocks, and the area occupied by the reference area is within the area occupied by the to-be-processed area to which the target image block belongs;

The entropy coding unit is used to encode the target image block according to the reference area.

15. A decoder, characterized in that the decoder comprises an entropy decoding unit and an inter-frame prediction unit, wherein:

The entropy decoding unit is used to parse the code stream to obtain a reference area of the target image block in the reference frame, wherein the target image block is any one of the one or more image blocks contained in the area to be processed, the area to be processed is obtained by segmenting any image frame in the image sequence of the original video, the area to be processed is an area of superimposed information, the reference frame is a frame that needs to be referenced when encoding the image frame, and the area occupied by the reference area is within the area occupied by the area to be processed to which the target image block belongs;

The inter-frame prediction unit is used to predict the target image block according to the reference area.

16. An encoder, characterized in that the encoder comprises a processor and a memory:

The processor is configured to execute instructions stored in the memory so that the encoder implements the method according to any one of claims 1 to 11.

17. A decoder, characterized in that the decoder comprises a processor and a memory:

The processor is configured to execute instructions stored in the memory so that the decoder implements the method according to any one of claims 12 or 13.

18. A computer-readable storage medium, characterized in that it comprises computer program instructions, and when the computer program instructions are executed by a computing device, the computing device implements the method according to any one of claims 1 to 13.

19. A computer program product comprising instructions, wherein when the instructions are executed by a computing device, the computing device is enabled to implement the method according to any one of claims 1 to 13.