Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
  • H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/117—Filters, e.g. for pre-processing or post-processing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/136—Incoming video signal characteristics or properties
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/167—Position within a video image, e.g. region of interest [ROI]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel

Definitions

• At least one of the present embodiments generally relates to a method or an apparatus for video encoding or decoding, compression, or decompression.
• BACKGROUND To achieve high compression efficiency, image and video coding schemes usually employ prediction, including motion vector prediction, and transform to leverage spatial and temporal redundancy in the video content.
• At least one of the present embodiments generally relates to a method or an apparatus for video encoding or decoding, and more particularly, to a method or an apparatus for improving the coding efficiency through adaptive in-loop filtering. According to a first aspect, there is provided a method.
• the method comprises steps for determining at least one in-loop filter for a video block based on a classification; filtering a reconstructed version of said video block with said determined at least one in-loop filter; signaling said determined at least one in-loop filter; and, encoding the video block using the filtered reconstructed video block
• the method comprises steps for deriving a classification for a video block; determining at least one in-loop filter for the video block based on the classification; filtering a reconstructed version of said video block with said determined at least one in-loop filter; and, decoding the video block using the filtered reconstructed video block.
• an apparatus comprises a processor.
• the processor can be configured to encode a block of a video or decode data by executing any of the aforementioned methods.
• a device comprising an apparatus according to any of the decoding embodiments; and at least one of (i) an antenna configured to receive a signal, the signal including the video block, (ii) a band limiter configured to limit the received signal to a band of frequencies that includes the video block, or (iii) a display configured to display an output representative of a video block.
• a non-transitory computer readable medium containing data content generated according to any of the described encoding embodiments or variants.
• a signal comprising video data generated according to any of the described encoding embodiments or variants.
• a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variants.
• a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the described decoding embodiments or variants.
• Figure 1 illustrates horizontal and vertical block boundaries on an 8x8 grid.
• Figure 2 illustrates an overview of HEVC deblocking decision workflow.
• Figure 3 illustrates sample position of pi,k and qi,k in the case of horizontal and vertical block boundaries.
• Figure 4 illustrates decisions for HEVC normal deblocking mode.
• Figure 5 illustrates four-sample long vertical boundary segment formed by blocks P and Q with VVC deblocking decisions based on lines 0 and 3.
• Figure 6 illustrates vertical and horizontal block boundaries on the 4x4 grid, 32x32 CU with Pus on 8x8 grid, and vertical boundary that may require long-tap deblocking.
• Figure 7 illustrates deblocking filtering (DBFV and DBFH) combined with SAOH and SAOV.
• DBFV and DBFH deblocking filtering
• Figure 8 illustrates ALF filter shapes: left: 5x5 diamond for chroma; right: 7x7 diamond for luma.
• Figure 9 illustrates subsampled Laplacian calculation.
• Figure 10 illustrates ALF process in ECM-3; F0 and F1 are pre-learned filters trained on classifiers C0 and C1, respectively; F2 signaled and uses either classifier C0 or C1.
• Figure 11 illustrates filter shape of signaled filter F2 in ECM-7.0.
• Figure 12 illustrates regular block grid (gray dotted lines) and shifted deblocking grid (solid black line).
• Figure 13 illustrates deblocking range examples (8x8 grid), where only gray samples are filter; samples in white are excluded from deblocking process.
• Figure 14 illustrates examples of spatial classification in the case of a 4x4 grid (left) and of an 8x8 grid (right).
• Figure 15 illustrates symmetries in a quadrant of the deblocking grid.
• Figure 16 illustrates examples of filter shapes (cross, diamond and square) of different sizes, where current to-be-filter sample is depicted in dark gray.
• Figure 17 illustrates examples of off-centered deblocking filter shapes for the top-left quadrant where current to-be-filtered sample is depicted in dark gray.
• Figure 18 illustrates two examples of spatially dependent filter sets (8x8 grid); (a) set of 16 square filters for top-left quadrant; (b) set of 16 cross-shaped filters for top-left quadrant.
• Figure 19 illustrates examples of 1D filter shapes; top row: Horizontal filters ( ⁇ ⁇ ⁇ ⁇ 0 and ⁇ ⁇ ⁇ ⁇ 0); Bottom row: Vertical filters ( ⁇ ⁇ ⁇ ⁇ ⁇ 0 and ⁇ ⁇ ⁇ ⁇ 0).
• Figure 20 illustrates one embodiment where modified ALF substitutes to legacy DBF.
• Figure 21 illustrates one embodiment of local deblocking control at encoder side.
• Figure 22 illustrates an example class merging algorithm.
• Figure 23 illustrates an embodiment of a first method under the described aspects.
• Figure 24 illustrates an embodiment of a second method under the described aspects.
• Figure 25 illustrates one embodiment of an apparatus under the described aspects.
• Figure 26 illustrates a standard, generic, video compression scheme.
• Figure 27 illustrates a standard, generic, video decompression scheme.
• FIG 28 illustrates a processor-based system for encoding/decoding under the general described aspects.
• DETAILED DESCRIPTION In-loop filters (265) are applied to the reconstructed picture in an encoder or a decoder to perform, for example, deblocking/SAO (Sample Adaptive Offset)/ALF (Adaptive Loop Filter) filtering to reduce encoding artifacts.
• the described embodiments relate to in-loop filters, corresponding to steps 265 at encoder side in Figure 26 and 365 at decoder side in Figure 27. In these embodiments, it is proposed to derive optimal deblocking filters in terms of mean squared error (MSE).
• MSE mean squared error
• optimal deblocking filters are pre-learned offline on a training dataset and hard-coded both in encoder and decoder.
• deblocking filters are derived at encoding time based on current tile/slice/frame statistics and signaled in the bitstream.
• a block-based video codec such as HEVC and VVC
• discontinuities can appear at the transform and prediction block boundaries. These discontinuities can lead to visual artifacts that are referred to as blocking artifacts. Removing or lessening these artifacts is referred to as deblocking, for instance using deblocking filters (DBF).
• a deblocking filter aims to achieve a smooth transition across the block boundaries while avoiding removal of natural edges.
• DBFs in HEVC and VVC are in-loop filters, that is, filters implemented within the reconstruction loop both at encoder and decoder sides.
• DBFs are driven by signal characteristics across block boundaries, together with coding information such as partitioning and prediction modes in such a way that no information/parameter is signaled.
• Adaptive Loop Filtering is another in-loop filter that aims to enhance reconstructed pictures with respect to original pictures in the sense of MSE.
• ALF filters are optimized based on current tile/slice/picture statistics, then signaled in the bitstream.
• DBF Deblocking filters
• block-based coding creates discontinuities at block boundaries. Because excessive filtering may lead to unnecessary smoothing of the picture details, whereas lack of filtering may leave noticeable blocking artifacts, prior art addresses mainly how to decide whether to filter a particular block boundary or not, and how to decide on the filtering strength to be applied.
• Deblocking is performed on a four-sample part of a block boundary when all of the following criteria are met: the block boundary is a PU or TU boundary, the boundary strength Bs (see definition in next subsection) is greater than zero, and variation of signal on both sides of the boundary is below a specified threshold. When certain additional conditions hold, a strong filter is applied instead of the normal deblocking filter.
• Figure 2 The whole decision workflow is depicted in Figure 2. The following subsections provide more detailed explanations on the different conditions tested, as well as specific deblocking implementations in the different modes.
• Boundary strength (Bs) and Edge-Level adaptability The boundary strength (Bs) can take one of the following three possible values: 0, 1, and 2.
• the definition of Bs is shown in Table 1.
• Bs At least one of the blocks is intra 2
• At least one of the blocks has non-zero coded residual coefficient 1 and boundary is a transform boundary
• Motion-compensated prediction for the two blocks refers to 1 different reference pictures or the number of motion vectors is different for the two blocks Otherwise 0
• Table 1 Definition of Bs values for the boundary between two neighboring Luma blocks. For Luma, only block boundaries with Bs values equal to 1 or 2 are filtered.
• Equation (1) evaluates how much signal on both sides of the block boundary deviates from a straight line (constant level or ramp). Only the first and fourth rows/columns of the block boundary are evaluated to reduce complexity.
• Normal and strong deblocking modes For block boundaries with an associate Bs greater than zero and for which (1) holds, deblocking filtering is performed. There are two deblocking modes in HEVC, namely the normal deblocking mode and the strong deblocking mode. Whether to apply strong or normal deblocking is also determined based on local signal characteristics, evaluated on the first and fourth rows/columns (respectively ⁇ ⁇ 0 and ⁇ ⁇ 3) of the four-sample length block boundary. ⁇ ⁇ , ⁇ ⁇ 2 ⁇ , ⁇ ⁇ ⁇ , ⁇ ⁇ 2 ⁇ , ⁇ ⁇ ⁇ , ⁇ ⁇ 8 (2)
• the threshold parameter ⁇ ⁇ also depends on QP as another monotonically non- decreasing function.
• Condition (2) checks that there is a low spatial activity on the side of block boundary (like (1) but using a lower threshold).
• Condition (3) checks that the signal on the block boundaries is flat.
• Condition (4) checks that the difference in intensities of samples on two sides of the block boundary does not exceed the threshold. Further decisions in normal deblocking mode Normal filtering has two modes differing in the number of pixels being modified on each side of the block boundary.
• the offset value ⁇ ⁇ is zero if the signal across the block boundary forms a ramp. Furthermore, the deblocking filtering is applied to the row or column of samples across the block boundary if and only if the following expression holds:
• the modified value ⁇ ⁇ ⁇ in each row/column across the block boundary is obtained by: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ (11)
• ⁇ ⁇ ⁇ is calculated as: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ (12) where the offset values ⁇ ⁇ and ⁇ ⁇ are obtained by clipping the corresponding ⁇ ⁇ and ⁇ ⁇ , which are calculated as: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 1 ⁇ ⁇ 1 ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 1 (13)
• the offset values ⁇ ⁇ , ⁇ ⁇ , and ⁇ ⁇ are added to pixels ⁇ ⁇ , ⁇ ⁇ , and ⁇ ⁇ , respectively, after clipping of the following ⁇ ⁇ , ⁇ ⁇ , and ⁇ ⁇ values: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 2 ⁇ ⁇ ⁇ 6 ⁇ ⁇ ⁇ 2 ⁇ ⁇ ⁇ ⁇ ⁇ 4 ⁇ ⁇ 3 (15) ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 2 ⁇ ⁇ 2 (16) ⁇ ⁇ ⁇ ⁇ 2 ⁇ ⁇ ⁇ 5 ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 4 ⁇ ⁇ 3 (17)
• the offset values for modifications of pixels ⁇ ⁇ ⁇ , ⁇ ⁇ , and ⁇ ⁇ ⁇ are added to pixels ⁇ ⁇ , ⁇ ⁇ , and ⁇ ⁇ , respectively, after clipping of the following ⁇ ⁇ ,
• Chroma deblocking As mentioned previously in discussing boundary strength, chroma deblocking is only performed when Bs is equal to two. In this case, no further deblocking decisions are done. Only pixels ⁇ ⁇ and ⁇ ⁇ are modified as in (7) and (8). The deblocking is performed with the ⁇ ⁇ value, which is obtained by clipping the following ⁇ ⁇ offset value: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 2 ⁇ ⁇ ⁇ ⁇ ⁇ 4 ⁇ ⁇ 3 (18) DBF in VVC
• the VVC DBF is largely based on the HEVC design. VVC essentially allows larger blocks than HEVC.
• the DBF in VVC basically extends HEVC DBF design to address the artifacts remaining in large smooth areas. Main contributions are briefly described below. The reader can refer to [JVET-T2002] for further details.
• Luma deblocking Due to flexible block sizes in VVC, the luma deblocking is applied on a 4x4 sample grid for boundaries between CUs and TUs, and on an 8x8 grid for boundaries between PUs inside CUs, as shown in Figure 6. Filtering decisions Before deriving Bs and performing filtering decisions based on spatial activity, VVC DBF decides whether to use long-tap filters and determine appropriate filter lengths. This new step is carried out to ensure that non spatial dependency exists with the adjacent vertical or horizontal block boundaries.
• the deblocking length is defined as the number of samples to be filtered for each line for adjacent blocks P and Q (see Figure 5).
• the deblocking length of P and Q is denoted as ⁇ ⁇ and ⁇ ⁇ , respectively.
• the number of samples used for filtering decisions and filtering operations are ⁇ ⁇ ⁇ ⁇ 1 ⁇ and ⁇ ⁇ ⁇ ⁇ 1 ⁇ .
• the value of ⁇ ⁇ and ⁇ ⁇ depends on the size of the block side orthogonal to block boundary for block P and Q, respectively. They can be set to 7, 5, 3 and 1 for CU/TU boundaries, and to 3, 2, and 1 for PU boundaries inside a CU. More details are to be found in [JVET-T2002] for the specific conditions at stake.
• the boundary strength bS is derived.
• bS derivation is modified to accommodate new VVC coding tools, such as block-level differential pulse code modulation (BDPCM), combined intra-inter prediction (CIIP), intra block copy (IBC), high motion vector accuracy, etc.
• BDPCM block-level differential pulse code modulation
• CIIP combined intra-inter prediction
• IBC intra block copy
• Specific bS adaptations are to be found in [JVET-T2002].
• decisions based on spatial activity are made for the cases of non-zero bS. If the deblocking filter length is larger than 3 on at least one side, the spatial activity decision for the long-tap deblocking filter applies. Otherwise, the decision process for short-tap deblocking filter applies.
• the short-tap deblocking filters are almost identical to the HEVC deblocking filters, the only difference consisting in the introduction of position-dependent clipping to control the differences between filtered values and the sample values before filtering.
• Non-standardized deblocking solutions Deblocking filter using adaptive weighting factors [JCTVC-H0148]
• JCTVC-H0148 Panasonic proposed to modify the deblocking equation (11) in the Luma case with the introduction of a weighting factor ⁇ : ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ (19)
• the optimal weighting factor ⁇ is determined on a slice basis and signaled on 6 bits in the slice header. In their implementation, the value of ⁇ is determined by minimizing the MSE for a subset of samples that are heuristically identified as subjectively important.
• HHI proposed an additional in-loop filter which operated jointly with DBF.
• This new filter adjusts Luma samples values after the application of DBF, by adding an offset similarly to Sample-Adaptive Offset (SAO).
• SAO Sample-Adaptive Offset
• SAOV two SAO-like filtering SAOV and SAOH are applied for vertical and horizontal DBF, respectively (see Figure 7).
• SAOV involves a classification based on sample-wise difference between sample value before and after DBFV (respectively DBFH) that derives from a predetermined threshold ⁇ .
• ALF Adaptive loop filter
• VVC Adaptive loop filter
• ALF is based on adaptive filters, which are typically applied to reduce the MSE between the original and the reconstructed samples using Wiener-based filtering.
• ALF is the last post-filter applied to the output of SAO.
• ALF in VVC A thorough description of VVC ALF can be found in [Karczewicz2021]. The sequel is largely based on that article to recap ALF main features.
• ALF includes a classification of non-overlapping 4x4 blocks based on their local sample gradients. For each class, a specific filter is applied among the different filters signaled in the bitstreams.
• filter shape In VVC, two filter shapes, 7 ⁇ 7 diamond shape and 5 ⁇ 5 diamond shape, are used for Luma and Chroma components, respectively, as shown in Figure 8. Each square corresponds to a Luma or a Chroma sample, and the center square denotes the current to-be-filtered sample.
• the filter coefficients use point-symmetry. Each integer filter coefficient ⁇ ⁇ is represented with 7-bit fractional precision. In addition, to preserve DC neutrality, the sum of coefficients of one filter must be equal to 128 ⁇ 1 ⁇ 7.
• Equation (20) the number of coefficients ⁇ is equal to 13 and 7 for 7x7 and 5x5 filter shape, respectively.
• Linear filtering Because of point-symmetry and DC neutrality, a filtered sample value ⁇ ⁇ ⁇ ⁇ , ⁇ is obtained by adding to the reconstructed sample ⁇ ⁇ , ⁇ a weighted sum of the differences between the to-be-filtered sample ⁇ ⁇ , ⁇ and its neighboring samples: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 64 ⁇ ⁇ 7 (21) reconstructed samples corresponding to the ⁇ -th coefficient ⁇ ⁇ (point symmetry).
• Equation (21) is modified as follows: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 64 ⁇ ⁇ 7 ⁇ ⁇ ⁇ ⁇ ⁇ min ⁇ ⁇ ⁇ , max ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ min ⁇ , max ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ (24) ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ Where ⁇ ⁇ is the clipping parameter for the coefficient ⁇ ⁇ , that is determined by a clipping index ⁇ ⁇ .
• the clipping parameter ⁇ ⁇ is derived as follows: ⁇ ⁇ ⁇ ⁇ ⁇ 2 ⁇ when ⁇ ⁇ 0 2 ⁇ otherwise (25) Where ⁇ ⁇ denotes the sample bit depth, and ⁇ ⁇ can be 0, 1, 2 or 3. Luma sub-block classification Filter adaptation at sub-block level only applies to Luma. Each 4 ⁇ 4 block is classified based on its directionality and Laplacian activity.
• Equations (27) the sum of sample gradients within a 10x10 Luma window that covers the target 4x4 block is used for classifying that block. To reduce the complexity, only gradients of every second sample in the 10x10 window is calculated as illustrated in Figure 9. The values of other sample gradients are set to 0.
• the ratio of the maximum and minimum of the sub-block horizontal and vertical gradients ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ max ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ min ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ (28) and the ratio of the maximum and the minimum of two sub-block diagonal gradients ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ max ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ min ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ (29) are compared against each other and with a set of thresholds ⁇ ⁇ and ⁇ ⁇ : Step 1.
• Step 2 If both ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ and ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ , ⁇ is set to 0 (block is categorized as Step 2. If ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , the directionality ⁇ is calculated in Step 3; otherwise in Step 4. Step 3.
• Step 4 If ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ , ⁇ is set to 2 (block is categorized as “strong horizontal/vertical”); otherwise ⁇ is set to 1 (block is categorized as “weak horizontal/vertical”). Step 4 If ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ , ⁇ is set to 4 (block is categorized as “strong diagonal”); otherwise ⁇ is set to 3 (block is categorized as “weak diagonal”).
• each 4x4 Luma block is categorized into one of the 25 classes: ⁇ ⁇ ⁇ 5 ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ (31) Each class can have its own filter assigned. Note that no classification method is applied for Chroma. Geometric transformations Before filtering each 4 ⁇ 4 luma block, a geometric transformation such as rotation or diagonal and vertical flipping is applied to the filter coefficients, depending on the sub-block gradient values as specified in Table 2. The goal is to align the directionally of the different blocks, to reduce the number of ALF classes.
• a Luma CTB can use a filter set calculated for the current slice or one of the filter sets calculated for the already coded slices. It can also use one of the 16 offline trained filter sets. Within each Luma CTB, which filter from the chosen filter set should be applied to each 4x4 block, is determined by the class ⁇ calculated in Equation (31) for this block. Chroma uses only CTB-level filter adaptation. Up to 8 filters can be used for Chroma components in a slice. Each CTB select one these filters. ALF syntax in VVC Filter coefficients and clipping indices are carried in ALF Adaptation Parameter Sets (APS). An ALF APS can include up to 8 Chroma filers and one Luma filter set with up to 25 filters.
• An index ⁇ ⁇ is also included for each of the 25 Luma classes. Classes having the same index ⁇ ⁇ share the same filter. By merging different classes, the number of bits required to represent the filter coefficients is reduced.
• the absolute value of a filter coefficient is represented using a 0 th order Exp-Golomb code followed by a sign bit for non-zero coefficients.
• a clipping index is also signaled for each filter coefficient using a two-bit fixed length code.
• Filter control syntax elements include two types of information. First, ALF On/Off flags are signaled at sequence, picture, slice and CTB levels. Chroma ALF can be enabled at picture and slice level only if Luma ALF is enabled at the corresponding level.
• filter usage information is signaled at picture, slice and CTB level, if ALF is enables at that level.
• Referenced ALF APS IDs are coded at a slice level or at picture level if all the slices within the picture use the same APSs.
• Luma component can reference up to seven ALF APSs and Chroma components can reference one ALF APS.
• an index is signaled indicating which ALF APS or offline trained Luma filter set is used.
• the index indicates which filter in the referenced APS is used.
• the encoder calculates values of ALF syntax (see previous subsection) by minimizing a rate-distortion cost, which is a weighted sum of the distortion, measured as the square error between the original samples and the sample after applying the ALF filter, and the number of bits required to transmit ALF syntax elements.
• Filter coefficients are calculated by solving Wiener-Hopf equations.
• the statistics required to calculate filter coefficients and estimate distortion are collected for all possible combinations of clipping indices. The statistics are collected separately for each CTB, and in case of Luma component, for each class of each CTB.
• a new filter set for Luma component based on this picture’s statistics denoted as ⁇ ⁇ , ⁇ , is obtained as follows: 1) The encoder derives the filter set ⁇ ⁇ , ⁇ by merging statistics of the CTBs for which ALF is enabled. At first iteration, it is assumed that ALF is enabled for all CTBs. 2) Whether to apply ALF filter is determined for each CTB based on the rate-distortion cost calculated using the derives filters and the statistics of the CTB. In VTM-9.0, steps 1) and 2) are repeated four times. Optimizing a Luma filter set When designing a Luma filter set in Step 1), the encoder first calculates a filter for each of the 25 Luma classes.
• a merging algorithm is applied to these 25 filters.
• the algorithm reduces the number of filters by one.
• the encoder redesigns a filter by merging two filters and the corresponding statistics. Using the redesigned filter, the distortion is then estimated. The encoder merges the pair with the smallest distortion. Twenty-five (25) filter sets are obtained, the first set having 25 filters, the second one 24 filters, and so on until the 25 th one contains a single filter. Eventually, the set which minimizes a rate- distortion cost, including bits required to code the filter coefficients, is selected.
• Luma filter optimization When designing a filter, the clipping indices and the ⁇ ⁇ ⁇ 1 ⁇ filter coefficients are calculated iteratively until there is no decrease of the square error. In each iteration, the values of the clipping indices are updated one by one, starting with ⁇ ⁇ and continuing till index ⁇ ⁇ is reached. When updating the index, up to three options are tested: keeping its value unchanged, incrementing by one or decrementing by one. The filter coefficients and the approximate distortion are calculated for these 3 values and the value which minimizes square error is selected. At first iteration, the values of clipping indices are initialized to 2. When merging two filters, the value of ⁇ ⁇ is set to the average of the corresponding clipping indices for these filters.
• Luma component can also use the offline trained filter sets and the Luma filter sets carried in available ALF APSs.
• the encoder By accessing APSs, from the most recently signaled to the last one, the encoder obtains up to 7 Luma filter sets, which are denoted as ⁇ ⁇ , ⁇ , where ⁇ ⁇ ⁇ 0; ⁇ ⁇ ⁇ 1 ⁇ .
• the final values of the Luma ALF syntax element are obtained by selecting the combination of filter sets ⁇ ⁇ , ⁇ , where ⁇ ⁇ ⁇ 0; 1 ⁇ and ⁇ ⁇ ⁇ 0; ⁇ ⁇ ⁇ among: a. 16 offline trained filter sets; b. ⁇ ⁇ , ⁇ , if ⁇ ⁇ 1; c.
• Chroma filter set ⁇ ⁇ , ⁇ is calculated. Eight (8) Chroma filter sets are derived, denoted as ⁇ ⁇ ⁇ , ⁇ where ⁇ ⁇ ⁇ 1; 8 ⁇ . Filter set ⁇ ⁇ ⁇ , ⁇ contains ⁇ filters. The filter set ⁇ ⁇ ⁇ , ⁇ is obtained as follows: 1) The current picture is uniformly partitioned into ⁇ regions. For each region, a Chroma filter is calculated by merging the statistics of all CTBs in this region.
• Step 2 For each Chroma CTB in the picture, ALF On/Off flag and the filter index from set ⁇ ⁇ ⁇ , ⁇ is determined as to minimize the estimated rate-distortion cost. 3) Each filter in ⁇ ⁇ ⁇ , ⁇ is re-designed, by merging statistics of Chroma CTBs for which the current to-be-redesigned filter was selected in Step 2). 4) Steps 2) and 3) are repeated ⁇ ⁇ ⁇ 1 ⁇ times. The filter set ⁇ ⁇ ⁇ , ⁇ that minimizes rate-distortion cost is selected as Chroma filter set ⁇ ⁇ , ⁇ .
• the encoder selects ⁇ ⁇ , ⁇ or one of the Chroma filter sets from the available APSs to be used for the Chroma components.
• ⁇ ⁇ ⁇ , ⁇ ⁇ and ⁇ ⁇ ⁇ , ⁇ ⁇ denote the intermediate sample values filtered by F0 and F1.
• the filter F2 is applied to ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ , ⁇ ⁇ ⁇ ⁇ , ⁇ , and the neighboring samples in order to derive a final filtered sample as: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ , and ⁇ denotes the clipped difference between ⁇ ⁇ ⁇ , ⁇ ⁇ and current sample value.
• the Figure 10 recaps the new process.
• T he filter coefficients ⁇ ⁇ ⁇ ⁇ are signaled.
• Latest developments Latest adoptions in ECM include changes in filter shapes and sizes, and taking into account samples before deblocking in the signaled filter, as depicted in Figure 11.
• Regarding contributions most involve further diversification of signaled filter input, with for instance collocated samples in reference pictures, pre-DBF samples once filtered by a fixed filter, or residual samples.
• a further classifier based on residue L1 has been proposed.
• Adaptive Wiener deblocking filters The described embodiments comprise using deblocking filters that depend on a novel local classification, where the deblocking filters may be adaptively selected and signaled in the bitstream for the different classes.
• the corresponding plurality of deblocking filters which is referred to as a deblocking filter set, is either signaled in the bitstream, or pre-learned and known both at encoder and decoder sides.
• a deblocking filter set is either signaled in the bitstream, or pre-learned and known both at encoder and decoder sides.
• the DBF activation and filters usage can be controlled at a lower level than the slice level, for instance at CTU level.
• Deblocking is performed along borders of Coding Units (CUs), Prediction Units (PUs) and Transform Units (TUs).
• the deblocking classification is sample-wise, and takes into account both the distance from the block boundaries and specific features of those boundaries.
• the corresponding deblocking filter sets are signaled in the bitstream, e.g., at tile, slice or picture level.
• signaled deblocking filter sets are determined at encoder side by minimizing a distortion error (for instance, the squared error, or the sum of absolute difference) between reconstructed samples and original values across block boundaries.
• a distortion error for instance, the squared error, or the sum of absolute difference
• deblocking filters are signaled in the bitstream and that they are derived from the pictures content.
• ALF lies in the classification used.
• the proposed solution may improve the signal fidelity of the filtered signal by integrating in the derivation of the deblocking filters constraints to remain close to the original signal, while current DBF only mostly targets visual reduction of blocking artefacts.
• Embodiment on combining ALF and DBF steps into one step the DBF process is removed, and the ALF process is enhanced by new classes and related filters specialized for reducing blocking artefacts. This advantageously simplifies the process design and may reduce the overall number of processing steps (the samples previously processed by DBF then ALF will now be only processed once by ALF).
• Block boundaries Deblocking is performed across block boundaries on borders of Coding Units, Prediction Units (or Prediction Blocks), and Transform Units (or Transform Blocks)
• the determination of block boundaries is made through scanning all CUs per CTU, for all CTUs.
• the current slice/picture is first partitioned into a regular block grid for convenience, then block boundaries of that first grid are parsed, looking for CU, PU and TU borders. The resulting deblocking grid is shifted from the first grid with vertical and horizontal block boundaries lying in the middle of each shifted block (see Figure 12).
• shifted blocks located on slice/pictures borders present a rectangular shape ⁇ ⁇ 2 ⁇ or 2 ⁇ ⁇ ⁇ .
• the described aspects are not limited on a specific grid size, such as 8x8, 4x4 or 2x2. Spatial range
• every sample can be deblocked.
• some samples are a priori excluded from the deblocking process depending on their position on the deblocking grid (see Figure 13).
• Classification The deblocking process derives from a local classification. In one variant, that classification is sample-wise. In another variant, the classification is block-based. In this variant, the classification may be achieved using the same block-size as used by ALF.
• Deblocking filter sets consist in a plurality of deblocking filters, and samples are processed with a specific filter depending on their class, similarly to ALF.
• class/filter merging is considered during deblocking filter optimization at encoder side.
• Class merging implies that the corresponding filters present the exact same shape and coefficients location (more details in next section).
• the described aspects are not limited on the classification used. For instance, it can rely on local signal features such as gradients, Laplacian activity, and/or Luma/Chroma bands.
• the deblocking classification depends at least partially on the distance from the sample to the closest horizontal and vertical block boundaries. Let ⁇ ⁇ and ⁇ ⁇ denote the distances to the closest horizontal and vertical block boundaries, respectively.
• the spatiality index ⁇ is defined as: ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ min ⁇ where Figure 14 depicts examples of such spatial classification for 4x4 and 8x8 grid sizes. Spatial classification exhibits horizontal, vertical, and diagonal symmetries. In the embodiment related to spatial range, where some specific spatial locations are excluded from the deblocking process), some of the higher spatiality indices are discarded, and the resulting number of spatial classes is reduced. Boundaries features In another embodiment, the deblocking classification depends locally on the characteristics of closest horizontal and vertical boundaries. Definition That dependency involves especially the knowledge of the tile/slice/picture partitioning and coding, e.g.: a.
• the deblocking classification also involves criteria that characterize signal variations across block boundaries. Such criteria measure the signal activity, based for instance on gradients or Laplacian operators.
• Boundary features index In an embodiment, a horizontal boundary features index ⁇ ⁇ ⁇ and a vertical boundary features index ⁇ ⁇ ⁇ are defined, and a boundary features index ⁇ ⁇ depends on both: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ ⁇ Horizontal and vertical boundary features indices ⁇ ⁇ ⁇ and ⁇ ⁇ ⁇ can take values in the integer range ⁇ 0; ⁇ ⁇ ⁇ 1 ⁇ , where ⁇ is a natural number greater than 1.
• boundary features index ⁇ ⁇ is defined as follows: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ Or alternatively: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇
• boundary features rely on the boundary strength criterion Bs defined in HEVC and VVC DBF [Norkin2012, Karczewicz2021, JVET-T2002] (see Table 1), which values are 0, 1 or 2 (that is, ⁇ ⁇ 3).
• D deblocking filter filter Table 3 Effects of reserved boundary features index values.
• deblocking is turned off depending on local classification, without requesting any syntax. This comes down to the way DBF is implemented in HEVC and VVC.
• deblocking control relies on the signaling of a local flag. That embodiment is similar to what happens in ALF.
• Deblocking class index and filter transposition A deblocking class index ⁇ ⁇ is defined, that depends on both spatiality and boundaries features.
• the deblocking class index ⁇ ⁇ derives from spatiality index ⁇ and boundary features index ⁇ ⁇ .
• ⁇ denote the range of unidirectional boundary features indices.
• the deblocking class index is computed as: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇
• ⁇ denotes the deblocking grid size: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 2. ⁇ ⁇ 2 ⁇ 1 ⁇ ⁇ ⁇ ⁇ Table 4 indexing in function of the decoding grid size and of the range of boundary features indices.
• Filter transposition and class clustering Transposition is introduced to reduce the total number of classes, as presented for spatiality above.
• deblocking filter transposition is also used for samples lying on the bisector of the deblocking grid (dark-gray samples in Figure 15), to further benefit from the symmetry between horizontal and vertical boundary features indices.
• the resulting class clustering policy is described in Table 5.
• Filter coefficients Regardless of its shape and size, a deblocking filter is determined by its coefficients. Let ⁇ denote their number. Filters coefficients are signaled as signed integers coded on ⁇ ⁇ ⁇ bits, similarly to ALF implementations in VVC and ECM. Straightforward deblocking implementation A reconstructed sample ⁇ ⁇ is filtered into its deblocked version ⁇ ⁇ as follows: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ .
• DC-neutrality of deblocking filters is enforced, that is, ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ 1, and a reconstructed sample ⁇ ⁇ is filtered into its deblocked version ⁇ ⁇ as follows: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ .
• Every sample is filtered with the same filter shape, of same size, with the same coefficient layout. This favors filter merging during optimization at encoder side.
• That shape can be a square, a cross, a diamond, or any combination of square, rectangle, cross and diamond.
• Centered filter shape In an embodiment, the common filter shape is odd-size and centered: the current to-be-filtered sample is located at the center of the filter shape (see Figure 16 for basic shapes).
• Off-centered filter shapes In another embodiment, the common filter shape is off-centered, that is: the filter shape covers more samples towards horizontal and vertical block boundaries than on the other side (see Figure 17). In that embodiment, filter shapes can be even- or odd-sized.
• filter shapes depend on the location of the current sample to be filtered. This prevents filter merging between spatial classes that are not transposed from each other.
• Figure 18 depicts two examples of spatially dependent filter sets for top-left quadrant.
• Boundary features dependent filter shapes In an embodiment, the dimensionality of the filter shape depends on boundary features (see section on no-deblocking boundary features class vs. local flag signaling). In that embodiment, a predetermined value of ⁇ ⁇ ⁇ and ⁇ ⁇ ⁇ (e.g., zero) turns off deblocking in that direction, that is, the filter shape is 1D, as described in Table 3.
• That embodiment can be combined with spatially dependent filter shapes (3.4.2.2), in which case deblocking filter shapes depend on both spatiality and boundaries features.
• deblocking filter shapes depend on both spatiality and boundaries features.
• default deblocking filters are defined (for instance, those already defined in VVC for DBF), and corrective coefficients, with limited amplitude, are signaled.
• the signaling of DBF corrective coefficients can be done in HLS, for instance in picture header, in slice header, in APS. It can also be made more local, for instance at tile, sub-picture, CTU levels. This gives more flexibility to adapt the filters locally to the spatial content changes.
• the corrective coefficients may be learned at encoder side to minimize the MSE between the filtered samples and the original samples, while keeping a constraint that they have to remain in a limited amplitude range (e.g., from -2 to +2 per coefficient).
• Deblocking syntax elements Deblocking filter set Deblocking filter sets are signaled for instance at picture, slice or tile level, for instance in a dedicated Adaptation Parameter Set (APS), similarly to ALF implementations in VVC and ECM.
• Deblocking filter coefficients Deblocking filter coefficients are signaled as signed integers, for instance with their absolute value Exp-Golomb-coded with order k equal to 0 followed by a sign-bit for non-zero coefficients.
• syntax include additional information, such as binary flags or indices that specify the length of the deblocking filters.
• Local deblocking control In another embodiment, a binary flag is signaled locally, e.g., for each shifted block, to turn off/on the deblocking.
• several deblocking filter sets are referenced. In an embodiment, several deblocking filter sets are referenced at slice or picture level, and an index is signaled locally, e.g., for each shifted block, to specify which deblocking filter set is used among the referenced deblocking filter sets.
• legacy DBF is not applied after picture reconstruction, and ALF is augmented with the deblocking classes described in the classification section. That is: samples lying across CU/PU/TUs borders that trigger deblocking conditions are filtered with adaptive Wiener deblocking filters, while the remaining samples are filtered with legacy ALF filters using legacy ALF classifiers (Laplacian-based or band- based).
• legacy ALF classifiers Laplacian-based or band- based.
• the Figure 20 depicts the corresponding workflow. This advantageously simplifies the process design and may reduce the overall number of processing steps (the samples previously processed by DBF then ALF will now be only processed once by ALF). Filters optimization at encoder side At encoder side, deblocking filter sets are optimized for a whole slice level or a whole picture.
• Filter coefficient computation Filter coefficients are calculated by solving Wiener-Hopf equations, that is: ⁇ . ⁇ ⁇ ⁇ é ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ . ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ . ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ... ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ . ⁇ ⁇ ⁇ ⁇ ⁇ ù ⁇ ⁇ samples, the filter coefficients, and the cross-correlation vector of the to-be-deblocked samples ⁇ and the source samples ⁇ , respectively.
• the deblocking filter set for the current slice/picture is determined as follows. First the slice/picture is split into shifted blocks (see Figure 12), that is, non- overlapping deblocking areas that contain each a vertical block boundary, a horizontal block boundary, or both. Then the statistics are collected separately for each deblocking class on the whole slice or picture. Wiener-Hopf equations (see section on substitution to legacy DBF) are solved for each class. The solutions yield the optimized filter set for the current slice/picture, with an optimal filter per deblocking class.
• the optimization at encoder side is an iterative two-step process. At initialization, deblocking is enabled in every shifted block. In a first step, an optimal deblocking filter set is determined based on the statistics of every shifted block where deblocking is enabled. Then in a second step, whether to apply deblocking or not is determined for each shifted block based on a rate-distortion optimization (RDO) condition.
• RDO rate-distortion optimization
• the encoder selects locally, e.g., for each shifted block, the best deblocking filter set in regards with a rate-distortion criterion.
• Filter and class merging In the embodiment where deblocking optimization includes filter/class merging, a plurality of deblocking filter sets with different number of classes are tested. Iterative merging algorithm First, an optimal deblocking filter set is computed, like in the main embodiment, with an optimal deblocking filter per deblocking class. Then a merging algorithm is applied iteratively.
• Filter merging can only be applied on classes that present the same filter shape, size, and layout (same position offsets corresponding to the same filter coefficients), so that their corresponding Wiener-Hopf matrices have the same rank and can be merged into a single one.
• a ‘potential merge pair’ denote a pair of deblocking classes that meets this condition: same filter shape, size and layout.
• every potential merge pair is considered successively, by merging their corresponding statistics.
• the pair that exhibits the lowest loss in terms of squared error is kept.
• the merging algorithm keeps running, as long as a potential pair remains.
• the number of classes is decremented by one.
• the merging algorithm yields as many deblocking filter sets as initial classes.
• the initial deblocking filter set contains as many filters as classes, then the next one has one less filter, then the next one another one less, and so on, until the last filter set that contains only one filter, shared by every class.
• Optimal deblocking filter set Eventually, the encoder selects the deblocking filter set that presents the best RDO performances, considering the error and the signaling cost.
• the overall class merging algorithm is illustrated in Figure 22.
• One embodiment of a method 2300 under the general aspects described here is shown in Figure 23.
• the method commences at start block 2301 and control proceeds to block 2310 for determining at least one in-loop filter for a video block based on a classification. Control proceeds from block 2310 to block 2320 for filtering a reconstructed version of said video block with said determined at least one in-loop filter. Control proceeds from block 2320 to block 2330 for signaling said determined at least one in-loop filter. Control proceeds from block 2330 to block 2340 for encoding the video block using the filtered reconstructed video block.
• One embodiment of a method 2400 under the general aspects described here is shown in Figure 24. The method commences at start block 2401 and control proceeds to block 2410 for deriving a classification for a video block.
• Control proceeds from block 2410 to block 2420 for determining at least one in-loop filter for the video block based on the classification. Control proceeds from block 2420 to block 2430 for filtering a reconstructed version of said video block with said determined at least one in-loop filter. Control proceeds from block 2430 to block 2440 for decoding the video block using the filtered reconstructed video block.
• Figure 25 shows one embodiment of an apparatus 2500 for encoding, decoding, compressing, or decompressing video data using any of the above methods, or variations.
• the apparatus comprises Processor 2510 and can be interconnected to a memory 2520 through at least one port. Both Processor 2510 and memory 2520 can also have one or more additional interconnections to external connections.
• Processor 2510 is also configured to either insert or receive information in a bitstream and, either compressing, encoding, or decoding using any of the described aspects.
• the embodiments described here include a variety of aspects, including tools, features, embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the application or scope of those aspects. Indeed, all of the different aspects can be combined and interchanged to provide further aspects. Moreover, the aspects can be combined and interchanged with aspects described in earlier filings as well. The aspects described and contemplated in this application can be implemented in many different forms.
• Figures 26, 27, and 28 provide some embodiments, but other embodiments are contemplated and the discussion of Figures 26, 27, and 28 does not limit the breadth of the implementations.
• At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded.
• These and other aspects can be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and/or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.
• the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, the terms “image,” “picture” and “frame” may be used interchangeably.
• the term “reconstructed” is used at the encoder side while “decoded” is used at the decoder side.
• Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and/or use of specific steps and/or actions may be modified or combined.
• the encoder 100 Before being encoded, the video sequence may go through pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to get a signal distribution more resilient to compression (for instance using a histogram equalization of one of the color components). Metadata can be associated with the pre-processing and attached to the bitstream. In the encoder 100, a picture is encoded by the encoder elements as described below.
• a color transform e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0
• Metadata can be associated with the pre-processing and attached to the bitstream.
• a picture is encoded by the encoder elements as described below.
• the picture to be encoded is partitioned (102) and processed in units of, for example, CUs.
• Each unit is encoded using, for example, either an intra or inter mode.
• intra prediction 160
• inter mode motion estimation (175) and compensation (170) are performed.
• the encoder decides (105) which one of the intra mode or inter mode to use for encoding the unit, and indicates the intra/inter decision by, for example, a prediction mode flag.
• Prediction residuals are calculated, for example, by subtracting (110) the predicted block from the original image block.
• the prediction residuals are then transformed (125) and quantized (130).
• the quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (145) to output a bitstream.
• the encoder can skip the transform and apply quantization directly to the non-transformed residual signal.
• the encoder can bypass both transform and quantization, i.e., the residual is coded directly without the application of the transform or quantization processes.
• the encoder decodes an encoded block to provide a reference for further predictions.
• the quantized transform coefficients are de-quantized (140) and inverse transformed (150) to decode prediction residuals. Combining (155) the decoded prediction residuals and the predicted block, an image block is reconstructed.
• In-loop filters (165) are applied to the reconstructed picture to perform, for example, deblocking/SAO (Sample Adaptive Offset)/ALF (Adaptive Loop Filtering) filtering to reduce encoding artifacts.
• the filtered image is stored at a reference picture buffer (180).
• Figure 27 illustrates a block diagram of a video decoder 200.
• a bitstream is decoded by the decoder elements as described below.
• Video decoder 200 generally performs a decoding pass reciprocal to the encoding pass as described in Figure 26.
• the encoder 100 also generally performs video decoding as part of encoding video data.
• the input of the decoder includes a video bitstream, which can be generated by video encoder 100.
• the bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information.
• the picture partition information indicates how the picture is partitioned.
• the decoder may therefore divide (235) the picture according to the decoded picture partitioning information.
• the transform coefficients are de-quantized (240) and inverse transformed (250) to decode the prediction residuals.
• Combining (255) the decoded prediction residuals and the predicted block an image block is reconstructed.
• the predicted block can be obtained (270) from intra prediction (260) or motion- compensated prediction (i.e., inter prediction) (275).
• In-loop filters (265) are applied to the reconstructed image.
• the filtered image is stored at a reference picture buffer (280).
• the decoded picture can further go through post-decoding processing (285), for example, an inverse color transform (e.g., conversion from YcbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre-encoding processing (101).
• the post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.
• Figure 28 illustrates a block diagram of an example of a system in which various aspects and embodiments are implemented.
• System 1000 can be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document.
• Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers.
• Elements of system 1000 singly or in combination, can be embodied in a single integrated circuit (IC), multiple ICs, and/or discrete components.
• the processing and encoder/decoder elements of system 1000 are distributed across multiple ICs and/or discrete components.
• the system 1000 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and/or output ports.
• the system 1000 is configured to implement one or more of the aspects described in this document.
• the system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document.
• Processor 1010 can include embedded memory, input output interface, and various other circuitries as known in the art.
• the system 1000 includes at least one memory 1020 (e.g., a volatile memory device, and/or a non- volatile memory device).
• System 1000 includes a storage device 1040, which can include non-volatile memory and/or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and/or optical disk drive.
• the storage device 1040 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and/or a network accessible storage device, as non-limiting examples.
• System 1000 includes an encoder/decoder module 1030 configured, for example, to process data to provide an encoded video or decoded video, and the encoder/decoder module 1030 can include its own processor and memory.
• the encoder/decoder module 1030 represents module(s) that can be included in a device to perform the encoding and/or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder/decoder module 1030 can be implemented as a separate element of system 1000 or can be incorporated within processor 1010 as a combination of hardware and software as known to those skilled in the art.
• processor 1010 Program code to be loaded onto processor 1010 or encoder/decoder 1030 to perform the various aspects described in this document can be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010.
• processor 1010, memory 1020, storage device 1040, and encoder/decoder module 1030 can store one or more of various items during the performance of the processes described in this document.
• Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.
• memory inside of the processor 1010 and/or the encoder/decoder module 1030 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding.
• a memory external to the processing device (for example, the processing device can be either the processor 1010 or the encoder/decoder module 1030) is used for one or more of these functions.
• the external memory can be the memory 1020 and/or the storage device 1040, for example, a dynamic volatile memory and/or a non-volatile flash memory.
• an external non-volatile flash memory is used to store the operating system of, for example, a television.
• a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO/IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).
• the input to the elements of system 1000 can be provided through various input devices as indicated in block 1130.
• Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and/or (iv) a High-Definition Multimedia Interface (HDMI) input terminal.
• RF radio frequency
• COMP Component
• USB Universal Serial Bus
• HDMI High-Definition Multimedia Interface
• the input devices of block 1130 have associated respective input processing elements as known in the art.
• the RF portion can be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band- limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets.
• a desired frequency also referred to as selecting a signal, or band-limiting a signal to a band of frequencies
• downconverting the selected signal for example
• band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments
• demodulating the downconverted and band- limited signal (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets
• the RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers.
• the RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband.
• the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band.
• Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter.
• the RF portion includes an antenna.
• the USB and/or HDMI terminals can include respective interface processors for connecting system 1000 to other electronic devices across USB and/or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented, for example, within a separate input processing IC or within processor 1010 as necessary. Similarly, aspects of USB or HDMI interface processing can be implemented within separate interface Ics or within processor 1010 as necessary.
• the demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 1010, and encoder/decoder 1030 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.
• Various elements of system 1000 can be provided within an integrated housing, Within the integrated housing, the various elements can be interconnected and transmit data therebetween using suitable connection arrangement, for example, an internal bus as known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.
• the system 1000 includes communication interface 1050 that enables communication with other devices via communication channel 1060.
• the communication interface 1050 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 1060.
• the communication interface 1050 can include, but is not limited to, a modem or network card and the communication channel 1060 can be implemented, for example, within a wired and/or a wireless medium.
• Data is streamed, or otherwise provided, to the system 1000, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers).
• IEEE 802.11 IEEE refers to the Institute of Electrical and Electronics Engineers.
• the Wi- Fi signal of these embodiments is received over the communications channel 1060 and the communications interface 1050 which are adapted for Wi-Fi communications.
• the communications channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the-top communications.
• inventions provide streamed data to the system 1000 using a set-top box that delivers the data over the HDMI connection of the input block 1130. Still other embodiments provide streamed data to the system 1000 using the RF connection of the input block 1130. As indicated above, various embodiments provide data in a non- streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.
• the system 1000 can provide an output signal to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120.
• the display 1100 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and/or a foldable display.
• OLED organic light-emitting diode
• the display 1100 can be for a television, a tablet, a laptop, a cell phone (mobile phone), or another device.
• the display 1100 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop).
• the other peripheral devices 1120 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and/or a lighting system.
• DVR digital versatile disc
• Various embodiments use one or more peripheral devices 1120 that provide a function based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.
• control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention.
• the output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to system 1000 using the communications channel 1060 via the communications interface 1050.
• the display 1100 and speakers 1110 can be integrated in a single unit with the other components of system 1000 in an electronic device such as, for example, a television.
• the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.
• the display 1100 and speaker 1110 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 1130 is part of a separate set-top box.
• the output signal can be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.
• the embodiments can be carried out by computer software implemented by the processor 1010 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments can be implemented by one or more integrated circuits.
• the memory 1020 can be of any type appropriate to the technical environment and can be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples.
• the processor 1010 can be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non- limiting examples.
• Various implementations involve decoding. “Decoding”, as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence to produce a final output suitable for display.
• such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding.
• processes also, or alternatively, include processes performed by a decoder of various implementations described in this application.
• decoding refers only to entropy decoding
• decoding refers only to differential decoding
• decoding refers to a combination of entropy decoding and differential decoding.
• decoding process is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.
• Various implementations involve encoding.
• encoding as used in this application can encompass all or part of the processes performed, for example, on an input video sequence to produce an encoded bitstream.
• processes include one or more of the processes typically performed by an encoder, for example, partitioning, differential encoding, transformation, quantization, and entropy encoding.
• processes also, or alternatively, include processes performed by an encoder of various implementations described in this application.
• encoding refers only to entropy encoding
• encoding refers only to differential encoding
• encoding refers to a combination of differential encoding and entropy encoding.
• Rate distortion optimization is usually formulated as minimizing a rate distortion function, which is a weighted sum of the rate and of the distortion.
• the approaches may be based on an extensive testing of all encoding options, including all considered modes or coding parameters values, with a complete evaluation of their coding cost and related distortion of the reconstructed signal after coding and decoding.
• Faster approaches may also be used, to save encoding complexity, in particular with computation of an approximated distortion based on the prediction or the prediction residual signal, not the reconstructed one.
• Mix of these two approaches can also be used, such as by using an approximated distortion for only some of the possible encoding options, and a complete distortion for other encoding options.
• Other approaches only evaluate a subset of the possible encoding options.
• implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program).
• An apparatus can be implemented in, for example, appropriate hardware, software, and firmware.
• a processor which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device.
• Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.
• PDAs portable/personal digital assistants
• this application may refer to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory. Further, this application may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.
• this application may refer to “receiving” various pieces of information.
• Receiving is, as with “accessing”, intended to be a broad term.
• Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory).
• “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.
• any of the following “/”, “and/or”, and “at least one of”, for example, in the cases of “A/B”, “A and/or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B).
• such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C).
• This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.
• the word “signal” refers to, among other things, indicating something to a corresponding decoder.
• the encoder signals a particular one of a plurality of transforms, coding modes or flags.
• the same transform, parameter, or mode is used at both the encoder side and the decoder side.
• an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter.
• signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter.
• signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.
• implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of a described embodiment.
• Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal.
• the formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream.
• the information that the signal carries can be, for example, analog or digital information.
• the signal can be transmitted over a variety of different wired or wireless links, as is known.
• the signal can be stored on a processor-readable medium.
• embodiments can include one or more of the following features, devices, or aspects, alone or in any combination, across various claim categories and types:
• One embodiment comprises an apparatus or method of determining a filter to be used as an in-loop filter in an encoding or decoding video environment.
• the filter is either a deblocking filter or an adaptive loop filter.
• One embodiment comprises the above method wherein a classification is used to determine the in-loop filter.
• One embodiment comprises the above method wherein filter selection is controllable at the coding tree unit level or higher.
• Some embodiments comprise any of the above methods wherein an index is signaled to indicate the determined filter.
• Some embodiments comprise any of the above methods wherein class merging of filters having the same exact shape and coefficient location occurs.
• Some embodiments comprise any of the above methods wherein at least one filter is pre-learned offline and hard coded within an encoder and decoder. Some embodiments comprise any of the above methods wherein deblock filtering is turned off in one direction. Some embodiments comprise any of the above methods wherein deblocking filter sets are signaled at a picture, slice or file level or in a parameter set. Some embodiments comprise any of the above methods wherein at least one filter is selected to be used per classification type. Some embodiments comprise any of the above methods wherein Wiener filtering is used.
• One embodiment comprises a bitstream or signal that includes one or more syntax elements to perform the above functions, or variations thereof.
• One embodiment comprises a bitstream or signal that includes syntax conveying information generated according to any of the embodiments described.
• One embodiment comprises creating and/or transmitting and/or receiving and/or decoding according to any of the embodiments described.
• One embodiment comprises a method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the embodiments described.
• One embodiment comprises inserting in the signaling syntax elements that enable the decoder to determine decoding information in a manner corresponding to that used by an encoder.
• One embodiment comprises creating and/or transmitting and/or receiving and/or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof.
• One embodiment comprises a TV, set-top box, cell phone, tablet, or other electronic device that performs transform method(s) according to any of the embodiments described.
• One embodiment comprises a TV, set-top box, cell phone, tablet, or other electronic device that performs transform method(s) determination according to any of the embodiments described, and that displays (e.g., using a monitor, screen, or other type of display) a resulting image.
• One embodiment comprises a TV, set-top box, cell phone, tablet, or other electronic device that selects, bandlimits, or tunes (e.g., using a tuner) a channel to receive a signal including an encoded image, and performs transform method(s) according to any of the embodiments described.
• One embodiment comprises a TV, set-top box, cell phone, tablet, or other electronic device that receives (e.g., using an antenna) a signal over the air that includes an encoded image, and performs transform method(s).

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=86605038

Family Applications (1)

Country Status (3)

Family Cites Families (1)

• 2024
  • 2024-04-17 EP EP24718249.6A patent/EP4702740A1/de active Pending
  • 2024-04-17 CN CN202480027812.7A patent/CN121002857A/zh active Pending
  • 2024-04-17 WO PCT/EP2024/060332 patent/WO2024223370A1/en not_active Ceased

Also Published As

Similar Documents

Legal Events