WO2023197181A1 - 解码方法、编码方法、解码器以及编码器 - Google Patents

解码方法、编码方法、解码器以及编码器 Download PDF

Info

Publication number
WO2023197181A1
WO2023197181A1 PCT/CN2022/086448 CN2022086448W WO2023197181A1 WO 2023197181 A1 WO2023197181 A1 WO 2023197181A1 CN 2022086448 W CN2022086448 W CN 2022086448W WO 2023197181 A1 WO2023197181 A1 WO 2023197181A1
Authority
WO
WIPO (PCT)
Prior art keywords
prediction mode
intra prediction
transformation
mode
block
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/086448
Other languages
English (en)
French (fr)
Inventor
王凡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Oppo Mobile Telecommunications Corp Ltd
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp Ltd filed Critical Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority to EP22936835.2A priority Critical patent/EP4510588A4/en
Priority to JP2024560810A priority patent/JP2025512523A/ja
Priority to PCT/CN2022/086448 priority patent/WO2023197181A1/zh
Priority to CN202280094735.8A priority patent/CN119013989A/zh
Publication of WO2023197181A1 publication Critical patent/WO2023197181A1/zh
Priority to US18/913,625 priority patent/US20250039454A1/en
Anticipated expiration legal-status Critical
Priority to MX2024012724A priority patent/MX2024012724A/es
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/18—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a set of transform coefficients
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/11—Selection of coding mode or of prediction mode among a plurality of spatial predictive coding modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
    • H04N19/122—Selection of transform size, e.g. 8x8 or 2x4x8 DCT; Selection of sub-band transforms of varying structure or type
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/46—Embedding additional information in the video signal during the compression process
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/593—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards

Definitions

  • the embodiments of the present application relate to the technical field of image and video encoding and decoding, and more specifically, to a decoding method, an encoding method, a decoder, and an encoder.
  • Digital video compression technology mainly compresses huge digital image and video data to facilitate transmission and storage.
  • Digital video compression standards can implement video decompression technology, there is still a need to pursue better digital video decompression technology to Improving compression efficiency.
  • Embodiments of the present application provide a decoding method, encoding method, decoder and encoder, which can improve compression efficiency.
  • this application provides a decoding method, including:
  • Predict the current block based on the first intra prediction mode and the second intra prediction mode derived by the prediction mode derivation mode, and obtain a prediction block of the current block;
  • a reconstructed block of the current block is obtained.
  • this application provides an encoding method, including:
  • Predict the current block based on the first intra prediction mode and the second intra prediction mode derived by the prediction mode derivation mode, and obtain the prediction block of the current block;
  • the fourth transform coefficient is encoded.
  • this application provides a decoder, including:
  • a decoding unit used to decode the code stream to obtain the first transform coefficient of the current block
  • Transform unit used for:
  • a prediction unit configured to predict the current block based on the first intra prediction mode and the second intra prediction mode derived from the prediction mode derivation mode, and obtain a prediction block of the current block;
  • a reconstruction unit configured to obtain a reconstruction block of the current block based on the prediction block of the current block and the residual block of the current block.
  • this application provides an encoder, including:
  • a prediction unit configured to predict the current block based on the first intra prediction mode and the second intra prediction mode derived from the prediction mode derivation mode, and obtain the prediction block of the current block;
  • a residual unit configured to obtain a residual block of the current block based on the prediction block of the current block
  • Transform unit used for:
  • Coding unit used to encode the fourth transform coefficient.
  • this application provides a decoder, including:
  • a processor adapted to implement computer instructions
  • the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for the processor to load and execute the decoding method in the above-mentioned first aspect or its respective implementations.
  • processors there are one or more processors and one or more memories.
  • the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
  • this application provides an encoder, including:
  • a processor adapted to implement computer instructions
  • the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for the processor to load and execute the encoding method in the above-mentioned second aspect or its respective implementations.
  • processors there are one or more processors and one or more memories.
  • the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
  • the present application provides a computer-readable storage medium that stores computer instructions.
  • the computer instructions When the computer instructions are read and executed by a processor of a computer device, the computer device performs the above-mentioned first aspect.
  • the present application provides a code stream, which code stream relates to the code stream in the above-mentioned first aspect or the code stream involved in the above-mentioned second aspect.
  • this application introduces the first transformation based on the prediction mode derivation mode and the second transformation, which can improve the decompression efficiency of the current block.
  • Figure 1 is a schematic block diagram of a coding framework provided by an embodiment of the present application.
  • Figure 2 is a schematic diagram of specific directions of the 65 angle prediction modes provided by the embodiment of the present application.
  • Figure 3 is an example of reference pixels in the wide-angle prediction mode provided by the embodiment of the present application.
  • FIG. 4 is a schematic diagram of the MIP mode provided by the embodiment of the present application.
  • Figure 5 is a schematic diagram of deriving a prediction mode based on DIMD provided by an embodiment of the present application.
  • Figure 6 is a schematic diagram of deriving prediction blocks based on DIMD provided by an embodiment of the present application.
  • Figure 7 is a schematic diagram of a template used by TIMD provided by an embodiment of the present application.
  • Figure 8 is an example of a weight map corresponding to 64 weight derivation modes of GPM on a square block provided by the embodiment of the present application.
  • Figure 9 is an example of dividing lines in the weight derivation mode provided by the embodiment of the present application.
  • Figure 10 is an example of a weight map corresponding to 56 weight derivation modes of AWP on a square block provided by the embodiment of the present application.
  • FIG 11 is a schematic diagram of a GPM or AWP provided by an embodiment of the present application.
  • FIG. 12 is an example of a DCT2-type base image provided by an embodiment of the present application.
  • Figure 13 is an example of LFNST provided by the embodiment of this application.
  • Figure 14 is an example of a transformation matrix group of LFNST provided by the embodiment of the present application.
  • Figure 15 is a schematic block diagram of a decoding framework provided by an embodiment of the present application.
  • Figure 16 is a schematic flow chart of the decoding method provided by the embodiment of the present application.
  • Figure 17 is a schematic flow chart of the encoding method provided by the embodiment of the present application.
  • Figure 18 is a schematic block diagram of a decoder provided by an embodiment of the present application.
  • Figure 19 is a schematic block diagram of an encoder provided by an embodiment of the present application.
  • Figure 20 is a schematic block diagram of an electronic device provided by an embodiment of the present application.
  • the solutions provided by the embodiments of this application can be applied to the field of digital video coding technology, including but not limited to: image coding and decoding, video coding and decoding, hardware video coding and decoding, dedicated circuit video coding and decoding, and real-time video coding and decoding. field.
  • the solution provided by the embodiments of the present application can be combined with the Audio Video Coding Standard (AVS), the second generation AVS standard (AVS2) or the third generation AVS standard (AVS3).
  • AVS Audio Video Coding Standard
  • VVC Very Video Coding
  • the solution provided by the embodiment of the present application can be used to perform lossy compression on images (lossy compression), or can also be used to perform lossless compression on images (lossless compression).
  • the lossless compression can be visually lossless compression (visually lossless compression) or mathematically lossless compression (mathematically lossless compression).
  • Video coding and decoding standards all adopt block-based hybrid coding framework.
  • Each frame in the video is divided into the largest coding unit (LCU) or coding tree unit (Coding Tree Unit, CTU) of the same size (such as 128x128, 64x64, etc.) squares.
  • Each maximum coding unit or coding tree unit can be divided into rectangular coding units (coding units, CU) according to rules.
  • the coding unit may also be divided into prediction unit (PU), transformation unit (TU), etc.
  • the hybrid coding framework includes prediction, transform, quantization, entropy coding, in loop filter and other modules.
  • the prediction module includes intra prediction and inter prediction.
  • Inter-frame prediction includes motion estimation (motion estimation) and motion compensation (motion compensation).
  • Intra-frame prediction only refers to the information of the same frame image and predicts the pixel information within the current divided block. Since there is a strong similarity between adjacent frames in the video, the interframe prediction method is used in video coding and decoding technology to eliminate the temporal redundancy between adjacent frames, thereby improving coding efficiency.
  • Inter-frame prediction can refer to image information of different frames and use motion estimation to search for motion vector information that best matches the current divided block. The transformation converts the predicted image blocks into the frequency domain and redistributes the energy. Combined with quantization, it can remove information that is insensitive to the human eye and is used to eliminate visual redundancy.
  • Entropy coding can eliminate character redundancy based on the current context model and the probability information of the binary code stream.
  • the encoder can first read a black-and-white image or color image from the original video sequence, and then encode the black-and-white image or color image.
  • the black-and-white image may include pixels of the brightness component
  • the color image may include pixels of the chrominance component.
  • the color image may also include pixels with a brightness component.
  • the color format of the original video sequence can be luminance-chrominance (YCbCr, YUV) format or red-green-blue (Red-Green-Blue, RGB) format, etc.
  • the encoder reads a black-and-white image or a color image, it divides it into blocks respectively, and uses intra-frame prediction or inter-frame prediction for the current block to generate a predicted block of the current block.
  • the prediction block is subtracted from the original block of the current block. block to obtain a residual block, transform and quantize the residual block to obtain a quantized coefficient matrix, entropy encode the quantized coefficient matrix and output it to the code stream.
  • the decoder uses intra prediction or inter prediction for the current block to generate a prediction block of the current block.
  • the decoding end decodes the code stream to obtain the quantization coefficient matrix, performs inverse quantization and inverse transformation on the quantization coefficient matrix to obtain the residual block, and adds the prediction block and the residual block to obtain the reconstruction block.
  • Reconstruction blocks can be used to compose a reconstructed image, and the decoder performs loop filtering on the reconstructed image based on images or blocks to obtain a decoded image.
  • the current block can be the current coding unit (CU) or the current prediction unit (PU), etc.
  • the encoding end also requires similar operations as the decoding end to obtain the decoded image.
  • the decoded image can be used as a reference frame for inter-frame prediction for subsequent frames.
  • the block division information determined by the encoding end, mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc., need to be output to the code stream if necessary.
  • the decoding end determines the same block division information as the encoding end through parsing and analyzing based on existing information, prediction, transformation, quantization, entropy coding, loop filtering and other mode information or parameter information, thereby ensuring the decoded image and decoding obtained by the encoding end
  • the decoded image obtained at both ends is the same.
  • the decoded image obtained at the encoding end is usually also called a reconstructed image.
  • the current block can be divided into prediction units during prediction, and the current block can be divided into transformation units during transformation.
  • the divisions between prediction units and transformation units can be the same or different.
  • the above is only the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules of the framework or some steps of the process may be optimized. This application is applicable to this block-based hybrid coding.
  • Figure 1 is a schematic block diagram of a coding framework 100 provided by an embodiment of the present application.
  • the coding framework 100 may include an intra prediction unit 180, an inter prediction unit 170, a residual unit 110, a transformation and quantization unit 120, an entropy coding unit 130, an inverse transformation and inverse quantization unit 140, and a loop. filter unit 150.
  • the encoding framework 100 may also include a decoded image buffer unit 160. This coding framework 100 may also be called a hybrid framework coding mode.
  • the intra prediction unit 180 or the inter prediction unit 170 may predict the image block to be encoded to output the prediction block.
  • the residual unit 110 may calculate a residual block, that is, a difference value between the prediction block and the image block to be encoded, based on the prediction block and the image block to be encoded.
  • the transformation and quantization unit 120 is used to perform operations such as transformation and quantization on the residual block to remove information that is insensitive to the human eye, thereby eliminating visual redundancy.
  • the residual block before transformation and quantization by the transformation and quantization unit 120 may be called a time domain residual block
  • the time domain residual block after transformation and quantization by the transformation and quantization unit 120 may be called a frequency residual block. or frequency domain residual block.
  • the entropy encoding unit 130 may output a code stream based on the transform quantization coefficient. For example, the entropy encoding unit 130 may eliminate character redundancy according to the target context model and probability information of the binary code stream. For example, the entropy coding unit 130 may be used for context-based adaptive binary arithmetic entropy coding (CABAC). The entropy encoding unit 130 may also be called a header information encoding unit.
  • CABAC context-based adaptive binary arithmetic entropy coding
  • the image block to be encoded can also be called an original image block or a target image block, and a prediction block can also be called a predicted image block or image prediction block, and can also be called a prediction signal or prediction information.
  • the reconstruction block may also be called a reconstructed image block or an image reconstruction block, and may also be called a reconstructed signal or reconstructed information.
  • the image block to be encoded may also be called an encoding block or a coded image block, and for the decoding end, the image block to be encoded may also be called a decoding block or a decoded image block.
  • the image block to be encoded may be a CTU or a CU.
  • the encoding framework 100 calculates the residual between the prediction block and the image block to be encoded to obtain the residual block, and then transmits the residual block to the decoder through processes such as transformation and quantization.
  • the decoder receives and decodes the code stream, it obtains the residual block through steps such as inverse transformation and inverse quantization.
  • the prediction block predicted by the decoder is superimposed on the residual block to obtain the reconstructed block.
  • the inverse transform and inverse quantization unit 140, the loop filter unit 150 and the decoded image buffer unit 160 in the encoding framework 100 may be used to form a decoder.
  • the intra prediction unit 180 or the inter prediction unit 170 can predict the image block to be encoded based on the existing reconstructed block, thereby ensuring that the encoding end and the decoding end have consistent understanding of the reference frame.
  • the encoder can replicate the decoder's processing loop and thus produce the same predictions as the decoder.
  • the quantized transform coefficients are inversely transformed and inversely quantized by the inverse transform and inverse quantization unit 140 to copy the approximate residual block at the decoding side.
  • the approximate residual block After the approximate residual block is added to the prediction block, it can pass through the loop filtering unit 150 to smoothly filter out block effects and other effects caused by block-based processing and quantization.
  • the image blocks output by the loop filter unit 150 may be stored in the decoded image buffer unit 160 for use in prediction of subsequent images.
  • Figure 1 is only an example of the present application and should not be understood as a limitation of the present application.
  • the loop filtering unit 150 in the coding framework 100 may include a deblocking filter (DBF) and sample adaptive compensation filtering (SAO).
  • DBF deblocking filter
  • SAO sample adaptive compensation filtering
  • the encoding framework 100 may adopt a neural network-based loop filtering algorithm to improve video compression efficiency.
  • the coding framework 100 may be a video coding hybrid framework based on a deep learning neural network.
  • a model based on a convolutional neural network can be used to calculate the result of filtering the pixels based on the deblocking filter and sample adaptive compensation filtering.
  • the network structures of the loop filter unit 150 on the luminance component and the chrominance component may be the same or different. Considering that the brightness component contains more visual information, the brightness component can also be used to guide the filtering of the chroma component to improve the reconstruction quality of the chroma component.
  • inter-frame prediction can refer to the image information of different frames and use motion estimation to search for the motion vector information that best matches the image block to be encoded to eliminate temporal redundancy;
  • the frames used for inter-frame prediction can be P frames and /or B frame, P frame refers to forward prediction frame, B frame refers to bidirectional prediction frame.
  • intra-frame prediction only refers to the information of the same frame image and predicts pixel information within the image block to be encoded to eliminate spatial redundancy;
  • the frame used for intra-frame prediction can be an I frame.
  • the image block to be encoded can refer to the upper left image block, the upper image block and the left image block as reference information to predict the image block to be encoded, and the image block to be encoded The block is used as reference information for the next image block, so that the entire image can be predicted.
  • the input digital video is in color format, such as YUV 4:2:0 format, then every 4 pixels of each image frame of the digital video consists of 4 Y components and 2 UV components.
  • the encoding framework can components (i.e. luma blocks) and UV components (i.e. chrominance blocks) are encoded separately.
  • the decoding end can also perform corresponding decoding according to the format.
  • intra prediction can predict the image block to be encoded with the help of angle prediction mode and non-angle prediction mode to obtain the prediction block. Based on the rate distortion information calculated between the prediction block and the image block to be encoded, the to-be-encoded image block can be filtered out.
  • the optimal prediction mode of the image block is determined and the prediction mode is transmitted to the decoder through the code stream.
  • the decoder parses the prediction mode, predicts the prediction block of the target decoding block, and superimposes the time domain residual block obtained through code stream transmission to obtain the reconstruction block.
  • non-angle prediction modes have remained relatively stable, including mean mode and plane mode.
  • Angle prediction modes continue to increase with the evolution of digital video coding and decoding standards.
  • the H.264/AVC standard only has 8 angle prediction modes and 1 non-angle prediction mode;
  • H.265/HEVC has been expanded to 33 angle prediction modes and 2 non-angle prediction modes. model.
  • the intra-frame prediction mode has been further expanded.
  • Matrix weighted intra-frame prediction (MIP) mode among which this traditional prediction modes include: planar mode, DC mode and 65 angle prediction modes.
  • planar mode is usually used to process some blocks with gradient textures
  • DC mode is usually used to process some flat areas
  • angle prediction mode is usually used to process blocks with obvious angular textures.
  • the current block used for intra prediction may be a square block or a rectangular block.
  • the probability of using the prediction mode at each angle is equal.
  • the probability of using the reference pixels above the horizontal block is greater than height
  • the usage probability of the left reference pixel is smaller than the usage probability of the left reference pixel for vertical blocks (height is greater than width).
  • this application introduces a wide-angle prediction mode.
  • the traditional angle prediction mode is converted into a wide-angle prediction mode.
  • the prediction angle range of the current block It is larger than the prediction angle range when predicting rectangular blocks using traditional angle prediction mode.
  • the decoder can convert the traditional angle prediction mode into the wide-angle prediction mode after receiving the signal, so that , the total number of intra prediction modes and the intra mode encoding method remain unchanged, and the intra mode encoding method remains unchanged.
  • Figure 2 is a schematic diagram of specific directions of the 65 angle prediction modes provided by the embodiment of the present application.
  • index 0 is used to identify the plane mode
  • index 1 is used to identify the DC mode
  • indexes -14 to 80 can be used to identify different angle prediction modes respectively.
  • indexes 2 to 66 are used to identify the traditional angle prediction mode
  • indexes -1 to -14 and indexes 67 to 80 are used to identify the wide angle prediction mode.
  • the traditional intra prediction modes identified by indices 2 to 66 can be used to predict square blocks
  • the wide angle prediction modes identified by indices -1 to -14 and 67 to 80 can be used to predict rectangular blocks.
  • prediction mode identified by the index x mentioned in this application may also be called prediction mode x.
  • intra prediction mode identified by index 2 may also be called intra prediction mode 2.
  • Figure 3 is an example of reference pixels in the wide-angle prediction mode provided by the embodiment of the present application.
  • the horizontal angle mode is replaced by some vertical-like angle modes near (larger than the intra prediction mode 66) the intra prediction mode 66 to expand the prediction angle range.
  • W ⁇ H for example, 4 ⁇ 8 CU
  • some points near the intra prediction mode 66 cannot be indexed to the reference.
  • pixels, and some pixels near intra prediction mode 2 can still be indexed to reference pixels), so some classes near intra prediction mode 66 (smaller than intra prediction mode 66) need to be vertically
  • the angle modes are replaced by some quasi-horizontal angle modes near intra prediction mode 2 (smaller than intra prediction mode 2) to expand the prediction angle range.
  • the intra prediction mode to be performed may be determined or selected based on the size of the current block; for example, the wide-angle prediction mode may be determined or selected based on the size of the current block to perform intra prediction on the current block; for example, in When the current block is a rectangular block (width and height have different dimensions), the wide-angle prediction mode can be used for intra prediction of the current block.
  • the aspect ratio of the current block may be used to determine the angle prediction mode to which the wide-angle prediction mode is replaced and the angle prediction mode after replacement. For example, when predicting the current block, any intra prediction mode having an angle not exceeding the diagonal corner of the current block (from the lower left corner to the upper right corner of the current block) may be selected as the replaced angle prediction mode.
  • the MIP mode can also be called the Matrix weighted Intra Prediction (Matrix weighted Intra Prediction) mode.
  • the process involved in the MIP mode can be divided into three main steps, which are the down-sampling process, the matrix multiplication process and the up-sampling process. Specifically, the spatially adjacent reconstructed samples are first downsampled through the downsampling process, and then the downsampled sample sequence is used as the input vector of the matrix multiplication process, that is, the output vector of the downsampling process is used as the input of the matrix multiplication process.
  • Vector multiply the preset matrix and add the offset vector, and output the calculated sample vector; use the output vector of the matrix multiplication process as the input vector of the upsampling process, and obtain the final prediction block through upsampling .
  • FIG. 4 is a schematic diagram of the MIP mode provided by the embodiment of the present application.
  • the MIP mode obtains the upper adjacent downsampled reconstructed sample vector bdry top by averaging the reconstructed samples adjacent to the top of the current coding unit, and obtains the left adjacent down sample vector bdry top by averaging the reconstructed samples adjacent to the left.
  • Sampling reconstructed sample vector bdry left After obtaining bdry top and bdry left , use them as the input vector bdry red of the matrix multiplication process.
  • the sample can be obtained through the top row vector bdry top red , bdry left , A k ⁇ bdry red +b k based on bdry red Vector, where A k is a preset matrix, b k is a preset bias vector, and k is the index of the MIP mode.
  • a k is a preset matrix
  • b k is a preset bias vector
  • k is the index of the MIP mode.
  • MIP in order to predict a block with width W and height H, MIP requires H reconstructed pixels in the left column of the current block and W reconstructed pixels in the upper row of the current block as input.
  • MIP generates prediction blocks in the following three steps: reference pixel averaging (Averaging), matrix multiplication (Matrix Vector Multiplication) and interpolation (Interpolation).
  • the core of MIP is matrix multiplication, which can be thought of as a process of generating prediction blocks using input pixels (reference pixels) in a matrix multiplication manner.
  • MIP provides a variety of matrices. Different prediction methods can be reflected in different matrices. Using different matrices for the same input pixel will yield different results.
  • reference pixel averaging and interpolation is a design that compromises performance and complexity. For larger blocks, an effect similar to downsampling can be achieved by reference pixel averaging, allowing the input to be adapted to a smaller matrix, while interpolation achieves an upsampling effect. In this way, there is no need to provide MIP matrices for blocks of each size, but only matrices of one or several specific sizes. As the demand for compression performance increases and hardware capabilities improve, more complex MIPs may appear in the next generation of standards.
  • the MIP mode can be simplified from the neural network.
  • the matrix used can be obtained based on training. Therefore, the MIP mode has strong generalization ability and prediction effects that traditional prediction models cannot achieve.
  • the MIP mode can be a model obtained through multiple simplifications of hardware and software complexity for an intra-frame prediction model based on a neural network. Based on a large number of training samples, multiple prediction modes represent a variety of models and parameters, which can compare Good coverage of natural sequences of textures.
  • MIP is somewhat similar to planar mode, but obviously MIP is more complex and more flexible than planar mode.
  • the number of MIP modes may be different. For example, for a coding unit of 4x4 size, the MIP mode has 16 prediction modes; for a coding unit of 8x8 with width equal to 4 or height equal to 4, the MIP mode has 8 prediction modes; for coding units of other sizes, the MIP mode has 6 prediction modes.
  • MIP mode has a transpose function. For prediction modes that conform to the current size, MIP mode can try transposition calculations on the encoder side. Therefore, MIP mode not only requires a flag bit to indicate whether the current coding unit uses MIP mode, but also, if the current coding unit uses MIP mode, an additional transposed flag bit needs to be transmitted to the decoder.
  • the main core point of the DIMD mode is that the decoder uses the same method as the encoder to derive the intra prediction mode. This avoids transmitting the intra prediction mode index of the current coding unit in the code stream to save bit overhead. Purpose.
  • DIMD mode The specific process of DIMD mode can be divided into the following two main steps:
  • Step 1 Export the prediction model.
  • Figure 5 is a schematic diagram of deriving a prediction mode based on DIMD provided by an embodiment of the present application.
  • DIMD derives a prediction mode using pixels in the template in the reconstruction area (reconstruction pixels on the left and upper sides of the current block).
  • the template may include three adjacent rows of reconstructed samples above the current block, three adjacent columns of reconstructed samples on the left, and corresponding adjacent reconstructed samples on the upper left.
  • the template can be configured according to the window (for example, as shown in (a) of Figure 5 or As shown in (b) of Figure 5), multiple gradient values are determined within the template, and each gradient value can be used to configure an intra prediction mode (Intra prediction mode, ipm) that is suitable for its gradient direction.
  • Intra prediction mode Intra prediction mode, ipm
  • the encoder can use the prediction mode adapted to the largest and second largest gradient values among multiple gradient values as the derived prediction mode. For example, as shown in Figure 5 (b), for a 4 ⁇ 4 block size, all pixels whose gradient values need to be determined are analyzed and the corresponding gradient histogram (histogram of gradients) is obtained, for example, as shown in Figure 5 As shown in (c), for blocks of other sizes, all pixels whose gradient values need to be determined are analyzed and the corresponding gradient histograms are obtained; finally, the prediction modes corresponding to the largest and second largest gradients in the gradient histogram are as a derived prediction model.
  • Figure 5 (b) for a 4 ⁇ 4 block size, all pixels whose gradient values need to be determined are analyzed and the corresponding gradient histogram (histogram of gradients) is obtained, for example, as shown in Figure 5
  • the prediction modes corresponding to the largest and second largest gradients in the gradient histogram are as a derived prediction model.
  • the gradient histogram in this application is only an example for determining the derived prediction mode, and can be implemented in a variety of simple forms, which is not specifically limited in this application.
  • this application does not limit the method of statistical gradient histograms.
  • the Sobel operator or other methods may be used to calculate the gradient histograms.
  • Step 2 Export prediction blocks.
  • Figure 6 is a schematic diagram of deriving prediction blocks based on DIMD provided by an embodiment of the present application.
  • the encoder can weight the prediction values of 3 intra prediction modes (planar mode and 2 intra prediction modes derived based on DIMD).
  • the codec uses the same prediction block derivation method to obtain the prediction block of the current block. Assume that the prediction mode corresponding to the largest gradient value is prediction mode 1, and the prediction mode corresponding to the second largest gradient value is prediction mode 2.
  • the encoder determines the following two conditions:
  • the gradient of prediction mode 2 is not 0;
  • Neither prediction mode 1 nor prediction mode 2 is planar mode or DC prediction mode.
  • prediction mode 1 is used to calculate the prediction sample value of the current block, that is, the ordinary prediction prediction process is applied to prediction mode 1; otherwise, that is, the above two conditions are established, then the weighted averaging method is used. Export the predicted block for the current block.
  • the specific method is: the plane mode occupies 1/3 of the weighted weight, and the remaining 2/3 is the total weight of prediction mode 1 and prediction mode 2. For example, divide the gradient amplitude value of prediction mode 1 by the sum of the gradient amplitude values of prediction mode 1. The sum of the gradient amplitude values of prediction mode 2 is used as the weighting weight of prediction mode 1.
  • the gradient amplitude value of prediction mode 2 is divided by the sum of the gradient amplitude value of prediction mode 1 and the gradient amplitude value of prediction mode 2 as the weighting of prediction mode 2.
  • the decoder follows the same steps to obtain the prediction block.
  • Weight(mode1) 2/3*(amp1/(amp1+amp2));
  • Weight(mode2) 1-Weight(PLANAR)-Weight(mode1);
  • mode1 and mode2 represent prediction mode 1 and prediction mode 2 respectively
  • amp1 and amp2 represent the gradient amplitude value of prediction mode 1 and the gradient amplitude value of prediction mode 2 respectively.
  • DIMD mode requires transmitting a flag bit to the decoder, which is used to indicate whether the current encoding unit uses DIMD mode.
  • DIMD uses gradient analysis of reconstructed pixels to screen intra prediction modes, and the two intra prediction modes plus the planar mode can be weighted according to the analysis results.
  • the advantage of DIMD is that if the DIMD mode is selected for the current block, there is no need to indicate which intra prediction mode is used in the code stream. Instead, the decoder itself derives it through the above process, which saves overhead to a certain extent.
  • the technical principles of the TIMD mode are similar to the technical principles of the above-mentioned DIMD mode. They both use the same operation of the codec to derive the prediction mode to save transmission mode index overhead.
  • the TIMD mode can be understood as two main parts. First, the cost information of each prediction mode is calculated according to the template. The prediction mode corresponding to the minimum cost and the second smallest cost will be selected.
  • the prediction mode corresponding to the minimum cost is recorded as prediction mode 1, and the second smallest cost is The prediction mode corresponding to the cost is recorded as prediction mode 2; if the ratio of the next smallest cost value (costMode2) to the minimum cost value (costMode1) meets the preset conditions, such as costMode2 ⁇ 2*costMode1, then prediction mode 1 and prediction mode 2
  • costMode2 next smallest cost value
  • costMode1 costMode2 ⁇ 2*costMode1
  • the corresponding weights of prediction mode 1 and prediction mode 2 are determined in the following manner:
  • weight1 costMode2/(costMode1+costMode2)
  • weight2 1-weight1
  • weight1 is the weight of the prediction block corresponding to prediction mode 1
  • weight2 is the weight of the prediction block corresponding to prediction mode 2.
  • weighted fusion between prediction blocks will not be performed, and the prediction block corresponding to prediction mode 1 will be the prediction block of TIMD.
  • the TIMD mode if the TIMD mode is used to perform intra prediction on the current block, if the reconstruction sample template of the current block does not contain available adjacent reconstruction samples, the TIMD mode selects the planar mode to perform intra prediction on the current block, that is, it does not Perform unweighted fusion. Same as DIMD mode, TIMD mode needs to transmit a flag bit to the decoder to indicate whether the current coding unit uses TIMD mode.
  • Figure 7 is a schematic diagram of a template used by TIMD provided by an embodiment of the present application.
  • the codec can be based on a coding unit with a width equal to 2(M+L1)+1 and a height equal to 2(N+L2)+1 Select the reference template (Reference of template) of the current block to calculate the template of the current block.
  • the TIMD mode selects the planar mode to perform intra prediction on the current block.
  • the available adjacent reconstruction samples may be samples adjacent to the left and upper sides of the current CU in FIG. 7 , that is, there are no available reconstruction samples in the diagonally filled area. That is to say, if there are no available reconstruction samples in the diagonal padding area, the TIMD mode selects the planar mode to perform intra prediction on the current block.
  • the left and upper sides of the current block can theoretically obtain reconstruction values, that is, the template of the current block contains available adjacent reconstruction samples.
  • the decoder can use a certain intra prediction mode to predict on the template, and compare the prediction value and the reconstructed value to obtain the cost of the intra prediction mode on the template.
  • a certain intra prediction mode to predict on the template, and compare the prediction value and the reconstructed value to obtain the cost of the intra prediction mode on the template.
  • TIMD predicts some candidate intra prediction modes on the template, obtains the cost of the candidate intra prediction mode on the template, and replaces the one or two intra prediction modes with the lowest cost as the intra prediction value of the current block. If the template cost difference between the two intra prediction modes is not large, the compression performance can be improved by weighting the prediction values of the two intra prediction modes.
  • the weights of the predicted values of the two prediction modes are related to the above-mentioned costs. For example, the weights are inversely proportional to the costs.
  • TIMD uses the prediction effect of the intra prediction mode on the template to screen the intra prediction mode, and can weight the two intra prediction modes according to the cost on the template.
  • the advantage of TIMD is that if the TIMD mode is selected for the current block, there is no need to indicate which intra prediction mode is used in the code stream. Instead, the decoder itself derives it through the above process, which saves overhead to a certain extent.
  • DIMD mode In the above simple introduction to several intra prediction modes, it is not difficult to find that the technical principles of DIMD mode are close to those of TIMD mode. They both use the decoder to perform the same operation as the encoder to infer the prediction of the current coding unit. model.
  • This prediction mode can save the transmission of the index of the prediction mode when the complexity is acceptable, thereby saving overhead and improving compression efficiency.
  • the DIMD mode and the TIMD mode work better in large areas with consistent texture characteristics. If the texture changes slightly or the template area cannot be covered, Then the prediction effect of this prediction mode is poor.
  • DIMD mode whether for DIMD mode or TIMD mode, prediction blocks obtained based on multiple traditional prediction modes are fused or weighted prediction blocks obtained based on multiple traditional prediction modes are performed.
  • the fusion of prediction blocks can Generating effects that cannot be achieved by a single prediction mode.
  • the DIMD mode introduces the planar mode as an additional weighted prediction mode to increase the spatial correlation between adjacent reconstructed samples and prediction samples, thereby improving the prediction effect of intra prediction, but , Since the prediction principle of planar mode is relatively simple, for some prediction blocks with obvious differences between the upper right corner and the lower left corner, using planar mode as an additional weighted prediction mode may have counterproductive effects.
  • GPM or AWP For GPM or AWP, it also uses two reference blocks with the same size as the current block, but some pixel positions 100% use the pixel value of the corresponding position of the first reference block, and some pixel positions 100% use the second
  • the pixel values at the corresponding positions of the reference block are used, and in the junction area or transition area, the pixel values at the corresponding positions of the two reference blocks are used according to a certain proportion.
  • the weight of the junction area also gradually transitions. How these weights are allocated is determined by the weight export mode of GPM or AWP. Determine the weight of each pixel position according to the weight derivation mode of GPM or AWP.
  • some GPM or AWP modes may not guarantee that some pixel positions must use 100% of the pixel values corresponding to the first reference block.
  • the pixel position 100% uses the pixel value at the corresponding position of the second reference block; at this time, it can also be considered that GPM or AWP uses two reference blocks that are different in size from the current block, that is, each takes the required part as the reference block; that is, The part with a weight other than 0 is used as a reference block, and the part with a weight of 0 is eliminated.
  • This application does not limit its specific implementation.
  • Figure 8 is an example of a weight map corresponding to 64 weight derivation modes of GPM on a square block provided by the embodiment of the present application.
  • GPM has weight maps corresponding to 64 weight derivation modes on square blocks.
  • black means that the weight value of the corresponding position of the first reference block is 0%
  • white means that the weight value of the corresponding position of the first reference block is 100%
  • the gray area is based on color.
  • the difference in depth means that the weight value of the corresponding position of the first reference block is a weight value greater than 0% and less than 100%.
  • the weight value of the corresponding position of the second reference block is 100% minus the weight value of the corresponding position of the first reference block.
  • Figure 9 is an example of dividing lines in the weight derivation mode provided by the embodiment of the present application.
  • the dividing line of the weight derivation mode may be a line composed of points with the same weight of the two prediction modes corresponding to the GPM.
  • the dividing line in the weight matrix of GPM, the dividing line is a line composed of points with the same weight of the two prediction modes corresponding to GPM.
  • the dividing line may be a line composed of points with the same weights of the two prediction modes corresponding to the GPM in the area where the weight changes in the weight matrix of the GPM.
  • the line composed of points with a median weight and the dividing line is a line composed of points corresponding to the median weight.
  • the point with the median weight may be located in the middle of the entire pixel, or it may not be in the middle of the entire pixel. ; Taking weights from 0 to 8 as an example, the median weight can be 4.
  • Figure 10 is an example of a weight map corresponding to 56 weight derivation modes of AWP on a square block provided by the embodiment of the present application.
  • the weight map corresponding to the 56 weight derivation modes of AWP on square blocks As shown in Figure 10, the weight map corresponding to the 56 weight derivation modes of AWP on square blocks.
  • black means that the weight value of the corresponding position of the first reference block is 0%
  • white means that the weight value of the corresponding position of the first reference block is 100%
  • the gray area is based on color.
  • the difference in depth means that the weight value of the corresponding position of the first reference block is a weight value greater than 0% and less than 100%.
  • the weight value of the corresponding position of the second reference block is 100% minus the weight value of the corresponding position of the first reference block.
  • GPM determines the angle and offset based on each weight derivation mode, and then calculates the weight map corresponding to each weight derivation mode.
  • AWP first determines a one-dimensional weight line based on each weight derivation mode, and then uses a method similar to intra-frame angle prediction to spread the one-dimensional weight line over the entire image to obtain a weight map corresponding to each weight derivation mode.
  • the weight map corresponding to each weight derivation mode may also be called a weight matrix.
  • the encoder can determine the corresponding dividing line according to each weight derivation mode, and then determine the corresponding weight matrix based on the dividing line. For example, the encoder can use Table 1 to determine the angle index variable angleIdx and distance index variable distanceIdx corresponding to the weight derivation mode to determine the weight derivation mode merge_gpm_partition_idx.
  • the angle index variable angleIdx and the distance index variable distanceIdx can be considered as variables used to determine the dividing line, that is, They are used to determine the angle and offset of the dividing line respectively.
  • the encoder determines the dividing lines corresponding to each weight derivation mode, it can determine the weight matrix corresponding to each weight derivation mode based on the dividing lines corresponding to each weight derivation mode.
  • each weight export mode can correspond to an angle index variable angleIdx and a distance index variable distanceIdx. Equivalently, each weight derivation mode can correspond to a dividing line. Of course, an angle index variable angleIdx or a distance index variable distanceIdx can correspond to the index of one or more weight derivation modes. Table 1 is only an example of this application and should not be understood as a limitation of this application.
  • the process of generating the GPM prediction sample matrix for one component can be divided into a sub-process, that is, the weighted sample prediction process for GPM for geometric partitioning mode), this process can be called for all three components, but the parameters called are different.
  • This application uses the brightness component as an example to illustrate.
  • nCbW is set to cbWidth
  • nCbH is set to cbHeight.
  • the inputs to the weighted prediction process of GPM include: the width of the current block nCbW; the height of the current block nCbH; 2 (nCbW)x(nCbH) prediction sample matrices predSamplesLA and predSamplesLB; GPM's "partition" angle index variable angleIdx; GPM's Distance index variable distanceIdx; component index variable cIdx, for example, when cIdx is 0, it can be used to represent the brightness component.
  • the prediction sample matrix pbSamples[x][y] can be derived as follows:
  • nW, nH, shift1, offset1, displacementX, displacementY, partFlip and shiftHor can be exported as follows:
  • variables offsetX and offsetY can be exported as follows:
  • offsetY ((-nH)>>1)+(angleIdx ⁇ 16?(distanceIdx*nH)>>3:-((distanceIdx*nH)>>3));
  • offsetX ((-nW)>>1)+(angleIdx ⁇ 16?(distanceIdx*nW)>>3:-((distanceIdx*nW)>>3);
  • weightIdx (((xL+offsetX) ⁇ 1)+1)*disLut[displacementX]+(((yL+offsetY) ⁇ 1)+1)*disLut[displacementY];
  • weightIdxL partFlip? 32+weightIdx:32–weightIdx;
  • pbSamples[x][y] Clip3(0,(1 ⁇ BitDepth)-1,(predSamplesLA[x][y]*wValue+predSamplesLB[x][y]*(8-wValue)+offset1)>> shift1).
  • pbSamples[x][y] represents the predicted sample at (x, y) point.
  • wValue represents the weight of the predicted value predSamplesLA[x][y] of the prediction matrix of a prediction mode at point (x, y), and (8-wValue) is another prediction mode at point (x, y) The weight of the predicted value predSamplesLB[x][y] of the prediction matrix.
  • weight derivation mode you can use it to derive a weight value wValue for each point, and then calculate a GPM predicted value pbSamples[x][y].
  • the weight wValue does not need to be written in the form of a matrix, but it can be understood that if the wValue of each position is saved in a matrix, then it is a weight matrix. Calculate the weight of each point separately and weight it to obtain the predicted value of GPM, or calculate all the weights and then weight them uniformly to obtain the predicted sample matrix of GPM.
  • the principle is the same.
  • the weight matrix is used in many descriptions in this article to make the expression easier to understand and to use the weight matrix to draw pictures more intuitively. In fact, it can also be described according to the weight of each position.
  • the weight matrix derivation mode can also be said to be the weight derivation mode, which is not specifically limited in this application.
  • GPM and AWP use masks of the weights of two reference blocks, namely the above-mentioned weight map or weight matrix. This mask determines the weight of the two reference blocks when generating the prediction block, or it can be simply understood as that part of the position of the prediction block comes from the first reference block and part of the position comes from the second reference block, while the transition area (blending area) is weighted by the corresponding positions of the two reference blocks, thereby making the transition smoother.
  • GPM and AWP do not divide the current block into two CUs or PUs, so the transformation, quantization, inverse transformation, inverse quantization, etc. of the residual after prediction are also processed on the current block as a whole.
  • GPM can use a weight matrix to combine two inter prediction blocks.
  • This application extends this to combining two arbitrary prediction blocks. For example, two inter prediction blocks, two intra prediction blocks, one inter prediction block plus one intra prediction block.
  • prediction blocks in intra block copy (IBC) mode or prediction blocks in palette (palette) mode can be used as one or two prediction blocks.
  • IBC intra block copy
  • palette palette mode
  • the prediction mode can be understood as information according to which the codec can generate a prediction block for the current block.
  • the prediction mode may be a certain intra prediction mode, such as DC mode, plane mode, various intra angle prediction modes, etc.
  • the prediction mode may be a fusion (merge) mode, a fusion with motion vector difference (MMVD) mode, or an advanced motion vector prediction mode (Advanced Motion Vector Prediction, AMVP).
  • the prediction mode may be unidirectional prediction, bidirectional prediction or multi-hypothesis prediction.
  • the inter-frame prediction mode uses unidirectional prediction and a piece of motion information can be determined, the prediction block can be determined based on the motion information. If the inter-frame prediction mode uses bidirectional prediction and two pieces of motion information can be determined, the prediction block can be determined based on the motion information.
  • FIG 11 is a schematic diagram of a GPM or AWP provided by an embodiment of the present application.
  • the information that GPM needs to determine can be expressed as 1 weight derivation mode and 2 prediction modes.
  • the weight derivation mode is used to determine the weight matrix or weight, and the two prediction modes determine a prediction block or prediction value respectively.
  • Weight derivation mode is also known as partition mode or weight matrix derivation mode.
  • the two prediction modes may be the same or different prediction modes, including but not limited to intra prediction mode, inter prediction mode, IBC mode and palette mode.
  • the current block When encoding, the current block will be predicted first.
  • the prediction uses spatial or temporal correlation performance to obtain an image that is the same or similar to the current block.
  • the predicted block and the current block are exactly the same, but it is difficult to guarantee that this is the case for all blocks in a video.
  • the predicted block is usually very similar to the current block, but there are differences.
  • due to irregular motion, distortion, occlusion, brightness, etc. changes in the video it is difficult to completely predict the current block.
  • the hybrid coding framework will subtract the predicted image from the original image of the current block to obtain the residual image, or the current block minus the predicted block will obtain the residual block.
  • Residual blocks are usually much simpler than the original image, so prediction can significantly improve compression efficiency.
  • the residual block is not directly encoded, but is usually transformed first. Transformation is to transform the residual image from the spatial domain to the frequency domain and remove the correlation of the residual image. After the residual image is transformed into the frequency domain, since the energy is mostly concentrated in the low-frequency region, most of the transformed non-zero coefficients are concentrated in the upper left corner. Next, quantization is used for further compression. And since the human eye is not sensitive to high frequencies, a larger quantization step size can be used in high-frequency areas.
  • Image transformation technology is a transformation of the original image in order to be able to represent the original image with an orthogonal function or orthogonal matrix. This transformation is two-dimensional linear and reversible.
  • the original image is generally called a spatial domain image
  • the transformed image is called a converted domain image (also called frequency domain).
  • the converted domain image can be inversely transformed into a spatial domain image. After image transformation, on the one hand, it can more effectively reflect the characteristics of the image itself, on the other hand, it can also concentrate energy on a small amount of data, which is more conducive to the storage, transmission and processing of images.
  • Transformation methods include but are not limited to: Discrete Cosine Transform (DCT) and Discrete Sine Transform (DST). Since DCT has strong energy concentration characteristics, only some areas (such as the upper left corner area) of the original image have non-zero coefficients after DCT transformation.
  • DCTs that can be used in video codecs include but are not limited to DCT2 and DCT8 types; DSTs that can be used in video codecs include but are not limited to DST7 types.
  • DCT2 type is a commonly used transformation in video compression standards
  • DCT8 type and DST7 type can be used in VVC. It is worth noting that transformation is very useful in general video compression, but not all blocks must be transformed. In some cases, transformation is not as good as compression without transformation. Therefore, in some cases, the encoder You can choose whether the current block uses transformations.
  • the encoder When the encoder transforms the current block in the current image, it can use a basis function or a basis image to transform the residual block of the current block.
  • the basis image is the graphical representation of the basis function.
  • FIG. 12 is an example of a DCT2-type base image provided by an embodiment of the present application.
  • the base image of DCT2 type can be a base image composed of 8 ⁇ 8 small blocks drawn according to the basis function, where each small block is composed of 8 ⁇ 8 elements (sub-blocks).
  • a base image composed of 8 ⁇ 8 small blocks can be used to transform a block of 8x8 block size, and an 8x8 transformation coefficient matrix is obtained.
  • DCT8 type and DST7 type to perform basic transformation on the residual block, that is, multiple transform mode selection (MTS) in VVC.
  • MTS multiple transform mode selection
  • the transformation type corresponding to the basis function used by the basic transformation can also be called the transformation kernel type used by the basic transformation.
  • MTS multiple transform mode selection
  • the basic transformation can also be called the core transform.
  • MTS can select the transformation core type through some syntax elements. An MTS that selects the transformation core type through syntax elements is listed below in conjunction with Table 3.
  • the transformation kernel type of the basic transformation in both the horizontal and vertical directions uses DCT2. If the value of MTS_CU_flag is 1, the value of MTS_Hor_flag is 0, and the value of MTS_Ver_flag is 0, then the transformation core type in the horizontal direction uses DST7, and the transformation core type in the vertical direction uses DST7.
  • VVC uses a syntax element mts_idx to determine the transformation core type of the basic transformation.
  • trTypeHor represents the transformation core type of horizontal transformation
  • trTypeVer represents the transformation core type of vertical transformation
  • trTypeHor and trTypeVer are 0 indicating DCT2 type transformation, 1 indicating DST7 type transformation, and 2 indicating DCT8 type transformation.
  • the basic transformation can also take advantage of this correlation.
  • One approach is to group the transform kernel types of MTS according to intra prediction mode. An example of grouping is shown in the table below.
  • the corresponding transform core type group with index 0 of MTS is selected.
  • the mode with index 0 in VVC is Planar, and the mode with index 1 in VVC is DC. , DC and Planar both produce flatter prediction values.
  • the index of the intra prediction mode is 2 to 12
  • the corresponding transform core type group with the index of MTS 1 is selected. According to the intra prediction mode picture, it can be seen that angles 2 to 12 point to the lower left direction.
  • each transformation kernel type group may have a selection of transformation kernel types for horizontal transformation and vertical transformation, or there may be multiple selections of transformation kernel types for horizontal transformation and vertical transformation, that is to say, After selecting which transform core type group according to the intra prediction mode, further subdivision can be made. For example, the transform kernel type can be further selected through some identification or block size information, etc., which will not be described again here.
  • the basic transform can select the transform core type group according to the intra prediction mode. It can also be seen that this method of selecting the transformation core type group of the basic transformation according to the intra prediction mode may also infer more detailed grouping of basic transformations in the future, which is not specifically limited in this application.
  • the transformation kernel types involved in the main transformation of this application can also be called transformation matrices, transformation types, transformation kernels, and other terms with similar or identical meanings.
  • the transformation kernel type group involved in the main transformation can also be called transformation matrices. Group, transformation type group, transformation core group and other terms with similar or identical meanings are not specifically limited in this application. That is to say, the selection transformation kernel type or transformation kernel type group involved in this application can also be called the selection transformation matrix or the transformation matrix group, it can also be called the selection transformation type or the selection transformation type group, and it can also be called the selection transformation. Kernel or transform kernel group.
  • the transformation core type or transformation type can include DCT2, DCT8, DST7, etc., and can also include DCT5, DST4, DST1 or identity transform (Identity Transform, IDTR), etc.
  • blocks of different sizes can use transformation kernel types of corresponding sizes, which will not be described in detail in this application.
  • the one-dimensional transformation of the horizontal and vertical directions is performed in two steps. For example, perform horizontal transformation first and then vertical transformation, or perform vertical transformation first and then horizontal transformation.
  • the above transformation method is more effective for horizontal and vertical textures, but is less effective for oblique textures. Since horizontal and vertical textures are the most common, the above transformation method is very useful for improving compression efficiency. However, with the development of technology, only processing the residual texture of horizontal and vertical directions can no longer meet the demand for compression efficiency.
  • this application introduces the concept of secondary transformation, that is, the encoder can perform secondary transformation based on the primary transform to improve compression efficiency.
  • the main transformation can be used to process textures in horizontal and vertical directions.
  • the main transformation can also be called a basic transformation.
  • the main transformation includes but is not limited to: the above-mentioned DCT2 type, DCT8 type, and DST7 type transformation.
  • Secondary transformation is used to process oblique textures.
  • secondary transformation includes but is not limited to: low frequency non-separable transform (LFNST).
  • LFNST low frequency non-separable transform
  • the secondary transform is used after the main transform and before quantization.
  • the secondary transform is used after the inverse quantization and before the inverse primary transform.
  • Figure 13 is an example of LFNST provided by the embodiment of this application.
  • LFNST performs a secondary transformation on the low-frequency coefficients in the upper left corner after the basic transformation.
  • the main transform concentrates the energy to the upper left corner by decorrelating the image.
  • the secondary transformation decorrelates the low-frequency coefficients of the primary transformation.
  • 16 coefficients are input to a 4x4 LFNST, 8 coefficients are output; when 64 coefficients are input to an 8x8 LFNST, 16 coefficients are output.
  • 8 coefficients are input to the 4x4 inverse LFNST, 16 coefficients are output; when 16 coefficients are input to the 8x8 inverse LFNST, 64 coefficients are output.
  • the encoder When the encoder performs secondary transformation on the current block in the current image, it can use a certain transformation matrix in the selected transformation matrix group to transform the residual block of the current block.
  • the transformation matrix may refer to a matrix used to transform a certain oblique texture
  • the transformation matrix group may include a matrix used to transform some similar oblique textures.
  • Figure 14 is an example of a transformation matrix group of LFNST provided by the embodiment of the present application.
  • LFNST may have 4 sets of transformation matrices, and the transformation matrices in the same set of transformation matrices have similar oblique textures.
  • the transformation matrix group shown in (a) in Figure 14 may be a transformation matrix group with an index of 0, and the transformation matrix group shown in (b) of Figure 14 may be a transformation matrix group with an index of 1,
  • the transformation matrix group shown in (c) in FIG. 14 may be a transformation matrix group with an index of 2
  • the transformation matrix group shown in (d) of FIG. 14 may be a transformation matrix group with an index of 3.
  • transformation matrix involved in the secondary transformation in this application can also be called transformation kernel, transformation kernel type or basis function and other terms with similar or identical meanings.
  • the transformation matrix group involved in the secondary transformation can also be called Terms such as transformation kernel group, transformation kernel type group or basis function group have similar or identical meanings, which are not specifically limited in this application. That is to say, the selection transformation matrix or transformation matrix group involved in this application can also be called the selection transformation kernel type or the transformation core type group. It can also be called the selection transformation type or the selection transformation type group. It can also be called the selection transformation. Kernel or transform kernel group.
  • Intra-frame prediction uses the reconstructed pixels around the current block as a reference to predict the current block. Since current videos are encoded from left to right and from top to bottom, the reference pixels that can be used by the current block are usually on the left and upper sides. . Angle prediction tiles the reference pixels to the current block according to the specified angle as the prediction value, which means that the predicted block will have obvious directional texture, and the residual of the current block after angle prediction will also reflect obvious angles statistically. characteristic. Therefore, the transformation matrix selected by LFNST can be bound to the intra prediction mode. That is, after the intra prediction mode is determined, LFNST can use a set of transformation matrices whose texture direction is adapted to the angular characteristics of the intra prediction mode. .
  • LFNST has a total of 4 groups of transformation matrices, each group has 2 transformation matrices.
  • Table 6 shows the correspondence between intra prediction modes and transformation matrix groups.
  • intra prediction modes 0 to 81 can be associated with four sets of transformation matrix indexes.
  • the cross-component prediction modes used by chroma intra-frame prediction are 81 to 83, while luma intra-frame prediction does not have these modes.
  • the transformation matrix of LFNST can be transposed to use one transformation matrix group to handle more angles.
  • intra prediction modes 13 to 23 and intra prediction modes 45 to 55 both correspond to transformation matrix group 2, but , intra prediction modes 13 to 23 are obviously close to horizontal modes, while intra prediction modes 45 to 55 are obviously close to vertical modes, and the corresponding modes of intra prediction modes 45 to 55 need to be transformed after inverse transformation Adapt by transposing.
  • the encoding end can determine which set of transformation matrices LFNST uses based on the intra prediction mode used by the current block, and then determine the transformation matrix to be used in the determined set of transformation matrices. Equivalently, the correlation between the intra prediction mode and the LFNST transformation matrix group can be utilized, thereby reducing the transmission of the selected LFNST transformation matrix in the code stream. Whether the current block will use LFNST, and if LFNST is used, whether to use the first or second one in a group can be determined by the code stream and some conditions.
  • LFNST can also be designed to be more complex. For example, use larger transformation matrices, more transformation matrix groups, and use more transformation matrices per transformation matrix group.
  • Table 7 shows another correspondence relationship between intra prediction modes and transformation matrix groups.
  • Index of the transformation matrix group 26 25 twenty four twenty three twenty two twenty one 20 19 18 17 16 15 14 13
  • each transformation matrix group uses 3 transformation matrices.
  • the corresponding relationship between the transformation matrix group and the intra prediction mode can be realized as follows: for the intra prediction mode 0 to 34, the forward direction corresponds to the transformation matrix group 0 to 34, that is, the larger the number of the prediction mode, the larger the index of the transformation matrix group; For intra prediction modes 35 to 67, due to transposition, they correspond inversely to 2 to 33, that is, the larger the number of the prediction mode, the smaller the index of the transformation matrix group; for the remaining prediction modes, they can all correspond uniformly to the transformation matrix group with index 2. That is to say, if transposition is not considered, an intra prediction mode corresponds to a transformation matrix group. According to this design, the residual corresponding to each intra prediction mode can obtain a more suitable LFNST transformation matrix, and the compression performance will also be improved.
  • the wide-angle mode can also achieve one-to-one design, but this design has a low cost performance, and this application will not explain it in detail.
  • the transformation matrix group to which the planar mode is adapted in this application can be used as the transformation matrix group adapted to the MIP.
  • LFNST is only an example of quadratic transformation and should not be understood as a limitation on quadratic transformation.
  • LFNST is a non-separable secondary transformation.
  • a separable secondary transformation can also be used to improve the compression efficiency of the residual of oblique textures.
  • Figure 15 is a schematic block diagram of the decoding framework 200 provided by the embodiment of the present application.
  • the decoding framework 200 may include an entropy decoding unit 210, an inverse transform and inverse quantization unit 220, a residual unit 230, an intra prediction unit 240, an inter prediction unit 250, a loop filtering unit 260, and a decoded image buffer. Unit 270.
  • the entropy decoding unit 210 After the entropy decoding unit 210 receives and decodes the code stream, it obtains the prediction block and the frequency domain residual block. For the frequency domain residual block, the inverse transform and inverse quantization unit 220 performs steps such as transformation and inverse quantization to obtain the time domain residual block. block, the residual unit 230 superposes the prediction block predicted by the intra prediction unit 240 or the inter prediction unit 250 to the time domain residual block after transformation and inverse quantization by the inverse transformation and inverse quantization unit 220, to obtain a reconstructed block .
  • Figure 16 is a schematic flow chart of the decoding method 300 provided by the embodiment of the present application. It should be understood that the decoding method 300 can be performed by a decoder. For example, it is applied to the decoding framework 200 shown in FIG. 15 . For the convenience of description, the following takes the decoder as an example.
  • the decoding method 300 may include some or all of the following:
  • the decoder decodes the code stream to obtain the first transform coefficient of the current block
  • S320 The decoder performs a first transformation on the first transform coefficient to obtain a second transform coefficient of the current block
  • the decoder performs a second transformation on the second transformation coefficient to obtain the residual block of the current block
  • S340 The decoder predicts the current block based on the first intra prediction mode and the second intra prediction mode derived from the prediction mode derivation mode, and obtains the prediction block of the current block;
  • the decoder obtains a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.
  • This application introduces the first transformation based on the prediction mode derivation mode and the second transformation, which can improve the decompression efficiency of the current block.
  • the first transformation may be LFNST, that is, this application combines LFNST into the prediction mode derivation mode, which can improve the compression efficiency of the residual of oblique textures.
  • the adaptation method of prediction mode derivation mode and LFNST is also applicable to other secondary transformation methods.
  • LFNST is a non-separable secondary transform.
  • the prediction mode derivation mode can also be applied to separable secondary transforms, which is not specifically limited in this application.
  • the prediction mode derivation mode may include a decoder-side intra mode derivation DIMD mode or a template-based intra mode derivation TIMD mode.
  • the S320 may include:
  • the first identifier is used to identify that the prediction mode derivation mode is allowed to be used to predict the image block in the current sequence
  • the second identifier is used to identify that the first transformation is allowed to be used to predict the current sequence
  • the current sequence is an image sequence including the current block.
  • the first identification may be used to control whether the current sequence uses the prediction mode to derive the mode.
  • the identifier allows the prediction mode derivation mode to be used to predict the image block in the current sequence. If the value of the first identifier is The second value indicates that the prediction mode derivation mode is not allowed to be used to predict the image blocks in the current sequence.
  • the first numerical value is 0 and the second numerical value is 1.
  • the first numerical value is 1 and the second numerical value is 0.
  • the first numerical value or the second numerical value can also be other values.
  • the second identification is used to control whether the current sequence uses the first transformation.
  • the identifier allows the image block in the current sequence to be transformed using the first transformation, and if the value of the second identifier is the third If the value is four, it indicates that the first transformation is not allowed to be used to transform the image blocks in the current sequence.
  • the third numerical value is 0 and the fourth numerical value is 1.
  • the third numerical value is 1 and the fourth numerical value is 0.
  • the third numerical value or the fourth numerical value can also be other values.
  • the first flag is recorded as sps_timd/dimd_enabled_flag and the second flag is recorded as sps_lfnst_enabled_flag, then when the values of sps_timd/dimd_enabled_flag and sps_lfnst_enabled_flag are both 1, the first transformation coefficient is performed, Obtain the second transform coefficient.
  • the first identification is used to identify that the prediction mode derivation mode is not allowed to be used to predict the image blocks in the current sequence
  • the second identification is used to identify that the use of the prediction mode derivation mode is not allowed
  • the first identifier and/or the second identifier can also be replaced by an image (picture), a slice (slice), the largest coding unit (LCU), or a coding unit.
  • Identification of levels such as tree unit (Coding Tree Unit, CTU), coding unit (coding unit, CU), prediction unit (prediction unit, PU), or transformation unit (transform unit, TU).
  • images, slices, LCUs, CTUs, CUs, and PUs may be added to determine whether to use the prediction mode derivation mode or whether to use the first transformation. or TU and other level identifiers, which are not specifically limited in the embodiments of this application.
  • the S320 may include:
  • the first identifier is used to identify that the prediction mode derivation mode is allowed to be used to predict the image block in the current sequence
  • the second identifier is used to identify that the first transformation is allowed to be used to predict the current sequence
  • a first transformation is performed on the first transformation coefficient to obtain the first transformation coefficient. Two transformation coefficients.
  • the third identifier is used to control whether the prediction mode derivation mode and the first transformation can be shared.
  • the identifier allows the image block in the current sequence to use the prediction mode derivation mode and the first transformation at the same time; if the third identifier is the sixth value, then the image blocks in the current sequence are not allowed to use the prediction mode derivation mode and the first transformation at the same time.
  • the fifth numerical value is 0 and the sixth numerical value is 1.
  • the fifth numerical value is 1 and the sixth numerical value is 0.
  • the fifth numerical value or the sixth numerical value can also be other values.
  • the first flag is recorded as sps_timd/dimd_enabled_flag
  • the second flag is recorded as sps_lfnst_enabled_flag
  • the third flag is recorded as sps_timd/dimd_lfnst_enabled_flag
  • the third identifier can also be replaced by a picture, a slice, a largest coding unit (LCU), a coding tree unit (CTU) ), coding unit (CU), prediction unit (PU) or transform unit (TU) and other levels of identification.
  • LCU largest coding unit
  • CTU coding tree unit
  • CU coding unit
  • PU prediction unit
  • TU transform unit
  • the S320 may include:
  • the decoder performs a first transformation on the first transformation coefficient to obtain the second transformation coefficient when the height and/or width of the current block is greater than or equal to the first threshold.
  • the device determines the height and/or width of the current block, and when the height and/or width of the current block is greater than or equal to the first threshold, performs a first transformation on the first transform coefficient to obtain the The second transformation coefficient.
  • the first threshold may be 4, 8, 16, 32, 64 or other values.
  • the method 300 may further include:
  • the decoder determines a set of transformation matrices used by the first transformation.
  • the prediction mode derivation mode combines two prediction modes (i.e., the first intra prediction mode and the second intra prediction mode) to predict the current block, different intra prediction modes are used to predict the current block.
  • the prediction blocks obtained by prediction may have different texture characteristics. Therefore, if the prediction mode derivation mode is selected for the current block, it means that the first intra prediction mode may cause the prediction block of the current block to exhibit a texture characteristic.
  • the second intra prediction mode may cause the prediction block of the current block to exhibit another texture characteristic; in other words, after predicting the current block, from a statistical point of view, the residual block of the current block will also exhibit two texture characteristics.
  • the decoder needs to first determine a transformation matrix group adapted to its characteristics.
  • the transformation matrix group used in the first transformation is usually a transformation matrix group defined based on a single intra prediction mode, the related scheme for determining the transformation matrix used in the first transformation needs to be further improved for the prediction mode derivation mode, as follows. Various implementations are illustrated.
  • the first transformation uses the same transformation matrix set as the transformation matrix set for planar mode or direct current DC mode adaptation.
  • the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder classifies the prediction mode derivation mode and the planar mode (or DC mode) into one category, and determines the derivation mode according to the planar mode ( or DC mode) adapts the set of transformation matrices used for the first transformation.
  • the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the encoder can return the prediction mode of the current block to planar mode (or DC mode), so that the decoder will match the planar mode ( or DC mode) adapted transformation matrix group, determined as the transformation matrix group used for the first transformation.
  • the decoder when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder can consider that the transformation matrix group used in the first transformation for the current block can be the same as the planar mode (or DC mode). ) adapted transformation matrix group.
  • the transformation matrix group adapted to the planar mode or the direct current DC mode is determined as the first transformation
  • the transformation matrix group used can not only decode the current block based on the prediction mode derivation mode and the first transformation, but also ensure that the texture characteristics of the transformation matrix group used by the first transformation are as close as possible to the texture of the residual block of the current block.
  • the decoder determines a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode; wherein the transformation matrix group used by the first transformation is the same as the transformation matrix group used by the first transformation.
  • the third intra prediction mode adaptation transformation matrix group is the same.
  • the decoder may determine the third intra prediction mode adapted transformation matrix group as the transformation matrix group used for the first transformation.
  • the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder determines the third intra prediction mode based on the first intra prediction mode and the second intra prediction mode. prediction mode, and group the prediction mode derivation mode and the third intra prediction mode into one category, so that the decoder can adapt the set of transformation matrices for the first transformation according to the third intra prediction mode.
  • the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the encoder can return the prediction mode of the current block to the third intra prediction mode, so that the decoder will match the third intra prediction mode.
  • the transformation matrix group for prediction mode adaptation is determined as the transformation matrix group used for the first transformation.
  • the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder can consider that the transformation matrix group used in the first transformation for the current block can be the same as the third intra prediction mode.
  • a set of adapted transformation matrices are provided.
  • the decoder may first determine the third intra prediction mode without explicitly determining the transformation matrix group used for the first transformation through the third intra prediction mode. . Instead, the transformation matrix group adapted to the third intra prediction mode is directly used as the transformation matrix group used for the first transformation.
  • the decoder determines the default prediction mode in the first intra prediction mode and the second intra prediction mode as the third intra prediction mode; or the decoder determines the default prediction mode in the first intra prediction mode and the second intra prediction mode based on the The weight of the first intra prediction mode and/or the weight of the second intra prediction mode determines the third intra prediction mode; or the decoder determines the third intra prediction mode based on the type of the first intra prediction mode and the second intra prediction mode.
  • the type of the intra prediction mode determines the third intra prediction mode; or the decoder determines the third frame based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode. Intra prediction mode.
  • the first intra prediction mode and the second intra prediction mode may be selected.
  • the prediction mode determines the set of transformation matrices used by the first transformation.
  • the first intra prediction mode can be used to determine in any case, that is, the transformation matrix group adapted to the first intra prediction mode is determined as the transformation used for the first transformation in any case.
  • Matrix group alternatively, it can be determined using the second intra prediction mode in any case, that is, the transformation matrix group adapted to the second intra prediction mode is determined as the transformation matrix used for the first transformation in any case. Group.
  • the first intra prediction mode can be used to determine in some cases, that is, in some cases, the transformation matrix group adapted to the first intra prediction mode is determined as the first transformation. Transformation matrix group; alternatively, it can be determined using the second intra-frame prediction mode in some cases, that is, in some cases, the transformation matrix group adapted to the second intra-frame prediction mode is determined as the first transformation. Transformation matrix group; even, in some cases, the plane mode or DC mode is used to determine, that is, in some cases, the transformation matrix group adapted to the plane mode or DC mode is determined as the first transformation. Transformation matrix group.
  • the so-called determination using a certain prediction mode means that the prediction mode derivation mode is classified into the same category as the certain prediction mode, so that the decoder can adapt the transformation matrix group used for the first transformation according to the certain prediction mode.
  • the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, it can return the certain prediction mode so that the decoder can adapt to the first prediction mode according to the certain prediction mode.
  • a set of transformation matrices for the transformation In other words, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder can consider that the transformation matrix group used in the first transformation of the current block can be the same as the certain prediction mode.
  • a set of adapted transformation matrices when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder can consider that the transformation matrix group used in the first transformation of the current block can be the same as the certain
  • the first frame when the decoder determines the third intra prediction mode based on the weight of the first intra prediction mode and/or the weight of the second intra prediction mode, the first frame may be The intra prediction mode with the largest weight among the intra prediction mode and the second intra prediction mode is determined as the third intra prediction mode.
  • the intra prediction mode with the greater weight is given priority.
  • the priority is higher than the intra prediction mode with small weight. For example, if the weight of the first intra prediction mode is greater than the weight of the second intra prediction mode, the first intra prediction mode is determined as the third intra prediction mode, and if the third intra prediction mode If the weight of the second intra prediction mode is greater than the weight of the first intra prediction mode, the second intra prediction mode is determined as the third intra prediction mode.
  • the decoder determines the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode, if the first intra prediction mode and the second intra prediction mode includes an angle prediction mode and a non-angle prediction mode, then the angle prediction mode is determined as the third intra prediction mode.
  • the angle prediction mode is used as the third intra prediction mode.
  • the priority is higher than the non-angle prediction mode as the third intra prediction mode.
  • the first intra-frame prediction mode is an angle prediction mode and the second intra-frame prediction mode is a non-angle prediction mode (such as planar mode or DC mode)
  • the first intra-frame prediction mode ie, angle prediction mode is mode
  • the first prediction The intra prediction mode corresponding to the angle is determined as the third intra prediction mode, wherein the first prediction angle is based on the prediction angle of the first intra prediction mode and the prediction of the second intra prediction mode.
  • Angle determination If the absolute value of the difference between the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode is greater than the second threshold, then the planar mode or the direct current DC mode is Determined to be the third intra prediction mode.
  • the decoder determines the third intra prediction mode based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode, if the first intra prediction mode If the prediction angle is relatively close to the prediction angle of the second intra prediction mode, the decoder can set the first intra prediction mode, the second intra prediction mode, or the prediction angle to be between the prediction angle of the first intra prediction mode and the second intra prediction mode.
  • Intra prediction mode Intra prediction mode with intermediate prediction angle. For example, if the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode are significantly different, the decoder may determine the planar mode or the DC mode as the third intra prediction mode.
  • the decoder determines the transformation matrix group adapted for the prediction mode derivation mode as the transformation matrix group used for the first transformation.
  • the decoder may derive a mode definition adapted or dedicated set of transform matrices for the prediction mode.
  • the first transformation is used to process textures in the current block in an oblique direction
  • the second transformation is used to process textures in the current block in a horizontal direction and in a vertical direction. texture on.
  • the first transformation at the decoding end is the inverse transformation of the first transformation at the encoding end
  • the second transformation at the decoding end is the inverse transformation of the second transformation at the encoding end
  • the first transformation is the secondary transformation mentioned above
  • the second transformation can be the basic transformation or main transformation mentioned above.
  • the first transformation can be the upper transformation.
  • the inverse transformation or inverse transformation of the quadratic transformation involved in this article For example, for the encoding end, the first transformation can be inverse LFNST, and the second transformation can be DCT2 type, DCT8 type or DST7 type, etc.; for the decoder, the first transformation can be inverse LFNST. (Inverse) LFNST, the second transformation may be an inverse (inverse) DCT2 type, an inverse (inverse) DCT8 type or an inverse (inverse) DST7 type, etc.
  • the decoding method according to the embodiment of the present application has been described in detail from the perspective of the decoder above.
  • the encoding method according to the embodiment of the present application will be described from the perspective of the encoder with reference to FIG. 17 below.
  • Figure 17 is a schematic flow chart of the encoding method 400 provided by the embodiment of the present application. It should be understood that the encoding method 400 can be performed by an encoder. For example, it is applied to the coding framework 100 shown in FIG. 1 . For ease of description, the following uses an encoder as an example.
  • the encoding method 400 may include:
  • S410 Predict the current block based on the first intra prediction mode and the second intra prediction mode derived from the prediction mode derivation mode, and obtain the prediction block of the current block;
  • the first transformation at the decoding end is the inverse transformation of the fourth transformation at the encoding end
  • the second transformation at the decoding end is the inverse transformation of the third transformation at the encoding end.
  • the third transformation is the basic transformation or the main transformation mentioned above
  • the fourth transformation is the secondary transformation mentioned above
  • the first transformation is the inverse transformation (or inverse transformation) of the secondary transformation.
  • Transform the second transformation may be the base transformation or the inverse transformation (or inverse transformation) of the main transformation.
  • the first transformation may be an inverse (inverse) LFNST
  • the second transformation may be an inverse (inverse) DCT2 type, an inverse (inverse) DCT8 type, or an inverse (inverse) DST7 type, etc.
  • the third transformation may be DCT2 type, DCT8 type or DST7 type, etc.
  • the fourth transformation may be LFNST.
  • the prediction mode derivation mode includes a decoder-side intra mode derivation mode or a template-based intra mode derivation mode.
  • the S450 may include:
  • the first identification is used to identify that the prediction mode derivation mode is allowed to be used to predict the image blocks in the current sequence
  • the second identification is used to identify that the fourth transformation is allowed to be used to predict the current sequence. Transform the image blocks in .
  • the S450 may include:
  • the third identification is used to identify that image blocks in the current sequence are allowed to use the prediction mode derivation mode and the fourth transformation at the same time.
  • the S440 may include:
  • the fourth transform is performed on the third transform coefficient to obtain the fourth transform coefficient.
  • the method 400 may further include:
  • the fourth transformation uses the same transformation matrix set as the transformation matrix set for planar mode or direct current DC mode adaptation.
  • determining the transformation matrix group used for the fourth transformation includes:
  • the transformation matrix group used in the fourth transformation is the same as the transformation matrix group used in the third intra prediction mode adaptation.
  • determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode includes:
  • the third intra prediction mode is determined based on the weight of the first intra prediction mode and/or the weight of the second intra prediction mode; or
  • the third intra prediction mode is determined based on the type of the first intra prediction mode and the type of the second intra prediction mode; or
  • the third intra prediction mode is determined based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode.
  • determining the third intra prediction mode based on the weight of the first intra prediction mode and/or the weight of the second intra prediction mode includes:
  • the intra prediction mode with the largest weight among the first intra prediction mode and the second intra prediction mode is determined as the third intra prediction mode.
  • determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode includes:
  • the angle prediction mode is determined as the third intra prediction mode.
  • determining the third intra prediction mode based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode includes:
  • the intra prediction corresponding to the first prediction angle is mode, determined to be the third intra prediction mode, wherein the first prediction angle is determined according to the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode:
  • the planar mode or the direct current DC mode is determined as the The third intra prediction mode is described.
  • determining the transformation matrix group used for the fourth transformation includes:
  • the transformation matrix group adapted to the prediction mode derivation mode is determined as the transformation matrix group used for the fourth transformation.
  • the fourth transformation is used to process the texture in the oblique direction in the current block
  • the third transformation is used to process the texture in the current block in the horizontal direction and in the vertical direction. texture on.
  • the encoding method can be understood as the reverse process of the decoding method. Therefore, for the specific solution of the encoding method 400, please refer to the relevant content of the decoding method 300. For the convenience of description, this application will not repeat it again.
  • the size of the sequence numbers of the above-mentioned processes does not mean the order of execution.
  • the execution order of each process should be determined by its functions and internal logic, and should not be used in this application.
  • the execution of the examples does not constitute any limitations.
  • Figure 18 is a schematic block diagram of the decoder 500 according to the embodiment of the present application.
  • the decoder 500 may include:
  • Decoding unit 510 used to decode the code stream to obtain the first transform coefficient of the current block
  • Transformation unit 520 used for:
  • Prediction unit 530 configured to predict the current block based on the first intra prediction mode and the second intra prediction mode derived from the prediction mode derivation mode, and obtain a prediction block of the current block;
  • the reconstruction unit 540 is configured to obtain a reconstruction block of the current block based on the prediction block of the current block and the residual block of the current block.
  • the prediction mode derivation mode includes a decoder-side intra mode derivation mode or a template-based intra mode derivation mode.
  • the transformation unit 520 is specifically used to:
  • the first identifier is used to identify that the prediction mode derivation mode is allowed to be used to predict the image block in the current sequence
  • the second identifier is used to identify that the first transformation is allowed to be used to predict the current sequence
  • the transformation unit 520 is specifically used to:
  • the first identifier is used to identify that the prediction mode derivation mode is allowed to be used to predict the image block in the current sequence
  • the second identifier is used to identify that the first transformation is allowed to be used to predict the current sequence
  • a first transformation is performed on the first transformation coefficient to obtain the first transformation coefficient. Two transformation coefficients.
  • the transformation unit 520 is specifically used to:
  • the first identifier is used to identify that the prediction mode derivation mode is allowed to be used to predict the image block in the current sequence
  • the second identifier is used to identify that the first transformation is allowed to be used to predict the current sequence
  • the transformation unit 520 performs a second transformation on the second transformation coefficient, and before obtaining the residual block of the current block, it is also used to:
  • the transformation unit 520 is specifically used to:
  • the first transformation uses the same transformation matrix set as the transformation matrix set for planar mode or direct current DC mode adaptation.
  • the transformation unit 520 is specifically used to:
  • the transformation matrix group used in the first transformation is the same as the transformation matrix group used in the third intra prediction mode adaptation.
  • the transformation unit 520 is specifically used to:
  • the third intra prediction mode is determined based on the weight of the first intra prediction mode and/or the weight of the second intra prediction mode; or
  • the third intra prediction mode is determined based on the type of the first intra prediction mode and the type of the second intra prediction mode; or
  • the third intra prediction mode is determined based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode.
  • the transformation unit 520 is specifically used to:
  • the intra prediction mode with the largest weight among the first intra prediction mode and the second intra prediction mode is determined as the third intra prediction mode.
  • the transformation unit 520 is specifically used to:
  • the angle prediction mode is determined as the third intra prediction mode.
  • the transformation unit 520 is specifically used to:
  • the intra prediction corresponding to the first prediction angle is mode, determined to be the third intra prediction mode, wherein the first prediction angle is determined according to the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode:
  • the planar mode or the direct current DC mode is determined as the The third intra prediction mode is described.
  • the transformation unit 520 is specifically used to:
  • the transformation matrix group adapted to the prediction mode derivation mode is determined as the transformation matrix group used for the first transformation.
  • the first transformation is used to process textures in the current block in an oblique direction
  • the second transformation is used to process textures in the current block in a horizontal direction and in a vertical direction. texture on.
  • Figure 19 is a schematic block diagram of the encoder 600 according to the embodiment of the present application.
  • the encoder 600 may include:
  • Prediction unit 610 configured to predict the current block based on the first intra prediction mode and the second intra prediction mode derived from the prediction mode derivation mode, and obtain the prediction block of the current block;
  • the residual unit 620 is used to obtain the residual block of the current block based on the prediction block of the current block;
  • Transformation unit 630 used for:
  • Encoding unit 640 used to encode the fourth transform coefficient.
  • the prediction mode derivation mode includes a decoder-side intra mode derivation mode or a template-based intra mode derivation mode.
  • the encoding unit 640 is specifically used to:
  • the first identification is used to identify that the prediction mode derivation mode is allowed to be used to predict the image blocks in the current sequence
  • the second identification is used to identify that the fourth transformation is allowed to be used to predict the current sequence. Transform the image blocks in .
  • the encoding unit 640 is specifically used to:
  • the third identification is used to identify that image blocks in the current sequence are allowed to use the prediction mode derivation mode and the fourth transformation at the same time.
  • the transformation unit 630 is specifically used to:
  • the fourth transform is performed on the third transform coefficient to obtain the fourth transform coefficient.
  • the transform unit 630 performs a fourth transform on the third transform coefficient, and before obtaining the fourth transform coefficient of the current block, it is also used for:
  • the fourth transformation uses the same transformation matrix set as the transformation matrix set for planar mode or direct current DC mode adaptation.
  • the transformation unit 630 is specifically used to:
  • the transformation matrix group used in the fourth transformation is the same as the transformation matrix group used in the third intra prediction mode adaptation.
  • the transformation unit 630 is specifically used to:
  • the third intra prediction mode is determined based on the weight of the first intra prediction mode and/or the weight of the second intra prediction mode; or
  • the third intra prediction mode is determined based on the type of the first intra prediction mode and the type of the second intra prediction mode; or
  • the third intra prediction mode is determined based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode.
  • the transformation unit 630 is specifically used to:
  • the intra prediction mode with the largest weight among the first intra prediction mode and the second intra prediction mode is determined as the third intra prediction mode.
  • the transformation unit 630 is specifically used to:
  • the angle prediction mode is determined as the third intra prediction mode.
  • the transformation unit 630 is specifically used to:
  • the intra prediction corresponding to the first prediction angle is mode, determined to be the third intra prediction mode, wherein the first prediction angle is determined according to the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode:
  • the planar mode or the direct current DC mode is determined as the The third intra prediction mode is described.
  • the transformation unit 630 is specifically used to:
  • the transformation matrix group adapted to the prediction mode derivation mode is determined as the transformation matrix group used for the fourth transformation.
  • the fourth transformation is used to process the texture in the oblique direction in the current block
  • the third transformation is used to process the texture in the current block in the horizontal direction and in the vertical direction. texture on.
  • the device embodiments and the method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, they will not be repeated here.
  • the decoder 500 shown in Figure 18 may correspond to the corresponding subject in performing the method 300 of the embodiment of the present application, and the foregoing and other operations and/or functions of each unit in the decoder 500 are respectively to implement the method 300, etc.
  • the encoder 600 shown in Figure 19 may correspond to the corresponding subject in performing the method 400 of the embodiment of the present application, that is, the foregoing and other operations and/or functions of each unit in the encoder 600 are respectively in order to implement the method 400, etc. Corresponding processes in each method.
  • each unit in the decoder 500 or encoder 600 involved in the embodiment of the present application can be separately or entirely combined into one or several other units to form, or some of the units (some) can also be disassembled. It is divided into multiple units with smaller functions, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application.
  • the above units are divided based on logical functions. In practical applications, the function of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the decoder 500 or the encoder 600 may also include other units. In practical applications, these functions may also be implemented with the assistance of other units, and may be implemented by multiple units in cooperation.
  • a general-purpose computing device including a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), etc.
  • Run a computer program capable of executing each step involved in the corresponding method to construct the decoder 500 or encoder 600 involved in the embodiment of the present application, and implement the encoding method or decoding method of the embodiment of the present application.
  • the computer program can be recorded on, for example, a computer-readable storage medium, loaded into an electronic device through the computer-readable storage medium, and run therein to implement the corresponding methods of the embodiments of the present application.
  • the units mentioned above can be implemented in the form of hardware, can also be implemented in the form of instructions in the form of software, or can be implemented in the form of a combination of software and hardware.
  • each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits of hardware in the processor and/or instructions in the form of software.
  • the steps of the methods disclosed in conjunction with the embodiments of the present application can be directly embodied in hardware.
  • the execution of the decoding processor is completed, or the execution is completed using a combination of hardware and software in the decoding processor.
  • the software can be located in a mature storage medium in this field such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc.
  • the storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.
  • FIG. 20 is a schematic structural diagram of an electronic device 700 provided by an embodiment of the present application.
  • the electronic device 700 at least includes a processor 710 and a computer-readable storage medium 720 .
  • the processor 710 and the computer-readable storage medium 720 may be connected through a bus or other means.
  • the computer-readable storage medium 720 is used to store a computer program 721
  • the computer program 721 includes computer instructions
  • the processor 710 is used to execute the computer instructions stored in the computer-readable storage medium 720.
  • the processor 710 is the computing core and the control core of the electronic device 700. It is suitable for implementing one or more computer instructions. Specifically, it is suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding functions.
  • the processor 710 may also be called a central processing unit (Central Processing Unit, CPU).
  • the processor 710 may include, but is not limited to: a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • the computer-readable storage medium 720 can be a high-speed RAM memory, or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory; optionally, it can also be at least one located far away from the aforementioned processor 710 Computer-readable storage media.
  • the computer-readable storage medium 720 includes, but is not limited to: volatile memory and/or non-volatile memory.
  • non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electrically removable memory.
  • Volatile memory may be Random Access Memory (RAM), which is used as an external cache.
  • RAM Random Access Memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • DRAM synchronous dynamic random access memory
  • DDR SDRAM double data rate synchronous dynamic random access memory
  • Enhanced SDRAM, ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous link dynamic random access memory
  • Direct Rambus RAM Direct Rambus RAM
  • the electronic device 700 may be the encoder or coding framework involved in the embodiment of the present application; the computer-readable storage medium 720 stores the first computer instructions; the computer-readable instructions are loaded and executed by the processor 710
  • the first computer instructions stored in the storage medium 720 are used to implement the corresponding steps in the encoding method provided by the embodiment of the present application; in other words, the first computer instructions in the computer-readable storage medium 720 are loaded by the processor 710 and execute the corresponding steps, To avoid repetition, they will not be repeated here.
  • the electronic device 700 may be the decoder or decoding framework involved in the embodiment of the present application; the computer-readable storage medium 720 stores second computer instructions; the computer-readable instructions are loaded and executed by the processor 710 The second computer instructions stored in the storage medium 720 are used to implement the corresponding steps in the decoding method provided by the embodiment of the present application; in other words, the second computer instructions in the computer-readable storage medium 720 are loaded by the processor 710 and execute the corresponding steps, To avoid repetition, they will not be repeated here.
  • embodiments of the present application also provide a coding and decoding system, including the above-mentioned encoder and decoder.
  • embodiments of the present application also provide a computer-readable storage medium (Memory).
  • the computer-readable storage medium is a memory device in the electronic device 700 and is used to store programs and data.
  • computer-readable storage medium 720 may include a built-in storage medium in the electronic device 700 , and of course may also include an extended storage medium supported by the electronic device 700 .
  • the computer-readable storage medium provides storage space that stores the operating system of the electronic device 700 .
  • one or more computer instructions suitable for being loaded and executed by the processor 710 are also stored in the storage space. These computer instructions may be one or more computer programs 721 (including program codes).
  • a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.
  • the data processing device 700 can be a computer.
  • the processor 710 reads the computer instructions from the computer-readable storage medium 720.
  • the processor 710 executes the computer instructions, so that the computer executes the encoding method provided in the above various optional ways. or decoding method.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transmitted from a website, computer, server, or data center to Transmission to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) methods.
  • wired such as coaxial cable, optical fiber, digital subscriber line (DSL)
  • wireless such as infrared, wireless, microwave, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Discrete Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请实施例提供了一种解码方法、编码方法、解码器以及编码器。所述解码方法包括：解码码流获取当前块的第一变换系数；对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数；对所述第二变换系数进行第二变换，得到所述当前块的残差块；基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对所述当前块进行预测，得到所述当前块的预测块；基于所述当前块的预测块和所述当前块的残差块，得到所述当前块的重建块。本申请在预测模式导出模式和第二变换的基础上，引入了第一变换，能够提升当前块的解压缩效率。

Description

解码方法、编码方法、解码器以及编码器 技术领域
本申请实施例涉及图像视频编解码技术领域，并且更具体地，涉及解码方法、编码方法、解码器以及编码器。
背景技术
数字视频压缩技术主要是将庞大的数字影像视频数据进行压缩，以便于传输以及存储等。随着互联网视频的激增以及人们对视频清晰度的要求越来越高，尽管已有的数字视频压缩标准能够实现视频的解压缩技术，但目前仍然需要追求更好的数字视频解压缩技术，以在提升压缩效率。
发明内容
本申请实施例提供了一种解码方法、编码方法、解码器以及编码器，能够提升压缩效率。
第一方面，本申请提供了一种解码方法，包括：
解码码流获取当前块的第一变换系数；
对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数；
对所述第二变换系数进行第二变换，得到所述当前块的残差块；
基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对所述当前块进行预测，得到所述当前块的预测块；
基于所述当前块的预测块和所述当前块的残差块，得到所述当前块的重建块。
第二方面，本申请提供了一种编码方法，包括：
基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对当前块进行预测，得到所述当前块的预测块；
基于所述当前块的预测块，得到所述当前块的残差块；
对所述当前块的残差块进行第三变换，得到所述当前块的第三变换系数；
对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数；
对所述第四变换系数进行编码。
第三方面，本申请提供了一种解码器，包括：
解码单元，用于解码码流获取当前块的第一变换系数；
变换单元，用于：
对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数；
对所述第二变换系数进行第二变换，得到所述当前块的残差块；
预测单元，用于基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对所述当前块进行预测，得到所述当前块的预测块；
重建单元，用于基于所述当前块的预测块和所述当前块的残差块，得到所述当前块的重建块。
第四方面，本申请提供了一种编码器，包括：
预测单元，用于基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对当前块进行预测，得到所述当前块的预测块；
残差单元，用于基于所述当前块的预测块，得到所述当前块的残差块；
变换单元，用于：
对所述当前块的残差块进行第三变换，得到所述当前块的第三变换系数；
对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数；
编码单元，用于对所述第四变换系数进行编码。
第五方面，本申请提供了一种解码器，包括：
处理器，适于实现计算机指令；以及，
计算机可读存储介质，计算机可读存储介质存储有计算机指令，计算机指令适于由处理器加载并执行上述第一方面或其各实现方式中的解码方法。
在一种实现方式中，该处理器为一个或多个，该存储器为一个或多个。
在一种实现方式中，该计算机可读存储介质可以与该处理器集成在一起，或者该计算机可读存储介质与处理器分离设置。
第六方面，本申请提供了一种编码器，包括：
处理器，适于实现计算机指令；以及，
计算机可读存储介质，计算机可读存储介质存储有计算机指令，计算机指令适于由处理器加载并执行上述第二方面或其各实现方式中的编码方法。
在一种实现方式中，该处理器为一个或多个，该存储器为一个或多个。
在一种实现方式中，该计算机可读存储介质可以与该处理器集成在一起，或者该计算机可读存储介质与处理器分离设置。
第七方面，本申请提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机指令，该计算机指令被计算机设备的处理器读取并执行时，使得计算机设备执行上述第一方面涉及的解码方法或上述第二方面涉及的编码方法。
第八方面，本申请提供了一种码流，该码流上述第一方面中涉及所述的码流或上述第二方面中涉及的码流。
基于以上技术方案，本申请在预测模式导出模式和第二变换的基础上，引入了第一变换，能够提升当前块的解压缩效率。
附图说明
图1是本申请实施例提供的编码框架的示意性框图。
图2为本申请实施例提供的65种角度预测模式的具体方向示意图。
图3是本申请实施例提供的宽角度预测模式的参考像素的示例。
图4是本申请实施例提供的MIP模式的示意图。
图5是本申请实施例提供的基于DIMD导出预测模式的示意图。
图6是本申请实施例提供的基于DIMD导出预测块的示意图。
图7是本申请实施例提供的TIMD使用的模板的示意图。
图8是本申请实施例提供的GPM在正方形的块上的64种权重导出模式对应的权重图的示例。
图9是本申请实施例提供的权重导出模式的划分线的示例。
图10是本申请实施例提供的AWP在正方形的块上的56种权重导出模式对应的权重图的示例。
图11是本申请实施例提供的GPM或AWP的示意图。
图12是本申请实施例提供的DCT2型的基图像的示例。
图13是本申请实施例提供的LFNST的示例。
图14是本申请实施例提供的LFNST的变换矩阵组的示例。
图15是本申请实施例提供的解码框架的示意性框图。
图16是本申请实施例提供的解码方法的示意性流程图。
图17是本申请实施例提供的编码方法的示意性流程图。
图18是本申请实施例提供的解码器的示意性框图。
图19是本申请实施例提供的编码器的示意性框图。
图20是本申请实施例提供的电子设备的示意性框图。
具体实施方式
下面将结合附图，对本申请实施例中的技术方案进行描述。
本申请实施例提供的方案可应用于数字视频编码技术领域，例如，包括但不限于：图像编解码领域、视频编解码领域、硬件视频编解码领域、专用电路视频编解码领域以及实时视频编解码领域。此外，本申请实施例提供的方案可结合至音视频编码标准(Audio Video coding Standard，AVS)、第二代AVS标准(AVS2)或第三代AVS标准(AVS3)。例如，包括但不限于：H.264/音视频编码(Audio Video coding，AVC)标准、H.265/高效视频编码(High Efficiency Video Coding，HEVC)标准以及H.266/多功能视频编码(Versatile Video Coding，VVC)标准。另外，本申请实施例提供的方案可以用于对图像进行有损压缩(lossy compression)，也可以用于对图像进行无损压缩(lossless compression)。其中该无损压缩可以是视觉无损压缩(visually lossless compression)，也可以是数学无损压缩(mathematically lossless compression)。
视频编解码标准都采用基于块的混合编码框架。视频中的每一帧被分割成相同大小(如128x128，64x64等)的正方形的最大编码单元(largest coding unit，LCU)或编码树单元(Coding Tree Unit，CTU)。每个最大编码单元或编码树单元可根据规则划分成矩形的编码单元(coding unit，CU)。编码单元可能还会划分为预测单元(prediction unit，PU)，变换单元(transform unit，TU)等。混合编码框架包括预测(prediction)、变换(transform)、量化(quantization)、熵编码(entropy coding)、环路滤波(in loop filter)等模块。预测模块包括帧内预测(intra prediction)和帧间预测(inter prediction)。帧间预测包括运动估计(motion estimation)和运动补偿(motion compensation)。由于视频的一个帧中的相邻像素之间存在很强的相关性，在视频编解码技术中使用帧内预测的方法消除相邻像素之间的空间冗余。帧内预测只参考同一帧图像的信息，预测当前划分块内的像素信息。由于视频中的相邻帧之间存在着很强的相似性，在视频编解码技术中使用帧间预测方法消除相邻帧之间的时间冗余，从而提高编码效率。帧间预测可以参考不同帧的图像信息，利用运动估计搜索最匹配当前划分块的运动矢量信息。变换将预测后的图像块转换到频率域，能量重新分布，结合量化可以将人眼不敏感的信息去除，用于消除视觉冗余。熵编码可以根据当前上下文模型以及二进制码流的概率信息消除字符冗余。
在数字视频编码过程中，编码器可以先从原始视频序列中读取一幅黑白图像或彩色图像，然后针对黑白图像或彩色图像进行编码。其中，黑白图像可以包括亮度分量的像素，彩色图像可以包括色度分量的像素。可选的，彩色图像还可以包括亮度分量的像素。原始视频序列的颜色格式可以是亮度色度(YCbCr，YUV)格式或红绿蓝(Red-Green-Blue，RGB)格式等。具体地，编码器读取一幅黑白图像或彩色图像之后，分别将其划分成块，并对当前块使用帧内预测或帧间预测产生当前块的预测块，当前块的原始块减去预测块得到残差块，对残差块进行变换、量化得到量化系数矩阵，对量化系数矩阵进行熵编码输出到码流中。在数字视频解码过程中，解码端对当前块使用帧内预测或帧间预测产生当前块的预测块。此外，解码端解码码流得到量化系数矩阵，对量化系数矩阵进行反量化、反变换得到残差块，将预测块和残差块相加得到重建块。重建块可用于组成重建图像，解码端基于图像或基于块对重建图像进行环路滤波得到解码图像。
当前块(current block)可以是当前编码单元(CU)或当前预测单元(PU)等。
需要说明的是，编码端同样需要和解码端类似的操作获得解码图像。解码图像可以为后续的帧作为帧间预测的参考帧。编码端确定的块划分信息，预测、变换、量化、熵编码、环路滤波等模式信息或者参数信息，如果有必要需要在输出到码流中。解码端通过解析及根据已有信息进行分析确定与编码端相同的块划分信息，预测、变换、量化、熵编码、环路滤波等模式信息或者参数信息，从而保证编码端获得的解码图像和解码端获得的解码图像相同。编码端获得的解码图像通常也叫做重建图像。在预测时可以将当前块划分成预测单元，在变换时可以将当前块划分成变换单元，预测单元和变换单元的划分可以相同也可以不同。当然，上述仅是基于块的混合编码框架下的视频编解码器的基本流程，随着技术的发展，框架的一些模块或流程的一些步骤可能会被优化，本申请适用于该基于块的混合编码框架下的视频编解码器的基本流程。
为了便于理解，先对本申请提供的编码框架进行简单介绍。
图1是本申请实施例提供的编码框架100的示意性框图。
如图1所示，该编码框架100可包括帧内预测单元180、帧间预测单元170、残差单元110、变换与量化单元120、熵编码单元130、反变换与反量化单元140、以及环路滤波单元150。可选的，该编码框架100还可包括解码图像缓冲单元160。该编码框架100也可称为混合框架编码模式。
其中，帧内预测单元180或帧间预测单元170可对待编码图像块进行预测，以输出预测块。残差单元110可基于预测块与待编码图像块计算残差块，即预测块和待编码图像块的差值。变换与量化单元120用于对残差块执行变换与量化等操作，以去除人眼不敏感的信息，进而消除视觉冗余。可选的，经过变换与量化单元120变换与量化之前的残差块可称为时域残差块，经过变换与量化单元120变换与量化之后的时域残差块可称为频率残差块或频域残差块。熵编码单元130接收到变换与量化单元120输出的变换量化系数后，可基于该变换量化系数输出码流。例如，熵编码单元130可根据目标上下文模型以及二进制码流的概率信息消除字符冗余。例如，熵编码单元130可以用于基于上下文的自适应二进制算术熵编码(CABAC)。熵编码单元130也可称为头信息编码单元。可选的，在本申请中，该待编码图像块也可称为原始图像块或目标图像块，预测块也可称为预测图像块或图像预测块，还可以称为预测信号或预测信息，重建块也可称为重建图像块或图像重建块，还可以称为重建信号或重建信息。此外，针对编码端，该待编码图像块也可称为编码块或编码图像块，针对解码端，该待编码图像块也可称为解码块或解码图像块。该待编码图像块可以是CTU或CU。
编码框架100将预测块与待编码图像块计算残差得到残差块经由变换与量化等过程，将残差块传输到解码端。相应的，解码端接收并解码码流后，经过反变换与反量化等步骤得到残差块，将解码端预测得到的预测块叠加残差块后得到重建块。
需要说明的是，编码框架100中的反变换与反量化单元140、环路滤波单元150以及解码图像缓冲单元160可用于形成一个解码器。相当于，帧内预测单元180或帧间预测单元170可基于已有的重建块对待编码图像块进行预测，进而能够保证编码端和解码端的对参考帧的理解一致。换言之，编码器可复制解码器的处理环路，进而可与解码端产生相同的预测。具体而言，量化的变换系数通过反变换与反量化单元140反变换与反量化来复制解码端的近似残差块。该近似残差块加上预测块后可经过环路滤波单元150，以平滑滤除由于基于块处理和量化产生的块效应等影响。环路滤波单元150输出的图像块可存储在解码图像缓存单元160中，以便用于后续图像的预测。
应理解，图1仅为本申请的示例，不应理解为对本申请的限制。
例如，该编码框架100中的环路滤波单元150可包括去块滤波器(DBF)和样点自适应补偿滤波(SAO)。DBF的作用是去块效应，SAO的作用是去振铃效应。在本申请的其他实施例中，该编码框架100可采用基于神经网络的环路滤波算法，以提高视频的压缩效率。或者说，该编码框架100可以是基于深度学习的神经网络的视频编码混合框架。在一种实现中，可以在去块滤波器和样点自适应补偿滤波基础上，采用基于卷积神经网络的模型计算对像素滤波后的结果。环路滤波单元150在亮度分量和色度分量上的网络结构可以相同，也可以有所不同。考虑到亮度分量包含更多的视觉信息，还可以采用亮度分量指导色度分量的滤波，以提升色度分量的重建质量。
下面对帧内预测和帧间预测的相关内容进行说明。
对于帧间预测，帧间预测可以参考不同帧的图像信息，利用运动估计搜索最匹配待编码图像块的运动矢量信息，用于消除时间冗余；帧间预测所使用的帧可以为P帧和/或B帧，P帧指的是向前预测帧，B帧指的是双向预测帧。
对于帧内预测，帧内预测只参考同一帧图像的信息，预测待编码图像块内的像素信息，用于消除空间冗余；帧内预测所使用的帧可以为I帧。例如，可根据从左至右、从上到下的编码顺序，待编码图像块可以参考左上方图像块，上方图像块以及左侧图像块作为参考信息来预测待编码图像块，而待编码图像块又作为下一个图像块的参考信息，如此，可对整幅图像进行预测。若输入的数字视频为彩色格式，例如YUV 4:2:0格式，则该数字视频的每一图像帧的每4个像素点由4个Y分量和2个UV分量组成，编码框架可对Y分量(即亮度块)和UV分量(即色度块)分别进行编码。类似的，解码端也可根据格式进行相应的解码。
针对帧内预测过程，帧内预测可借助角度预测模式与非角度预测模式对待编码图像块进行预测，以得到预测块，根据预测块与待编码图像块计算得到的率失真信息，筛选出待编码图像块最优的预测模式，并将该预测模式经码流传输到解码端。解码端解析出预测模式，预测得到目标解码块的预测块并叠加经码流传输而获取的时域残差块，可得到重建块。
经过历代的数字视频编解码标准发展，非角度预测模式保持相对稳定，有均值模式和平面模式。角度预测模式则随着数字视频编解码标准的演进而不断增加。以国际数字视频编码标准H系列为例，H.264/AVC标准仅有8种角度预测模式和1种非角度预测模式；H.265/HEVC扩展到33种角度预测模式和2种非角度预测模式。在H.266/VVC中，帧内预测模式被进一步拓展，对于亮度块共有67种传统预测模式和非传统的预测模式矩阵加权帧内预测(Matrix weighted intra-frame prediction，MIP)模式，其中这67种传统预测模式包括：平面(planar)模式、直流(DC)模式和65种角度预测模式。其中，平面模式通常用于处理一些纹理存在渐变的块，DC模式顾名思义通常用于处理一些平坦区域，而角度预测模式通常用于处理角度纹理比较明显的块。
需要说明的是，本申请中，用于帧内预测的当前块可以是正方形块，也可以是矩形块。
进一步的，由于帧内预测块都是正方形的所以各个角度预测模式使用的概率是相等的，当前块的长宽不等时，对于水平类的块(宽大于高)上边的参考像素使用概率大于左边参考像素的使用概率，对于垂直类的块(高大于宽)上边的参考像素使用概率小于左边参考像素的使用概率。基于此，本申请引入了宽角度预测模式，在对矩形块预测时，将传统的角度预测模式转换为宽角度预测模式，利用宽角度预测模式对矩形块进行预测时，当前块的预测角度范围大于利用传统角度预测模式对矩形块进行预测时的预测角度范围。可选的，使用宽角度预测模式时，可以仍然使用传统角度预测模式的索引发出信号，相应的，解码端在在收到信号后可将传统角度预测模式再转换为宽角度预测模式，由此，帧内预测模式的总数和帧内模式编码方法保持不变，并且帧内模式编码方法不变。
图2为本申请实施例提供的65种角度预测模式的具体方向示意图。
如图2所示，索引0用于标识平面模式，索引1用于标识DC模式，索引-14至80可以分别用于标识不同的角度预测模式。具体地，索引2～66用于标识传统角度预测模式，索引-1～-14以及索引67～80用于标识宽角度预测模式。或者 说，索引2～66标识的传统的帧内预测模式可用于预测正方形块，索引-1～-14以及67～80标识的宽角度预测模式可用于预测矩形块。
应当理解，本申请中涉及的索引x标识的预测模式也可称为预测模式x。例如，索引2标识的帧内预测模式也可称为帧内预测模式2。
图3是本申请实施例提供的宽角度预测模式的参考像素的示例。
如图3所示，在宽角度预测模式下，对于尺寸为W×H的CU，其上方的参考像素点的个数为2W+1，其左侧的参考样点的个数为2H+1。具体地，如图3中的(a)所示，当W>H(例如8×4的CU)时，帧内预测模式2附近(大于帧内预测模式2)存在CU的右下角点无法索引到参考像素的情况，而帧内预测模式66附近(大于帧内预测模式66)仍有部分能够索引到参考像素，因此需要将帧内预测模式2附近(大于帧内预测模式2)的一些类水平角度模式替换成为帧内预测模式66附近(大于帧内预测模式66)的一些类垂直角度模式，以扩展预测角度范围。类似的，如图3中的(b)所示，当W＜H(例如4×8的CU)时，由于帧内预测模式66附近(小于帧内预测模式66)存在部分点无法索引到参考像素的情况，而帧内预测模式2附近(小于帧内预测模式2)仍有部分能够索引到参考像素)，因此需要将帧内预测模式66附近(小于帧内预测模式66)的一些类垂直角度模式替换成为帧内预测模式2附近(小于帧内预测模式2)的一些类水平角度模式，以扩展预测角度范围。
在一些情况下，可以基于当前块的尺寸来确定或选择要执行的帧内预测模式；例如，可以基于当前块的尺寸来确定或选择宽角度预测模式对当前块进行帧内预测；例如，在当前块是矩形块(宽度和高度具有不同的尺寸)时，可以使用宽角度预测模式对当前块进行帧内预测。其中，当前块的宽高比可以用于确定宽角度预测模式被替换的角度预测模式和替换后的角度预测模式。例如在预测当前块时，可以选择具有不超过当前块的对角(从当前块的左下角到右上角)的角度的任何帧内预测模式，作为替换后的角度预测模式。
下面对本申请涉及的其他帧内预测模式进行介绍：
(1)、基于矩阵的帧内预测(Matrix based Intra Prediction，MIP)模式。
MIP模式也可称为矩阵加权帧内预测(Matrix weighted Intra Prediction)模式，MIP模式涉及的流程可以分为三个主要步骤，其分别是下采样过程、矩阵相乘过程以及上采样过程。具体来说，首先通过下采样过程下采样空间相邻重建样本，然后，将得到下采样后的样本序列作为矩阵相乘过程的输入向量，即将下采样过程的输出向量作为矩阵相乘过程的输入向量，与预先设定好的矩阵相乘并加上偏置向量，并输出计算之后的样本向量；将矩阵相乘过程的输出向量作为上采样过程的输入向量，通过上采样得到最终的预测块。
图4是本申请实施例提供的MIP模式的示意图。
如图4所示，MIP模式在下采样过程中通过平均当前编码单元上边相邻的重建样本后得到上相邻下采样重建样本向量bdry top，通过平均左相邻的重建样本后得到左相邻下采样重建样本向量bdry left。得到bdry top和bdry left后，将其作为矩阵相乘过程的输入向量bdry red，具体地，可通基于bdry red的顶行向量bdry top red、bdry left、A k·bdry red+b k得到样本向量，其中A k为预先设定好的矩阵，b k为预先设定好的偏置向量，k为MIP模式的索引。得到样本向量后通过线性插值对其进行上采样，以得到与实际编码单元样本数相符的预测样本块。
换言之，为了对一个宽度为W高度为H的块进行预测，MIP需要当前块左侧一列的H个重建像素和当前块上侧一行的W个重建像素作为输入。MIP按如下3个步骤生成预测块：参考像素平均(Averaging)，矩阵乘法(Matrix Vector Multiplication)和插值(Interpolation)。其中MIP的核心是矩阵乘法，其可以认为是用一种矩阵乘法的方式用输入像素(参考像素)生成预测块的过程。MIP提供了多种矩阵，预测方式的不同可以体现在矩阵的不同上，相同的输入像素使用不同的矩阵会得到不同的结果。而参考像素平均和插值的过程是一种性能和复杂度折中的设计。对于尺寸较大的块，可以通过参考像素平均来实现一种近似于降采样的效果，使输入能适配到比较小的矩阵，而插值则实现一种上采样的效果。这样就不需要对每一种尺寸的块都提供MIP的矩阵，而是只提供一种或几种特定的尺寸的矩阵即可。随着对压缩性能的需求的提高，以及硬件能力的提高，下一代的标准中也许会出现复杂度更高的MIP。
对于MIP模式而言，MIP模式可以由神经网络简化而来，例如其采用的矩阵可以是基于训练得到，因此，MIP模式拥有较强的泛化能力和传统预测模式达不到的预测效果。MIP模式可以是对一个基于神经网络的帧内预测模型经过多次硬件和软件复杂度简化而得到的模型，在大量训练样本的基础上，多种预测模式代表着多种模型和参数，能够较好的覆盖自然序列的纹理情况。
MIP有些类似于平面模式，但显然MIP比平面模式更复杂，灵活性也更强。
需要说明的是，对于不同块尺寸的编码单元，MIP模式的个数可以有所不同。示例性地，对于4x4大小的编码单元，MIP模式有16种预测模式；对于8x8、宽等于4或高等于4的编码单元，MIP模式有8种预测模式；其他尺寸的编码单元，MIP模式有6种预测模式。同时，MIP模式有一个转置功能，对于符合当前尺寸的预测模式，MIP模式在编码器侧可以尝试转置计算。因此，MIP模式不仅需要一个标志位来表示当前编码单元是否使用MIP模式，同时，若当前编码单元使用MIP模式，则额外还需要传输一个转置标志位到解码器。
(2)、解码器侧帧内模式导出(Decoder side Intra Mode Derivation，DIMD)模式。
DIMD模式主要核心点在于帧内预测的模式在解码器使用与编码器相同的方法导出帧内预测模式，以此避免在码流中传输当前编码单元的帧内预测模式索引，达到节省比特开销的目的。
DIMD模式的具体流程可分为以下两个主要步骤：
步骤1：导出预测模式。
图5是本申请实施例提供的基于DIMD导出预测模式的示意图。
如图5的(a)所示，DIMD利用重建区域中模板中的像素(当前块左侧和上侧的重建像素)导出预测模式。例如，模板可以包括当前块的上方三行相邻重建样本、左侧三列相邻重建样本以及左上方对应相邻重建样本，基于此，可按照窗口(例如图5的(a)所示或如图5的(b)所示的窗口)在模板内确定出多个梯度值，其中每一个梯度值适可用于配出与其梯度方向相适应的一种帧内预测模式(Intra prediction mode，ipm)，基于此，编码器可将多个梯度值中最大和次大的梯度值适配的预测模式作为导出的预测模式。例如，如图5的(b)所示，对于4×4大小的块，针对所有需要确定梯度值的像 素进行分析并得到的对应的梯度直方图(histogram of gradients)，例如，如图5的(c)所示，对于其它大小的块，对所有需要确定梯度值的像素进行分析并得到的对应的梯度直方图；最终，将梯度直方图中梯度最大和次大的梯度所对应的预测模式作为导出的预测模式。
当然，本申请中的梯度直方图仅为用于确定导出的预测模式的示例，具体实现时可以用多种简单的形式实现，本申请对此不作具体限定。此外，本申请对统计梯度直方图的方式不作限定，例如，可以利用可利用索贝尔算子或其他方式统计梯度直方图。
步骤2：导出预测块。
图6是本申请实施例提供的基于DIMD导出预测块的示意图。
如图6所示，编码器可3个帧内预测模式(平面模式以及基于DIMD导出的2个帧内预测模式)的预测值进行加权。编解码器使用同样的预测块导出方式得到当前块的预测块。假设最大梯度值对应的预测模式为预测模式1，次大梯度值对应的预测模式为预测模式2，编码器判断以下两个条件：
1、预测模式2的梯度不为0；
2、预测模式1和预测模式2均不为平面模式或者DC预测模式。
若上述两个条件不同时成立，则仅使用预测模式1计算当前块的预测样本值，即对预测模式1应用普通预测预测过程；否则，即上述两个条件均成立，则使用加权求平均方式导出当前块的预测块。具体方法为：平面模式占据1/3的加权权重，剩下2/3为预测模式1和预测模式2的总权重，例如将预测模式1的梯度幅度值除以预测模式1的梯度幅度值和预测模式2的梯度幅度值的和作为预测模式1的加权权重，将预测模式2的梯度幅度值除以预测模式1的梯度幅度值和预测模式2的梯度幅度值的和作为预测模式2的加权权重；将基于上述三种预测模式得到的预测块，即对基于平面模式、预测模式1和预测模式2分别得到的预测块1、预测块2以及预测块3进行加权求平均得到当前编码单元的预测块。解码器以同样步骤得到预测块。
换言之，上述步骤2中具体权重计算如下：
Weight(PLANAR)＝1/3；
Weight(mode1)＝2/3*(amp1/(amp1+amp2))；
Weight(mode2)＝1–Weight(PLANAR)–Weight(mode1)；
其中，mode1和mode2分别代表预测模式1和预测模式2，amp1和amp2分别代表预测模式1的梯度幅度值和预测模式2的梯度幅度值。DIMD模式需要传输一个标志位传输到解码器，该标志位用于表示当前编码单元是否使用DIMD模式。
当然，上述加权求平均方式仅为本申请的示例，不应理解为对本申请的限定。
总结来说，DIMD利用重建像素的梯度分析来筛选帧内预测模式，而且可以将2个帧内预测模式再加上平面模式根据分析结果进行加权。DIMD的好处在于如果当前块选择了DIMD模式，那么码流中不需要再去指示具体使用了哪种帧内预测模式，而是由解码器自己通过上述流程导出，一定程度上节省了开销。
(3)、基于模板的帧内模式导出(Template based Intra Mode Derivation，TIMD)模式。
TIMD模式的技术原理与上述DIMD模式的技术原理比较近似，都是利用编解码器同样操作导出预测模式的方式来节省传输模式索引开销。TIMD模式主要可以理解成两个主要部分，首先根据模板计算各预测模式的代价信息，最小代价及次小代价对应的预测模式将被选中，最小代价对应的预测模式记为预测模式1，次小代价对应的预测模式记为预测模式2；若次小代价的数值(costMode2)与最小代价的数值(costMode1)比例满足预设条件，如costMode2<2*costMode1，则将预测模式1与预测模式2各对应的预测块可按照预测模式1与预测模式2各对应的权重进行加权融合，进而得到最终的预测块。
示例性地，预测模式1与预测模式2各对应的权重根据以下方式确定：
weight1＝costMode2/(costMode1+costMode2)；
weight2＝1-weight1；
其中，weight1即为预测模式1对应预测块的加权权重，weight2即为预测模式2对应预测块的加权权重。但若次小代价的数值costMode2与最小代价的数值costMode1比例不满足预设条件，则不做预测块之间的加权融合，预测模式1对应预测块即为TIMD的预测块。
需要说明的是，若采用TIMD模式对当前块进行帧内预测时，若当前块的重建样本模板中不包含可用相邻重建样本，则TIMD模式选择平面模式对当前块进行帧内预测，即不进行不加权融合。与DIMD模式相同，TIMD模式需要传输一个标志位到解码器，以表示当前编码单元是否使用TIMD模式。
图7是本申请实施例提供的TIMD使用的模板的示意图。
如图7所示，若当前块为宽等于M且高等于N的编码单元，编解码器可以基于宽等于2(M+L1)+1且高等于2(N+L2)+1的编码单元中选择当前块的参考模板(Reference of template)计算当前块的模板，此时，若当前块的模板中不包含可用相邻重建样本，则TIMD模式选择平面模式对当前块进行帧内预测。例如，所述可用相邻重建样本可以是图7中与当前CU的左侧和上侧相邻的样本，即斜线填充区域中没有可用重建样本。也即是说，若斜线填充区域中没有可用重建样本，则TIMD模式选择平面模式对当前块进行帧内预测。
除了边界情况，在编解码当前块时，当前块的左侧和上侧理论上是可以得到重建值的，即当前块的模板中包含可用相邻重建样本。具体实现中，解码器可以使用某一个帧内预测模式在模板上进行预测，并且将预测值和重建值进行比较，以得到该帧内预测模式在模板上的代价。比如SAD，SATD，SSE等。由于模板和当前块是相邻的，因此，模板内的重建样本和当前块内的像素具有相关性，因此，可以用一个预测模式在模板上的表现来估计这个预测模式在当前块上的表现。TIMD将一些候选的帧内预测模式在模板上进行预测，得到候选的帧内预测模式在模板上的代价，取代价最低的一个或2个帧内预测模式作为当前块的帧内预测值。如果2个帧内预测模式在模板上的代价差距不大，将2个帧内预测模式的预测值进行加权平均可以得到压缩性能的提升。可选的，2个预测模式的预测值的权重跟上述的代价有关，例如权重跟代价成反比。
总结来说，TIMD利用帧内预测模式在模板上的预测效果来筛选帧内预测模式，而且可以将2个帧内预测模式根据模板上的代价进行加权。TIMD的好处在于如果当前块选择了TIMD模式，则码流中不需要再去指示具体使用了哪种帧内预测模式，而是由解码器自己通过上述流程导出，一定程度上节省了开销。
通过上述简单地对几个帧内预测模式的介绍不难发现，DIMD模式的技术原理和TIMD模式的技术原理接近，都是利用解码器执行与编码器相同的操作来推断出当前编码单元的预测模式。这种预测模式在复杂度可接受的情况下能够省去对预测模式的索引的传输，达到节省开销的作用，提高压缩效率。但受限于可参考信息的局限性和本身并没有太多提高预测质量的部分，DIMD模式和TIMD模式在大面积纹理特性一致的区域效果较好，若纹理略有变化或者模板区域不能覆盖，则这种预测模式的预测效果较差。
此外，不管针对DIMD模式还是针对TIMD模式，其都对基于多种传统预测模式得到的预测块进行了融合或者都对基于多种传统预测模式得到的预测块进行了加权处理，预测块的融合可以生成单一预测模式所达不到的效果，DIMD模式虽然引入平面模式作为额外加权预测模式，以增加相邻重建样本与预测样本之间的空间关联性，进而能够提升帧内预测的预测效果，但是，由于平面模式的预测原理相对简单，对于一些右上角与左下角相差明显的预测块，将平面模式作为额外加权预测模式可能会带来反作用。
(4)、几何划分模式(geometric partitioning mode，GPM)以及角度加权预测(angular weighted prediction，AWP)。
在视频编解码标准中，传统的单向预测只找一个与当前块大小相同的参考块，传统的双向预测使用两个与当前块大小相同的参考块，且预测块每个点的像素值为两个参考块对应位置的平均值，即每一个参考块的所有点都占50％的比例。进一步的，双向加权预测使得两个参考块的比例可以不同，如第一个参考块中所有点都占75％的比例，第二个参考块中所有点都占25％的比例，但同一个参考块中的所有点的比例都相同。此外，对于一些优化方法，例如，解码器侧运动矢量的细化(decoder side motion vector refinement，DMVR)以及双向光流(bi-directional optical flow，BIO或BDOF)，其会使得参考像素或预测像素产生一些变化。
对于GPM或AWP而言，其也使用两个与当前块大小相同的参考块，但某些像素位置100％使用第一个参考块对应位置的像素值，某些像素位置100％使用第二个参考块对应位置的像素值，而在交界区域或者称过渡区域，按一定比例使用这两个参考块对应位置的像素值。交界区域的权重也是逐渐过渡的。具体这些权重如何分配，由GPM或AWP的权重导出模式决定。根据GPM或AWP的权重导出模式确定每个像素位置的权重。
当然，在某些情况下，比如说块尺寸很小的情况，可能某些GPM或AWP的模式下不能保证一定有某些像素位置100％使用第一个参考块对应位置的像素值，某些像素位置100％使用第二个参考块对应位置的像素值；此时，也可以认为GPM或AWP使用两个与当前块大小不相同的参考块，即各取所需的一部分作为参考块；即将权重不为0的部分作为参考块，而将权重为0的部分剔除出来，本申请对其具体实现不作限定。
图8是本申请实施例提供的GPM在正方形的块上的64种权重导出模式对应的权重图的示例。
如图8所示，GPM在正方形的块上存在64种权重导出模式对应的权重图。其中，对于每一个权重导出模式对应的权重图，黑色表示第一个参考块对应位置的权重值为0％，白色表示第一个参考块对应位置的权重值为100％，灰色区域则按颜色深浅的不同表示第一个参考块对应位置的权重值为大于0％小于100％的某一个权重值。第二个参考块对应位置的权重值则为100％减去第一个参考块对应位置的权重值。
图9是本申请实施例提供的权重导出模式的划分线的示例。
如图9所示，权重导出模式的划分线可以是GPM对应的两个预测模式的权重相同的点所组成的线。换言之，在GPM的权重矩阵中，划分线为GPM对应的两个预测模式的权重相同的点所组成的线。或者说，所述划分线可以是GPM的权重矩阵中权重发生变化的区域中、GPM对应的两个预测模式的权重相同的点所组成的线。或者说，权重为中值的点组成的线，划分线为中值权重对应的点组成的线，其中，中值权重的点有可能位于整个像素的中间位置，也有可能不在整像素的中间位置；以权重为0到8为例，中值权重可以是4。
图10是本申请实施例提供的AWP在正方形的块上的56种权重导出模式对应的权重图的示例。
如图10所示，AWP在正方形的块上的56种权重导出模式对应的权重图。其中，对于每一个权重导出模式对应的权重图，黑色表示第一个参考块对应位置的权重值为0％，白色表示第一个参考块对应位置的权重值为100％，灰色区域则按颜色深浅的不同表示第一个参考块对应位置的权重值为大于0％小于100％的某一个权重值。第二个参考块对应位置的权重值则为100％减去第一个参考块对应位置的权重值。
需要说明的是，GPM和AWP的权重导出方法可以不同。例如，GPM根据每种权重导出模式确定角度及偏移量，而后计算出每个权重导出模式对应的权重图。AWP首先根据每种权重导出模式确定一维的权重的线，然后使用类似于帧内角度预测的方法将一维的权重的线铺满整个图像，得到每个权重导出模式对应的权重图。当然，在其他可替代实施例中，每一个权重导出模式对应的权重图也可称为权重矩阵。
下面以GPM为例对权重导出方法进行说明。
编码器可根据每一个权重导出模式确定出对应的划分线，然后基于划分线确定出相应的权重矩阵。例如，编码器可利用表1确定权重导出模式对应的角度索引变量angleIdx和距离索引变量distanceIdx确定权重导出模式merge_gpm_partition_idx，角度索引变量angleIdx和距离索引变量distanceIdx可以认为是用于确定划分线的变量，即其分别用于确定划分线的角度和偏移量。编码器确定出每一个权重导出模式对应的划分线后，可基于每一个权重导出模式对应的划分线，确定每一个权重导出模式对应的权重矩阵。
表1
Figure PCTCN2022086448-appb-000001
Figure PCTCN2022086448-appb-000002
如表1所示，权重导出模式包括64种(例如图8所示的64种模式)，其索引(merge_gpm_partition_idx)的取值为0～63，其中，这64种权重导出模式中的每一种权重导出模式可对应一个角度索引变量angleIdx和距离索引变量distanceIdx。相当于，每一个权重导出模式可对应到一个划分线。当然，一个角度索引变量angleIdx或一个距离索引变量distanceIdx可以对应到一个或多个权重导出模式的索引，表1仅为本申请的示例，不应理解为对本申请的限制。
由于三个分量(如Y、Cb、Cr)都可以使用GPM，因此，可以将一个分量产生GPM的预测样本矩阵的过程分装到了一个子流程，即对于GPM的加权预测过程(Weighted sample prediction process for geometric partitioning mode)，三个分量都可以调用这个流程，只是调用的参数不同，本申请以亮度分量为例进行说明。示例性地，可以由GPM的加权预测过程导出当前亮度块的预测矩阵predSamplesL[xL][yL]；其中xL＝0..cbWidth–1，yL＝0..cbHeight-1。其中，nCbW设为cbWidth，nCbH设为cbHeight。
GPM的加权预测过程的输入包括：当前块的宽度nCbW；当前块的高度nCbH；2个(nCbW)x(nCbH)的预测样本矩阵predSamplesLA和predSamplesLB；GPM的“划分”角度索引变量angleIdx；GPM的距离索引变量distanceIdx；分量索引变量cIdx，例如cIdx为0时，可以用于表示亮度分量。GPM的加权预测过程的输出是：(nCbW)x(nCbH)的GPM预测样本矩阵pbSamples[x][y]；其中x＝0..nCbW–1，y＝0..nCbH-1。
预测样本矩阵pbSamples[x][y]可按如下方法导出：
示例性地，首先，可以按照如下方法导出变量nW，nH,shift1,offset1,displacementX,displacementY,partFlip以及shiftHor：
nW＝(cIdx＝＝0)？nCbW:nCbW*SubWidthC。
nH＝(cIdx＝＝0)？nCbH:nCbH*SubHeightC。
shift1＝Max(5,17-BitDepth)，其中BitDepth是编解码的比特深度。
offset1＝1<<(shift1-1)。
displacementX＝angleIdx。
displacementY＝(angleIdx+8)％32。
partFlip＝(angleIdx>＝13&&angleIdx<＝27)？0:1。
shiftHor＝(angleIdx％16＝＝8||(angleIdx％16！＝0&&nH>＝nW))？0:1。
接着，可按照以下方式导出变量offsetX和offsetY：
如果shiftHor的值为0：
offsetX＝(-nW)>>1；
offsetY＝((-nH)>>1)+(angleIdx<16？(distanceIdx*nH)>>3:-((distanceIdx*nH)>>3))；
否则(即shiftHor的值为1)：
offsetX＝((-nW)>>1)+(angleIdx<16？(distanceIdx*nW)>>3:-((distanceIdx*nW)>>3)；
offsetY＝(-nH)>>1。
然后，可按照如下方法导出预测样本矩阵pbSamples[x][y]；其中x＝0..nCbW–1，y＝0..nCbH-1)：
变量xL和yL按如下方法导出:
xL＝(cIdx＝＝0)？x:x*SubWidthC；
yL＝(cIdx＝＝0)？y:y*SubHeightC；
weightIdx＝(((xL+offsetX)<<1)+1)*disLut[displacementX]+(((yL+offsetY)<<1)+1)*disLut[displacementY]；
其中，可通过表2disLut[displacementX]：
表2
idx 0 2 3 4 5 6 8 10 11 12 13 14
disLut[idx] 8 8 8 4 4 2 0 -2 -4 -4 -8 -8
idx 16 18 19 20 21 22 24 26 27 28 29 30
disLut[idx] -8 -8 -8 -4 -4 -2 0 2 4 4 8 8
weightIdxL＝partFlip？32+weightIdx:32–weightIdx；
wValue＝Clip3(0,8,(weightIdxL+4)>>3)；
pbSamples[x][y]＝Clip3(0,(1<<BitDepth)-1,(predSamplesLA[x][y]*wValue+predSamplesLB[x][y]*(8-wValue)+offset1)>>shift1)。
其中，pbSamples[x][y]表示(x，y)点的预测样本。wValue表示为(x，y)点的一种预测模式的预测矩阵的预测值predSamplesLA[x][y]的权重，而(8-wValue)即为(x，y)点的另一种预测模式的预测矩阵的预测值predSamplesLB[x][y]的权重。
需要注意的是，对于一个权重导出模式，可以利用其针对每一个点推导出一个权重值wValue，然后计算一个GPM 的预测值pbSamples[x][y]。这种方式权重wValue不必写成一个矩阵的形式，但是可以理解如果把每个位置的wValue都保存到一个矩阵里，那它就是一个权重矩阵。每个点分别计算权重并加权得到GPM的预测值，或者计算出所有的权重再统一加权得到GPM的预测样本矩阵其原理是一样的。而本文中的诸多描述中使用权重矩阵的说法，是为了表述更容易理解，用权重矩阵画图更直观，其实也可以按每个位置的权重来描述。比如权重矩阵导出模式也可以说成权重导出模式，本申请对此不作具体限定。
此外，对于CU、PU以及TU的划分，其均属于基于矩形的划分方式；然而，GPM和AWP在没有划分的情况下实现了预测的非矩形的划分效果。GPM和AWP使用了2个参考块的权重的蒙版(mask)，即上述的权重图或者权重矩阵。这个蒙版确定了两个参考块在产生预测块时的权重，或者可以简单地理解为预测块的一部分位置来自于第一个参考块一部分位置来自于第二个参考块，而过渡区域(blending area)用两个参考块的对应位置加权得到，从而使过渡更平滑。GPM和AWP没有将当前块划分成两个CU或PU，于是在预测之后的残差的变换、量化、反变换、反量化等也都是将当前块作为一个整体来处理。
需要说明的是，GPM可以用权重矩阵组合两个帧间预测块。本申请将其扩展到组合两个任意的预测块。如两个帧间预测块，两个帧内预测块，一个帧间预测块加一个帧内预测块。甚至在屏幕内容编码中，还可以使用帧内块复制(intra block copy，IBC)模式的预测块或调色板(palette)模式的预测块作为其中的1个或2个预测块。为了表述方便，本申请将帧内模式、帧间模式、IBC模式、调色板模式统称为预测模式。预测模式可以理解为根据它编解码器可以产生当前块的一个预测块的信息。示例性地，在帧内预测中，预测模式可以是某个帧内预测模式，如DC模式，平面模式，各种帧内角度预测模式等。当然，也可以叠加某个或某些辅助的信息，比如帧内参考像素的优化方法，产生初步的预测块以后的优化方法(比如滤波)等。示例性地，在帧间预测中，预测模式可以是融合(merge)模式、运动矢量差的融合(Merge with Motion Vector Difference，MMVD)模式、高级运动矢量预测模式(Advanced Motion Vector Prediction，AMVP)。示例性地，预测模式可以是单向预测，也可以是双向预测或多假设预测。进一步的，如果帧间的预测模式使用单向预测，且能够确定一个运动信息，则可以根据运动信息能确定预测块。如果帧间的预测模式使用双向预测，且能够确定两个运动信息，则可以根据运动信息能确定预测块。
图11是本申请实施例提供的GPM或AWP的示意图。
如图11所示，GPM需要确定的信息可以表述为1个权重导出模式和2个预测模式。权重导出模式用来确定权重矩阵或权重，2个预测模式分别确定一个预测块或预测值。权重导出模式也被称为划分模式或权重矩阵导出模式。2个预测模式可以是相同或不同的预测模式，例如包括但不限于帧内预测模式、帧间预测模式、IBC模式以及调色板模式。
下面对与残差块进行变换相关的内容进行说明。
在进行编码时，会先对当前块进行预测，预测利用空间或者时间上的相关性能得到一个跟当前块相同或相似的图像。对一个块来说，预测块和当前块是完全相同的情况是有可能出现的，但是很难保证一个视频中的所有块都如此。特别是对自然视频，或者说相机拍摄的视频，因为图像的纹理复杂，且图像有噪音的存在等因素，通常预测块和当前块很像，但是有差异。而且视频中不规则的运动，扭曲形变，遮挡，亮度等的变化，当前块很难被完全预测。因此混合编码框架会将当前块的原始图像减去预测图像得到残差图像，或者说当前块减去预测块得到残差块。残差块通常要比原始图像简单很多，因而预测可以显著提升压缩效率。对残差块也不是直接进行编码，而是通常先进行变换。变换是把残差图像从空间域变换到频率域，去除残差图像的相关性。残差图像变换到频率域以后，由于能量大多集中在低频区域，变换后的非零系数大多集中在左上角。接下来利用量化来进一步压缩。而且由于人眼对高频不敏感，高频区域可以使用更大的量化步长。
图像变换技术是为了能够用正交函数或正交矩阵表示原始图像而对原图像所作的变换，该变换是二维线性可逆的。一般称原始图像为空间域图像，称变换后的图像为转换域图像(也称为频率域)，转换域图像可反变换为空间域图像。经过图像变换后，一方面能够更有效地反映图像自身的特征，另一方面也可使能量集中在少量数据上，更有利于图像的存储、传输及处理。
结合至图像视频编码领域，编码器在得到残差块后，可对残差块进行变换。变换的方式包括但不限于：离散余弦变换(Discrete Cosine Transform，DCT)和离散正弦变换(Discrete Sine Transform,DST)。由于DCT具有很强的能量集中特性，因此，原始图像经过DCT变换以后只有部分区域(例如左上角区域)存在非零系数。当然，在视频编解码中，图像是分割成块来处理的，因而变换也是基于块来进行的。视频编解码中可使用的DCT包括但不限于DCT2、DCT8型；视频编解码中可使用的DST包括但不限于DST7型。其中，DCT2型是视频压缩标准中常用的变换；VVC中可以使用DCT8型和DST7型。值得注意的是，变换在通常的视频的压缩中非常有用，但是也并非所有的块都必须要做变换，有些情况下变换反而不如不变换压缩效果好，因而，在某些情况下，编码器可以选择当前块是否使用变换。
编码器对当前图像中的当前块进行变换时，可以采用基函数或基图像对当前块的残差块进行变换。基图像是基函数的图像表达形式。
图12是本申请实施例提供的DCT2型的基图像的示例。
如图12所示，DCT2型的基图像可以是根据基函数绘制的8×8个小块组成的基图像，其中每个小块由8×8个元素(子块)组成。在具体实现中，可以利用8×8个小块组成的基图像对8x8块大小的块进行变换，得到的是一个8x8的变换系数矩阵。
如上文所述，VVC中除了可以使用DCT2型对残差块进行基础变换，还可以使用DCT8型和DST7型对残差块进行基础变换，即VVC中的多变换模式选择(multiple transform selection，MTS)技术，其中基础变换使用的基函数对应的变换类型也可称为基础变换使用的变换核类型。编码器进行基础变换时，基于不同的残差分布特性，选择最合适的变换核类型能提高压缩性能。基础变换也可以叫主变换(core transform)。MTS可以通过一些语法元素来选择变换核类型。下面结合表3列出了一种通过语法元素选择变换核类型的MTS。
表3
Figure PCTCN2022086448-appb-000003
Figure PCTCN2022086448-appb-000004
如表3所示，如果MTS_CU_flag的值为0，则水平方向和竖直方向的基础变换的变换核类型都用DCT2。如果MTS_CU_flag的值为1，且MTS_Hor_flag的值为0，MTS_Ver_flag的值为0，则水平方向的变换核类型用DST7，竖直方向的变换核类型用DST7。
在VVC标准中，还可以对MTS的语法进行改写或者说简化，即VVC用一个语法元素mts_idx来确定基础变换的变换核类型。
表4
mts_idx 0 1 2 3 4
trTypeHor 0 1 2 1 2
trTypeVer 0 1 1 2 2
如表4所示，trTypeHor表示水平变换的变换核类型，trTypeVer表示竖直变换的变换核类型，trTypeHor和trTypeVer为0表示DCT2型变换，1表示DST7型变换，2表示DCT8型变换。
由于残差的分布和帧内预测模式有一定的相关性，因此基础变换也可以利用这种相关性。一种方法是根据帧内预测模式对MTS的变换核类型进行分组。一个分组的例子如下表所示。
表5
0 0<＝mode<＝1
1 2<＝mode<＝12
2 13<＝mode<＝23
3 24<＝mode<＝34
如表5所示，如果帧内预测模式的索引为0或1，对应选择MTS的索引为0的变换核类型组，VVC中索引为0的模式是Planar，VVC中索引为1的模式是DC，DC和Planar都会产生较平坦的预测值。如果帧内预测模式的索引为2到12，对应选择MTS的索引为1的变换核类型组。根据帧内预测模式的图可以看到，2到12的角度都指向左下方向。
需要说明的是，每一个变换核类型组里可能有一种水平变换和竖直变换的变换核类型的选择，也可能有多种水平变换和竖直变换的变换核类型的选择，也就是说，在根据帧内预测模式选择了哪一变换核类型组以后还可以做进一步的细分，比如说再通过一些标识或块尺寸信息等进一步选择变换核类型，这里就不再赘述。重点是基础变换可以根据帧内预测模式选择变换核类型组。也可以看到这种根据帧内预测模式选择基础变换的变换核类型组的方法，在将来也有可能推测出更细致的基础变换的分组，本申请对此不作具体限定。
此外，本申请在主变换中涉及的变换核类型也可称为变换矩阵、变换类型或变换核等具有类似或相同含义的术语，在主变换中涉及的变换核类型组也可称为变换矩阵组、变换类型组或变换核组等具有类似或相同含义的术语，本申请对此不作具体限定。也即是说，本申请涉及的选择变换核类型或变换核类型组，也可以称为选择变换矩阵或变换矩阵组，还可以称为选择变换类型或选择变换类型组，还可以称为选择变换核或变换核组。变换核类型或者变换类型可以包括DCT2，DCT8，DST7等，还可以包括DCT5,DST4,DST1或标识变换(Identity Transform，IDTR)等。
另外，不同大小的块可以用对应大小的变换核类型，本申请对此不再进行赘述。
值得注意的是，由于图像都是2维的，而直接进行二维的变换运算量和内存开销都是硬件条件所不能接受的，因此上述DCT2型，DCT8型，DST7型变换都是拆分成水平方向和竖直方向的一维变换，即分成两步进行的。如先进行水平方向的变换再进行竖直方向的变换，或者先进行竖直方向的变换再进行水平方向的变换。上述变换方法对水平方向和竖直方向的纹理比较有效，但是对斜向的纹理效果就会差一些。由于水平和竖直方向的纹理是最常见的，因而上述的变换方法对提升压缩效率是非常有用的。但是，随着对技术发展，仅处理水平和竖直方向的纹理的残差，已经满足不了对压缩效率需求。
基于此，本申请引入了二次变换的概念，即编码器可以在主变换(primary transform)的基础上进行二次变换，以提升压缩效率。
示例性地，主变换可用于处理水平和竖直方向的纹理，主变换也可称为基础变换，例如主变换包括但不限于：上述DCT2型，DCT8型，DST7型变换。二次变换用于处理斜向纹理，例如，二次变换包括但不限于：低频不可分变换(low frequency non-separable transform，LFNST)。在编码端，二次变换用于主变换之后且量化之前。在解码端，二次变换用于反量化之后且反主变换之前。
图13是本申请实施例提供的LFNST的示例。
如图13所示，在编码端，LFNST对基础变换后的左上角的低频系数进行二次变换。主变换通过对图像进行去相关性，把能量集中到左上角。而二次变换对主变换的低频系数再去相关性。在编码端，将16个系数输入到4x4的LFNST时，输出的是8个系数；将64个系数输入到8x8的LFNST时，输出的是16个系数。在解码端，将8个系数输入到4x4的反LFNST时，输出的是16个系数；将16个系数输入到8x8的反LFNST时，输出的是64个系数。
编码器对当前图像中的当前块进行二次变换时，可以采用选择的变换矩阵组中的某一个变换矩阵对当前块的残差块进行变换。以二次变换为LFNST为例，变换矩阵可以指用于对某个斜向纹理进行变换的矩阵，变换矩阵组可以包括用于对某些类似的斜向纹理进行变换的矩阵。
图14是本申请实施例提供的LFNST的变换矩阵组的示例。
如图14的(a)至(d)所示，LFNST可具有4组变换矩阵，且同一组变换矩阵中的变换矩阵具有类似的斜向纹理。示例性地，图14中的(a)所示的变换矩阵组可以是索引为0的变换矩阵组，图14中的(b)所示的变换矩阵组可以是索引为1的变换矩阵组，图14中的(c)所示的变换矩阵组可以是索引为2的变换矩阵组，图14中的(d)所示的变换矩阵组可以是索引 为3的变换矩阵组。
应当理解，本申请在二次变换中涉及的变换矩阵也可称为变换核、变换核类型或基函数等具有类似或相同含义的术语，在二次变换中涉及的变换矩阵组也可称为变换核组、变换核类型组或基函数组等具有类似或相同含义的术语，本申请对此不作具体限定。也即是说，本申请涉及的选择变换矩阵或变换矩阵组，也可以称为选择变换核类型或变换核类型组，还可以称为选择变换类型或选择变换类型组，还可以称为选择变换核或变换核组。
下面对LFNST应用于帧内编码的块的相关方案进行说明。
帧内预测使用当前块周边已重建的像素作为参考对当前块进行预测，由于目前视频都是从左向右从上向下编码的，因而当前块可使用的参考像素通常在左侧和上侧。角度预测按照指定的角度将参考像素平铺到当前块作为预测值，这意味着预测块会有明显的方向纹理，而当前块经过角度预测后的残差在统计上也会体现出明显的角度特性。因而，LFNST所选用的变换矩阵可以跟帧内预测模式进行绑定，即确定了帧内预测模式以后，LFNST可以使用纹理方向与帧内预测模式的角度特征相适应的一组(set)变换矩阵。
示例性地，假设LFNST总共有4组变换矩阵，每组有2个变换矩阵。表6给出了帧内预测模式和变换矩阵组的对应关系。
表6
Figure PCTCN2022086448-appb-000005
如表6所示，帧内预测模式0～81可与4组变换矩阵的索引关联。
值得注意的是，色度帧内预测使用的跨分量预测模式为81到83，亮度帧内预测并没有这几种模式。LFNST的变换矩阵可以通过转置来用一个变换矩阵组对应处理更多的角度，举例来说，帧内预测模式13～23和帧内预测模式45～55的模式都对应变换矩阵组2，但是，帧内预测模式13～23明显是接近于水平的模式，而帧内预测模式45～55明显是接近于竖直的模式，帧内预测模式45～55的模式对应的变换后反变换后需要通过转置来进行适配。
在具体实现中，由于LFNST共有4组变换矩阵，编码端可根据当前块使用的帧内预测模式确定LFNST使用哪一组变换矩阵，进而在确定的一组变换矩阵中确定使用的变换矩阵。相当于，可以利用帧内预测模式和LFNST的变换矩阵组之间的相关性，从而减少了选择LFNST的变换矩阵在码流中的传输。而当前块是否会使用LFNST，以及如果使用LFNST，是使用一个组中的第一个还是第二个，可以通过码流和一些条件来确定的。
当然，考虑到普通帧内预测模式有67种，而LFNST只有4组变换矩阵，因此，多种相近的角度预测模式只能对应一组LFNST变换矩阵，这是一种性能和复杂度折中的考虑，因为每种变换矩阵都需要占用存储空间来保存变换矩阵的系数。而随着对压缩效率要求的提升，以及硬件能力的提升，LFNST也可以设计的更加复杂。比如使用更大的变换矩阵，更多的变换矩阵组，以及每个变换矩阵组使用更多的变换矩阵。示例性地，表7给出了帧内预测模式和变换矩阵组的另一种对应关系。
表7
帧内预测模式 -14 -13 -12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1
变换矩阵组的索引 2 2 2 2 2 2 2 2 2 2 2 2 2 2
帧内预测模式 0 1 2 3 4 5 6 7 8 9 10 11 12 13
变换矩阵组的索引 0 1 2 3 4 5 6 7 8 9 10 11 12 13
帧内预测模式 14 15 16 17 18 19 20 21 22 23 24 25 26 27
变换矩阵组的索引 14 15 16 17 18 19 20 21 22 23 24 25 26 27
帧内预测模式 28 29 30 31 32 33 34 35 36 37 38 39 40 41
变换矩阵组的索引 28 29 30 31 32 33 34 33 32 31 30 29 28 27
帧内预测模式 42 43 44 45 46 47 48 49 50 51 52 53 54 55
变换矩阵组的索引 26 25 24 23 22 21 20 19 18 17 16 15 14 13
帧内预测模式 56 57 58 59 60 61 62 63 64 65 66 67 68 69
变换矩阵组的索引 12 11 10 9 8 7 6 5 4 3 2 2 2 2
帧内预测模式 70 71 72 73 74 75 76 77 78 79 80      
变换矩阵组的索引 2 2 2 2 2 2 2 2 2 2 2      
如表7所示，使用35个变换矩阵组，每个变换矩阵组使用3个变换矩阵。变换矩阵组与帧内预测模式的对应关系可以实现为：对于帧内预测模式0～34，正向对应变换矩阵组0～34，即预测模式的编号越大则变换矩阵组的索引越大；对于帧内预测模式35～67，由于转置的原因，其反向对应2～33，即预测模式的编号越大则变换矩阵组的索引越小；对于剩余的预测模式，其均可以统一对应到索引为2的变换矩阵组。也就是说如果不考虑转置，一种帧内预测模式对应一个变换矩阵组。按这种设计，每一个帧内预测模式对应的残差可以得到更适配的LFNST变换矩阵，压缩性能也会提高。
当然，理论上宽角度模式也可以做到一对一，但是这种设计的性价比较低，本申请对此不再作具体说明。
此外，对于LFNST而言，为了能够使得MIP适配到变换矩阵组，本申请将平面模式适配到的变换矩阵组可作为与MIP适配的变换矩阵组。
需要指出的是，LFNST仅是二次变换的一种示例，不应理解为对二次变换的限制。例如，LFNST是不可分离的二次变换，在其他可替代实施例中，还可以采用可分离的二次变换提升斜向纹理的残差的压缩效率。
图15是本申请实施例提供的解码框架200的示意性框图。
如图15所示，该解码框架200可包括熵解码单元210、反变换反量化单元220、残差单元230、帧内预测单元240、帧间预测单元250、环路滤波单元260、解码图像缓存单元270。
熵解码单元210接收并解码码流后，以获取预测块和频域残差块，针对频域残差块，通过反变换反量化单元220进行变换与反量化等步骤，可获取时域残差块，残差单元230将帧内预测单元240或帧间预测单元250预测得到的预测块叠加至经过通过反变换反量化单元220进行变换与反量化之后的时域残差块，可得到重建块。
图16是本申请实施例提供的解码方法300的示意性流程图。应理解，该解码方法300可由解码器执行。例如应用于图15所示的解码框架200。为便于描述，下面以解码器为例进行说明。
如图16所示，所述解码方法300可包括以下中的部分或全部：
S310，解码器解码码流获取当前块的第一变换系数；
S320，解码器对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数；
S330，解码器对所述第二变换系数进行第二变换，得到所述当前块的残差块；
S340，解码器基于预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对所述当前块进行预测，得到所述当前块的预测块；
S350，解码器基于所述当前块的预测块和所述当前块的残差块，得到所述当前块的重建块。
本申请在预测模式导出模式和第二变换的基础上，引入了第一变换，能够提升当前块的解压缩效率。
示例性地，所述第一变换可以是LFNST，即本申请将LFNST结合至预测模式导出模式，可以提升斜向纹理的残差的压缩效率。
当然，预测模式导出模式与LFNST的适配方法也适用于其他的二次变换方法。例如，LFNST是不可分离的二次变换，在其他可替代实施例中，预测模式导出模式也可适用于可分离的二次变换，本申请对此不作具体限定。
在一些实施例中，所述预测模式导出模式可以包括解码器侧帧内模式导出DIMD模式或基于模板的帧内模式导出TIMD模式。
在一些实施例中，所述S320可包括：
解码所述码流，获取第一标识和第二标识；
若所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则对所述第一变换系数进行第一变换，得到所述第二变换系数。
示例性地，所述当前序列为包括所述当前块的图像序列。
示例性地，所述第一标识可用于控制当前序列是否使用预测模式导出模式。
示例性地，若所述第一标识的取值为第一数值，则标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，若所述第一标识的取值为第二数值，则标识不允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测。作为一种实现方式，所述第一数值为0且所述第二数值为1，作为另一种实现方式，所述第一数值为1且所述第二数值为0。当然，所述第一数值或所述第二数值也可为其他取值。
示例性地，所述第二标识用于控制当前序列是否使用第一变换。
示例性地，若所述第二标识的取值为第三数值，则标识允许使用所述第一变换对所述当前序列中的图像块进行变换，若所述第二标识的取值为第四数值，则标识不允许使用所述第一变换对所述当前序列中的图像块进行变换。作为一种实现方式，所述第三数值为0且所述第四数值为1，作为另一种实现方式，所述第三数值为1且所述第四数值为0。当然，所述第三数值或所述第四数值也可为其他取值。
示例性地，若将第一标识记为sps_timd/dimd_enabled_flag，将第二标识记为sps_lfnst_enabled_flag，则在sps_timd/dimd_enabled_flag和sps_lfnst_enabled_flag的取值均为1时，对所述第一变换系数进行第一变换，得到所述第二变换系数。
示例性地，若所述第一标识用于标识不允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，和/或，所述第二标识用于标识不允许使用所述第一变换对所述当前序列中的图像块进行变换，则对所述第一变换系数不进行所述第一变换，或者说，可以直接对所述第一变换系数进行所述第二变换得到所述当前块的残差值。
当然，在其他可替代实施例中，也可以将所述第一标识和/或所述第二标识替换为图像(picture)、切片(slice)、最大编码单元(largest coding unit，LCU)、编码树单元(Coding Tree Unit，CTU)、编码单元(coding unit，CU)、预测单元(prediction unit，PU)或变换单元(transform unit，TU)等层级的标识。或者，也可以在所述第一标识和所述第二标识的基础上，针对是否使用所述预测模式导出模式或是否使用所述第一变换，增加图像、切片、LCU、CTU、CU、PU或TU等层级的标识，本申请实施例对此不作具体限定。
在一些实施例中，所述S320可包括：
若所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则解码所述码流，获取第三标识；
若所述第三标识用于标识允许所述当前序列中的图像块同时使用所述预测模式导出模式和所述第一变换，则对所述第一变换系数进行第一变换，得到所述第二变换系数。
示例性地，所述第三标识用于控制预测模式导出模式和第一变换是否可以共用。
示例性地，若所述第三标识的取值为第五数值，则标识允许所述当前序列中的图像块同时使用所述预测模式导出模式和所述第一变换；若所述第三标识的取值为第六数值，则不允许所述当前序列中的图像块同时使用所述预测模式导出模式和所述第一变换。作为一种实现方式，所述第五数值为0且所述第六数值为1，作为另一种实现方式，所述第五数值为1且所述第六数值为0。当然，所述第五数值或所述第六数值也可为其他取值。
示例性地，若将第一标识记为sps_timd/dimd_enabled_flag，将第二标识记为sps_lfnst_enabled_flag，并将第三标识记为sps_timd/dimd_lfnst_enabled_flag，则在sps_timd/dimd_enabled_flag和sps_lfnst_enabled_flag的取值均为1时，确定sps_timd/dimd_lfnst_enabled_flag是否为1，并在sps_timd/dimd_lfnst_enabled_flag为1时，对所述第一变换系数进行第一变换，得到所述第二变换系数。
当然，在其他可替代实施例中，也可以将所述第三标识替换为图像(picture)、切片(slice)、最大编码单元(largest coding unit，LCU)、编码树单元(Coding Tree Unit，CTU)、编码单元(coding unit，CU)、预测单元(prediction unit，PU)或变换单元(transform unit，TU)等层级的标识。或者，也可以在所述第三标识的基础上，针对是否使用所述预测模式导出模式或是否使用所述第一变换，增加图像、切片、LCU、CTU、CU、PU或TU等层级的标识，本申请实施例对此不作具体限定。
在一些实施例中，所述S320可包括：
若所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则解码器在所述当前块的高和/或宽大于或等于第一阈值的情况下，对所述第一变换系数进行第一变换，得到所述第二变换系数。
示例性地，若将第一标识记为sps_timd/dimd_enabled_flag，将第二标识记为sps_lfnst_enabled_flag，并将第三标识记为sps_timd/dimd_lfnst_enabled_flag，则在sps_timd/dimd_enabled_flag和sps_lfnst_enabled_flag的取值均为1时，解码器确定所述当前块的高和/或宽，并在所述当前块的高和/或宽大于或等于第一阈值的情况下，对所述第一变换系数进行第一变换，得到所述第二变换系数。
示例性地，所述第一阈值可以为4、8、16、32、64或其他取值。
在一些实施例中，所述S320之前，所述方法300还可包括：
解码器确定所述第一变换使用的变换矩阵组。
需要说明的是，由于预测模式导出模式是结合2种预测模式(即第一帧内预测模式和第二帧内预测模式)对当前块进行预测的，而利用不同的帧内预测模式对当前块进行预测得到的预测块可能具有不同的纹理特征，因此，如果当前块选择了预测模式导出模式，则说明第一帧内预测模式有可能会使得当前块的预测块会表现出一种纹理特性，而第二帧内预测模式有可能会使得当前块的预测块表现出另一种纹理特性；换言之，对当前块进行预测后，从统计角度来说，当前块的残差块也会表现出两种纹理特性，也即是说，当前块的残差块并不一定是符合某一种预测模式能够体现的规律。因此，针对预测模式导出模式，解码器在对第一变换系数进行第一变换之前，需要先确定与其特性适配的变换矩阵组。然而，由于第一变换使用的变换矩阵组通常是基于单个帧内预测模式定义的变换矩阵组，因此，针对预测模式导出模式，需要进一步完善确定第一变换使用的变换矩阵的相关方案，下面其各种实现方式进行示例性说明。
在一些实施例中，所述第一变换使用的变换矩阵组与平面模式或直流DC模式适配的变换矩阵组相同。
示例性地，解码器测检查当前块的预测模式时，如果当前块使用预测模式导出模式，则解码器将预测模式导出模式和平面模式(或DC模式)归为一类，并根据平面模式(或DC模式)适配用于第一变换的变换矩阵组。或者说，解码器测检查当前块的预测模式时，如果当前块使用预测模式导出模式，则编码器可以返回当前块的预测模式为平面模式(或DC模式)，以便解码器将与平面模式(或DC模式)适配的变换矩阵组，确定为所述第一变换使用的变换矩阵组。或者说，解码器测检查当前块的预测模式时，如果当前块使用预测模式导出模式，则解码器可以认为针对当前块的第一变换中采用的变换矩阵组可以为与平面模式(或DC模式)适配的变换矩阵组。
本实施例中，由于平面模式(或DC模式)与预测模式导出模式均能够体现出多种纹理特性，因此，将与平面模式或直流DC模式适配的变换矩阵组确定为所述第一变换使用的变换矩阵组，不仅能够实现基于预测模式导出模式和第一变换对当前块进行解码，还能够保证第一变换使用的变换矩阵组的纹理特性尽可能的接近当前块的残差块的纹理特性，提升了解压缩效率。
在一些实施例中，解码器基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式；其中，所述第一变换使用的变换矩阵组与所述第三帧内预测模式适配的变换矩阵组相同。
示例性地，解码器可将所述第三帧内预测模式适配的变换矩阵组，确定为所述第一变换使用的变换矩阵组。
示例性地，解码器测检查当前块的预测模式时，如果当前块使用预测模式导出模式，则解码器基于所述第一帧内预测模式和所述第二帧内预测模式确定第三帧内预测模式，并将预测模式导出模式和第三帧内预测模式归为一类，以便解码器可以根据第三帧内预测模式适配用于第一变换的变换矩阵组。或者说，解码器测检查当前块的预测模式时，如果当前块使用预测模式导出模式，则编码器可以返回当前块的预测模式为第三帧内预测模式，以便解码器将与第三帧内预测模式适配的变换矩阵组，确定为所述第一变换使用的变换矩阵组。或者说，解码器测检查当前块的预测模式时，如果当前块使用预测模式导出模式，则解码器可以认为针对当前块的第一变换中采用的变换矩阵组可以为与第三帧内预测模式适配的变换矩阵组。
当然，在其他可替代实施例中，解码器也可以不显式地先确定所述第三帧内预测模式，再通过所述第三帧内预测模式确定所述第一变换使用的变换矩阵组。而是直接把所述第三帧内预测模式适配的变换矩阵组作为所述第一变换使用的变换矩阵组。
在一些实施例中，解码器将所述第一帧内预测模式和所述第二帧内预测模式中的缺省预测模式，确定为所述第三帧内预测模式；或解码器基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式；或解码器基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式；或解码器基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式。
示例性地，解码器测检查当前块的预测模式时，如果当前块使用预测模式导出模式，则在选择第一变换使用的变换矩阵组时，可基于第一帧内预测模式和第二帧内预测模式确定第一变换使用的变换矩阵组。作为一种实现方式，可以在任何情况下都用第一帧内预测模式来确定，即在任何情况下都将与第一帧内预测模式适配的变换矩阵组确定为第一变换使用的变换矩阵组；或者，可以在任何情况下都用第二帧内预测模式来确定，即在任何情况下都将与第二帧内预测模式适配的变换矩阵组确定为第一变换使用的变换矩阵组。作为另一种实现方式，可以在某些情况下用第一帧内预测模式来 确定，即在某些情况下将与第一帧内预测模式适配的变换矩阵组确定为第一变换使用的变换矩阵组；或者，可以在某些情况下用第二帧内预测模式来确定，即在某些情况下都将与第二帧内预测模式适配的变换矩阵组确定为第一变换使用的变换矩阵组；甚至于，在以在某些情况下用平面模式或DC模式来确定，即在某些情况下都将与平面模式或DC模式适配的变换矩阵组确定为第一变换使用的变换矩阵组。所谓用某个预测模式来确定，即使将预测模式导出模式与所述某个预测模式归为一类，以便解码器可以根据所述某个预测模式适配用于第一变换的变换矩阵组。或者说，解码器在检查当前块的预测模式时，如果当前块使用预测模式导出模式，则可以返回所述某个预测模式，以便解码器可以根据所述某个预测模式适配用于第一变换的变换矩阵组。或者说，解码器在检查当前块的预测模式时，如果当前块使用预测模式导出模式，则解码器可以认为针对当前块的第一变换中采用的变换矩阵组可以为与所述某个预测模式适配的变换矩阵组。
在一些实施例中，解码器基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式时，可将所述第一帧内预测模式和所述第二帧内预测模式中权重最大的帧内预测模式，确定为所述第三帧内预测模式。
示例性地，解码器基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式时，权重大的帧内预测模式的优先级高于权重小的帧内预测模式的优先级。例如，若所述第一帧内预测模式的权重大于所述第二帧内预测模式的权重，则将所述第一帧内预测模式确定为所述第三帧内预测模式，若所述第二帧内预测模式的权重大于所述第一帧内预测模式的权重，则将所述第二帧内预测模式确定为所述第三帧内预测模式。
在一些实施例中，解码器基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式时，若所述第一帧内预测模式和所述第二帧内预测模式包括角度预测模式和非角度预测模式，则将所述角度预测模式，确定为所述第三帧内预测模式。
示例性地，解码器基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式时，角度预测模式作为第三帧内预测模式的优先级高于非角度预测模式作为第三帧内预测模式的优先级。例如，若所述第一帧内预测模式为角度预测模式，所述第二帧内预测模式为非角度预测模式(例如平面模式或DC模式)，则将第一帧内预测模式(即角度预测模式)，确定为所述第三帧内预测模式。
在一些实施例中，若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值小于或等于第二阈值，则将第一预测角度对应的帧内预测模式，确定为所述第三帧内预测模式，其中，所述第一预测角度根据所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度确定：若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值大于所述第二阈值，则将平面模式或直流DC模式确定为所述第三帧内预测模式。
示例性地，解码器基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式时，若第一帧内预测模式的预测角度和第二帧内预测模式的预测角度比较接近，则解码器可以将第一帧内预测模式、第二帧内预测模式、或预测角度位于第一帧内预测模式的预测角度和第二帧内预测模式的预测角度中间的帧内预测模式。例如，若第一帧内预测模式的预测角度和第二帧内预测模式的预测角度差别较大，则解码器可以将平面模式或者DC模式确定为第三帧内预测模式。
在一些实施例中，解码器将所述预测模式导出模式适配的变换矩阵组，确定为所述第一变换使用的变换矩阵组。
示例性地，解码器可针对预测模式导出模式定义适配或专用的变换矩阵组。
在一些实施例中，所述第一变换用于处理所述当前块中沿倾斜方向上的纹理，所述第二变换用于处理所述当前块中沿水平方向上的纹理和沿竖直方向上的纹理。
应当理解，解码端的第一变换为编码端的第一变换的反变换，解码端的第二变换为编码端的第二变换的反变换。例如，针对编码端而言，第一变换为上文涉及的二次变换，第二变换可以为上文涉及的基础变换或主变换，则针对解码端而言，所述第一变换可以是上文涉及的二次变换的反变换或逆变换。例如，针对编码端而言，所述第一变换可以是反LFNST，所述第二变换可以是DCT2型，DCT8型或DST7型等；则对解码器而言，所述第一变换可以是反(逆)LFNST，所述第二变换可以是反(逆)DCT2型，反(逆)DCT8型或反(逆)DST7型等。
上文中从解码器的角度详细描述了根据本申请实施例的解码方法，下面将结合图17，从编码器的角度描述根据本申请实施例的编码方法。
图17是本申请实施例提供的编码方法400的示意性流程图。应理解，该编码方法400可由编码器执行。例如应用于图1所示的编码框架100。为便于描述，下面以编码器为例进行说明。
如图17所示，所述编码方法400可包括：
S410，基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对当前块进行预测，得到所述当前块的预测块；
S420，基于所述当前块的预测块，得到所述当前块的残差块；
S430，对所述当前块的残差块进行第三变换，得到所述当前块的第三变换系数；
S440，对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数；
S450，对所述第四变换系数进行编码。
应当理解，解码端的第一变换为编码端的第四变换的反变换，解码端的第二变换为编码端的第三变换的反变换。例如，所述第三变换为上文涉及的基础变换或主变换，所述第四变换为上文涉及的二次变换，相应的，所述第一变换为二次变换的反变换(或逆变换)，所述第二变换可以为基础变换或主变换的反变换(或逆变换)。例如，所述第一变换可以是反(逆)LFNST，所述第二变换可以是反(逆)DCT2型，反(逆)DCT8型或反(逆)DST7型等；相应的，所述第三变换可以是DCT2型，DCT8型或DST7型等，所述第四变换可以是LFNST。
在一些实施例中，所述预测模式导出模式包括解码器侧帧内模式导出模式或基于模板的帧内模式导出模式。
在一些实施例中，所述S450可包括：
对第一标识、第二标识以及所述第四变换系数进行编码；
其中，所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，所述第二标识 用于标识允许使用所述第四变换对所述当前序列中的图像块进行变换。
在一些实施例中，所述S450可包括：
对所述第一标识、所述第二标识、所述第四变换系数以及第三标识进行编码；
其中，所述第三标识用于标识允许所述当前序列中的图像块同时使用所述预测模式导出模式和所述第四变换。
在一些实施例中，所述S440可包括：
在所述当前块的高和/或宽大于或等于第一阈值的情况下，对所述第三变换系数进行所述第四变换，得到所述第四变换系数。
在一些实施例中，所述S440之前，所述方法400还可包括：
确定所述第四变换使用的变换矩阵组。
在一些实施例中，所述第四变换使用的变换矩阵组与平面模式或直流DC模式适配的变换矩阵组相同。
在一些实施例中，所述确定所述第四变换使用的变换矩阵组，包括：
基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式；
其中，所述第四变换使用的变换矩阵组与所述第三帧内预测模式适配的变换矩阵组相同。
在一些实施例中，所述基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式，包括：
将所述第一帧内预测模式和所述第二帧内预测模式中的缺省预测模式，确定为所述第三帧内预测模式；或
基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式；或
基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式；或
基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式。
在一些实施例中，所述基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式，包括：
将所述第一帧内预测模式和所述第二帧内预测模式中权重最大的帧内预测模式，确定为所述第三帧内预测模式。
在一些实施例中，所述基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式，包括：
若所述第一帧内预测模式和所述第二帧内预测模式包括角度预测模式和非角度预测模式，则将所述角度预测模式，确定为所述第三帧内预测模式。
在一些实施例中，所述基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式，包括：
若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值小于或等于第二阈值，则将第一预测角度对应的帧内预测模式，确定为所述第三帧内预测模式，其中，所述第一预测角度根据所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度确定：
若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值大于所述第二阈值，则将平面模式或直流DC模式确定为所述第三帧内预测模式。
在一些实施例中，所述确定所述第四变换使用的变换矩阵组，包括：
将所述预测模式导出模式适配的变换矩阵组，确定为所述第四变换使用的变换矩阵组。
在一些实施例中，所述第四变换用于处理所述当前块中沿倾斜方向上的纹理，所述第三变换用于处理所述当前块中沿水平方向上的纹理和沿竖直方向上的纹理。
应当理解，编码方法可以理解为解码方法的逆过程，因此，所述编码方法400的具体方案可参见解码方法300的相关内容，为便于描述，本申请对此不再赘述。
以上结合附图详细描述了本申请的优选实施方式，但是，本申请并不限于上述实施方式中的具体细节，在本申请的技术构思范围内，可以对本申请的技术方案进行多种简单变型，这些简单变型均属于本申请的保护范围。例如，在上述具体实施方式中所描述的各个具体技术特征，在不矛盾的情况下，可以通过任何合适的方式进行组合，为了避免不必要的重复，本申请对各种可能的组合方式不再另行说明。又例如，本申请的各种不同的实施方式之间也可以进行任意组合，只要其不违背本申请的思想，其同样应当视为本申请所公开的内容。还应理解，在本申请的各种方法实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
上文详细描述了本申请的方法实施例，下文结合图18至图20，详细描述本申请的装置实施例。
图18是本申请实施例的解码器500的示意性框图。
如图18所示，所述解码器500可包括：
解码单元510，用于解码码流获取当前块的第一变换系数；
变换单元520，用于：
对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数；
对所述第二变换系数进行第二变换，得到所述当前块的残差块；
预测单元530，用于基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对所述当前块进行预测，得到所述当前块的预测块；
重建单元540，用于基于所述当前块的预测块和所述当前块的残差块，得到所述当前块的重建块。
在一些实施例中，所述预测模式导出模式包括解码器侧帧内模式导出模式或基于模板的帧内模式导出模式。
在一些实施例中，所述变换单元520具体用于：
解码所述码流，获取第一标识和第二标识；
若所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则对所述第一变换系数进行第一变换，得到所述第二变换系数。
在一些实施例中，所述变换单元520具体用于：
若所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则解码所述码流，获取第三标识；
若所述第三标识用于标识允许所述当前序列中的图像块同时使用所述预测模式导出模式和所述第一变换，则对所述第一变换系数进行第一变换，得到所述第二变换系数。
在一些实施例中，所述变换单元520具体用于：
若所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则在所述当前块的高和/或宽大于或等于第一阈值的情况下，对所述第一变换系数进行第一变换，得到所述第二变换系数。
在一些实施例中，所述变换单元520对所述第二变换系数进行第二变换，得到所述当前块的残差块之前，还用于：
确定所述第一变换使用的变换矩阵组。
在一些实施例中，所述变换单元520具体用于：
在一些实施例中，所述第一变换使用的变换矩阵组与平面模式或直流DC模式适配的变换矩阵组相同。
在一些实施例中，所述变换单元520具体用于：
基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式；
其中，所述第一变换使用的变换矩阵组与所述第三帧内预测模式适配的变换矩阵组相同。
在一些实施例中，所述变换单元520具体用于：
将所述第一帧内预测模式和所述第二帧内预测模式中的缺省预测模式，确定为所述第三帧内预测模式；或
基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式；或
基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式；或
基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式。
在一些实施例中，所述变换单元520具体用于：
将所述第一帧内预测模式和所述第二帧内预测模式中权重最大的帧内预测模式，确定为所述第三帧内预测模式。
在一些实施例中，所述变换单元520具体用于：
若所述第一帧内预测模式和所述第二帧内预测模式包括角度预测模式和非角度预测模式，则将所述角度预测模式，确定为所述第三帧内预测模式。
在一些实施例中，所述变换单元520具体用于：
若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值小于或等于第二阈值，则将第一预测角度对应的帧内预测模式，确定为所述第三帧内预测模式，其中，所述第一预测角度根据所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度确定：
若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值大于所述第二阈值，则将平面模式或直流DC模式确定为所述第三帧内预测模式。
在一些实施例中，所述变换单元520具体用于：
将所述预测模式导出模式适配的变换矩阵组，确定为所述第一变换使用的变换矩阵组。
在一些实施例中，所述第一变换用于处理所述当前块中沿倾斜方向上的纹理，所述第二变换用于处理所述当前块中沿水平方向上的纹理和沿竖直方向上的纹理。
图19是本申请实施例的编码器600的示意性框图。
如图19所示，所述编码器600可包括：
预测单元610，用于基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对当前块进行预测，得到所述当前块的预测块；
残差单元620，用于基于所述当前块的预测块，得到所述当前块的残差块；
变换单元630，用于：
对所述当前块的残差块进行第三变换，得到所述当前块的第三变换系数；
对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数；
编码单元640，用于对所述第四变换系数进行编码。
在一些实施例中，所述预测模式导出模式包括解码器侧帧内模式导出模式或基于模板的帧内模式导出模式。
在一些实施例中，所述编码单元640具体用于：
对第一标识、第二标识以及所述第四变换系数进行编码；
其中，所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，所述第二标识用于标识允许使用所述第四变换对所述当前序列中的图像块进行变换。
在一些实施例中，所述编码单元640具体用于：
对所述第一标识、所述第二标识、所述第四变换系数以及第三标识进行编码；
其中，所述第三标识用于标识允许所述当前序列中的图像块同时使用所述预测模式导出模式和所述第四变换。
在一些实施例中，所述变换单元630具体用于：
在所述当前块的高和/或宽大于或等于第一阈值的情况下，对所述第三变换系数进行所述第四变换，得到所述第四变换系数。
在一些实施例中，所述变换单元630对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数之前，还用于：
确定所述第四变换使用的变换矩阵组。
在一些实施例中，所述第四变换使用的变换矩阵组与平面模式或直流DC模式适配的变换矩阵组相同。
在一些实施例中，所述变换单元630具体用于：
基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式；
其中，所述第四变换使用的变换矩阵组与所述第三帧内预测模式适配的变换矩阵组相同。
在一些实施例中，所述变换单元630具体用于：
将所述第一帧内预测模式和所述第二帧内预测模式中的缺省预测模式，确定为所述第三帧内预测模式；或
基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式；或
基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式；或
基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式。
在一些实施例中，所述变换单元630具体用于：
将所述第一帧内预测模式和所述第二帧内预测模式中权重最大的帧内预测模式，确定为所述第三帧内预测模式。
在一些实施例中，所述变换单元630具体用于：
若所述第一帧内预测模式和所述第二帧内预测模式包括角度预测模式和非角度预测模式，则将所述角度预测模式，确定为所述第三帧内预测模式。
在一些实施例中，所述变换单元630具体用于：
若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值小于或等于第二阈值，则将第一预测角度对应的帧内预测模式，确定为所述第三帧内预测模式，其中，所述第一预测角度根据所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度确定：
若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值大于所述第二阈值，则将平面模式或直流DC模式确定为所述第三帧内预测模式。
在一些实施例中，所述变换单元630具体用于：
将所述预测模式导出模式适配的变换矩阵组，确定为所述第四变换使用的变换矩阵组。
在一些实施例中，所述第四变换用于处理所述当前块中沿倾斜方向上的纹理，所述第三变换用于处理所述当前块中沿水平方向上的纹理和沿竖直方向上的纹理。
应理解，装置实施例与方法实施例可以相互对应，类似的描述可以参照方法实施例。为避免重复，此处不再赘述。具体地，图18所示的解码器500可以对应于执行本申请实施例的方法300中的相应主体，并且解码器500中的各个单元的前述和其它操作和/或功能分别为了实现方法300等各个方法中的相应流程。类似的，图19所示的编码器600可以对应于执行本申请实施例的方法400中的相应主体，即编码器600中的各个单元的前述和其它操作和/或功能分别为了实现方法400等各个方法中的相应流程。
还应当理解，本申请实施例涉及的解码器500或编码器600中的各个单元可以分别或全部合并为一个或若干个另外的单元来构成，或者其中的某个(些)单元还可以再拆分为功能上更小的多个单元来构成，这可以实现同样的操作，而不影响本申请的实施例的技术效果的实现。上述单元是基于逻辑功能划分的，在实际应用中，一个单元的功能也可以由多个单元来实现，或者多个单元的功能由一个单元实现。在本申请的其它实施例中，该解码器500或编码器600也可以包括其它单元，在实际应用中，这些功能也可以由其它单元协助实现，并且可以由多个单元协作实现。根据本申请的另一个实施例，可以通过在包括例如中央处理单元(CPU)、随机存取存储介质(RAM)、只读存储介质(ROM)等处理元件和存储元件的通用计算机的通用计算设备上运行能够执行相应方法所涉及的各步骤的计算机程序(包括程序代码)，来构造本申请实施例涉及的解码器500或编码器600，以及来实现本申请实施例的编码方法或解码方法。计算机程序可以记载于例如计算机可读存储介质上，并通过计算机可读存储介质装载于电子设备中，并在其中运行，来实现本申请实施例的相应方法。
换言之，上文涉及的单元可以通过硬件形式实现，也可以通过软件形式的指令实现，还可以通过软硬件结合的形式实现。具体地，本申请实施例中的方法实施例的各步骤可以通过处理器中的硬件的集成逻辑电路和/或软件形式的指令完成，结合本申请实施例公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件组合执行完成。可选地，软件可以位于随机存储器，闪存、只读存储器、可编程只读存储器、电可擦写可编程存储器、寄存器等本领域的成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法实施例中的步骤。
图20是本申请实施例提供的电子设备700的示意结构图。
如图20所示，该电子设备700至少包括处理器710以及计算机可读存储介质720。其中，处理器710以及计算机可读存储介质720可通过总线或者其它方式连接。计算机可读存储介质720用于存储计算机程序721，计算机程序721包括计算机指令，处理器710用于执行计算机可读存储介质720存储的计算机指令。处理器710是电子设备700的计算核心以及控制核心，其适于实现一条或多条计算机指令，具体适于加载并执行一条或多条计算机指令从而实现相应方法流程或相应功能。
作为示例，处理器710也可称为中央处理器(Central Processing Unit，CPU)。处理器710可以包括但不限于：通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等等。
作为示例，计算机可读存储介质720可以是高速RAM存储器，也可以是非不稳定的存储器(Non-VolatileMemory)，例如至少一个磁盘存储器；可选的，还可以是至少一个位于远离前述处理器710的计算机可读存储介质。具体而言，计算机可读存储介质720包括但不限于：易失性存储器和/或非易失性存储器。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM， DDR SDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synch link DRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DR RAM)。
在一种实现方式中，该电子设备700可以是本申请实施例涉及的编码器或编码框架；该计算机可读存储介质720中存储有第一计算机指令；由处理器710加载并执行计算机可读存储介质720中存放的第一计算机指令，以实现本申请实施例提供的编码方法中的相应步骤；换言之，计算机可读存储介质720中的第一计算机指令由处理器710加载并执行相应步骤，为避免重复，此处不再赘述。
在一种实现方式中，该电子设备700可以是本申请实施例涉及的解码器或解码框架；该计算机可读存储介质720中存储有第二计算机指令；由处理器710加载并执行计算机可读存储介质720中存放的第二计算机指令，以实现本申请实施例提供的解码方法中的相应步骤；换言之，计算机可读存储介质720中的第二计算机指令由处理器710加载并执行相应步骤，为避免重复，此处不再赘述。
根据本申请的另一方面，本申请实施例还提供了一种编解码系统，包括上文涉及的编码器和解码器。
根据本申请的另一方面，本申请实施例还提供了一种计算机可读存储介质(Memory)，计算机可读存储介质是电子设备700中的记忆设备，用于存放程序和数据。例如，计算机可读存储介质720。可以理解的是，此处的计算机可读存储介质720既可以包括电子设备700中的内置存储介质，当然也可以包括电子设备700所支持的扩展存储介质。计算机可读存储介质提供存储空间，该存储空间存储了电子设备700的操作系统。并且，在该存储空间中还存放了适于被处理器710加载并执行的一条或多条的计算机指令，这些计算机指令可以是一个或多个的计算机程序721(包括程序代码)。
根据本申请的另一方面，提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。例如，计算机程序721。此时，数据处理设备700可以是计算机，处理器710从计算机可读存储介质720读取该计算机指令，处理器710执行该计算机指令，使得该计算机执行上述各种可选方式中提供的编码方法或解码方法。
换言之，当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。该计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行该计算机程序指令时，全部或部分地运行本申请实施例的流程或实现本申请实施例的功能。该计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。该计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质进行传输，例如，该计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(digital subscriber line，DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元以及流程步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
最后需要说明的是，以上内容，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (34)

  1. 一种解码方法，其特征在于，包括：
    解码码流获取当前块的第一变换系数；
    对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数；
    对所述第二变换系数进行第二变换，得到所述当前块的残差块；
    基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对所述当前块进行预测，得到所述当前块的预测块；
    基于所述当前块的预测块和所述当前块的残差块，得到所述当前块的重建块。
  2. 根据权利要求1所述的方法，其特征在于，所述预测模式导出模式包括解码器侧帧内模式导出模式或基于模板的帧内模式导出模式。
  3. 根据权利要求1或2所述的方法，其特征在于，所述对所述第二变换系数进行第二变换，得到所述当前块的残差块，包括：
    解码所述码流，获取第一标识和第二标识；
    若所述第一标识用于标识允许使用所述预测模式导出模式对当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则对所述第一变换系数进行第一变换，得到所述第二变换系数。
  4. 根据权利要求3所述的方法，其特征在于，所述对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数，包括：
    若所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则解码所述码流，获取第三标识；
    若所述第三标识用于标识允许所述当前序列中的图像块同时使用所述预测模式导出模式和所述第一变换，则对所述第一变换系数进行第一变换，得到所述第二变换系数。
  5. 根据权利要求3所述的方法，其特征在于，所述对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数，包括：
    若所述第一标识用于标识允许使用所述预测模式导出模式对所述当前序列中的图像块进行预测，且所述第二标识用于标识允许使用所述第一变换对所述当前序列中的图像块进行变换，则在所述当前块的高和/或宽大于或等于第一阈值的情况下，对所述第一变换系数进行第一变换，得到所述第二变换系数。
  6. 根据权利要求1至5中任一项所述的方法，其特征在于，所述对所述第二变换系数进行第二变换，得到所述当前块的残差块之前，所述方法还包括：
    确定所述第一变换使用的变换矩阵组。
  7. 根据权利要求6所述的方法，其特征在于，所述第一变换使用的变换矩阵组与平面模式或直流DC模式适配的变换矩阵组相同。
  8. 根据权利要求6所述的方法，其特征在于，所述确定所述第一变换使用的变换矩阵组，包括：
    基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式；
    其中，所述第一变换使用的变换矩阵组与所述第三帧内预测模式适配的变换矩阵组相同。
  9. 根据权利要求8所述的方法，其特征在于，所述基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式，包括：
    将所述第一帧内预测模式和所述第二帧内预测模式中的缺省预测模式，确定为所述第三帧内预测模式；或
    基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式；或
    基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式；或
    基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式。
  10. 根据权利要求9所述的方法，其特征在于，所述基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式，包括：
    将所述第一帧内预测模式和所述第二帧内预测模式中权重最大的帧内预测模式，确定为所述第三帧内预测模式。
  11. 根据权利要求9所述的方法，其特征在于，所述基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式，包括：
    若所述第一帧内预测模式和所述第二帧内预测模式包括角度预测模式和非角度预测模式，则将所述角度预测模式，确定为所述第三帧内预测模式。
  12. 根据权利要求9所述的方法，其特征在于，所述基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式，包括：
    若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值小于或等于第二阈值，则将第一预测角度对应的帧内预测模式，确定为所述第三帧内预测模式，其中，所述第一预测角度根据所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度确定：
    若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值大于所述第二阈值，则将平面模式或直流DC模式确定为所述第三帧内预测模式。
  13. 根据权利要求6所述的方法，其特征在于，所述确定所述第一变换使用的变换矩阵组，包括：
    将所述预测模式导出模式适配的变换矩阵组，确定为所述第一变换使用的变换矩阵组。
  14. 根据权利要求1至13中任一项所述的方法，其特征在于，所述第一变换用于处理所述当前块中沿倾斜方向上的纹理，所述第二变换用于处理所述当前块中沿水平方向上的纹理和沿竖直方向上的纹理。
  15. 一种编码方法，其特征在于，包括：
    基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对当前块进行预测，得到所述当前块的预测块；
    基于所述当前块的预测块，得到所述当前块的残差块；
    对所述当前块的残差块进行第三变换，得到所述当前块的第三变换系数；
    对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数；
    对所述第四变换系数进行编码。
  16. 根据权利要求15所述的方法，其特征在于，所述预测模式导出模式包括解码器侧帧内模式导出模式或基于模板的帧内模式导出模式。
  17. 根据权利要求15或16所述的方法，其特征在于，所述对所述第四变换系数进行编码，包括：
    对第一标识、第二标识以及所述第四变换系数进行编码；
    其中，所述第一标识用于标识允许使用所述预测模式导出模式对当前序列中的图像块进行预测，所述第二标识用于标识允许使用所述第四变换对所述当前序列中的图像块进行变换。
  18. 根据权利要求17所述的方法，其特征在于，所述对第一标识、第二标识以及所述第四变换系数进行编码，包括：
    对所述第一标识、所述第二标识、所述第四变换系数以及第三标识进行编码；
    其中，所述第三标识用于标识允许所述当前序列中的图像块同时使用所述预测模式导出模式和所述第四变换。
  19. 根据权利要求15至18中任一项所述的方法，其特征在于，所述对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数，包括：
    在所述当前块的高和/或宽大于或等于第一阈值的情况下，对所述第三变换系数进行所述第四变换，得到所述第四变换系数。
  20. 根据权利要求15至19中任一项所述的方法，其特征在于，所述对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数之前，所述方法还包括：
    确定所述第四变换使用的变换矩阵组。
  21. 根据权利要求20所述的方法，其特征在于，所述第四变换使用的变换矩阵组与平面模式或直流DC模式适配的变换矩阵组相同。
  22. 根据权利要求20所述的方法，其特征在于，所述确定所述第四变换使用的变换矩阵组，包括：
    基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式；
    其中，所述第四变换使用的变换矩阵组与所述第三帧内预测模式适配的变换矩阵组相同。
  23. 根据权利要求22所述的方法，其特征在于，所述基于所述第一帧内预测模式和所述第二帧内预测模式，确定第三帧内预测模式，包括：
    将所述第一帧内预测模式和所述第二帧内预测模式中的缺省预测模式，确定为所述第三帧内预测模式；或
    基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式；或
    基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式；或
    基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式。
  24. 根据权利要求23所述的方法，其特征在于，所述基于所述第一帧内预测模式的权重和/或所述第二帧内预测模式的权重确定所述第三帧内预测模式，包括：
    将所述第一帧内预测模式和所述第二帧内预测模式中权重最大的帧内预测模式，确定为所述第三帧内预测模式。
  25. 根据权利要求23所述的方法，其特征在于，所述基于所述第一帧内预测模式的类型和所述第二帧内预测模式的类型确定所述第三帧内预测模式，包括：
    若所述第一帧内预测模式和所述第二帧内预测模式包括角度预测模式和非角度预测模式，则将所述角度预测模式，确定为所述第三帧内预测模式。
  26. 根据权利要求23所述的方法，其特征在于，所述基于所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度，确定所述第三帧内预测模式，包括：
    若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值小于或等于第二阈值，则将第一预测角度对应的帧内预测模式，确定为所述第三帧内预测模式，其中，所述第一预测角度根据所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度确定：
    若所述第一帧内预测模式的预测角度和所述第二帧内预测模式的预测角度之间的差值的绝对值大于所述第二阈值，则将平面模式或直流DC模式确定为所述第三帧内预测模式。
  27. 根据权利要求20所述的方法，其特征在于，所述确定所述第四变换使用的变换矩阵组，包括：
    将所述预测模式导出模式适配的变换矩阵组，确定为所述第四变换使用的变换矩阵组。
  28. 根据权利要求15至27中任一项所述的方法，其特征在于，所述第四变换用于处理所述当前块中沿倾斜方向上的纹理，所述第三变换用于处理所述当前块中沿水平方向上的纹理和沿竖直方向上的纹理。
  29. 一种解码器，其特征在于，包括：
    解码单元，用于解码码流获取当前块的第一变换系数；
    变换单元，用于：
    对所述第一变换系数进行第一变换，得到所述当前块的第二变换系数；
    对所述第二变换系数进行第二变换，得到所述当前块的残差块；
    预测单元，用于基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对所述当前块进行预测，得到所述当前块的预测块；
    重建单元，用于基于所述当前块的预测块和所述当前块的残差块，得到所述当前块的重建块。
  30. 一种编码器，其特征在于，包括：
    预测单元，用于基于由预测模式导出模式导出的第一帧内预测模式和第二帧内预测模式对当前块进行预测，得到所 述当前块的预测块；
    残差单元，用于基于所述当前块的预测块，得到所述当前块的残差块；
    变换单元，用于：
    对所述当前块的残差块进行第三变换，得到所述当前块的第三变换系数；
    对所述第三变换系数进行第四变换，得到所述当前块的第四变换系数；
    编码单元，用于对所述第四变换系数进行编码。
  31. 一种电子设备，其特征在于，包括：
    处理器，适于执行计算机程序；
    计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序，所述计算机程序被所述处理器执行时，实现如权利要求1至14中任一项所述的方法或如权利要求15至28中任一项所述的方法。
  32. 一种计算机可读存储介质，其特征在于，用于存储计算机程序，所述计算机程序使得计算机执行如权利要求1至14中任一项所述的方法或如权利要求15至28中任一项所述的方法。
  33. 一种计算机程序产品，包括计算机程序/指令，其特征在于，所述计算机程序/指令被处理器执行时实现如权利要求1至14中任一项所述的方法或如权利要求15至28中任一项所述的方法。
  34. 一种码流，其特征在于，所述码流如权利要求1至14中任一项所述的方法中所述的码流或如权利要求15至28中任一项所述的方法生成的码流。
PCT/CN2022/086448 2022-04-12 2022-04-12 解码方法、编码方法、解码器以及编码器 Ceased WO2023197181A1 (zh)

Priority Applications (6)

Application Number Priority Date Filing Date Title
EP22936835.2A EP4510588A4 (en) 2022-04-12 2022-04-12 DECODING METHOD, CODING METHOD, DECODERS AND CODERS
JP2024560810A JP2025512523A (ja) 2022-04-12 2022-04-12 復号化方法、符号化方法、デコーダ、エンコーダ
PCT/CN2022/086448 WO2023197181A1 (zh) 2022-04-12 2022-04-12 解码方法、编码方法、解码器以及编码器
CN202280094735.8A CN119013989A (zh) 2022-04-12 2022-04-12 解码方法、编码方法、解码器以及编码器
US18/913,625 US20250039454A1 (en) 2022-04-12 2024-10-11 Decoding method, encoding method, decoder and encoder
MX2024012724A MX2024012724A (es) 2022-04-12 2024-10-14 Metodo de decodificacion, metodo de codificacion, decodificador y codificador

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/086448 WO2023197181A1 (zh) 2022-04-12 2022-04-12 解码方法、编码方法、解码器以及编码器

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/913,625 Continuation US20250039454A1 (en) 2022-04-12 2024-10-11 Decoding method, encoding method, decoder and encoder

Publications (1)

Publication Number Publication Date
WO2023197181A1 true WO2023197181A1 (zh) 2023-10-19

Family

ID=88328719

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/086448 Ceased WO2023197181A1 (zh) 2022-04-12 2022-04-12 解码方法、编码方法、解码器以及编码器

Country Status (6)

Country Link
US (1) US20250039454A1 (zh)
EP (1) EP4510588A4 (zh)
JP (1) JP2025512523A (zh)
CN (1) CN119013989A (zh)
MX (1) MX2024012724A (zh)
WO (1) WO2023197181A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025140624A1 (en) * 2023-12-29 2025-07-03 Douyin Vision Co., Ltd. Method, apparatus, and medium for video processing

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200059669A1 (en) * 2017-04-25 2020-02-20 Panasonic Intellectual Property Corporation Of America Encoder, decoder, encoding method, and decoding method
CN110881127A (zh) * 2018-09-06 2020-03-13 腾讯美国有限责任公司 控制残差编码的方法、装置、计算机设备和存储介质
CN111869218A (zh) * 2018-03-07 2020-10-30 腾讯美国有限责任公司 视频编解码的方法和装置
CN113728640A (zh) * 2019-04-23 2021-11-30 北京字节跳动网络技术有限公司 帧内预测和残差编解码
CN114073081A (zh) * 2019-06-25 2022-02-18 弗劳恩霍夫应用研究促进协会 使用基于矩阵的帧内预测和二次变换进行编码

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4030762A4 (en) * 2019-09-10 2023-09-06 Samsung Electronics Co., Ltd. IMAGE DECODING DEVICE USING A SET OF TOOLS AND CORRESPONDING IMAGE DECODING METHOD, AND IMAGE ENCODING DEVICE AND CORRESPONDING IMAGE ENCODING METHOD
US12615365B2 (en) * 2021-04-12 2026-04-28 Qualcomm Incorporated Intra-mode dependent multiple transform selection for video coding
US11943432B2 (en) * 2021-04-26 2024-03-26 Tencent America LLC Decoder side intra mode derivation
US12375663B2 (en) * 2021-06-27 2025-07-29 Alibaba Innovation Private Limited Methods and systems for performing combined inter and intra prediction
US12506881B2 (en) * 2021-06-28 2025-12-23 Wilus Institute Of Standards And Technology Inc. Video signal processing method using intra prediction and device therefor
WO2023274372A1 (en) * 2021-07-01 2023-01-05 Beijing Bytedance Network Technology Co., Ltd. Method, device, and medium for video processing
US12160561B2 (en) * 2021-08-02 2024-12-03 Tencent America LLC Method and apparatus for improved intra prediction
CN117941350A (zh) * 2021-09-13 2024-04-26 鸿颖创新有限公司 用于视频编码中的帧内预测的设备和方法
US20230090025A1 (en) * 2021-09-22 2023-03-23 Alibaba Singapore Holding Private Limited Methods and systems for performing combined inter and intra prediction
CN118044182A (zh) * 2021-09-29 2024-05-14 抖音视界有限公司 用于视频处理的方法、设备和介质
US12531985B2 (en) * 2021-10-01 2026-01-20 Lg Electronics Inc. CIIP-based prediction method and device
WO2023059056A1 (ko) * 2021-10-05 2023-04-13 엘지전자 주식회사 비분리 1차 변환에 기반한 영상 부호화/복호화 방법, 장치 및 비트스트림을 저장하는 기록 매체
US12010316B2 (en) * 2021-10-05 2024-06-11 Tencent America LLC Modification on fusion of intra prediction
CN118525515A (zh) * 2021-10-28 2024-08-20 抖音视界有限公司 视频处理方法、设备和介质
US12126822B2 (en) * 2021-12-07 2024-10-22 Tencent America LLC Most probable mode (mpm) list sorting
WO2023128615A1 (ko) * 2021-12-29 2023-07-06 엘지전자 주식회사 영상 인코딩/디코딩 방법 및 장치, 그리고 비트스트림을 저장한 기록 매체
US12355982B2 (en) * 2022-01-04 2025-07-08 Alibaba (China) Co., Ltd Decoder-side chroma intra prediction mode gradient-based derivation
WO2023164245A1 (en) * 2022-02-28 2023-08-31 Beijing Dajia Internet Information Technology Co., Ltd. Methods and devices for decoder-side intra mode derivation

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200059669A1 (en) * 2017-04-25 2020-02-20 Panasonic Intellectual Property Corporation Of America Encoder, decoder, encoding method, and decoding method
CN111869218A (zh) * 2018-03-07 2020-10-30 腾讯美国有限责任公司 视频编解码的方法和装置
CN110881127A (zh) * 2018-09-06 2020-03-13 腾讯美国有限责任公司 控制残差编码的方法、装置、计算机设备和存储介质
CN113728640A (zh) * 2019-04-23 2021-11-30 北京字节跳动网络技术有限公司 帧内预测和残差编解码
CN114073081A (zh) * 2019-06-25 2022-02-18 弗劳恩霍夫应用研究促进协会 使用基于矩阵的帧内预测和二次变换进行编码

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4510588A4 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2025140624A1 (en) * 2023-12-29 2025-07-03 Douyin Vision Co., Ltd. Method, apparatus, and medium for video processing

Also Published As

Publication number Publication date
CN119013989A (zh) 2024-11-22
EP4510588A4 (en) 2026-02-18
EP4510588A1 (en) 2025-02-19
US20250039454A1 (en) 2025-01-30
JP2025512523A (ja) 2025-04-17
MX2024012724A (es) 2024-11-08

Similar Documents

Publication Publication Date Title
TWI834773B (zh) 使用適應性迴路濾波器以編碼和解碼影像之一或多個影像部分的方法、裝置和電腦可讀儲存媒體
EP3149941B1 (en) Method, apparatus and system for de-blocking a block of video samples
KR20180083389A (ko) 비디오 코딩 방법 및 장치
WO2024007116A1 (zh) 解码方法、编码方法、解码器以及编码器
WO2023193253A1 (zh) 解码方法、编码方法、解码器以及编码器
CN118592025A (zh) 解码方法、编码方法、解码器、编码器和编解码系统
WO2023123398A1 (zh) 滤波方法、滤波装置以及电子设备
US20250039452A1 (en) Decoding method, encoding method, decoder, and encoder
WO2023193254A1 (zh) 解码方法、编码方法、解码器以及编码器
US20250039454A1 (en) Decoding method, encoding method, decoder and encoder
CN118160306A (zh) 帧内预测方法、解码器、编码器及编解码系统
WO2023197180A1 (zh) 解码方法、编码方法、解码器以及编码器
RU2859876C2 (ru) Способ декодирования, способ кодирования и энергонезависимый машиночитаемый носитель данных
WO2025236218A1 (zh) 编解码方法、码流、编码器、解码器以及存储介质
WO2025166744A1 (zh) 编解码方法、编解码器、码流以及存储介质
WO2025091378A1 (zh) 编解码方法、编解码器以及存储介质
WO2025199802A1 (zh) 编解码方法、码流、编码器、解码器以及存储介质
WO2024212086A1 (zh) 解码方法、编码方法、解码器以及编码器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22936835

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202280094735.8

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: MX/A/2024/012724

Country of ref document: MX

WWE Wipo information: entry into national phase

Ref document number: 2024560810

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202417086796

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2022936835

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022936835

Country of ref document: EP

Effective date: 20241112