WO2023197717A1 - 一种图像解码方法、编码方法及装置 - Google Patents

一种图像解码方法、编码方法及装置 Download PDF

Info

Publication number
WO2023197717A1
WO2023197717A1 PCT/CN2023/071928 CN2023071928W WO2023197717A1 WO 2023197717 A1 WO2023197717 A1 WO 2023197717A1 CN 2023071928 W CN2023071928 W CN 2023071928W WO 2023197717 A1 WO2023197717 A1 WO 2023197717A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature map
feature
image
optical flow
map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/071928
Other languages
English (en)
French (fr)
Inventor
葛运英
王晶
师一博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP23787366.6A priority Critical patent/EP4498678A4/en
Publication of WO2023197717A1 publication Critical patent/WO2023197717A1/zh
Priority to US18/914,881 priority patent/US20250037317A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/18—Image warping, e.g. rearranging pixels individually
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/771—Feature selection, e.g. selecting representative features from a multi-dimensional feature space
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/577—Motion compensation with bidirectional frame interpolation, i.e. using B-pictures

Definitions

  • the present application relates to the field of video processing technology, and in particular, to an image decoding method, encoding method and device.
  • Video compression systems perform spatial (intra-image) prediction and/or temporal (inter-image) prediction to reduce or remove redundant information inherent in video sequences; video enhancement techniques are used to improve the display quality of images.
  • the decoder uses the optical flow mapping (warping) method to decode the image frames contained in the video.
  • Optical flow mapping means that the decoding end obtains the image domain optical flow between the image frame and the reference frame, and decodes the image frame based on this optical flow.
  • Image domain optical flow is used to indicate the motion speed and motion direction between corresponding pixels in two adjacent frames of images.
  • optical flow mapping is more sensitive to optical flow accuracy, and subtle changes in optical flow accuracy will affect the accuracy of warping. Since the error in optical flow prediction between two adjacent frames is large, such as an error of 5 pixels or more, the accuracy of decoding the image frame based on the image domain optical flow at the decoder is low, resulting in decoding The clarity of the images obtained is affected. Therefore, how to provide a more effective image decoding method has become an urgent problem that needs to be solved.
  • the present application provides an image decoding method, encoding method and device, which solves the problem that the accuracy of decoding image frames based on image domain optical flow is low, causing the clarity of the decoded image to be affected.
  • this application provides an image decoding method, which method is applied to a video coding and decoding system, or a physical device that supports the implementation of the video coding and decoding system to implement the image decoding method.
  • the physical device is a decoding end or a video decoder. (decoder), in some cases, the physical device may include a chip system.
  • the image decoding method performed by the decoding end is used as an example.
  • the image decoding method includes: first, the decoding end parses the code stream to obtain at least one optical flow set; the at least one optical flow set includes a first optical flow set, and the first optical flow set
  • the flow set corresponds to the first feature map of the reference frame of the first image frame, and the first optical flow set includes one or more feature domain optical flows, any one of the one or more feature domain optical flows.
  • the stream is used to indicate motion information between the feature map of the first image frame and the aforementioned first feature map.
  • the decoding end processes the first feature map based on one or more feature domain optical flows included in the first optical flow set, and obtains one or more intermediate feature maps corresponding to the first feature map; and, the decoding end fuses the aforementioned one or more A plurality of intermediate feature maps are used to obtain a first predicted feature map of the first image frame. Finally, the decoder decodes the first image frame according to the first prediction feature map to obtain the first image.
  • the error of the pixels in the feature domain optical flow is smaller than the error of the pixels in the image domain optical flow. Therefore, the intermediate feature map determined by the decoding end based on the feature domain optical flow brings The resulting decoding error is lower than the decoding error caused by image domain optical flow in common technology.
  • the decoding end decodes the image frame based on the feature domain optical flow, which reduces the decoding error caused by the image domain optical flow between two adjacent frames. Decoding errors improve the accuracy of image decoding.
  • the decoder fuses multiple intermediate feature maps determined by the optical flow set and the feature map of the reference frame to obtain a predicted feature map of the image frame.
  • This predicted feature map contains more image information than the optical flow of a single image domain.
  • the decoder decodes the image frame based on the prediction feature map obtained by fusion, it avoids the difficulty of accurately representing the information indicated by the optical flow in a single image domain.
  • the problem of reaching the first image is improved, and the accuracy of image decoding and image quality (such as image clarity, etc.) are improved.
  • the decoder processes the first feature map based on one or more feature domain optical flows included in the first optical flow set, and obtains one or more intermediate feature maps corresponding to the first feature map, including :
  • the decoding end parses the code stream to obtain the first feature map of the aforementioned reference frame; and, for one or more feature domain optical flows included in the first optical flow set, the decoding end analyzes the first feature domain optical flow based on the first feature domain optical flow.
  • the feature map is subjected to optical flow mapping (warping) to obtain an intermediate feature map corresponding to the first feature domain optical flow; the first feature domain optical flow is any one of the aforementioned one or more feature domain optical flows.
  • the decoder performs warping on the first feature map based on a set of feature domain optical flows. Then, after the decoder obtains the image corresponding to the reference frame, the decoder can warp the first feature map in the image corresponding to the reference frame according to a set of The feature domain optical flow interpolates the corresponding positions of the set of feature domain optical flows in the image corresponding to the reference frame to obtain the predicted value of the pixel value in the first image, thereby obtaining the first image, avoiding the need for the decoder to rely on the image domain optical flow. Predicting all pixel values of the first image reduces the amount of calculation required by the decoder for image decoding and improves the efficiency of image decoding.
  • the aforementioned first optical flow set also corresponds to the second feature map of the reference frame.
  • the image decoding method before the decoding end decodes the first image frame according to the first prediction feature map to obtain the first image, the image decoding method further includes: a first step, the decoding end includes based on the first optical flow set: One or more feature domain optical flows process the second feature map to obtain one or more intermediate feature maps corresponding to the second feature map.
  • the decoder fuses one or more intermediate feature maps corresponding to the second feature map to obtain the second predicted feature map of the first image frame. In this way, the decoder can decode the first image frame according to the first prediction feature map and the second prediction feature map to obtain the first image.
  • multiple feature maps (or a group of feature maps) of the reference frame can correspond to an optical flow set (or a set of feature domain optical flows).
  • the decoding end processes the feature map according to a set of feature domain optical flows corresponding to the feature map, thereby obtaining an intermediate feature map corresponding to the feature map.
  • the decoder end processes the intermediate feature map corresponding to the feature map. Fusion is performed to obtain the predicted feature map corresponding to the feature map.
  • the decoder decodes the image frame according to the prediction feature maps corresponding to all feature maps of the reference frame to obtain the target image.
  • the decoder can divide the multiple feature maps into one group or multiple groups.
  • the feature maps belonging to the same group share an optical flow set, and
  • the intermediate feature map corresponding to the feature map is fused to obtain the predicted feature map, which avoids the problem that the decoder reconstructs the image based on the feature map with low accuracy and slow speed when the reference frame or image frame has more information, and improves the image quality. Decoding accuracy. It is worth noting that the number of channels of feature maps belonging to different groups can be different.
  • the decoder fuses one or more intermediate feature maps to obtain the first predicted feature map of the first image frame, including: the decoder obtains one or more of the one or more intermediate feature maps. Multiple weights, where one intermediate feature map corresponds to one weight; and, the decoder processes the intermediate feature maps corresponding to the one or more weights based on the one or more intermediate feature map weights, and all processes The subsequent intermediate feature maps are added to obtain the first predicted feature map.
  • the weight is used to indicate the weight of the intermediate feature map in the first predicted feature map.
  • the weight of each intermediate feature map can be different.
  • the decoder can set different weights for the intermediate feature maps according to the requirements of image decoding. , For example, if the images corresponding to some intermediate feature maps are relatively blurry, the weights of these intermediate feature maps are reduced, thereby improving the clarity of the first image.
  • the decoder fuses one or more intermediate feature maps to obtain the first predicted feature map of the first image frame, including: converting one or more intermediate feature maps corresponding to the first feature map Input the feature fusion model to obtain the first predicted feature map.
  • the feature fusion model includes a convolutional network layer, which is used to fuse intermediate feature maps.
  • the decoder obtains multiple image domain optical flows between image frames and reference frames, and obtains multiple images based on the multiple image domain optical flows and reference frames, thereby fusing multiple images to obtain image frames.
  • the decoding end needs to predict the pixel values of multiple images when decoding one frame of image, and fuse multiple images to obtain the target image, resulting in large computing resources required for image decoding. Streaming decoded video is less efficient.
  • the decoder uses a feature fusion model to fuse multiple intermediate feature maps, and then , the decoding end decodes the image frame based on the prediction feature map obtained by fusing multiple intermediate feature maps, that is, the decoding end only needs to predict the pixel value of the image position indicated by the prediction feature map in the image based on the prediction feature map, without prediction. All pixel values of multiple images reduce the computing resources required for image decoding and improve the efficiency of image decoding.
  • the image decoding method provided by this application also includes: first, the decoding end obtains the feature map of the first image. Secondly, the decoding end obtains the enhanced feature map based on the feature map of the first image, the first feature map and the first predicted feature map. Finally, the decoder processes the first image according to the enhanced feature map to obtain the second image.
  • the second image has a higher definition than the first image.
  • the decoder can fuse the feature map of the first image, the first feature map and the first predicted feature map, and perform video enhancement processing on the first image based on the enhanced feature map obtained through the fusion to obtain better clarity. Excellent second image, thereby improving the decoded image clarity and image display effect.
  • the decoder processes the first image according to the enhanced feature map to obtain the second image, including: the decoder obtains the enhancement layer image of the first image based on the enhanced feature map, and reconstructs the image based on the enhancement layer image. Construct the first image and obtain the second image.
  • the enhancement layer image may refer to an image determined by the decoder based on the reference frame and the enhancement feature map.
  • the decoder adds part or all of the information of the enhancement layer image to the first image to obtain the second image; or , the decoder uses the enhancement layer image as the reconstructed image of the first image, that is, the aforementioned second image.
  • the decoder obtains multiple feature maps of the image at different stages, and obtains the enhanced feature map determined by these multiple feature maps, so as to reconstruct and enhance the first image based on the enhanced feature map. Improved image clarity and image display after decoding.
  • the second aspect provides an image coding method, which method is applied to a video coding and decoding system, or a physical device that supports the implementation of the video coding and decoding system to implement the image coding method.
  • the physical device is an encoding end or a video encoder (encoder). ), in some cases, the physical device may include a system-on-a-chip.
  • an image coding method performed by the encoding end is used as an example.
  • the image encoding method includes: in the first step, the encoding end obtains the feature map of the first image frame and the first feature map of the reference frame of the first image frame.
  • the encoding end obtains at least one optical flow set based on the feature map and the first feature map of the first image frame; the at least one optical flow set includes a first optical flow set, and the first optical flow set corresponds to the aforementioned first optical flow set.
  • feature map, and the first optical flow set includes one or more feature domain optical flows, any one of the aforementioned one or more feature domain optical flows is used to indicate the feature map and the first feature of the first image frame Motion information between graphs.
  • the encoding end processes the first feature map based on one or more feature domain optical flows included in the first optical flow set, and obtains one or more intermediate feature maps corresponding to the first feature map.
  • the encoding end fuses the aforementioned one or more intermediate feature maps to obtain the first predicted feature map of the first image frame.
  • the encoding end encodes the first image frame according to the first prediction feature map to obtain a code stream.
  • the code stream includes: a feature domain optical flow code stream corresponding to at least one optical flow set, and a residual code stream of the image area corresponding to the first prediction feature map.
  • the error of pixels in the feature domain optical flow is smaller than that of the image domain optical flow.
  • the error of pixels in the flow Therefore, the encoding error caused by the intermediate feature map determined by the encoding end based on the feature domain optical flow is lower than the encoding error caused by the image domain optical flow in the common technology.
  • the encoding end determines based on the feature domain optical flow.
  • Encoding image frames reduces coding errors caused by image domain optical flow between two adjacent frames and improves the accuracy of image coding.
  • the encoding end processes the feature maps of the reference frame based on multiple feature domain optical flows to obtain multiple intermediate feature maps, and fuses the multiple intermediate feature maps to determine the predicted feature map of the image frame.
  • the encoding end combines the optical flow set and the reference frame.
  • the multiple intermediate feature maps determined by the feature map are fused to obtain the prediction feature map of the image frame.
  • the prediction feature map contains more image information, so that when the encoding end encodes the image frame based on the prediction feature map obtained by the fusion, This avoids the problem that a single intermediate feature map is difficult to accurately express the first image, and improves the accuracy of image coding and image quality (such as image clarity, etc.).
  • the encoding end processes the first feature map based on one or more feature domain optical flows included in the first optical flow set, and obtains one or more intermediate feature maps corresponding to the first feature map, including : For one or more feature domain optical flows included in the first optical flow set, optical flow mapping is performed on the first feature map based on the first feature domain optical flow, and an intermediate feature map corresponding to the first feature domain optical flow is obtained. .
  • the first feature map is any one of one or more feature domain optical flows included in the first optical flow set.
  • the encoding end uses a feature fusion model to fuse multiple intermediate feature maps, and then , the encoding end encodes the image frame based on the prediction feature map obtained by fusing multiple intermediate feature maps, that is, the encoding end only needs to predict the pixel value of the image position indicated by the prediction feature map in the image based on the prediction feature map, without prediction. All pixel values of multiple images reduce the computing resources required for image encoding and improve the efficiency of image encoding.
  • the first optical flow set also corresponds to the second feature map of the reference frame.
  • the image encoding method provided in this embodiment further includes: a first step, the encoding end bases one or more of the first optical flow set on The feature domain optical flow processes the second feature map to obtain one or more intermediate feature maps corresponding to the second feature map.
  • the encoding end fuses one or more intermediate feature maps corresponding to the second feature map to obtain the second predicted feature map of the first image frame.
  • the aforementioned coding end encoding the first image frame according to the first prediction feature map to obtain a code stream may include: the encoding end encoding the first image frame according to the first prediction feature map and the second prediction feature map to obtain a code stream.
  • multiple feature maps (or a group of feature maps) of the reference frame can correspond to an optical flow set (or a set of feature domain optical flows).
  • the encoding end processes the feature map according to a set of feature domain optical flows corresponding to the feature map, thereby obtaining an intermediate feature map corresponding to the feature map.
  • the encoding end converts the intermediate feature map corresponding to the feature map Fusion is performed to obtain the predicted feature map corresponding to the feature map.
  • the encoding end encodes the image frame according to the prediction feature maps corresponding to all feature maps of the reference frame to obtain the target code stream. In this way, during the image encoding process, if the reference frame corresponds to multiple feature maps, the encoding end can divide the multiple feature maps into one group or multiple groups.
  • the feature maps belonging to the same group share an optical flow set, and
  • the intermediate feature map corresponding to the feature map is fused to obtain the predicted feature map, which avoids the problem that when the reference frame or image frame has more information, the encoding end obtains a code stream based on the feature map encoding with more redundancy and lower accuracy, and improves improve the accuracy of image coding. It is worth noting that the number of channels of feature maps belonging to different groups can be different.
  • the encoding end fuses one or more intermediate feature maps to obtain the first predicted feature map of the first image frame, including: the encoding end obtains one or more of the one or more intermediate feature maps. Multiple weights, wherein one intermediate feature map corresponds to one weight; and, the encoding end processes the intermediate feature maps corresponding to the one or more weights based on the aforementioned one or more weights, and converts all processed The intermediate feature maps are added to obtain the first predicted feature map.
  • the weight is used to indicate the weight of the intermediate feature map in the first predicted feature map. In this embodiment, for the first For multiple intermediate feature maps corresponding to the feature map, the weight of each intermediate feature map can be different.
  • the encoding end can set different weights for the intermediate feature map according to the requirements of image encoding. For example, if some intermediate features When the image corresponding to the image is relatively blurry, the weights of these intermediate feature maps are reduced to improve the clarity of the first image.
  • the encoding end fuses one or more intermediate feature maps to obtain the first predicted feature map of the first image frame, including: the encoding end fuses one or more intermediate feature maps corresponding to the first feature map.
  • the feature map is input into the feature fusion model to obtain the first predicted feature map.
  • the feature fusion model includes a convolutional network layer, which is used to fuse intermediate feature maps.
  • the encoding end uses a feature fusion model to fuse multiple intermediate feature maps.
  • the encoding end uses a feature fusion model to fuse multiple intermediate feature maps.
  • the predicted feature map obtained from the intermediate feature map encodes the image frame, reducing the computing resources required for image encoding and improving the efficiency of image encoding.
  • an image decoding device which may be applied to a decoding end or support a video coding and decoding system that implements the foregoing image decoding method.
  • the image decoding device includes various modules for executing the image decoding method in the first aspect or any possible implementation of the first aspect.
  • the image decoding device includes: a code stream unit, a processing unit, a fusion unit and a decoding unit.
  • the code stream unit is used to parse the code stream to obtain at least one optical flow set.
  • the at least one optical flow set includes a first optical flow set corresponding to the first feature map of the reference frame of the first image frame.
  • the first optical flow set includes one or more feature domain optical flows, as described above. Any one of the one or more feature domain optical flows is used to indicate motion information between the feature map of the first image frame and the first feature map.
  • a processing unit configured to process the first feature map based on one or more feature domain optical flows included in the first optical flow set, and obtain one or more intermediate feature maps corresponding to the first feature map.
  • the fusion unit is used to fuse one or more intermediate feature maps to obtain the first predicted feature map of the first image frame.
  • a decoding unit configured to decode the first image frame according to the first prediction feature map to obtain the first image.
  • the image decoding device has the function of implementing the behaviors in the method examples of any of the above first aspects.
  • the functions described can be implemented by hardware, or can be implemented by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the processing unit is specifically configured to: parse the code stream to obtain the first feature map of the reference frame, and perform optical flow mapping (warping) on the first feature map based on the first feature domain optical flow. , obtain an intermediate feature map corresponding to the first feature domain optical flow; the first feature domain optical flow is any one of one or more feature domain optical flows included in the aforementioned first optical flow set.
  • the first optical flow set also corresponds to the second feature map of the reference frame.
  • the processing unit is also configured to process the second feature map based on one or more feature domain optical flows included in the first optical flow set, and obtain one or more intermediate feature maps corresponding to the second feature map.
  • the fusion unit is also used to fuse one or more intermediate feature maps corresponding to the second feature map to obtain the second predicted feature map of the first image frame.
  • the decoding unit is specifically configured to decode the first image frame according to the first prediction feature map and the second prediction feature map to obtain the first image.
  • the fusion unit is specifically used to: obtain one or more weights of the aforementioned one or more intermediate feature maps, where one intermediate feature map corresponds to a weight, and the weight is expressed as Indicates the weight of the intermediate feature map in the first predicted feature map. and, based on one or more weights, process intermediate feature maps respectively corresponding to the aforementioned one or more weights, and add all processed intermediate feature maps to obtain a first prediction feature map.
  • the fusion unit is specifically configured to input one or more intermediate feature maps corresponding to the first feature map into the feature fusion model to obtain the first predicted feature map.
  • the feature fusion model includes a convolutional network layer, which is used to fuse intermediate feature maps.
  • the image decoding device further includes: an acquisition unit and an enhancement unit.
  • An acquisition unit is used to acquire the feature map of the first image.
  • the fusion unit is also configured to obtain an enhanced feature map based on the feature map of the first image, the first feature map and the first predicted feature map.
  • the enhancement unit is used to process the first image according to the enhanced feature map to obtain the second image.
  • the second image has a higher definition than the first image.
  • the enhancement unit is specifically configured to: obtain the enhancement layer image of the first image based on the enhancement feature map. And, reconstruct the first image based on the enhancement layer image to obtain the second image.
  • the fourth aspect provides an image encoding device, which can be applied to the encoding end, or supports a video encoding and decoding system that implements the aforementioned image encoding method.
  • the image encoding device includes various modules for executing the image encoding method in the second aspect or any possible implementation of the second aspect.
  • the image coding device includes: an acquisition unit, a processing unit, a fusion unit and a coding unit.
  • the obtaining unit is configured to obtain the feature map of the first image frame and the first feature map of the reference frame of the first image frame.
  • a processing unit configured to obtain at least one optical flow set according to the feature map of the first image frame and the first feature map.
  • the at least one optical flow set includes a first optical flow set corresponding to the first feature map of the reference frame of the first image frame.
  • the first optical flow set includes one or more feature domain optical flows, as described above. Any one of the one or more feature domain optical flows is used to indicate motion information between the feature map of the first image frame and the first feature map.
  • the processing unit is also configured to process the first feature map based on one or more feature domain optical flows included in the first optical flow set, and obtain one or more intermediate feature maps corresponding to the first feature map.
  • the fusion unit is used to fuse one or more intermediate feature maps to obtain the first predicted feature map of the first image frame.
  • An encoding unit configured to encode the first image frame according to the first prediction feature map to obtain a code stream.
  • the code stream includes: a feature domain optical flow code stream corresponding to at least one optical flow set, and a residual code stream of the image area corresponding to the first prediction feature map.
  • the image encoding device has the function of implementing the behaviors in the method examples of any of the above-mentioned first aspects.
  • the functions described can be implemented by hardware, or can be implemented by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the processing unit is specifically configured to: and, for one or more feature domain optical flows included in the first optical flow set, calculate the first feature based on the first feature domain optical flow therein.
  • Optical flow mapping (warping) is performed on the graph to obtain an intermediate feature map corresponding to the first feature domain optical flow; the first feature domain optical flow is any one of one or more feature domain optical flows included in the first optical flow set.
  • the first optical flow set also corresponds to the second feature map of the reference frame.
  • the processing unit is also configured to process the second feature map based on one or more feature domain optical flows included in the first optical flow set, and obtain one or more intermediate feature maps corresponding to the second feature map.
  • the fusion unit is also used to fuse one or more intermediate feature maps corresponding to the second feature map to obtain the second predicted feature map of the first image frame.
  • the encoding unit is specifically configured to encode the first image frame according to the first prediction feature map and the second prediction feature map to obtain a code stream.
  • the fusion unit is specifically configured to: obtain one or more weights of the aforementioned one or more intermediate feature maps, where one intermediate feature map corresponds to a weight, and the weight is expressed as to indicate that the intermediate feature map is in The weight of the first predicted feature map. and, based on the one or more weights, process the intermediate feature maps respectively corresponding to the one or more weights, and add all the processed intermediate feature maps to obtain the first prediction feature map.
  • the fusion unit is specifically configured to input one or more intermediate feature maps corresponding to the first feature map into the feature fusion model to obtain the first predicted feature map.
  • the feature fusion model includes a convolutional network layer, which is used to fuse intermediate feature maps.
  • an image decoding device in a fifth aspect, includes at least one processor and a memory.
  • the memory is used to store program code.
  • the processor calls the program code, the first aspect or The operation steps of the image decoding method in any possible implementation of the first aspect, for example, the image decoding device may be a decoding end or a video decoder included in the video encoding and decoding system.
  • the image decoding device may be a coding end or a video encoder included in the video coding and decoding system.
  • a computer-readable storage medium In a sixth aspect, a computer-readable storage medium is provided. Computer programs or instructions are stored in the storage medium. When the computer program or instructions are executed by an electronic device, it is possible to implement the first aspect or any one of the first aspects.
  • the operation steps of the image decoding method in the manner, and/or, the operation steps of the image encoding method in the second aspect or any possible implementation manner of the second aspect are performed.
  • the electronic device refers to the aforementioned image decoding device.
  • another computer-readable storage medium which stores a code stream obtained according to the image encoding method in the second aspect or any of the possible implementations of the second aspect.
  • the code stream may include: a feature domain optical flow code stream corresponding to at least one optical flow set in the second aspect, and a residual code stream of the image area corresponding to the first prediction feature map.
  • An eighth aspect provides a video encoding and decoding system, including an encoding end and a decoding end.
  • the decoding end can perform the operation steps of the image decoding method in the first aspect or any possible implementation of the first aspect.
  • the encoding end can Execute the operational steps of the image encoding method in the second aspect or any possible implementation manner of the second aspect.
  • beneficial effects please refer to the description of any one of the first aspects, or the description of any one of the second aspects, and will not be described again here.
  • a computer program product When the computer program product is run on an electronic device, it causes the electronic device to perform the operation steps of the method described in the first aspect or any possible implementation of the first aspect, And/or, when the computer program product is run on the computer, the electronic device is caused to perform the operation steps of the method described in the first aspect or any possible implementation manner of the first aspect.
  • the electronic device refers to the aforementioned image decoding device.
  • a chip including a control circuit and an interface circuit.
  • the interface circuit is used to receive signals from other devices outside the chip and transmit them to the processor, or to send signals from the control circuit to devices outside the chip.
  • Other devices, control circuits, logic circuits or execution code instructions are used to implement: the above-mentioned first aspect or the operational steps of the method in any possible implementation of the first aspect, and/or the above-mentioned second aspect or the second aspect. Operational steps of the method in any possible implementation.
  • the chip refers to the aforementioned image decoding device.
  • Figure 1 is an exemplary block diagram of the video encoding and decoding system provided by this application.
  • Figure 2 is a mapping relationship diagram between optical flow and color provided by this application.
  • FIG. 3 is a schematic diagram of optical flow mapping provided by this application.
  • FIG. 4 is a schematic diagram of the architecture of the video compression system provided by this application.
  • Figure 5 is a schematic flow chart of the image encoding method provided by this application.
  • Figure 6 is a schematic structural diagram of the feature extraction network provided by this application.
  • Figure 7 is a schematic structural diagram of the optical flow estimation network provided by this application.
  • Figure 8 is a schematic flow chart of optical flow mapping and feature fusion provided by this application.
  • Figure 9 is a schematic flow chart 1 of the image decoding method provided by this application.
  • Figure 10 is a schematic structural diagram of the feature encoding network and feature decoding network provided by this application.
  • Figure 11 is a schematic flow chart 2 of the image decoding method provided by this application.
  • Figure 12 is a schematic structural diagram of the image reconstruction network provided by this application.
  • Figure 13 is a schematic diagram of the variable convolution network provided by this application.
  • FIG 14 is a schematic diagram of the efficiency comparison provided by this application.
  • Figure 15 is a schematic structural diagram of the image encoding device provided by this application.
  • Figure 16 is a schematic structural diagram of the image decoding device provided by this application.
  • Figure 17 is a schematic structural diagram of an image decoding device provided by this application.
  • Embodiments of the present application provide an image decoding (encoding) method, including: the decoding end (or encoding end) obtains a set of feature domain optical flows (or an optical flow set) of the image frame and the feature map of the reference frame. multiple intermediate feature maps between them, and fuse the multiple intermediate feature maps to obtain the predicted feature map of the image frame, so that the decoding end (or encoding end) decodes (or encodes) the image frame based on the predicted feature map , obtain the target image (or code stream) corresponding to the image frame.
  • the image decoding process as an example, for a single feature domain optical flow, the error of the pixels in the feature domain optical flow is smaller than the error of the pixels in the image domain optical flow.
  • the decoding end determines the intermediate feature map based on the feature domain optical flow.
  • the decoding error caused is lower than the decoding error caused by the image domain optical flow in common technology.
  • the decoder decodes the image frame based on the feature domain optical flow, which reduces the image domain optical flow between two adjacent frames.
  • the decoding error improves the accuracy of image decoding.
  • the decoding end processes the feature maps of the reference frame based on multiple feature domain optical flows to obtain multiple intermediate feature maps, and fuses the multiple intermediate feature maps to determine the predicted feature map of the image frame. That is, the decoding end combines the optical flow set and the reference frame.
  • the multiple intermediate feature maps determined by the feature map are fused to obtain the predicted feature map of the image frame.
  • the predicted feature map contains more image information, so that when the decoder decodes the image frame based on the predicted feature map obtained by fusion, This avoids the problem that a single intermediate feature map is difficult to accurately express the first image, and improves the accuracy of image decoding and image quality (such as image clarity, etc.).
  • image decoding and image quality such as image clarity, etc.
  • FIG. 1 is an exemplary block diagram of a video encoding and decoding system provided by this application.
  • the term "video decoder” generally refers to both a video encoder and a video decoder.
  • video coding or “decoding” may generally refer to video encoding or video decoding.
  • the encoding end or the decoding end may be collectively referred to as an image decoding device.
  • the video encoding and decoding system includes an encoding end 10 and a decoding end 20.
  • Encoding end 10 generates encoded video data. Therefore, the encoding end 10 may be called a video encoding device.
  • the decoding end 20 may decode the encoded video data (such as a video including one or more image frames) generated by the encoding end 10 . Therefore, the decoding end 20 may be called a video decoding device.
  • Various implementations of encoding end 10, decoding end 20, or both may include one or more processors and memory coupled to the one or more processors.
  • the memory may include, but is not limited to, random access memory (random access memory, RAM), flash memory, read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (erasable PROM, EPROM), electronic Electrically erasable programmable read-only memory (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures accessible by a computer.
  • RAM random access memory
  • ROM read-only memory
  • PROM programmable ROM
  • EPROM erasable programmable read-only memory
  • EEPROM electronic Electrically erasable programmable read-only memory
  • flash memory or any other medium that can be used to store desired program code in the form of instructions or data structures accessible by a computer.
  • Encoding end 10 and decoding end 20 may include a variety of devices, including desktop computers, mobile computing devices, notebook (eg, laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called “smart” phones. , televisions, cameras, display devices, digital media players, video game consoles, vehicle-mounted computers, or the like.
  • Decoding end 20 may receive encoded video data from encoding end 10 via link 30 .
  • Link 30 may include one or more media or devices capable of moving encoded video data from encoding end 10 to decoding end 20 .
  • link 30 may include one or more communication media that enables encoding end 10 to transmit encoded video data directly to decoding end 20 in real time.
  • the encoding end 10 may modulate the encoded video data according to a communication standard (eg, a wireless communication protocol), and may transmit the modulated video data to the decoding end 20 .
  • the one or more communication media may include wireless and/or wired communication media, such as the radio frequency (RF) spectrum or one or more physical transmission lines.
  • RF radio frequency
  • the one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (eg, the Internet).
  • the one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from the encoding end 10 to the decoding end 20.
  • the encoded data may be output from output interface 140 to storage device 40.
  • encoded data may be accessed from storage device 40 through input interface 240.
  • Storage device 40 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray Disc, digital video disc (DVD), compact disc read-only memory, CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.
  • storage device 40 may correspond to a file server or another intermediate storage device that may hold encoded video generated by encoding end 10 .
  • the decoding end 20 may access the stored video data from the storage device 40 via streaming or downloading.
  • the file server may be any type of server capable of storing encoded video data and transmitting the encoded video data to decoding end 20 .
  • Example file servers include network servers (for example, for websites), file transfer protocol (FTP) servers, network attached storage (NAS) devices, or local disk drives. Decoder 20 may access the encoded video data through any standard data connection, including an Internet connection.
  • the transmission of encoded video data from storage device 40 may be a streaming transmission, a download transmission, or a combination of both.
  • the image decoding method provided by the present application can be applied to video encoding and decoding to support a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, streaming video transmission (for example, via the Internet), for storage in data Encoding of video data on a storage medium, decoding of video data stored on a data storage medium, or other applications.
  • a video codec system may be used to support one-way or two-way video transmission to support applications such as video streaming, video enhancement, video playback, video broadcasting, and/or video telephony.
  • the video codec system illustrated in FIG. 1 is only an example, and the techniques of this application are applicable to video coding arrangements (eg, video encoding or video decoding) that do not necessarily involve any data communication between encoding devices and decoding devices.
  • the data is retrieved from local storage, streamed over the network, and so on.
  • the video encoding device can process the data
  • the data is encoded and stored to memory, and/or the video decoding device may retrieve the data from memory and decode the data.
  • encoding and decoding are performed by devices that do not communicate with each other but merely encode data to and/or retrieve data from memory and decode data.
  • the encoding end 10 includes a video source 120 , a video encoder 100 and an output interface 140 .
  • output interface 140 may include a regulator/demodulator (modem) and/or a transmitter.
  • Video source 120 may include a video capture device (e.g., a video camera), a video archive containing previously captured video data, a video feed interface to receive video data from a video content provider, and/or a computer for generating video data. graphics system, or a combination of these sources of video data.
  • Video encoder 100 may encode video data from video source 120 .
  • encoder 10 transmits the encoded video data directly to decoder 20 via output interface 140 .
  • the encoded video data may also be stored on the storage device 40 for later access by the decoder 20 for decoding and/or playback.
  • the decoding terminal 20 includes an input interface 240 , a video decoder 200 and a display device 220 .
  • input interface 240 includes a receiver and/or modem.
  • Input interface 240 may receive encoded video data via link 30 and/or from storage device 40.
  • the display device 220 may be integrated with the decoding terminal 20 or may be external to the decoding terminal 20 . Generally speaking, display device 220 displays decoded video data.
  • the display device 220 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • video encoder 100 and video decoder 200 may each be integrated with an audio encoder and decoder, and may include appropriate multiplexer-demultiplexer units or other hardware and software to handle the encoding of both audio and video in a common data stream or in separate data streams.
  • the demultiplexer (MUX-DEMUX) unit may conform to the ITU H.223 multiplexer protocol, or other protocols such as user datagram protocol (UDP), if applicable.
  • Video encoder 100 and video decoder 200 may each be implemented as any of a variety of circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (application-specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA), discrete logic, hardware, or any combination thereof. If the present application is implemented partly in software, the device may store instructions for the software in a suitable non-volatile computer-readable storage medium, and the instructions may be executed in hardware using one or more processors Thereby implementing the technology of this application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered one or more processors. Each of video encoder 100 and video decoder 200 may be included in one or more encoders or decoders, either of which may be integrated as a combined encoder in the respective device. / part of the decoder (codec).
  • codec part of the decoder
  • Video encoder 100 may be generally referred to herein as "signaling” or “transmitting” certain information to another device, such as video decoder 200 .
  • the term “signaling” or “transmission” may generally refer to the transmission of syntax elements and/or other data used to decode compressed video data. This transfer can occur in real time or near real time. Alternatively, this communication may occur over a period of time, such as when encoding to store the syntax elements in the encoded codestream to a computer-readable storage medium, and the decoding device may then occur after the syntax elements are stored to such media. retrieve the syntax element at any time.
  • a video sequence usually contains a series of video frames or images.
  • a group of pictures exemplarily includes a series, one or more video images.
  • the GOP can be in the header information of the GOP, in one or more of the images
  • the header information or elsewhere contains syntax data describing the number of images contained in the GOP.
  • Each slice of an image may contain slice syntax data describing the encoding mode of the corresponding image.
  • Video encoder 100 typically operates on video blocks within individual video slices in order to encode video data. Video blocks may correspond to decoding nodes within a CU. Video blocks may be of fixed or varying size, and may differ in size depending on the specified decoding standard.
  • video encoder 100 may scan the quantized transform coefficients using a predefined scan order to produce serialized vectors that may be entropy encoded. In other possible implementations, video encoder 100 may perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 100 may perform context-based adaptive variable-length code (CAVLC), context-adaptive binary arithmetic decoding (context-based adaptive variable-length code). based adaptive binary arithmetic coding (CABAC), syntax-based adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) decoding or other entropy decoding methods. Entropy decodes a one-dimensional vector. Video encoder 100 may also entropy encode syntax elements associated with the encoded video data for use by video decoder 200 in decoding the video data.
  • CAVLC context-based adaptive variable-length code
  • CABAC based adaptive binary arithmetic coding
  • SBAC syntax-based adaptive binary arithmetic
  • video encoder 100 may assign context within the context model to symbols to be transmitted.
  • the context may relate to whether the adjacent values of the symbol are non-zero.
  • video encoder 100 may select a variable length code of symbols to be transmitted. Codewords in variable-length code (VLC) can be constructed such that relatively short codes correspond to more likely symbols, while longer codes correspond to less likely symbols. In this way, the use of VLC can achieve code rate savings relative to using equal length codewords for each symbol to be transmitted. Probabilities in CABAC can be determined based on the context assigned to the symbol.
  • This application may refer to the image currently being decoded by the video decoder as the current image.
  • the video encoding and decoding system provided by this application can be applied to video compression scenarios, such as artificial intelligence (artificial intelligence, AI) video encoding/decoding modules.
  • artificial intelligence artificial intelligence, AI
  • the video encoding and decoding system provided by this application can be used to store compressed video files for different services, such as data (image or video) storage of terminal albums, video surveillance, Huawei TM Cloud, etc.
  • the video encoding and decoding system provided by this application can be used to transmit compressed video files, such as Huawei TM Cloud, live video, etc.
  • the process of the output interface 140 sending the data stream to the outside can also be called push streaming, and the server cluster sends the data stream to the input interface.
  • the process of sending data streams can also be called distribution.
  • the encoding end 10 may use optical flow mapping to encode and compress the image frames.
  • the decoding end 20 may also use optical flow mapping to decode and reconstruct image frames.
  • optical flow represents the movement speed and direction of pixels in two adjacent frames of images.
  • image domain optical flow represents the motion information between pixels in two adjacent frames of images.
  • the motion information may include the motion speed and motion direction between pixels in two adjacent frames of images.
  • Feature domain optical flow It represents the motion information between the feature maps of two adjacent image frames.
  • the motion information indicates: the motion speed and direction of motion between the feature maps of the image frame and the feature maps of adjacent image frames of the image frame.
  • the adjacent image frame eg, image frame 2 of the aforementioned image frame (eg, image frame 1) may also be called a reference frame of the image frame 1.
  • Optical flow has two directions in the time dimension: direction 1, the optical flow from the previous frame image to the next frame image; direction 2, the optical flow from the next frame image to the previous frame image.
  • Optical flow in one direction is usually represented digitally, such as using a three-dimensional array [2, h, w].
  • “2" means that the optical flow contains 2 channels, and the first channel represents the deflection of the image in the x direction. Shift direction and size, the second channel represents the shift direction and size of the image in the y direction.
  • h is the height of the image
  • w is the width of the image.
  • a positive value means the object moves to the left, while a negative value means the object moves to the right; in the y direction, a positive value means the object moves upward, while a negative value means the object moves downward.
  • Figure 2 is a mapping relationship between optical flow and color provided by this application.
  • the arrow in (A) of Figure 2 shows the direction of the optical flow in the image, and the length of the arrow shows the size of the optical flow in the image; in Figure 2 (B) shows the color and brightness of the image determined based on the optical flow.
  • the color represents the direction of the optical flow
  • the brightness represents the size of the optical flow.
  • the greater the brightness the greater the value corresponding to the optical flow.
  • (B) in Figure 2 takes grayscale as an example to illustrate the color of the image, but this should not be understood as a limitation of the present application.
  • the method of predicting the current frame through the reference frame and the optical flow between the two frames is called optical flow mapping (Warping), usually expressed as in, is the predicted frame, x t-1 is the reference frame, and v t is the optical flow from the reference frame to the predicted frame.
  • Warping optical flow mapping
  • the pixel points of the current frame correspond to the position of the reference frame. According to the corresponding By interpolating the position of the reference frame, an estimate of the pixel value of the current frame can be obtained.
  • Optical flow mapping includes forward mapping (forward warping) and backward mapping (backward warping).
  • Figure 3 is a schematic diagram of the optical flow mapping provided by this application:
  • a in Figure 3 is forward mapping, which means that the decoder uses the optical flow between the previous frame image (image frame 1) and the previous frame. The flow predicts the next frame of image (Image Frame 2);
  • B in Figure 3 is the backward mapping, which means that the decoder predicts the previous frame of image (Image Frame 2) based on the optical flow between the next frame of image (Image Frame 2) and the previous and next frames.
  • Image frame 1 A in Figure 3 is forward mapping, which means that the decoder uses the optical flow between the previous frame image (image frame 1) and the previous frame. The flow predicts the next frame of image (Image Frame 2);
  • B in Figure 3 is the backward mapping, which means that the decoder predicts the previous frame of image (Image Frame 2) based on the optical flow between the next frame of image (Image Frame 2) and the previous and next frames. Image frame 1).
  • FIG. 4 is a schematic architectural diagram of the video compression system provided by this application.
  • the video compression system 400 includes: a motion estimation module 410, an optical flow compression module 420, a motion compensation module 430, and a residual Compression (residual compression) module 440, entropy coding (entropy coding) module 450, and multi-frame feature fusion (multi-frame feature fusion) module 460.
  • the motion estimation module 410, the optical flow compression module 420 and the motion compensation module 430 may be collectively referred to as variable motion compensation components.
  • the image frame X t obtains the feature map F t of the image frame after feature extraction.
  • the reference frame X t-1 obtains the feature map F t-1 of the reference frame after feature extraction.
  • the reference frame X t-1 Can be stored in the reconstructed frame buffer (decoded frame buffer), which is used to provide data storage space for multiple frames.
  • the motion estimation module 410 determines the feature domain optical flow O t between the image frame and the reference frame based on F t and F t-1 .
  • the optical flow compression module 420 compresses O t to obtain the feature domain optical flow code stream O' t .
  • the motion compensation module 430 performs feature prediction to determine the predicted feature map F t_pre corresponding to the image frame X t based on the feature map F t-1 of the reference frame and the decoded feature domain optical stream O ⁇ t .
  • the residual compression module 440 outputs the compressed decoding residual R ⁇ t according to the obtained feature domain residual R t .
  • the prediction feature map F t_pre and the decoding residual R ⁇ t can be used to determine the initial reconstruction feature F ⁇ t_intial of the image frame.
  • the multi-frame feature fusion module 460 calculates the reconstructed feature maps corresponding to the initial reconstructed feature F ⁇ t_intial and multiple reference frames (F t-1_ref , F t-2_ref and F t-3_ref shown in Figure 4) , determine the final reconstructed feature map F ⁇ t_final of the image frame X t .
  • the final reconstructed feature map F ⁇ t_final undergoes frame reconstruction to obtain the reconstructed frame X ⁇ t corresponding to the image frame X t .
  • the entropy encoding module 450 is used to encode at least one of the feature domain optical flow O t and the feature domain residual R t to obtain a binary code stream.
  • Figures 1 to 4 are only examples provided by this application.
  • the encoding end 10, the decoding end 20, the video encoder 100, the video decoder 200 and the video encoding and decoding system may include more or fewer components or Unit, this application is not limited to this.
  • Figure 5 is a schematic flow chart of an image coding method provided by this application.
  • the image coding method can be applied to the video encoding and decoding system shown in Figure 1 or the video compression system shown in Figure 4.
  • Example the image encoding method can be executed by the encoding end 10 or the video encoder 100.
  • the encoding end 10 executes the image encoding method provided in this embodiment as an example for description.
  • the image encoding method provided by this embodiment includes the following steps S510 to S560.
  • S510 The encoding end obtains the feature map of image frame 1 and the first feature map of the reference frame of image frame 1.
  • the image frame 1 and the reference frame may belong to the same GOP included in the video.
  • the video includes one or more image frames.
  • Image frame 1 can be any image frame in the video.
  • the reference frame can be an image frame adjacent to image frame 1 in the video.
  • the reference frame is in image frame 1 The previous adjacent image frame, or the reference frame is the adjacent image frame after image frame 1. It is worth noting that in some cases, image frame 1 may also be called the first image frame.
  • the way in which the encoding end obtains the feature map may include, but is not limited to: the encoding end is implemented based on a neural network model. Assume that the size of each image frame in the video is [3, H, W], that is, the image frame has 3 channels, a height of H, and a width of W.
  • Figure 6 is a schematic structural diagram of the feature extraction network provided by this application.
  • the feature extraction network includes 1 convolutional layer (conv) and 3 residual blocks (residual block). , resblock) processing layer, the convolution kernel of the convolution layer is 64 ⁇ 5 ⁇ 5/2, and the convolution kernel size of the resblock processing layer is 64 ⁇ 3 ⁇ 3.
  • the encoding end determines that the feature map corresponding to image frame 1 is f t , the first feature map of the reference frame is f t-1 .
  • Figure 6 is only an example of a feature extraction method provided in this embodiment and should not be understood as a limitation of this application.
  • the features shown in Figure 6 The parameters of each network layer in the extracted network can also be different.
  • the encoding end may also use other methods to extract features of the image frame 1 and the reference frame, which is not limited in this application.
  • S520 The encoding end obtains at least one optical flow set based on the feature map and the first feature map of image frame 1.
  • the first optical flow set in the at least one optical flow set corresponds to the aforementioned first characteristic map.
  • the first optical flow set may be the optical flow set 1 shown in FIG. 5 .
  • the first optical flow set may include one or more feature domain optical flows v t .
  • the feature domain optical flows v t are used to indicate the relationship between the feature map of the image frame 1 (or the first image frame) and the first feature map. movement information. This motion information can be used to indicate motion information and motion direction between the feature map of image frame 1 and the first feature map.
  • the process of obtaining the optical flow set by the encoding end is actually an optical flow estimation process.
  • the optical flow estimation process can be implemented by the motion estimation module 410 shown in Figure 4. This operation The motion estimation module 410 can be supported by the encoding end.
  • the encoding end may utilize an optical flow estimation network to determine the aforementioned optical flow set.
  • Figure 7 is a schematic structural diagram of an optical flow estimation network provided by this application.
  • the optical flow estimation network includes an upsampling network and a downsampling network, where the upsampling network includes three network layers 1 , Network layer 1 includes: 3 residual block processing layers (convolution kernel is 64 ⁇ 3 ⁇ 3) and 1 convolution layer (convolution kernel is 64 ⁇ 5 ⁇ 5/2).
  • the downsampling network includes 3 network layers 2, which in turn include: 1 convolution layer (convolution kernel is 64 ⁇ 5 ⁇ 5/2) and 3 residual block processing layers (convolution kernel is 64 ⁇ 3 ⁇ 3).
  • a residual block processing layer includes: a convolution layer (convolution kernel is 64 ⁇ 3 ⁇ 3), an activation layer and a convolution layer (convolution kernel is 64 ⁇ 3 ⁇ 3).
  • the activation layer may refer to a linear rectified layer (rectified linear units layer, ReLU), or a parametric rectified linear unit (PReLU), etc.
  • Figure 7 is only an example of the optical flow estimation network provided by the embodiment of the present application and should not be understood as a limitation of the present application.
  • the size of the convolution kernel of each network layer, the number of feature map channels of the input network layer, The downsampling position, number of convolutional layers, and network activation layers can all be adjusted.
  • the optical flow estimation network can also use more complex network structures.
  • S530 The encoding end performs feature domain optical flow coding on the optical flow set obtained in S520 to obtain a feature domain optical flow code stream.
  • the feature domain optical flow code stream includes the code stream corresponding to optical flow set 1.
  • the feature domain optical stream code stream may be a binary file, or the feature domain optical stream code stream may also be other types of files that comply with the multimedia transmission protocol, without limitation.
  • S540 The encoding end processes the first feature map based on the feature domain optical flow included in the optical flow set 1, and obtains one or more intermediate feature maps corresponding to the first feature map.
  • a feasible processing method is provided here.
  • the encoding end performs optical flow mapping on the first feature map based on the first feature domain optical flow (v 1 ). (warping), obtain the intermediate feature map corresponding to the optical flow in the first feature domain. It is worth noting that warping a feature domain optical flow with the first feature map will obtain an intermediate feature map.
  • the number of intermediate feature maps corresponding to the first feature map is consistent with the number of feature domain optical flows included in optical flow set 1.
  • Figure 8 is a schematic process diagram of optical flow mapping and feature fusion provided by this application.
  • Optical flow set 1 contains multiple feature domain optical flows, such as v 1 to v m , and m is a positive integer.
  • the encoding end warps each feature domain optical flow included in the optical flow set 1 with the first feature map f t to obtain m intermediate feature maps.
  • S550 The encoding end fuses one or more intermediate feature maps to obtain the first predicted feature map of the first image frame.
  • the decoder obtains multiple image domain optical flows between image frames and reference frames, and obtains multiple images based on multiple image domain optical flows and reference frames, thereby fusing multiple images to obtain image frame correspondence.
  • the target image Therefore, the decoder decodes one frame of image and needs to predict the pixel values of multiple images and fuse the multiple images to obtain the target image. This results in large computing resources required for image decoding.
  • the decoder uses image domain optical flow Decoding video is less efficient.
  • the encoding end inputs the aforementioned one or more intermediate feature maps into the feature fusion model to obtain the first predicted feature map.
  • the feature fusion model includes a convolutional network layer, which is used to fuse intermediate feature maps.
  • the decoder uses a feature fusion model to fuse multiple intermediate feature maps.
  • the decoder uses a feature fusion model to fuse multiple intermediate feature maps.
  • the prediction feature map obtained from the intermediate feature map decodes the image frame, that is, the decoder only needs to predict the pixel value of the image position indicated by the prediction feature map in the image based on the prediction feature map, and does not need to predict all pixels of multiple images. value, reducing the computing resources required for image decoding and improving the efficiency of image decoding.
  • the encoding end obtains one or more weights of the aforementioned one or more intermediate feature maps, and processes the one or more weights based on the one or more weights.
  • the intermediate feature maps corresponding to respective values, and the first predicted feature map is obtained by adding all processed intermediate feature maps.
  • the weight is used to indicate the weight of the intermediate feature map in the first predicted feature map.
  • the weight of each intermediate feature map can be different.
  • the encoding end can set different weights for the intermediate feature maps according to the requirements of image encoding. , For example, if the images corresponding to some intermediate feature maps are relatively blurry, the weights of the intermediate feature maps corresponding to these blurred images are reduced, thereby improving the clarity of the first image.
  • the weight may refer to the mask value corresponding to each optical flow.
  • the optical flow set 1 includes 9 feature domain optical flows.
  • the weights of each feature domain optical flow are as follows: For example, the size of the optical flow in the feature domain is [2, H/s, W/s], and the size of the mask is [1, H/s, W/s].
  • the encoding end fuses the intermediate feature maps corresponding to the optical flows in these nine feature domains to obtain the first predicted feature map corresponding to the first feature map.
  • the feature map fusion in this application can only adopt the above two methods.
  • the encoding end can warping each feature domain optical flow included in the optical flow set 1 with the first feature map f t respectively.
  • Four intermediate feature maps are obtained; furthermore, the encoding end performs feature fusion on the four intermediate feature maps to obtain the first predicted feature map.
  • the image encoding method provided in this embodiment also includes the following step S560.
  • S560 The encoding end encodes the residual corresponding to image frame 1 according to the first prediction feature map to obtain a residual code stream.
  • the residual code stream of the image area corresponding to the first prediction feature map and the feature domain optical flow code stream determined in S530 can be collectively referred to as the code stream corresponding to image frame 1 (or the first image frame).
  • the code stream includes: a feature domain optical flow code stream corresponding to the aforementioned optical flow set and a residual code stream of the image area corresponding to the first prediction feature map.
  • the encoding end determines a set of feature domain optical flows ( As the aforementioned optical flow set 1), and process the first feature map of the reference frame to obtain one or more intermediate feature maps; secondly, obtain one or more intermediate feature maps corresponding to the first feature map at the encoding end Finally, the one or more intermediate feature maps are fused to obtain the first prediction feature map corresponding to the first image frame; finally, the encoding end encodes the first image frame according to the first prediction feature map to obtain a code stream.
  • a set of feature domain optical flows As the aforementioned optical flow set 1
  • the encoding end encodes the image frame according to the feature domain optical flow, which reduces the coding error caused by the image domain optical flow between two adjacent frames and improves the coding error.
  • Image encoding accuracy Moreover, the encoding end processes the feature maps of the reference frame based on multiple feature domain optical flows to obtain multiple intermediate feature maps, and fuses the multiple intermediate feature maps to determine the predicted feature map of the image frame.
  • the encoding end combines the optical flow set and the reference frame.
  • the multiple intermediate feature maps determined by the feature map are fused to obtain the prediction feature map of the image frame.
  • the prediction feature map contains more image information, so that when the encoding end encodes the image frame based on the prediction feature map obtained by fusion, This avoids the problem that a single intermediate feature map is difficult to accurately express the first image, and improves the accuracy of image coding and image quality (such as image clarity, etc.).
  • the reference frame may correspond to multiple feature maps, such as the aforementioned first feature map and the second feature map.
  • the first feature map and the second feature map may refer to different channels among the multiple channels included in the reference frame. Two feature maps of the same channel.
  • a set of feature maps corresponding to the reference frame may correspond to an optical flow set.
  • the aforementioned optical flow set 1 may also correspond to the second feature map.
  • the encoding end can also process the second feature map based on the feature domain optical flow included in the optical flow set 1 to obtain the second feature map.
  • One or more intermediate feature maps corresponding to the feature map (f ⁇ t-1 to f ⁇ tm as shown in Figure 8, m is a positive integer).
  • the encoding end also fuses one or more intermediate feature maps corresponding to the second feature map to obtain the second predicted feature map of the first image frame.
  • the encoding end encoding the first image frame may include the following content: the encoding end bases the first prediction feature map and the second feature map on The second prediction feature map is used to encode the first image frame to obtain a code stream.
  • the code stream may include the residual code stream of the image area corresponding to the first feature map and the second feature map in the first image frame, and the feature domain optical flow code stream corresponding to the aforementioned optical flow set 1.
  • multiple feature maps (or a set of feature maps, or a feature map group, etc.) of the reference frame may correspond to an optical flow set (or a set of feature domain optical flows). For example, for Multiple feature maps belonging to the same group share a set of feature domain flows.
  • the encoding end processes the feature map according to a set of feature domain optical flows corresponding to the feature map, thereby obtaining the intermediate feature map corresponding to the feature map. Further, the encoding end will The intermediate feature map corresponding to the feature map is fused to obtain the predicted feature map corresponding to the feature map. Finally, the encoding end encodes the image frame according to the prediction feature maps corresponding to all feature maps of the reference frame to obtain the target code stream.
  • the encoding end can divide the multiple feature maps into one group or multiple groups.
  • the feature maps belonging to the same group share an optical flow set, and
  • the intermediate feature map corresponding to the feature map is fused to obtain the predicted feature map, which avoids the problem that when the reference frame or image frame has more information, the code stream obtained by the encoding end based on the feature map encoding has more redundancy and lower accuracy, and improves improve the accuracy of image coding. It is worth noting that the number of channels of feature maps belonging to different groups can be different.
  • the encoding end can use different feature fusion methods for different feature maps of the reference frame.
  • the intermediate feature map corresponding to the feature map of the reference frame is fused to obtain a predicted feature map corresponding to the feature map of the reference frame.
  • the reference frame corresponds to feature map 1 and feature map 2.
  • the encoding end inputs multiple intermediate feature maps corresponding to feature map 1 into the feature fusion model to determine the predicted feature map corresponding to feature map 1; the encoding end obtains feature map 2.
  • the corresponding weight of each intermediate feature map is processed, and the corresponding intermediate feature map is processed according to these weights to obtain the predicted feature map corresponding to feature map 2.
  • the encoding end can set the confidence level for the predicted feature map output by each feature fusion method, so as to meet the different coding requirements of the user.
  • FIG. 9 is a schematic flowchart 1 of the image decoding method provided by this application.
  • the image decoding method can be applied to the video encoding and decoding system shown in Figure 1 or the video encoding and decoding system shown in Figure 4.
  • Video compression system, exemplary image encoding The method may be executed by the decoding end 20 or the video decoder 200.
  • the decoding end 20 executes the image decoding method provided in this embodiment as an example for description.
  • the image decoding method provided by this embodiment includes the following steps S910 to S940.
  • the decoding end parses the code stream to obtain at least one optical flow set.
  • the at least one optical flow set includes a first optical flow set (optical flow set 1 as shown in Figure 9), the optical flow set 1 corresponds to the first feature map of the reference frame, and the optical flow set 1 includes one or more A feature domain optical flow, wherein any one of the one or more feature domain optical flows is used to indicate motion information between the feature map of the first image frame and the aforementioned first feature map.
  • a first optical flow set optical flow set 1 as shown in Figure 9
  • the optical flow set 1 corresponds to the first feature map of the reference frame
  • the optical flow set 1 includes one or more A feature domain optical flow, wherein any one of the one or more feature domain optical flows is used to indicate motion information between the feature map of the first image frame and the aforementioned first feature map.
  • the reference frame may refer to the image frame adjacent to the first image frame that has been decoded by the decoding end (the image frame before the first image frame, or the image frame after the first image frame), or,
  • the reference frame is an image frame carried in the code stream that contains complete information of an image.
  • the decoding end processes the first feature map based on the feature domain optical flow included in the optical flow set 1, and obtains one or more intermediate feature maps corresponding to the first feature map.
  • the process by which the decoder processes the feature map of the reference frame based on the feature domain optical flow is also called feature alignment, feature prediction, or feature align, and this application is not limited to this.
  • the decoding end can also process the first feature map in the same manner as the aforementioned S520, and obtain one or more features corresponding to the first feature map. an intermediate feature map.
  • the decoder performs warping on the first feature map based on a set of feature domain optical flows.
  • the decoder can interpolate based on the position of a set of feature domain optical flows corresponding to the reference frame to obtain the predicted value of the pixel value in the first image to avoid This eliminates the need for the decoder to predict all pixel values of the first image based on the image domain optical flow, reduces the amount of calculation required for image decoding at the decoder, and improves the efficiency of image decoding.
  • S930 The decoder fuses one or more intermediate feature maps determined in S920 to obtain the first predicted feature map of the first image frame.
  • the decoding end may input the one or more intermediate feature maps determined in S920 to the feature fusion model to obtain the first predicted feature map.
  • the feature fusion model includes a convolutional network layer, which is used to fuse intermediate feature maps.
  • the decoder uses a feature fusion model to fuse multiple intermediate feature maps.
  • the decoder uses a feature fusion model to fuse multiple intermediate feature maps.
  • the predicted feature map obtained from the intermediate feature map decodes the image frame, reducing the computing resources required for image decoding and improving the efficiency of image decoding.
  • the decoder may also obtain one or more weights of the aforementioned one or more intermediate feature maps, where one intermediate feature map corresponds to one weight; and the decoder may process the intermediate feature map based on the weight of the intermediate feature map, and All processed intermediate feature maps are added to obtain the first predicted feature map.
  • the weight is used to indicate the weight of the intermediate feature map in the first predicted feature map.
  • the weight of each intermediate feature map can be different.
  • the decoder can set different weights for the intermediate feature maps according to the requirements of image decoding.
  • the weights of these intermediate feature maps are reduced, thereby improving the clarity of the first image.
  • the weight of the intermediate feature map please refer to the relevant content of S550 and will not be described in detail here.
  • the decoder decodes the first image frame according to the first prediction feature map determined in S930 to obtain the first image.
  • the decoder can decode the first image frame according to the reconstructed feature map f_res to obtain the first image.
  • the decoder determines the first feature of the reference frame based on a set of feature domain optical flows corresponding to the first image frame (such as the aforementioned first optical flow set).
  • the image is processed to obtain one or more intermediate feature maps; secondly, after the decoder obtains one or more intermediate feature maps corresponding to the first feature map, the one or more intermediate feature maps are fused to obtain the first The first prediction feature map corresponding to the image frame; finally, the decoder decodes the first image frame according to the first prediction feature map to obtain the first image.
  • the decoder decodes the image frame according to the feature domain optical flow, which reduces the decoding error caused by the image domain optical flow between two adjacent frames and improves improve the accuracy of image decoding.
  • the decoding end processes the feature maps of the reference frame based on multiple feature domain optical flows to obtain multiple intermediate feature maps, and fuses the multiple intermediate feature maps to determine the predicted feature map of the image frame. That is, the decoding end combines the optical flow set and the reference frame.
  • the multiple intermediate feature maps determined by the feature map are fused to obtain the predicted feature map of the image frame.
  • the predicted feature map contains more image information, so that when the decoder decodes the image frame based on the predicted feature map obtained by fusion, This avoids the problem that a single intermediate feature map is difficult to accurately express the first image, and improves the accuracy of image decoding and image quality (such as image clarity, etc.).
  • the encoding end can encode the feature map through a feature encoding network
  • the decoding end can encode the code stream corresponding to the feature map through a feature decoding network.
  • Figure 10 is the feature encoding network provided by this application.
  • the feature encoding network includes 3 network layers 1.
  • the network layer 1 includes: 3 residual block processing layers (convolution kernel is 64 ⁇ 3 ⁇ 3) and 1 convolution layer (convolution layer).
  • the core is 64 ⁇ 5 ⁇ 5/2).
  • the feature decoding network includes 3 network layers 2.
  • Network layer 2 includes: 1 convolution layer (convolution kernel is 64 ⁇ 5 ⁇ 5/2) and 3 residual block processing layers (convolution kernel is 64 ⁇ 3 ⁇ 3).
  • a residual block processing layer includes in sequence: a convolution layer (the convolution kernel is 64 ⁇ 3 ⁇ 3), an activation layer and a convolution layer (the convolution kernel is 64 ⁇ 3 ⁇ 3). 64 ⁇ 3 ⁇ 3).
  • the activation layer may refer to the ReLU layer or PReLU, etc.
  • Figure 10 is only an example of the feature encoding network and the feature decoding network provided by the embodiment of the present application, and should not be understood as a limitation of the present application.
  • the convolution kernel size of each network layer, the feature map of the input network layer The number of channels, downsampling positions, number of convolutional layers, and network activation layers can all be adjusted. In some more complex scenes, the optical flow estimation network can also use more complex network structures.
  • the reference frame may correspond to multiple feature maps, such as the aforementioned first feature map and the second feature map.
  • a set of feature maps corresponding to the reference frame may correspond to an optical flow set.
  • the aforementioned optical flow set 1 may also correspond to the second feature map of the reference frame.
  • the decoder can also process the second feature map based on the feature domain optical flow included in the optical flow set 1 to obtain the second feature map.
  • One or more intermediate feature maps corresponding to the feature map (f ⁇ t-1 to f ⁇ tm as shown in Figure 8, m is a positive integer).
  • the decoder also fuses one or more intermediate feature maps corresponding to the second feature map to obtain the second predicted feature map of the first image frame.
  • multiple feature maps (or a group of feature maps) of the reference frame can correspond to an optical flow set (or a set of feature domain optical flows).
  • the decoding end processes the feature map according to a set of feature domain optical flows corresponding to the feature map, thereby obtaining an intermediate feature map corresponding to the feature map.
  • the decoder end processes the intermediate feature map corresponding to the feature map. Fusion is performed to obtain the predicted feature map corresponding to the feature map.
  • the decoder decodes the image frame according to the prediction feature maps corresponding to all feature maps of the reference frame to obtain the target image.
  • the decoder can divide the multiple feature maps into one group or multiple groups.
  • the feature maps belonging to the same group share an optical flow set, and
  • the intermediate feature map corresponding to the feature map is fused to obtain the predicted feature map, which avoids the problem of low accuracy in reconstructing the image based on the feature map at the decoder when the reference frame or image frame has more information, and improves the accuracy of image decoding.
  • FIG 11 is a schematic flow chart 2 of the image decoding method provided by this application.
  • the image decoding method shown in Figure 11 can be compared with the image encoding method provided by the previous embodiment. Combined with the image decoding method, it can also be implemented separately.
  • the decoder executes the image decoding method shown in Figure 11 as an example.
  • the image decoding method provided by this embodiment includes the following steps S1110 to S1130.
  • the decoder obtains the feature map of the first image.
  • the decoder can obtain the feature map of the first image according to the feature extraction network shown in Figure 6, which will not be described again here.
  • the decoder obtains an enhanced feature map based on the feature map of the first image, the first feature map of the reference frame, and the first predicted feature map.
  • the decoder can use a feature fusion model to fuse multiple feature maps included in S1120 to obtain an enhanced feature map.
  • the feature fusion model may refer to the feature fusion model provided by the aforementioned S550 or S930, or may refer to other models including a convolution layer.
  • the convolution kernel of the convolution layer is 3 ⁇ 3, which is not limited in this application. .
  • the decoder can also set different weights for the feature map of the first image, the first feature map of the reference frame, and the first prediction feature map, so as to fuse these multiple feature maps to obtain enhanced features. picture.
  • the enhancement feature map may be used to determine the enhancement layer image of the first image.
  • the video quality of the enhancement layer image is higher than the video quality of the first image.
  • the video quality may refer to or include image signal-to-noise ratio (SNR), resolution (image resolution) and peak signal-to-noise ratio (SNR). Peak signal-to-noise ratio, PSNR) at least one.
  • SNR image signal-to-noise ratio
  • PSNR peak signal-to-noise ratio
  • the image signal-to-noise ratio refers to the ratio of the signal mean of the image to the background standard deviation.
  • the "signal mean” here generally refers to the grayscale mean of the image, and the background standard deviation can be used as the variance of the background signal value of the image.
  • the variance of the background signal value refers to the noise power; for images, the greater the image signal-to-noise ratio, the better the image quality.
  • Resolution is the number of pixels per unit area of a single frame image. The higher the resolution of the image, the better the quality of the image.
  • PSNR is used to indicate the subjective quality of an image. The larger the PSNR, the better the quality of the image. For more information about SNR, resolution and PSNR, please refer to the relevant descriptions of the prior art, which will not be described again here.
  • the decoder processes the first image according to the enhanced feature map to obtain the second image.
  • the second image indicates the same content as the first image, but the second image is sharper than the first image.
  • image sharpness may be indicated by peak signal-to-noise ratio (PSNR),
  • PSNR peak signal-to-noise ratio
  • the decoder can fuse the feature map of the first image, the first feature map and the first predicted feature map, and perform video enhancement processing on the first image based on the enhanced feature map obtained through the fusion to obtain better clarity. Excellent second image, thereby improving the decoded image clarity and image display effect.
  • the decoder processes the first image according to the enhanced feature map to obtain the second image, including: the decoder obtains the enhancement layer image of the first image based on the enhanced feature map, and reconstructs the first image based on the enhancement layer image to obtain the second image.
  • the enhancement layer image may refer to an image determined by the decoder based on the reference frame and the enhancement feature map.
  • the decoder adds part or all of the information of the enhancement layer image to the first image to obtain the second image; or , the decoder uses the enhancement layer image as the reconstructed image of the first image, that is, the aforementioned second image.
  • the decoder obtains multiple feature maps of the image at different stages, and obtains the enhanced feature map determined by these multiple feature maps, so as to reconstruct and enhance the first image based on the enhanced feature map. Improved image clarity and image display after decoding.
  • the decoding end may reconstruct the first image based on an image reconstruction network, as shown in Figure 12.
  • Figure 12 is a schematic structural diagram of the image reconstruction network provided by this application.
  • the image The reconstruction network includes: 3 residual block processing layers (convolution kernel is 64 ⁇ 3 ⁇ 3) and 1 deconvolution layer (convolution kernel is 3 ⁇ 5 ⁇ 5/2), in which the residual block
  • the processing layer can include 1 convolution layer (convolution kernel is 64 ⁇ 3 ⁇ 3), an activation layer and 1 convolution layer (convolution kernel is 64 ⁇ 3 ⁇ 3).
  • the encoding end/decoding end predicts the feature map from an intermediate feature map set determined by performing optical flow mapping on the feature map of the reference frame.
  • the encoding end/decoding end can also predict the feature map based on the deformable Networks (deformable convolutional networks, DCN) process feature maps.
  • DCN is implemented based on convolution. The following is a mathematical expression of convolution:
  • n is the convolution kernel size
  • w is the convolution kernel weight
  • F is the input feature map
  • p is the convolution position
  • p k is the enumeration value of the position relative to p in the convolution kernel.
  • DCN is based on a network learning offset (offset), so that the convolution kernel is offset at the sampling point of the input feature map and concentrated on the region of interest (ROI) or target area. Its mathematical expression is:
  • ⁇ p k is the offset relative to p k , so that the convolution sampling position becomes an irregular position.
  • m(p k ) indicates that the mask value of position p k is the penalty term for position p k .
  • the convolution operation of convolving the most central point through points in the neighborhood (a) is the common sampling method of 3x3 convolution kernel, (b) is after sampling deformable convolution plus offset changes in the sampling points, where (c) and (d) are special forms of deformable convolution.
  • the multiple pixel points in (c) are scaled to obtain the pixel prediction value of the corresponding position in the target image, and as (c) The multiple pixels in d) are rotated and scaled to obtain the pixel prediction value of the corresponding position in the target image.
  • the image encoding method and image decoding method provided by the embodiments of the present application can also be implemented by DCN. It can be understood that when the reference frame corresponds to multiple channel different feature maps, each feature map can use the same or different DCN processing methods, as shown in Figure 13, the four possible DCN processing methods.
  • Figure 14 is a schematic diagram of the efficiency comparison provided by this application.
  • Figure 14 provides two indicators to compare the technical solution provided by this application and common technology. These two The indicators are: PSNR and bits per pixel (BPP).
  • BPP refers to the number of bits used to store each pixel. BPP is also used to indicate the resolution of the image.
  • the BPP of the solution provided by the common technology is higher than the technical solution provided by the embodiment of the present application, that is, the storage occupied by the code stream generated by the common technology
  • the space is larger and the network bandwidth required to transmit the code stream is larger; however, the PSNR of the solution provided by the general technology is lower than that of the technical solution provided by the embodiment of the present application. That is, in the technical solution provided by the embodiment of the present application, after the code stream is decoded
  • the subjective quality of the video (or image) obtained is better.
  • the end-to-end time of the technical solution to test the frame image is usually 0.3206 seconds (second, s).
  • the technical solution test provided by the embodiment of the present application The end-to-end time of this frame image is 0.2188s. That is to say, during the video encoding and decoding process, the technical solution provided by the embodiment of the present application can not only save the bit rate (reflected by BPP), but also ensure the quality (reflected by PSNR), and also reduce the encoding and decoding time of a single frame image. delay, thereby improving the overall efficiency of video encoding and decoding.
  • image encoding method and image decoding method can not only be applied to video encoding and decoding, video enhancement, video compression and other scenarios, but also can be applied to all video frame requirements such as video prediction, video frame insertion, and video analysis.
  • the encoding end and the decoding end include corresponding hardware structures and/or software modules for performing each function.
  • the units and method steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software driving the hardware depends on the specific application scenarios and design constraints of the technical solution.
  • Figure 15 is a schematic structural diagram of an image encoding device provided by the present application.
  • the image encoding device can be used to implement the functions of the encoding end or video encoder in the above method embodiments, and therefore can also achieve the beneficial effects of the above image encoding method embodiments.
  • the image encoding device may be the encoding end 10 or the video encoder 100 as shown in FIG. 1 , or may be a module (such as a chip) applied to the encoding end 10 or the video encoder 100 .
  • the image encoding device 1500 includes: an acquisition unit 1510 , a processing unit 1520 , a fusion unit 1530 and an encoding unit 1540 .
  • the image encoding device 1500 is used to implement the functions of the encoding end or video encoder in the method embodiments shown in FIG. 5 and FIG. 8 .
  • the acquisition unit 1510 is used to perform S510
  • the processing unit 1520 is used to perform S520 and S540
  • the fusion unit 1530 is used to perform S550
  • the encoding unit 1540 is used to perform S530 and S560.
  • the processing unit 1520 and the fusion unit 1530 are used to implement the functions of optical flow mapping and feature fusion.
  • FIG. 16 is a schematic structural diagram of the image decoding device provided by the present application.
  • the image decoding device can be used to implement decoding in the above method embodiments.
  • the function of the terminal or video decoder can also achieve the beneficial effects of the above image decoding method embodiments.
  • the image decoding device may be the decoding terminal 20 or the video decoder 200 as shown in FIG. 1 , or may be a module (such as a chip) applied to the decoding terminal 20 or the video decoder 200 .
  • the image decoding device 1600 includes: a code stream unit 1610, a processing unit 1620, a fusion unit 1630 and a decoding unit 1640.
  • the image decoding device 1600 is used to implement the functions of the decoding end or video decoder in the method embodiments shown in FIG. 8 and FIG. 9 .
  • the processing unit 1620 and the fusion unit 1630 are used to implement the functions of optical flow mapping and feature fusion.
  • the code stream unit 1610 is used to perform S910
  • the processing unit 1620 is used to perform S920
  • the fusion unit 1630 is used to perform S930
  • the decoding unit 1640 is used to perform S940 .
  • the image decoding device 1600 may further include an enhancement unit configured to process the first image according to the enhanced feature map to obtain the second image.
  • the enhancement unit is specifically configured to: obtain the enhancement layer image of the first image based on the enhancement feature map. And, reconstruct the first image based on the enhancement layer image to obtain the second image.
  • the image encoding (or image decoding) device and its respective units may also be software modules.
  • the software module is called by the processor to implement the above image encoding (or image decoding) method.
  • the processor can be a central processing unit (CPU), an application-specific integrated circuit (ASIC) implementation, or a programmable logic device (PLD), which can be a complex program Logic device (complex programmable logical device, CPLD), field programmable gate array (field programmable gate array, FPGA), general array logic (generic array logic, GAL) or any combination thereof.
  • image encoding (or image decoding) device For a more detailed description of the above image encoding (or image decoding) device, please refer to the embodiments shown in the preceding figures. The relevant descriptions are not repeated here. It can be understood that the image encoding (or image decoding) device shown in the foregoing drawings is only an example provided by this embodiment.
  • the image encoding (or image decoding) device may vary depending on the image encoding (or image decoding) process or service. More or fewer units may be included, which is not limited by this application.
  • the hardware may be implemented by a processor or a chip.
  • the chip includes interface circuit and control circuit.
  • the interface circuit is used to receive data from other devices other than the processor and transmit it to the control circuit, or to send data from the control circuit to other devices other than the processor.
  • control circuit is used to implement any of the possible implementation methods in the above embodiments through logic circuits or executing code instructions.
  • the beneficial effects can be found in the description of any aspect in the above embodiments and will not be described again here.
  • processor in the embodiments of the present application may be a CPU, a neural processing unit (NPU) or a graphics processing unit (GPU), or other general-purpose processors, digital signal Processor (digital signal processor, DSP), ASIC, FPGA or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.
  • a general-purpose processor can be a microprocessor or any conventional processor.
  • the method steps in the embodiments of the present application can be implemented by means of hardware.
  • the hardware is an image decoding device, as shown in Figure 17.
  • Figure 17 is a schematic structural diagram of the image decoding device provided by the present application.
  • the image decoding device 1700 includes a memory 1710 and at least one processor 1720.
  • the processor 1720 can implement the image encoding method and the image decoding method provided in the above embodiments.
  • the memory 1710 is used to store the correspondence between the above image encoding method and the image decoding method. software instructions.
  • the image decoding device 1700 may refer to a chip or chip system encapsulating one or more processors 1720 .
  • the processor 1720 included in the image decoding device 1700 executes the steps of the above method and its possible sub-steps.
  • the image decoding device 1700 may also include a communication interface 1730, which may be used to send and receive data.
  • the communication interface 1730 is used to receive the user's encoding request, decoding request, or send code stream, receive code stream, etc.
  • the communication interface 1730, the processor 1720, and the memory 1710 can be connected through a bus 1740.
  • the bus 1740 can be divided into an address bus, a data bus, a control bus, etc.
  • the image decoding device 1700 can also perform the functions of the image encoding device 1500 shown in FIG. 15 and the functions of the image decoding device 1600 shown in FIG. 16 , which will not be described again here.
  • the image decoding device 1700 provided in this embodiment may be a server, a personal computer, or other image decoding device 1700 with data processing functions, which is not limited by this application.
  • the image decoding device 1700 may be the aforementioned encoding end 10 (or video encoder 100), or the decoding end 20 (or video decoder 200).
  • the image decoding device 1700 may also have the functions of the aforementioned encoding end 10 and decoding end 20.
  • the image decoding device 1700 refers to a video codec system (or video compression system) with a video codec function. ).
  • the method steps in the embodiments of the present application can also be implemented by a processor executing software instructions.
  • Software instructions can be composed of corresponding software modules.
  • Software modules can be stored in random access memory (random access memory, RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or other well-known in the art any other form of storage media.
  • An exemplary storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium.
  • the storage medium may also be an integral part of the processor.
  • the processor and storage media may be located in an ASIC. Additionally, the ASIC can be located in network equipment or terminal equipment. Of course, the processor and the storage medium can also exist as discrete components in network equipment or terminal equipment.
  • this application also provides a computer-readable storage medium, which stores a code stream obtained according to the image encoding method provided in any of the foregoing embodiments.
  • the computer-readable storage medium may be, but is not limited to: RAM, flash memory, ROM, PROM, EPROM, EEPROM, register, hard disk, removable hard disk, CD-ROM or any other form of storage medium well known in the art.
  • the computer program product includes one or more computer programs or instructions.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment, or other programmable device.
  • the computer program or instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another.
  • the computer program or instructions may be transmitted from a website, computer, A server or data center transmits via wired or wireless means to another website site, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media.
  • the available media may be magnetic media, such as floppy disks, hard disks, and magnetic tapes; they may also be optical media, such as digital video discs (DVDs); they may also be semiconductor media, such as solid state drives (solid state drives). ,SSD).
  • “at least one” refers to one or more, and “plurality” refers to two or more.
  • “And/or” describes the association of associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A, B can be singular or plural.
  • the character “/” generally indicates that the related objects before and after are an “or” relationship; in the formula of this application, the character “/” indicates that the related objects before and after are a kind of "division” Relationship.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

一种图像解码方法、编码方法及装置，涉及视频处理技术领域。解码端基于图像帧对应的一组特征域光流(如光流集)，对参考帧的第一特征图进行处理获得一组中间特征图；解码端融合该组中间特征图进行获得预测特征图；解码端根据该预测特征图解码图像帧获得目标图像。该特征域光流中像素误差小于图像域光流中像素误差，因此，由特征域光流确定的中间特征图所带来的解码误差低于通常技术中图像域光流所导致的解码误差。图像帧的预测特征图是解码端融合多个中间特征图确定的，该预测特征图包含了更多的图像信息，解码端基于预测特征图解码图像帧时，避免了单个中间特征图难以准确表达目标图像的问题，提高了图像解码的准确性和图像质量。

Description

一种图像解码方法、编码方法及装置
本申请要求于2022年04月15日提交国家知识产权局、申请号为202210397258.4、申请名称为“图像解码方法、编码方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及视频处理技术领域，尤其涉及一种图像解码方法、编码方法及装置。
背景技术
在视频编解码技术中，视频压缩和视频增强技术尤为重要。视频压缩系统执行空间(图像内)预测和/或时间(图像间)预测，以减少或移除视频序列中固有的冗余信息；视频增强技术用于提高图像的显示质量。针对于视频压缩或者视频增强对应的视频解码过程，解码端采用光流映射(warping)的方法对视频包含的图像帧进行解码。光流映射是指解码端获取图像帧和参考帧之间的图像域光流，并根据该光流对图像帧进行解码。图像域光流用于指示相邻两帧图像中的相应像素之间的运动速度和运动方向。然而，光流映射对光流精度较为敏感，光流精度的细微变化会影响warping的准确性。由于相邻两帧图像之间的光流预测的误差较大，如5个像素或更多像素的误差，因此，解码端基于图像域光流对图像帧进行解码的准确性较低，导致解码获得的图像的清晰度受到影响。因此，如何提供一种更有效的图像解码方法成为目前亟需解决的问题。
发明内容
本申请提供一种图像解码方法、编码方法及装置，解决了基于图像域光流对图像帧进行解码的准确性较低，导致解码获得的图像的清晰度受到影响的问题。
本申请采用如下技术方案。
第一方面，本申请提供了一种图像解码方法，该方法应用于视频编解码系统，或支持实现该视频编解码系统实现图像解码方法的物理设备，例如该物理设备为解码端或者视频解码器(decoder)，在一些情况中，该物理设备可包括芯片系统。这里以解码端执行图像解码方法为例进行说明，该图像解码方法包括：首先，解码端解析码流获得至少一个光流集；该至少一个光流集包括第一光流集，该第一光流集对应前述第一图像帧的参考帧的第一特征图，且该第一光流集包括一个或多个特征域光流，该一个或多个特征域光流中的任一个特征域光流用于指示第一图像帧的特征图与前述第一特征图之间的运动信息。其次，解码端基于第一光流集包括的一个或多个特征域光流处理第一特征图，获得第一特征图对应的一个或多个中间特征图；以及，解码端融合前述的一个或多个中间特征图，获得第一图像帧的第一预测特征图。最后，解码端根据该第一预测特征图解码第一图像帧获得第一图像。
在本实施例中，对于单个特征域光流而言，该特征域光流中像素的误差小于图像域光流中像素的误差，因此，解码端根据特征域光流确定的中间特征图所带来的解码误差低于通常技术中图像域光流所导致的解码误差，换言之，解码端根据特征域光流对图像帧进行解码，降低了相邻两帧图像之间的图像域光流导致的解码误差，提高了图像解码的准确性。而且，解码端将光流集和参考帧的特征图确定的多个中间特征图进行融合，获得图像帧的预测特征图，该预测特征图包含了比单个图像域光流更多的图像信息，使得解码端基于融合获得的预测特征图对图像帧进行解码时，避免了单个图像域光流指示的信息难以准确表 达第一图像的问题，提高了图像解码的准确性和图像质量(如图像清晰度等)。
在一种可选的实现方式中，解码端基于第一光流集包括的一个或多个特征域光流处理第一特征图，获得第一特征图对应的一个或多个中间特征图，包括：解码端解析码流获得前述参考帧的第一特征图；以及，针对于第一光流集包括的一个或多个特征域光流，解码端基于其中的第一特征域光流对第一特征图进行光流映射(warping)，获得第一特征域光流对应的中间特征图；该第一特征域光流为前述一个或多个特征域光流中的任一个。
在本实施例中，解码端基于一组特征域光流对第一特征图进行warping，继而，在解码端获得参考帧对应的图像后，解码端可在参考帧对应的图像中，根据一组特征域光流对该组特征域光流在参考帧对应的图像中相应位置进行插值，获得第一图像中像素值的预测值，从而获得第一图像，避免了解码端需根据图像域光流对第一图像所有像素值进行预测，减少了解码端进行图像解码所需的计算量，提高了图像解码的效率。
在另一种可选的实现方式中，前述的第一光流集还对应参考帧的第二特征图。本实施例提供的图像解码方法中，在解码端根据第一预测特征图解码第一图像帧获得第一图像之前，该图像解码方法还包括：第一步，解码端基于第一光流集包括的一个或多个特征域光流处理第二特征图，获得第二特征图对应的一个或多个中间特征图。第二步，解码端融合第二特征图对应的一个或多个中间特征图，获得第一图像帧的第二预测特征图。如此，解码端可根据第一预测特征图和第二预测特征图，解码第一图像帧获得第一图像。
在本实施例中，参考帧的多个特征图(或称一组特征图)可对应一个光流集(或称一组特征域光流)，换言之，针对于属于同一组的多个特征图共用一组特征域流，解码端根据特征图对应的一组特征域光流处理该特征图，从而获得该特征图对应的中间特征图，进一步的，解码端将该特征图对应的中间特征图进行融合获得该特征图对应的预测特征图。最后，解码端根据参考帧的所有特征图对应的预测特征图对图像帧进行解码获得目标图像。如此，在图像解码过程中，若参考帧对应多个特征图，解码端可将该多个特征图划分为一个组或多个组，属于同一个组的特征图共用一个光流集，并将特征图对应的中间特征图进行融合获得预测特征图，避免了当参考帧或图像帧具有较多信息时，解码端根据特征图重构图像的精度较低、速度较慢的问题，提高了图像解码的准确性。值得注意的是，属于不同组的特征图的通道数可以不相同。
在另一种可选的实现方式中，解码端融合一个或多个中间特征图，获得第一图像帧的第一预测特征图，包括：解码端获取前述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值；以及，解码端基于一个或多个中间特征图权值处理与该一个或多个权值各自对应的中间特征图，并将所有处理后的中间特征图相加获得第一预测特征图。其中的权值用于指示中间特征图在第一预测特征图中所占的权重。
在本实施例中，针对于第一特征图对应的多个中间特征图，每个中间特征图的权值可以不同，换言之，解码端可根据图像解码的需求为中间特征图设置不同的权值，示例性的，若在一些中间特征图对应的图像较为模糊的情况下，降低这些中间特征图的权值，从而提高第一图像的清晰度。
在另一种可选的实现方式中，解码端融合一个或多个中间特征图，获得第一图像帧的第一预测特征图，包括：将第一特征图对应的一个或多个中间特征图输入特征融合模型，获得第一预测特征图。特征融合模型包括卷积网络层，卷积网络层用于融合中间特征图。
在通常技术中，解码端获取图像帧和参考帧之间的多个图像域光流，并根据多个图像域光流与参考帧获取到多张图像，从而将多张图像进行融合获得图像帧对应的目标图像，因此，解码端解码一帧图像需预测多张图像的像素值，并将多张图像进行融合获得目标图像，导致图像解码所需的计算资源较大，解码端根据图像域光流解码视频的效率较低。
在本实施例中，如在图像解码过程中，由于特征图融合所需的计算资源小于特征图解码所需的计算资源，因此，解码端采用特征融合模型对多个中间特征图进行融合，进而，解码端基于融合多个中间特征图获得的预测特征图对图像帧进行解码，即解码端仅需根据预测特征图对该预测特征图在图像中指示的图像位置的像素值进行预测，无需预测多张图像的所有像素值，减少了图像解码所需的计算资源，提高了图像解码的效率。
在另一种可选的实现方式中，本申请提供的图像解码方法还包括：首先，解码端获取第一图像的特征图。其次，解码端根据第一图像的特征图、第一特征图和第一预测特征图，获得增强特征图。最后，解码端根据增强特征图处理第一图像，获得第二图像。第二图像的清晰度高于第一图像的清晰度。
本实施例中，解码端可将第一图像的特征图、第一特征图和第一预测特征图进行融合，并基于融合获得的增强特征图对第一图像进行视频增强处理，获得清晰度更优的第二图像，从而提高了解码后的图像清晰度及图像显示效果。
在另一种可选的实现方式中，解码端根据增强特征图处理第一图像，获得第二图像，包括：解码端依据增强特征图获得第一图像的增强层图像，并基于增强层图像重构第一图像，获得第二图像。示例性的，该增强层图像可以是指解码端依据参考帧和该增强特征图确定的图像，解码端在第一图像的基础上加入增强层图像的部分或全部信息，获得第二图像；或者，解码端将增强层图像作为第一图像的重构图像，即前述的第二图像。在本示例中，解码端获取到图像在不同阶段的多个特征图，并获得了这多个特征图确定的增强特征图，从而依据该增强特征图对第一图像进行重构和视频增强，提高了解码后的图像清晰度和图像显示效果。
第二方面，提供了一种图像编码方法，该方法应用于视频编解码系统，或支持实现该视频编解码系统实现图像编码方法的物理设备，例如该物理设备为编码端或者视频编码器(encoder)，在一些情况中，该物理设备可包括芯片系统。这里以编码端执行图像编码方法为例进行说明，该图像编码方法包括：第一步，编码端获取第一图像帧的特征图，以及第一图像帧的参考帧的第一特征图。第二步，编码端根据第一图像帧的特征图和第一特征图，获得至少一个光流集；该至少一个光流集包括第一光流集，该第一光流集对应前述第一特征图，且该第一光流集包括一个或多个特征域光流，前述一个或多个特征域光流中的任一个特征域光流用于指示第一图像帧的特征图与第一特征图之间的运动信息。第三步，编码端基于第一光流集包括的一个或多个特征域光流处理第一特征图，获得第一特征图对应的一个或多个中间特征图。第四步，编码端融合前述一个或多个中间特征图，获得第一图像帧的第一预测特征图。第五步，编码端根据第一预测特征图对第一图像帧进行编码获得码流。
可选的，该码流包括：至少一个光流集对应的特征域光流码流，以及第一预测特征图对应的图像区域的残差码流。
在本实施例中，对于单个特征域光流而言，该特征域光流中像素的误差小于图像域光 流中像素的误差，因此，编码端根据特征域光流确定的中间特征图所带来的编码误差低于通常技术中图像域光流所导致的编码误差，换言之，编码端根据特征域光流对图像帧进行编码，降低了相邻两帧图像之间的图像域光流导致的编码误差，提高了图像编码的准确性。而且，编码端基于多个特征域光流处理参考帧的特征图获得多个中间特征图，以及融合该多个中间特征图确定图像帧的预测特征图，即编码端将光流集和参考帧的特征图确定的多个中间特征图进行融合，获得图像帧的预测特征图，该预测特征图包含了更多的图像信息，使得编码端基于融合获得的预测特征图对图像帧进行编码时，避免了单个中间特征图难以准确表达第一图像的问题，提高了图像编码的准确性和图像质量(如图像清晰度等)。
在一种可选的实现方式中，编码端基于第一光流集包括的一个或多个特征域光流处理第一特征图，获得第一特征图对应的一个或多个中间特征图，包括：针对于第一光流集包括的一个或多个特征域光流，基于其中的第一特征域光流对第一特征图进行光流映射，获得第一特征域光流对应的中间特征图。该第一特征图为第一光流集包括的一个或多个特征域光流中任一个。
在本实施例中，如在图像编码过程中，由于特征图融合所需的计算资源小于特征图编码所需的计算资源，因此，编码端采用特征融合模型对多个中间特征图进行融合，进而，编码端基于融合多个中间特征图获得的预测特征图对图像帧进行编码，即编码端仅需根据预测特征图对该预测特征图在图像中指示的图像位置的像素值进行预测，无需预测多张图像的所有像素值，减少了图像编码所需的计算资源，提高了图像编码的效率。
在另一种可选的实现方式中，第一光流集还对应参考帧的第二特征图。在编码端根据第一预测特征图对第一图像帧进行编码获得码流之前，本实施例提供的图像编码方法还包括：第一步，编码端基于第一光流集包括的一个或多个特征域光流处理第二特征图，获得第二特征图对应的一个或多个中间特征图。第二步，编码端融合第二特征图对应的一个或多个中间特征图，获得第一图像帧的第二预测特征图。如此，前述的编码端根据第一预测特征图对第一图像帧进行编码获得码流，可包括：编码端根据第一预测特征图和第二预测特征图，编码第一图像帧获得码流。在本实施例中，参考帧的多个特征图(或称一组特征图)可对应一个光流集(或称一组特征域光流)，换言之，针对于属于同一组的多个特征图共用一组特征域流，编码端根据特征图对应的一组特征域光流处理该特征图，从而获得该特征图对应的中间特征图，进一步的，编码端将该特征图对应的中间特征图进行融合获得该特征图对应的预测特征图。最后，编码端根据参考帧的所有特征图对应的预测特征图对图像帧进行编码获得目标码流。如此，在图像编码过程中，若参考帧对应多个特征图，编码端可将该多个特征图划分为一个组或多个组，属于同一个组的特征图共用一个光流集，并将特征图对应的中间特征图进行融合获得预测特征图，避免了当参考帧或图像帧具有较多信息时，编码端根据特征图编码获得码流的冗余较多、精度较低的问题，提高了图像编码的准确性。值得注意的是，属于不同组的特征图的通道数可以不相同。
在另一种可选的实现方式中，编码端融合一个或多个中间特征图，获得第一图像帧的第一预测特征图，包括：编码端获取前述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值；以及，编码端基于前述一个或多个权值处理与该一个或多个权值各自对应的中间特征图，并将所有处理后的中间特征图相加获得第一预测特征图。该权值用于指示中间特征图在第一预测特征图中所占的权重。在本实施例中，针对于第一 特征图对应的多个中间特征图，每个中间特征图的权值可以不同，换言之，编码端可根据图像编码的需求为中间特征图设置不同的权值，示例性的，若在一些中间特征图对应的图像较为模糊的情况下，降低这些中间特征图的权值，从而提高第一图像的清晰度。
在另一种可选的实现方式中，编码端融合一个或多个中间特征图，获得第一图像帧的第一预测特征图，包括：编码端将第一特征图对应的一个或多个中间特征图输入特征融合模型，获得第一预测特征图。特征融合模型包括卷积网络层，卷积网络层用于融合中间特征图。在图像编码过程中，由于特征图融合所需的计算资源小于特征图编码所需的计算资源，因此，编码端采用特征融合模型对多个中间特征图进行融合，进而，编码端基于融合多个中间特征图获得的预测特征图对图像帧进行编码，减少了图像编码所需的计算资源，提高了图像编码的效率。
第三方面，提供了一种图像解码装置，该图像解码装置可应用于解码端，或者支持实现前述图像解码方法的视频编解码系统。该图像解码装置包括用于执行第一方面或第一方面任一种可能实现方式中的图像解码方法的各个模块。示例的，图像解码装置包括：码流单元、处理单元、融合单元和解码单元。
码流单元，用于解析码流获得至少一个光流集。该至少一个光流集包括第一光流集，该第一光流集对应第一图像帧的参考帧的第一特征图，该第一光流集包括一个或多个特征域光流，前述一个或多个特征域光流中任一个特征域光流用于指示第一图像帧的特征图与第一特征图之间的运动信息。
处理单元，用于基于第一光流集包括的一个或多个特征域光流处理第一特征图，获得第一特征图对应的一个或多个中间特征图。
融合单元，用于融合一个或多个中间特征图，获得第一图像帧的第一预测特征图。
解码单元，用于根据第一预测特征图解码第一图像帧获得第一图像。
有益效果可以参见第一方面中任一方面的描述，此处不再赘述。所述图像解码装置具有实现上述第一方面中任一方面的方法实例中行为的功能。所述功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的模块。
在一种可选的实现方式中，处理单元，具体用于：解析码流获得参考帧的第一特征图，以及，基于第一特征域光流对第一特征图进行光流映射(warping)，获得第一特征域光流对应的中间特征图；该第一特征域光流为前述第一光流集包括的一个或多个特征域光流中的任一个。
在另一种可选的实现方式中，第一光流集还对应参考帧的第二特征图。处理单元，还用于基于第一光流集包括的一个或多个特征域光流处理第二特征图，获得第二特征图对应的一个或多个中间特征图。融合单元，还用于融合第二特征图对应的一个或多个中间特征图，获得第一图像帧的第二预测特征图。解码单元，具体用于根据第一预测特征图和第二预测特征图，解码第一图像帧获得第一图像。
在另一种可选的实现方式中，融合单元，具体用于：获取前述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值，该权值用于指示中间特征图在第一预测特征图中所占的权重。以及，基于一个或多个权值处理与前述一个或多个权值各自对应的中间特征图，并将所有处理后的中间特征图相加获得第一预测特征图。
在另一种可选的实现方式中，融合单元，具体用于：将第一特征图对应的一个或多个中间特征图输入特征融合模型，获得第一预测特征图。特征融合模型包括卷积网络层，卷积网络层用于融合中间特征图。
在另一种可选的实现方式中，图像解码装置还包括：获取单元和增强单元。获取单元，用于获取第一图像的特征图。融合单元，还用于根据第一图像的特征图、第一特征图和第一预测特征图，获得增强特征图。增强单元，用于根据增强特征图处理第一图像，获得第二图像。第二图像的清晰度高于第一图像的清晰度。
在另一种可选的实现方式中，增强单元，具体用于：依据增强特征图获得第一图像的增强层图像。以及，基于增强层图像重构第一图像，获得第二图像。
第四方面，提供了一种图像编码装置，该图像编码装置可应用于编码端，或者支持实现前述图像编码方法的视频编解码系统。该图像编码装置包括用于执行第二方面或第二方面任一种可能实现方式中的图像编码方法的各个模块。示例的，图像编码装置包括：获取单元、处理单元、融合单元和编码单元。
获取单元，用于获取第一图像帧的特征图，以及第一图像帧的参考帧的第一特征图。
处理单元，用于根据第一图像帧的特征图和第一特征图，获得至少一个光流集。该至少一个光流集包括第一光流集，该第一光流集对应第一图像帧的参考帧的第一特征图，该第一光流集包括一个或多个特征域光流，前述一个或多个特征域光流中任一个特征域光流用于指示第一图像帧的特征图与第一特征图之间的运动信息。
处理单元，还用于基于第一光流集包括的一个或多个特征域光流处理第一特征图，获得第一特征图对应的一个或多个中间特征图。
融合单元，用于融合一个或多个中间特征图，获得第一图像帧的第一预测特征图。
编码单元，用于根据第一预测特征图对第一图像帧进行编码获得码流。
可选的，该码流包括：至少一个光流集对应的特征域光流码流，以及第一预测特征图对应的图像区域的残差码流。
有益效果可以参见第二方面中任一方面的描述，此处不再赘述。所述图像编码装置具有实现上述第一方面中任一方面的方法实例中行为的功能。所述功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的模块。
在一种可选的实现方式中，处理单元，具体用于：以及，针对于第一光流集包括的一个或多个特征域光流，基于其中的第一特征域光流对第一特征图进行光流映射(warping)，获得第一特征域光流对应的中间特征图；该第一特征域光流为第一光流集包括的一个或多个特征域光流中任一个。
在另一种可选的实现方式中，第一光流集还对应参考帧的第二特征图。处理单元，还用于基于第一光流集包括的一个或多个特征域光流处理第二特征图，获得第二特征图对应的一个或多个中间特征图。融合单元，还用于融合第二特征图对应的一个或多个中间特征图，获得第一图像帧的第二预测特征图。编码单元，具体用于根据第一预测特征图和第二预测特征图，编码第一图像帧获得码流。
在另一种可选的实现方式中，融合单元，具体用于：获得前述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值，该权值用于指示中间特征图在 第一预测特征图中所占的权重。以及，基于前述一个或多个权值处理与该一个或多个权值各自对应的中间特征图，并将所有处理后的中间特征图相加获得第一预测特征图。
在另一种可选的实现方式中，融合单元，具体用于：将第一特征图对应的一个或多个中间特征图输入特征融合模型，获得第一预测特征图。特征融合模型包括卷积网络层，卷积网络层用于融合中间特征图。
第五方面，提供了一种图像译码的装置，该图像译码的装置包括至少一个处理器和存储器，存储器用于存储程序代码，当处理器调用所述程序代码时，执行第一方面或第一方面任一种可能实现方式中的图像解码方法的操作步骤，示例性的，该图像译码的装置可以是视频编解码系统包括的解码端或视频解码器。
或者，当处理器调用存储器中的程序代码时，执行第二方面或第二方面任一种可能实现方式中的图像编码方法的操作步骤。示例性的，该图像译码的装置可以是视频编解码系统包括的编码端或视频编码器。
第六方面，提供了一种计算机可读存储介质，存储介质中存储有计算机程序或指令，当所述计算机程序或指令被电子设备执行时，执行第一方面或第一方面任一种可能实现方式中的图像解码方法的操作步骤，和/或，执行第二方面或第二方面任一种可能实现方式中的图像编码方法的操作步骤。示例性的，该电子设备是指前述图像译码的装置。
第七方面，提供了另一种计算机可读存储介质，该计算机读存储介质中存储有根据第二方面或第二方面中任一种可能实现方式中的图像编码方法获取的码流。示例性的，该码流可以包括：前述第二方面中至少一个光流集对应的特征域光流码流，以及第一预测特征图对应的图像区域的残差码流。
第八方面，提供了一种视频编解码系统，包括编码端和解码端，该解码端可执行第一方面或第一方面任一种可能实现方式中的图像解码方法的操作步骤，编码端可执行第二方面或第二方面任一种可能实现方式中的图像编码方法的操作步骤。有益效果可以参见第一方面中任一方面的描述，或者，第二方面中任一方面的描述，此处不再赘述
第九方面，提供了一种计算机程序产品，当计算机程序产品在电子设备上运行时，使得电子设备执行如第一方面或第一方面任意一种可能的实现方式中所述方法的操作步骤，和/或，当计算机程序产品在计算机上运行时，使得电子设备执行如第一方面或第一方面任意一种可能的实现方式中所述方法的操作步骤。示例性的，该电子设备是指前述图像译码的装置。
第十方面，提供了一种芯片，包括控制电路和接口电路，接口电路用于接收来自芯片之外的其它设备的信号并传输至处理器，或将来自控制电路的信号发送给芯片之外的其它设备，控制电路通过逻辑电路或执行代码指令用于实现：上述第一方面或第一方面中任一种可能实现方式中方法的操作步骤，和/或，上述第二方面或第二方面中任一种可能实现方式中方法的操作步骤。示例性的，该芯片是指前述图像译码的装置。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
图1为本申请提供的视频编解码系统的示例性框图；
图2为本申请提供的光流和颜色的映射关系图；
图3为本申请提供的光流映射的示意图；
图4为本申请提供的视频压缩系统的架构示意图；
图5为本申请提供的图像编码方法的流程示意图；
图6为本申请提供的特征提取网络的结构示意图；
图7为本申请提供的光流估计网络的结构示意图；
图8为本申请提供的光流映射和特征融合的流程示意图；
图9为本申请提供的图像解码方法的流程示意图一；
图10为本申请提供的特征编码网络和特征解码网络的结构示意图；
图11为本申请提供的图像解码方法的流程示意图二；
图12为本申请提供的图像重构网络的结构示意图；
图13为本申请提供的可变性卷积网络的示意图；
图14为本申请提供的效率比对的示意图；
图15为本申请提供的图像编码装置的结构示意图；
图16为本申请提供的图像解码装置的结构示意图；
图17为本申请提供的图像译码的装置的结构示意图。
具体实施方式
本申请的实施例提供了一种图像解码(编码)方法，包括：解码端(或者编码端)获取到图像帧的一组特征域光流(或称一个光流集)与参考帧的特征图之间的多个中间特征图，并将该多个中间特征图进行融合获得图像帧的预测特征图，从而，解码端(或者编码端)根据该预测特征图对图像帧进行解码(或者编码)，获得图像帧对应的目标图像(或者码流)。以图像解码过程为例，对于单个特征域光流而言，该特征域光流中像素的误差小于图像域光流中像素的误差，因此，解码端根据特征域光流确定的中间特征图所带来的解码误差低于通常技术中图像域光流所导致的解码误差，换言之，解码端根据特征域光流对图像帧进行解码，降低了相邻两帧图像之间的图像域光流导致的解码误差，提高了图像解码的准确性。而且，解码端基于多个特征域光流处理参考帧的特征图获得多个中间特征图，以及融合该多个中间特征图确定图像帧的预测特征图，即解码端将光流集和参考帧的特征图确定的多个中间特征图进行融合，获得图像帧的预测特征图，该预测特征图包含了更多的图像信息，使得解码端基于融合获得的预测特征图对图像帧进行解码时，避免了单个中间特征图难以准确表达第一图像的问题，提高了图像解码的准确性和图像质量(如图像清晰度等)。下面结合实施例对本申请提供的方案进行说明，为了下述各实施例的描述清楚简洁，首先给出相关技术的简要介绍。
图1为本申请提供的视频编解码系统的示例性框图，如本文所使用，术语“视频译码器”一般是指视频编码器和视频解码器两者。在本申请中，术语“视频译码”或“译码”可一般地指代视频编码或视频解码，示例性的，编码端或解码端可统称为图像译码的装置。
如图1中所示，视频编解码系统包含编码端10和解码端20。编码端10产生经编码视频数据。因此，编码端10可被称为视频编码装置。解码端20可对由编码端10所产生的经编码的视频数据(如包含一个或多个图像帧的视频)进行解码。因此，解码端20可被称为视频解码装置。编码端10、解码端20或两个的各种实施方案可包含一或多个处理器以及耦合到所述一个或多个处理器的存储器。所述存储器可包含但不限于随机存取存储器 (random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)、快闪存储器或可用于以可由计算机存取的指令或数据结构的形式存储所要的程序代码的任何其它媒体。
编码端10和解码端20可以包括各种装置，包含桌上型计算机、移动计算装置、笔记型(例如，膝上型)计算机、平板计算机、机顶盒、例如所谓的“智能”电话等电话手持机、电视机、相机、显示装置、数字媒体播放器、视频游戏控制台、车载计算机或其类似者。
解码端20可经由链路30从编码端10接收经编码视频数据。链路30可包括能够将经编码视频数据从编码端10移动到解码端20的一或多个媒体或装置。在一个实例中，链路30可包括使得编码端10能够实时将经编码视频数据直接发射到解码端20的一或多个通信媒体。在此实例中，编码端10可根据通信标准(例如无线通信协议)来调制经编码视频数据，且可将经调制的视频数据发射到解码端20。所述一或多个通信媒体可包含无线和/或有线通信媒体，例如射频(radio frequency，RF)频谱或一或多个物理传输线。所述一或多个通信媒体可形成基于分组的网络的一部分，基于分组的网络例如为局域网、广域网或全球网络(例如，因特网)。所述一或多个通信媒体可包含路由器、交换器、基站或促进从编码端10到解码端20的通信的其它设备。
在另一实例中，可将经编码数据从输出接口140输出到存储装置40。类似地，可通过输入接口240从存储装置40存取经编码数据。存储装置40可包含多种分布式或本地存取的数据存储媒体中的任一者，例如硬盘驱动器、蓝光光盘、数字通用光盘(digital video disc，DVD)、只读光盘(compact disc read-only memory，CD-ROM)、快闪存储器、易失性或非易失性存储器，或用于存储经编码视频数据的任何其它合适的数字存储媒体。
在另一实例中，存储装置40可对应于文件服务器或可保持由编码端10产生的经编码视频的另一中间存储装置。解码端20可经由流式传输或下载从存储装置40存取所存储的视频数据。文件服务器可为任何类型的能够存储经编码的视频数据并且将经编码的视频数据发射到解码端20的服务器。实例文件服务器包含网络服务器(例如，用于网站)、文件传输协议(file transfer protocol，FTP)服务器、网络附接式存储(network attached storage，NAS)装置或本地磁盘驱动器。解码端20可通过任何标准数据连接(包含因特网连接)来存取经编码视频数据。这可包含无线信道(例如，无线保真(wIreless-fidelity，Wi-Fi)连接)、有线连接(例如，数字用户线路(digital subscriber line，DSL)、电缆调制解调器等)，或适合于存取存储在文件服务器上的经编码视频数据的两者的组合。经编码视频数据从存储装置40的传输可为流式传输、下载传输或两者的组合。
本申请提供的图像的解码方法可应用于视频编解码以支持多种多媒体应用，例如空中电视广播、有线电视发射、卫星电视发射、串流视频发射(例如，经由因特网)、用于存储于数据存储媒体上的视频数据的编码、存储在数据存储媒体上的视频数据的解码，或其它应用。在一些实例中，视频编解码系统可用于支持单向或双向视频传输以支持例如视频流式传输、视频增强、视频回放、视频广播和/或视频电话等应用。
图1中所说明的视频编解码系统仅为实例，并且本申请的技术可适用于未必包含编码装置与解码装置之间的任何数据通信的视频译码设置(例如，视频编码或视频解码)。在其它实例中，数据从本地存储器检索、在网络上流式传输等等。视频编码装置可对数据进行 编码并且将数据存储到存储器，和/或视频解码装置可从存储器检索数据并且对数据进行解码。在许多实例中，由并不彼此通信而是仅编码数据到存储器和/或从存储器检索数据且解码数据的装置执行编码和解码。
在图1的实例中，编码端10包含视频源120、视频编码器100和输出接口140。在一些实例中，输出接口140可包含调节器/解调器(调制解调器)和/或发射器。视频源120可包括视频捕获装置(例如，摄像机)、含有先前捕获的视频数据的视频存档、用以从视频内容提供者接收视频数据的视频馈入接口，和/或用于产生视频数据的计算机图形系统，或视频数据的此些来源的组合。
视频编码器100可对来自视频源120的视频数据进行编码。在一些实例中，编码端10经由输出接口140将经编码视频数据直接发射到解码端20。在其它实例中，经编码视频数据还可存储到存储装置40上，供解码端20以后存取来用于解码和/或播放。
在图1的实例中，解码端20包含输入接口240、视频解码器200和显示装置220。在一些实例中，输入接口240包含接收器和/或调制解调器。输入接口240可经由链路30和/或从存储装置40接收经编码视频数据。显示装置220可与解码端20集成或可在解码端20外部。一般来说，显示装置220显示经解码视频数据。显示装置220可包括多种显示装置，例如，液晶显示器(liquid crystal display，LCD)、等离子显示器、有机发光二极管(organic light-emitting diode，OLED)显示器或其它类型的显示装置。
尽管图1中未图示，但在一些方面，视频编码器100和视频解码器200可各自与音频编码器和解码器集成，且可包含适当的多路复用器-多路分用器单元或其它硬件和软件，以处置共同数据流或单独数据流中的音频和视频两者的编码。在一些实例中，如果适用的话，那么解复用器(MUX-DEMUX)单元可符合ITU H.223多路复用器协议，或例如用户数据报协议(user datagram protocol，UDP)等其它协议。
视频编码器100和视频解码器200各自可实施为例如以下各项的多种电路中的任一者：一个或多个微处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application-specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array，FPGA)、离散逻辑、硬件或其任何组合。如果部分地以软件来实施本申请，那么装置可将用于软件的指令存储在合适的非易失性计算机可读存储媒体中，且可使用一或多个处理器在硬件中执行所述指令从而实施本申请技术。前述内容(包含硬件、软件、硬件与软件的组合等)中的任一者可被视为一或多个处理器。视频编码器100和视频解码器200中的每一者可包含在一或多个编码器或解码器中，所述编码器或解码器中的任一者可集成为相应装置中的组合编码器/解码器(编码解码器)的一部分。
本申请可大体上将视频编码器100称为将某些信息“发信号通知”或“发射”到例如视频解码器200的另一装置。术语“发信号通知”或“发射”可大体上指代用以对经压缩视频数据进行解码的语法元素和/或其它数据的传送。此传送可实时或几乎实时地发生。替代地，此通信可经过一段时间后发生，例如可在编码时在经编码码流中将语法元素存储到计算机可读存储媒体时发生，解码装置接着可在所述语法元素存储到此媒体之后的任何时间检索所述语法元素。
视频序列通常包含一系列视频帧或图像。图像群组(group of picture，GOP)示例性地包括一系列、一个或多个视频图像。GOP可在GOP的头信息中、图像中的一者或多者的 头信息中或在别处包含语法数据，语法数据描述包含于GOP中的图像的数目。图像的每一条带可包含描述相应图像的编码模式的条带语法数据。视频编码器100通常对个别视频条带内的视频块进行操作以便编码视频数据。视频块可对应于CU内的解码节点。视频块可具有固定或变化的大小，且可根据指定解码标准而在大小上不同。
在一些可行的实施方式中，视频编码器100可利用预定义扫描次序来扫描经量化变换系数以产生可经熵编码的串行化向量。在其它可行的实施方式中，视频编码器100可执行自适应性扫描。在扫描经量化变换系数以形成一维向量之后，视频编码器100可根据上下文自适应性可变长度解码(context-based adaptive variable-length code，CAVLC)、上下文自适应性二进制算术解码(context-based adaptive binary arithmetic coding，CABAC)、基于语法的上下文自适应性二进制算术解码(syntax-based adaptive binary arithmetic coding，SBAC)、概率区间分割熵(probability interval partitioning entropy，PIPE)解码或其他熵解码方法来熵解码一维向量。视频编码器100也可熵编码与经编码视频数据相关联的语法元素以供视频解码器200用于解码视频数据。
为了执行CABAC，视频编码器100可将上下文模型内的上下文指派给待传输的符号。上下文可与符号的相邻值是否为非零有关。为了执行CAVLC，视频编码器100可选择待传输的符号的可变长度码。可变长度解码(variable-length code，VLC)中的码字可经构建以使得相对较短码对应于可能性较大的符号，而较长码对应于可能性较小的符号。以这个方式，VLC的使用可相对于针对待传输的每一符号使用相等长度码字达成节省码率的目的。基于指派给符号的上下文可以确定CABAC中的概率。
本申请可将视频解码器当前在解码的图像称作当前图像。
在一种示例中，本申请提供的视频编解码系统可应用于视频压缩场景，如人工智能(artificial intelligence，AI)的视频编码/解码模块。
在另一种示例中，本申请提供的视频编解码系统可用于将压缩好的视频文件针对不同业务进行存储，如终端相册的数据(图像或视频)存储、视频监控、华为TM云等。
在又一种示例中，本申请提供的视频编解码系统可用于将压缩好的视频文件进行传输，如华为TM云、视频直播(live)等。示例性的，当视频编解码系统应用于直播(live)技术中时，输出接口140向外(如支持直播技术的服务器集群)发送数据流的过程也可称为推流，服务器集群向输入接口240发送数据流的过程也可以称为分发。
以上几种示例仅为本实施例提供的视频编解码系统可能的应用场景，不应理解为对本申请的限定。
在本申请提供的实施例中，为减少码流的数据量，提高码流在多个设备之间的传输速度，编码端10可采用光流映射的方式对图像帧进行编码和图像压缩。相对应的，解码端20也可采用光流映射的方式对图像帧进行解码和图像重构。
其中，光流表示的是相邻两帧图像中像素的运动速度和运动方向。示例性的，图像域光流表示的是相邻两帧图像中像素之间的运动信息，该运动信息可以包括的相邻两帧图像的像素之间的运动速度和运动方向，特征域光流表示的是相邻两帧图像的特征图之间的运动信息，该运动信息指示了：图像帧的特征图以及该图像帧的相邻图像帧的特征图之间的运动速度和运动方向。示例性的，前述图像帧(如图像帧1)的相邻图像帧(如图像帧2)也可称为该图像帧1的参考帧。
光流在时间维度上有两个方向：方向①，前一帧图像到后一帧图像的光流；方向②，后一帧图像到前一帧图像的光流。在一个方向上的光流通常进行数字化表示，如使用一个三维的数组[2，h，w]表示，“2”表示光流包含2个通道，其中的第一通道表示图像在x方向的偏移方向和大小，第二通道表示图像在y方向的偏移方向和大小。h为图像的高度，w为图像的宽度。在x方向上，正值表示物体向左移动，而负值表示物体向右移动；在y方向上，正值表示物体向上移动，而负值表示物体向下移动。
图2为本申请提供的光流和颜色的映射关系图，图2的(A)中箭头示出了图像中光流的方向，箭头的长度示出了图像中光流的大小；图2中(B)中示出了依据该光流确定的图像的颜色和亮度，示例性的，颜色表示光流的方向，亮度表示光流的大小，如亮度越大，光流对应的值越大。值得注意的是，图2中(B)以灰度为例对图像的颜色进行说明，但不应理解为对本申请的限定，关于光流和图像的映射关系的更多内容可参照通常技术的相关内容，此处不予赘述。
通过参考帧和两帧之间的光流预测当前帧的方法称为光流映射(Warping)，通常表示为其中，为预测帧，xt-1为参考帧，vt为参考帧向预测帧的光流，解码端在已知参考帧和光流，可以推断出当前帧的像素点对应参考帧的位置，根据对应参考帧的位置进行插值就可以得到当前帧的像素值的估计值。
光流映射包括前向映射(forward warping)和后向映射(backward warping)。如图3所示，图3为本申请提供的光流映射的示意图：图3中的A为前向映射，是指解码端根据前一帧图像(图像帧1)和前后帧之间的光流预测后一帧图像(图像帧2)；图3中的B为后向映射，是指解码端根据后一帧图像(图像帧2)和前后帧之间的光流预测前一帧图像(图像帧1)。
下面在图1至图3的基础上，本申请的实施例提供了一种面向特征空间的深度视频压缩框架(framework towards deep video compression in feature space，简称：FVC)，如图4所示，图4为本申请提供的视频压缩系统的架构示意图，该视频压缩系统400包括：运动估计(motion estimation)模块410、光流压缩(motion compression)模块420、运动补偿(motion compensation)模块430、残差压缩(residual compression)模块440、熵编码(entropy coding)模块450，以及多帧特征融合(multi-frame feature fusion)模块460。其中的运动估计模块410、光流压缩模块420和运动补偿模块430可统称为可变运动补偿组件。
图像帧Xt在经过特征提取(feature extraction)后获得图像帧的特征图Ft，参考帧Xt-1在经过特征提取后获得参考帧的特征图Ft-1，参考帧Xt-1可存储在重建帧缓存(decoded frame buffer)，重建帧缓存用于提供多个帧的数据存储空间。
运动估计模块410根据Ft和Ft-1确定图像帧与参考帧之间的特征域光流Ot。
光流压缩模块420对Ot进行压缩获得特征域光流码流O`t。
运动补偿模块430根据参考帧的特征图Ft-1和解码特征域光流码流O`t，做特征预测确定图像帧Xt对应的预测特征图Ft_pre。
特征域残差为Rt＝Ft-Ft_pre，残差压缩模块440根据获得特征域残差Rt输出压缩后的解码残差R`t。该预测特征图Ft_pre和解码残差R`t可用于确定图像帧的初始重建特征F`t_intial。
进一步的，多帧特征融合模块460根据该初始重建特征F`t_intial和多个参考帧对应的重建特征图(如图4所示出的Ft-1_ref、Ft-2_ref和Ft-3_ref)，确定图像帧Xt的最终重建特征图F`t_final。 该最终重建特征图F`t_final经过帧重建(frame reconstruction)获得图像帧Xt对应的重建帧X`t。
熵编码模块450用于将特征域光流Ot和特征域残差Rt中至少一个进行编码，获得二进制码流。
图1至图4仅为本申请提供的示例，在一些示例中，编码端10、解码端20、视频编码器100、视频解码器200以及视频编解码系统可以包括更多或更少的部件或单元，本申请对此不予限定。
下面将结合附图对本实施例提供的图像编码方法的具体实现方式进行详细描述。
如图5所示，图5为本申请提供的图像编码方法的流程示意图，该图像编码方法可应用于图1所示出的视频编解码系统或者图4所示出的视频压缩系统，示例性的，该图像编码方法可由编码端10或者视频编码器100执行，这里以编码端10执行本实施例提供的图像编码方法为例进行说明。如图5所示，本实施例提供的图像编码方法包括以下步骤S510至S560。
S510，编码端获取图像帧1的特征图，以及图像帧1的参考帧的第一特征图。
该图像帧1和参考帧可同属于一个视频包括的GOP。举例来说，视频包括一个或多个图像帧，图像帧1可以为视频中任一个图像帧，该参考帧可以是视频中与图像帧1相邻的图像帧，如参考帧为在图像帧1之前的相邻图像帧，或者，参考帧为在图像帧1之后的相邻图像帧。值得注意的是，在一些情形中，图像帧1也可称为第一图像帧。
在一种可选的示例中，编码端获取特征图的方式可以包括，但不限于：编码端基于神经网络模型来实现。假设视频中每个图像帧的大小为[3，H，W]，即图像帧具有3个通道、高度为H、宽度为W，图像帧对应的特征图大小为[N，H/s，W/s]，其中的s为正整数，这里假设s＝2，N＝64，神经网络模型为特征提取网络，对本实施例提供的图像帧1的特征图和参考帧的第一特征图的特征提取过程进行说明：如图6所示，图6为本申请提供的特征提取网络的结构示意图，该特征提取网络包括1个卷积层(convolutional layer，conv)和3个残差块(residual block，resblock)处理层，卷积层的卷积核为64×5×5/2，resblock处理层的卷积核大小为64×3×3，进而，编码端确定图像帧1对应的特征图为ft，参考帧的第一特征图为ft-1。
需要说明的是，图6仅为本实施例提供的一种特征提取方式的示例，不应理解为对本申请的限定，当图像帧和参考帧的大小为其他时，图6所示出的特征提取网络中各网络层的参数也可不同。在一些可选的实现方式中，编码端还可以采用其他方式来提取图像帧1和参考帧的特征，本申请对此不予限定。
S520，编码端根据图像帧1的特征图和第一特征图，获得至少一个光流集。
该至少一个光流集中的第一光流集对应前述的第一特征图，如该第一光流集可以为图5所示出的光流集1。
该第一光流集可包括一个或多个特征域光流vt，该特征域光流vt用于指示图像帧1(或称第一图像帧)的特征图与第一特征图之间的运动信息。该运动信息可用于指示图像帧1的特征图和第一特征图之间的运动信息和运动方向。
在本申请的实施例中，编码端根据获取光流集的过程实际是一种光流估计过程。作为一种可行的实现方式，该光流估计过程可由图4所示出的运动估计模块410来实现，该运 动估计模块410可由编码端支持实现。
在一些可选的示例中，编码端可利用光流估计网络来确定前述的光流集。举例来说，如图7所示，图7为本申请提供的光流估计网络的结构示意图，该光流估计网络包括上采样网络和下采样网络，其中的上采样网络包括3个网络层1，网络层1依次包括：3个残差块处理层(卷积核为64×3×3)和1个卷积层(卷积核为64×5×5/2)。下采样网络包括3个网络层2，网络层2依次包括：1个卷积层(卷积核为64×5×5/2)和3个残差块处理层(卷积核为64×3×3)。作为一种可选的情形，1个残差块处理层依次包括：1个卷积层(卷积核为64×3×3)、一个激活层和1个卷积层(卷积核为64×3×3)。其中的激活层可以是指线性整流层(rectified linear units layer，ReLU)，或参数化修正线性单元(parametric rectified linear unit，PReLU)等。
值得注意的是，图7仅为本申请实施例提供的光流估计网络的示例，不应理解为对本申请的限定，其中各网络层的卷积核大小、输入网络层的特征图通道数、下采样位置、卷积层数、网络激活层均可调整。在一些较为复杂的场景中，光流估计网络还可以使用更加复杂的网络结构。
S530，编码端对S520获得的光流集进行特征域光流编码，获得特征域光流码流。
该特征域光流码流包括光流集1对应的码流。
在一些情形中，该特征域光流码流可以是二进制文件，或者，该特征域光流码流也可以是符合多媒体传输协议的其他类型文件，不予限定。
S540，编码端基于光流集1包括的特征域光流处理第一特征图，获得第一特征图对应的一个或多个中间特征图。
这里提供一种可行的处理方式，针对于光流集1包括的一个或多个特征域光流，编码端基于其中的第一特征域光流(v1)对第一特征图进行光流映射(warping)，获得第一特征域光流对应的中间特征图。值得注意的是，一个特征域光流与第一特征图warping，会获得一个中间特征图，第一特征图对应的中间特征图数量与光流集1包含的特征域光流的数量一致。如图8所示，图8为本申请提供的光流映射和特征融合的流程示意图，光流集1包含多个特征域光流，如v1至vm，m为正整数。针对于参考帧的第一特征图ft，编码端将光流集1包含的每个特征域光流分别与第一特征图ft进行warping，获得m个中间特征图。
S550，编码端融合一个或多个中间特征图，获得第一图像帧的第一预测特征图。
通常技术中，解码端获取图像帧和参考帧之间的多个图像域光流，并根据多个图像域光流与参考帧获取到多张图像，从而将多张图像进行融合获得图像帧对应的目标图像，因此，解码端解码一帧图像需预测多张图像的像素值，并将多张图像进行融合获得目标图像，导致图像解码所需的计算资源较大，解码端根据图像域光流解码视频的效率较低。
相比之下，在本实施例提供的一种可能的情形中，编码端将前述的一个或多个中间特征图输入至特征融合模型，获得该第一预测特征图。如特征融合模型包括卷积网络层，该卷积网络层用于融合中间特征图。如在图像解码过程中，由于特征图融合所需的计算资源小于特征图解码所需的计算资源，因此，解码端采用特征融合模型对多个中间特征图进行融合，进而，解码端基于融合多个中间特征图获得的预测特征图对图像帧进行解码，即解码端仅需根据预测特征图对该预测特征图在图像中指示的图像位置的像素值进行预测，无需预测多张图像的所有像素值，减少了图像解码所需的计算资源，提高了图像解码的效率。
在本实施例提供的另一种可能的情形中，编码端获取前述一个或多个中间特征图的一个或多个权值，并基于该一个或多个权值处理与该一个或多个权值各自对应的中间特征图，以及将所有处理后的中间特征图相加获得第一预测特征图。该权值用于指示中间特征图在第一预测特征图中所占的权重。
在本实施例中，针对于第一特征图对应的多个中间特征图，每个中间特征图的权值可以不同，换言之，编码端可根据图像编码的需求为中间特征图设置不同的权值，示例性的，若在一些中间特征图对应的图像较为模糊的情况下，降低这些模糊图像所对应的中间特征图的权值，从而提高第一图像的清晰度。
例如，该权值可以是指每个光流对应的mask值，假设光流集1包括9个特征域光流，各特征域光流的权值依次为：如特征域域光流的大小为[2，H/s，W/s]，mask大小为[1，H/s，W/s]。进而，编码端将这9个特征域光流所对应的中间特征图进行融合，获得第一特征图对应的第一预测特征图。
以上两种可能的情形仅为本实施例提供的示例，不应理解为本申请中的特征图融合仅能采用上述两种方式。如图8所示，假设m＝4，针对于参考帧的第一特征图ft，编码端可将光流集1包含的每个特征域光流分别与第一特征图ft进行warping，获得4个中间特征图；进而，编码端将4个中间特征图进行特征融合获得第一预测特征图。
请继续参见图5，本实施例提供的图像编码方法还包括以下步骤S560。
S560，编码端根据第一预测特征图对图像帧1对应的残差进行编码获得残差码流。
该第一预测特征图对应的图像区域的残差码流，以及前述S530确定的特征域光流码流可统称为图像帧1(或称第一图像帧)对应的码流。
可选的，该码流包括：前述光流集对应的特征域光流码流和第一预测特征图对应的图像区域的残差码流。
在本实施例中，对于第一图像帧(如前述的图像帧1)的参考帧，编码端基于该第一图像帧的特征图和参考帧的第一特征图确定一组特征域光流(如前述的光流集1)，并对该参考帧的第一特征图进行处理，获得一个或多个中间特征图；其次，在编码端获得第一特征图对应的一个或多个中间特征图后，将该一个或多个中间特征图进行融合，从而获得第一图像帧对应的第一预测特征图；最后，编码端根据第一预测特征图对第一图像帧进行编码获得码流。
如此，对于单个特征域光流而言，该特征域光流中像素的误差小于图像域光流中像素的误差，因此编码端根据特征域光流确定的中间特征图所带来的编码误差低于通常技术中图像域光流所导致的编码误差，换言之，编码端根据特征域光流对图像帧进行编码，降低了相邻两帧图像之间的图像域光流导致的编码误差，提高了图像编码的准确性。而且，编码端基于多个特征域光流处理参考帧的特征图获得多个中间特征图，以及融合该多个中间特征图确定图像帧的预测特征图，即编码端将光流集和参考帧的特征图确定的多个中间特征图进行融合，获得图像帧的预测特征图，该预测特征图包含了更多的图像信息，使得编码端基于融合获得的预测特征图对图像帧进行编码时，避免了单个中间特征图难以准确表达第一图像的问题，提高了图像编码的准确性和图像质量(如图像清晰度等)。
作为一种可选的实现方式，参考帧可对应多个特征图，如前述的第一特征图，以及第二特征图。示例性的，第一特征图和第二特征图可以是指参考帧包含的多个通道中属于不 同通道的两个特征图。
在本申请的实施例中，参考帧对应的一组特征图可对应一个光流集，如前述的光流集1还可以对应该第二特征图。结合图8所示出的内容，若参考帧的特征图ft-1为第二特征图，则编码端还可基于光流集1包括的特征域光流处理第二特征图，获得第二特征图对应的一个或多个中间特征图(如图8所示出的f`t-1至f`t-m，m为正整数)。此外，编码端还融合第二特征图对应的一个或多个中间特征图，获得第一图像帧的第二预测特征图。
由于参考帧对应了多个特征图，如前述的第一特征图和第二特征图，因此，编码端对第一图像帧进行编码的过程可包括以下内容：编码端根据第一预测特征图和第二预测特征图，编码第一图像帧获得码流。该码流可包括第一图像帧中与第一特征图、第二特征图对应的图像区域的残差码流，以及前述光流集1对应的特征域光流码流。
例如，假设参考帧的每8个特征图共享9个特征域光流，这8个特征图看作一个特征图的分组，在图像编码的过程中，属于同一个分组的8个特征图可以共用该9个特征域光流，从而获得8×9＝72个中间特征图。进一步的，在编码端对中间特征图进行特征融合的过程中，将一个特征图对应的9个中间特征图进行融合获得该一个特征图对应的预测特征图。值得注意的是，在本申请中，几个特征图共享几个特征域多光流在此不加限制，每个特征图分组的特征通道数是否一致也不加限制。
在本实施例中，参考帧的多个特征图(或称：一组特征图，或一个特征图分组等)可对应一个光流集(或称一组特征域光流)，例如，针对于属于同一组的多个特征图共用一组特征域流，编码端根据特征图对应的一组特征域光流处理该特征图，从而获得该特征图对应的中间特征图，进一步的，编码端将该特征图对应的中间特征图进行融合获得该特征图对应的预测特征图。最后，编码端根据参考帧的所有特征图对应的预测特征图对图像帧进行编码获得目标码流。
如此，在图像编码过程中，若参考帧对应多个特征图，编码端可将该多个特征图划分为一个组或多个组，属于同一个组的特征图共用一个光流集，并将特征图对应的中间特征图进行融合获得预测特征图，避免了当参考帧或图像帧具有较多信息时，编码端根据特征图编码获得的码流冗余较多、精度较低的问题，提高了图像编码的准确性。值得注意的是，属于不同组的特征图的通道数可以不相同。
作为一种可选的实现方式，当参考帧对应多个特征图，且这多个特征图对应一组特征域光流时，对于参考帧的不同特征图，编码端可采用不同的特征融合方式对参考帧的特征图所对应的中间特征图进行融合，从而获得参考帧的特征图所对应的预测特征图。示例性的，参考帧对应了特征图1和特征图2，编码端将特征图1对应多个中间特征图输入特征融合模型，从而确定特征图1对应的预测特征图；编码端获取特征图2对应的每个中间特征图的权值，并根据这些权值处理各自对应的中间特征图，获得特征图2对应的预测特征图。在本实施例中，针对于不同的特征融合方式，在图像的编码需求不同时，编码端可为每种特征融合方式输出的预测特征图设置置信度，从而满足用户的不同编码需求。
对应于前述实施例提供的图像编码方法，为对包含残差码流和特征域光流码流的码流进行解码，获得该码流对应的目标图像或视频，本申请的实施例还提供一种图像解码方法，如图9所示，图9为本申请提供的图像解码方法的流程示意图一，该图像解码方法可应用于图1所示出的视频编解码系统或者图4所示出的视频压缩系统，示例性的，该图像编码 方法可由解码端20或者视频解码器200执行，这里以解码端20执行本实施例提供的图像解码方法为例进行说明。
如图9所示，本实施例提供的图像解码方法包括以下步骤S910至S940。
S910，解码端解析码流获得至少一个光流集。
该至少一个光流集包括第一光流集(如图9所示出的光流集1)，该光流集1对应参考帧的第一特征图，且该光流集1包括一个或多个特征域光流，其中该一个或多个特征域光流中任一个特征域光流用于指示第一图像帧的特征图与前述第一特征图之间的运动信息。关于光流集和特征域光流更多的内容可参照前述S510的相关阐述，在此不予赘述。
可选的，参考帧可以是指解码端已解码的与该第一图像帧相邻的图像帧(在第一图像帧之前的图像帧，或者在第一图像帧之后的图像帧)，或者，该参考帧为码流中携带的包含一张图像的完整信息的图像帧。
S920，解码端基于光流集1包括的特征域光流处理第一特征图，获得第一特征图对应的一个或多个中间特征图。
值得注意的是，解码端根据特征域光流处理参考帧的特征图的过程，也称为特征对齐、特征预测、或者特征align，本申请对此不予限定。
对应于S520中编码端基于光流集1处理参考帧的特征图的过程，解码端也可采用和前述S520相同的方式对第一特征图进行处理，获取到第一特征图对应的一个或多个中间特征图。例如，解码端基于一组特征域光流对第一特征图进行warping，如解码端可根据一组特征域光流对应参考帧的位置进行插值，获得第一图像中像素值的预测值，避免了解码端需根据图像域光流对第一图像所有像素值进行预测，减少了解码端进行图像解码所需的计算量，提高了图像解码的效率。
S930，解码端融合S920确定的一个或多个中间特征图，获得第一图像帧的第一预测特征图。
对应于前述编码端执行的S550，解码端可将S920确定的一个或多个中间特征图输入至特征融合模型，获得该第一预测特征图。如特征融合模型包括卷积网络层，该卷积网络层用于融合中间特征图。在图像解码过程中，由于特征图融合所需的计算资源小于特征图解码所需的计算资源，因此，解码端采用特征融合模型对多个中间特征图进行融合，进而，解码端基于融合多个中间特征图获得的预测特征图对图像帧进行解码，减少了图像解码所需的计算资源，提高了图像解码的效率。
或者，解码端还可获取前述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值；以及解码端基于中间特征图的权值处理中间特征图，并将所有处理后的中间特征图相加获得第一预测特征图。其中的权值用于指示中间特征图在第一预测特征图中所占的权重。在本实施例中，针对于第一特征图对应的多个中间特征图，每个中间特征图的权值可以不同，换言之，解码端可根据图像解码的需求为中间特征图设置不同的权值，示例性的，若在一些中间特征图对应的图像较为模糊的情况下，降低这些中间特征图的权值，从而提高第一图像的清晰度。关于中间特征图的权值可参照S550的相关内容，在此不予赘述。
S940，解码端根据S930确定的第一预测特征图对第一图像帧进行解码，获得第一图像。
示例性的，码流中与第一预测特征图对应的残差为r`t，第一预测特征图为f`t，则解码端对第一图像帧进行解码所需的重建特征图f_res＝f`t+r`t，解码端可根据该重建特征图f_res对第一图像帧进行解码获得第一图像。
在本实施例中，对于第一图像帧的参考帧，解码端基于该第一图像帧对应的一组特征域光流(如前述的第一光流集)，对该参考帧的第一特征图进行处理，获得一个或多个中间特征图；其次，在解码端获得第一特征图对应的一个或多个中间特征图后，将该一个或多个中间特征图进行融合，从而获得第一图像帧对应的第一预测特征图；最后，解码端根据该第一预测特征图对第一图像帧进行解码获得第一图像。
如此，对于单个特征域光流而言，该特征域光流中像素的误差小于图像域光流中像素的误差，因此，解码端根据特征域光流确定的中间特征图所带来的解码误差低于通常技术中图像域光流所导致的解码误差，换言之，解码端根据特征域光流对图像帧进行解码，降低了相邻两帧图像之间的图像域光流导致的解码误差，提高了图像解码的准确性。
而且，解码端基于多个特征域光流处理参考帧的特征图获得多个中间特征图，以及融合该多个中间特征图确定图像帧的预测特征图，即解码端将光流集和参考帧的特征图确定的多个中间特征图进行融合，获得图像帧的预测特征图，该预测特征图包含了更多的图像信息，使得解码端基于融合获得的预测特征图对图像帧进行解码时，避免了单个中间特征图难以准确表达第一图像的问题，提高了图像解码的准确性和图像质量(如图像清晰度等)。
可选的，编码端对特征图进行编码可通过特征编码网络实现，解码端对特征图对应的码流进行编码可通过特征解码网络实现。示例性的，针对于编码端所需的特征编码网络和解码端所需的特征解码网络，这里提供了一种可能的实施例，如图10所示，图10为本申请提供的特征编码网络和特征解码网络的结构示意图，特征编码网络包括3个网络层1，网络层1依次包括：3个残差块处理层(卷积核为64×3×3)和1个卷积层(卷积核为64×5×5/2)。
特征解码网络包括3个网络层2，网络层2依次包括：1个卷积层(卷积核为64×5×5/2)和3个残差块处理层(卷积核为64×3×3)。
在一种可选的情形中，1个残差块处理层依次包括：1个卷积层(卷积核为64×3×3)、一个激活层和1个卷积层(卷积核为64×3×3)。其中的激活层可以是指ReLU层或PReLU等。
值得注意的是，图10仅为本申请实施例提供的特征编码网络和特征解码网络的示例，不应理解为对本申请的限定，其中各网络层的卷积核大小、输入网络层的特征图通道数、下采样位置、卷积层数、网络激活层均可调整。在一些较为复杂的场景中，光流估计网络还可以使用更加复杂的网络结构。
在一种可选的实现方式中，参考帧可对应多个特征图，如前述的第一特征图，以及第二特征图。
在本申请的实施例中，参考帧对应的一组特征图可对应一个光流集，如前述的光流集1还可以对应该参考帧的第二特征图。结合图8所示出的内容，若参考帧的特征图ft-1为第二特征图，则解码端还可基于光流集1包括的特征域光流处理第二特征图，获得第二特征图对应的一个或多个中间特征图(如图8所示出的f`t-1至f`t-m，m为正整数)。此外，解码端还融合第二特征图对应的一个或多个中间特征图，获得第一图像帧的第二预测特征图。
由于参考帧对应了多个特征图，如前述的第一特征图和第二特征图，因此，解码端对第一图像帧进行解码的过程可包括以下内容：解码端根据第一预测特征图和第二预测特征图，解码第一图像帧获得第一图像。例如，假设参考帧的每8个特征图共享9个特征域光流，这8个特征图看作一个特征图的分组，在图像解码的过程中，属于同一个分组的8个特征图可以共用该9个特征域光流，从而获得8×9＝72个中间特征图。进一步的，在解码端对中间特征图进行特征融合的过程中，将一个特征图对应的9个中间特征图进行融合获得该一个特征图对应的预测特征图。值得注意的是，属于不同组的特征图的通道数可以不相同。
在本实施例中，参考帧的多个特征图(或称一组特征图)可对应一个光流集(或称一组特征域光流)，换言之，针对于属于同一组的多个特征图共用一组特征域流，解码端根据特征图对应的一组特征域光流处理该特征图，从而获得该特征图对应的中间特征图，进一步的，解码端将该特征图对应的中间特征图进行融合获得该特征图对应的预测特征图。最后，解码端根据参考帧的所有特征图对应的预测特征图对图像帧进行解码获得目标图像。如此，在图像解码过程中，若参考帧对应多个特征图，解码端可将该多个特征图划分为一个组或多个组，属于同一个组的特征图共用一个光流集，并将特征图对应的中间特征图进行融合获得预测特征图，避免了当参考帧或图像帧具有较多信息时，解码端根据特征图重构图像的精度较低问题，提高了图像解码的准确性。
在一些情形中，若第一图像的清晰度未能达到预期的清晰度，导致解码后的第一图像较为模糊，为提高第一图像的清晰度，改善视频显示效果，本申请的实施例还提供了一种视频增强的技术方案，如图11所示，图11为本申请提供的图像解码方法的流程示意图二，图11所示出的图像解码方法可以和前述实施例提供的图像编码方法和图像解码方法相结合，也可以单独实现，这里以解码端执行图11所示出的图像解码方法为例进行说明，如图11所示，本实施例提供的图像解码方法包括以下步骤S1110至S1130。
S1110，解码端获取第一图像的特征图。
示例性的，解码端可根据图6所示出的特征提取网络来获取第一图像的特征图，在此不予赘述。
S1120，解码端根据第一图像的特征图、参考帧的第一特征图和第一预测特征图，获得增强特征图。
示例性的，解码端可通过特征融合模型来对S1120包含的多个特征图进行融合，获得增强特征图。特征融合模型可以是指前述S550或者S930所提供的特征融合模型，也可以是指包含卷积层的其他模型，如该卷积层的卷积核为3×3，本申请对此不予限定。
在一些可能的情形中，解码端还可为第一图像的特征图、参考帧的第一特征图和第一预测特征图设置不同的权值，从而将这多个特征图进行融合获得增强特征图。
在一些可能的示例中，该增强特征图可用于确定第一图像的增强层图像。该增强层图像的视频质量高于第一图像的视频质量，视频质量可以是指或包括图像信噪比(signal-to-noise ratio，SNR)、分辨率(image resolution)和峰值信噪比(Peak signal-to-noise ratio，PSNR)中至少一种。在本文中，针对于一帧图像的视频质量可称为该图像的图像质量。
其中，图像信噪比是指图像的信号均值与背景标准偏差的比值，对于图像，这里的“信号均值”一般是指图像的灰度平均值，背景标准偏差可以用图像的背景信号值的方差来表 示，背景信号值的方差是指噪声功率；对于图像而言，图像信噪比越大，图像的质量越好。分辨率是单帧图像在单位面积内的像素点数，图像的分辨率越高，图像的质量越好。PSNR用于指示图像的主观质量，PSNR越大，图像的质量越好。关于SNR、分辨率和PSNR的更多内容可以参考现有技术的相关描述，此处不予赘述。
S1130，解码端根据增强特征图处理第一图像，获得第二图像。
第二图像和第一图像指示的内容相同，但第二图像的清晰度高于第一图像的清晰度。在一些实施例中，图像的清晰度可以用峰值信噪比(PSNR)来指示，
本实施例中，解码端可将第一图像的特征图、第一特征图和第一预测特征图进行融合，并基于融合获得的增强特征图对第一图像进行视频增强处理，获得清晰度更优的第二图像，从而提高了解码后的图像清晰度及图像显示效果。
可选的，解码端根据增强特征图处理第一图像，获得第二图像，包括：解码端依据增强特征图获得第一图像的增强层图像，并基于增强层图像重构第一图像，获得第二图像。示例性的，该增强层图像可以是指解码端依据参考帧和该增强特征图确定的图像，解码端在第一图像的基础上加入增强层图像的部分或全部信息，获得第二图像；或者，解码端将增强层图像作为第一图像的重构图像，即前述的第二图像。在本示例中，解码端获取到图像在不同阶段的多个特征图，并获得了这多个特征图确定的增强特征图，从而依据该增强特征图对第一图像进行重构和视频增强，提高了解码后的图像清晰度和图像显示效果。
在一种可能的情形中，解码端对第一图像进行重构可以是基于图像重构网络实现的，如图12所示，图12为本申请提供的图像重构网络的结构示意图，该图像重构网络依次包括：3个残差块处理层(卷积核为64×3×3)和1个反卷积层(卷积核为3×5×5/2)，其中的残差块处理层可以包括1个卷积层(卷积核为64×3×3)、一个激活层和1个卷积层(卷积核为64×3×3)。
可以理解的是，图6、图7、图10和图12所示出的各网络是本实施例以图像帧的大小是[3，H，W]为例进行说明的，不应理解为对本申请的限定。例如，若图像帧的大小发生了变化，前述的各网络中卷积层、残差块处理层、反卷积层的参数也会根据视频编解码、视频增强等的需求而改变，本申请对此不予限定。
在本申请的实施例中，编码端/解码端是通过对参考帧的特征图进行光流映射确定的中间特征图集预测特征图，在一些情形中，编码端/解码端也可基于可变形网络(deformable convolutional networks，DCN)对特征图进行处理，DCN是基于卷积实现的，下面是一种卷积的数学表达式：
其中n为卷积核大小，w为卷积核权重，F是输入特征图(input feature map)，p是卷积位置，pk是相对p的位置在卷积核中的枚举值。DCN是基于一个网络学习偏移(offset)，使得卷积核在input feature map的采样点发生偏移，集中于感兴趣的区域(region of interest，ROI)或者目标区域，其数学表达形式为:
另外带有mask的数学表达形式为：
其中Δpk为相对于pk的偏移量，这样卷积采样位置就变成了不规则位置，m(pk)表示位置pk的mask值是对位置pk的惩罚项。如图13所示通过邻域内的点卷积出最中心点的卷积操作：(a)是常见的3x3卷积核的采样方式，(b)是采样可变形卷积加上偏移量之后的采样点的变化，其中(c)、(d)是可变形卷积的特殊形式，如(c)中的多个像素点经过缩放后获得目标图像中相应位置的像素预测值，又如(d)中的多个像素点经过旋转和缩放后获得目标图像中相应位置的像素预测值。
将特征域光流设置为前述表达式中的偏移量Δpk，则本申请实施例提供的图像编码方法、图像解码方法也可由DCN实现。可以理解的，当参考帧对应多个通道不同的特征图时，每个特征图可采用相同或不同的DCN处理方式，如图13所示出的4种可能的DCN处理方式。
针对于通常技术和本方案提供的视频增强的过程，图14为本申请提供的效率比对的示意图，图14提供了两个指标来对本申请提供的技术方案和通常技术进行比较，这两个指标为：PSNR和每像素比特数(bits per pixel，BPP)，BPP是指存储每个像素所用的位数，BPP也用于指示图像的分辨率。
参照图14可知，在多个码点(编码或解码的节点位置)中，通常技术所提供的方案的BPP高于本申请实施例提供的技术方案，即通常技术产生的码流所占用的存储空间更大，传输码流所需的网络带宽更大；但通常技术所提供的方案的PSNR低于本申请实施例提供的技术方案，即本申请实施例提供的技术方案中，码流解码后获得的视频(或图像)的主观质量更优。
举例来说，对于一帧分辨率为1080P(1920×1080)的图像，通常技术方案测试该帧图像的端到端的时间为0.3206秒(second，s)，而本申请实施例提供的技术方案测试该帧图像的端到端的时间为0.2188s。也就是说，在视频编解码过程中，本申请实施例提供的技术方案不仅可以节省码率(由BPP体现)，又保证了质量(由PSNR体现)，还降低了单帧图像的编解码的时延，从而提高了视频编解码的整体效率。
值得注意的是，本申请提供的图像编码方法、图像解码方法不仅可以应用于视频编解码、视频增强、视频压缩等场景，还可应用于视频预测、视频插帧、视频分析等所有需要视频帧间特征融合或特征对齐的视频处理技术领域。
可以理解的是，为了实现上述实施例中功能，编码端和解码端包括了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本申请中所公开的实施例描述的各示例的单元及方法步骤，本申请能够以硬件或硬件和计算机软件相结合的形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用场景和设计约束条件。
图15为本申请提供的图像编码装置的结构示意图，图像编码装置可以用于实现上述方法实施例中编码端或视频编码器的功能，因此也能实现上述图像编码方法实施例所具备的有益效果。在本申请的实施例中，该图像编码装置可以是如图1所示的编码端10或者视频编码器100，也可以是应用于编码端10或者视频编码器100的模块(如芯片)。
如图15所示，图像编码装置1500包括：获取单元1510、处理单元1520、融合单元1530和编码单元1540。图像编码装置1500用于实现上述图5和图8所示的方法实施例中编码端或视频编码器的功能。
当图像编码装置1500用于实现图5所示的方法实施例时，获取单元1510用于执行S510，处理单元1520用于执行S520和S540，融合单元1530用于执行S550，编码单元1540用于执行S530和S560。
当图像编码装置1500用于实现图8所示的方法实施例时，处理单元1520和融合单元1530用于实现光流映射和特征融合的功能。
有关上述获取单元1510、处理单元1520、融合单元1530和编码单元1540更详细的描述可以直接参考图5和图8所示的方法实施例中相关描述直接得到，这里不加赘述。
相应的，本申请的实施例还提供了一种图像解码装置，如图16所示，图16为本申请提供的图像解码装置的结构示意图，图像解码装置可以用于实现上述方法实施例中解码端或视频解码器的功能，因此也能实现上述图像解码方法实施例所具备的有益效果。在本申请的实施例中，该图像解码装置可以是如图1所示的解码端20或者视频解码器200，也可以是应用于解码端20或者视频解码器200的模块(如芯片)。
如图16所示，图像解码装置1600包括：码流单元1610、处理单元1620、融合单元1630和解码单元1640。图像解码装置1600用于实现上述图8和图9所示的方法实施例中解码端或视频解码器的功能。
当图像解码装置1600用于实现图8所示的方法实施例时，处理单元1620和融合单元1630用于实现光流映射和特征融合的功能。
当图像解码装置1600用于实现图9所示的方法实施例时，码流单元1610用于执行S910，处理单元1620用于执行S920，融合单元1630用于执行S930，解码单元1640用于执行S940。
在一些可选的情形中，图像解码装置1600还可以包括增强单元，该增强单元用于根据增强特征图处理第一图像，获得第二图像。在一种可能的具体示例中，增强单元，具体用于：依据增强特征图获得第一图像的增强层图像。以及，基于增强层图像重构第一图像，获得第二图像。
有关上述码流单元1610、处理单元1620、融合单元1630、解码单元1640和增强单元更详细的描述可以直接参考图8和图9所示的方法实施例中相关描述直接得到，这里不加赘述。
图像编码(或图像解码)装置通过软件实现前述附图中任一所示的图像编码(或图像解码)方法时，图像编码(或图像解码)装置及其各个单元也可以为软件模块。通过处理器调用该软件模块实现上述的图像编码(或图像解码)方法。该处理器可以是中央处理单元(central processing unit，CPU)，特定应用集成电路(application-specific integrated circuit，ASIC)实现，或可编程逻辑器件(programmable logic device，PLD)，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)、现场可编程门阵列(field programmable gate array，FPGA)、通用阵列逻辑(generic array logic，GAL)或其任意组合。
有关上述图像编码(或图像解码)装置更详细的描述可以参考前述附图所示的实施例 中相关描述，这里不加赘述。可以理解的，前述附图所示出的图像编码(或图像解码)装置仅为本实施例提供的示例，根据图像编码(或图像解码)过程或者业务的不同，图像编码(或图像解码)装置可包括更多或更少的单元，本申请对此不予限定。
当图像编码(或图像解码)装置通过硬件实现时，该硬件可以通过处理器或芯片实现。芯片包括接口电路和控制电路。接口电路用于接收来自处理器之外的其它设备的数据并传输至控制电路，或将来自控制电路的数据发送给处理器之外的其它设备。
控制电路通过逻辑电路或执行代码指令用于实现上述实施例中任一种可能实现方式的方法。有益效果可以参见上述实施例中任一方面的描述，此处不再赘述。
可以理解的是，本申请的实施例中的处理器可以是CPU、神经处理器(neural processing unit，NPU)或图形处理器(graphic processing unit，GPU)，还可以是其它通用处理器、数字信号处理器(digital signal processor，DSP)、ASIC、FPGA或者其它可编程逻辑器件、晶体管逻辑器件，硬件部件或者其任意组合。通用处理器可以是微处理器，也可以是任何常规的处理器。
本申请的实施例中的方法步骤可以通过硬件的方式来实现，如该硬件为图像译码的装置，如图17所示，图17为本申请提供的图像译码的装置的结构示意图，该图像译码的装置1700包括存储器1710和至少一个处理器1720，该处理器1720可以实现上述实施例提供的图像编码方法和图像解码方法，该存储器1710用于存储上述图像编码方法和图像解码方法对应的软件指令。作为一种可选的实现方式，在硬件实现上，图像译码的装置1700可以是指封装有一个或多个处理器1720的芯片或芯片系统。示例的，当图像译码的装置1700用于实现上述实施例中方法步骤时，图像译码的装置1700包括的处理器1720执行上述方法的步骤及其可能的子步骤。在一种可选的情形中，图像译码的装置1700还可以包括通信接口1730，该通信接口1730可以用于收发数据。例如，通信接口1730用于接收用户的编码请求、解码请求，或者发送码流、接收码流等。
本申请的实施例中，通信接口1730、处理器1720以及存储器1710之间可通过总线1740连接，所述总线1740可以分为地址总线、数据总线、控制总线等。
值得注意的是，图像译码的装置1700还可以执行图15所示出的图像编码装置1500的功能，以及图16所示出的图像解码装置1600的功能，此处不予赘述。
本实施例提供的图像译码的装置1700可以是服务器，个人计算机，或者具有数据处理功能的其他图像译码的装置1700，本申请对此不予限定。例如，该图像译码的装置1700可以是前述的编码端10(或视频编码器100)，或者解码端20(或视频解码器200)。又如，该图像译码的装置1700也可以同时具有前述编码端10和解码端20的功能，如该图像译码的装置1700是指具有视频编解码功能的视频编解码系统(或者视频压缩系统)。
本申请的实施例中的方法步骤也可以由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存 储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于网络设备或终端设备中。当然，处理器和存储介质也可以作为分立组件存在于网络设备或终端设备中。
另外，本申请还提供一种计算机可读存储介质，该计算机可读存储介质中存储有根据前述任一实施例提供的图像编码方法获得的码流。如该计算机可读存储介质可以为，但不限于：RAM、闪存、ROM、PROM、EPROM、EEPROM、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的存储介质。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机程序或指令。在计算机上加载和执行所述计算机程序或指令时，全部或部分地执行本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、网络设备、用户设备或者其它可编程装置。所述计算机程序或指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机程序或指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，数字视频光盘(digital video disc，DVD)；还可以是半导体介质，例如，固态硬盘(solid state drive，SSD)。
在本申请的各个实施例中，如果没有特殊说明以及逻辑冲突，不同的实施例之间的术语和/或描述具有一致性、且可以相互引用，不同的实施例中的技术特征根据其内在的逻辑关系可以组合形成新的实施例。
本申请中，“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B的情况，其中A，B可以是单数或者复数。在本申请的文字描述中，字符“/”，一般表示前后关联对象是一种“或”的关系；在本申请的公式中，字符“/”，表示前后关联对象是一种“相除”的关系。
可以理解的是，在本申请的实施例中涉及的各种数字编号仅为描述方便进行的区分，并不用来限制本申请的实施例的范围。上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定。

Claims (30)

  1. 一种图像解码方法，其特征在于，所述方法包括：
    解析码流获得至少一个光流集；其中，所述至少一个光流集包括第一光流集，所述第一光流集包括一个或多个特征域光流，所述第一光流集对应第一图像帧的参考帧的第一特征图，所述一个或多个特征域光流中的任一个特征域光流用于指示第一图像帧的特征图与所述第一特征图之间的运动信息；
    基于所述一个或多个特征域光流处理所述第一特征图，获得所述第一特征图对应的一个或多个中间特征图；
    融合所述一个或多个中间特征图，获得所述第一图像帧的第一预测特征图；
    根据所述第一预测特征图解码所述第一图像帧获得第一图像。
  2. 根据权利要求1所述的方法，其特征在于，所述基于所述一个或多个特征域光流处理所述第一特征图，获得所述第一特征图对应的一个或多个中间特征图，包括：
    解析所述码流获得所述第一特征图；
    基于第一特征域光流对所述第一特征图进行光流映射warping，获得所述第一特征域光流对应的中间特征图，其中，所述第一特征域光流为所述一个或多个特征域光流中的任一个。
  3. 根据权利要求1或2所述的方法，其特征在于，所述第一光流集还对应所述参考帧的第二特征图；
    在所述根据所述第一预测特征图解码所述第一图像帧获得第一图像之前，所述方法还包括：
    基于所述一个或多个特征域光流处理所述第二特征图，获得所述第二特征图对应的一个或多个中间特征图；
    融合所述第二特征图对应的一个或多个中间特征图，获得所述第一图像帧的第二预测特征图；
    所述根据所述第一预测特征图解码所述第一图像帧获得第一图像，包括：
    根据所述第一预测特征图和所述第二预测特征图，解码所述第一图像帧获得第一图像。
  4. 根据权利要求1至3中任一项所述的方法，其特征在于，所述融合所述一个或多个中间特征图，获得所述第一图像帧的第一预测特征图，包括：
    获取所述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值，所述权值用于指示所述中间特征图在所述第一预测特征图中所占的权重；
    基于所述一个或多个权值处理与所述一个或多个权值各自对应的中间特征图，并将所有处理后的中间特征图相加获得所述第一预测特征图。
  5. 根据权利要求1至3中任一项所述的方法，其特征在于，所述融合所述一个或多个中间特征图，获得所述第一图像帧的第一预测特征图，包括：
    将所述第一特征图对应的一个或多个中间特征图输入特征融合模型，获得第一预测特征图；所述特征融合模型包括卷积网络层，所述卷积网络层用于融合中间特征图。
  6. 根据权利要求1至5中任一项所述的方法，其特征在于，所述方法还包括：
    获取所述第一图像的特征图；
    根据所述第一图像的特征图、所述第一特征图和所述第一预测特征图，获得增强特征 图；
    根据所述增强特征图处理所述第一图像，获得第二图像；所述第二图像的清晰度高于所述第一图像的清晰度。
  7. 根据权利要求6所述的方法，其特征在于，所述根据所述增强特征图处理所述第一图像，获得第二图像，包括：
    依据所述增强特征图获得所述第一图像的增强层图像；
    基于所述增强层图像重构所述第一图像，获得所述第二图像。
  8. 一种图像编码方法，其特征在于，所述方法包括：
    获取第一图像帧的特征图，以及所述第一图像帧的参考帧的第一特征图；
    根据所述第一图像帧的特征图和所述第一特征图，获得至少一个光流集；所述至少一个光流集包括第一光流集，所述第一光流集包括一个或多个特征域光流，所述第一光流集对应所述第一特征图，所述一个或多个特征域光流中的任一个特征域光流用于指示所述第一图像帧的特征图与所述第一特征图之间的运动信息；
    基于所述一个或多个特征域光流处理所述第一特征图，获得所述第一特征图对应的一个或多个中间特征图；
    融合所述一个或多个中间特征图，获得所述第一图像帧的第一预测特征图；
    根据所述第一预测特征图对所述第一图像帧进行编码获得码流。
  9. 根据权利要求8所述的方法，其特征在于，所述基于所述一个或多个特征域光流处理所述第一特征图，获得所述第一特征图对应的一个或多个中间特征图，包括：
    基于第一特征域光流对所述第一特征图进行光流映射warping，获得所述第一特征域光流对应的中间特征图，其中，所述第一特征域光流为所述一个或多个特征域光流中的任一个。
  10. 根据权利要求8或9所述的方法，其特征在于，所述第一光流集还对应所述参考帧的第二特征图；
    在所述根据所述第一预测特征图对所述第一图像帧进行编码获得码流之前，所述方法还包括：
    基于所述一个或多个特征域光流处理所述第二特征图，获得所述第二特征图对应的一个或多个中间特征图；
    融合所述第二特征图对应的一个或多个中间特征图，获得所述第一图像帧的第二预测特征图；
    所述根据所述第一预测特征图对所述第一图像帧进行编码获得码流，包括：
    根据所述第一预测特征图和所述第二预测特征图，编码所述第一图像帧获得码流。
  11. 根据权利要求8至10中任一项所述的方法，其特征在于，所述融合所述一个或多个中间特征图，获得所述第一图像帧的第一预测特征图，包括：
    获取所述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值，所述权值用于指示所述中间特征图在所述第一预测特征图中所占的权重；
    基于所述一个或多个权值处理与所述一个或多个权值各自对应的中间特征图，并将所有处理后的中间特征图相加获得所述第一预测特征图。
  12. 根据权利要求8至11中任一项所述的方法，其特征在于，所述融合所述一个或多 个中间特征图，获得所述第一图像帧的第一预测特征图，包括：
    将所述第一特征图对应的一个或多个中间特征图输入特征融合模型，获得第一预测特征图；所述特征融合模型包括卷积网络层，所述卷积网络层用于融合中间特征图。
  13. 一种图像解码装置，其特征在于，所述装置包括：
    码流单元，用于解析码流获得至少一个光流集；所述至少一个光流集包括第一光流集，所述第一光流集包括一个或多个特征域光流，其中的第一光流集对应第一图像帧的参考帧的第一特征图，所述一个或多个特征域光流中的任一个特征域光流用于指示所述第一图像帧的特征图与所述第一特征图之间的运动信息；
    处理单元，用于基于所述一个或多个特征域光流处理所述第一特征图，获得所述第一特征图对应的一个或多个中间特征图；
    融合单元，用于融合所述一个或多个中间特征图，获得所述第一图像帧的第一预测特征图；
    解码单元，用于根据所述第一预测特征图解码所述第一图像帧获得第一图像。
  14. 根据权利要求13所述的装置，其特征在于，所述处理单元，具体用于：解析所述码流获得所述第一特征图；以及，基于第一特征域光流对所述第一特征图进行光流映射warping，获得所述第一特征域光流对应的中间特征图；所述第一特征域光流为所述一个或多个特征域光流中的任一个。
  15. 根据权利要求13或14所述的装置，其特征在于，所述第一光流集还对应所述参考帧的第二特征图；
    所述处理单元，还用于基于所述一个或多个特征域光流处理所述第二特征图，获得所述第二特征图对应的一个或多个中间特征图；
    所述融合单元，还用于融合所述第二特征图对应的一个或多个中间特征图，获得所述第一图像帧的第二预测特征图；
    所述解码单元，具体用于根据所述第一预测特征图和所述第二预测特征图，解码所述第一图像帧获得第一图像。
  16. 根据权利要求13至15中任一项所述的装置，其特征在于，所述融合单元，具体用于：获取所述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值，所述权值用于指示所述中间特征图在所述第一预测特征图中所占的权重；以及，基于所述一个或多个权值处理与所述一个或多个权值各自对应的中间特征图，并将所有处理后的中间特征图相加获得所述第一预测特征图。
  17. 根据权利要求13至15中任一项所述的装置，其特征在于，所述融合单元，具体用于：将所述第一特征图对应的一个或多个中间特征图输入特征融合模型，获得第一预测特征图；所述特征融合模型包括卷积网络层，所述卷积网络层用于融合中间特征图。
  18. 根据权利要求13至17中任一项所述的装置，其特征在于，所述装置还包括：获取单元和增强单元；
    所述获取单元，用于获取所述第一图像的特征图；
    所述融合单元，还用于根据所述第一图像的特征图、所述第一特征图和所述第一预测特征图，获得增强特征图；
    所述增强单元，用于根据所述增强特征图处理所述第一图像，获得第二图像；所述第 二图像的清晰度高于所述第一图像的清晰度。
  19. 根据权利要求18所述的装置，其特征在于，所述增强单元，具体用于：依据所述增强特征图获得所述第一图像的增强层图像；以及，基于所述增强层图像重构所述第一图像，获得所述第二图像。
  20. 一种图像编码装置，其特征在于，所述装置包括：
    获取单元，用于获取第一图像帧的特征图，以及所述第一图像帧的参考帧的第一特征图；
    处理单元，用于根据所述第一图像帧的特征图和所述第一特征图，获得至少一个光流集；所述至少一个光流集包括第一光流集，所述第一光流集包括一个或多个特征域光流，所述第一光流集对应所述第一特征图，所述一个或多个特征域光流中的任一个特征域光流用于指示所述第一图像帧的特征图与所述第一特征图之间的运动信息；
    所述处理单元，还用于基于所述一个或多个特征域光流处理所述第一特征图，获得所述第一特征图对应的一个或多个中间特征图；
    融合单元，用于融合所述一个或多个中间特征图，获得所述第一图像帧的第一预测特征图；
    编码单元，用于根据所述第一预测特征图对所述第一图像帧进行编码获得码流。
  21. 根据权利要求20所述的装置，其特征在于，所述处理单元，具体用于：基于第一特征域光流对所述第一特征图进行光流映射warping，获得所述第一特征域光流对应的中间特征图，所述第一特征域光流为所述一个或多个特征域光流中的任一个。
  22. 根据权利要求20或21所述的装置，其特征在于，所述第一光流集还对应所述参考帧的第二特征图；
    所述处理单元，还用于基于所述一个或多个特征域光流处理所述第二特征图，获得所述第二特征图对应的一个或多个中间特征图；
    所述融合单元，还用于融合所述第二特征图对应的一个或多个中间特征图，获得所述第一图像帧的第二预测特征图；
    所述编码单元，具体用于根据所述第一预测特征图和所述第二预测特征图，编码所述第一图像帧获得码流。
  23. 根据权利要求20至22中任一项所述的装置，其特征在于，所述融合单元，具体用于：获得所述一个或多个中间特征图的一个或多个权值，其中，一个中间特征图对应一个权值，所述权值用于指示所述中间特征图在所述第一预测特征图中所占的权重；以及，基于所述一个或多个权值处理与所述一个或多个权值各自对应的中间特征图，并将所有处理后的中间特征图相加获得所述第一预测特征图。
  24. 根据权利要求20至22中任一项所述的装置，其特征在于，所述融合单元，具体用于：将所述第一特征图对应的一个或多个中间特征图输入特征融合模型，获得第一预测特征图；所述特征融合模型包括卷积网络层，所述卷积网络层用于融合中间特征图。
  25. 一种图像译码的装置，其特征在于，包括：存储器和处理器；所述存储器用于存储程序代码，所述处理器用于调用所述程序代码实现权利要求1至7中任意一项所述的方法，或者实现权利要求8至12中任意一项所述的方法。
  26. 一种计算机可读存储介质，其特征在于，所述存储介质中存储有计算机程序或指令， 当所述计算机程序或指令被电子设备执行时，实现权利要求1至7中任意一项所述的方法，或者实现权利要求8至12中任意一项所述的方法。
  27. 一种计算机可读存储介质，其特征在于，所述存储介质中存储有根据权利要求8-12任意一项所述的方法获取的码流。
  28. 一种视频编解码系统，其特征在于，包括：编码端和解码端；
    所述编码端用于执行权利要求1至7中任意一项所述的方法；
    所述解码端用于执行权利要求8至12中任意一项所述的方法。
  29. 一种计算机程序产品，其特征在于，当所述计算机程序产品在电子设备上运行时，所述电子设备实现权利要求1至7中任意一项所述的方法，和/或，所述电子设备实现权利要求8至12中任意一项所述的方法。
  30. 一种芯片，其特征在于，包括：控制电路和接口电路；
    所述接口电路用于：接收来自所述芯片之外的其它设备的信号并传输至处理器，或将来自所述控制电路的信号发送给所述芯片之外的其它设备；
    所述控制电路用于：通过逻辑电路或执行代码指令来实现权利要求1至7中任意一项所述的方法，或者实现权利要求8至12中任意一项所述的方法。
PCT/CN2023/071928 2022-04-15 2023-01-12 一种图像解码方法、编码方法及装置 Ceased WO2023197717A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23787366.6A EP4498678A4 (en) 2022-04-15 2023-01-12 IMAGE DECODING METHOD AND APPARATUS, AND IMAGE ENCODING METHOD AND APPARATUS
US18/914,881 US20250037317A1 (en) 2022-04-15 2024-10-14 Image Decoding Method, Image Encoding Method, and Apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210397258.4 2022-04-15
CN202210397258.4A CN116962706B (zh) 2022-04-15 2022-04-15 图像解码方法、编码方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/914,881 Continuation US20250037317A1 (en) 2022-04-15 2024-10-14 Image Decoding Method, Image Encoding Method, and Apparatus

Publications (1)

Publication Number Publication Date
WO2023197717A1 true WO2023197717A1 (zh) 2023-10-19

Family

ID=88328782

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/071928 Ceased WO2023197717A1 (zh) 2022-04-15 2023-01-12 一种图像解码方法、编码方法及装置

Country Status (4)

Country Link
US (1) US20250037317A1 (zh)
EP (1) EP4498678A4 (zh)
CN (1) CN116962706B (zh)
WO (1) WO2023197717A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180098066A1 (en) * 2016-10-05 2018-04-05 Qualcomm Incorporated Systems and methods of switching interpolation filters
CN110913218A (zh) * 2019-11-29 2020-03-24 合肥图鸭信息科技有限公司 一种视频帧预测方法、装置及终端设备
US20210314474A1 (en) * 2020-04-01 2021-10-07 Samsung Electronics Co., Ltd. System and method for motion warping using multi-exposure frames
CN113542651A (zh) * 2021-05-28 2021-10-22 北京迈格威科技有限公司 模型训练方法、视频插帧方法及对应装置
CN113949883A (zh) * 2020-07-16 2022-01-18 武汉Tcl集团工业研究院有限公司 一种双向预测帧的编码方法、解码方法以及编解码系统
CN114339219A (zh) * 2021-12-31 2022-04-12 浙江大华技术股份有限公司 帧间预测方法、装置、编解码方法、编解码器及电子设备

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110163370B (zh) * 2019-05-24 2021-09-17 上海肇观电子科技有限公司 深度神经网络的压缩方法、芯片、电子设备及介质
CN110689558B (zh) * 2019-09-30 2022-07-22 清华大学 多传感器图像增强方法及装置
CN111083501A (zh) * 2019-12-31 2020-04-28 合肥图鸭信息科技有限公司 一种视频帧重构方法、装置及终端设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180098066A1 (en) * 2016-10-05 2018-04-05 Qualcomm Incorporated Systems and methods of switching interpolation filters
CN110913218A (zh) * 2019-11-29 2020-03-24 合肥图鸭信息科技有限公司 一种视频帧预测方法、装置及终端设备
US20210314474A1 (en) * 2020-04-01 2021-10-07 Samsung Electronics Co., Ltd. System and method for motion warping using multi-exposure frames
CN113949883A (zh) * 2020-07-16 2022-01-18 武汉Tcl集团工业研究院有限公司 一种双向预测帧的编码方法、解码方法以及编解码系统
CN113542651A (zh) * 2021-05-28 2021-10-22 北京迈格威科技有限公司 模型训练方法、视频插帧方法及对应装置
CN114339219A (zh) * 2021-12-31 2022-04-12 浙江大华技术股份有限公司 帧间预测方法、装置、编解码方法、编解码器及电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4498678A4

Also Published As

Publication number Publication date
EP4498678A1 (en) 2025-01-29
EP4498678A4 (en) 2025-07-02
US20250037317A1 (en) 2025-01-30
CN116962706A (zh) 2023-10-27
CN116962706B (zh) 2025-12-12

Similar Documents

Publication Publication Date Title
TWI893191B (zh) 圖像編碼方法、圖像解碼方法及相關裝置
JP2012508485A (ja) Ｇｐｕ加速を伴うソフトウエアビデオトランスコーダ
TWI882138B (zh) 圖像編碼方法、圖像解碼方法及相關裝置
KR20060043115A (ko) 베이스 레이어를 이용하는 영상신호의 엔코딩/디코딩 방법및 장치
CN113747242B (zh) 图像处理方法、装置、电子设备及存储介质
US20060072837A1 (en) Mobile imaging application, device architecture, and service platform architecture
CN110545433B (zh) 视频编解码方法和装置及存储介质
CN110572673B (zh) 视频编解码方法和装置、存储介质及电子装置
CN114125448B (zh) 视频编码方法、解码方法及相关装置
CN110662071B (zh) 视频解码方法和装置、存储介质及电子装置
US9681129B2 (en) Scalable video encoding using a hierarchical epitome
CN110636295B (zh) 视频编解码方法和装置、存储介质及电子装置
CN110572677B (zh) 视频编解码方法和装置、存储介质及电子装置
CN110572672B (zh) 视频编解码方法和装置、存储介质及电子装置
CN116962706B (zh) 图像解码方法、编码方法及装置
CN114051140B (zh) 视频编码方法、装置、计算机设备及存储介质
WO2024108931A1 (zh) 视频编解码方法及装置
CN116405665A (zh) 编码方法、装置、设备及存储介质
CN115866297A (zh) 视频处理方法、装置、设备及存储介质
US8982948B2 (en) Video system with quantization matrix coding mechanism and method of operation thereof
CN115334305A (zh) 视频数据传输方法、装置、电子设备和介质
US20250234043A1 (en) Training with corruption for quality propagation in low-delay video coders
US20250233904A1 (en) Discrete cosine hyperprior in neural image coding
US20240244229A1 (en) Systems and methods for predictive coding
WO2025156771A1 (zh) 编解码方法以及相关装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23787366

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202417079056

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2023787366

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2023787366

Country of ref document: EP

Effective date: 20241021

NENP Non-entry into the national phase

Ref country code: DE