WO2022001469A1 - 图像处理方法、装置、电子设备以及存储介质 - Google Patents

图像处理方法、装置、电子设备以及存储介质 Download PDF

Info

Publication number
WO2022001469A1
WO2022001469A1 PCT/CN2021/095054 CN2021095054W WO2022001469A1 WO 2022001469 A1 WO2022001469 A1 WO 2022001469A1 CN 2021095054 W CN2021095054 W CN 2021095054W WO 2022001469 A1 WO2022001469 A1 WO 2022001469A1
Authority
WO
WIPO (PCT)
Prior art keywords
frame
image frame
target image
padding
frames
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/095054
Other languages
English (en)
French (fr)
Inventor
李潇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to JP2022552630A priority Critical patent/JP7471725B2/ja
Priority to EP21834613.8A priority patent/EP4044608B1/en
Publication of WO2022001469A1 publication Critical patent/WO2022001469A1/zh
Priority to US17/728,810 priority patent/US12217462B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
    • H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/156—Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4781—Games

Definitions

  • the present application relates to the field of computer technology, and in particular, to an image processing method, apparatus, electronic device, and storage medium.
  • games can also display corresponding content to users in the form of video playback, and use the cloud to play games.
  • the game does not run on the player's game terminal, but runs on the cloud server, and the cloud server renders the game scene into a video and audio stream, which is transmitted to the player's terminal through the network.
  • the player's terminal does not need to have powerful graphics computing and data processing capabilities, but only needs to have basic streaming media playback capabilities and the ability to obtain player input instructions and send them to the cloud server.
  • Embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium, which can reduce the time delay caused by decoding an image sequence and improve the smoothness of image processing.
  • the embodiment of the present application provides an image processing method, including:
  • the image frame set is input into the cache queue of the decoder frame by frame, and it is determined that the image frame currently positioned at the first position of the cache queue is the target image frame;
  • a padding frame is inserted between the target image frame and the next image frame of the target image frame, so that in the cache queue, except for the target image frame, the rest of the frames are the fill frame;
  • the next image frame of the target image frame is taken as a new target image frame, and a comparison between the target image frame and the target image frame based on the filling frame information is performed.
  • the decoded image frame is processed and the processing result is displayed.
  • an image processing apparatus including:
  • a receiving module configured to receive padding frame information sent by a server and an image frame set of an image sequence to be processed, where the padding frame information is generated by the server according to the image frame set;
  • a determination module for inputting the set of image frames into the cache queue of the decoder frame by frame, and determining that the image frame currently positioned at the top of the cache queue is the target image frame;
  • An inserting module configured to insert a padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information, so that in the cache queue, except for the target image frame, the rest of the The frame is the padding frame;
  • the detection module is configured to use the next image frame of the target image frame as a new target image frame when it is detected that the decoding of the target image frame is completed, and perform a comparison between the target image frame and the target image frame based on the filling frame information. the step of inserting a filling frame between the next frame of image frames of the target image frame, until all the image frames in the image frame set are decoded;
  • the processing module is used to process the decoded image frame and display the processing result.
  • the plug-in module includes:
  • a first extraction unit configured to extract a set of fill frames generated by the server according to the set of image frames from the fill frame information
  • a selection unit used for selecting a filling frame corresponding to the target image frame in the filling frame set to obtain a target filling frame
  • An inserting unit configured to insert a preset number of target padding frames between the target image frame and the next image frame of the target image frame.
  • the insertion unit is specifically used for:
  • a number of target padding frames corresponding to the padding quantity are inserted between the target image frame and the next image frame of the target image frame.
  • an assignment module is also included, and the assignment module includes:
  • a second extraction unit for extracting the frame number of the target image frame
  • the assignment unit is configured to assign a value to the frame sequence number of the filled frame in the cache queue based on the frame sequence number and the position of the filled frame in the cache queue.
  • the assignment unit is specifically used for:
  • the frame sequence number of the filled frame in the cache queue is assigned based on the frame sequence number, the position of the filled frame in the cache queue, and the offset corresponding to the filled frame in the cache queue.
  • the detection module is specifically used for:
  • the next image frame of the target image frame is used as a new target image frame to obtain the current processing object
  • the frame number of the current processing object is used as the benchmark, and the frame number between the target image frame and the next image frame of the target image frame is based on the filling frame information. Insert padding frames between;
  • the frame number of the current processing object is adjusted according to the frame number corresponding to the previous frame of the current processing object, and the frame number of the current processing object is adjusted based on the filling frame information. Inserting a padding frame between the target image frame and the next frame of the target image frame;
  • the step of obtaining the current processing object by taking the next image frame of the target image frame as a new target image frame is performed until all the image frames are decoded.
  • the determining module is specifically used for:
  • the target image frame is stored in the first position of the buffer queue.
  • An embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the above image processing when executing the program steps of the method.
  • Embodiments of the present application further provide a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the image processing method described above are implemented.
  • FIG. 1a is a schematic diagram of a scene of an image processing method provided by an embodiment of the present application.
  • FIG. 1b is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • Fig. 2a is another schematic flowchart of the image processing method provided by the embodiment of the present application.
  • FIG. 2b is a schematic diagram of an image processing system provided by an embodiment of the present application.
  • 2c is a schematic flowchart of generating a padding frame in an image processing method provided by an embodiment of the present application
  • FIG. 2d is a schematic flowchart of processing an image frame set by an image processing method provided by an embodiment of the present application
  • FIG. 3a is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application.
  • FIG. 3b is another schematic structural diagram of an image processing apparatus provided by an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of the distributed system 100 provided in an embodiment of the present application applied to a blockchain system;
  • FIG. 6 is a schematic diagram of a block structure (Block Structure) provided by an embodiment of the present application.
  • the video hardware decoding frame buffer will greatly affect the overall delay, resulting in the final playback of the video may be stuck.
  • Embodiments of the present application provide an image processing method, apparatus, electronic device, and storage medium, which can reduce the time delay caused by decoding an image sequence and improve the smoothness of image processing.
  • the padding frame information is generated by the server according to the image frame set, and the image frame set is input frame by frame. to the cache queue of the decoder, and determine that the image frame currently located at the top of the cache queue is the target image frame, and then, based on the filling frame information, insert the image frame between the target image frame and the next frame of the target image frame.
  • the image processing device may be specifically integrated in a terminal, and the terminal may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto.
  • the terminal and the server may be directly or indirectly connected through wired or wireless communication, which is not limited in this application.
  • the image processing apparatus is integrated on the terminal, and the terminal can receive the filling frame information sent by the server and the image frame set of the image sequence to be processed, wherein the filling frame information is generated by the server according to the image frame set.
  • the server is a cloud server.
  • the cloud server renders the game scene into a video and audio stream (ie, video stream data), and transmits it to the user's terminal through the network. Then, the terminal inputs the image frame set to the decoding frame by frame.
  • the terminal In the cache queue of the controller, and determine that the image frame currently at the head of the cache queue is the target image frame, then, the terminal inserts the padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information, so that the Except for the target image frame in the cache queue, the rest of the frames are filled frames.
  • the terminal detects that the decoding of the target image frame is completed, the next frame of the target image frame is used as a new target image frame, and Execute the step of inserting a filling frame between the target image frame and the next image frame of the target image frame based on the filling frame information, until all the image frames in the image frame set are decoded, and finally, the terminal renders the decoded image frame to The sequence of images to be processed is played.
  • a padding frame is inserted between the target image frame and the next image frame of the target image frame, so that when the target image frame is located in the cache queue, it is at the first position of the cache queue, and in the cache queue Except for the target image frame, the rest of the frames are filler frames, which improves the decoding speed of the image frame by the terminal, reduces the time delay caused by decoding the video, and further improves the efficiency of image processing.
  • An image processing method comprising: receiving filling frame information sent by a server and an image frame set of an image sequence to be processed, inputting the image frame set into a buffer queue of a decoder frame by frame, and determining the image frame currently at the top of the buffer queue. For the target image frame, insert a padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information. When it is detected that the decoding of the target image frame is completed, the next image frame of the target image frame is used as A new target image frame is created, and the step of inserting a filling frame between the target image frame and the next image frame of the target image frame based on the filling frame information is performed until all the image frames in the image frame set are decoded. The image frames are processed and the processing results are displayed.
  • FIG. 1b is a schematic flowchart of an image processing method provided by the present application.
  • the image processing method can be executed by a terminal, and the specific process can be as follows:
  • the padding frame information is generated by the server according to the image frame set, and the terminal can receive the video stream data sent by the server through a wired network or wireless network, and the video stream data carries the padding frame information and the image frame set of the image sequence to be processed, After receiving the video stream data sent by the server, the video stream data is parsed to obtain the image frame set of the image sequence to be processed and the filling frame information generated by the server according to the image frame set.
  • the format of the video data source can be AVC (Advanced Video Coding, that is, H.264 encoding standard), or HEVC (High Efficiency Video Coding, that is, H.265 encoding standard).
  • the so-called video coding method refers to compressing the original video image into a binary byte stream through prediction, change, quantization, recombination and entropy coding in sequence through compression technology.
  • the coding method includes: based on context Adaptive Variable Length Coding (Context-Adaptive Variable-Length Coding, CAVLC) and Context-based Adaptive Binary Arithmetic Coding (CABAC), the server obtains the image sequence header of the image sequence to be processed. After the information, the image sequence header information is parsed to obtain the image quality level of the image sequence to be processed, the number of reference frames, whether there are bidirectional prediction frames and the corresponding length of the image frame sequence, etc. To generate a padding frame, when the server generates a padding frame, it will package the generated padding frame, the sequence number and offset corresponding to the padding frame into padding frame information, and write it into the image sequence header information.
  • the picture sequence header information is sent to the terminal.
  • the volume of the video content usually needs to be reduced, that is, the original video image needs to be compressed, and the compression algorithm is also referred to as the encoding format.
  • the server can use the H.264 encoding format to compress the video image.
  • the compressed video content needs to be decompressed, also called decoding.
  • the encoder encodes multiple images and produces a group of pictures (GOPs), and the decoder reads the GOPs to decode them, and then reads the images and then renders them for display.
  • GOP Group of Pictures
  • GOP is a group of continuous pictures, which consists of a key frame and multiple non-key frames.
  • the key frame is the basic frame (the first frame) of the GOP, and there is only one key frame in a group.
  • the key frame includes the forward reference frame and the bidirectional reference frame, where the key frame is a complete picture, and the forward reference frame and the bidirectional reference frame record the changes relative to the key frame.
  • the reference frame needs to rely on the previous frame of image for decoding, and the bidirectional reference frame needs to rely not only on the previous frame of image, but also on the next frame of image for decoding, that is, when the video needs to be played, it is necessary to locate a key frame first.
  • the position of the video can be played, that is, in some embodiments, the step "input the image frame set into the cache queue of the decoder frame by frame, and determine the image frame currently at the top of the cache queue as the target image frame” , which can include:
  • the user's terminal can cache video data (ie, image frame sets) between the user's terminal and the server through a cache queue, that is, the cache queue is used for each image frame to be decoded in the image processing process. and the terminal decodes the first image frame in the buffer queue only when all buffer spaces in the buffer queue are buffered with image frames.
  • video data ie, image frame sets
  • a padding frame is inserted between the target image frame and the next image frame of the target image frame, so that the remaining frames in the cache queue are padding frames except the target image frame.
  • cloud Cloud gaming also known as gaming on demand
  • the set of image frames sent by the server generally only includes key frame and forward reference frame, because the bidirectional reference frame records the difference between the current frame and the previous frame, that is to say, to decode the bidirectional reference frame, not only the cached picture before the bidirectional reference frame is obtained, but also after the bidirectional reference frame is decoded.
  • Picture the final picture is obtained by superimposing the front and rear pictures with the data of this frame.
  • the bidirectional reference frame When the bidirectional reference frame needs to be decoded, it is also necessary to obtain the data of the next frame of the bidirectional reference frame. At this time, the next frame of the bidirectional reference frame is still in the cache. In the queue, therefore, the bidirectional reference frame cannot be decoded, causing the picture to freeze and other phenomena.
  • the padding frame information includes the padding frame corresponding to each image frame in the image frame set. Therefore, the padding frame corresponding to the target image frame can be determined according to the padding frame information, and the determined padding frame can be inserted into the gap between the target image frame and the target image frame. Between the next image frames, so that the target image frame is at the head of the cache queue, and the other frames in the cache queue are filled frames except the target image frame, that is, in some embodiments, the step "based on”
  • the padding frame information inserts a padding frame between the target image frame and the next image frame of the target image frame", which may specifically include:
  • the padding frame is a forward prediction frame
  • the frame mode of the padding frame is skip mode.
  • the P-Skip macroblock is a special P macroblock.
  • Both the pixel residual and the motion vector residual are written into the code stream and transmitted from the encoding end to the decoding end, but the special feature of the P-Skip macroblock is that neither the pixel residual nor the motion vector residual is transmitted.
  • the encoding end does not need to transmit other information about the macroblock.
  • the motion vector residual is equal to the difference between the motion vector and the predicted motion vector, in the encoding segment
  • the motion vector residual is zero, and the predicted motion vector can be obtained during decoding, that is to say, during decoding, the motion vector can also be obtained.
  • the decoding end there are reconstructed pixels of the macroblock corresponding to the reference frame. According to the reconstructed pixels and The motion vector can restore the pixel value of the macroblock in this frame. This is the so-called P-skip macroblock principle. Literally, this macroblock is skipped, which is equivalent to this macroblock is not encoded, and is taken at the decoding end.
  • the pixel value of this macroblock in this frame can be directly recovered by an approximate substitution method, for example, there is a ping-pong ball in the first frame, and there is also a ping-pong ball in the second frame, the ping-pong ball in the second frame is this Macroblocks are likely to be encoded into PSkip macroblocks.
  • the filler frame inserted into the buffer queue from the perspective before encoding is exactly the same as the target image frame.
  • a padding frame is inserted between the target image frame and the next image frame of the target image frame, so that the buffer queue is filled (up to the maximum buffer number of the buffer queue), you can According to the maximum buffer number of the cache queue and the target image frame, the padding quantity is generated, and then, the target padding frame corresponding to the padding quantity is inserted between the target image frame and the next image frame of the target image frame, that is, in some implementations
  • the step "insert a preset number of target padding frames between the target image frame and the next image frame of the target image frame" may specifically include:
  • the frame number of the target image frame is 5, and 4 padding frames are inserted into the cache queue, which are padding frame A, padding frame B, padding frame C, and padding frame D, and the positions between padding frames are padding frames.
  • A-filling B-filling frame C-filling frame D, the corresponding frame numbers of the stuffing frame A, stuffing frame B, stuffing frame C and stuffing frame D are assigned in an increasing order.
  • the frame serial number of stuffing frame A is 6.
  • the frame serial number of the stuffing frame B is 7, the frame serial number of the stuffing frame C is 8, and the frame serial number of the stuffing frame D is 9.
  • an H.264 slice refers to an integer number of macroblocks or macroblock pairs arranged in a raster scan order within a specific slice group, but these macroblocks or macroblock pairs It is not necessarily consecutively arranged in the raster scan order within the image, and the addresses of the macroblocks are obtained through the address of the first macroblock of the slice (described in the slice header) and the mapping from macroblocks to slice groups.
  • the H.264 slice header refers to a part of the coding slice, which contains data elements related to the first or all macroblocks in the slice, that is, the information corresponding to the filling frame in the buffer queue can be extracted from the filling frame information.
  • Offset which refers to the offset of the padding frame in the slice header, and then, based on the frame sequence number, the position of the padding frame in the buffer queue and the offset corresponding to the padding frame in the buffer queue, for all
  • the frame sequence number of the filled frame in the cache queue is assigned, that is, in some embodiments, the step "based on the frame sequence number and the position of the filled frame in the cache queue, assigns the frame sequence number of the filled frame in the cache queue, so that in the The frame sequence number of the last frame filled frame in the cache queue is the second frame sequence number", which may include:
  • the frame sequence number of the padding frame generated by the server is extracted from the padding frame information, and according to the frame sequence number, the position of the padding frame in the cache queue, and the offset corresponding to the padding frame in the buffer queue, the buffer queue is filled with The frame serial number of the frame is updated to the corresponding frame serial number, so as to ensure the continuity of the image sequence to be processed after the filling frame is inserted into the buffer queue.
  • the next image frame of the target image frame is taken as the new target image frame, and the image frame between the target image frame and the next image frame of the target image frame is executed based on the filling frame information.
  • the step of inserting padding frames between the image frames until all image frames in the image frame set are decoded.
  • the next image frame of the target image frame is taken as the new target image frame, that is, the next image frame of the target image frame is taken as the currently processed image. frame, and then, insert a padding frame between the currently processed image frame and the next image frame of the currently processed image frame, that is, perform the interpolation between the target image frame and the next image frame of the target image frame based on the padding frame information.
  • the step of inserting padding frames between the image frames until all image frames in the image frame set are decoded.
  • the frame number of the current processing object is adjusted according to the frame number corresponding to the previous frame of the current processing object, and the current processing object and the current processing object are based on the filling frame information. Insert a padding frame between the next frame of image frames;
  • the frame sequence number of the current processing object is determined as the starting sequence number.
  • the frame number of the frame is assigned; if the current processing object is not a key frame, the frame number corresponding to the previous frame of the current processing object is obtained, where the previous frame of the current processing object is the filling frame, that is, according to the previous frame
  • the frame number of the filled frame adjusts the frame number of the current processing object, and assigns the frame number of the filled frame in the cache queue based on the adjusted frame number of the current processing object and the position of the filled frame in the cache queue.
  • the current processing object is not a key frame, based on the adjusted frame number of the current processing object and the position of the filled frame in the cache queue, after assigning the frame number of the filled frame in the cache queue, it is also necessary to detect the assigned filled frame. Whether the corresponding frame sequence number is greater than the preset threshold, if so, the frame sequence number greater than the preset threshold will be reset to zero.
  • the decoded image frame can be rendered to play the image sequence to be processed.
  • the filling frame is a P-skip frame
  • the P-skip frame is only decoded without rendering, and the Each frame of video is decoded, rendered, and played, ultimately reducing the delay caused by decoding the video and improving the smoothness of image processing.
  • the image frame set is input into the cache queue of the decoder frame by frame, and the image frame currently at the top of the cache queue is determined as the target image frame. , and then insert a padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information, so that in addition to the target image frame in the cache queue, the rest of the frames are padding frames.
  • the image frame decoding When the target image frame is detected When the image frame decoding is completed, take the next image frame of the target image frame as a new target image frame, and perform the step of inserting a filling frame between the target image frame and the next image frame of the target image frame based on the filling frame information , until all the image frames in the image frame set are decoded, and finally, the decoded image frames are processed and the processing results are displayed.
  • a padding frame is inserted between the target image frame and the next image frame of the target image frame, so that when the target image frame is located in the cache queue, it is at the first position of the cache queue, and in the cache queue Except for the target image frame, the rest of the frames are filler frames, which improves the decoding speed of the image frame by the terminal, reduces the time delay caused by decoding the video, and further improves the efficiency of image processing.
  • the image processing apparatus is specifically integrated in a terminal as an example for description.
  • the terminal receives the padding frame information and the image frame set of the to-be-processed image sequence sent by the server.
  • the terminal inputs the set of image frames frame by frame into the buffer queue of the decoder, and determines the image frame currently at the head of the buffer queue as the target image frame.
  • the terminal When the terminal receives the compressed video content sent by the server, the terminal needs to decompress the compressed video content. That is, when the terminal needs to play the video, it first needs to locate the position of a key frame before the video can be played. .
  • the terminal inserts a padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information.
  • the padding frame information includes the padding frame corresponding to each image frame in the image frame set. Therefore, the padding frame corresponding to the target image frame can be determined according to the padding frame information, and the determined padding frame can be inserted into the gap between the target image frame and the target image frame. between the next image frames, so that the target image frame is located at the head of the buffer queue, and the other frames in the buffer queue except the target image frame are filled frames.
  • the terminal detects that the decoding of the target image frame is completed, the terminal takes the next image frame of the target image frame as a new target image frame, and executes the next frame of the target image frame and the next image frame of the target image frame based on the filling frame information. The steps of inserting padding frames in between until all the image frames in the image frame set have been decoded.
  • the frame sequence number of the current processing object is determined as the starting sequence number.
  • the frame number of the filling frame is assigned; if the current processing object is not a key frame, the terminal obtains the frame number corresponding to the previous frame of the current processing object, wherein the previous frame of the current processing object is the filling frame, that is, the terminal according to The frame number of the filled frame of the previous frame adjusts the frame number of the current processing object, and assigns the frame number of the filled frame in the cache queue based on the adjusted frame number of the current processing object and the position of the filled frame in the cache queue, which needs to be explained.
  • the terminal needs to detect the assignment after assigning the frame number of the filled frame in the cache queue based on the frame number adjusted by the current processing object and the position of the filled frame in the cache queue. Whether the frame sequence number corresponding to the subsequent padding frame is greater than the preset threshold, and if so, the terminal will perform zero reset processing on the frame sequence number greater than the preset threshold.
  • the terminal processes the decoded image frame, and displays the processing result
  • the decoded image frame is rendered to play the image sequence to be processed.
  • the filling frame is a P-skip frame
  • the P-skip frame is only decoded but not rendered, and each frame of video in the image frame set is decoded, rendered and played, and finally the decoding of the video is reduced. The delay brought about by the purpose of improving the fluency of image processing.
  • the terminal of the present application After the terminal of the present application receives the fill frame information sent by the server and the image frame set of the image sequence to be processed, the terminal inputs the image frame set into the buffer queue of the decoder frame by frame, and determines that the image frame currently at the top of the buffer queue is the target image frame, and then, the terminal inserts a padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information, so that the remaining frames except the target image frame in the cache queue are padding frames, When the terminal detects that the decoding of the target image frame is completed, it takes the next image frame of the target image frame as a new target image frame, and executes the process between the target image frame and the next image frame of the target image frame based on the filling frame information.
  • the step of inserting filling frames is performed until all image frames are decoded.
  • the terminal processes the decoded image frames and displays the processing results.
  • the terminal provided by the present application inserts a padding frame between the target image frame and the next image frame of the target image frame, so that when the target image frame is located in the buffer queue, it is at the first position of the buffer queue, and the target image frame is removed from the buffer queue.
  • the rest of the frames are filled frames, which improves the decoding speed of the image frame by the terminal, reduces the delay caused by decoding the video, and further improves the efficiency of image processing.
  • the present application provides an image processing system (hereinafter referred to as a processing system), the processing system includes: a first terminal 10, a second terminal 20, and a server 30.
  • the terminal 10 After collecting the to-be-processed image sequence encoding to generate the H.264 video stream, send the image sequence header information of the video to be played to the server 30, wherein the image sequence header information includes the sequence parameter set and image parameter set of the video to be played, and the sequence parameter A set of global parameters of an encoded video sequence is centrally stored.
  • the so-called encoded video sequence is a sequence composed of the encoded pixel data of one frame and one frame of the original video, and the parameters on which the encoded data of each frame depends are stored.
  • the server 30 parses the image sequence header information to obtain the basic format information of the image sequence to be processed, including the number of reference frames, the video level, the maximum frame sequence, and the frame sequence number. The offset in the corresponding strip and the number of bidirectional reference frame image frames, and based on the analysis result, it is determined whether the to-be-processed image sequence can generate a padding frame. Specifically, please refer to FIG.
  • the padding frame generation module corresponding to the CAVLC mode is selected, and the entropy coding mode is CABAC mode, then Select the padding frame generation module corresponding to the CABAC mode; then, based on the selected padding frame generation module, specify the encoder to generate a forward reference frame without any motion vector estimation, and all macroblock prediction modes are forced to P-skip mode (skip mode). Pass mode), generate the corresponding padding frame by specifying the entropy coding mode (CAVLC/CABAC), and finally, the padding frame information packaging module will generate the padding frame, the frame sequence number in the parsing result, and the frame sequence number in the corresponding strip.
  • the offset is encapsulated to obtain the padding frame information, and the server 30 sends the image sequence header information and the padding frame information to the second terminal 20 .
  • the second terminal 20 receives the image sequence header information and the padding frame information sent by the server 30, and the second terminal 20 parses the image sequence header information to obtain the image frame set of the image sequence to be processed.
  • the second terminal 20 inserts the image frame set into the decoding module of the second terminal 20 frame by frame, if there is a buffer queue during decoding, the second terminal 20 extracts the filling frame generated by the server 30 from the filling frame information.
  • the second terminal 20 determines, according to the set of image frames, that the image frame currently at the head of the cache queue is the target image frame, and inserts a filling frame between the target image frame and the next image frame of the target image frame based on the filling frame information, and then , the second terminal 20 respectively updates the frame sequence numbers of the filled frames and the target image frames in the cache queue according to the offset in the filled frame information, so as to ensure the continuity of the images of the image sequence to be processed, please refer to the previous embodiment for details,
  • the filling frame generation module in the server 30 sends the image sequence header information to the relay proxy server, so that the second terminal 20 receives the image sent by the relay proxy server.
  • the sequence header information shows the image sequence to be
  • the present application further provides an image processing apparatus based on the above method.
  • the meanings of the nouns are the same as those in the above image processing method, and the specific implementation details can refer to the description in the method embodiment.
  • FIG. 3a is a schematic structural diagram of an image processing apparatus provided by the present application, wherein the image processing apparatus may include a receiving module 301, a determining module 302, an inserting module 303, a detecting module 304, and a rendering module 305, which may be specifically as follows:
  • the receiving module 301 is configured to receive the padding frame information sent by the server and the image frame set of the image sequence to be processed.
  • the padding frame information is generated by the server according to the image frame set, and the receiving module 301 can receive the video stream data sent by the server through a wired network or wireless network, and the video stream data carries the padding frame information and the image frame set of the image sequence to be processed. , after receiving the video stream data sent by the server, analyze the video stream data, thereby obtaining the image frame set of the image sequence to be processed and the filling frame information generated by the server according to the image frame set.
  • the format of the video data source can be AVC (Advanced Video Coding, namely H.264 encoding standard), can also be HEVC (High Efficiency Video Coding, namely H.265 encoding standard).
  • the determining module 302 is configured to input the image frame set into the buffer queue of the decoder frame by frame, and determine the image frame currently located at the head of the buffer queue as the target image frame.
  • the volume of the video content usually needs to be reduced, that is, the original video image needs to be compressed, and the compression algorithm is also referred to as the encoding format.
  • the H.264 encoding format can be used to compress the video image, and when the determining module 302 receives the compressed video content sent by the server, the terminal needs to decompress the compressed video content, that is, when the determining module 302 needs to play
  • the terminal when creating a video, it is first necessary to locate the position of a key frame before the video can be played.
  • the determining module 302 may be specifically configured to: obtain the key frame of the currently rendered image sequence to be processed, obtain the target key frame, and determine the next image frame of the target key frame in the image frame set as the target image frame, Store the target image frame at the head of the buffer queue.
  • the inserting module 303 is configured to insert a padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information.
  • the padding frame information includes a padding frame corresponding to each image frame in the image frame set. Therefore, the inserting module 303 can determine a padding frame corresponding to the target image frame according to the padding frame information, and insert the determined padding frame into the target image frame and the target image frame. between the next image frame of the image frame, so that the target image frame is at the head of the buffer queue, and the other frames in the buffer queue except the target image frame are filled frames.
  • the plug-in module 303 may specifically include:
  • the first extraction unit is used for extracting the filling frame set generated by the server according to the image frame set from the filling frame information
  • the selection unit is used to select the filling frame corresponding to the target image frame in the filling frame set to obtain the target filling frame;
  • the inserting unit is used for inserting a preset number of target filling frames between the target image frame and the next image frame of the target image frame.
  • the inserting unit may be specifically configured to: detect the maximum buffer quantity corresponding to the buffer queue, generate the filling quantity according to the target image frame and the maximum buffer quantity, and generate a filling quantity between the target image frame and the next image frame of the target image frame. Interpolate a number of target padding frames corresponding to the padding amount.
  • the image processing apparatus may further include an assignment module 306, and the assignment module 306 includes:
  • the second extraction unit is used to extract the frame sequence number of the target image frame
  • the assignment unit is used to assign the frame sequence number of the filled frame in the cache queue based on the frame sequence number and the position of the filled frame in the cache queue.
  • the assignment unit may be specifically configured to: extract the offset corresponding to the filled frame in the buffer queue from the fill frame information, and fill the frame in the buffer queue based on the frame sequence number, the position of the filled frame in the buffer queue, and the buffer queue.
  • the corresponding offset is assigned to the frame number of the filled frame in the buffer queue.
  • the detection module 304 is configured to use the next frame of the target image frame as a new target image frame when detecting that the decoding of the target image frame is completed, and perform the next step between the target image frame and the target image frame based on the filling frame information. The step of inserting stuffing frames between the frame image frames until all the image frames in the image frame set are decoded.
  • the frame sequence number of the current processing object is determined as the starting sequence number, and the detection module 304 is based on the frame sequence number of the current processing object and the position of the filled frame in the cache queue.
  • the frame number of the filling frame in the queue is assigned; if the current processing object is not a key frame, the terminal obtains the frame number corresponding to the previous frame of the current processing object, where the previous frame of the current processing object is the filling frame, that is,
  • the detection module 304 adjusts the frame serial number of the current processing object according to the frame serial number of the filling frame of the previous frame, and based on the adjusted frame serial number of the current processing object and the position of the filling frame in the buffer queue, performs the frame serial number of the filling frame in the buffer queue.
  • the detection module 304 assigns the frame number of the filled frame in the cache queue based on the frame sequence number adjusted by the current processing object and the position of the filled frame in the cache queue. Afterwards, it is also necessary to detect whether the frame sequence number corresponding to the assigned padding frame is greater than the preset threshold, and if so, the terminal performs zero-returning processing on the frame sequence number greater than the preset threshold.
  • the detection module 304 can be specifically configured to: when it is detected that the decoding of the target image frame is completed, use the next image frame of the target image frame as a new target image frame, obtain the current processing object, and detect the current processing object When the detection result indicates that the current processing object is a key frame, the frame number of the current processing object is used as the benchmark, and the filling frame is inserted between the target image frame and the next image frame of the target image frame based on the filling frame information; When it is detected that the frame type of the target image frame is not a key frame, the frame serial number of the current processing object is adjusted according to the frame serial number corresponding to the previous frame of the current processing object, and based on the filling frame information, the target image frame and the target Insert a filling frame between the next image frame of the image frame; perform the steps of taking the next image frame of the target image frame as a new target image frame to obtain the current processing object, until all the image frames in the image frame set are decoded complete.
  • the rendering module 305 is configured to render the decoded image frame to play the image sequence to be processed.
  • the rendering module 305 since the filling frame is a P-skip frame, the rendering module 305 only decodes the P-skip frame but does not render it, but decodes, renders and plays each frame of video in the image frame set, and finally achieves a reduced
  • the purpose of improving the fluency of image processing is the delay caused by video decoding.
  • the receiving module 301 of the present application After receiving the filling frame information sent by the server and the image frame set of the to-be-processed image sequence, the receiving module 301 of the present application, the determining module 302 inputs the image frame set frame by frame into the buffer queue of the decoder, and determines that it is currently at the top of the buffer queue.
  • the image frame is the target image frame
  • the inserting module 303 inserts the padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information, so that in the cache queue, except the target image frame, the rest
  • the frame is a filling frame
  • the detection module 304 detects that the decoding of the target image frame is completed
  • the next image frame of the target image frame is used as a new target image frame
  • the target image frame and the target image frame are executed based on the filling frame information.
  • the rendering module 305 renders the decoded image frames to play the image sequence to be processed.
  • the image processing device inserts a padding frame between the target image frame and the next image frame of the target image frame, so that when the target image frame is located in the cache queue, it is at the first position of the cache queue, and in the cache queue Except for the target image frame, the rest of the frames are filler frames, which improves the decoding speed of the image frame by the terminal, reduces the time delay caused by decoding the video, and further improves the efficiency of image processing.
  • FIG. 4 shows a schematic structural diagram of the electronic device involved in the present application, specifically:
  • the electronic device may include a processor 401 of one or more processing cores, a memory 402 of one or more computer-readable storage media, a power supply 403 and an input unit 404 and other components.
  • a processor 401 of one or more processing cores may include a processor 401 of one or more processing cores, a memory 402 of one or more computer-readable storage media, a power supply 403 and an input unit 404 and other components.
  • FIG. 4 does not constitute a limitation to the electronic device, and may include more or less components than the one shown, or combine some components, or arrange different components. in:
  • the processor 401 is the control center of the electronic device, uses various interfaces and lines to connect various parts of the entire electronic device, runs or executes the software programs and/or modules stored in the memory 402, and invokes the software programs and/or modules stored in the memory 402. Data, perform various functions of electronic equipment and process data, so as to conduct overall monitoring of electronic equipment.
  • the processor 401 may include one or more processing cores; the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and application programs, and the modem processor mainly handles Handle wireless communications. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 401.
  • the memory 402 can be used to store software programs and modules, and the processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402 .
  • the memory 402 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playback function, an image playback function, etc.) required for at least one function, and the like; Data created by the use of electronic equipment, etc.
  • memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, memory 402 may also include a memory controller to provide processor 401 access to memory 402 .
  • the electronic device also includes a power supply 403 for supplying power to various components.
  • the power supply 403 can be logically connected to the processor 401 through a power management system, so that functions such as charging, discharging, and power consumption management are implemented through the power management system.
  • Power source 403 may also include one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.
  • the electronic device may also include an input unit 404 that may be used to receive input numerical or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
  • an input unit 404 may be used to receive input numerical or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
  • the electronic device may further include a display unit and the like, which will not be described here.
  • the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 executes them and stores them in the memory 402 .
  • Receive the fill frame information sent by the server and the image frame set of the image sequence to be processed input the image frame set into the cache queue of the decoder frame by frame, and determine the image frame currently at the top of the cache queue as the target image frame.
  • the frame information inserts a padding frame between the target image frame and the next image frame of the target image frame.
  • the next image frame of the target image frame is used as a new target image frame, and
  • the step of inserting a filling frame between the target image frame and the next image frame of the target image frame based on the filling frame information is performed until all the image frames are decoded, and the decoded image frame is rendered to play the image sequence to be processed.
  • the image frame set is input into the cache queue of the decoder frame by frame, and the image frame currently at the top of the cache queue is determined as the target image frame. , and then insert a padding frame between the target image frame and the next image frame of the target image frame based on the padding frame information, so that in addition to the target image frame in the cache queue, the rest of the frames are padding frames.
  • a padding frame is inserted between the target image frame and the next image frame of the target image frame, so that when the target image frame is located in the cache queue, it is at the first position of the cache queue, and in the cache queue Except for the target image frame, the rest of the frames are filler frames, which improves the decoding speed of the image frame by the terminal, reduces the time delay caused by decoding the video, and further improves the efficiency of image processing.
  • the present application provides a non-volatile computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any of the image processing methods provided by the present application .
  • the instruction can perform the following steps:
  • Receive the fill frame information sent by the server and the image frame set of the image sequence to be processed input the image frame set into the cache queue of the decoder frame by frame, and determine the image frame currently at the head of the cache queue as the target image frame, based on the fill frame
  • the information inserts a padding frame between the target image frame and the next image frame of the target image frame.
  • the next image frame of the target image frame is taken as the new target image frame, and the execution
  • the storage medium may include: a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • magnetic disk or an optical disk and the like.
  • the system involved in this application may be a distributed system formed by connecting a client and multiple nodes (computing devices of any form in an access network, such as a server and a user terminal) through network communication.
  • FIG. 5 is a schematic structural diagram of the distributed system 100 provided by the present application applied to the blockchain system.
  • a peer-to-peer (P2P, Peer To Peer) network is formed between nodes.
  • the P2P protocol is a protocol that runs on the Transmission Control Protocol (TCP, Transmission Control Protocol) protocol. application layer protocol.
  • TCP Transmission Control Protocol
  • application layer protocol In a distributed system, any machine such as a server and a terminal can join to become a node, and a node includes a hardware layer, a middle layer, an operating system layer and an application layer.
  • the involved functions include:
  • Routing a basic function that a node has to support communication between nodes.
  • a node can also have the following functions:
  • Application used to deploy in the blockchain, implement specific business according to actual business needs, record data related to the realization of functions to form record data, carry a digital signature in the record data to indicate the source of the task data, and send the record data To other nodes in the blockchain system, for other nodes to add the record data to the temporary block when verifying the source and integrity of the record data successfully.
  • the business implemented by the application can include: blockchain, including a series of blocks (Block) that are connected to each other in the chronological order of generation. Once a new block is added to the blockchain, it will not be removed. The record data submitted by the nodes in the blockchain system is recorded.
  • blockchain including a series of blocks (Block) that are connected to each other in the chronological order of generation. Once a new block is added to the blockchain, it will not be removed.
  • the record data submitted by the nodes in the blockchain system is recorded.
  • FIG. 6 is a schematic diagram of the block structure (Block Structure) provided by this application, and each block includes the hash value of this block storage transaction record (the hash value of this block), and the previous block.
  • the hash value of the block, each block is connected by the hash value to form a blockchain.
  • the block may also include information such as a timestamp when the block was generated.
  • Blockchain (Blockchain) is essentially a decentralized database, which is a series of data blocks associated with cryptographic methods. Each data block contains relevant information to verify the validity of its information. (anti-counterfeiting) and generate the next block

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

本申请公开了一种图像处理方法、装置、电子设备以及存储介质，包括：接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合，将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得缓存队列中除目标图像帧之外，其余的帧为填充帧；当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中的所有图像帧均解码完毕，对解码后的图像帧进行处理，并展示处理结果，该方案可以降低对图像序列进行解码所带来的时延。

Description

图像处理方法、装置、电子设备以及存储介质
本申请要求于2020年7月1日提交中国专利局、申请号为2020106262817，发明名称为“图像处理方法、装置、电子设备以及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，具体涉及一种图像处理方法、装置、电子设备以及存储介质。
背景技术
越来越多的用户开始通过手机、平板电脑及个人电脑等终端在线收看视频，并且，随着移动终端及网络技术的发展，游戏也可以以视频播放的方式向用户展示相应的内容，以云游戏为例，在云游戏的场景下，游戏并不在玩家游戏终端，而是在云端服务器中运行，并由云端服务器将游戏场景渲染为视频音频流，通过网络传输给玩家的终端。玩家的终端无需拥有强大的图形运算与数据处理能力，仅需拥有基本的流媒体播放能力与获取玩家输入指令并发送给云端服务器的能力即可。
技术内容
本申请实施例提供一种图像处理方法、装置、电子设备以及存储介质，可以降低对图像序列进行解码所带来的时延，提高图像处理的流畅度。
本申请实施例提供了一种图像处理方法，包括：
接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合，所述填充帧信息由所述服务器根据所述图像帧集合生成的；
将所述图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧；
基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧，以使得所述缓存队列中除所述目标图像帧之外，其余的帧为所述填充帧；
当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所述图像帧集合中的所有图像帧均解码完毕；
对解码后的图像帧进行处理，并展示处理结果。
相应的，本申请实施例还提供了一种图像处理装置，包括：
接收模块，用于接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合， 所述填充帧信息由所述服务器根据所述图像帧集合生成的；
确定模块，用于将所述图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧；
插入模块，用于基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧，以使得所述缓存队列中除所述目标图像帧之外，其余的帧为所述填充帧；
检测模块，用于当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所述图像帧集合中的所有图像帧均解码完毕；
处理模块，用于对解码后的图像帧进行处理，并展示处理结果。
在本申请的一些实施例中，所述插入模块包括：
第一提取单元，用于从所述填充帧信息中提取由所述服务器根据所述图像帧集合生成的填充帧集合；
选择单元，用于在所述填充帧集合中选择与所述目标图像帧对应的填充帧，得到目标填充帧；
插入单元，用于在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入预设数量的目标填充帧。
在本申请的一些实施例中，所述插入单元具体用于：
检测所述缓存队列对应的最大缓存数量；
根据所述目标图像帧以及最大缓存数量，生成填充数量；
在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入与填充数量对应数量的目标填充帧。
在本申请的一些实施例中，还包括赋值模块，所述赋值模块包括：
第二提取单元，用于提取所述目标图像帧的帧序号；
赋值单元，用于基于帧序号以及在所述缓存队列中填充帧的位置，对所述缓存队列中填充帧的帧序号进行赋值。
在本申请的一些实施例中，所述赋值单元具体用于：
从所述填充帧信息中提取在所述缓存队列中填充帧对应的偏移量；
基于帧序号、在所述缓存队列中填充帧的位置以及在所述缓存队列中填充帧对应的偏移量，对所述缓存队列中填充帧的帧序号进行赋值。
在本申请的一些实施例中，所述检测模块具体用于：
当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象；
检测所述当前处理对象的类型；
当检测结果指示所述当前处理对象为关键帧时，则以所述当前处理对象的帧序号为基准，基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧；
当检测到所述目标图像帧的帧类型不为关键帧时，则根据所述当前处理对象的上一帧对应的帧序号，调整所述当前处理对象的帧序号，并基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧；
执行将所述目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象的步骤，直至所有图像帧均解码完毕。
在本申请的一些实施例中，所述确定模块具体用于：
获取当前渲染的所述待处理图像序列的关键帧，得到目标关键帧；
在所述图像帧集合确定所述目标关键帧的下一帧图像帧为目标图像帧；
将所述目标图像帧存放至缓存队列的首位。
本申请实施例还提供了一种电子设备，包括存储器，处理器及存储在存储器上并可在处理器上运行的计算机程序，其中，所述处理器执行所述程序时实现如上所述图像处理方法的步骤。
本申请实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，其中，所述计算机程序被处理器执行时实现如上所述图像处理方法的步骤。
附图简要说明
为了更清楚地说明本申请中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1a是本申请实施例提供的图像处理方法的场景示意图；
图1b是本申请实施例提供的图像处理方法的流程示意图；
图2a是本申请实施例提供的图像处理方法的另一流程示意图；
图2b是本申请实施例提供的图像处理系统的示意图；
图2c是本申请实施例提供的图像处理方法中生成填充帧的流程示意图；
图2d是本申请实施例提供的图像处理方法对图像帧集合进行处理的流程示意图；
图3a是本申请实施例提供的图像处理装置的结构示意图；
图3b是本申请实施例提供的图像处理装置的另一结构示意图；
图4是本申请实施例提供的电子设备的结构示意图；
图5是本申请实施例提供的分布式系统100应用于区块链系统的一个结构示意图；
图6是本申请实施例提供的区块结构(Block Structure)一个示意图。
具体实施方式
下面将结合本申请中的附图，对本申请中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
在一些云游戏场景中，受限于玩家的终端的解码能力，在解码方案中，视频硬件解码帧缓存会极大地影响整体延迟，导致最终播放的视频可能会存在卡顿的现象。
本申请实施例提供一种图像处理方法、装置、电子设备和存储介质，可以降低对图像序列进行解码所带来的时延，提高图像处理的流畅度。
本申请实施例在接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合后，所述填充帧信息由所述服务器根据所述图像帧集合生成的，将所述图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，接着，基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧，以使得所述缓存队列中除所述目标图像帧之外，其余的帧为所述填充帧，当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所有图像帧均解码完毕，最后，对解码后的图像帧进行处理，并展示处理结果。因此，该方案可以降低对视频进行解码所带来的时延，提高图像处理的流畅度。
其中，该图像处理装置具体可以集成在终端中，终端可以是智能手机、平板电脑、笔记本电脑、台式计算机、智能音箱、智能手表等，但并不局限于此。终端以及服务器可以通过有线或无线通信方式进行直接或间接地连接，本申请在此不做限制。
例如，请参阅图1a，该图像处理装置集成在终端上，终端可以接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合，其中，填充帧信息由所述服务器根据图像帧集合生成的，服务器为云端服务器，在云游戏场景下，由云端服务器将游戏场景渲染为视 频音频流(即视频流数据)，通过网络传输给用户的终端，然后，终端将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，接着，终端基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得缓存队列中除所述目标图像帧之外，其余的帧为填充帧，接着，终端当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中所有图像帧均解码完毕，最后，终端渲染解码后的图像帧，以播放所述待处理图像序列。
本申请提供的图像处理方法，通过在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得目标图像帧位于缓存队列时，处于缓存队列的首位，并且，缓存队列中除目标图像帧之外，其余的帧为填充帧，提高终端对图像帧解码的速度，降低对视频进行解码所带来的时延，进而提高了图像处理的效率。
以下分别进行详细说明。需说明的是，以下实施例的描述顺序不作为对实施例优先顺序的限定。
一种图像处理方法，包括：接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合，将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中的所有图像帧均解码完毕，对解码后的图像帧进行处理，并展示处理结果。
请参阅图1b，图1b为本申请提供的图像处理方法的流程示意图。该图像处理方法可以由终端执行，具体流程可以如下：
101、接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合。
其中，填充帧信息是由服务器根据图像帧集合生成的，终端可以通过有线网络或无线网络接收服务器发送的视频流数据，该视频流数据携带有填充帧信息以及待处理图像序列的图像帧集合，在接收服务器发送的视频流数据后，对该视频流数据进行解析，从而得到待处理图像序列的图像帧集合以及服务器根据图像帧集合生成的填充帧信息。视频数据源的格式可以是AVC(Advanced Video Coding，即H.264编码标准)，也可以是HEVC(High Efficiency Video Coding，即H.265编码标准)。
首先，介绍以下视频编码的概念，所谓视频编码方式就是指通过压缩技术，将原始 视频图像依次通过预测、变化、量化、重组以及熵编码压缩成二进制字节流，其中，编码方式包括：基于上下文的自适应变长编码(Context-Adaptive Varialbe-Length Coding，CAVLC)和基于上下文的自适应二进制算术编码(Context-based Adaptive Binary Arithmetic Coding，CABAC)，服务器在获取到待处理图像序列的图像序列头信息后，对图像序列头信息进行解析，得到待处理图像序列的画质级别、参考帧数量、是否有双向预测帧以及图像帧序列对应的长度等信息，并基于解析得到的信息，判断是否能够生成填充帧，服务器当生成填充帧时，将生成的填充帧、填充帧对应的序号以及偏移量，打包成填充帧信息，并写入图像序列头信息内，随后，将包含填充帧信息的图像序列头信息发送至终端。
102、将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧。
为了便于视频内容的存储和传输，通常需要减少视频内容的体积，也就是需要将原始的视频图像经过压缩，压缩算法也简称编码格式。例如，服务器可以采用H.264编码格式对视频图像进行压缩，当接收到服务器发送的压缩后的视频内容，需要对压缩后的视频内容进行解压缩，也称解码，就视频图像编码和解码而言，编码器将多张图像进行编码后生产成一段一段的图像组(Group of Pictures，GOP)，解码器在播放时则是读取一段一段的GOP进行解码后读取画面再渲染显示。GOP(Group of Pictures)是一组连续的画面，由一个关键帧和多个非关键帧组成，其中，关键帧是GOP的基础帧(第一帧)，在一组中只有一个关键帧，非关键帧包括前向参考帧和双向参考帧，其中，关键帧是一个完整的画面，前向参考帧和双向参考帧记录的是相对于关键帧的变化，因此，关键帧可以独立解码，而前向参考帧需要依赖前一帧图像进行解码，双向参考帧不仅需要依赖前一帧图像，还需要依赖下一帧图像才能进行解码，即，当需要播放视频时，首先需要定位出一帧关键帧的位置，才能实现视频的播放，也即，在一些实施例中，步骤“将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧”，具体可以包括：
(11)获取当前渲染的待处理图像序列的关键帧，得到目标关键帧；
(12)在图像帧集合确定目标关键帧的下一帧图像帧为目标图像帧；
(13)将目标图像帧存放至缓存队列的首位。
目前，用户的终端的可以与服务器之间通过缓存队列进行视频数据(即图像帧集合)的缓存，即，缓存队列用于对在图像处理过程中的每一个待解码的图像帧按照先进先出的顺序进行缓存，并且，仅当缓存队列中的所有的缓存空间均缓存有图像帧时，终端才对位 于缓存队列首位的图像帧进行解码。
103、基于填充帧信息在目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧。
其中，在目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧，以使得缓存队列中除目标图像帧之外，其余的帧为填充帧，以云游戏为例，云游戏(Cloud gaming)又可称为游戏点播(gaming on demand)，是一种以云计算技术为基础的在线游戏技术，并且，在云游戏中，服务器所发送的图像帧集合中一般只包括关键帧和前向参考帧，由于双向参考帧记录的是本帧与前后帧的差别，也就是说要解码双向参考帧，不仅要取得双向参考帧之前的缓存画面，还要解码双向参考帧之后的画面，通过前后画面的与本帧数据的叠加取得最终的画面，当双向参考帧需要解码时，还需要获取双向参考帧下一帧的数据，此时，双向参考帧的下一帧还位于缓存队列中，因此，并不能对双向参考帧进行解码，从而导致画面出现卡顿等现象。
填充帧信息包含与图像帧集合中每个图像帧对应的填充帧，因此，可以根据填充帧信息确定与目标图像帧对应的填充帧，并将确定的填充帧插入目标图像帧与目标图像帧的下一帧图像帧之间，以使得目标图像帧位于缓存队列的首位，且缓存队列中除所述目标图像帧之外，其余的帧为填充帧，即，在一些实施例中，步骤“基于填充帧信息在目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧”，具体可以包括：
(21)从填充帧信息中提取由服务器根据所述图像帧集合生成的填充帧集合；
(22)在填充帧集合中选择与目标图像帧对应的填充帧，得到目标填充帧；
(23)在目标图像帧与所述目标图像帧的下一帧图像帧之间插入预设数量的目标填充帧。
需要说明的是，该填充帧为前向预测帧，并且，该填充帧的帧模式为跳过模式，P-Skip宏块是一种特殊的P宏块，对于一般的P宏块而言，像素残差和运动矢量残差都会写入码流，从编码端传到解码端，但是P-Skip宏块的特殊之处就在于，既不传送像素残差，也不传送运动矢量残差，编码端除了传送一些标识该宏块是PSkip宏块的少量的字节外，不在需要传送关于该宏块的其他信息，由于运动矢量残差等于运动矢量与预测的运动矢量之差，在编码段的运动矢量残差为零，而解码时可以得到预测的运动矢量，也就是说，在解码时，也能得到运动矢量，在解码端有参考帧对应宏块的重建像素，根据这个重建像素和运动矢量，就可以恢复出本帧本宏块的像素值，这就是所谓的P-skip宏块原理，从字面理解就是跳过了这个宏块，相当于这个宏块没有编码，在解码端采取了近似替代的恢复办法，如果 本帧的一宏块和参考帧的一宏块像素(不要求两宏块在同一位置)几乎完全一致，很显然，当前帧的宏块就不用编码，在解码端，直接就可以用近似替代的办法恢复出本帧本宏块的像素值，比如在第一帧中有个乒乓球，在第二帧中也有个乒乓球，那个第二帧中乒乓球这个宏块就很可能编成了PSkip宏块，在本申请中，从编码前的角度插入缓存队列中的填充帧与目标图像帧是完全一样的。
进一步的，为了提高图像帧解码的效率，因此，在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，使得缓存队列被填满(达到缓存队列的最大缓存数量)，可以根据缓存队列的最大缓存数量以及目标图像帧，生成填充数量，然后，在目标图像帧与目标图像帧的下一帧图像帧之间插入与填充数量对应数量的目标填充帧，即，在一些实施例中，步骤“在目标图像帧与所述目标图像帧的下一帧图像帧之间插入预设数量的目标填充帧”，具体可以包括：
(31)检测所述缓存队列对应的最大缓存数量；
(32)根据所述目标图像帧以及最大缓存数量，生成填充数量；
(33)在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入与填充数量对应数量的目标填充帧。
为了保证在缓存队列插入填充帧后，待处理图像序列画面的连续性，因此，需要对缓存队列中填充帧的帧序号进行赋值，即，在一些实施例中，步骤“在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入与填充数量对应数量的目标填充帧”，具体还包括：
(41)提取目标图像帧的帧序号；
(42)基于帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值。
比如，目标图像帧的帧序号为5，在缓存队列中插入了4个填充帧，分别为填充帧A、填充帧B、填充帧C以及填充帧D，且填充帧之间的位置为填充帧A-填充B-填充帧C-填充帧D，按照依次递增的顺序依次赋予填充帧A、填充帧B、填充帧C以及填充帧D相应的帧序号，此时，填充帧A的帧序号为6，填充帧B的帧序号为7，填充帧C的帧序号为8，填充帧D的帧序号为9。
在实际应用中，比如，在H.264编码格式中，H.264条带是指特定条带组内部按照光栅扫描顺序排列的整数个宏块或宏块对，但这些宏块或宏块对在图像内部并不一定是按照光栅扫描顺序连续排列的，上述宏块的地址是通过条带第一个宏块的地址(条带头中描述) 以及宏块到条带组的映射得到的。H.264条带头是指编码条带的一部分，包含与该条带中第一个或者全部宏块有关的数据元素，也就是说，可以从填充帧信息中提取在缓存队列中填充帧对应的偏移量，该偏移量指的是填充帧在条带头中的偏移量，然后，基于帧序号在缓存队列中填充帧的位置以及在缓存队列中填充帧对应的偏移量，对所述缓存队列中填充帧的帧序号进行赋值，即，在一些实施例中，步骤“基于帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值，以使得在缓存队列中最后一帧填充帧的帧序号为第二帧序号”，具体可以包括：
(51)从填充帧信息中提取在缓存队列中填充帧对应的偏移量；
(52)基于帧序号、在缓存队列中填充帧的位置以及在缓存队列中填充帧对应的偏移量，对缓存队列中填充帧的帧序号进行赋值。
具体的，从填充帧信息中提取由服务器生成的填充帧的帧序号，并根据帧序号、在缓存队列中填充帧的位置以及在缓存队列中填充帧对应的偏移量，将缓存队列中填充帧的帧序号更新为相应的帧序号，以保证在缓存队列插入填充帧后，待处理图像序列画面的连续性。
104、当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中的所有图像帧均解码完毕。
例如，具体的，当检测到目标图像帧解码完成时，则将目标图像帧的下一帧图像帧作为新的目标图像帧，即，将目标图像帧的下一帧图像帧作为当前处理的图像帧，然后，在当前处理的图像帧与当前处理的图像帧的下一帧图像帧之间插入填充帧，即，执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中的所有图像帧均解码完毕。
需要说明的是，不同的关键帧所表示的图像是完全不同，因此，在插入填充帧时，还需要检测当前处理的图像帧是否为关键帧，即，在一些实施例中，步骤“当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中的所有图像帧均解码完毕”，具体可以包括：
(61)当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象；
(62)检测当前处理对象的类型；
(63)当检测结果指示当前处理对象为关键帧时，则以当前处理对象的帧序号为基准，基于填充帧信息在当前处理对象与当前处理对象的下一帧图像帧之间插入填充帧；
(64)当检测到当前处理对象不为关键帧时，则根据当前处理对象的上一帧对应的帧序号，调整当前处理对象的帧序号，并基于填充帧信息在当前处理对象与当前处理对象的下一帧图像帧之间插入填充帧；
(65)执行将目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象的步骤，直至所有图像帧均解码完毕。
例如，具体的，若当前处理对象为关键帧时，则将当前处理对象的帧序号确定为起始序号，基于当前处理对象的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值；若当前处理对象不为关键帧时，则获取当前处理对象的上一帧对应的帧序号，其中，当前处理对象的上一帧为填充帧，即，根据上一帧填充帧的帧序号调整当前处理对象的帧序号，并基于当前处理对象调整后的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值，需要说明的是，若当前处理对象不为关键帧时，基于当前处理对象调整后的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值之后，还需要检测赋值后的填充帧对应的帧序号是否大于预设阈值，若是，则将大于预设阈值的帧序号进行归零处理。
105、对解码后的图像帧进行处理，并展示处理结果。
其中，可以渲染解码后的图像帧，以播放待处理图像序列，在本申请中，由于填充帧为P-skip帧，因此，对P-skip帧只解码不渲染，而对图像帧集合中的每一帧视频解码、渲染以及播放，最终达到降低对视频进行解码所带来的时延，提高图像处理的流畅度的目的。
本申请在接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合后，将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，然后，基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得缓存队列中除目标图像帧之外，其余的帧为填充帧，当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中的所有图像帧均解码完毕，最后，对解码后的图像帧进行处理，并展示处理结果。本申请提供的图像处理方法，通过在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得目标图像帧位于缓存队列时，处于缓存队列的首位，并且，缓存队列中除目标图像帧之外， 其余的帧为填充帧，提高终端对图像帧解码的速度，降低对视频进行解码所带来的时延，进而提高了图像处理的效率。
根据实施例所述的方法，以下将举例进一步详细说明。
在本实施例中将以该图像处理装置具体集成在终端中为例进行说明。
请参阅图2a，一种图像处理方法，具体流程可以如下：
201、终端接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合。
202、终端将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧。
终端当接收到服务器发送的压缩后的视频内容，终端需要对压缩后的视频内容进行解压缩，即，终端当需要播放视频时，首先需要定位出一帧关键帧的位置，才能实现视频的播放。
203、终端基于填充帧信息在目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧。
填充帧信息包含与图像帧集合中每个图像帧对应的填充帧，因此，可以根据填充帧信息确定与目标图像帧对应的填充帧，并将确定的填充帧插入目标图像帧与目标图像帧的下一帧图像帧之间，以使得目标图像帧位于缓存队列的首位，且缓存队列中除所述目标图像帧之外，其余的帧为填充帧。
204、终端当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中的所有图像帧均解码完毕。
例如，具体的，若当前处理对象为关键帧时，则将当前处理对象的帧序号确定为起始序号，终端基于当前处理对象的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值；若当前处理对象不为关键帧时，终端则获取当前处理对象的上一帧对应的帧序号，其中，当前处理对象的上一帧为填充帧，即，终端根据上一帧填充帧的帧序号调整当前处理对象的帧序号，并基于当前处理对象调整后的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值，需要说明的是，若当前处理对象不为关键帧时，终端基于当前处理对象调整后的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值之后，还需要检测赋值后的填充帧对应的帧序号是否大于预设阈值，若是，终端则将大于预设阈值的帧序号进行归零处理。
205、终端对解码后的图像帧进行处理，并展示处理结果
例如，具体的，渲染解码后的图像帧，以播放待处理图像序列。在本申请中，由于填充帧为P-skip帧，因此，对P-skip帧只解码不渲染，而对图像帧集合中的每一帧视频解码、渲染以及播放，最终达到降低对视频进行解码所带来的时延，提高图像处理的流畅度的目的。
本申请的终端在接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合后，终端将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，然后，终端基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得缓存队列中除目标图像帧之外，其余的帧为填充帧，终端当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所有图像帧均解码完毕，最后，终端对解码后的图像帧进行处理，并展示处理结果。本申请提供的终端通过在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得目标图像帧位于缓存队列时，处于缓存队列的首位，并且，缓存队列中除目标图像帧之外，其余的帧为填充帧，提高终端对图像帧解码的速度，降低对视频进行解码所带来的时延，进而提高了图像处理的效率。
为了便于进一步理解本申请的图像处理方案，请参阅图2b，本申请提供一种图像处理系统(以下简称处理系统)，该处理系统包括：第一终端10、第二终端20以及服务器30，终端10采集待处理图像序列编码生成H.264视频流后，将待播放视频的图像序列头信息发送至服务器30，其中，图像序列头信息包括待播放视频的序列参数集和图像参数集，序列参数集中保存了一组编码视频序列的全局参数，所谓的编码视频序列即原始视频的一帧一帧的像素数据经过编码之后的结构组成的序列，而每一帧的编码后数据所依赖的参数保存于图像参数集中，然后，服务器30接收到图像序列头信息后，对图像序列头信息进行解析，得到待处理图像序列的基本格式信息，包括参考帧个数、视频级别、最大帧序列、帧序号在对应的条带中的偏移量以及双向参考帧图像帧的数量，并基于解析结果判断该待处理图像序列是否可以生成填充帧。具体的，请结合参阅图2c，当待处理图像序列中不存在双向参考帧且最大参考帧数量为1时，则确定待处理图像序列可以生成填充帧，否则，则结束流程；进一步的，当确定待处理图像序列可以生成填充帧时，根据解析结果确定相应的填充帧生成模块，比如，熵编码模式为CAVLC模式，则选择CAVLC模式对应的填充帧生成模块，熵编码模式为CABAC模式，则选择CABAC模式对应的填充帧生成模块；再然后，基于选择的填充帧生成模块指定编码器生成一个前向参考帧，不做任何运动向量估计，所 有宏块预测模式强制为P-skip模式(跳过模式)，经过指定熵编码模式(CAVLC/CABAC)生成相应的填充帧，最后，填充帧信息打包模块将生成的填充帧、解析结果中的帧序号以及帧序号在对应的条带中的偏移量进行封装，得到填充帧信息，服务器30将图像序列头信息以及填充帧信息发送至第二终端20。
请参阅图2d，并结合参阅图2b，第二终端20接收服务器30发送的图像序列头信息以及填充帧信息，第二终端20对图像序列头信息进行解析，得到待处理图像序列的图像帧集合，然后，第二终端20将图像帧集合逐帧插入第二终端20的解码模块中，若解码时存在缓存队列，第二终端20则从填充帧信息中提取出由服务器30生成的填充帧，接着，第二终端20根据图像帧集合确定当前位于缓存队列首位的图像帧为目标图像帧，并基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，随后，第二终端20根据填充帧信息中的偏移量分别对缓存队列中的填充帧和目标图像帧的帧序号进行更新，以保证待处理图像序列画面的连续性，具体请参阅前面实施例，在此不再赘述；另外，若解码时不存在缓存队列，服务器30中的填充帧生成模块则将图像序列头信息发送中继代理服务器，以便第二终端20接收到中继代理服务器发送的图像序列头信息展示待处理图像序列，并且，若解码出的视频帧为填充帧，则不对填充帧进行渲染，并执行解码的步骤，以播放待播放视频。
为便于更好的实施本申请的图像处理方法，本申请还提供一种基于上述方法的图像处理装置。其中名词的含义与上述图像处理方法中相同，具体实现细节可以参考方法实施例中的说明。
请参阅图3a，图3a为本申请提供的图像处理装置的结构示意图，其中该图像处理装置可以包括接收模块301、确定模块302、插入模块303、检测模块304以及渲染模块305，具体可以如下：
接收模块301，用于接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合。
其中，填充帧信息由服务器根据图像帧集合生成的，接收模块301可以通过有线网络或无线网络接收服务器发送的视频流数据，该视频流数据携带有填充帧信息以及待处理图像序列的图像帧集合，在接收服务器发送的视频流数据后，对该视频流数据进行解析，从而得到待处理图像序列的图像帧集合以及服务器根据图像帧集合生成的填充帧信息视频数据源的格式可以是AVC(Advanced Video Coding，即H.264编码标准)，也可以是HEVC(High Efficiency Video Coding，即H.265编码标准)。
确定模块302，用于将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于 缓存队列首位的图像帧为目标图像帧。
为了便于视频内容的存储和传输，通常需要减少视频内容的体积，也就是需要将原始的视频图像经过压缩，压缩算法也简称编码格式。例如，可以采用H.264编码格式对视频图像进行压缩，确定模块302当接收到服务器发送的压缩后的视频内容，终端需要对压缩后的视频内容进行解压缩，即，确定模块302当需要播放视频时，首先需要定位出一帧关键帧的位置，才能实现视频的播放。
在一些实施例中，确定模块302具体可以用于：获取当前渲染的待处理图像序列的关键帧，得到目标关键帧，在图像帧集合确定目标关键帧的下一帧图像帧为目标图像帧，将目标图像帧存放至缓存队列的首位。
插入模块303，用于基于填充帧信息在目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧。
填充帧信息包含与图像帧集合中每个图像帧对应的填充帧，因此，插入模块303可以根据填充帧信息确定与目标图像帧对应的填充帧，并将确定的填充帧插入目标图像帧与目标图像帧的下一帧图像帧之间，以使得目标图像帧位于缓存队列的首位，且缓存队列中除所述目标图像帧之外，其余的帧为填充帧。
在一些实施例中，插入模块303具体可以包括：
第一提取单元，用于从填充帧信息中提取由服务器根据图像帧集合生成的填充帧集合；
选择单元，用于在填充帧集合中选择与目标图像帧对应的填充帧，得到目标填充帧；
插入单元，用于在目标图像帧与目标图像帧的下一帧图像帧之间插入预设数量的目标填充帧。
在一些实施例中，插入单元具体可以用于：检测缓存队列对应的最大缓存数量，根据目标图像帧以及最大缓存数量，生成填充数量，在目标图像帧与目标图像帧的下一帧图像帧之间插入与填充数量对应数量的目标填充帧。
在一些实施例中，请参阅图3b，该图像处理装置还可以包括赋值模块306，赋值模块306包括：
第二提取单元，用于提取目标图像帧的帧序号；
赋值单元，用于基于帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值。
在一些实施例中，赋值单元具体可以用于：从填充帧信息中提取在缓存队列中填充 帧对应的偏移量，基于帧序号、在缓存队列中填充帧的位置以及在缓存队列中填充帧对应的偏移量，对缓存队列中填充帧的帧序号进行赋值。
检测模块304，用于当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至图像帧集合中的所有图像帧均解码完毕。
例如，具体的，若当前处理对象为关键帧时，则将当前处理对象的帧序号确定为起始序号，检测模块304基于当前处理对象的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值；若当前处理对象不为关键帧时，终端则获取当前处理对象的上一帧对应的帧序号，其中，当前处理对象的上一帧为填充帧，即，检测模块304根据上一帧填充帧的帧序号调整当前处理对象的帧序号，并基于当前处理对象调整后的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值，需要说明的是，若当前处理对象不为关键帧时，检测模块304基于当前处理对象调整后的帧序号以及在缓存队列中填充帧的位置，对缓存队列中填充帧的帧序号进行赋值之后，还需要检测赋值后的填充帧对应的帧序号是否大于预设阈值，若是，终端则将大于预设阈值的帧序号进行归零处理。
在一些实施例中，检测模块304具体可以用于：当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象，检测当前处理对象的类型，当检测结果指示当前处理对象为关键帧时，则以当前处理对象的帧序号为基准，基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧；当检测到目标图像帧的帧类型不为关键帧时，则根据当前处理对象的上一帧对应的帧序号，调整当前处理对象的帧序号，并基于填充帧信息在目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧；执行将目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象的步骤，直至图像帧集合中的所有图像帧均解码完毕。
渲染模块305，用于渲染解码后的图像帧，以播放待处理图像序列。
在本申请中，由于填充帧为P-skip帧，因此，渲染模块305对P-skip帧只解码不渲染，而对图像帧集合中的每一帧视频解码、渲染以及播放，最终达到降低对视频进行解码所带来的时延，提高图像处理的流畅度的目的。
本申请的接收模块301在接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合后，确定模块302将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，然后，插入模块303基于填充帧信息在目标图像帧 与目标图像帧的下一帧图像帧之间插入填充帧，以使得缓存队列中除目标图像帧之外，其余的帧为填充帧，检测模块304当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所有图像帧均解码完毕，最后，渲染模块305渲染解码后的图像帧，以播放待处理图像序列。本申请提供的图像处理装置，通过在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得目标图像帧位于缓存队列时，处于缓存队列的首位，并且，缓存队列中除目标图像帧之外，其余的帧为填充帧，提高终端对图像帧解码的速度，降低对视频进行解码所带来的时延，进而提高了图像处理的效率。
此外，本申请还提供一种电子设备，如图4所示，其示出了本申请所涉及的电子设备的结构示意图，具体来讲：
该电子设备可以包括一个或者一个以上处理核心的处理器401、一个或一个以上计算机可读存储介质的存储器402、电源403和输入单元404等部件。本领域技术人员可以理解，图4中示出的电子设备结构并不构成对电子设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。其中：
处理器401是该电子设备的控制中心，利用各种接口和线路连接整个电子设备的各个部分，通过运行或执行存储在存储器402内的软件程序和/或模块，以及调用存储在存储器402内的数据，执行电子设备的各种功能和处理数据，从而对电子设备进行整体监控。处理器401可包括一个或多个处理核心；处理器401可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器401中。
存储器402可用于存储软件程序以及模块，处理器401通过运行存储在存储器402的软件程序以及模块，从而执行各种功能应用以及数据处理。存储器402可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序(比如声音播放功能、图像播放功能等)等；存储数据区可存储根据电子设备的使用所创建的数据等。此外，存储器402可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他易失性固态存储器件。相应地，存储器402还可以包括存储器控制器，以提供处理器401对存储器402的访问。
电子设备还包括给各个部件供电的电源403，电源403可以通过电源管理系统与处理器401逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。电源403还可以包括一个或一个以上的直流或交流电源、再充电系统、电源故障检测电路、 电源转换器或者逆变器、电源状态指示器等任意组件。
该电子设备还可包括输入单元404，该输入单元404可用于接收输入的数字或字符信息，以及产生与用户设置以及功能控制有关的键盘、鼠标、操作杆、光学或者轨迹球信号输入。
尽管未示出，电子设备还可以包括显示单元等，在此不再赘述。具体在本实施例中，电子设备中的处理器401会按照如下的指令，将一个或一个以上的应用程序的进程对应的可执行文件加载到存储器402中，并由处理器401来运行存储在存储器402中的应用程序，从而实现各种功能，如下：
接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合，将所述图像帧集合逐帧输入解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所有图像帧均解码完毕，渲染解码后的图像帧，以播放待处理图像序列。
以上各个操作的具体实施可参见前面的实施例，在此不再赘述。
本申请在接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合后，将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，然后，基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得缓存队列中除目标图像帧之外，其余的帧为填充帧，当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所有图像帧均解码完毕，最后，渲染解码后的图像帧，以播放待处理图像序列。本申请提供的图像处理方法，通过在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，以使得目标图像帧位于缓存队列时，处于缓存队列的首位，并且，缓存队列中除目标图像帧之外，其余的帧为填充帧，提高终端对图像帧解码的速度，降低对视频进行解码所带来的时延，进而提高了图像处理的效率。
本领域普通技术人员可以理解，上述实施例的各种方法中的全部或部分步骤可以通过指令来完成，或通过指令控制相关的硬件来完成，该指令可以存储于一计算机可读存储介质中，并由处理器进行加载和执行。
为此，本申请提供一种非易失性计算机可读存储介质，其中存储有多条指令，该指 令能够被处理器进行加载，以执行本申请所提供的任一种图像处理方法中的步骤。例如，该指令可以执行如下步骤：
接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合，将图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧，当检测到目标图像帧解码完成时，将目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在目标图像帧与目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所有图像帧均解码完毕，渲染解码后的图像帧，以播放待处理图像序列。
以上各个操作的具体实施可参见前面的实施例，在此不再赘述。
其中，该存储介质可以包括：只读存储器(ROM，Read Only Memory)、随机存取记忆体(RAM，Random Access Memory)、磁盘或光盘等。
由于该存储介质中所存储的指令，可以执行本申请所提供的任一种图像处理方法中的步骤，因此，可以实现本申请所提供的任一种图像处理方法所能实现的有益效果，详见前面的实施例，在此不再赘述。
本申请涉及的系统可以是由客户端、多个节点(接入网络中的任意形式的计算设备，如服务器、用户终端)通过网络通信的形式连接形成的分布式系统。
以分布式系统为区块链系统为例，参见图5，图5是本申请提供的分布式系统100应用于区块链系统的一个结构示意图，由多个节点(接入网络中的任意形式的计算设备，如服务器、用户终端)和客户端形成，节点之间形成组成的点对点(P2P，Peer To Peer)网络，P2P协议是一个运行在传输控制协议(TCP，Transmission Control Protocol)协议之上的应用层协议。在分布式系统中，任何机器如服务器、终端都可以加入而成为节点，节点包括硬件层、中间层、操作系统层和应用层。
参见图5示出的区块链系统中各节点的功能，涉及的功能包括：
1)路由，节点具有的基本功能，用于支持节点之间的通信。
节点除具有路由功能外，还可以具有以下功能：
2)应用，用于部署在区块链中，根据实际业务需求而实现特定业务，记录实现功能相关的数据形成记录数据，在记录数据中携带数字签名以表示任务数据的来源，将记录数据发送到区块链系统中的其他节点，供其他节点在验证记录数据来源以及完整性成功时，将记录数据添加到临时区块中。
例如，应用实现的业务可以包括：区块链，包括一系列按照产生的先后时间顺序相 互接续的区块(Block)，新区块一旦加入到区块链中就不会再被移除，区块中记录了区块链系统中节点提交的记录数据。
参见图6，图6是本申请提供的区块结构(Block Structure)一个示意图，每个区块中包括本区块存储交易记录的哈希值(本区块的哈希值)、以及前一区块的哈希值，各区块通过哈希值连接形成区块链。另外，区块中还可以包括有区块生成时的时间戳等信息。区块链(Blockchain)，本质上是一个去中心化的数据库，是一串使用密码学方法相关联产生的数据块，每一个数据块中包含了相关的信息，用于验证其信息的有效性(防伪)和生成下一个区块
以上对本申请所提供的一种图像处理方法、装置、电子设备以及存储介质进行了详细介绍，本文中应用了具体个例对本发明的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本发明的方法及其核心思想；同时，对于本领域的技术人员，依据本发明的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本发明的限制。

Claims (15)

  1. 一种图像处理方法，包括：
    接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合，所述填充帧信息由所述服务器根据所述图像帧集合生成的；
    将所述图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于所述缓存队列首位的图像帧为目标图像帧；
    基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧，以使得所述缓存队列中除所述目标图像帧之外，其余的帧为所述填充帧；
    当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，并执行在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所述图像帧集合中的所有图像帧均解码完毕；
    对解码后的图像帧进行处理，并展示处理结果。
  2. 根据权利要求1所述的方法，所述基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧，包括：
    从所述填充帧信息中提取由所述服务器根据所述图像帧集合生成的填充帧集合；
    在所述填充帧集合中选择与所述目标图像帧对应的填充帧，得到目标填充帧；
    在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入预设数量的目标填充帧。
  3. 根据权利要求2所述的方法，所述在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入预设数量的目标填充帧，包括：
    检测所述缓存队列对应的最大缓存数量；
    根据所述目标图像帧以及最大缓存数量，生成填充数量；
    在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入与填充数量对应数量的目标填充帧。
  4. 根据权利要求3所述的方法，所述在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入与填充数量对应数量的目标填充帧之后，还包括：
    提取所述目标图像帧的帧序号；
    基于帧序号以及在所述缓存队列中填充帧的位置，对所述缓存队列中填充帧的帧序号进行赋值。
  5. 根据权利要求4所述的方法，所述基于帧序号以及在所述缓存队列中填充帧的位 置，对所述缓存队列中填充帧的帧序号进行赋值，包括：
    从所述填充帧信息中提取在所述缓存队列中填充帧对应的偏移量；
    基于帧序号、在所述缓存队列中填充帧的位置以及在所述缓存队列中填充帧对应的偏移量，对所述缓存队列中填充帧的帧序号进行赋值。
  6. 根据权利要求1至5任一项所述的方法，所述当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所有图像帧均解码完毕，包括：
    当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象；
    检测所述当前处理对象的类型；
    当检测结果指示所述当前处理对象为关键帧时，则以所述当前处理对象的帧序号为基准，基于填充帧信息在所述当前处理对象与所述当前处理对象的下一帧图像帧之间插入填充帧；
    当检测到所述当前处理对象不为关键帧时，则根据所述当前处理对象的上一帧对应的帧序号，调整所述当前处理对象的帧序号，并基于填充帧信息在所述当前处理对象与所述当前处理对象的下一帧图像帧之间插入填充帧；
    执行将所述目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象的步骤，直至所有图像帧均解码完毕。
  7. 根据权利要求1至5任一项所述的方法，所述将所述图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧，包括：
    获取当前渲染的所述待处理图像序列的关键帧，得到目标关键帧；
    在所述图像帧集合确定所述目标关键帧的下一帧图像帧为目标图像帧；
    将所述目标图像帧存放至缓存队列的首位。
  8. 一种图像处理装置，包括：
    接收模块，用于接收服务器发送的填充帧信息以及待处理图像序列的图像帧集合，所述填充帧信息由所述服务器根据所述图像帧集合生成的；
    确定模块，用于将所述图像帧集合逐帧输入至解码器的缓存队列中，并确定当前位于缓存队列首位的图像帧为目标图像帧；
    插入模块，用于基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像 帧之间插入填充帧，以使得所述缓存队列中除所述目标图像帧之外，其余的帧为所述填充帧；
    检测模块，用于当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，并执行基于填充帧信息在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入填充帧的步骤，直至所述图像帧集合中的所有图像帧均解码完毕；
    处理模块，用于对解码后的图像帧进行处理，并展示处理结果。
  9. 根据权利要求8所述的图像处理装置，所述插入模块包括：
    第一提取单元，用于从所述填充帧信息中提取由服务器根据所述图像帧集合生成的填充帧集合；
    选择单元，用于在所述填充帧集合中选择与所述目标图像帧对应的填充帧，得到目标填充帧；
    插入单元，用于在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入预设数量的目标填充帧。
  10. 根据权利要求9所述的图像处理装置，其中所述插入单元进一步用于：
    检测所述缓存队列对应的最大缓存数量，根据所述目标图像帧以及所述最大缓存数量，生成填充数量，在所述目标图像帧与所述目标图像帧的下一帧图像帧之间插入与填充数量对应数量的目标填充帧。
  11. 根据权利要求10的图像处理装置，进一步包括赋值模块，所述赋值模块包括：
    第二提取单元，用于提取所述目标图像帧的帧序号；
    赋值单元，用于基于帧序号以及在所述缓存队列中填充帧的位置，对所述缓存队列中填充帧的帧序号进行赋值。
  12. 根据权利要求11所述的图像处理装置，其中，所述赋值单元进一步用于：
    从所述填充帧信息中提取在所述缓存队列中填充帧对应的偏移量，基于帧序号、在所述缓存队列中填充帧的位置以及在所述缓存队列中填充帧对应的偏移量，对所述缓存队列中填充帧的帧序号进行赋值。
  13. 根据权利要求8至12任一项所述的方法，其中，所述检测模块进一步用于：
    当检测到所述目标图像帧解码完成时，将所述目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象；
    检测所述当前处理对象的类型；
    当检测结果指示所述当前处理对象为关键帧时，则以所述当前处理对象的帧序号为 基准，基于填充帧信息在所述当前处理对象与所述当前处理对象的下一帧图像帧之间插入填充帧；
    当检测到所述当前处理对象不为关键帧时，则根据所述当前处理对象的上一帧对应的帧序号，调整所述当前处理对象的帧序号，并基于填充帧信息在所述当前处理对象与所述当前处理对象的下一帧图像帧之间插入填充帧；
    执行将所述目标图像帧的下一帧图像帧作为新的目标图像帧，得到当前处理对象的步骤，直至所有图像帧均解码完毕。
  14. 一种电子设备，包括存储器，处理器及存储在存储器上并可在处理器上运行的计算机程序，其中，所述处理器执行所述程序时实现如权利要求1-7任一项所述图像处理方法的步骤。
  15. 一种计算机可读存储介质，其上存储有计算机程序，其中，所述计算机程序被处理器执行时实现如权利要求1-7任一项所述图像处理方法的步骤。
PCT/CN2021/095054 2020-07-01 2021-05-21 图像处理方法、装置、电子设备以及存储介质 Ceased WO2022001469A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2022552630A JP7471725B2 (ja) 2020-07-01 2021-05-21 画像処理方法、装置、電子機器及びコンピュータプログラム
EP21834613.8A EP4044608B1 (en) 2020-07-01 2021-05-21 Image processing method and apparatus, electronic device and storage medium
US17/728,810 US12217462B2 (en) 2020-07-01 2022-04-25 Image processing method and apparatus, electronic device, and storage medium for reducing time delay caused by decoding an image sequence and improving the smoothness of image processing

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010626281.7A CN111901666B (zh) 2020-07-01 2020-07-01 图像处理方法、装置、电子设备以及存储介质
CN202010626281.7 2020-07-01

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/728,810 Continuation US12217462B2 (en) 2020-07-01 2022-04-25 Image processing method and apparatus, electronic device, and storage medium for reducing time delay caused by decoding an image sequence and improving the smoothness of image processing

Publications (1)

Publication Number Publication Date
WO2022001469A1 true WO2022001469A1 (zh) 2022-01-06

Family

ID=73192864

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/095054 Ceased WO2022001469A1 (zh) 2020-07-01 2021-05-21 图像处理方法、装置、电子设备以及存储介质

Country Status (5)

Country Link
US (1) US12217462B2 (zh)
EP (1) EP4044608B1 (zh)
JP (1) JP7471725B2 (zh)
CN (1) CN111901666B (zh)
WO (1) WO2022001469A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115278308A (zh) * 2022-07-28 2022-11-01 北京字跳网络技术有限公司 媒体流处理方法、装置、设备及存储介质

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111901666B (zh) * 2020-07-01 2021-05-11 腾讯科技（深圳）有限公司 图像处理方法、装置、电子设备以及存储介质
CN112422873B (zh) * 2020-11-30 2022-09-16 Oppo（重庆）智能科技有限公司 插帧方法、装置、电子设备及存储介质
CN113240577B (zh) * 2021-05-13 2024-03-15 北京达佳互联信息技术有限公司 一种图像生成方法、装置、电子设备及存储介质
CN114257811A (zh) * 2021-12-21 2022-03-29 杭州雾联科技有限公司 一种云手机延时的处理方法、装置及介质
US12496526B1 (en) 2022-10-06 2025-12-16 A&C Technology, Inc. Image drawing non-fungible token video game
CN115671713A (zh) * 2022-11-14 2023-02-03 咪咕互动娱乐有限公司 云游戏画面补偿方法、系统、设备和存储介质
CN116962714A (zh) * 2023-01-12 2023-10-27 腾讯科技（深圳）有限公司 视频解码方法、装置、电子设备及计算机可读存储介质
CN117150059B (zh) * 2023-10-27 2024-02-13 湖南视觉伟业智能科技有限公司 一种基于nvr设备的碎片化图像数据存储方法
CN119342222B (zh) * 2024-10-23 2025-10-24 上海砹芯科技有限公司 一种视频编码方法、装置及电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103780908A (zh) * 2014-02-25 2014-05-07 成都佳发安泰科技股份有限公司 一种高效的h264解码方法
CN104113778A (zh) * 2014-08-01 2014-10-22 广州金山网络科技有限公司 一种视频流解码方法及装置
US9967579B2 (en) * 2012-11-28 2018-05-08 Cisco Technology, Inc. Fast switching hybrid video decoder
CN111901666A (zh) * 2020-07-01 2020-11-06 腾讯科技（深圳）有限公司 图像处理方法、装置、电子设备以及存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1374429A4 (en) 2001-03-05 2009-11-11 Intervideo Inc SYSTEMS AND METHOD FOR CODING AND DECODING REDUNDANT MOTION VECTORS IN COMPRESSED VIDEO BITSTRAMS
US8107531B2 (en) 2003-09-07 2012-01-31 Microsoft Corporation Signaling and repeat padding for skip frames
US7822123B2 (en) * 2004-10-06 2010-10-26 Microsoft Corporation Efficient repeat padding for hybrid video sequence with arbitrary video resolution
US8879635B2 (en) * 2005-09-27 2014-11-04 Qualcomm Incorporated Methods and device for data alignment with time domain boundary
US8218640B2 (en) * 2006-10-31 2012-07-10 Sony Computer Entertainment Inc. Picture decoding using same-picture reference for pixel reconstruction
WO2009078686A2 (en) 2007-12-18 2009-06-25 Humax Co., Ltd. Method and device for video coding and decoding
CN104113777B (zh) * 2014-08-01 2018-06-05 广州猎豹网络科技有限公司 一种音频流解码方法及装置
JP2016059015A (ja) 2014-09-12 2016-04-21 株式会社東芝 画像出力装置
CA3173525A1 (en) 2017-04-21 2018-10-25 Zenimax Media Inc. Systems and methods for game-generated motion vectors
CN110366033B (zh) * 2019-07-17 2021-10-22 腾讯科技（深圳）有限公司 一种视频播放方法、装置、设备及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9967579B2 (en) * 2012-11-28 2018-05-08 Cisco Technology, Inc. Fast switching hybrid video decoder
CN103780908A (zh) * 2014-02-25 2014-05-07 成都佳发安泰科技股份有限公司 一种高效的h264解码方法
CN104113778A (zh) * 2014-08-01 2014-10-22 广州金山网络科技有限公司 一种视频流解码方法及装置
CN111901666A (zh) * 2020-07-01 2020-11-06 腾讯科技（深圳）有限公司 图像处理方法、装置、电子设备以及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4044608A4 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115278308A (zh) * 2022-07-28 2022-11-01 北京字跳网络技术有限公司 媒体流处理方法、装置、设备及存储介质
CN115278308B (zh) * 2022-07-28 2023-12-01 北京字跳网络技术有限公司 媒体流处理方法、装置、设备及存储介质

Also Published As

Publication number Publication date
US20220254069A1 (en) 2022-08-11
EP4044608B1 (en) 2025-08-13
JP7471725B2 (ja) 2024-04-22
CN111901666B (zh) 2021-05-11
CN111901666A (zh) 2020-11-06
JP2023517003A (ja) 2023-04-21
EP4044608A1 (en) 2022-08-17
EP4044608A4 (en) 2023-01-04
US12217462B2 (en) 2025-02-04

Similar Documents

Publication Publication Date Title
CN111901666B (zh) 图像处理方法、装置、电子设备以及存储介质
JP7221957B2 (ja) ビデオエンコーダレンダリング向けのゲームエンジンアプリケーション
CN112073737B (zh) 在直播视频流应用中重新编码预测的图像帧
CN110784740A (zh) 视频处理方法、装置、服务器及可读存储介质
CN107634930B (zh) 一种媒体数据的获取方法和装置
CN109963176B (zh) 视频码流处理方法、装置、网络设备和可读存储介质
US11217279B2 (en) Method and device for adjusting video playback speed
CN112533059B (zh) 图像渲染方法、装置、电子设备以及存储介质
EP3410302B1 (en) Graphic instruction data processing method, apparatus
CN111404882B (zh) 媒体流处理方法及装置
CN111726624A (zh) 用于视频递送的纹理的序列的生成
CN106791850A (zh) 视频编码方法及装置
TWI605704B (zh) 重組影像檔案的方法
KR20230144620A (ko) 포인트 클라우드 인코딩 및 디코딩 방법, 포인트 클라우드 인코딩 및 디코딩 장치, 컴퓨터로 판독 가능한 매체, 그리고 전자 디바이스
CN103929682B (zh) 一种在视频直播系统中设置关键帧的方法及装置
US12432362B2 (en) Encoding and decoding video data
TWI505717B (zh) 彩色影像之接合純量嵌入圖形寫碼
CN115225902A (zh) 基于分散编码的高分辨率vr云游戏解决方法及计算机设备
KR20160023777A (ko) 그래픽 처리기를 이용한 비디오 디코딩에 대한 픽처 참조 제어
JP2017192080A (ja) 画像圧縮装置、画像復号装置、画像圧縮方法及び画像圧縮プログラム
JP7768593B2 (ja) データ処理方法、装置、コンピュータ機器及びコンピュータプログラム
CN113785495B (zh) 用于视频编解码的方法和装置、存储介质
US20240244229A1 (en) Systems and methods for predictive coding
CN116648904B (zh) 视频编码的方法、视频解码的方法、装置及存储介质
CN117440159A (zh) 一种视频解码方法、装置、计算机设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21834613

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021834613

Country of ref document: EP

Effective date: 20220512

ENP Entry into the national phase

Ref document number: 2022552630

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

WWG Wipo information: grant in national office

Ref document number: 2021834613

Country of ref document: EP