Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/103—Selection of coding mode or of prediction mode
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/103—Selection of coding mode or of prediction mode
  • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/174—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a slice, e.g. a line of blocks or a group of blocks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/187—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a scalable video layer
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
  • H04N19/34—Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/46—Embedding additional information in the video signal during the compression process
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards

Definitions

• the invention lies in the field of video compression, and more particularly immersive or omnidirectional video coding and decoding techniques (eg 180 °, 360 ° in 2D or 3D).
• immersive or omnidirectional video coding and decoding techniques eg 180 °, 360 ° in 2D or 3D.
• Omnidirectional video content allows you to represent a scene from a central point in any direction. We are talking about 360 ° video content when the entire field is captured. A subset of the field can also be captured, for example 180 ° only.
• Content can be captured monoscopically (2D) or stereoscopically (3D). This type of content can be generated by assembling sequences of images captured by different cameras, or computer-generated (ex: video games in VR). The images of such video content make it possible to render via a suitable device the video in any direction.
• a user can control the direction in which the captured scene is displayed and navigate continuously in all possible directions.
• Such 360 ° video contents may for example be rendered using a virtual reality headset offering the user an impression of immersion in the scene captured by the 360 ° video content.
• Such 360 ° video contents require reception devices adapted to this type of content (virtual reality headset for example) in order to offer the immersion and control functions of the view displayed by the user.
• the broadcasting of 360 ° video contents is not retro-compatible with the existing video receiver park and is limited to receivers adapted to this type of content.
• the content captured specifically for a 360 ° video broadcast can already be captured for a 2D or 3D video broadcast. In this case, all 360 content projected on a map is broadcast.
• layered video encoding techniques for encoding a 2D video stream into multiple successive refinement layers providing different levels of reconstruction of the 2D video.
• spatial scalability makes it possible to encode a video signal in several layers of increasing spatial resolution.
• Scalability in PSNR for Peak Signal to Noise Ratio
• Scalability in color space makes it possible to encode a video signal in several layers represented in color spaces that are larger and wider.
• none of the existing coding techniques makes it possible to generate a video stream representative of a scene that can be decoded at the same time by a conventional 2D or 3D video decoder and by a 360 ° video decoder.
• Document US 2016/156917 describes a method of scalable coding of a video which may be a multi-view video and in which each view of the multi-view video is encoded in one layer of the stream and predicted by another view of the video. multi-view video.
• the invention improves the state of the art. To this end, it relates to a method of encoding a data stream representative of an omnidirectional video, comprising:
• the at least one enhancement layer being coded by prediction with respect to the at least one base layer.
• the invention thus makes it possible to reduce the transmission cost of the video streams when the video contents have to be transmitted both in 2D and 360 ° view or in 3D and 3D-360 ° views.
• a conventional 2D or 3D video decoder will decode only one or more of the base layers to reconstruct a 2D or 3D video of the scene and a 360 ° compatible decoder will decode the base layer (s) and at least one enhancement layer for rebuild the 360 ° video.
• the use of a prediction of the at least one base layer for coding the enhancement layer thus makes it possible to reduce the cost of coding the enhancement layer.
• the invention also relates to a method of decoding a data stream representative of an omnidirectional video, comprising:
• omnidirectional video we mean here as well a video of a scene whose entire field (360 °) is captured, a video of a scene of which a subfield of the 360 ° field is captured, for example 180 °, 160 °, 255.6 °, or other.
• the omnidirectional video is therefore representative of a scene captured on at least one continuous part of the field at 360 °.
• the prediction of the enhancement layer with respect to the at least one base layer comprises, for encoding or reconstructing at least one image of the enhancement layer:
• the prediction in the enhancement layer is achieved by the addition during coding or decoding of an image of the enhancement layer of a reference image in which the images reconstructed from the base layers are projected.
• a new reference image is added to the reference image memory of the enhancement layer. This new reference image is generated by geometric projection of all the basic images reconstructed from the base layers at a time instant.
• the data stream comprises information representative of a type of geometric projection used to represent the omnidirectional video.
• the view represented by the 2D or 3D video is a view extracted from the omnidirectional video.
• the data stream comprises information representative of a type of geometric projection used to extract a view of the omnidirectional video and its location parameters.
• such information representative of the projection and location parameters of said base image is encoded in the data stream at each image of the 360 ° video.
• this variant makes it possible to take into account a displacement in the scene of a view serving as a prediction for the raising layer.
• the images of the video of the base layer can correspond to images captured by moving in the scene, for example to follow a moving object in the scene.
• the view can be captured by a moving camera or successively by several cameras located at different points of view in the scene, to follow a balloon or a player during a football match for example.
• the data stream comprises at least two basic layers, each base layer being representative of a 2D or 3D video, each base layer being respectively representative of a view of the scene, the at least two base layers being coded independently of one another.
• an image of the enhancement layer is coded using a group of tiles, each tile covering a region of the image of the enhancement layer, each region being distinct and disjunct from the other regions of the enhancement layer image, each tile being prediction encoded with respect to at least one base layer.
• the decoding of the enhancement layer includes the reconstruction of a portion of the image of the enhancement layer, the reconstruction of said portion of the image comprising the decoding of the enhancement layer tiles covering the portion of the image of the enhancement layer to be reconstructed, and the decoding of the at least one base layer comprising the decoding of the base layers used to predict the tiles covering the portion of the image of the enhancement layer to be reconstructed.
• Such a particular embodiment of the invention makes it possible to reconstruct only part of the omnidirectional image, and not the entire image. Typically, only the part being viewed by the user is reconstructed. Thus, it is not necessary to decode all the basic layers of the video stream, or even send them to the receiver. Indeed, a user can not simultaneously see the entire image of the omnidirectional video, it is thus possible to encode an omnidirectional image by a A tile mechanism for independently encoding regions of the omnidirectional image to subsequently decode only those regions of the omnidirectional image visible to the user.
• the independent coding of the basic layers thus makes it possible to reconstruct the tiles of the omnidirectional image separately and to limit the complexity to decoding by avoiding the decoding of unnecessary basic layers.
• information identifying the at least one base layer used to predict the tile is decoded from the data stream.
• the invention also relates to a device for encoding a data stream representative of an omnidirectional video.
• the coding device comprises coding means in said stream of at least one base layer representative of a 2D or 3D video, the 2D or 3D video being representative of a view of the same scene captured by the omnidirectional video , and encoding means in said stream of at least one enhancement layer representative of the omnidirectional video, said enhancement layer coding means comprising means for predicting the enhancement layer with respect to the at least one layer basic.
• the invention also relates to a device for decoding a data stream representative of an omnidirectional video.
• the decoding device comprises means for decoding in said stream of at least one base layer representative of a 2D or 3D video, the 2D or 3D video being representative of a view of the same scene captured by the omnidirectional video , and decoding means in said stream of at least one enhancement layer representative of the omnidirectional video, said enhancement layer decoding means comprising means for predicting the enhancement layer with respect to the at least one layer basic.
• the coding or decoding device is particularly suitable for implementing the coding or decoding method described above.
• the coding or decoding device may, of course, comprise the various characteristics relating to the coding or decoding method according to the invention.
• the characteristics and advantages of this encoding device, respectively of decoding are the same as those of the coding method, respectively of decoding, and are not detailed further.
• the decoding device is included in a terminal.
• the invention also relates to a signal representative of an omnidirectional video, comprising coded data of at least one base layer representative of a 2D or 3D video, the 2D or 3D video being representative of a view of the same scene captured by the omnidirectional video, and coded data of at least one enhancement layer representative of the omnidirectional video, the at least one enhancement layer being coded by prediction with respect to the at least one base layer.
• an image of the enhancement layer is coded using a group of tiles, each tile covering a region of the image of the enhancement layer, each region being distinct. and disjoined from the other regions of the enhancement layer image, each tile being prediction encoded with respect to at least one base layer.
• the signal also comprises for each tile information identifying the at least one base layer used to predict the tile.
• the invention also relates to a computer program comprising instructions for implementing the coding method or the decoding method according to any one of the particular embodiments described above, when said program is executed by a processor.
• a program can use any programming language. It can be downloaded from a communication network and / or saved on a computer-readable medium.
• This program can use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other form desirable shape.
• the invention relates to a computer-readable recording medium or information medium comprising instructions of a computer program as mentioned above.
• the recording media mentioned above can be any entity or device capable of storing the program.
• the medium may comprise storage means, such as a read-only memory (ROM), for example a CD-ROM or a microelectronic circuit ROM, a flash memory mounted on a removable storage medium , such as a USB key, or a magnetic mass memory type Hard-Disk Drive (HDD) or Solid-State Drive (SSD), or a combination of memories operating according to one or more data recording technologies.
• the recording media may correspond to a transmissible medium such as an electrical signal or optical, which can be routed via an electrical or optical cable, by radio or by other means.
• the proposed computer program can be downloaded on an Internet type network.
• the recording media may correspond to an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method in question.
• the coding or decoding method according to the invention can therefore be implemented in various ways, in particular in hard-wired form or in software form. 4. List of figures
• FIG. 1A illustrates steps of the coding method according to a particular embodiment of the invention
• FIG. 1B illustrates an example of a signal generated according to the coding method implemented according to a particular embodiment of the invention
• FIG. 2A illustrates an image of a view of a scene captured by a 360 ° video encoded in a base layer
• FIG. 2B illustrates the image illustrated in FIG. 2A projected in the reference frame of an image of the 360 ° video
• FIG. 2C illustrates an image of the 360 ° video coded in an enhancement layer
• FIGS. 2D and 2E each illustrate an image of two views of a scene captured by a 360 ° video and each coded in a base layer
• FIG. 2F illustrates the images of the two views illustrated in FIGS. 2D and 2E projected in the reference frame of an image of the 360 ° video
• FIG. 2G illustrates an image of the 360 ° video coded in an enhancement layer
• FIG. 3 illustrates steps of the decoding method according to a particular embodiment of the invention
• FIG. 4A illustrates an example of an encoder configured to implement the coding method according to a particular embodiment of the invention
• FIG. 4B illustrates a device adapted to implement the coding method according to another particular embodiment of the invention
• FIG. 5A illustrates an example of a decoder configured to implement the decoding method according to a particular embodiment of the invention
• FIG. 5B illustrates a device adapted to implement the decoding method according to another particular embodiment of the invention.
• FIGS. 6A and 6B respectively illustrate an image of the 360 ° omnidirectional video coded by independent tiles and a reference image generated from two views of two base layers and used to code the image of FIG. 6A,
• FIGS. 7A-C respectively show a projection in a 2D plane of a 360 ° omnidirectional video with a cubemap projection, a 3D spherical representation in an XYZ repository of the 360 ° omnidirectional video, and a view extracted from the immersive content 360 ° in a 2D plane according to a rectilinear projection,
• FIG. 7D illustrates the relationship between different geometric projections
• FIGS. 2A, CE and G and FIGS. 7A-B are taken from 360 ° videos made available by LetlnVR as part of the JVET exploration group (for Joint Video Exploration Team in English, JVT-D0179: Test Sequences for Virtual Reality Video Coding from Letin VR, October 15-21, 2016). 5. Description of an embodiment of the invention
• the general principle of the invention is to encode a data stream in a scalable manner thus making it possible to reconstruct and restore a 360 ° video when a receiver is adapted to receive and render such a 360 ° video and reconstruct and render a 2D video or 3D when the receiver is only suitable for rendering a 2D or 3D video.
• the 2D or 3D video is encoded in a base layer and the 360 ° video is coded in a predicted enhancement or enhancement layer from the basecoat.
• the stream may comprise several basic layers each corresponding to a 2D or 3D video corresponding to a view of the scene.
• the improvement layer is thus coded by prediction from all or part of the base layers included in the stream. 5.
• FIG. 1A illustrates steps of the coding method according to a particular embodiment of the invention.
• a 360 ° video is scalable by extracting 360 ° video views and encoding each view into a base layer.
• view we mean here a sequence of images acquired from a point of view of the scene captured by the 360 ° video.
• Such an image sequence can be a monoscopic image sequence in the case of a 2D 360 ° video or a stereoscopic image sequence in the case of a 3D 360 ° video.
• each image includes a left view and a right view coded jointly, for example in the form of an image generated using left and right views placed side by side or one at the left. above the other.
• the encoder encoding such a stereoscopic image sequence in a base layer or enhancement layer will then encode each image including a left view and a right view as a conventional 2D image sequence.
• the omnidirectional video is a 2D 360 ° video.
• the number of base layers is independent of the number of views used to generate 360 ° video.
• the number of base layers encoded in the scalable data stream is for example determined during the production of the content, or may be determined by the encoder for rate optimization purposes.
• first and second views are extracted from the 360 video.
• the views [1] and [2] are respectively coded during a coding step 12 of a base layer BL [1] and a coding step 13 of a base layer BL [2].
• the base layers BL [1] and BL [2] are coded independently of each other, ie there is no coding dependency (prediction, coding context , etc.) between the coding of the images of the base layer BL [1] and the coding of the images of the base layer BL [2].
• Each base layer BL [1] or BL [2] is decodable independently of the others.
• this particular embodiment it is possible to code the base layers BL [1] and BL [2] in a dependent manner, for example to gain efficiency in compression.
• this particular embodiment of the invention requires the decoder to be able to decode the two basic layers to render a conventional 2D video.
• Figs. 2A-2C illustrate an embodiment in which a single base layer is used.
• the images of the 360 ° video have a spatial resolution of 3840x1920 pixels and are generated by an equi-rectangular projection and the 360 ° image sequence has a frequency of 30 frames per second.
• Figure 2C illustrates an image of 360 ° video at a time instant t coded in the enhancement layer.
• FIG. 2A An image at the time instant t of the view extracted from the 360 ° video is illustrated in FIG. 2A.
• the coordinates Yaw and Pitch correspond to the coordinates of the center (P in FIG. 2B) of the geometric projection of an image of the view of the base layer, the coordinates Yaw and Pitch respectively correspond to the angle ⁇ and to the angle ⁇ of the point P illustrated in the pivot format in FIG. 7B.
• the parameters Horizontal FOV and Vertical FOV correspond respectively to the horizontal and vertical size of an image of the extracted view centered at point P in the pivot format illustrated in FIG. 7B, this image of the extracted view is represented in FIG. 7C.
• FIG. 2B illustrates reference image 1 ref used to predict the image of the 360 ° video at time t after equirectangular geometric projection of the image of the base layer illustrated in FIG. 2A.
• Figures 2D-2G illustrate an embodiment in which two base layers are used.
• the images of the 360 ° video have a spatial resolution of 3840x1920 pixels and are generated by an equi-rectangular projection and the 360 ° image sequence at a frequency of 30 frames per second.
• Figure 2G illustrates an image of 360 ° video at a time instant t encoded in the enhancement layer.
• FIG. 2D An image at the time instant t of a first view extracted from the 360 ° video is illustrated in FIG. 2D.
• FIG. 2E An image at the time instant t of a second view extracted from the 360 ° video is illustrated in FIG. 2E.
• FIG. 2F illustrates the reference image 1 ref used to predict the image of the 360 ° video at the instant t after equirectangular geometric projection of the images of the first view and the second view respectively illustrated in FIGS. 2D and 2E.
• the representation of a 360 ° omnidirectional video in a plane is defined by a geometric transformation characterizing the way in which a 360 ° omnidirectional content represented in a sphere is adapted to a representation in a plane.
• the spherical representation of the data is used as a pivot format, it makes it possible to represent the points captured by an omnidirectional video device. Such a 3D XYZ spherical representation is illustrated in FIG. 7B.
• the 360 ° video is represented using an equirectangular geometric transformation that can be seen as the projection of points on a cylinder surrounding the sphere.
• Other geometrical transformations are of course possible, for example the projection in CubeMap, corresponding to a projection of the points on a cube enclosing the sphere, the faces of the cubes finally being unfolded on a plane to form the 2D image.
• Such a CubeMap projection is for example illustrated in FIG. 7A.
• Figure 7D illustrates in more detail the relationship between the different formats mentioned above.
• the transition from an equirectangular A format to a cubemap B format is done through a pivotal format C characterized by a representation of the samples in an XYZ spherical system illustrated in FIG. 7B.
• the extraction of a view D from the format A is done through this pivotal format C.
• the extraction of a view of the immersive content is characterized by a geometrical transformation, for example by operating a projection rectilinear points of the sphere on a plane shown by the plane ABCD in Figure 7C. This projection is characterized by location parameters such as yaw, pitch, and horizontal and vertical field of view (FOV).
• location parameters such as yaw, pitch, and horizontal and vertical field of view (FOV).
• FIG. 8 illustrates the different steps allowing the passage between two formats.
• a look-up table is first constructed in E80 to match the position of each sample in the destination image (l ref ) with its position corresponding in the source format (corresponding to the reconstructed images of the base layers BL [1] and BL [2] in the example described with FIG. 1A). For each position (u, v) in the destination image the following steps apply:
• each pixel (u, v) in the destination image (l ref ) is interpolated with respect to the value at the corresponding position (u ', v') in the image source during a step E84 (corresponding to the reconstructed images of the base layers BL [1] and BL [2] in the example described with FIG. 1A).
• An interpolation can be performed in (uV) before the value is assigned, by applying a Lanczos type interpolation filter to the decoded base layer image at the matched position.
• the 360 ° video is coded in an improvement layer EL by prediction with respect to the BL [1] and BL [2] base layers using the image reference ref generated from the base layers .
• a step 17 the data encoded in steps 12, 13 and 16 are multiplexed to form a bit stream comprising the coded data of the BL [1] and BL [2] base layers and the enhancement layer. EL.
• the projection data making it possible to construct the reference image 1 ref are also coded in the bit stream and transmitted to the decoder.
• FIG. 1B illustrates an example of a bit stream generated according to the method described with reference to FIG. 1A.
• the bit stream comprises:
• a PRJ information representative of the type of geometric projection used to represent the omnidirectional content for example a value indicating an equirectangular projection, an information PRJ_B1, respectively PRJ_B2, representative of the projection used to extract the view and its location parameters in the 360 ° video from the view of the base layer BL [1], respectively BL [2].
• the information representative of the projection and location parameters of a view of the base layer can for example be coded in the form of the coordinates of the view (Yaw, Pitch, HFOV, VFOV) with the projection type (rectilinear). used to extract the view.
• the information representative of the projection and location parameters of a view of a base layer can be coded once in the bit stream. It is thus valid for the entire sequence of images.
• the information representative of the projection and location parameters of a view of a base layer can be coded several times in the bit stream, for example at each image, or at each group of images. It is thus valid only for an image or group of images.
• such a variant provides the advantage that the view extracted at each time point of the sequence may correspond to a view of an object in progress. movement in the scene and followed over time.
• FIG. 3 illustrates steps of the decoding method according to a particular embodiment of the invention.
• the scalable bitstream representative of the 360 ° video is de-multiplexed during a step 30.
• the coded data of the base layers, BL [1] and BL [2] in FIG. the example described here are sent to a decoder for decoding (steps 31, 33 respectively).
• the reconstructed images of the base layers are projected (steps 32, 34 respectively) in a manner similar to the coding method on a ref reference image to serve as a prediction to the enhancement layer EL.
• the geometric projection is performed from the projection data provided in the bit stream (projection type, projection information and view location).
• the coded data of the enhancement layer EL are decoded (step 35) and the images of the 360 ° video are reconstructed using the ref reference images generated from geometric projections made on the base layers, as previously specified.
• the scalable bitstream representative of the 360 ° video makes it possible to address any type of receiver. Such a scalable flow also allows each receiver to decode and reconstruct a 2D video or a 360 ° video according to its capabilities.
• receivers such as PC, TV, tablet, etc. will only decode a base layer, and will restore a sequence of 2D images.
• receivers adapted for 360 ° video such as virtual reality headphones, smartphones, etc., will decode base layers and enhancement layer and render 360 ° video.
• FIG. 4A illustrates in greater detail the coding steps of a base layer and a process improvement layer described above according to a particular embodiment of the invention.
• the case of coding an improvement layer coding a 360 ° omnidirectional video by prediction from a base layer encoding a view k is described here.
• Each image of the view k to be encoded is divided into blocks of pixels and each block of pixels is then conventionally encoded by spatial or temporal prediction using a previously reconstructed reference image of the image sequence of the view k.
• a prediction module P determines a prediction for a current block B k c .
• the current block B k c is coded by spatial prediction with respect to other blocks of the same image or by temporal prediction with respect to a block of a previously coded and reconstructed reference image of the view k and stored in the MEM memory b .
• a prediction residue is obtained by calculating the difference between the current block B k c and the prediction determined by the prediction module P.
• This prediction residue is then transformed by a transformation module T implementing for example a transformation of DCT (Discrete Cosine Transform) type.
• the transformed coefficients of the residue block are then quantized by a quantization module Q, and then coded by the entropy coding module C to form the coded data of the base layer BL [k].
• DCT Discrete Cosine Transform
• the prediction residue is reconstructed, via an inverse quantization performed by the Q "1 module and an inverse transformation performed by the T " module 1 and added to the prediction determined by the prediction module P to reconstruct the current block.
• the reconstructed current block is then stored in order to reconstruct the current image and this reconstructed current image can be used as a reference when encoding subsequent images of the view k.
• a projection module PROJ performs a geometrical projection of the reconstructed image in the reference image 1 ref of the 360 ° video, as illustrated in FIG. 2B and according to FIG. geometric transformation described previously.
• the reference image l ref obtained by projection of the reconstructed image of the base layer is stored in the memory of the enhancement layer MEM e .
• 360 ° omnidirectional video is frame-by-frame and block-wise coded.
• Each block of pixels is conventionally encoded by spatial or temporal prediction using a reference image previously reconstructed and stored in the memory MEM e .
• a prediction module P determines a prediction for a current block B e c of a current image of the 360 ° omnidirectional video.
• the current block B e c is coded by spatial prediction with respect to other blocks of the same image or by temporal prediction with respect to a block of a previously coded and reconstructed reference image of the 360 ° video and stored in MEM memory e .
• the current block B e c can also be coded by inter-layer prediction with respect to a block co-located in the reference image 1 ref obtained from the base layer.
• a coding mode is indicated in the coded EL data of the improvement layer by an INTER coding mode signaling a block time coding, a zero motion vector, and a reference index indicating the image of the coding layer. reference of the memory MEM e used indicating the image l ref .
• This information is coded by the entropy coder C.
• the prediction mode determined for coding a current block B e c is for example selected from all possible prediction modes and selecting the one minimizing a rate / distortion criterion.
• a prediction residual is obtained by calculating the difference between the current block B e c and the prediction determined by the prediction module P.
• This prediction residue is then transformed by a transformation module T implementing for example a transformation of DCT (Discrete Cosine Transform) type.
• the transformed coefficients of the residue block are then quantized by a quantization module Q, and then coded by the entropy coding module C to form the coded data of the enhancement layer EL.
• the prediction residue is reconstructed, via an inverse quantization performed by the Q "1 module and an inverse transformation performed by the module T 1 and added to the prediction determined by the prediction module P to reconstruct the current block.
• the reconstructed current block is then stored in order to reconstruct the current image and this reconstructed current image can be used as a reference when encoding subsequent images of the 360 ° omnidirectional video.
• the coding has been described here in the case of a single view k encoded in a base layer.
• the method is easily transposable to the case of several views coded in as many basic layers.
• Each image reconstructed at a time t of a base layer is projected onto the same reference image 1 ref of the 360 ° video to encode an image of the 360 video at time t.
• FIG. 4B shows the simplified structure of a coding device COD adapted to implement the coding method according to any one of the particular embodiments of the invention described above.
• Such an encoding device comprises a memory MEM4, a processing unit UT4, equipped for example with a processor PROC4.
• the coding method is implemented by a computer program PG4 stored in memory MEM4 and driving the processing unit UT4.
• the computer program PG4 includes instructions for implementing the steps of the encoding method as described above, when the program is executed by the processor PROC4.
• the code instructions of the computer program PG4 are for example loaded into a memory (not shown) before being executed by the processor PROC4.
• the processor PROC4 of the processing unit UT4 implements in particular the steps of the coding method described in relation with FIGS. 1A or 4A, according to the instructions of the computer program PG4.
• the coding method is implemented by functional modules (P, T, Q, Q "1 , T " 1 , C, PROJ).
• the processing unit UT4 cooperates with the various functional modules and the memory MEM4 in order to implement the steps of the coding method.
• the memory MEM4 can in particular comprise the memories MEM b , MEM e .
• a functional module may include a processor, a memory, and program code instructions for implementing the function corresponding to the module when the code instructions are executed by the processor.
• a functional module can be implemented by any type of suitable encoding circuits, such as for example and without limitation microprocessors, signal processing processors (DSP for Digital Signal Processor in English), integrated circuits specific to applications (ASICs for Application Specified Integrated) Circuit in English), FPGA circuits for Field Programmable Gate Arrays in English, logic unit wiring.
• DSP Digital Signal Processor
• ASICs Application Specified Integrated Circuit
• FIG. 5A illustrates in more detail the decoding steps of a base layer and a process improvement layer described above according to a particular embodiment of the invention.
• the case of the decoding of an enhancement layer EL coding a 360 ° omnidirectional video by prediction from a base layer BL [k] coding a view k is described here.
• the k view and the 360 ° omnidirectional video are decoded frame by frame and block by block.
• the data of the base layer BL [k] are decoded by an entropy decoding module D.
• a prediction residue is reconstructed via an inverse quantization of coefficients decoded entropically by an inverse quantization module Q "1 and an inverse transformation by an inverse transformation module T 1.
• a prediction module P determines a prediction for the current block from the signaling data decoded by the entropy decoding module D. The prediction is added to the reconstructed prediction residue to reconstruct the current block.
• the reconstructed current block is then stored in order to reconstruct the current image and that this reconstructed current image is stored in the reference image memory of the base layer MEM b and that it can serve as a reference during the decoding of the current image. following images of view k.
• a projection module PROJ carries out a geometric projection of the reconstructed image in the reference image 1 ref of the 360 ° omnidirectional video, as illustrated in FIG. 2B and according to FIG. the geometric transformation described above.
• the reference image 1 ref obtained by projection of the reconstructed image of the base layer is stored in the reference image memory of the enhancement layer MEM e .
• the data of the enhancement layer EL are decoded by an entropy decoding module D.
• a prediction residue is reconstructed via an inverse quantization of the decoded coefficients entropically implemented. by an inverse quantization module Q "1 and an inverse transformation implemented by an inverse transformation module T " 1 .
• a prediction module P determines a prediction for the current block from the signaling data decoded by the entropic decoding module D.
• the decoded syntax data indicates that the current block B e c is coded by inter-layer prediction with respect to a block co-located in the reference picture l ref obtained from the base layer.
• the prediction module therefore determines that the prediction corresponds to the block co-located at the current block B e c in the reference image l ref .
• the prediction is added to the reconstructed prediction residue to reconstruct the current block.
• the reconstructed current block is then stored in order to reconstruct the current image of the enhancement layer.
• This reconstructed image is stored in the reference image memory of the enhancement layer MEM e to serve as a reference when decoding subsequent images of the 360 ° video.
• FIG. 5B shows the simplified structure of a decoding device DEC adapted to implement the decoding method according to any one of the particular embodiments of the invention described above.
• Such a decoding device comprises a memory MEM5, a processing unit UT5, equipped for example with a processor PROC5.
• the decoding method is implemented by a computer program PG5 stored in memory MEM5 and driving the processing unit UT5.
• the computer program PG5 includes instructions for implementing the steps of the decoding method as described above, when the program is executed by the processor PROC5.
• the code instructions of the computer program PG5 are for example loaded into a memory (not shown) before being executed by the processor PROC5.
• the processor PROC5 of the processing unit UT5 notably implements the steps of the decoding method described in relation with FIG. 3 or 5A, according to the instructions of the computer program PG5.
• the decoding method is implemented by functional modules (P, Q "1 , T 1 , D, PROJ), for which the UT5 processing unit cooperates with the different functional modules and the memory MEM5 in order to implement the steps of the decoding method
• the memory MEM5 may in particular comprise the memories MEM b , MEM e .
• a functional module may include a processor, a memory, and program code instructions for implementing the function corresponding to the module when the code instructions are executed by the processor.
• a functional module can be implemented by any type of suitable encoding circuits, such as, for example, and without limitation microprocessors, signal processing processors (DSPs for Digital Signal Processor), application-specific integrated circuits (ASICs for Application Specific Integrated Circuit), FPGAs for Field Programmable Gate Arrays in English, cabling logical units.
• DSPs Digital Signal Processor
• ASICs Application Specific Integrated Circuit
• FPGAs Field Programmable Gate Arrays in English, cabling logical units.
• the blocks of an image of the enhancement layer are coded in groups of blocks, such a group of blocks is also called a tile.
• Each group of blocks i.e. each tile is coded independently of the other tiles.
• Each tile can then be decoded independently of other tiles.
• Such tiles TE0-TE1 1) are illustrated in FIG. 6A showing an image of the 360 ° omnidirectional video at a time instant in which 12 tiles are defined and completely cover the image.
• independent coding of the tiles here is meant a coding of the blocks of a tile not using spatial prediction from a block of another tile of the image, or of temporal prediction from a block a tile of the reference image not co-located with the current tile.
• Each tile is coded by temporal or inter-layer prediction from one or more of base layers as illustrated in FIGS. 6A and 6B.
• the tiles TE4 and TE7 are coded by inter-layer prediction with respect to the image projected in the reference image 1 ref of the view 1 and the tiles TE3 and TE6 are coded by inter-layer prediction. relative to the image projected in the reference image l ref of the view 2.
• a receiver adapted to decode and render a 360 ° video can only decode the tiles necessary for the current area of the 360 ° image displayed by a user. Indeed, when rendering a 360 ° video, a user can not view at a time t, the entire image of the video, ie he can not look in all directions at once and only a moment t that the area of the image in front of his eyes.
• such a viewing area is represented by the area ZV in Figure 6A.
• step 31 only the base layers having served to predict the area visualized by the user are decoded in step 31.
• step 31 only the layer of base corresponding to the view 1 is decoded in the step 31, and only the tiles TE4, TE5, TE7 and TE8 are decoded in the step 35 of Figure 3, from the enhancement layer EL.
• step 35 only the part of the image of the enhancement layer corresponding to the tiles TE4, TE5, TE7 and TE8 is reconstructed.
• a improvement layer tile EL can be prediction coded from several base layers, depending for example on rate / distortion optimization choices made during block coding of the enhancement layer, a block of a tile that can be prediction coded with respect to a first base layer, and another block of the same tile that can be encoded by another base layer distinct from the first base layer. In this case, all the base layers used to predict the blocks of a tile of the enhancement layer must be decoded.
• the coded data stream includes for each tile of the enhancement layer information identifying the base layers used to predict the tile. For example, for each tile, syntax elements indicating the number of base layers used and an identifier of each base layer used are encoded in the data stream. Such syntax elements are decoded for each tile of the enhancement layer to be decoded during step 35 of decoding the enhancement layer.
• such a reference image has undefined areas, for example set to 0 by default, large size, then using memory resources unnecessarily.
• the reconstructed images of the base layers projected on the enhancement layer can be stored in reference subpictures.
• a subimage can be used for each base layer.
• Each subimage is stored in association with offset information enabling the encoder and / or decoder to determine the location of the subimage in the enhancement image.
• Such a variant can be implemented independently to the encoder and / or the decoder.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=61187409

Family Applications (1)

Country Status (5)

Families Citing this family (1)

Citations (3)

Family Cites Families (22)

• 2017
  • 2017-10-19 FR FR1759822A patent/FR3072850B1/fr active Active
• 2018
  • 2018-10-12 WO PCT/EP2018/077922 patent/WO2019076764A1/fr not_active Ceased
  • 2018-10-12 US US16/756,755 patent/US11172223B2/en active Active
  • 2018-10-12 EP EP18783032.8A patent/EP3698546A1/de active Pending
  • 2018-10-12 CN CN201880068327.9A patent/CN111357292B/zh active Active
• 2021
  • 2021-10-13 US US17/500,362 patent/US11736725B2/en active Active

Patent Citations (3)

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events