JP7439251B2 - Adjustable tradeoff between quality and computational complexity in video codecs - Google Patents

Description

Cross-reference to related applications This application is based on U.S. Provisional Patent Application No. 62/916,579, filed on October 17, 2019, and European Patent Application No. 19203773.7, filed on October 17, 2019. each of which is incorporated herein by reference in its entirety.

TECHNICAL FIELD This disclosure relates generally to images. More particularly, one embodiment of the present disclosure relates to an adjustable trade-off between quality and computational complexity in video codecs.

As used herein, the term "dynamic range" (DR) refers to the range of intensities (e.g., luminance, luma) in an image, e.g., from the darkest black (dark) to the brightest white (highlight). It may be related to the ability of the human visual system (HVS) to perceive. In this sense, DR relates to "scene reference" intensity. DR may also relate to the ability of a display device to fully or approximately render an intensity range of a particular width. In this sense, DR relates to "display reference" intensity. Unless a specific meaning is explicitly provided at any point in the description to have a specific meaning, the terms may be used interchangeably in either sense, e.g. It should be assumed that

As used herein, the term "high dynamic range (HDR)" refers to the DR width of the human visual system (HVS), which spans approximately 14-15 orders of magnitude or more. In practice, DR, which allows humans to simultaneously perceive a wide range of intensities, may be somewhat truncated in relation to HDR. As used herein, the term enhanced dynamic range (EDR) or visual dynamic range (VDR) refers to the human visual system (HVS), which allows light adaptation changes across a scene or image to include eye movements. can be individually or interchangeably related to the DR perceptible within a scene or image. As used herein, EDR can refer to 5-6 orders of magnitude DR. Therefore, although EDR is probably somewhat narrow in relation to true scene-based HDR, EDR nevertheless represents a wide DR width and may be referred to as HDR.

In reality, an image contains one or more color components of a color space (e.g. luminance Y and chroma Cb and Cr), each color component represented by n bits of precision per pixel (e.g. n=8). be done. Using nonlinear intensity coding (e.g., gamma encoding), images with n≦8 (e.g., color 24-bit JPEG images) are considered standard dynamic range images, and images with n>8 are considered as enhanced dynamic range images. I can do it.

The reference electro-optic transfer function (EOTF) for a given display describes the relationship between the color values of the input video signal (e.g., luminance) and the color values of the screen produced by the display (e.g., screen luminance). characterize. For example, ITU Rec. ITU-R BT. 1886 "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011), which is incorporated herein by reference in its entirety. It is incorporated into the EOTF standard for flat panel displays. Define. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. The term "metadata" herein refers to any auxiliary information sent as part of a coded bitstream that assists a decoder in rendering the decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as described herein.

The term "PQ" as used herein refers to perceptual luminance amplitude quantization. The human visual system responds to increases in light level in a highly non-linear manner. A person's ability to see a stimulus is influenced by the brightness of the stimulus, the size of the stimulus, the spatial frequencies that make up the stimulus, and the brightness level to which the eye has adapted when viewing the stimulus. In some embodiments, the perceptual quantization function maps linear input gray levels to output gray levels that better match contrast sensitivity thresholds in the human visual system. Exemplary PQ mapping functions are described in SMPTE ST 2084:2014 “High Dynamic Range EOTF of Mastering Reference Displays” (“SMPTE”), which is incorporated herein by reference in its entirety and herein incorporated by reference. , given a fixed stimulus size, for each luminance level (e.g. stimulus level), the minimum visible contrast step at that luminance level is determined according to the most sensitive adaptation level and the most sensitive spatial frequency (HVS model ) is selected.

Displays that support brightness between 200 and 1,000 cd/m ² or nits typically exhibit lower dynamic range (LDR), also referred to as standard dynamic range (SDR) in conjunction with EDR (or HDR). EDR content can be displayed on an EDR display that supports higher dynamic range (eg, 1000 to 5000 nits or more). Such displays may be defined using alternative EOTFs that support high brightness capabilities (eg, 0 to 10,000 nits or more). Examples of such EOTFs are SMPTE 2084 and Rec. ITU-R BT. 2100 “Image parameter values for high dynamic range television for use in production and international program exchange” (2 (June 2017). As appreciated by the inventors herein, improved techniques for structuring video content data that can be used to support the display capabilities of a wide range of SDR and HDR display devices are desired. There is.

The approaches described in this section are approaches that could be pursued, but are not necessarily approaches that have been previously devised or pursued. Therefore, unless otherwise noted, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless specifically stated, it should not be assumed that any problems identified with respect to one or more approaches were recognized in any prior art based on this section.

An embodiment of the invention is illustrated by way of example, and not by way of limitation, in the accompanying drawings, in which: FIG. In the drawings, like reference numbers refer to like elements.

3 illustrates an example process of a video distribution pipeline. An example of the trade-off between the quality of the reconstructed image and the computational complexity of the generative mapping to create the reconstructed image is shown. An example codec is shown. An example codec is shown. An example codec is shown. An example of prediction error is shown. Figure 3 shows an example set of image processing operations to generate different color grades. An example of color correction that can be applied to different color grades is shown. 3 illustrates an example of color correction performed in conjunction with a mapping table update operation. An example of backward reshaping mapping generation with brightness update, backup lookup table modification, and color correction is shown. 3 illustrates an example process flow. 3 illustrates an example process flow. 3 illustrates an example process flow. 1 depicts a simplified block diagram of an example hardware platform on which a computer or computing device described herein may be implemented. FIG.

DESCRIPTION OF THE EMBODIMENTS In the following description, numerous specific details are set forth for purposes of explanation and to provide a thorough understanding of the disclosure. However, it will be obvious that the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices have not been described in exhaustive detail to avoid unnecessarily obscuring, obscuring, or obscuring the present disclosure.

Base layer (BL) image data with a relatively narrow dynamic range (e.g., relatively low bit depth, 8 bits, 10 bits, etc.) is used to generate/reconstruct HDR image data from the BL image data. It may be encoded into the coded bitstream along with metadata. The BL image data and image metadata can be provided to a receiving decoding and playback device, and the BL image data (e.g., SDR image data etc.), or reconstruct HDR image data from decoded BL image data and image metadata and display HDR image data on a relatively high dynamic range display (e.g., an HDR display). can be rendered.

In one example (eg, high fidelity, etc.) approach, the quality of the reconstructed HDR image data may be maximized at the expense of significantly increasing computational cost and video delivery delay. In another example (e.g., high efficiency, etc.) approach, the computational cost of generating the image metadata used to reconstruct HDR image data significantly reduces the quality of the reconstructed HDR image data. can be minimized at the expense of allowing A relatively large gap exists between high-fidelity and high-efficiency approaches in terms of decoder-side quality and encoder-side computational cost (or efficiency) of rendering reconstructed HDR images. Exemplary high-fidelity and high-efficiency image metadata generation is described in PCT Patent Application No. PCT/US2019/031620, filed May 9, 2019, the entire contents of which are incorporated herein by reference. Incorporated herein by reference as if fully set forth.

quality and computational complexity in order to better utilize the capabilities of different codecs and media processing systems, and also to provide greater flexibility to support different media consuming applications in a timely manner at the highest achievable quality. Coding tools that implement trade-offs between high-fidelity and high-efficiency approaches are provided under the techniques described herein to fill the relatively large gap between the aforementioned high-fidelity and high-efficiency approaches. For example, these coding tools assess the achievable decoder-side quality of reconstructed HDR image data and the encoder-side computational cost of producing the image metadata used to generate the reconstructed HDR image. can be used to provide a trade-off between

The content provider and/or content consumer has a resource budget (computation cost budget, end-to-end or individual system latency budget, end-to-end or individual delay budget, etc.) and a visual budget for the reconstructed HDR image data. Some or all of the coding tools can be selected based on specific quality goals or objectives.

Some or all of these coding tools may implement techniques to achieve color accuracy in both BL image data and reconstructed HDR image data. Additionally, optionally or alternatively, some or all of these coding tools implement techniques that reduce or reduce the risk of banding artifacts that are likely to occur in some operating scenarios (e.g., SDR image data, etc.) be able to. Banding reduction as described in U.S. Provisional Patent Application No. 62/885,921 filed August 13, 2019, also published as WO/2020/072651 in response to PCT/US2019/054299. For further improvements, this technique can be used, the entire contents of which are incorporated herein by reference.

Dynamic (e.g. image-dependent, image-specific, scene-dependent, scene-specific, etc.) forward mapping tables (e.g. mapping HDR codewords to corresponding SDR codewords) such as three-dimensional mapping tables (d3DMT) Forward reshaping mappings (e.g., forward reshaping functions/curves or polynomial sets, forward lookup tables or FLUTs, forward reshaping multivariate multiple regression (MMR) coefficients, etc.) A source HDR image can be generated and forward reshaped into an SDR image (or BL image data).

A backward mapping table (e.g. containing mapping pairs mapping an SDR codeword to a corresponding HDR codeword, etc.), such as a backward d3DMT, is used to create a backward reshaping mapping (e.g. backward reshaping function/curve or polynomial set). , backward lookup table or BLUT, backward reshaping MMR coefficients, etc.) to backward reshape the SDR image into an HDR image (or reconstructed HDR image data) that approximates the source HDR image. .

Image metadata specifying the forward reshaped SDR image and the backward reshaping mapping is encoded into the coded bitstream so that a receiving decoding/playback device can directly render the SDR image or Allows to render HDR constructed from backward reshaping mappings (backward reshaping).

The backward d3DMT can be constructed computationally efficiently from the corresponding forward mapping table (forward d3DMT). Additionally, optionally or alternatively, the posterior d3DMT is constructed directly using the source HDR image and the anterior reshaped SDR image (once the anterior reshaped SDR image is available) to map the reshaped Accuracy can be improved.

The d3DMT described herein may be updated with one of the appropriate subsets of channels or planes in the set of all channels or planes of the color space before being used to generate the backward reshaping mapping. . In some operational scenarios, the d3DMT may be updated only in the luminance (or luminance) channel/plane of the color space, thereby improving the color accuracy of the BL and/or reconstructed HDR image data to some extent. In some operational scenarios, the d3DMT may be updated with luminance and chromaticity (or chroma) channels/planes (e.g., all, etc.), thereby improving the color accuracy of the BL and/or reconstructed HDR image data. relatively significantly.

Therefore, the reshaping mapping accuracy, including but not limited to color accuracy and corresponding computational cost, can be reduced by updating only the luma channel/plane of the forward d3DMT or by updating both the luma and chroma planes of the forward d3DMT. or by directly constructing the posterior d3DMT from the source HDR image and the anterior reshaped SDR image.

Some or all of noise injection, BLUT modification, etc. may be performed to reduce banding artifacts. Exemplary noise injection and/or BLUT modifications are described in the aforementioned US Provisional Patent Application No. 62/885,921.

It is observed that the modification/update of luminance backward reshaping mapping like luminance BLUT may affect the color accuracy of the reconstructed HDR image and cause unintended changes in the color appearance of the reconstructed HDR image. .

Color correction operations as described herein include BLUT corrections (e.g., after mitigating banding artifacts) such that the colors in the reconstructed HDR image (e.g., after mitigating banding artifacts) more closely resemble the colors in the source HDR image. , used to mitigate banding artifacts) can be implemented or performed to correct the color of pixels affected by the banding artifacts. Additionally, optionally or alternatively, the noise injection operation is improved to make the injected noise more visually pleasing.

Example embodiments described herein relate to generating and encoding backward reshaping mappings for image reconstruction. The backward reshape mapping table is initially generated as the inverse of the forward reshape mapping table. The forward reshape table is used to generate a forward reshape mapping to generate a first dynamic range forward reshape image from a second dynamic range source image. The first dynamic range is lower than the second dynamic range. The forward reshaping mapping table includes a second dynamic range sampled luma codeword, a second dynamic range sampled chroma codeword, and a second dynamic range sampled luma codeword corresponding to the second dynamic range sampled luma codeword. a content-mapped luma codeword of one dynamic range; and a content-mapped chroma codeword of a first dynamic range corresponding to a sampled chroma codeword of a second dynamic range. The backward reshaping mapping table is updated by replacing the content mapping intensity codeword with the forward reshaping intensity codeword. The forward reconstructed luminance codeword is generated by applying luminance forward mapping to the sampled luminance codeword of the second dynamic range. The brightness forward mapping consists of a forward reshaping mapping table. A backward reshaping mapping table for creating a second dynamic range reconstructed image from the first dynamic range forward reshaping image is generated using the backward reshaping mapping table and the luminance forward mapping. The forward reshaping image is encoded within the video signal along with image metadata that specifies the backward reshaping mapping. A device receiving the video signal applies the backward reshaping mapping to the forward reshaping image to generate a second dynamic range reconstructed image.

Example embodiments described herein relate to decoding backward reshaping mappings for image reconstruction and rendering. A first dynamic range forward reshaped image is decoded from the video signal. Image metadata including backward reshaping mapping is decoded from the video signal. The backward reshaping mapping was generated by an upstream image processing device from the backward reshaping mapping table and the luminance forward mapping. The backward reshaping mapping table was initially generated by an upstream image processing device as the inverse of the forward reshaping mapping table that generates the luminance forward mapping. The content mapping intensity codeword in the backward reshaping mapping table has been updated with the forward reshaping intensity codeword. The forward reshaped luminance codeword was generated by applying luminance forward mapping to the sampled source luminance codeword in the source image. Apply backward reshaping mapping to the forward reshape image to generate a reconstructed image. The displayed image is derived from a reconstructed image rendered using a display device.

Example Video Distribution Processing Pipeline FIG. 1A shows an example process of a video distribution pipeline (100) showing various stages from video capture to video content display. A series of video frames (102) are captured or generated using an image generation block (105). Video frames (102) may be captured digitally (eg, by a digital camera) or generated by a computer (eg, using computer animation) to provide video data (107). Further, optionally or alternatively, the video frames (102) may be captured on film by a film camera. The film is converted to digital format to provide video data (107). In the production phase (110), video data (107) is edited to provide a video production stream (112).

The video data of the production stream (112) is then provided to a processor for post-production editing (115). Post-production editing (115) adjusts or modifies the color or brightness in specific areas of the image to improve the quality of the image or achieve a particular look of the image, according to the creative intent of the video creator. It may also include doing. This is sometimes called "color timing" or "color grading." Other edits (e.g., scene selection and sequencing, manual and/or automatic scene cut information generation, image cropping, addition of computer-generated visual special effects, etc.) may be performed on HDR images (117-1) or SDR (or comparison Post-production editing (115) may be performed to generate a release version (e.g., SDR, etc.) of the image (117). In some embodiments, during post-production editing (115), the HDR image (117-1) is highly dynamically edited by a colorist performing post-production editing operations on the HDR image (117-1). Viewed on a reference HDR display that supports range. Additionally, optionally, or in the alternative, during post-production editing (115), the SDR image (117) is configured to have a standard dynamic range (or the reference display (125) supporting a relatively narrow dynamic range). Additionally, optionally or alternatively, the SDR image (117) may be content mapped from the HDR image (117-1).

In some embodiments, the coding block (120) provides various adjustable trade-offs between the decoder-side quality of the decoded HDR image data and the encoder-side computational complexity of the backward reshaping mapping. Can be implemented in part or in full. The coding block (120) receives the HDR image (117-1) from the post-production edit (115) and forwardly reshapes the HDR image (117-1) into a (forward)reformed SDR image. .

The reshaped SDR image may be compressed/encoded by a coding block (120) into a coded bitstream (122), eg, in a single layer. Exemplary single-layer video coding operations are described in U.S. patent application Ser. The entire contents are incorporated herein by reference.

In some embodiments, the coding block (120) is configured to generate a coded bitstream (122), such as those defined by ATSC, DVB, DVD, Blu-Ray, and other distribution formats. May include audio and video encoders.

The reshaped SDR image is encoded into video data in a video signal (e.g., 8-bit SDR video signal, 10-bit SDR video signal, etc.) that is backward compatible with a wide range of SDR display devices (e.g., SDR displays, etc.) be able to. In a non-limiting example, the video signal encoded with the reshaped SDR image may be a single layer backward compatible video signal. As used herein, a "single layer backward compatible video signal" may refer to a video signal that carries an SDR image that is specifically optimized for SDR display or color graded in a single signal layer.

In some embodiments, the coded bitstream (122) output by the coding block (120) includes image metadata, including, but not limited to, a backward reshaping mapping produced by the coding block (120). An output 8-bit SDR video signal can be represented along with the data. Under the techniques described herein, the backward reshaping mapping (or compositor metadata) is based on the decoder-side quality of the reconstructed HDR image data and the encoder-side computational complexity of the backward reshaping mapping. A particular set of adjustable trade-off options between generated based on

Backward reshaping mapping involves performing backward reshaping (e.g., inverse toning) on the reshaped SDR image to produce a backward reshaping image that can be optimized for rendering on HDR (e.g., reference) displays. mapping, etc.) by a downstream decoder. In some embodiments, the backward reshaping image is processed using one or more SDR-to-HDR conversion tools that perform inverse tone mapping based at least in part on the backward reshaping mapping (or composer metadata). It may be generated from a reshaped SDR image (or a decoded version thereof). As used herein, backward reshaping returns the requantized image to its original EOTF domain (e.g., gamma, PQ, hybrid log gamma or HLG, etc.) for further downstream processing such as display management. Refers to image processing operations that convert images into images. An exemplary posterior reshaping operation is described in U.S. Provisional Application No. 62/136,402, filed March 20, 2015 (also published January 18, 2018, as U.S. Patent Application Publication No. 2018/0020224). ), the entire contents of which are incorporated herein by reference as if fully set forth herein.

Additionally, optionally or alternatively, the coded bitstream (122) is used by a downstream decoder to perform display management operations on the backward reshaped image for HDR reference display and to perform non-reference HDR The images are encoded with additional image metadata including, but not limited to, display management metadata that can generate display images that are optimized for rendering on other displays, such as a display.

The coded bitstream (122) is then sent downstream to a receiver such as a decoding and playback device, media source device, media streaming client device, television set, set-top box, movie theater, etc. At the receiver (or downstream device), the coded bitstream (122) is decoded by a decoding block (130) to produce a decoded image 182. This may be the same as the reshaped SDR image, subject to quantization errors caused in the compression performed by the coding block (120) and the decompression performed by the decoding block (130). .

In an operating scenario in which the receiver operates with (or is attached to) a target display 140 that supports a standard dynamic range or a relatively narrow dynamic range equal to or less than the standard dynamic range, the decoding block (130) decodes the coded bits. A reshaped SDR image can be decoded from the stream (122) and used directly or indirectly for rendering on a target display (140). In embodiments where the target display (140) has similar characteristics to the SDR reference display (125), the reshaped SDR image is directly viewable on the target display (140).

In some embodiments, the receiver operates with (or is attached to) an HDR target display 140-1 that supports high dynamic range (e.g., 400 nits, 1000 nits, 4000 nits, 10,000 nits or more). By extracting the constructor metadata from the encoded bitstream (122) and using the backward reshaping metadata (composer metadata), A backward reshaped image 132-1 is created from the reshaped SDR image by rearwardly reshaping the SDR image reshaped, and the backward reshaped image (132-1) is directly or indirectly set as an HDR target. It can be used for rendering on the display (140-1).

The backward reshaped image (132-1) is displayed on an HDR display (e.g., a reference) that is not identical and supports a value greater than the maximum or peak brightness value of the HDR target display (140-1). can be optimized for A display management block (such as 135-1) in the receiver, HDR target display (140-1), or another device generates a display map signal (137-1) adapted to the characteristics of the HDR target display (140-1). 1), the backward reshaped image (132-1) is further adjusted to match the characteristics of the HDR target display (140-1).

Video Codecs and Quality-Complexity Tradeoffs FIG. 1B shows an example of the tradeoff between the decoder side of the quality of the reconstructed HDR image and the encoder side computational complexity of the generation of the backward reshaping mapping. The coding blocks described herein (e.g., video encoders, video transcoders, media streaming systems, content delivery systems, etc.) may be selected by content providers and/or content consumers operating on video processing systems that include the coding blocks. (e.g., dynamically, statically, adaptively, scheduled, independently, etc.).

As shown in Figure 1B, example tradeoffs include enabling or disabling (turning on or off) any of (a) noise injection, (b) BLUT modification, (c) color correction, etc. , and (d) selecting different options in updating (or building) the d3DMT. In some operational scenarios, enabling BLUT modification and color correction can alleviate banding artifacts, improve color appearance, and improve the overall quality of reconstructed HDR images. .

Exemplary trade-off options (e.g., selectable or adjustable in coding blocks) in updating (or constructing) a d3DMT include, but are not necessarily limited to: (1) not performing a forward update of the d3DMT; (2) updating the luma channels/planes in front of the d3DMT; (3) updating the luma channels/planes and chroma channels/planes in front of the d3DMT; (4) updating the d3DMT based on the source HDR image and the forward reshaped SDR image. This can include backward construction, etc. As shown in FIG. 1B, these different options provide different color grades of decoder-side quality of the decoded HDR image and/or different complexity (or efficiency) of encoder-side generation of the backward reshaping mapping.

2A and 2B illustrate an example video codec, such as a video encoder (eg, coding block (120) of FIG. 1A) in an end-to-end video distribution pipeline. An end-to-end video delivery pipeline as described herein, or a device therein, comprises one or more computing devices, such as software, hardware, or a combination of software and hardware. It may be implemented with more or fewer processing blocks than those shown in this document. By way of example and not limitation, some or all processing flows and/or data flows are marked with numbers or letters in parentheses. As shown in FIGS. 2A and 2B, any, some, or all of these processing blocks may be selected or selected (e.g., by a content provider user) to configure the video code within an end-to-end video delivery pipeline. may be arbitrary, depending on the particular set of trade-off options (e.g., as shown in FIG. 1B) selected or created (e.g., by a content consumer user, etc.).

As shown in FIGS. 2A and 2B, the video encoder described herein uses a "mapping information" block, a reversible SDR image (or forward reshaped SDR image), to generate HDR to SDR mapping information. a "forward reshape" block for generating image metadata (referred to as "RPU" or "rpu"); It may include a backward reshaping mapping to be used by the receiving device to reconstruct an HDR image that approximates the HDR image (eg, backward reshaping from reversible SDR).

In the video encoder of Figure 2A, some or all of (a) noise injection, (b) BLUT modification, and (c) color correction as shown in Figure 1B may be enabled, as shown in Figure 1B. One of options (d)(2) or (d)(3) for d3DMT update may be selected. As mentioned above, option (d)(2) in FIG. 1B corresponds to updating (e.g. only) the luminance channel/plane in the forward d3DMT, and option (d)(3) in FIG. Corresponds to updates of luma channel/plane and chroma channel/plane.

The "Mapping Information" block of FIG. 2A samples (or extracts) HDR color samples from a reference HDR image, and content mapping (CM; operations/actions performed based on artistic or colorist intent, for example). , color grading operations, etc.) to these HDR color samples to generate corresponding SDR color samples, etc. Examples of sampling strategies used to sample HDR color samples from a source or reference HDR image include, but are not limited to, any of the following: sampling, sampling a relatively small set of reference HDR images (e.g., 1/2, 1/4, 1/8, 1/16, etc.) uniformly or non-uniformly in the spatial dimensions of the reference HDR image; sampling, uniformly or non-uniformly sampling in the luminance distribution of the luminance codewords/values of the reference HDR image, etc.

The mapping information stage constructs a 3D sampling grid (e.g., a linear 3D grid, a sparse 3D grid, a 3D grid with uniform or non-uniform distribution/density of vertices, etc.) based at least in part on the sampling strategy. can do. In various embodiments, sample points may or may not be selected based on a 3D sampling grid. In some embodiments, sample points may be selected based on a density distribution, e.g., sample points may be selected based on the presence or absence of color present in the image, as indicated by the density distribution. good. The CM may be used to generate a content mapping (eg, only, etc.) from HDR sample points to corresponding SDR sample points. Content mapping is not content mapping from all HDR pixels (e.g. 2 million pixels or more for a 1920x1080 spatial resolution image) to all corresponding SDR pixels, but rather a relatively small number of sample points (e.g. 10, 000 sample points) and can therefore be much lighter to generate than all content mappings for all pixels.

HDR (e.g., color, codeword, etc.) samples and corresponding SDR (e.g., color, codeword, etc.) samples are used as samples to construct or generate forward d3DMT (e.g., multidimensional mapping table, histogram, etc.) Used to form mapping pairs. The "Mapping Information" block of FIG. 2A further includes information on whether the character box is present in the HDR image and, if so, the pixel values of the character box. It may include collecting box information.

A video encoder may implement a High Efficiency Reduction Reference (HERR) codec architecture to reduce computational complexity, as described in PCT/US2019/031620. Mapping side information - instead of a reference SDR image containing pixel values of individual pixels (e.g. all, substantially all, etc.) - is transferred from the mapping information stage to the forward reshaping stage of the video encoder. , to a subsequent processing stage, such as a posterior reshaping stage. In a HERR encoding operation, the source (or reference) HDR image and the mapping side information related to the source (or reference) HDR image are used in a later processing step to perform a forward reshaping function (e.g., FLUT, MMR coefficients, etc.). ) and generate a (forward) reshaped SDR image by forwardly reshaping the source HDR image based on a forward reshaping function, and generate a reconstructed HDR image that approximates the source HDR image. generate backward reshaped image metadata for use by a receiving device (or receiver video decoder) to encode the reshaped SDR image along with the backward reshaped image metadata in the SLBC video signal. In some operational scenarios, the information on the mapping side can reduce the number of colors compared to what is represented in a full reference SDR image.

In High Fidelity Full Reference (HFFR) mode, described in PCT/US2019/031620, a reference SDR image is generated by applying content mapping (e.g. color mapping) to each pixel of the source HDR image, and the chromaticity Used to construct d3DMT for anterior reshaping. In HFFR mode, the reference SDR image is not encoded into the SLBC video signal and serves as an approximate reference for constructing the d3DMT for chromatic forward reshaping. Therefore, in HFFR mode, there are many encoding-related operations performed at each individual pixel level (eg, each of the million pixels in the image).

In contrast, in the HERR mode described in PCT/US2019/031620, d3DMT generates a reduced number of points (e.g., 10, 000 points, much less than 1 million points, etc.) and can be generated from information on the mapping side, such as content mapping information (eg, color mapping information, etc.). As a result, large amounts of computation or related operations can be saved or avoided.

個の３Ｄビンがある。正規化されていない最小コードワード値を、

と示し、正規化されていない最大コードワード値を、

と示し、ここで

である。ビン

の範囲は次のように表される。

ここで、

であり、

であり、

はＨＤＲ（またはＥＤＲ）のビット深度である。 As in the full reference mode, to generate the mapping side information in the mapping information stage of the reduced reference mode, the codewords in the source HDR image (referred to as frame t) are It can be divided into Q bins based on the minimum and maximum codeword values for each channel. as a whole,

There are 3D bins. The minimum unnormalized codeword value is

, and the maximum unnormalized codeword value is

and here

It is. bottle

The range of is expressed as follows.

here,

and

and

is the HDR (or EDR) bit depth.

として示される）は、ビンインデックス

によって指定される３Ｄヒストグラム内の各３Ｄビンとともに収集される。各３ＤビンにおけるＨＤＲ画素値のチャネル固有の和（

として示される）がそれぞれ計算される。（３Ｄヒストグラム

非ゼロの画素数を有する空でないビンについては、チャネル固有の平均ＨＤＲ画素値（またはコードワード値）（

として示される）は、それぞれ、すべてのカラーチャネルについて計算することができ、空でないビンに表されるＨＤＲ画素値についてのマッピングされたＳＤＲ画素値は、それぞれ、すべてのカラーチャネルについての平均ＨＤＲ画素値におけるコンテンツマッピングを用いて決定または計算することもできる。

をＫｔ個のビンとして示し、ここで、

である。マッピングされたＳＤＲ画素値を

として示し、チャネル固有の平均ＨＤＲ画素値

およびマッピングされたＳＤＲ画素値

などのマッピング統計を収集するための例示的な手順を、下記の表１に示す。

3D histogram (

) is the bin index

is collected with each 3D bin in the 3D histogram specified by . Channel-specific sum of HDR pixel values in each 3D bin (

) are calculated respectively. (3D histogram

For non-empty bins with non-zero number of pixels, the channel-specific average HDR pixel value (or codeword value) (

) can be computed for all color channels, respectively, and the mapped SDR pixel values for HDR pixel values represented in non-empty bins are the average HDR pixel values for all color channels, respectively. It can also be determined or calculated using content mapping in values.

is denoted as Kt bins, where

It is. The mapped SDR pixel value

and the channel-specific average HDR pixel value

and the mapped SDR pixel value

An exemplary procedure for collecting mapping statistics such as is shown in Table 1 below.

の非ゼロのビンにのみ適用される可能性がある。サンプリングされたＨＤＲ及びＳＤＲコードワード統計

を含む３Ｄヒストグラム

によって表されるｄ３ＤＭＴは、ヒストグラム

と共に、マッピング情報ステージから前方再整形ステージ及び／又は後方再整形ステージへのマッピング側情報として送られ、前方再整形ステージ及び後方再整形ステージにおける前方及び後方再整形関数を構築するために使用され得る。 As seen in Table 1 above, unlike the full reference mode, the mapped SDR pixel values in the reduced reference mode are calculated using the HDR pixel values at the sampled points instead of averaging the individual SDR pixel values in the reference SDR image. Obtained by applying content mapping (such as color mapping) to pixel values. Such content mapping is a 3D histogram representing d3DMT.

May only apply to non-zero bins. Sampled HDR and SDR codeword statistics

3D histogram including

The d3DMT represented by the histogram

and may be sent as mapping side information from the mapping information stage to the forward reshaping stage and/or the backward reshaping stage and used to construct the forward and backward reshaping functions in the forward reshaping stage and the backward reshaping stage. .

The "forward reshaping" block of FIG. 2A uses the forward d3DMT to analyze the HDR luminance samples (e.g., the luminance components of the HDR samples) obtained from the forward d3DMT and the SDR luminance samples (e.g., the luminance components of the corresponding SDR samples). etc.) and applying cumulative density function (CDF) matching to form or generate forward look-up tables (FLUTs) for the luminance channels/planes, etc. Exemplary CDF matching operations include PCT Application No. PCT/US2017/50980, filed on September 11, 2017; (also published on April 5, 2018 as Patent Application Publication No. 2018/0098094), the entire contents of which are incorporated herein by reference.

In some embodiments, CDF matching may be used to construct a forward reshaping lookup table. The 1D intensity histogram is generated from source HDR codeword values and SDR codeword values at sampling points (e.g., sampling grids in each of the HDR and SDR image frames), rather than from the source HDR image and the reference SDR image at each unsampled level. It may be constructed using the generated d3DMT. Since the 3D histogram representing the d3DMT is already available in the mapping side information from the mapping information stage, the forward modification stage sums all 3D bins of the 3D histogram whose luminance values correspond to the same luminance bin of the 1D luminance histogram. By doing so, a 1D brightness histogram can be constructed.

CDF matching may be performed by a CDF matching block to generate an interpolated FLUT based on the 1D intensity histogram. The interpolated FLUT may be smoothed to produce a smoothed FLUT. Additionally, a backward reshaping LUT (BLUT) may be constructed by the BLUT construction block 456 using a codeword mapping or a curve (e.g., an 8-piece quadratic polynomial, etc.) represented in the smoothed FLUT. good.

Two exemplary methods or procedures for constructing 1D intensity histograms with different computational costs are shown in Tables 2 and 3 below. In the first method, shown in Table 2, the centroid of each bin is calculated. CDF matching is performed using centroids. This requires relatively expensive computation, but produces a relatively accurate mapping. In the second method, shown in Table 3, each bin is represented by its midpoint, which can be easily determined with relatively low cost calculations. Two methods are shown below.

In some operational scenarios, block standard deviation (denoted "BLKSTD") is calculated from HDR images. The risk of banding artifacts is estimated from BLKSTD (calculated from HDR images) and FLUT (created with CDF matching). Banding artifacts and/or noise depending on the risk of brightness levels of the HDR image may be injected (in the "(a) Noise injection" sub-block) into the dark areas of the HDR image or into subranges of the HDR brightness channels/planes.

Exemplary block standard deviation calculations and banding artifact risk estimation are described in U.S. Pat. No. 10,032,262, the entire contents of which are incorporated herein by reference. incorporated herein by.

Additionally, the letterboxing operation performs a letterboxing operation to process any letterboxing that may be present in the reference HDR image to forward the reshaped SDR image and/or the reconstructed HDR image. We can help ensure accurate colors within. Examples of letterboxing operations are described in the aforementioned PCT Patent Application No. PCT/US2019/031620.

By applying a FLUT to the dithered (or noise-injected) HDR luminance channels/planes of the HDR image, the BL luminance channels/planes/components of the corresponding forward reshaped SDR image can be generated.

By applying the forward MMR coefficients to the HDR chroma channels/planes of the HDR image, the BL chroma channels/planes/components of the corresponding forward reshaped SDR image can be generated. Forward MMR coefficients (representing chromatic forward reshaping mapping) can be calculated from the forward d3DMT and character box information.

The d3DMT from the mapping information stage may be received by unconstrained MMR matrix construction in the forward reshaping stage. The unconstrained least squares problem combines the chroma codewords in the source HDR image (including any letterboxed source HDR chroma codewords, if applicable) with the reshaped chroma codewords in the reshaped SDR image (including any letterboxed source HDR chroma codewords, if applicable). can be formulated to solve for the MMR coefficients used to transfer any letterbox-reformed SDR (including chroma codewords).

From the d3DMT provided in the mapping side information, two vectors can be constructed as follows using the SDR chroma values of the non-zero (or non-empty) bins of the 3D histogram representing the d3DMT:

ここで、

これには、サポートされるすべてのＭＭＲ項が含まれている。 Additionally, a matrix can be constructed using the average HDR pixel values of non-zero bins as follows:

here,

This includes all supported MMR terms.

Assume that the following relational expression holds.

The unconstrained MMR coefficients are obtained in closed form by solving an unconstrained least squares problem with the unconstrained MMR matrix as follows:

The "backwards reshaping" block of FIG. 2A uses a histogram-based method, such as the histogram-enhanced BLUT construction method described in the aforementioned PCT patent application no. Build a backward lookup table and perform BLUT correction (referred to as "(b)") to reduce the risk of banding artifacts in the reconstructed HDR image generated from backward reshaping of the forward reshape SDR image reduce or reduce

The FLUT and forward MMR coefficients generated by the "Forward Reshaping" block of FIG. and/or used to update SDR codewords/samples in chroma channels/planes. The modified BLUT (generated in the "(b) BLUT modification" sub-block) is used in the "(c) Color correction" sub-block to modify the HDR chroma codeword in the backward d3DMT. (e.g., values, samples, etc.), thereby improving the color appearance of the reconstructed HDR image. The backward reshaping MMR coefficient can be calculated from the updated backward d3DMT and character box information.

The d3DMT may be received by an unconstrained MMR matrix configuration in the backward reshaping stage. The unconstrained least squares problem combines the chromaticity codewords in the reconstructed SDR image (including the letterboxed SDR chromaticity codeword, if applicable) with the chromaticity code in the reconstructed HDR image. can be formulated to solve for the MMR coefficients used to restore the word (including the letterbox reconstructed HDR chroma codeword, if applicable).

From the d3DMT, two vectors can be constructed using the average HDR chroma values of the non-zero (or non-empty) bins of the 3D histogram representing the d3DMT as follows:

ここで、

これには、サポートされるすべてのＭＭＲ項が含まれている。 Additionally, a matrix can be constructed using the SDR pixel values of non-zero bins as follows:

here,

This includes all supported MMR terms.

Suppose that the following relational expression holds.

The unconstrained MMR coefficients are obtained in closed form by solving an unconstrained least squares problem with the unconstrained MMR matrix as follows:

The backward reshape mapping, which includes (or specifies) the BLUT and the backward reshape MMR coefficients, is a bit coded as part of the image metadata (e.g., "rpu", etc.) that accompanies the reshaped SDR image. May be output to a stream.

Each of some or all of the trade-off options shown in FIG. They can be individually selected or fabricated to configure the encoder. A first trade-off option may be made to enable/disable noise injection by retaining/removing processing subblock (a) of FIG. 2A. A second trade-off option can be made to enable/disable BLUT modification by retaining/removing processing subblock (b) of FIG. 2A. A third trade-off option may be to enable/disable color correction by retaining/removing processing sub-block (c) of FIG. 2A. Several trade-off options can be made regarding d3DMT updates. For example, a fourth trade-off option can be made to enable/disable processing sub-block (d) of FIG. 2A. A fifth trade-off option may be made to preserve data flow (2) and processing sub-block (d) of FIG. 2A. A sixth trade-off option may be made to preserve data flow (3) and processing sub-block (d) of FIG. 2A.

A seventh trade-off option is shown in Figure 2B. Under this option, the backward d3DMT used to generate the backward reshaping mapping consists of the reference HDR image and the forward reshaping SDR image provided in the data flow shown as (4) in Figure 2B. be done.

In summary, adjustable tradeoff techniques as described herein may be used to generate backward reshaping mappings and support end-to-end video delivery between video codecs. According to the availability of computational resources on the encoder side, various trade-off options can be selected or made to configure the video encoder to achieve the optimal quality on the decoder side of the reconstructed HDR image. can.

The availability of computational resources on the encoder side and/or the decoder side is used (e.g., dynamically, statically, adaptively, etc.) to select or create certain adjustable trade-off options. obtain. Exemplary computational resources described herein include CPU consumption, DSP processing power, memory size, cache, data stores, network resources, latency, end-to-end video delivery pipeline delay, etc. Not limited to these.

FIG. 2C shows an example video codec, such as a video decoder (e.g., decoding block (130) of FIG. 1A), that may be used in one or more downstream video decoders (e.g., a receiver, etc.). It can also be implemented with several computing processors.

In some operating scenarios, such as shown in FIG. 2C, the video signal encoded by the (forward) reshaped SDR image and image metadata 152 in the single layer 144 is not necessarily used as input by the video decoder upstream. Including, but not limited to, backward shaping mappings generated by video encoders.

The decompression block 154 (eg, part of the decoding block (130) of FIG. 1A) decodes/decodes compressed video data within a single layer (144) of the video signal into a decoded SDR image (182). The decoded SDR image (182) may be identical to the reshaped SDR image, subject to quantization errors in the coding block (120) and restoration block (154), which is optimal for SDR display devices. It is possible that it may have been morphed. The decoded SDR image (182) is output to an output SDR video signal 156 and rendered to an SDR display device.

Further, a backward reshaping block 158 extracts a backward reshaping mapping from the input video signal, constructs an optimal backward reshaping function based on the extracted backward reshaping mapping in the image metadata (152), and generates a reconstructed HDR image. A backward reshaping operation is performed on the reshaped SDR image based on the optimal backward reshaping function to generate a backward reshaped HDR image (eg, 132-1 of FIG. 1A, approximation of the HDR reference image, etc.).

In some implementations, the backward reshaped HDR image represents a production quality or near production quality HDR image that is optimized for an HDR target/reference display device. The backward reshaped HDR image is output to an output HDR video signal 160 and rendered to an HDR display device. In some operations, DM may not be implemented at the receiver to reduce cost or latency.

Additionally, optionally or alternatively, in some operational scenarios, DM metadata may be transmitted to the receiver within the image metadata (152) and the reshaped SDR image. HDR display device-specific display management operations may be performed based at least in part on DM metadata within the image metadata (152), e.g., to generate an HDR display image to be rendered on the HDR display device. It may also be performed on a reshaped HDR image.

For purposes of illustration, a single layer codec architecture is described. Note that the techniques described herein can be used with different single-layer codec architectures than those shown in FIGS. 2A-2C. Additionally, optionally or alternatively, these techniques can be used in a multi-layer codec architecture. Accordingly, these and other variations of single-layer or multi-layer codec architectures are operable with some or all of the techniques described herein.

Updating d3DMT for Posterior Reshaping Many of the relatively large gaps between high-fidelity and high-efficiency approaches, such as those described in the aforementioned PCT Patent Application No. PCT/US2019/031620, are due to reconstruction Compute backward reshaping mappings (or functions) such as backward reshaping MMR coefficients for backward reshaping of SDR images in chroma channels/planes in terms of decoder side quality and encoder side computational cost (or efficiency) of HDR images. or may be due to the method used to generate it.

On the other hand, high-efficiency approaches calculate the backward reshaping MMR coefficients directly from the forward d3DMT, thereby introducing errors that significantly affect the quality of the reconstructed HDR image at the decoder side. This is because the anterior d3DMT is accurate enough for anterior reshaping, but may not be accurate enough for posterior reshaping.

Since it is performed in a single (luminance) channel/plane of color space, luminance reshaping may be prone to some error in the reshaped codeword/value. Additional errors may occur in MMR prediction for chroma reshaping.

FIG. 3A shows an example of prediction errors in Y, Cb (denoted as C0 or C0) and Cr (denoted as C1 or C1) caused by forward reshaping for each entry in the forward d3DMT. These prediction errors represent the difference or deviation between the original codeword/value in the forward d3DMT and the FLUT/MMR predicted codeword/value. Luminance prediction errors can be more significant than chromaticity prediction errors. These prediction errors further propagate to the backward reshaping, thereby causing relatively significant errors in the reconstructed HDR image.

On the other hand, high-fidelity approaches construct a new posterior d3DMT from a reference (or source) HDR and a corresponding SDR image (eg, a forward reshaped SDR image, etc.), thereby incurring significant computational cost and latency.

First solved/obtained the FLUT used to predict the SDR codeword/value in the luma channel/plane and the forward MMR coefficients used to predict the SDR codeword/value in the chroma channel/plane. . The backward d3DMT corresponding to the forward d3DMT is then combined with the FLUT predicted luma SDR codeword/value and the forward MMR predicted chroma codeword to correct or minimize the prediction error introduced by the forward reshaping in the backward reshape mapping. /value (generated by content mapping the HDR codeword/value/sample in the "Mapping Information" block of FIG. 2A or FIG. 2B) in the SDR entry of the forward d3DMT using can be generated by replacing or modifying the

ここでＫは、前方ｄ３ＤＭＴのエントリまたは行の総数であり、

は、マッピングテーブルのｋ番目のエントリのＹ、Ｃ０およびＣ１チャネル／平面におけるソースＨＤＲ値をそれぞれ示し、

は、前方ｄ３ＤＭＴのｋ番目のエントリのＹ、Ｃ０およびＣ１チャネル／平面におけるコンテンツマッピングＳＤＲ値を示す。ｋは０と（Ｋ－１）の間の整数である。 The mapping pair in the forward d3DMT generated from the HDR samples (or source HDR values of Y, C0 and C1 channels/planes) of the tth HDR reference image (or frame) and the content mapping of the HDR samples as follows: Describe the mapping pair of corresponding SDR samples (or content mapping SDR values) generated from:

where K is the total number of entries or rows in the forward d3DMT,

denote the source HDR values in the Y, C0 and C1 channels/planes of the kth entry of the mapping table, respectively;

denotes the content mapping SDR values in the Y, C0 and C1 channels/planes of the kth entry of the forward d3DMT. k is an integer between 0 and (K-1).

A mapping pair in the mapping table described herein is a pair of entries for HDR values (e.g., the left side of equation (1) above) and a corresponding entry for SDR values (e.g., the right side of equation (1) above). ) refers to the pair. Under the techniques described herein, these values in the mapping pair may be updated for the purpose of producing a relatively high quality backward reshaping mapping.

と記す。色度チャネル／平面Ｃ０およびＣ１におけるＨＤＲコードワード／値からＳＤＲコードワード／値を予測するために使用される前方ＭＭＲ係数（前方ｄ３ＤＭＴから生成される）を、それぞれ、

と記すと、以下のようになる：

ここで、ＭはＭＭＲに基づく前方再整形マッピングにおける項の総数を表す。 The forward FLUT (generated from the forward d3DMT) used to predict the SDR codeword/value from the HDR codeword/value in the luminance channel/plane,

It is written as The forward MMR coefficients (generated from the forward d3DMT) used to predict the SDR codewords/values from the HDR codewords/values in the chroma channels/planes C0 and C1, respectively, are

, it becomes as follows:

Here, M represents the total number of terms in the MMR-based forward reshaping mapping.

The backward d3DMT containing the SDR to HDR pair mapping can initially be generated as the inverse of the forward d3DMT. For example, for the purpose of constructing the backward d3DMT as the inverse of the forward d3DMT first, use the HDR values mapped to the SDR values in the forward d3DMT as the corresponding HDR values to which the SDR values in the backward d3DMT are mapped. be able to.

An exemplary procedure for replacing content-mapped SDR codewords/values in the luminance channel/plane with FLUT-predicted forward reconstructed SDR codewords/values in the backward d3DMT is shown in Table 4 below. .

In some operational scenarios, after updating the SDR brightness code/value of the backward d3DMT, which is first generated as the inverse of the forward d3DMT, a modified backward d3DMT from HDR to SDR can be generated as follows:

As mentioned above, to construct this backward d3DMT, the HDR values mapped to the SDR values in the forward d3DMT may be used as the corresponding HDR values to which the SDR values in the backward d3DMT are mapped.

An exemplary procedure for replacing content-mapped SDR codewords/values in chroma channels/planes with forward reshaped SDR codewords/values predicted with forward MMR coefficients in the backward d3DMT is shown in Table 5 below. Shown below.

In some operational scenarios, after updating the SDR chroma codewords/values in the backward d3DMT, which was first generated as the inverse of the forward d3DMT, a modified backward d3DMT from SDR to HDR may be generated as follows: can:

Additionally, optionally or alternatively, after updating the SDR luminance and chroma codewords/values of the backward d3DMT, an SDR to HDR modified backward d3DMT can be generated as follows:

ここで、

は、ＭＭＲ予測演算でサポートされるすべての（Ｍ個の）項を含む。 The backward MMR coefficients used to predict the reconstructed HDR codeword/value may be calculated from the modified backward d3DMT as expressed in equations (3) to (5). Take the updated backward d3DMT in equation (5) as an example. Construct the matrix as follows:

here,

contains all (M) terms supported in the MMR prediction operation.

Assume that the following relational expression holds:

The posterior MMR coefficient for posterior reshaping can be calculated as follows:

In some operational scenarios, the prediction error in the backward reshaping in the luma channel/plane is more important than the prediction error in the backward reshaping chroma channel/plane. Furthermore, generating and updating SDR luma codes or values in backward d3DMT updates/modifications is computationally more efficient than generating and updating SDR chroma codes or values. In these operational scenarios, the processes shown in Table 4 may be given higher priority than the processes shown in Table 5.

Figure 4A shows an example for an adjustable trade-off between the decoder-side quality of the reconstructed HDR image and the encoder-side computational cost of updating or constructing the d3DMT to generate the backward reconstruction mapping. Show the process flow. In some embodiments, a video encoder implemented on one or more computing devices can perform this process flow.

Block 402 is responsible for delivering video data to a decoder/playback device, etc. to support updating/building a forward d3DMT (or vice versa, updating/building a corresponding backward d3DMT that was originally generated). determining whether computational resources of one or more video codecs (e.g., a video encoder, a video encoder of a coding block) are available.

Block 404 directly uses the inverse of the forward d3DMT as the backward d3DMT to calculate backward MMR coefficients in response to a determination that computational resources are not available to support an update/build of the forward d3DMT (or backward d3DMT). d3DMT to avoid making changes to the forward d3DMT. A first color grade of the reconstructed HDR image is generated by a video decoder receiving the forward reshaped SDR image and a corresponding backward reshaping mapping generated at least in part by the backward MMR coefficients. I can do it.

To generate the first color grade, as shown in Figure 3B, the video encoder first maps the HDR samples of each source (or input) HDR image and the corresponding content map of the SDR image corresponding to the source HDR image. A forward d3DMT is created from the SDR samples. The video encoder then computes a backward reshaping mapping, including but not limited to a chromatic backward reshaping function represented by the backward reshaping MMR coefficients derived from the backward d3DMT, as the inverse of the forward d3DMT (the forward without further updates to d3DMT or backward d3DMT).

As used herein, different color grades may be generated to depict the same visual semantic content. A color grade may refer to a particular version of the reconstructed HDR image (eg, encoder generated, etc.) that depicts the same visual semantic content.

Block 406 includes determining a particular level of available computational resources in response to determining that computational resources are available to support forward d3DMT (or backward d3DMT) updates/builds. including.

In some operational scenarios, multiple computational resource level thresholds may be configured for a video encoder and used by the video encoder to compare to a particular level of available computational resources.

Block 408 identifies a corresponding particular trade-off option among the plurality of trade-off options for updating/building the d3DMT based on the particular level of available computing resources and the plurality of computing resource level thresholds. including determining the

For example, in response to determining a particular level of available computing resources, if the first computing resource level threshold is exceeded but below the second computing resource level threshold (the first computing resource level threshold), the video encoder may update the SDR luminance code/value of the forward d3DMT (or the corresponding backward d3DMT) as shown in Table 4. A second color grade of the reconstructed HDR image includes a forward reshaped SDR image and a corresponding backward reshaped mapping generated by backward MMR coefficients derived at least in part from the updated d3DMT. can be generated by a receiving video decoder.

To generate the second color grade, as shown in Figure 3B, the video encoder first extracts the HDR samples of each source (or input) HDR image, and the content map SDR image correspondence corresponding to the source HDR image. A forward d3DMT is created from the SDR samples. The video encoder generates a luma forward reshaping function based on the forward d3DMT and uses the luma forward reshaping function to update the luma codewords/values (but not update the chroma codes/values) in the forward d3DMT. To do. The video encoder then computes a backward reshaping mapping including, but not limited to, a chromatic backward reshaping function represented by backward reshaping MMR coefficients derived from the updated backward d3DMT.

If the particular level of available computing resources is above a second computing resource level threshold but below a third computing resource level threshold (above the second computing resource level threshold), the video encoder: The SDR luminance and chroma codewords/values of the forward d3DMT (or corresponding backward d3DMT) may be updated as shown in Tables 4 and 5. A third color grade of the reconstructed HDR image is a backward MMR derived from the updated d3DMT, at least in part, by a video decoder that receives the forward reshaped SDR image and the corresponding backward reshaped mapping. It can be generated by a coefficient.

To generate the third color grade, as shown in Figure 3B, the video encoder first extracts the HDR samples of each source (or input) HDR image, and the content map SDR image correspondence corresponding to the source HDR image. A forward d3DMT is created from the SDR samples. The video encoder generates a luma forward reshaping function based on the forward d3DMT and a chroma forward reshaping function (eg, forward reshaping MMR coefficients, etc.) based on the forward d3DMT. The video encoder uses the luma and chroma forward reconstruction functions to update the luma codes/values as well as the luma codes/values in the forward d3DMT or the backward d3DMT that was originally generated as backward from the forward d3DMT. The video encoder then computes a backward reshaping mapping including, but not limited to, a chromatic backward reshaping function represented by backward reshaping MMR coefficients derived from the updated backward d3DMT.

In response to the particular level of available computational resources being above a third computational resource level threshold, the video encoder uses the HDR source (or reference) image and the corresponding forward reshaped SDR image to , one can construct a (new) backward d3DMT for backward reshaping (e.g., without using the inverse of the forward d3DMT, etc.) and calculate the backward MMR coefficients based on the constructed backward d3DMT. The fourth color grade of the reconstructed HDR image is a forward reshaped SDR image and a corresponding backward reshaped mapping generated, at least in part, by the backward MMR coefficients derived from the constructed backward d3DMT. can be generated by a receiving video decoder.

To generate the fourth color grade, as shown in Figure 3B, the video encoder first extracts the HDR samples of each source (or input) HDR image and the content maps of the SDR image corresponding to the source HDR image. A forward d3DMT is created from the SDR samples. The video encoder generates a luma forward reshaping function based on the forward d3DMT and a chroma forward reshaping function (eg, forward reshaping MMR coefficients, etc.) based on the forward d3DMT. The video encoder uses forward reshaping functions of luminance and chroma to generate a forward reshape SDR image. The video encoder then includes a chromatic backward reshaping function, represented by backward reshaping MMR coefficients derived from a backward d3DMT constructed directly based on the source HDR image and the forward reshaping SDR image. Compute an unbounded backward reshaping mapping.

BLUT Modification In some operational scenarios, an example trade-off option is BLUT modification as shown in FIG. 1B. The maximum brightness range that can be represented by the available codewords of the SDR bin without exhibiting banding artifacts was estimated using BLKSTD. The actual brightness range represented by the available codewords (which may exhibit banding artifacts, etc., for example) is calculated from the BLUT. The BLUT portion (e.g. brightest, brightest below ceiling brightness codeword/value, etc.) above a particular brightness codeword/value (e.g., denoted as ) is the maximum brightness range and the actual brightness range may be modified based on the ratio between. Additionally, optionally or alternatively, the dark (eg, darkest and darkest above the floor brightness code/value) BLUT portion may be modified as part of the noise injection operation. Exemplary BLUT modifications and noise injection are described in the aforementioned US Provisional Patent Application No. 62/885,921.

As described herein, enabling BLUT modification in the video encoder can alleviate or reduce banding artifacts in bright regions of reconstructed HDR images, thereby reducing these reconstructions. The visual quality of the constructed HDR images can be significantly improved.

Color Correction for BLUT Correction In some operational scenarios, BLUT correction can alleviate banding artifacts in bright areas, but the color of collided pixels may change, for example, where the brightness code/value changes due to BLUT correction. It is observed that the appearance of may change. For example, if banding artifacts are removed from the sky around the sun in a reconstructed HDR image by changing the BLUT, the sky color in the reconstructed HDR image will be different from that of the reconstructed HDR without changing the BLUT. It may appear more saturated than the image.

In general, decreasing a pixel's luminance value can make the pixel's color appear more saturated, and increasing a pixel's luminance value can make the pixel's color appear more saturated if the pixel's chromaticity value remains the same. may become more saturated and invisible.

Although it may not mean that the saturation actually changes, the appearance of a pixel's color visually perceived by the viewer is different for different luminance values for the same chromaticity value.

In some operational scenarios, the corresponding chromaticity values of the pixels (corresponding to the luminance values of the pixels changed by the BLUT modification) are determined by the color appearance of these pixels from the source (e.g. reference, input, etc.) HDR The color may be modified to appear relatively close to the color of the image. The ratio of the modified BLUT to the original BLUT (unchanged by the BLUT modification) is used as a desaturation (or chromaticity scaling) function to adjust the chroma value.

と記し、バンディングアーチファクトを抑制する改良型ＢＬＵＴを、

と記すことにする。ＢＬＵＴは（例えば、正規化、非正規化等された）ＳＤＲ輝度値を（例えば、正規化、非正規化等された）ＨＤＲ輝度値にマッピングする。 The original BLUT,

, an improved BLUT that suppresses banding artifacts,

I will write this as BLUT maps SDR luminance values (eg, normalized, denormalized, etc.) to HDR luminance values (eg, normalized, denormalized, etc.).

For illustrative purposes only, chromaticity values are expressed in color spaces including, but not limited to, YCbCr, ICtCp, IPTPQ, etc., and chromaticity values in the normalized domain [0 1] are expressed with an offset of 0.5. be done. Therefore, a chromaticity value of 0.5 means a neutral color (grayscale). To adjust the chromaticity value (or its saturation/color appearance), an offset can be removed from the chromaticity value.

An exemplary procedure for replacing an input (or reference) HDR chroma codeword/value in a chroma channel/plane with an unsaturated HDR codeword/value in the backward d3DMT is shown in Table 6 below. This ensures that instead of the input (or reference) HDR chroma codeword/value, the unsaturated HDR codeword/value is the reconstructed HDR codeword produced by backward reconstruction mapping as described herein. / value.

In some operational scenarios, after updating the HDR chroma codewords/values (and updating the content-mapped SDR codewords/values) in the backward d3DMT, which was originally generated as backward of the forward d3DMT, (after replacing codewords/values), the modified backward d3DMT from HDR to SDR can be generated as follows:

The backward MMR coefficients in the backward reshaping mapping can then be calculated from the backward d3DMT.

のパラメータβは、元の（例えば、トレーニング中などの）ＨＤＲ画像のカラー外観と、色度値が不飽和で色度値が不飽和でない対応する再構成されたＨＤＲ画像のカラー外観の経験的研究を通して、実際の値が決定またはチューニングされ得るチューニングパラメータを表す。パラメータβの値の例としては、１．５、２、２．５、３、３．５などのうちの１つが挙げられるが、必ずしもこれらに限定されない。 Unsaturated functions as shown in Table 6

The parameter β is the empirical difference between the color appearance of the original (e.g., during training) HDR image and the corresponding reconstructed HDR image with unsaturated chromaticity values and unsaturated chromaticity values. represents a tuning parameter whose actual value may be determined or tuned through research. Examples of the value of the parameter β include, but are not necessarily limited to, one of 1.5, 2, 2.5, 3, 3.5, etc.

は、２つの項を含む。第１項

は、ニュートラルグレー値０．５からの不飽和色度値の偏差を表す。したがって、Ｃ０またはＣｂチャネルの不飽和色度値

の第１項が比較的小さい場合、不飽和色度値

は、ニュートラルグレー値０．５に比較的近い値に調整される。その結果、この非飽和色度値

を有する画素の色は、ニュートラルグレー値０．５に比較的近く調整されたので、対応する調整前の色度値

を有する画素よりも不飽和化される。 Unsaturated chromaticity value of C0 or Cb channel as shown in Table 6

contains two terms. Section 1

represents the deviation of the unsaturated chromaticity value from the neutral gray value of 0.5. Therefore, the unsaturated chromaticity value of the C0 or Cb channel

If the first term of is relatively small, the unsaturated chromaticity value

is adjusted to a value relatively close to the neutral gray value of 0.5. As a result, this unsaturated chromaticity value

The color of the pixel with has been adjusted relatively close to the neutral gray value of 0.5, so the corresponding unadjusted chromaticity value

It is less saturated than a pixel with .

は、２つの項を含む。第１項

は、ニュートラルグレー値０．５からの不飽和色度値の偏差を表す。したがって、Ｃ１またはＣｂチャネルの不飽和色度値

の第１項が比較的小さい場合、不飽和色度値

は、ニュートラルグレー値０．５に比較的近い値に調整される。その結果、この非飽和色度値

を有する画素の色は、ニュートラルグレー値０．５に比較的近く調整されたので、対応する調整前の色度値

を有する画素よりも不飽和化される。 Similarly, the unsaturated chromaticity value of the C1 or Cr channel

contains two terms. Section 1

represents the deviation of the unsaturated chromaticity value from the neutral gray value of 0.5. Therefore, the unsaturated chromaticity value of C1 or Cb channel

If the first term of is relatively small, the unsaturated chromaticity value

is adjusted to a value relatively close to the neutral gray value of 0.5. As a result, this unsaturated chromaticity value

The color of the pixel with has been adjusted relatively close to the neutral gray value of 0.5, so the corresponding unadjusted chromaticity value

It is less saturated than a pixel with .

FIG. 3C shows an example of color correction that can be applied to any color grade in various color grades (eg, color grades 2-4 as shown in FIG. 3B). As an example, a video encoder generates a luminance backward reshaping function, such as a BLUT (eg, before modification, etc.) based on the backward d3DMT, as described herein. The video encoder then modifies the BLUT to reduce banding artifacts (eg, the risk thereof, etc.). The video encoder uses the unmodified BLUT and the modified BLUT to generate an unsaturated function (e.g., as shown in Table 6 above) to update the HDR chroma codeword/value in the backward d3DMT. (as a percentage). The video encoder then computes a backward reshaping mapping including, but not limited to, a chromatic backward reshaping function represented by backward reshaping MMR coefficients derived from the updated backward d3DMT.

FIG. 3D shows an example color correction performed in conjunction with a d3DMT update operation for color grade 2 (as shown in FIG. 3B). The video encoder first constructs a forward d3DMT from the HDR samples of each source (or input) HDR image and the corresponding SDR samples of the content map SDR image corresponding to the source HDR image. The video encoder constructs or generates a luminance forward reshaping function based on the forward d3DMT. The video encoder uses the luminance forward reorganization function to update the luminance codes/values of the forward d3DMT or the backward d3DMT that were originally generated as an inverse transform from the forward d3DMT. The video encoder then constructs or generates a luminance backward reshaping function based on the backward d3DMT. The video encoder further modifies the BLUT to reduce banding artifacts. The video encoder uses the pre-modified BLUT and the modified BLUT to generate an unsaturated function to update the HDR chroma codeword/value in the backward d3DMT (e.g., in Table 6 above). ratios shown). Subsequently, the video encoder calculates a backward reshaping mapping including, but not limited to, a chromatic backward reshaping function represented by backward reshaping MMR coefficients derived from the updated backward d3DMT.

Backward Reshaping Mapping and Temporal Stability Figure 3E shows an example of backward reshaping mapping generation with brightness update, BLUT correction, and color correction. Block 302 includes constructing a d3DMT using the sampled HDR codewords of the source HDR image and the corresponding content-mapped SDR codewords, each of which It includes a plurality of mapping pairs between HDR codewords and SDR codewords. Block 304 includes constructing a corresponding luminance forward reshaping function, such as a forward lookup table, based on the plurality of mapping pairs within each d3DMT. Block 306 includes constructing corresponding forward chroma mappings represented by forward MMR coefficients based on the plurality of mapping pairs within each d3DMT. Block 308 includes updating the SDR luminance codes in the plurality of mapping pairs within each d3DMT by replacing the content-mapped SDR luminance codewords with the FLUT-predicted forward reconstructed SDR codewords. include. Block 310 includes constructing a luminance backward reshaping function (BLUT) from the FLUT. Block 312 includes modifying the BLUT to reduce or reduce the risk of banding artifacts. Block 314 performs HDR chroma codewords in the plurality of mapping pairs within each d3DMT by replacing the sampled HDR chroma codewords with unsaturated HDR chroma codewords generated based on the original BLUT and the modified BLUT. Including rectifying the chromaticity codeword. Block 316 performs each forward reshaping process corresponding to one of the source HDR images based on an updated d3DMT that includes a plurality of mapping pairs having an updated luma SDR codeword and an updated chroma HDR codeword. chromatic backward reshaping mapping (e.g., chromatic backward reshaping function, backward (reshaping) MMR coefficients, etc.) for the SDR image.

In some operational scenarios, a linear segment-based structure may be used in computing/generating/including backward reshaping mappings of image metadata for the purpose of maintaining temporal stability of image metadata. Exemplary linear segment-based structures are described in U.S. Patent Application No. 2018/0007356, published January 4, 2018, the entire contents of which are incorporated herein by reference. is incorporated herein by reference.

Some or all of the techniques described herein may be implemented and/or as part of a real-time operation to produce suitable color-grade video content for broadcast video applications, real-time streaming applications, etc. can be executed. Additionally, some or all of the techniques described herein may optionally or alternatively be implemented and/or performed as part of a time-delayed or offline operation for non-real-time streaming applications, cinema applications, etc. It is possible to generate video content with appropriate color grade.

Example Process Flow FIG. 4B is a diagram illustrating an example process flow according to one embodiment. In some embodiments, one or more computing devices or components (e.g., encoding devices/modules, transcoding devices/modules, decoding devices/modules, reverse tone mapping devices/modules, tone mapping devices/modules, media devices) /modules, inverse mapping generation and application systems, etc.) may perform this process flow. At block 422, the image processing system first generates a backward reshaping table as the inverse of the forward reshaping table.

The forward reshape table is used to generate a forward reshape mapping to generate a first dynamic range forward reshape image from a second dynamic range source image. The first dynamic range is lower than the second dynamic range. The forward reorganization mapping table includes a second dynamic range sampled luma codeword, a second dynamic range sampled chroma codeword, and a second dynamic range sampled luma codeword corresponding to the second dynamic range sampled luma codeword. a content-mapped luma codeword of one dynamic range; and a content-mapped chroma codeword of a first dynamic range corresponding to a sampled chroma codeword of a second dynamic range.

At block 424, the image processing system updates the backward reconstruction mapping table by replacing the content-mapped intensity codeword with the forward reconstructed intensity codeword. The forward reconstructed luminance codeword is generated by applying luminance forward mapping to the sampled luminance codeword of the second dynamic range. The brightness forward mapping consists of a forward reshaping mapping table.

At block 426, the image processing system generates a backward reshaping mapping table to generate a backward reshaping mapping to create a second dynamic range reconstructed image from the first dynamic range forward reshaping image. and using luminance forward mapping.

At block 428, the image processing system encodes the forward reshaping image with the video signal along with image metadata specifying the backward reshaping mapping. A device receiving the video signal applies the backward reshaping mapping to the forward reshaping image to generate a second dynamic range reconstructed image.

In one embodiment, the image processing system determines available computational resources in an end-to-end video delivery pipeline, and uses the available computational resources to decode a second dynamic range decoded image. determining a specific set of trade-off options between side quality and encoder-side computational complexity for generating a backward reshaping mapping; The method is further configured to perform: performing the particular set of processing operations.

In one embodiment, the image processing system is further configured to derive a display image from the reconstructed image and render it at the video signal receiving device.

In one embodiment, the image processing system is further configured to update the backward reshaping mapping table by replacing the content mapped chroma codeword with the forward reshape chroma codeword. A forward reconstructed chroma codeword is generated by applying multivariate multiple regression (MMR) chroma forward mapping to the second dynamic range sampled luma and chroma codewords. The MMR chromaticity forward mapping consists of a forward reshaping mapping table.

In one embodiment, the image processing system is further configured to update the backward reshaping mapping table by replacing the sampled chroma codeword with an unsaturated chroma codeword.

In one embodiment, the unsaturated chroma codeword is generated by applying an unsaturated function to the sampled chroma codeword, and the unsaturated function is combined with the modified luminance backward reshaping mapping and the original is constructed as the ratio between the luminance backward reshaping mapping of .

In one embodiment, the image processing system is further configured to apply a backward lookup table modification to reduce banding artifacts in the reconstructed image.

In one embodiment, at least one of the backward mapping table and the forward mapping table represents a three-dimensional mapping table that is at least partially dynamically constructed from the source image and the forward reshaped image.

In one embodiment, the intensity forward mapping is represented by an intensity lookup table.

In one embodiment, the second dynamic range reconstructed image approximates the second dynamic range source image.

In one embodiment, the video signal represents a single layer backward compatible video signal.

FIG. 4C is a diagram illustrating an exemplary process flow according to one embodiment of the invention. In some embodiments, one or more computing devices or components (e.g., encoding devices/modules, transcoding devices/modules, decoding devices/modules, inverse tone mapping devices/modules, tone mapping devices/modules, media devices/modules, predictive models and feature selection systems, inverse mapping generation and application systems, etc.) may perform this process flow. At block 442, the video decoding system decodes a first dynamic range forward reshaped image from the video signal.

At block 444, the video decoding system decodes image metadata, including backward reshaping mapping, from the video signal.

The backward reshaping mapping was generated by an upstream image processing device from the backward reshaping mapping table and the luminance forward mapping. The backward reshaping mapping table was initially generated by an upstream image processing device as the inverse of the forward reshaping mapping table that generates the luminance forward mapping. The content mapping intensity codeword in the backward reshaping mapping table has been updated with the forward reshaping intensity codeword. The forward reshaped luminance codeword was generated by applying luminance forward mapping to the sampled source luminance codeword in the source image.

At block 446, the video decoding system applies a backward reshaping mapping to the forward reshaping image to generate a reconstructed image.

At block 448, the video decoding system causes a display image derived from the reconstructed image to be rendered on a display device.

In one embodiment, a computing device, such as a display device, mobile device, set-top box, multimedia device, etc., is configured to perform any of the aforementioned methods. In one embodiment, an apparatus comprises a processor and is configured to perform any of the methods described above. In one embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause performance of any of the methods described above.

In one embodiment, a computing device comprising one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, cause performance of any of the methods.

Note that although separate embodiments are discussed herein, any combination of embodiments and/or sub-embodiments discussed herein may be combined to form further embodiments. I want to be

Example Implementations of Computer Systems Embodiments of the present invention may be implemented in computer systems, systems configured into electronic circuits and components, integrated circuits such as microcontrollers, field programmable gate arrays, or other configurable or programmable logic devices, discrete-time or can be implemented using digital signal processors, application specific ICs, and/or apparatus including one or more of these systems, devices, or components. The computer and/or IC may execute, control, or execute instructions related to adaptive perceptual quantization of images with extended dynamic range as described herein. A computer and/or IC may calculate any of various parameters or values associated with the adaptive perceptual quantization process described herein. Image and video embodiments may be implemented in hardware, software, firmware, and various combinations thereof.

Certain implementations of the present invention include a computer processor executing software instructions that cause the processor to perform the methods of the present disclosure. For example, one or more processors in displays, encoders, set-top boxes, transcoders, etc. may be associated with adaptive perceptual quantization of HDR images as described above by executing software instructions in a program memory accessible to the processor. The method can be implemented. Embodiments of the invention may also be provided in the form of a program product. The program product may include any non-transitory medium carrying a set of computer-readable signals containing instructions that, when executed by a data processor, cause the data processor to perform methods of embodiments of the present invention. . Program products according to embodiments of the invention may take any of a wide variety of forms. The program product may include physical media such as, for example, magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD-ROMs, DVDs, electronic data storage media including ROMs, flash RAM, and the like. Computer readable signals on the program product may be optionally compressed or encrypted.

When a component (e.g., software module, processor, assembly, device, circuit, etc.) is mentioned above, references to that component (including references to "means") are , any components that perform the functions (e.g., are functionally equivalent) of the described components, including components that are not structurally equivalent to the disclosed structures that perform the functions in exemplary embodiments of the invention. shall be construed as equivalents of such constituent elements, including constituent elements.

According to one embodiment, the techniques described herein are implemented by one or more special purpose computing devices. A special-purpose computing device may be hard-wired to perform the technique, or may include one or more application-specific integrated circuits or field-programmable gate arrays that are permanently programmed to perform the technique. or may include one or more general purpose hardware processors programmed to perform the techniques according to a combination of program instructions, firmware, memory, other storage devices, or a combination of programmed instructions. Such special purpose computing devices may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to achieve the technology. The special purpose computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that incorporates hardwired and/or program logic to implement the techniques.

For example, FIG. 5 is a block diagram illustrating a computer system 500 in which one embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled with bus 502 for processing information. Hardware processor 504 may be, for example, a general purpose microprocessor.

Computer system 500 also includes main memory 506, such as random access memory or other dynamic storage device, coupled to bus 502 for storing information and instructions executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during execution of instructions executed by processor 504. Such instructions, when stored on a non-transitory storage medium accessible to processor 504, render computer system 500 into a specialized machine customized to perform the operations specified in the instructions.

Computer system 500 further includes a read-only memory 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic or optical disk, is provided and coupled to bus 502 for storing information and instructions.

Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. Input devices 514, including alphanumeric and other keys, are coupled to bus 502 for communicating information and command selections to processor 504. Another type of user input device is a cursor control device 516, such as a mouse, trackball, or cursor direction keys, that communicates directional information and command selections to processor 504 and controls movement of a cursor on display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), and the device specifies a position in a plane. make it possible to

Computer system 500 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware, and/or program logic, which may be used to implement the techniques described herein. In combination with the system, computer system 500 becomes a special purpose machine or is programmed. According to one embodiment, the techniques described herein are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Ru. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

The term "storage medium" as used herein refers to any non-transitory medium that stores data and/or instructions that cause a device to operate in a particular manner. Such storage media may include non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 510. Volatile media includes dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, floppy disks, hard disks, solid state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, Any physical media with a hole pattern, including RAM, PROM, and EPROM, FLASH®-EPROM, NVRAM, and any other memory chip or cartridge.

Storage media are distinct from, but can be used in conjunction with, transmission media. Transmission media participate in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that make up bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 504 for execution. For example, the instructions may first be executed on a remote computer's magnetic disk or solid state drive. A remote computer can load the instructions into its dynamic memory and transmit the instructions over a telephone line using a modem. A modem local to computer system 500 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried within the infrared signal, and appropriate circuitry can place the data on bus 502. Bus 502 carries data to main memory 506 from where processor 504 retrieves and executes instructions. Instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by processor 504.

Computer system 500 also includes a communications interface 518 coupled to bus 502. Communication interface 518 provides bi-directional data communication coupling to network link 520 connected to local network 522. For example, communications interface 518 may be an integrated services digital network card, cable modem, satellite modem, or modem that provides a data communications connection to a corresponding type of telephone line. As another example, communications interface 518 may be a local area network card for providing data communications connectivity to a compatible LAN. A wireless link may also be implemented. In such implementations, communication interface 518 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connection through local network 522 to a data device operated by host computer 524 or Internet service provider 526. ISP 526, in turn, provides data communication services via the worldwide packet data communication network, now commonly referred to as the “Internet” 528. Local network 522 and Internet 528 both use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks and signals on network link 520 and communication interface 518 that carry digital data to and from computer system 500 are example forms of transmission media.

Computer system 500 can send messages and receive data, including program code, through the network, network link 520 and communication interface 518. In the Internet example, server 530 may transmit the requested code of the application program via Internet 528, ISP 526, local network 522, and communication interface 518.

The received code is executed by processor 504 as it is received and/or stored in storage device 510, or other non-volatile storage device, for later execution.

Equivalents, Extensions, Alternatives, and Others In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the only and exclusive indication of the claimed embodiments of this invention, and what is intended by applicant to be the claimed embodiments of this invention, is from this application A set of claims in a particular form that has been made, including any subsequent amendments. The definitions expressly set forth in this Agreement for terms contained in such claims shall control the meaning of such terms as used in such claims. Therefore, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Various aspects of the invention can be understood from the examples (EEE) listed below:
EEE1
initially generating a backward reshaping mapping table as the inverse of a forward reshaping mapping table, wherein the forward reshaping table connects the forward reshaping image of a first dynamic range to a source of a second dynamic range; is used to generate a forward reshaping mapping for generating from an image, the first dynamic range being lower than the second dynamic range, and the forward reshaping mapping table being the sampled luminance codeword, the second dynamic range sampled chroma codeword, the first dynamic range content mapped corresponding to the second dynamic range sampled luminance codeword; a content-mapped chroma codeword of the first dynamic range corresponding to the sampled chroma codeword of the second dynamic range;
updating the backward reshaping mapping table by replacing the content mapped brightness codeword with a forward reshaped brightness codeword, wherein the forward reshaped brightness codeword is one of the second brightness codewords; generated by applying a luminance forward mapping to the sampled luminance codewords of a dynamic range, the luminance forward mapping being constructed from the forward reshaping mapping table;
the backward reshaping mapping table and the luminance front to generate a backward reshaping mapping for creating a reconstructed image of the second dynamic range from the forward reshaping image of the first dynamic range; Steps to use mapping and
encoding the forward reshaped image with image metadata specifying the backward reshaping mapping into a video signal.
EEE2
The method of EEE1, wherein the video signal receiving device applies the backward reshaping mapping to the forward reshaping image to create the reconstructed image of the second dynamic range.
EEE3
determining available computational resources in an end-to-end video delivery pipeline;
between the quality of the reconstructed image of the second dynamic range on the decoder side and the computational complexity of generating the backward reshaping mapping on the encoder side using the available computational resources. determining a particular set of trade-off options;
and performing a particular set of image processing operations according to the particular set of trade-off options to generate the backward reshaping mapping.
EEE4
4. A method according to EEE 2 or 3, further comprising the step of deriving a display image from the reconstructed image and rendering it at a receiving device of the video signal.
EEE5
updating the backward reshaping mapping table by replacing the content mapped chroma code with a forward reshape chroma codeword, wherein the forward reshape chroma codeword is generated by applying a multivariate multiple regression (MMR) chroma forward mapping to the sampled luma and chroma codewords of a dynamic range of 2, the MMR chroma forward mapping from the forward reshaping mapping table. 5. The method according to any one of EEE1-4, further comprising the step of constructing.
EEE6
6. The method of any one of EEE1-5, further comprising updating the backward reshaping mapping table by replacing the sampled chroma codeword with a desaturated chroma codeword.
EEE7
The desaturated chroma codeword is generated by applying a desaturation function to the sampled chroma codeword, the desaturation function combining the modified luminance backward reshaping mapping and the original The method according to EEE6, constructed as the ratio between the luminance backward reshaping mapping of .
EEE8
8. The method according to any one of EEE1-7, further comprising applying a backward look-up table modification to reduce banding artifacts in the reconstructed image.
EEE9
at least one of the backward mapping table and the forward mapping table represents a three-dimensional mapping table (3DMT) dynamically constructed from the source image and the forward reshaped image, at least in part; The method described in any one of EEE 1 to 8.
EEE10
A method according to any one of EEE1-9, wherein the brightness forward mapping is represented by a brightness lookup table.
EEE11
11. The method of any one of EEE1-10, wherein the reconstructed image in the second dynamic range approximates the source image in the second dynamic range.
EEE12
12. The method according to any one of EEE1-11, wherein the video signal represents a single layer backward compatible video signal.
EEE13
decoding the first dynamic range forward reshaped image from the video signal;
decoding image metadata from the video signal including a backward reshaping mapping, the backward reshaping mapping being generated by an upstream image processing device from a backward reshaping mapping table and a luminance forward mapping; A reshaping mapping table is initially generated by the upstream image processing device as the inverse of the forward reshaping mapping table that generates the luminance forward mapping, and the content mapped luminance codewords in the backward reshaping mapping table are , updated with a forward reshaped luminance codeword, the forward reshaped luminance codeword generated by applying the luminance forward mapping to the sampled luminance codeword in the source image. The steps are:
applying the backward reshaping mapping to the forward reshaping image to generate the reconstructed image;
causing a display device to render a display image derived from the reconstructed image.
EEE14
A computer system configured to perform any one of the methods described in EEE1-13.
EEE15
An apparatus comprising a processor and configured to perform any one of the methods described in EEE1-13.
EEE16
A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for performing a method by one or more processors according to any of the methods described in EEE1-13.

Claims (17)

initially generating a backward reshaping mapping table as the inverse of a forward reshaping mapping table, the forward reshaping mapping table combining a forward reshaping image of a first dynamic range with a forward reshaping mapping table of a second dynamic range; the forward reshaping mapping table is used to generate a forward reshaping mapping for generating from a source image, the first dynamic range being lower than the second dynamic range; the second dynamic range luminance codeword of and the second dynamic range chroma codeword of , and the correspondence from the first dynamic range reference image produced by color grading the source image. a first dynamic range of luminance codewords of samples, and a first dynamic range of chroma codewords of samples, the number of samples from the source image being compared to the number of pixels in the source image. and a step that is reduced by
updating the backward reshaping mapping table by replacing the luminance codewords of the first dynamic range in the backward reshaping mapping table with forward reshaped luminance codewords; a luminance codeword of the second dynamic range is generated by applying a luminance forward reshaping function to the luminance codeword of the second dynamic range, the luminance forward reshaping function extracting a cumulative density function (CDF) from the forward reshaping mapping table. ) is constructed using a matching step;
constructing a luminance backward reshaping function from the luminance forward reshaping function that maps luminance codewords of the first dynamic range to luminance codewords of the second dynamic range;
a particular luminance code of said first dynamic range based on a ratio between an estimated maximum luminance range that can be represented without exhibiting banding artifacts and an actual luminance range represented by said luminance backward reshaping function; modifying a portion of the luminance backward reshaping function above the word;
updating the backward reshaping mapping table by replacing the chroma codeword of the second dynamic range in the backward reshaping mapping table with a desaturated chroma codeword, The modified chroma codeword is applied by applying a desaturation function to the chroma codeword of the second dynamic range, the desaturation function being the modified luminance backward reshaping function. and the original luminance backward reshaping function.
the backward reshaping mapping table and the luminance front to generate a backward reshaping mapping for creating a reconstructed image of the second dynamic range from the forward reshaping image of the first dynamic range; a step of using a reshaping function;
A method comprising encoding the forward reshaped image with image metadata specifying the backward reshaping mapping into a video signal. determining available computational resources in an end-to-end video delivery pipeline;
between the quality of the reconstructed image of the second dynamic range on the decoder side and the computational complexity of generating the backward reshaping mapping on the encoder side using the available computational resources. determining a particular set of trade-off options;
2. The method of claim 1, further comprising: performing a particular set of image processing operations according to the particular set of trade-off options to generate the backward reshaping mapping. 3. A method according to claim 1 or 2, further comprising the step of deriving a display image from the reconstructed image and rendering it at a receiving device of the video signal. updating the backward reshaping mapping table by replacing the chroma codeword of the first dynamic range in the backward reshaping mapping table with a forward reshaping chroma codeword; A reshaped chroma codeword is generated by applying multivariate multiple regression (MMR) chroma forward mapping to the luminance and chroma codewords of the second dynamic range, the MMR chroma forward mapping A method according to any one of claims 1 to 3, further comprising the step of: is constructed from the forward reshaping mapping table. A method according to any preceding claim, further comprising applying a backward look-up table modification to reduce banding artifacts in the reconstructed image. At least one of the backward reshaped mapping table and the forward reshaped mapping table is a three-dimensional mapping table (3DMT) dynamically constructed at least in part from the source image and the forward reshaped image. The method according to any one of claims 1 to 5, wherein the method represents: A method according to any preceding claim, wherein the intensity forward reshaping function is represented by an intensity lookup table. A method according to any preceding claim, wherein the reconstructed image of the second dynamic range approximates the source image of the second dynamic range. A method according to any preceding claim, wherein the video signal represents a single layer backward compatible video signal. The video signal receiving device applies the backward reshaping mapping to the forward reshaping image to create the reconstructed image of the second dynamic range. The method described in Section 1. decoding a forward reshaped image of a first dynamic range from the video signal generated by the method according to any one of claims 1 to 10, the step of decoding a forward reshaped image of a first dynamic range; is generated from a source image of a second dynamic range by a forward reshaping mapping table that is generated from a forward reshaping mapping table, the forward reshaping mapping table being configured to calculate the second dynamic range of samples from the source image. a luminance codeword and a chromaticity codeword of the second dynamic range and the first dynamic range of corresponding samples from the first dynamic range reference image generated by color grading the source image; a luma codeword of the first dynamic range and a chroma codeword of the first dynamic range, the number of samples from the source image being reduced compared to the number of pixels in the source image;
decoding image metadata including backward reshaping mapping from the video signal;
applying the backward reshaping mapping to the forward reshaping image to generate a reconstructed image;
causing a display device to render a display image derived from the reconstructed image. A computer system configured to carry out the method according to any one of claims 1 to 10 . A computer system configured to perform the method of claim 11. Apparatus comprising a processor and configured to carry out a method according to any one of claims 1 to 10 . 12. An apparatus comprising a processor and configured to perform the method of claim 11. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for performing the method by one or more processors according to the method of any one of claims 1-10 . 12. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon for performing the method by one or more processors in accordance with the method of claim 11.

Images