EP4567783A1 - Bildanzeigeverbesserung in aufgehellten betrachtungsumgebungen - Google Patents

Bildanzeigeverbesserung in aufgehellten betrachtungsumgebungen Download PDF

Info

Publication number
EP4567783A1
EP4567783A1 EP23215315.5A EP23215315A EP4567783A1 EP 4567783 A1 EP4567783 A1 EP 4567783A1 EP 23215315 A EP23215315 A EP 23215315A EP 4567783 A1 EP4567783 A1 EP 4567783A1
Authority
EP
European Patent Office
Prior art keywords
input
value
function
image
display
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP23215315.5A
Other languages
English (en)
French (fr)
Inventor
Johannes Yzebrand Tichelaar
Roeland Focco Everhard Goris
Philippe Antoine Maurice Van Overmeire
Jeroen Sebastiaan VAN GASTEL
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Koninklijke Philips NV
Original Assignee
Koninklijke Philips NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips NV filed Critical Koninklijke Philips NV
Priority to EP23215315.5A priority Critical patent/EP4567783A1/de
Priority to PCT/EP2024/084019 priority patent/WO2025119781A1/en
Publication of EP4567783A1 publication Critical patent/EP4567783A1/de
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/10—Intensity circuits
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/02—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the way in which colour is displayed
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/06—Adjustment of display parameters
    • G09G2320/0626—Adjustment of display parameters for control of overall brightness
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/06—Colour space transformation
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2360/00—Aspects of the architecture of display systems
    • G09G2360/14—Detecting light within display terminals, e.g. using a single or a plurality of photosensors
    • G09G2360/144—Detecting light within display terminals, e.g. using a single or a plurality of photosensors the light being ambient light

Definitions

  • the invention relates to the displaying of in particular high dynamic range images or videos (sequences of images) which need some brightening of the displaying to adjust for non-dark viewing environments, which have an illumination above a typical illumination level as expected and for which the images were prepared by color grading.
  • LDR Low Dynamic Range
  • SDR Standard Dynamic Range
  • the earliest television standards (NTSC, PAL) communicated the color components as three voltage signals (which defined the amount of a color component between 0 and 700mV), where the time positions along the voltage signal corresponded by using a scan path with pixels on the screen.
  • the control signals generated at the creation side directly instructed what the display should make as proportion (apart from their being an accidental fixed gamma pre-correction at the transmitter, because the physics of the cathode ray tube took approximately a square power of the input voltage, which would have made the dark colors much blacker than they were intended e.g. as seen by a camera) at the creation side). So a 60%, 30%, 25% color (which is a dark red) in the scene being captured, would look substantially similar on the display, since it would be re-generated as a 60%, 30%, 25% color (note that the absolute brightness didn't matter much, since the eye of the viewer would adapt to the white and the average brightness of the colors being displayed on the screen).
  • RGB and YCbCr are easy ones, namely they can be calculated into each other by using a simple fixed 3x3 matrix (the coefficients of which depend on the emission spectra of the three primaries, and are standardized, i.e. also to be emulated electronically internally by LCDs which actually may have different optical characteristics, so that from image communication point of view all SDR displays are alike).
  • relative brightness as a percentage of something, which may be undefined until a choice is made, e.g. by the consumer buying a certain display, and setting its brightness setting to e.g. 120%, which will make e.g. the backlight emit a certain amount of light, and so also the white and colored pixels
  • absolute brightness on the other hand absolute brightness.
  • the latter can be characterized by the universal physical quantity luminance (which is measured technically in the unit nit, which is also candela per square meter).
  • the luminance can be stated as an amount of photons coming out of a patch on an object, such as a pixel on the screen, towards the eye (and it is related to the lighting concept of illuminance, since such patch will receive a certain illuminance, and send some fraction of it towards the viewer).
  • a color space is the mathematical 3D space to represent colors (defining geometric positions of colors by coordinate numbers, e.g. a red, green and blue value of the weighed combination of primary intensities in the produced total color), often presented in such a shape that the base is defined by the triangle of 3 primaries.
  • Color model refers to the election of the categories of numerical values one uses to define the space, e.g. red, green and blue being a natural representation for specifying additive color generators, yet the same colors can be defined by three other coordinate ranges in the Hue, Saturation, and Value model, which model characterizes the colors in a more human-related manner.
  • color gamut is the set of all colors that can be technically defined or displayed (i.e. a space may be e.g. a 3D coordinate system going to infinity whereas the gamut may be a cube or tent-shape of some size in that space).
  • a space may be e.g. a 3D coordinate system going to infinity whereas the gamut may be a cube or tent-shape of some size in that space.
  • luminance range more commonly worded as " dynamic range ", spanning from some minimum brightness to its maximum brightness.
  • the present technologies will not primarily be about chromatic (i.e. color per se, such as more specifically its saturation) aspects, but rather about brightness aspects, so the chromatic aspects will only be mentioned to the extent needed for the relevant embodiments.
  • HDR High Dynamic Range
  • the luminance of SDR white for videos (a.k.a. the SDR White Point Luminance (WP) or maximum luminance (ML)), is standardized to be 100 nit (not to be confused with the reference luminance of white text in 1000 nit HDR images being 200 nit).
  • WP White Point Luminance
  • ML maximum luminance
  • a 1000 nit ML HDR image representation can make up to 10x brighter (glowing) object colors. What one can make with this are e.g. specular reflections on metals, such as a boundary of a metal window frame: in SDR the luminance has to end at 100 nit, making them visually on slightly brighter than the e.g. 70 nit light gray colors of the part of the window frame that does not specularly reflect.
  • HDR high definition of images
  • how a display which can only display whites as bright as 650 nit (the display maximum luminance ML_D) is to actually display the images is an entirely different matter, namely one of display adaptation a.k.a. display tuning, not of image (de)coding.
  • the relationship with how a camera captures HDR scene colors may be tight or loose: we will in general assume that HDR colors have already been defined in the HDR image when talking about such technologies as coding, communication, dynamic range conversion and the like.
  • the original camera-captured colors or specifically their luminances may have been changed into different values by e.g. a human color grader (who defines the ultimate look of an image, i.e. which color triplet values each pixel color of the image(s) should have), or some automatic algorithm.
  • a human color grader who defines the ultimate look of an image, i.e. which color triplet values each pixel color of the image(s) should have
  • some automatic algorithm e.g. a human color grader (who defines the ultimate look of an image, i.e. which color triplet values each pixel color of the image(s) should have), or some automatic algorithm.
  • the chromatic gamut size may stretch with less than a factor 2
  • the brightness range e.g. luminance range may stretch by a factor 100, one expects different technical rationales and solutions for the two improvement technologies.
  • a HDR image may be associated with a metadatum called mastering display white point luminance (MDWPL), a.k.a. ML_V .
  • MDWPL mastering display white point luminance
  • ML_V ML_V
  • This value which is typically communicated in metadata of the signal, and is a characterizer of the HDR video images (rather than of a specific display, as it is an element of a virtual display associated specifically with the video, being some ideal intended display for which the video pixel colors have been optimized to be conforming).
  • This is an electable parameter of the video, which can be contemplated as similar to the election of the painting canvas aspect ratio by a painter: first the painter chooses an appropriate AR, e.g.
  • the creator will then, after having established that the MDWPL is e.g. 5000 nit, make his secondary elections that in a specific scene this lamp shade should be at 700 nit, the flames in the hearth distributed around 500 nit, etc.
  • the MDWPL is e.g. 5000 nit
  • HDR image creation can also involve deeper blacks, up to as deep as e.g. 0.0001 nit (although that is mostly relevant for dark viewing environments, such as in cinema theatres).
  • the other objects e.g. the objects which merely reflect the scene light, will be coordinated to be e.g. at least 40x darker in a 5000 nit MDWPL graded video, and e.g. at least 20x darker in a 2000 nit video, etc. So the distribution of all image pixel luminances will typically depend on the MDWPL value (not making most of the pixels very bright).
  • OETF Opto-electronic Transfer Function
  • B_relative is a float number ranging from 0 to 1.0, so will Y_float.
  • signal value Y_float is quantized, because we want 8 bit digital representations, ergo, the Y_dig value that is communicated to receivers over e.g. airways DVB (or internet-supplied video on demand, or blu-ray disk, etc.) has a value between 0 and 255 (i.e. power(2;8)-1).
  • the vertical axis represents normalized luminances (in a linear gamut representation), or normalized lumas (in a non-linear representation (coding) of those luminances, e.g via a psychovisually uniformized OETF or its inverse the EOTF). Since after normalization (i.e. division by the respective MDWPL values, e.g. 2000 for an HDR image of a particular video and 100 for an SDR image), the common representation will become easy, and one can define luminance (or luma) mapping functions on normalized axes as shown in Fig. 1D (i.e.
  • Fig. 1B In the representation of Fig. 1B one can show the mapping from a HDR color (C_H) to an LDR color (C_L) of a pixel as a vertical shift (assuming that both colors should have the same proper color, i.e. hue and saturation, which usually is the desired technical requirement, i.e. on the circular ground plane they will project to the same point).
  • Ye means the color yellow, and its complementary color on the opposite side of the achromatic axis of luminances (or lumas) is blue (B), and W signifies white (the brightest color in the gamut a.k.a. the white point of the gamut, with the darkest colors, the blacks being at the bottom).
  • the receiving side in the old days, or today will know it has an SDR video, if it gets this format.
  • the maximum white (of SDR) will be by definition the brightest color that SDR can define. So if one now wants to make brighter image colors (e.g of real luminous lamps), that should be done with a different codec (as one can show the math of the Rec. 709 OETF allows only a coding of up to 1000:1 and no more).
  • HDR codecs start by defining an Electro-optical transfer function instead of its inverse, the OETF. Then one can at least basically define brighter (and darker) colors. That as such is not enough for a professional HDR coding system, since because it is different from SDR, and there are even various flavors, one wants more (new compared to SDR coding) technical information relating to the HDR images, which will be metadata.
  • HDR EOTFs are much steeper, to encode a much larger range of needed to be coded HDR luminances, and a significant part of that range coding specifically darker colors (relatively darker, since although one may be coding absolute luminances with e.g. the Perceptual Quantizer (PQ) EOTF (standardized in SMPTE 2084) , one applies the function after normalization).
  • PQ Perceptual Quantizer
  • EOTFs standardized in SMPTE 2084
  • the EOTF will be able to decode the pixel lumas in the plane of lumas spanning the image (i.e. having a width of e.g. 4000 pixels and a height of 2000), which will simply be binary numbers.
  • HDR images will also have a larger word length, e.g. 10 bit.
  • a linear (bit-represented) code e.g. a DMD pixel, indeed to reach e.g.
  • the receiving side may in both situations get as input a coded pixel color (luma and Cb, Cr; or in some systems by matrixing equivalent non-linear R'G'B' component values) which lie between 0 and 255, or 0 and 1023, but it will know the kind of signal it is getting (hence what ought to be displayed) from the metadata, such as the metadata (e.g. MPEG VUI metadata) co-communicated EOTF (e.g.
  • Fig. 1 for a typical nice HDR scene image, of a monster being fought in a cave with a flame thrower, the master grading (Mstr HDR) of which is shown spatially in Fig. 1A , and the range of occurring pixel luminances on the left of Fig. 1C ).
  • the master grading or master graded image is where the image creator can make his image look as impressive (e.g. realistic) as desired.
  • the image creator can make his image look as impressive (e.g. realistic) as desired.
  • a baker's shop window look somewhat illuminated by making the yellow walls somewhat brighter than paper white e.g.
  • a chromaticity is composed of a certain (rotation angle) hue h (e.g. bluish-green e.g. "teal”), and a saturation sat, which is the amount of pure color mixed in a grey, e.g.
  • the two dotted horizontal lines represent the limitations of the SDR codable image, when associating 100 nit with the 100% of SDR white.
  • the monster will be strongly illuminated by the light of the flames, so we will give it an (average) luminance of 300 nit (with some spread, due to the square power law of light dimming, skin texture, etc.).
  • the soldier may be 20 nit, since that is a nicely slightly dark value, still giving some good basic visibility.
  • a vehicle may be hidden in some shadowy corner, and therefore in an archetypical good impact HDR scene of a cave e.g. have a luminance of 0.01 nit.
  • the camera operator would open his iris so that the soldier comes out at "20 nit", or in fact more precisely 20%. Since the flames are much brighter (note: we didn't actually show the real world scene luminances, since master HDR video Mstr_HDR is already an optimal grading to have best impact in a typical living room viewing scenario, but also in the real world the flames would be quite brighter than the soldier, and certainly the vehicle), they would all clip to maximum white.
  • HDR maximum luminance i.e. ML_V
  • EOTF e.g. PQ for coding
  • the problem is that, unless the receiving side has a display which can display pixels at least as bright as 5000 nit, there is still a question of how to display those pixels.
  • Some (DR adaptation) luminance down-mapping must be performed in the TV, to make darker pixels which are displayable.
  • the display has a (end-user) display maximum luminance ML_D of 1500 nit, one could somehow try to calculate 1200 nit yellow pixels for the flame (potentially with errors, like some discoloration, e.g. changing the oranges into yellows).
  • This luminance down-mapping is not really an easy task, especially to do very accurately instead of sufficiently well, and therefore various technologies have been invented (also for the not necessarily similar task of luminance up-mapping, to create an output image of larger dynamic range and in particular maximum luminance than the input image).
  • mapping function (generically, i.e. used for simplicity of elucidation) of a convex shape in a normalized luminance (or brightness) plot, as shown in Fig. 1D .
  • Both input and output luminances are defined here on a range normalized to a maximum equaling one, but one must mind that on the input axis this one corresponds to e.g. 5000 nit, and on the output axis e.g. 200 nit (which to and fro can be easily implemented by division respectfully multiplication).
  • the darkest colors will typically be too dark for the grading with the lower dynamic range of the two images (here for down-conversion shown on the vertical output axis, of normalized output luminances L_out, the horizontal axis showing all possible normalized input luminances L_in).
  • Ergo to have a satisfactory output image corresponding to the input image, we must relatively boost those darkest luminances, e.g. by multiplying by 3x, which is the slope of this luminance compression function F_comp for its darkest end. But one cannot boost forever if one wants no colors to be clipped to maximum output, ergo, the curve must get an increasingly lower slope for brighter input luminances, e.g. it may typically map input 1.0 to output 1.0. In any case the luminance compression function F_comp for down-grading will lie above the 45 degree diagonal (diag) typically.
  • the general desired shape for the brightening of the colors may still be the function F_comp (e.g. determined by the video creator, when grading a secondary image corresponding to his master HDR image already optimally graded), one wants a more savvy down-mapping.
  • F_comp e.g. determined by the video creator, when grading a secondary image corresponding to his master HDR image already optimally graded
  • Fig. 1B for many scenarios one may desire a re-grading which merely changes the brightness of the normalized luminance component (L), but now the innate type of color, i.e. its chromaticity (hue and saturation). If both SDR and HDR are represented with the same red, green and blue color primaries, they will have a similarly shaped gamut tent, only one being higher than the other in absolute luminance representation.
  • both gamuts will exactly overlap.
  • the desired mapping from a HDR color C_H to a corresponding output SDR color C_L (or vice versa) will simply be a vertical shifting, whilst the projection to the chromaticity plane circle stays the same.
  • communication image Im_comm instead of just making some final secondary grading from the master image, e.g. in a television, one can make a lower dynamic range image version for communication , communication image Im_comm.
  • this image was defined with its communication image maximum luminance ML_C equal to 200 nit.
  • the original 5000 nit image can then be reconstructed (a.k.a. decoded) as a reconstructed image Rec_HDR (i.e.
  • the proxy image for actually communicating an image of a higher dynamic range (DR_H, e.g. spanning from 0.001 nit to 5000 nit) is an image of a different, lower dynamic range (DR_L).
  • the communication image can even elect the communication image to be a 100 nit LDR (i.e. SDR) image , which is immediately ready (without further color processing) to be displayed on legacy LDR images (which is a great advantage, because legacy displays don't have HDR knowledge on board).
  • SDR 100 nit LDR
  • legacy LDR images which is immediately ready (without further color processing) to be displayed on legacy LDR images (which is a great advantage, because legacy displays don't have HDR knowledge on board).
  • the legacy TV doesn't recognize the MDWPL metadatum (cos that didn't exist in the SDR video standard, so the TV is also not arranged to go look for it somewhere in the signal, e.g. in a Supplemental Enhancement Information message, which is MPEG's mechanism to introduce all kinds of pre-agreed new technical information). It is also not going to look for the function. It just looks at the YCbCr e.g.
  • the image can be tuned for any possible connected tv, i.e. any ML_D, because one can double the function of the coding function FL_enc as some guidance function for the up-mapping from 100 nit Im_comm not to a 5000 nit reconstructed image, but to e.g. a 1500 nit image.
  • the concave function which is substantially the inverse of F_comp (note, for display tuning there is no requirement of exact inversion as there is for reconstruction), will now have to be scaled to be somewhat less steep (i.e.
  • a display adapted luminance mapping function FL_DA will be calculated), since we expand to only 1500 nit instead of 5000 nit.
  • an image of tertiary dynamic range (DR_T) can be calculated, e.g. optimized for a particular display in that the maximum luminance of that tertiary dynamic range is typically the same as the maximum displayable luminance of a particular display.
  • Fig. 2 shows -in general, without desiring to be limiting- a few typical creations of video where the present teachings may be usefully deployed.
  • an intermediate dynamic range format calculate and format all the needed metadata, convert to some broadcasting format like DVB or ATSC, packetize in chunks for distribution, etc. (the etc. indicating there may be tables added for signaling available content, sub-titling, encryption, but at least some of that will be of lesser interest to understand the details of the present technical innovations).
  • the video (a television broadcast in the example) is communicated via a television satellite 250 to a satellite dish 260 and a satellite signal capable set-top-box 261. Finally it will be displayed on an end-user display 263.
  • This display may be showing this first video, but it may also show other video feed, potentially even at the same time, e.g. in Picture-in-Picture windows (or some data of the first HDR video program may come via some distribution mechanism and other data via another).
  • a second production is typically an off-line production.
  • a Hollywood movie but it can also be a show of somebody having a race through a jungle.
  • Such a production may be shot with other optimal cameras, e.g. steadicam 211 and drone 210.
  • the camera feeds (which may be raw, or already converted to some HDR production format like HLG) are stored somewhere on network 212, for later processing.
  • the video may be uploaded to some internet-based video service 251. For professional video distribution this may be e.g. Netflix.
  • a third example is consumer video production.
  • the user will have e.g. when making a vlog a ring lighter 221, and will capture via a mobile phone 220, but (s)he may also be capturing in some exterior location without supplementary lighting. She/he will typically also upload to the internet, but now maybe to youtube, or tiktok, etc.
  • the display 263 In case of reception via the internet, the display 263 will be connected via a modem, or router 262 or the like (more complicated setups like in-house wifi and the like are not shown in this mere elucidation).
  • Another user may be viewing the video content on a portable display (271), such as a laptop (or similarly other users may use a non-portable desktop PC), or a mobile phone etc. They may access the content over a wireless connection (270), such as Wifi, 5G, etc.
  • a portable display such as a laptop (or similarly other users may use a non-portable desktop PC), or a mobile phone etc. They may access the content over a wireless connection (270), such as Wifi, 5G, etc.
  • Fig. 3 shows an example of an absolute (nit-level-defined) dynamic range conversion circuit 300 for a (HDR) image or video decoder (the encoder working similarly but with inverted functions typically, i.e. the function to be applied being the function of the other side mirrored over the diagonal). It is based on coding a primary image (e.g. a master HDR grading) with a primary luminance dynamic range (DR_Prim) as another (so-called proxy) image with a different secondary range of pixel luminances (DR_Sec).
  • a primary image e.g. a master HDR grading
  • DR_Prim primary luminance dynamic range
  • proxy secondary range of pixel luminances
  • the encoder and all its supply-able decoders have pre-agreed or know that the proxy image has a maximum luminance of 100 nit, this need not be communicated as an SDR_WPL metadatum.
  • the proxy image is e.g. a 200 nit maximum image, this will be indicated by filling its proxy white point luminance P_WPL with the value 200, or similarly for 80 nit etc.
  • the various pixel lumas will typically come in as a luma image plane, i.e. the sequential pixels will have first luma Y11, second Y21, etc. (typically these will be scanned, and the dynamic range conversion circuit will convert pixel by pixel to output pixel color triplets (Y_out, Cb_out, Cr out).
  • Y_out, Cb_out, Cr out pixel color triplets
  • Various dynamic range conversion circuits may internally work differently, to achieve basically the same thing: a correctly reconstructed output luminance L_out for all image pixels (the actual details don't matter for this innovation, and the embodiments will focus on teaching only aspects as far as needed).
  • the mapping of luminances from the secondary dynamic range to the primary dynamic range may happen on the luminances themselves, but also on any luma representation (i.e. according to any EOTF, or OETF), provided it is done correctly, e.g. not separately on non-linear R'G'B' components.
  • the internal luma representation need not even be the one of the input (i.e. of Y_in), or for that manner of whatever output the dynamic range conversion circuitry or its encompassing decoder may deliver (e.g. a format luma Y_sigfm for a particular communication format or communication system, "communicating" including storage to a memory, e.g. inside a PC, a hard disk, an optical storage medium, etc.).
  • a luma conversion circuit 301 which turns the input lumas Y_in into perceptionally uniformized lumas Y_pc.
  • Y_pc log_ 10 1 + RHO WPL_inrep ⁇ 1 ⁇ power Ln_in ; 1 / 2.4 / log_ 10 RHO WPL_inrep
  • RHO WPL_inrep 1 + 32 ⁇ power WPL_inrep / 10000 ; 1 / 2.4
  • the value WPL_inrep is the maximum luminance of the range that needs to be converted to psychovisually uniformized lumas, so for the 100 nit SDR image this value would be 100, and for the to be reconstructed output image (or the originally coded image at the creation side) the value would be 1000.
  • Ln_in are the luminances along that whichever range which need to be converted, after normalization by dividing by its respective maximum luminance, i.e. within range [0,1].
  • this mapping function had been specifically chosen by the encoder of the image (at least for yielding good quality reconstructability, and maybe also a reduced amount of needed bits when MPEG compressing, but sometimes also fulfilling further criteria like e.g. the SDR proxy image being of correct luminance distribution for the particular scene -a dark cave, or a daytime explosion- on a legacy SDR display, etc.).
  • this function F_dec (or its inverse) will be extracted from metadata of the input image signal or representation, and supplied to the dynamic range conversion circuit for doing the actual per pixel luma mapping.
  • the function F_dec directly specifies the needed mapping in the perceptual luma domain, but other variants are of course possible, as the various conversions can also be applied on the functions.
  • the lower and higher dynamic range image will in general have object pixels of different brightness, and oftentimes at least some of the pixels will have different saturation, but ideally the hue of the pixels in both image versions will be the same).
  • This color function will typically specify a multiplier which has a value dependent on a brightness code Y (e.g. the Y_pc, or other codes in other variants).
  • a multiplier establishment circuit 305 will yield the correct multiplier m for the brightness situation of the pixel being processed.
  • a formatting circuit 310 so that the output color triplet (Y_out, Cb_out, Cr_out) can be converted to whatever needed output format (e.g. an RGB format, or a communication YCbCr format, Y_sigfm, Cb_ sigfm, Cr_sigfm).
  • a communication channel 379 which is an HDMI cable
  • such cables typically use PQ-based YCbCr pixel color coding, ergo, the lumas will again be converted from the perceptual domain to the PQ domain by the formatting circuit.
  • a display tuning circuit 380 which calculates ultimate pixel colors and luminances to be displayed at the screen of some display, e.g. a 450 nit tv which some consumer has at home.
  • Fig. 3B for some typical HDR image being an indoors/outdoors image.
  • the outdoors luminances may typically be 100 times brighter than the indoors luminances
  • in an actual master graded HDR image it may be better to make them e.g. 10 ⁇ brighter, since the viewer will be viewing all together on a screen, in a fixed viewing angle, even typically in a dimly illuminated room in the evening, and not in the real world.
  • a convex function as shown in Fig. 1 , or inside luma mapper 302, is used which squeezes in the brighter luminances, due to the limitations of the smaller luminance dynamic range.
  • the indoors objects will display (assuming for the moment an 100 or 200 nit display would faithfully display those luminances as coded, and not e.g. do some arbitrary beautification processing which brightens them) darker, darker than ideally desired, i.e.
  • the outdoors colors may also be somewhat pastellized, i.e. of lowered saturation. But of course if the grader at the creation side has control over all the functions (F_enc, FCOL), he may balance those features, so that some have a lesser deviation at the detriment of others. E.g. if the outdoors shows a plain blue sky, the grader may opt for making it brighter, yet less blue.
  • the middle graph shows what the lumas would look like for the proxy luminances, and that may typically give a more uniform histogram, with e.g. approximately the same span for the indoors and outdoors image object luminances.
  • the lumas are however only relevant to the extent of coding the luminances, or in case some calculations are actually performed in the luma domain (which has advantages for the size of the word length on the processing circuitry). Note that whereas the absolute formalism can allocate luminances on the input side between zero and 100 nit, one can also treat the SDR luminances as relative brightnesses (which is what a legacy display would do, when discarding all the HDR knowledge and communicated metadata, and looking merely at the 0-255 luma and chroma codes).
  • So brightness (grey) driving signal s_driv "0" would show the darkest possible color on the display, and that would be black (e.g. an OLED display would have the local OLED not emitting light), 10% should give 10% output, and 100% would give 100% output, whatever that would be (relative displaying such as legacy SDR displaying would e.g. show white as 150 nit, if that was the display one bought, and the brightness settings one made if not sticking to factory settings; and absolute displays should ideally display the maximum indicated pixel luminance of an image, or some reduced version thereof).
  • the display in a surround illumination which causes the average reflecting grey object around the display to be e.g. 10 nit.
  • his viewing environment may change considerably (usually substantially statically on average, but potentially even dynamically if e.g. the sun gets replaced by incoming storm clouds).
  • One user may be watching that same movie in almost pitch black circumstances, maybe only with the LEDs indicators of his appliances and some other stray light illuminating the room, and another viewer, or the same viewer at a different time, may be watching the movie during sunlit daytime.
  • the primary issue here would be reflections from the lighter environments (e.g. walls) on the front screen of the display, which would mask the image detail in the darkest emitted screen luminances, at least be above zero nit.
  • L_screen_electronically_displayed is the amount of colored light and luminance that gets generated electronically be the display controlled by the incoming image, i.e. eq. 1
  • control processor (510) arranged to adjust the brightness of displaying on a display (520) of at least a dark sub-set of colors in an input image (IMG_in) in response to a measurement of a level of illumination (Lev_illumdrown) of an environment in which the display resides, wherein the adjustment comprises the establishment of an initial adjustment function (F_adj), for shifting one or more color components of input color component triplets (R'_PQ, G'_PQ, B'_PQ) of pixels of the input image by at least one correction value (Adj_brightn) to obtain corresponding output color component triplets (R'_out, G'_out, B'_out), wherein the adjustment comprises multiplying for values of the one or more color components of input color component triplets (R'_PQ, G'_PQ, B'_PQ) below a threshold value (GAM1) the initial adjustment function (F_adj) by a correction function (F_corr) to obtain a
  • a standard surround adjustment function (as explained with Fig. 4 ) would work nicely on many types of image content, since the distribution of the values suggest a reasonable black (black is a qualia in the brain, that can vary to a certain degree: besides the absolute minimum black, dark greys start looking blackish already if their brightness falls below 5% of white).
  • black is a qualia in the brain, that can vary to a certain degree: besides the absolute minimum black, dark greys start looking blackish already if their brightness falls below 5% of white).
  • some part of the movie may be reflecting a somewhat brighter surround object on a screen that reflects specularly, say some lamp shade in the room, and also that bright spot may change if somebody walks in front of the lamp shade.
  • the classical solution may not be preferable, as those objects may look rather greyish instead of true black.
  • the inventor invented a correction strategy, that leaves most of the processing intact (since there may be highly tuned surround light compensation going on in some of the embodiments), but makes at least for the true black objects as defined in the input video a true black to be displayed.
  • the actual color processing can be performed in various color spaces. What is common is that additive display mechanisms typically require three color components to make substantially any color as desired. However, the basis a.k.a. primary colors may vary. E.g., when retaining SDR chromaticities and a so-called narrow gamut, one may re-use the EBU primaries, however, one may give those a substantially increased maximum brightness (compared to the classical CRT, which could make red pixels with a maximum brightness of about 30 nit, and no more, current e.g.
  • OLEDs can make much brighter red primaries, hence make contributions in the color triplet that defines any color of the red component which are brighter than before, hence a larger gamut of brighter colors can be made, despite the components still being defined in an RGB color model).
  • the saturation of the colors which can be made i.e. the color gamut
  • the saturation of the colors which can be made must still fall within the boundaries of the triangle spanned by the elected EBU primaries. If one wants to be able to make more saturated (still RGB) colors, one can use e.g. DCI_P3 or Rec. 2020 color primaries.
  • RGB color model of any non-linear Electro-Optical Transfer Function specification, i.e. by e.g. converting the linear red, green and blue percentual or absolute contributions by a Perceptual Quantizer inverse EOTF which is standardized in SMPTE 2084), which is sometimes necessary in some color processing architectures, then the three color components will get the function-based adjustment as taught.
  • the present improvement method can work in any scenario, as it corrects the initial adjustment function (F_adj) for dark input values (defining dark image colors) below an electable threshold value (GAM1). So in the luma processing this threshold value will be used for the input lumas, and RGB processing will use three times the same threshold value when processing each color channel
  • the threshold value may be chosen to be a pre-fixed small value (e.g. 0.1 nit or smaller, or 0.0001 relatively), but some embodiments may elect it to be equal to a measured surround illumination level (Lev_illumdrown).
  • This value may be evaluated once, e.g. read on startup as the previous value when watching television, and then corrected to be a new value for today's watching, after measuring for e.g. 5 minutes, and then keeping this value, or it can be continuously adapted, but not on too fine a scale as one doesn't; want to see the black level dancing around, but e.g. once every 10 minutes or less often.
  • the correction function multiplies with the initial adjustment function (F_adj), for all possible input values, to get a final adjustment function, which will be the one applied for doing the surround illumination adaptive brightening of mainly the darker pixels (though as explained, if one uses a pure constant addition as initial adjustment this will also change the brightest pixels, but as explained in a visually less significant manner).
  • control processor (510) applies the correction function which is strictly monotonically increasing (note that the initial adjustment function can have e.g. clipping above a maximum value).
  • the initial adjustment function can have e.g. clipping above a maximum value.
  • a strictly monotonically increasing function when making its input higher, cannot have a lower output for a higher input than for a lower input value (though it could have a constant output for some range of increasing input values).
  • a strictly monotonically increasing function must always have a higher output value for a higher input value than for a previous lower input value, i.e. it cannot have flat output value regions.
  • control processor (510) applies its illumination adjustment and hence its correction function on an input and output domain of psychovisually uniformized color component values.
  • a function is the perceptual quantizer functions of SMPTE 2084, i.e. when doing processing on only a luma component one would pre-apply the inverse PQ EOTF on the luminance of a pixel, and then do the illumination adjustment on the luma in the uniformized domain.
  • processing three RGB components in parallel one would apply the inverse PQ EOTF on the three linear red, green and blue component contributions to any incoming pixel color being processed, and then do the illumination adjustment on those three resulting non-linear representations of the color component (the non-linearity indicated typically by a prime, e.g.
  • R' instead of the linear R.
  • the value WPL_inrep is the maximum luminance of the range that needs to be converted to psychovisually uniformized lumas, so for the 100 nit SDR image this value would be 100, and for a 1000 nit maximum image the value would be 1000.
  • Ln_in are the luminances along that whichever range which need to be converted, after normalization by dividing by its respective maximum luminance, i.e. within range [0,1].
  • control processor (510) uses an embodiment of the correction function which has a number of higher order derivatives equal to zero at an input value equal to the threshold value (GAM1).
  • GAM1 threshold value
  • the control processor may reside in any apparatus preparing for display, such as e.g. a computer, or in a display itself.
  • the present innovations may also be embodied in a method of adjusting the brightness of displaying on a display (520) of at least a dark sub-set of colors in an input image (IMG_in) in response to a measurement of a level of illumination (Lev_illumdrown) of an environment in which the display resides, wherein the adjustment comprises the establishment of an initial adjustment function (F_adj), for shifting one or more color components of input color component triplets (R'_PQ, G'_PQ, B'_PQ) of pixels of the input image by at least one correction value (Adj_brightn) to obtain corresponding output color component triplets (R'_out, G'_out, B'_out), wherein the adjustment comprises multiplying for values of the one or more color components of input color component triplets (R'_PQ, G'_PQ, B'_PQ) below a threshold value (GAM1) the initial adjustment function (F_adj) by a correction function (F_corr) to
  • a useful embodiment of the method of adjusting the brightness of displaying as has an embodiment of the correction function is strictly monotonically increasing.
  • a useful embodiment of the method of adjusting the brightness of displaying applies the correction function on an input and output domain of psychovisually uniformized color component values (i.e. both input and output values are pre-transformed from their reference linear representations to any such a domain, typically defined by a corresponding EOTF or OETF), such as by pre-applying an inverse perceptual quantizer electro-optical transfer functions on linear versions of the one or more color components of input color component triplets.
  • the various methods may be embodied as computer program products comprising software code for instructing a processor to execute one of the methods, and e.g. delivered over internet (e.g. a firmware upgrade for the display) etc.
  • Fig. 5 shows a first embodiment of how to realize the improved surround illumination compensation.
  • the control processor (510) is realized as a post-processor, and acts on input color component triplets (red non-linear component R'_PQ, and green and blue non-linear component G'_PQ and B'_PQ) which form pixels of an input image IMG_in of the present innovation's surround illumination adjusting stage, and are outputted from a pre-processing stage, which non-linear components in this example are non-linearized according to the inverse perceptual quantizer EOTF.
  • the control processor comprises three function-based mapping circuits (first 511, second 512, and third 513 function-based mapping circuit) for the respective input component, to yield a respective brightened output component, e.g. non-linear red output R'_out (respectively G'_out, B'_out).
  • a respective brightened output component e.g. non-linear red output R'_out (respectively G'_out, B'_out).
  • R'_out non-linear red output
  • B'_out non-linear red output
  • the shape of the adjustment function (F_adj) depends on a measurement of the surround illumination (Lev_illumdrown), by illumination meter 580. How to use an illumination meter is a detail beyond the scope of the present invention, e.g.
  • the meter may comprise a white light homogenizing dome before sending the incoming light to a measuring electronic component which captures the amount of incoming photons and converts them to a voltage or current, which is typically factory-calibrated into a representative digital value, and a corresponding luminance Lev_illumdrown.
  • the meter may be e.g. on the front of a display.
  • the pre-processing circuit 501 may typically e.g. implement the image processing as described above, i.e. decode and display optimize an input HDR image or video without taking into account a particular (higher) level of surround illumination (i.e. assuming a dark room, or some typical reference viewing illumination, of e.g. dim or dark home movie viewing).
  • the pre-processing circuit 501 will typically contain a luma processing circuit 502, which will adapt the incoming absolute luminances or relative percentual brightnesses, by using some luma mapping function, which will typically depend on characteristics of the display to be served with the images, such as the maximum luminance capability ML_D of the display.
  • the shape of the function may also depend on properties of the image, such as global properties like its average luma or average luminance, or more detailed properties, which may be communicated as metadata (e.g. a guidance luma mapping function F_Lguid).
  • a chroma processing circuit may do chroma processing, ideally corresponding to the luma processing, so that the colors do not look over- or undersaturated.
  • a color transformation circuit 504 transforms the color representation consisting of the mapped luma Y_LM and the processed chromas Cb_pr, and Cr_pr into the needed input format for the control processor, i.e. PQ-based non-linear R'G'B' components.
  • Fig. 6 shows another possible embodiment of the control processor, now as a total color mapping circuit 5101.
  • This circuit can again do some luma mapping in luma mapping circuit 550.
  • a luma mapping function as received as an exterior guiding luma mapping F_Lguid, which may be further optimized into another function shape (e.g. taking into account metadata MET such as a maximum luminance ML_D of a connected display) before applying the present techniques, or already constitute a fully optimized initial adjustment function F_adj for the connected display capabilities and the current viewing environment characterization (i.e. the currently stored or determined level of illumination Lev_illumdrown).
  • the function is however multiplied by the correction function to yield a new bottom part 555 of the final version of the adjustment function F_adj. That function may then be e.g. stored in a LUT, and will thereafter be applied to any of the incoming pixels sequentially scanned from the input image, until e.g. anew scene occurs, and at least the optimal luma mapping for the new scene luma distribution will require a new function shape, or, until a new measurement of the environment illumination requires a new bottom part (and/or a new additive constant Adj_brightn), even when a static mapping curve is used for the entire movie for the display adaptation of the video to the maximum luminance capabilities of the display.
  • the chroma processing circuit 551 may similarly apply any of the chroma adaptation variants already mentioned above, either independent of the measured level of illumination Lev_illumdrown or also in dependence on that level. E.g., preferably the chroma correction may take luminance dependencies into account. One may e.g. similarly as for the RGB parallel processing, use a same or similar correction function also for the bottom part of the Cb and Cr component. Finally a color transformer 552 will be arranged to again do the static color transformation to the needed output format (R'_out, G'_out, B'_out) for driving the display, e.g. PQ-based for communication over HDMI cable, etc.
  • Fig. 7 shows the numerical behaviour of the processing, in an example of the Fig. 5 example.
  • Each of the three function-based mapping circuits (511, 512, and 513) applies such a function, where Ein is the normalized value in the PQ domain of the input to the function and Eout the normalized value of the output result of the function.
  • the initial adjustment function F_adj is here simply a linear compressive function (after having re-distributed all image object lumas of a scene corresponding to the capabilities of the connected display in the pre-processor), in the psychovisually uniformized PQ domain, which maps the zero image pixel values to an output value equalling the measured level of illumination Lev_illumdrown (i.e.
  • F_std would again be the function without taking account of the specific surround illumination of the viewing environment i.e. it would be an identity transform (up to 1000 nit).
  • Eout PQ GAM + min Ein , PQ ML_D ⁇ 1 ⁇ PQ GAM / PQ ML_D in which PQ stands for the inverse Perceptual Quantizer EOTF, GAM is in this example 1 nit (which is relatively dark illumination), and the value of the display maximum luminance ML_D is here chosen to be 1000 nit, i.e. the display can display pixels as bright as 1000 nit
  • this function of Eq. 2 is multiplied by the correction function F_BCor for the darkest pixel values, i.e. values below GAM1 on the input axis.
  • Fig. 9 gives the shapes of these polynomials (from which one may typically want to use order 4 up to 8). We see that if one desires a steep convergence towards the initial adjustment function F_adj, one should use a higher order polynomial, e.g. order 8 or higher.
  • Fig. 8 gives another example of the same procedure of multiplying with a corrective function to obtain good blacks for the actual fully blacks in an image, but now for an illumination level (equivalent luminance) of 10 nit (i.e. a somewhat bright viewing environment). Now it has a higher second threshold value GAM2.
  • the algorithmic components disclosed in this text may (entirely or in part) be realized in practice as hardware (e.g. parts of an application specific integrated circuit) or as software running on a special digital signal processor, or a generic processor, etc. At least some of the elements of the various embodiments may be running on a fixed or configurable CPU, GPU, Digital Signal Processor, FPGA, Neural Processing Unit, Application Specific Integrated Circuit, microcontroller, SoC, etc.
  • the images may be temporarily or for long term stored in various memories, in the vicinity of the processor(s) or remotely accessible e.g. over the internet.
  • the computer program product denotation should be understood to encompass any physical realization of a collection of commands enabling a generic or special purpose processor, after a series of loading steps (which may include intermediate conversion steps, such as translation to an intermediate language, and a final processor language) to enter the commands into the processor, and to execute any of the characteristic functions of an invention.
  • the computer program product may be realized as data on a carrier such as e.g. a disk, data present in a memory, data travelling via a network connection -wired or wireless.
  • characteristic data required for the program may also be embodied as a computer program product.
  • Some of the technologies may be encompassed in signals, typically control signals for controlling one or more technical behaviors of e.g. a receiving apparatus, such as a television.
  • Some circuits may be reconfigurable, and temporarily configured for particular processing by software. Some parts of the apparatuses may be specifically adapted to receive, parse and/or understand innovative signals.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Of Color Television Signals (AREA)
EP23215315.5A 2023-12-08 2023-12-08 Bildanzeigeverbesserung in aufgehellten betrachtungsumgebungen Pending EP4567783A1 (de)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23215315.5A EP4567783A1 (de) 2023-12-08 2023-12-08 Bildanzeigeverbesserung in aufgehellten betrachtungsumgebungen
PCT/EP2024/084019 WO2025119781A1 (en) 2023-12-08 2024-11-29 Image display improvement in brightened viewing environments

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP23215315.5A EP4567783A1 (de) 2023-12-08 2023-12-08 Bildanzeigeverbesserung in aufgehellten betrachtungsumgebungen

Publications (1)

Publication Number Publication Date
EP4567783A1 true EP4567783A1 (de) 2025-06-11

Family

ID=89164174

Family Applications (1)

Application Number Title Priority Date Filing Date
EP23215315.5A Pending EP4567783A1 (de) 2023-12-08 2023-12-08 Bildanzeigeverbesserung in aufgehellten betrachtungsumgebungen

Country Status (2)

Country Link
EP (1) EP4567783A1 (de)
WO (1) WO2025119781A1 (de)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150213781A1 (en) * 2014-01-24 2015-07-30 Acer Incorporated Image processing circuit and method thereof
WO2017108906A1 (en) 2015-12-21 2017-06-29 Koninklijke Philips N.V. Optimizing high dynamic range images for particular displays
WO2017157977A1 (en) 2016-03-18 2017-09-21 Koninklijke Philips N.V. Encoding and decoding hdr videos
US20180102107A1 (en) * 2016-10-10 2018-04-12 Dell Products L.P. Adaptive brightness control for dark display content
WO2022233612A1 (en) 2021-05-07 2022-11-10 Koninklijke Philips N.V. Display-optimized hdr video contrast adaptation

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150213781A1 (en) * 2014-01-24 2015-07-30 Acer Incorporated Image processing circuit and method thereof
WO2017108906A1 (en) 2015-12-21 2017-06-29 Koninklijke Philips N.V. Optimizing high dynamic range images for particular displays
WO2017157977A1 (en) 2016-03-18 2017-09-21 Koninklijke Philips N.V. Encoding and decoding hdr videos
US20180102107A1 (en) * 2016-10-10 2018-04-12 Dell Products L.P. Adaptive brightness control for dark display content
WO2022233612A1 (en) 2021-05-07 2022-11-10 Koninklijke Philips N.V. Display-optimized hdr video contrast adaptation

Also Published As

Publication number Publication date
WO2025119781A1 (en) 2025-06-12

Similar Documents

Publication Publication Date Title
US11887285B2 (en) Encoding and decoding HDR videos
EP3619678B1 (de) Optimierte decodierte hochkontrastbildsättigung
US10902567B2 (en) Handling multiple HDR image sources
EP3430806A1 (de) Codierung und decodierung von hdr-videos
EP3764346A1 (de) Anpassung des anzeigeoptimierungsverhaltens für hdr-bilder
US12462360B2 (en) Display-optimized HDR video contrast adaptation
US12541830B2 (en) Display-optimized ambient light HDR video adapation
EP4567783A1 (de) Bildanzeigeverbesserung in aufgehellten betrachtungsumgebungen
EP4636683A1 (de) Verbesserte luma- und chroma-kartierung für bilder
EP4607917A1 (de) Verbesserte codierung und decodierung für bilder
EP4657423A1 (de) Visuelle zustandsabhängige koordinierte luminanzverarbeitung von assets
US12462359B2 (en) Display-optimized HDR video contrast adaptation
WO2025176561A1 (en) Luminance mapping for images
EP4415343A1 (de) Hdr-ausserfarbskalen-tonabbildung

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN