TWI882003B - Low-latency, low-frequency effects codec - Google Patents

Abstract

In some implementations, a method of encoding a low-frequency effect (LFE) channel comprises: receiving a time-domain LFE channel signal; filtering, using a low-pass filter, the time-domain LFE channel signal; converting the filtered time-domain LFE channel signal into a frequency-domain representation of the LFE channel signal that includes a number of coefficients representing a frequency spectrum of the LFE channel signal; arranging coefficients into a number of subband groups corresponding to different frequency bands of the LFE channel signal; quantizing coefficients in each subband group according to a frequency response curve of the low-pass filter; encoding the quantized coefficients in each subband group using an entropy coder tuned for the subband group; and generating a bitstream including the encoded quantized coefficients; and storing the bitstream on a storage device or streaming the bitstream to a downstream device.

Description

Low latency, low frequency effect codec

The present invention relates generally to audio signal processing, and more particularly to processing low frequency effects (LFE) channels.

For example, standardization efforts for immersive services include the development of an Immersive Sound and Audio Services (IVAS) codec for sound, multi-stream teleconferencing, virtual reality (VR), user-generated live and non-live content streaming. One of the goals of the IVAS standard is to develop a single codec with excellent audio quality, low latency, support for spatial audio coding, an appropriate bit rate range, high-quality error resilience, and a practical implementation complexity. To achieve this goal, it is desirable to develop an IVAS codec that can handle low-latency LFE operations based on IVAS-capable devices or any other device capable of processing LFE signals. The LFE channel is used for deep bass-pitched sounds in the 20Hz to 120Hz range and is typically sent to a speaker designed to reproduce low-frequency audio content.

An implementation scheme of a configurable low-latency LFE codec is disclosed.

In some implementations, a method for encoding a low frequency effects (LFE) channel includes: receiving a time domain LFE channel signal using one or more processors; filtering the time domain LFE channel signal using a low pass filter; converting the filtered time domain LFE channel signal into a frequency domain representation of the LFE channel signal including a certain number of coefficients representing a spectrum of the LFE channel signal using the one or more processors; arranging the coefficients to correspond to different frequency bands of the LFE channel signal using the one or more processors; quantize the coefficients in each sub-band group according to a frequency response curve of the low-pass filter using the one or more processors; encode the quantized coefficients in the sub-band group using an entropy encoder tuned for each sub-band group using the one or more processors; and generate a bit stream including the encoded quantized coefficients using the one or more processors; and store the bit stream on a storage device or stream the bit stream to a downstream device using the one or more processors.

In some implementations, quantizing the coefficients in each sub-band grouping further includes generating a scale shift factor based on a maximum number of available quantization points and a sum of the absolute values of the coefficients; and quantizing the coefficients using the scale shift factor.

In some implementations, if a quantized coefficient exceeds the maximum number of quantization points, the scaling factor is reduced and the coefficients are quantized again.

In some implementations, the quantization points are different for each band grouping.

In some implementations, the coefficients in each sub-band grouping are quantized according to a fine quantization scheme or a coarse quantization scheme, wherein more quantization points are allocated to the respective sub-band groupings using the fine quantization scheme than quantization points assigned to the one or more sub-band groupings according to the coarse quantization scheme.

In some implementations, the sign bits of the coefficients are coded separately from the coefficients.

In some embodiments, there are four sub-band groups, and a first sub-band group corresponds to a first frequency range of 0 Hz to 100 Hz, a second sub-band group corresponds to a second frequency range of 100 Hz to 200 Hz, a third sub-band group corresponds to a third frequency range of 200 Hz to 300 Hz, and a fourth sub-band group corresponds to a fourth frequency range of 300 Hz to 400 Hz.

In some implementations, the entropy coder is an arithmetic entropy coder.

In some implementations, converting the filtered time-domain LFE channel signal into a frequency-domain representation of the LFE channel signal including a certain number of coefficients representing a spectrum of the LFE channel signal further includes: determining a first step length of the LFE channel signal; specifying a first window size of a window function based on the first step length; applying the first window size to one or more frames of the time-domain LFE channel signal; and applying a modified discrete cosine transform (MDCT) to the windowed frames to generate the coefficients.

In some implementations, the method further includes: determining a second step size of the LFE channel signal; specifying a second window size of the window function based on the second step size; and applying the second window size to the one or more frames of the time-domain LFE channel signal.

In some implementations, the first step length is N milliseconds (ms), N is greater than or equal to 5ms and less than or equal to 60ms, the first window size is greater than or equal to 10ms, the second step length is 5ms and the second window size is 10ms.

In some implementations, the first step length is 20 milliseconds (ms), the first window size is 10ms or 20ms or 40ms, the second step length is 10ms and the second window size is 10ms or 20ms.

In some implementations, the first step size is 10 milliseconds (ms), the first window size is 10ms or 20ms, the second step size is 5ms, and the second window size is 10ms.

In some implementations, the first step size is 20 milliseconds (ms), the first window size is 10ms, 20ms, or 40ms, the second step size is 5ms, and the second window size is 10ms.

In some embodiments, the window function is a Kaiser-Besso derived (KBD) window function with a configurable gradient length.

In some embodiments, the low-pass filter is a fourth-order Butterworth low-pass filter having a cutoff frequency of about 130 Hz or less.

In some implementations, the method further includes: using the one or more processors to determine whether an energy level of a frame of the LFE channel signal is lower than a threshold value; generating a silence frame indicator to indicate the decoder according to the energy level being lower than a threshold level; inserting the silence frame indicator into the metadata of the LFE channel bit stream; and reducing an LFE channel bit rate when a silence frame is detected.

In some implementations, a method for decoding a low frequency effect (LFE) channel bit stream includes: receiving, using one or more processors, an LFE channel bit stream, the LFE channel bit stream comprising entropy-coded coefficients representing a spectrum of a time-domain LFE channel signal; decoding, using the one or more processors, the quantized coefficients using an entropy decoder; and dequantizing, using the one or more processors, the quantized coefficients, wherein the coefficients quantized in a sub-band group corresponding to a frequency band according to a frequency response curve of a low-pass filter used to filter the time-domain LFE channel signal in an encoder; converting the inversely quantized coefficients into a time-domain LFE channel signal using the one or more processors; adjusting a delay of the time-domain LFE channel signal using the one or more processors; and filtering the delay-adjusted LFE channel signal using a low-pass filter.

In some implementations, an order of the low pass filter is configured to ensure that a first total algorithm delay due to encoding and decoding the LFE channel in a multi-channel audio signal including the LFE channel signal is less than or equal to a second total algorithm delay due to encoding and decoding other channels.

In some implementations, the method further includes: determining whether the second total algorithm delay exceeds a critical value; and configuring the low-pass filter as an N-order low-pass filter according to the second total algorithm delay exceeding the critical value, wherein N is an integer greater than or equal to 2; and configuring the order of the low-pass filter to be less than N according to the second total algorithm delay not exceeding the critical value.

Other embodiments disclosed herein relate to a system, apparatus, and computer-readable medium. Details of one or more disclosed embodiments are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

Specific embodiments disclosed herein provide one or more of the following advantages. The disclosed low-latency LFE codec: 1) is primarily targeted at the LFE channel; 2) is primarily targeted at a frequency range of 20 Hz to 120 Hz, but carries audio up to 300 Hz in low/medium bit rate scenarios and up to 400 Hz in high bit rate scenarios; 3) achieves a low bit rate by applying a quantization scheme based on a frequency response curve of an input low-pass filter; 4) has a low algorithmic delay and is designed to operate in a step size of 20 milliseconds (ms) and has a total algorithmic delay of 33 ms (including framing); 5) can be configured to smaller steps and lower algorithm delays to support other scenarios, including configuration to steps as low as 5 ms and 13 ms total algorithm delay (including framing); 6) automatically select a low pass filter at the decoder output based on the available delay of the LFE codec; 7) have a silent mode with a low bit rate of 50 bits per second (bps) during silent periods; and 8) during active frames, the bit rate fluctuates between 2 kilobits per second (kbps) and 4 kbps based on the quantization level used, and during silent frames the bit rate is 50 bps.

100: Immersive sound and audio service codec

101: Audio data

102: Spatial analysis and downmixing unit

103: Main audio channel encoding unit/main audio channel encoder

104: Spatial metadata coding unit

105: Low frequency effect channel coding unit

106: Spatial metadata decoding unit

107: Main audio channel decoding unit

108: Low frequency effect channel decoding unit

109: Spatial synthesis/upmixing/presentation unit

201: Input low-pass filter/fourth-order input low-pass filter/low-pass filter

202: Modified Discrete Cosine Transformation Unit

203: Quantization and entropy coding unit

204: Entropy decoding and inverse quantization unit

205: Inverse modified discrete cosine transform and windowing unit

206: Delay adjustment unit

207: Output low-pass filter/Low-pass filter/Fourth-order output low-pass filter/Second-order output low-pass filter

900:Procedure

901: Steps

902: Steps

903: Steps

904: Steps

905: Steps

906: Steps

907: Steps

908: Steps

1000:Program

1001: Steps

1002: Steps

1003: Steps

1004: Steps

1005: Steps

1100:System

1101: Central processing unit

1102: Read-only memory

1103: Random Access Memory

1104: Bus

1105: Input/output interface

1109: Communication unit

In the drawings, the specific configuration or order of schematic elements (e.g., elements representing devices, units, instruction blocks, and data elements) is shown for ease of illustration. However, those skilled in the art will understand that the specific order or configuration of schematic elements in the drawings is not intended to imply the need for a particular processing order or sequence or process separation. Furthermore, the inclusion of a schematic element in a drawing is not intended to imply that such element is required in all embodiments, or that the features represented by such elements may not be included in some embodiments or combined with other elements in some embodiments.

In addition, in the drawings, connecting elements (such as solid or dashed lines or arrows) are used to illustrate a connection, relationship or association between or among two or more other schematic elements, and the absence of any such connecting elements is not intended to imply that the connection, relationship or association cannot exist. In other words, some connections, relationships or associations between elements are not shown in the drawings to avoid obscuring the present invention. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, when a connecting element represents a communication of a signal, data or instruction, those skilled in the art should understand that such elements represent one or more signal paths that affect the communication as needed.

FIG. 1 illustrates an IVAS codec for encoding and decoding IVAS and LFE bit streams according to one or more implementations.

FIG. 2A is a block diagram illustrating LFE encoding according to one or more implementations.

FIG. 2B is a block diagram illustrating LFE decoding according to one or more implementations.

FIG. 3 is a graph illustrating a frequency response of a fourth-order Butterworth low-pass filter having a corner cutoff point of 130 Hz according to one or more embodiments.

FIG. 4 is a graph illustrating a Fielder window according to one or more implementations.

FIG5 illustrates the variation of fine quantization points with frequency according to one or more implementations.

FIG6 illustrates the variation of coarse quantization points with frequency according to one or more implementations.

FIG. 7 illustrates a probability distribution of quantized MDCT coefficients with fine quantization according to one or more implementations.

FIG8 illustrates a probability distribution of quantized MDCT coefficients with coarse quantization according to one or more implementations.

FIG. 9 is a flow chart of a process for encoding modified discrete cosine transform (MDCT) coefficients according to one or more implementations.

FIG. 10 is a flow chart of a process for decoding modified discrete cosine transform (MDCT) coefficients according to one or more implementation schemes.

FIG. 11 is a block diagram of a system for implementing the features and procedures described with reference to FIGS. 1 to 10 according to one or more implementation schemes.

The same reference symbols are used in the various drawings to indicate similar elements.

In the following detailed description, numerous specific details are set forth to provide a thorough understanding of one of the various described embodiments. Those skilled in the art will appreciate that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits are not described in detail to avoid unnecessarily obscuring the aspects of the embodiments. Several features are described below that may be used independently of each other or in any combination with other features.

Nomenclature

As used herein, the term "including" and its variations are interpreted as open-ended terms, meaning "including but not limited to". The term "or" is interpreted as "and/or", unless the context clearly indicates otherwise. The term "based on" is interpreted as "based at least in part on". The terms "an exemplary embodiment" and "exemplary embodiment" are interpreted as "at least one exemplary embodiment". The term "another embodiment" is interpreted as "at least one other embodiment". The term "determined" or "determined" is interpreted as obtaining, receiving, calculating, computing, estimating, predicting or deriving. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the invention belongs.

System Overview

FIG. 1 illustrates an IVAS codec 100 for encoding and decoding an IVAS bitstream (including an LFE channel bitstream) according to one or more embodiments. During encoding, the IVAS codec 100 receives N+1 channels of audio data 101, wherein the N channels of audio data 101 are input to a spatial analysis and downmix unit 102, and one LFE channel is input to an LFE channel encoding unit 105. The audio data 101 includes, but is not limited to: a mono signal, a stereo signal, a binaural signal, a spatial audio signal (e.g., a multi-channel spatial audio object), first-order Ambisonics (FoA), higher-order Ambisonics (HoA), and any other audio data.

In some implementations, the spatial analysis and downmix unit 102 is configured to implement complex advanced coupling (CACPL) for analysis/downmixing of stereo audio data, and/or spatial reconstruction (SPAR) for analysis/downmixing of FoA audio data. In other implementations, the spatial analysis and downmix unit 102 implements other formats. The output of the spatial analysis and downmix unit 102 includes spatial metadata and 1 to N channels of audio data. The spatial metadata is input to the spatial metadata encoding unit 104, which is configured to quantize and entropy code the spatial metadata. In some implementations, quantization may include fine, moderate, coarse, and extra-coarse quantization strategies, and entropy coding may include Huffman or arithmetic coding.

1 to N channels of audio data are input to the main audio channel coding unit 103, which is configured to encode 1 to N channels of audio data into one or more enhanced voice service (EVS) bit streams. In some implementations, the main audio channel coding unit 103 complies with 3GPP TS 26.445 and provides a variety of functionalities, such as enhancing quality and coding efficiency for narrowband (EVS-NB) and wideband (EVS-WB) voice services, enhancing quality of voice using ultra-wideband (EVS-SWB), enhancing the quality of mixed content and music in conversational applications, robustness to packet loss and delay jitter, and backward compatibility with AMR-WB codecs.

In some embodiments, the main audio channel coding unit 103 includes a pre-processing and mode selection unit that selects between a speech codec for encoding speech signals and a perceptual codec for encoding audio signals at a specified bit rate based on mode/bit rate control. In some embodiments, the speech codec is a modified variation of Algebraic Coded Excitation Linear Prediction (ACELP) that is extended with specialized LP-based modes for different speech categories.

In some implementations, the audio codec is a modified discrete cosine transform (MDCT) codec that has improved efficiency at low latency/low bit rates and is designed to perform seamless and reliable switching between the speech codec and the audio codec.

As previously explained, the LFE channel signal is used for deep bass sound in the range of 20 Hz to 120 Hz and is typically sent to a speaker (e.g., a subwoofer) designed to reproduce low-frequency audio content. The LFE channel signal is input to the LFE channel signal encoding unit 105, which is configured to encode the LFE channel signal as explained with reference to FIG. 2A.

In some embodiments, an IVAS decoder includes: a spatial metadata decoding unit 106 configured to recover spatial metadata; and a main audio channel decoding unit 107 configured to recover 1 to N channel audio signals. The recovered spatial metadata and the recovered 1 to N channel audio signals are input into a spatial synthesis/upmixing/presentation unit 109, which is configured to use the spatial metadata to synthesize and present the 1 to N channel audio signals into N or more channel output audio signals for playback on speakers of various audio systems, including but not limited to: home theater systems, video conference room systems, virtual reality (VR) equipment, and any other audio system capable of presenting audio. The LFE channel decoding unit 108 receives the LFE bit stream and is configured to decode the LFE bit stream, as described with reference to FIG. 2B .

Although the exemplary implementation of LFE encoding/decoding described above is implemented by an IVAS codec, the low-latency LFE codec described below may be a standalone LFE codec, or it may be included in any dedicated or standardized audio codec that encodes and decodes low frequency signals in audio applications where low latency and configurability are required or desired.

FIG. 2A is a block diagram illustrating functional components of the LFE channel encoding unit 105 shown in FIG. 1 according to one or more embodiments. FIG. 2B is a block diagram illustrating functional components of the LFE channel decoder 108 shown in FIG. 1 according to one or more embodiments. The LFE channel decoder 108 includes an entropy decoding and inverse quantization unit 204, an inverse MDCT and windowing unit 205, a delay adjustment unit 206, and an output LPF 207. The delay adjustment unit 206 may be located before or after the LPF 207 and performs delay adjustment (e.g., by buffering the decoded LFE channel signal) to match the decoded LFE channel signal with the decoded output of the main encoder/decoder. In the following text, the LFE channel encoding unit 105 and the LFE channel decoding unit 108 described in reference to FIG. 2B are collectively referred to as an LFE codec.

The LFE channel coding unit 105 includes an input low pass filter (LPF) 201, a windowing and MDCT unit 202, and a quantization and entropy coding unit 203. In one embodiment, the input audio signal is a pulse code modulation (PCM) audio signal, and the LFE channel coding unit 105 expects an input audio signal with a step size of 5 ms, 10 ms, or 20 ms. Internally, the LFE channel coding unit 105 operates on 5 ms or 10 ms subframes, and performs windowing and MDCT on a combination of these subframes. In one embodiment, the LFE channel coding unit 105 operates with a 20 ms input step size and internally divides the input into two subframes of equal length. The last subframe of the previous input frame to the LFE is concatenated with the first subframe of the current input frame to the LFE and is windowed. The first subframe of the current input frame to the LFE is concatenated with the second subframe of the current input frame to the LFE and is windowed. The MDCT is performed twice, once on each windowed block.

In one embodiment, the algorithmic delay (excluding the framing delay) is equal to 8 ms + the delay caused by the input LPF 103 + the delay caused by the output LPF 207. In the case of a four-stage input LPF 201 and a four-stage output LPF 207, the total system delay is about 15 ms. In the case of a four-stage input LPF 201 and a two-stage output LPF 207, the total LFE codec delay is about 13 ms.

FIG. 3 is a graph illustrating a frequency response of an exemplary input LPF 201 according to one or more embodiments. In the example shown, LPF 201 is a fourth-order Butterworth filter with a cutoff frequency of 130 Hz. Other embodiments may use a different type of LPF (e.g., Chebyshev, Bessel) with the same or different order and the same or different cutoff frequency.

FIG. 4 is a graph illustrating a plot of a Field window according to one or more embodiments. In one embodiment, the window function applied by the windowing and MDCT unit 202 is a Field window function with a gradual concealment length of 8 milliseconds. The Field window is a Kaiser-Besso derived (KBD) window with α=5, which is constructed to satisfy the Princen-Bradley condition of the MDCT and is therefore a window used in the Advanced Audio Coding (AAC) digital audio format. Other window functions may also be used.

Quantization and entropy coding

In one embodiment, the quantization and entropy coding unit 203 implements a quantization strategy that conforms to the frequency response curve of the input LPF 201 to more efficiently quantize the MDCT coefficients. In one embodiment, the frequency range is divided into 4 sub-band groups representing 4 frequency bands: 0Hz to 100Hz, 100Hz to 200Hz, 200Hz to 300Hz, and 300Hz to 400Hz. These frequency bands are examples, and more or fewer frequency bands can be used with the same or different frequency ranges. More specifically, the MDCT coefficients are quantized using a scaling factor that is dynamically calculated based on the MDCT coefficient values in a particular frame, and the quantization points are selected according to the LPF frequency response curve, as shown in Figures 5 to 8. This quantization strategy helps reduce the quantization points of the MDCT coefficients belonging to the 100Hz to 200Hz, 200Hz to 300Hz, and 300Hz to 400Hz bands, while retaining the best quantization points for the main LFE band 0Hz to 100Hz, where the lowest frequency effect energy (e.g., rumble) will exist.

In one embodiment, a quantization strategy for an F _len millisecond (ms) input PCM step (input frame length) to the LFE channel coding unit 105 is described below, where the frame length F _len can take any value given by 5* f ms, where 1<= f <=12.

First, divide the input PCM stride into N subframes of equal length, each subframe width (S _w ) = F _len /N ms. N should be selected so that each Sw is a multiple of 5 _ms (for example, if F _len =20 ms, then N can be 1, 2 or 4; if F _len =10 ms, then N can be 1 or 2; and if F _len =5 ms, then N is equal to 1). Let S _i be the i- th subframe in any given frame, where i is an integer in the range 0 <= i <= N , where S ₀ corresponds to the last subframe in the previous input frame to the LFE coding unit 105, and S ₁ to S _N are the N subframes in the current frame.

Next, each Si and Si _{+ 1} subframe is _concatenated and windowed using a Field window (see FIG. 4 ), and then MDCT is performed on these windowed samples. This results in a total of N MDCTs being performed on each frame. The number of MDCT coefficients from each MDCT ( num_coeffs ) = sampling frequency * Sw /1000. The frequency resolution of each MDCT (width of each _{MDCT coefficient) (Wmdct} ) is approximately 1000/(2* Sw ) Hz _. Given that subwoofers typically have an LPF cutoff point of approximately 100 Hz to 120 Hz, and the post-LPF energy after 400 Hz is typically very low, the MDCT coefficients up to 400 Hz are quantized and sent to the LFE decoding unit 108, while the remainder of the _MDCT coefficients are quantized to zero. Sending MDCT coefficients up to 400 Hz ensures high quality reconstruction up to 120 Hz at the LFE decoding unit 108. Therefore, the total number of MDCT coefficients used for quantization and _coding ( Nquant ) is equal to _N*400/Wmdct .

Next, the MDCT coefficients are arranged in M sub-band groups, where the width of each sub-band group is a multiple of W _mdct and the sum of the widths of all sub-band groups is equal to 400 Hz. Let the width of each sub-band be SBW _m Hz, where m is an integer in the range 1 <= m <= M. At this width, the number of coefficients in the mth sub-band group = SN _quant = N * SBW _m /W _mdct (i.e., the SBW _m /W _mdct coefficients from each MDCT). The MDCT coefficients in each sub-band group are then scaled according to a shift scaling factor (shift) as explained below, which is determined by the sum or maximum of the absolute values of all N _quant MDCT coefficients. The scaled MDCT coefficients in each sub-band group are then quantized and coded individually at the encoder input using a quantization scheme that conforms to the LPF curve. The quantized MDCT coefficients are coded using an entropy coder (e.g., an arithmetic or Huffman coder). Each sub-band group is coded using a different entropy coder, and each entropy coder uses an appropriate probability distribution model to efficiently code the respective sub-band group.

An exemplary quantization strategy with a 20 ms stride ( F _len = 20 ms), 2 subframes ( N = 2) and sampling frequency = 48000 will now be described. In this exemplary input configuration, the subframe width Sw ₌ 10 ms and the number of MDCTs = N = 2. The first MDCT is performed on a 20 ms block. This block is formed by concatenating a 10 ms to 20 ms subframe in the previous 20 ms input with a 0 ms to 10 ms subframe in the current 20 ms input, and then windowing a 20 ms long Field window (see FIG. 4 ). In the case of N = 1 and N = 4, the Field window is scaled accordingly and the gradient length is changed to 16/N ms. The second MDCT is performed on a 20ms block formed by windowing the current 20ms input frame using a 20ms long Field window. The number of MDCT coefficients per MDCT ( num_coeffs ) = 480, the width of each MDCT coefficient Wmdct = _50Hz , the total number of _quantized and coded coefficients Nquant = 16, and the total number of quantized and coded coefficients/MDCT = 16/N = 8.

Next, the MDCT coefficients are arranged in 4 sub-band groups ( _M =4), where each sub-band group corresponds to a 100 Hz band (0 to 100, 100 to 200, 200 to 300, 300 to 400, SBWm = _100Hz , the number of coefficients in each sub- _band group = SNquant = N * SBWm / _Wmdct = 4). Let _a1 , _a2 , _a3 , _a4 , _a5 , _a6 , _a7 , _a8 be the first 8 MDCT coefficients to be quantized from the first MDCT, and _b1 , _b2 , _b3 , _b4 , _b5 , _b6 , _b7 , _b8 be the first 8 MDCT coefficients to be quantized from the second MDCT. The four sub-band groups are configured to have the following coefficients: sub-band group 1 = {a ₁ , a ₂ , b ₁ , b ₂ }, sub-band group 2 = {a ₃ , a ₄ , b ₃ , b ₄ }, sub-band group 3 = {a ₅ , a ₆ , b ₅ , b ₆ }, sub-band group 4 = {a ₇ , a ₈ , b ₇ , b ₈ }, where each sub-band group corresponds to a 100 Hz frequency band.

A frame with a gain of approximately -30 dB (or less than -30 dB) may have MDCT coefficients with values greater than ^10-2 or ^10-1 or less, while a frame with full -scale gain may have MDCT coefficients with values of 20 or more. To accommodate this wide range of values, a scaling shift factor (shift ) is calculated based on the maximum available quantization point ( max_value ) and a sum of the absolute values of the MDCT coefficients ( lfe_dct_new ), as follows : shift = floor ( shift s_per_double *log2( max_value / sum(abs (lfe_dct_new ) ))).

In one embodiment, lfe_dct_new is an array of 16 MDCT coefficients, shifts_per_double is a constant (e.g., 4 ) , max_value is an integer selected for fine quantization (e.g., 63 quantization values) and coarse quantization ( e.g., 31 quantization values), and shifts are limited to 5-bit values from 4 to 35 in fine quantization and 2 to 33 in coarse quantization.

The quantized MDCT coefficients are then calculated as follows: vals = round( lfe _ dct _ new *(2^( shift/shifts _ per _ double ))), where the round() operation rounds the result to the nearest integer value.

If the quantized value ( vals ) exceeds the maximum allowed number of available quantization points ( max_val ), the scaling shift factor ( shift ) is reduced and the quantized value ( vals ) is calculated again. In other implementation schemes, instead of the sum function sum ( abs ( lfe_dct_new ))), the maximum function max(abs( lfe_dct_new ) )) can be used to calculate the scaling shift factor ( shift ), but using the max() function will make the quantized values more dispersed, making it more difficult to design an efficient entropy coder.

In the quantization step described above, the quantized values of each sub-band grouping are calculated together in one loop, but the quantization point is different for each sub-band grouping. If the first sub-band grouping exceeds the allowed range, the scaling factor is reduced. If any of the other sub-band groups exceeds the allowed range, the sub-band grouping is deleted to max_value . The sign bits of all MDCT coefficients and the absolute value of the quantized MDCT coefficients are coded separately for each sub-band grouping.

FIG. 5 illustrates the variation of fine quantization points with frequency according to one or more embodiments. In fine quantization, subband group 1 (0 Hz to 100 Hz) has 64 quantization points, subband group 2 (100 Hz to 200 Hz) has 32 quantization points, subband group 3 (200 Hz to 300 Hz) has 8 quantization points, and subband group 4 (300 Hz to 400 Hz) has 2 quantization points. In one embodiment, each subband group is entropy-coded using an entropy coder (e.g., an arithmetic or Huffman entropy coder), where each entropy coder uses a different probability distribution. Therefore, the main 0 Hz to 100 Hz range is assigned the most quantization points.

Note that the assignment of quantization points to subband group 1 to subband group 4 follows the shape of the LPF frequency response curve, which has more information in lower frequencies than in higher frequencies and no information outside the cutoff frequency. To properly reconstruct frequencies up to 130 Hz, the MDCT coefficients corresponding to frequencies above 130 Hz are also encoded to avoid or minimize frequency overlap. In some implementations, MDCT coefficients up to 400 Hz are encoded so that frequencies up to 130 Hz can be properly reconstructed at the decoding unit.

FIG6 illustrates the variation of coarse quantization points with frequency according to one or more implementations. In coarse quantization, subband group 1 (0 Hz to 100 Hz) has 32 quantization points, subband group 2 (100 Hz to 200 Hz) has 16 quantization points, subband group 3 (200 Hz to 300 Hz) has 4 quantization points, and subband group 4 (300 Hz to 400 Hz) is not quantized and entropy coded. In one embodiment, entropy coding is performed on each subband group using a separate entropy coder using a different probability distribution.

FIG. 7 illustrates a probability distribution of quantized MDCT coefficients in fine quantization according to one or more implementations. The y-axis is the frequency of occurrence and the x-axis is the number of quantization points. Sg1 is subband group 1 corresponding to quantized MDCT coefficients in the 0 Hz to 100 Hz band, Sg2 is subband group 2 corresponding to quantized MDCT coefficients in the 100 Hz to 200 Hz band. Sg3 is subband group 3 corresponding to quantized MDCT coefficients in the 200 Hz to 300 Hz band. Sg4 is subband group 4 corresponding to quantized MDCT coefficients in the 300 Hz to 400 Hz band.

FIG8 illustrates a probability distribution of quantized MDCT coefficients during coarse quantization according to one or more implementations. The y-axis is the frequency of occurrence and the x-axis is the number of quantization points. Sg1 is subband group 1 corresponding to quantized MDCT coefficients in the 0 Hz to 100 Hz band, Sg2 is subband group 2 corresponding to quantized MDCT coefficients in the 100 Hz to 200 Hz band. Sg3 is subband group 3 corresponding to quantized MDCT coefficients in the 200 Hz to 300 Hz band. Sg4 is subband group 4 corresponding to quantized MDCT coefficients in the 300 Hz to 400 Hz band.

Note that the main band (0 Hz to 100 Hz) is the band where the LFE effect is found most and therefore more quantization points are allocated to achieve greater resolution. However, fewer bits are allocated to the main band in coarse quantization than fine quantization. In one embodiment, whether fine quantization or coarse quantization is used for a frame of MDCT coefficients depends on the desired target bit rate set by the main audio channel encoder 103. The main audio channel encoder 103 sets this value once during initialization or dynamically on a frame-by-frame basis based on the bits required or used to encode the main audio channel in each frame.

Silent message frame

In some implementations, a signal is added to the LFE channel bitstream to indicate a silent frame. A silent frame is a frame with energy below a specified threshold. In some implementations, 1 bit is included in the LFE channel bitstream transmitted to the decoder (e.g., inserted in a frame header) to indicate a silent frame, and all MDCT coefficients in the LFE channel bitstream are set to 0. This technique can reduce the bit rate to 50bps during silent frames.

Decoder LPF

Two options for implementing LPF 207 (see FIG. 2B ) are provided at the output of the LFE channel decoding unit 108. LPF 207 is selected based on the available delay (total delay of other audio channels minus LFE fade delay minus input LPF delay). Note that other channels are expected to be encoded/decoded by the main audio channel encoding unit 103/main audio channel decoding unit 107 and their delays depend on the algorithmic delay of the main audio channel encoding unit 103/main audio channel decoding unit 107.

In one embodiment, if the available delay is less than 3.5ms, a second-order Butterworth LPF with a cutoff point of 130Hz is used; otherwise, a fourth-order Butterworth LPF with a cutoff point of 130Hz is used. Therefore, at the LFE channel decoding unit 108, a trade-off needs to be made between removing the folded frequency energy outside the cutoff frequency and the algorithm delay. In some embodiments, LPF 207 can be completely removed because the subwoofer usually has an LPF. LPF 20 helps to reduce the folded frequency energy outside the cutoff point of the LFE decoder output itself, and can help efficient post-processing.

Example procedures

FIG. 9 is a flow chart of a process 900 for encoding MDCT coefficients according to one or more implementations. Process 900 may be implemented using, for example, system 1100 as described with reference to FIG. 11 .

The process 900 includes the following steps: receiving a time domain LFE channel signal (901); filtering the time domain LFE channel signal using a low pass filter (902); converting the filtered time domain LFE channel signal into a frequency domain representation of the LFE channel signal including a certain number of coefficients representing a spectrum of the LFE channel signal (903); allocating the coefficients to a certain number of sub-band groups corresponding to different frequency bands of the LFE channel signal (904); 904); quantizing the coefficients in each sub-band group using a scaling factor according to a frequency response curve of a low pass filter (905); encoding the quantized coefficients in each sub-band group using an entropy encoder configured for the sub-band group (906); generating a bit stream including the encoded quantized coefficients (907); and storing the bit stream on a storage device or transmitting the bit stream to a downstream device (908).

FIG. 10 is a flow chart of a process 1000 for decoding MDCT coefficients according to one or more implementations. Process 1000 may be implemented using, for example, system 1100 as described with reference to FIG. 11 .

Process 1000 includes the following steps: receiving an LFE channel bit stream (1001), wherein the LFE channel bit stream includes entropy coded coefficients representing a spectrum of a time-domain LFE channel signal; decoding and inverse quantizing the coefficients (1002), wherein the coefficients are quantized in sub-band groups corresponding to different frequency bands using a scale shift factor according to a frequency response curve of a low-pass filter; converting the decoded and inverse quantized coefficients into a time-domain LFE channel signal (1003); adjusting a delay of the time-domain LFE channel signal (1004); and filtering the delay-adjusted LFE channel signal using a low-pass filter (1005). In one embodiment, the order of the low pass filter may be configured based on a total algorithmic delay available from a main codec for encoding/decoding the full bandwidth channels of a multi-channel audio signal including a time domain LFE channel signal. In some embodiments, the decoding unit only needs to know whether the coding unit encodes the MDCT coefficients using fine quantization or coarse quantization. The quantization type may be indicated using a bit in the LFE bit stream header or any other suitable signaling mechanism.

In some implementations, decoding of the inverse quantized coefficients into time domain PCM samples is performed as follows: The inverse quantized coefficients in each sub-band grouping are re-arranged into N groups (N being the number of MDCTs operated at the coding unit), where each group has coefficients corresponding to a respective MDCT. According to the exemplary implementation scheme described above, the encoding unit encodes the following four sub-band groups: sub-band group 1 = {a ₁ , a ₂ , b ₁ , b ₂ }, sub-band group 2 = {a ₃ , a ₄ , b ₃ , b ₄ }, sub-band group 3 = {a ₅ , a ₆ , b ₅ , b ₆ }, sub-band group 4 = {a ₇ , a ₈ , b ₇ , b ₈ }.

_The decoding unit decodes and reconfigures the 4 subband groups back to {a ₁ ,a ₂ ,a ₃ ,a 4 ,a ₅ ,a ₆ ,a ₇ ,a ₈ } and {b ₁ ,b ₂ ,b ₃ ,b ₄ ,b ₅ ,b ₆ ,b ₇ ,b ₈ } and then pads the groups with zeros to get the desired inverse MDCT (iMDCT) input length. The iMDCT is performed N times to inverse transform the MDCT coefficients in each group into time domain blocks. In this example, each block is 2*Sw ms wide, where Sw is the subframe width defined above. Next, this block is windowed by the same Field window used by the LFE encoding unit shown in Figure 4. Each subframe S _i (i is an integer between 1 <= i <= N ) is reconstructed by appropriately superimposing the windowed data of the previous iMDCT output with the current iMDCT output. Finally, the output of (1003) is reconstructed by concatenating all N subframes.

Example system architecture

FIG. 11 is a block diagram of a system 1100 for implementing the features and procedures described in reference to FIG. 1 to FIG. 10 according to one or more implementation schemes. System 1100 includes one or more server computers or any client devices, including but not limited to: call servers, user equipment, conference room systems, home theater systems, virtual reality (VR) equipment, and immersive content capture devices. System 1100 includes any consumer device, including but not limited to: smart phones, tablet computers, wearable computers, car computers, game consoles, surround systems, information stations, etc.

As shown, the system 1100 includes a central processing unit (CPU) 1101, which can execute various programs according to a program stored in, for example, a read-only memory (ROM) 1102 or a program loaded from, for example, a storage unit 1108 to a random access memory (RAM) 1103. Data required when the CPU 1101 executes various programs is also stored in RAM 1103 as needed. CPU 1101, ROM 1102, and RAM 1103 are connected to each other via a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.

The following components are connected to the I/O interface 1105: an input unit 1106, which may include a keyboard, a mouse, etc.; an output unit 1107, which may include a display, such as a liquid crystal display (LCD) and one or more speakers; a storage unit 1108, which includes a hard disk or another suitable storage device; and a communication unit 1109, which includes a network interface card, such as a network card (e.g., wired or wireless).

In some implementations, input unit 1106 includes one or more microphones located in different locations (depending on the host device) and capable of capturing audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

In some implementations, output unit 1107 includes a system having a varying number of speakers. Output unit 1107 (depending on the capabilities of the host device) can present audio signals in a variety of formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

The communication unit 1109 is configured to communicate with other devices (e.g., via a network). A disk drive 1110 is also connected to the I/O interface 1105 as needed. A removable medium 1111 (e.g., a disk, an optical disk, a magneto-optical disk, a flash disk drive, or another suitable removable medium) is installed on the disk drive 1110 so that a computer program read from the removable medium 1111 can be installed into the storage unit 1108 as needed. Those skilled in the art should understand that although the system 1100 is described as including the components described above, in actual applications, some of these components may be added, removed, and/or replaced and all such modifications or changes are within the scope of the present invention.

According to exemplary embodiments of the present invention, the procedures described above may be implemented as a computer software program or implemented on a computer-readable storage medium. For example, an embodiment of the present invention includes a computer program product, which includes a computer program tangibly embodied on a machine-readable medium, and the computer program includes a program code for implementing the method. In these embodiments, the computer program can be downloaded and installed from the network via the communication unit 1109, and/or installed from the removable medium 1111.

In general, various exemplary embodiments of the present invention may be implemented as hardware or special purpose circuits (e.g., control circuit systems), software, logic, or any combination thereof. For example, the units discussed above may be executed by a control circuit system (e.g., a CPU in combination with other components of FIG. 11 ), so that the control circuit system may be performing the actions described in the present invention. Some aspects may be implemented as hardware, while other aspects may be implemented as firmware or software that may be executed by a controller, microprocessor, or other computing device (e.g., control circuit system). Although various aspects of exemplary embodiments of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that these blocks, devices, systems, techniques, or methods described herein may be implemented in (as non-limiting examples) hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers, or other computing devices, or some combination thereof.

In addition, the various blocks shown in the flowchart may be viewed as method steps and/or operations achieved by the operation of computer program code, and/or as a plurality of coupled logic circuit elements constructed to perform related functions. For example, an embodiment of the present invention includes a computer program product, which includes a computer program tangibly embodied on a machine-readable medium, the computer program containing program code configured to implement the method described above.

In the context of the present invention, a machine/computer readable medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine/computer readable medium may be a machine/computer readable signal medium or a machine/computer readable storage medium. A machine/computer readable medium may be non-transitory and may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of machine/computer readable storage media would include an electrical connection having one or more wiring, a portable computer disk, a hard drive, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Computer program code for implementing the method of the present invention may be written in any combination of one or more programming languages. Such computer program code may be provided to a general purpose computer, a processor of a special purpose computer, or other programmable data processing device having a control circuit system so that when the program code is executed by the processor of the computer or other programmable data processing device, the functions/operations specified in the flowchart and/or block diagram will be implemented. The program code may be executed entirely on a computer, partially on a computer, partially on a computer and partially on a remote computer, or entirely on a remote computer or server as a stand-alone software package, or distributed on one or more remote computers and/or servers.

Although this document contains many specific implementation details, these details should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be unique to particular embodiments. Specific features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in a particular combination and even initially claimed as such, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be directed to a subcombination or variations of a subcombination. The logical flow depicted in the figure does not require the particular order or sequential order shown to achieve the desired results. Additionally, other steps may be provided, or steps may be eliminated from the described flow, and other components may be added to or removed from the described system. Thus, other implementations are within the scope of the appended claims.

100: Immersive sound and audio service codec

101: Audio data

102: Spatial analysis and downmixing unit

103: Main audio channel encoding unit/main audio channel encoder

104: Spatial metadata coding unit

105: Low frequency effect channel coding unit

106: Spatial metadata decoding unit

107: Main audio channel decoding unit

108: Low frequency effect channel decoding unit

109: Spatial synthesis/upmixing/presentation unit

Claims (22)

A method for encoding a low frequency effect (LFE) channel, comprising: Using one or more processors to receive a time domain LFE channel signal; Using a low pass filter to filter the time domain LFE channel signal; Using the one or more processors to convert the filtered time domain LFE channel signal into a frequency domain representation of the LFE channel signal including a certain number of coefficients representing a spectrum of the LFE channel signal; Using the one or more processors to allocate the coefficients to a certain number of coefficients corresponding to different frequency bands of the LFE channel signal; number of sub-band groups; quantizing the coefficients in each sub-band group according to a frequency response curve of the low-pass filter using the one or more processors; encoding the quantized coefficients in the sub-band group using an entropy encoder tuned for each sub-band group using the one or more processors; and generating a bit stream including the encoded quantized coefficients using the one or more processors; and storing the bit stream on a storage device or streaming the bit stream to a downstream device using the one or more processors. The method of claim 1, wherein quantizing the coefficients in each frequency band group further comprises: generating a scaling factor based on a maximum number of available quantization points and a sum of the absolute values of the coefficients; and quantizing the coefficients using the scaling factor. As in the method of claim 2, if a quantized coefficient exceeds the maximum number of quantization points, the scaling factor is reduced and the coefficients are quantized again. The method of any one of claims 1 to 3 above, wherein the quantization points are different for each frequency band grouping. A method as in any of claims 1 to 3 above, wherein the coefficients in each sub-band group are quantized according to a fine quantization scheme or a coarse quantization scheme, wherein the fine quantization scheme is used to allocate more quantization points to the respective sub-band groups than the quantization points assigned to one or more sub-band groups according to the coarse quantization scheme. A method as in any one of claims 1 to 3 above, wherein the sign bits of the coefficients are coded separately from the coefficients. A method as in any of claims 1 to 3 above, wherein there are four sub-band groups, and a first sub-band group corresponds to a first frequency range of 0 Hz to 100 Hz, a second sub-band group corresponds to a second frequency range of 100 Hz to 200 Hz, a third sub-band group corresponds to a third frequency range of 200 Hz to 300 Hz, and a fourth sub-band group corresponds to a fourth frequency range of 300 Hz to 400 Hz. A method as in any one of claims 1 to 3 above, wherein the entropy coder is an arithmetic entropy coder. A method as in any one of claims 1 to 3 above, wherein converting the filtered time-domain LFE channel signal into a frequency-domain representation of the LFE channel signal comprising a certain number of coefficients representing a spectrum of the LFE channel signal further comprises: Determining a first step length of the LFE channel signal; Specifying a first window size of a window function based on the first step length; Applying the first window size to one or more frames of the time-domain LFE channel signal; and Applying a modified discrete cosine transform (MDCT) to the windowed frames to generate the coefficients. The method of claim 9 further comprises: determining a second step size of the LFE channel signal; specifying a second window size of the window function based on the second step size; and applying the second window size to the one or more frames of the time domain LFE channel signal. The method of claim 10, wherein: the first step size is N milliseconds (ms); N is greater than or equal to 5 ms and less than or equal to 60 ms; the first window size is greater than or equal to 10 ms; the second step size is 5 ms; and the second window size is 10 ms. The method of claim 10, wherein: the first step size is 20 milliseconds (ms); the first window size is 10 ms, 20 ms, or 40 ms; the second step size is 10 ms; and the second window size is 10 ms or 20 ms. A method as claimed in claim 10, wherein: the first step size is 10 milliseconds (ms); the first window size is 10 ms or 20 ms; the second step size is 5 ms; and the second window size is 10 ms. The method of claim 10, wherein: the first step size is 20 milliseconds (ms); the first window size is 10 ms, 20 ms, or 40 ms; the second step size is 5 ms; and the second window size is 10 ms. The method of claim 9, wherein the window function is a Kaiser-Besso derived (KBD) window function having a configurable gradual hidden length. A method as in any of claims 1 to 3 above, wherein the low pass filter is a fourth order Butterworth low pass filter having a cutoff frequency of about 130 Hz or less. The method of any one of claims 1 to 3 further comprises: Using the one or more processors to determine whether an energy level of a frame of the LFE channel signal is lower than a critical value; Based on the energy level being lower than a critical level, Generate a silence frame indicator to indicate the decoder; Insert the silence frame indicator into the metadata of the LFE channel bit stream; and Reduce an LFE channel bit rate when a silence frame is detected. A method for decoding a low frequency effect (LFE) channel bit stream, comprising: Using one or more processors to receive an LFE channel bit stream, the LFE channel bit stream comprising entropy coded coefficients representing a spectrum of a time domain LFE channel signal; Using the one or more processors to decode the quantized coefficients using an entropy decoder; Using the one or more processors to inverse quantize the quantized coefficients, wherein the coefficients have been quantized according to the entropy decoder; A frequency response curve of a low pass filter for filtering the time domain LFE channel signal in an encoder is quantized in a sub-band group corresponding to the frequency band; The inverse quantized coefficients are converted into a time domain LFE channel signal using the one or more processors; A delay of the time domain LFE channel signal is adjusted using the one or more processors; and The delay-adjusted LFE channel signal is filtered using a low pass filter. A method as in claim 18, wherein an order of the low pass filter is configured to ensure that a first total algorithm delay due to encoding and decoding the LFE channel is less than or equal to a second total algorithm delay due to encoding and decoding other channels in a multi-channel audio signal including the LFE channel signal. The method of claim 19 further includes: Determining whether the second total algorithm delay exceeds a critical value; and Based on the second total algorithm delay exceeding the critical value, Configuring the low-pass filter as an N-order low-pass filter, where N is an integer greater than or equal to 2; and Based on the second total algorithm delay not exceeding the critical value, Configuring the order of the low-pass filter to be less than N. A low latency low frequency effect (LFE) encoder/decoder comprising: one or more processors; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform any of method requests 1 to 20. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the operations of any one of method claims 1 to 20.