ES3069236T3 - Backward-compatible integration of high frequency reconstruction techniques for audio signals - Google Patents

Abstract

A method for decoding an encoded audio bitstream is described. The method includes receiving the encoded audio bitstream and decoding the audio data to generate a decoded low-band audio signal. The method also includes extracting high-frequency reconstruction metadata and filtering the decoded low-band audio signal with a bank of analysis filters to generate a filtered low-band audio signal. Furthermore, the method includes extracting a flag indicating whether a spectral translation or harmonic transposition should be performed on the audio data and regenerating a high-band portion of the audio signal using the filtered low-band audio signal and the high-frequency reconstruction metadata, according to that flag.

Description

[0001] DESCRIPTION

[0002] Backward compatible integration of high-frequency reconstruction techniques for audio signals. Cross-reference to related application

[0003] This application is a European divisional application of European patent application EP 23210525.4 (reference: D17806EP07), for which EPO Form 1001 was filed on November 17, 2023.

[0004] Technical field

[0005] The realizations refer to a method and an audio processing unit for performing high-frequency reconstruction of an audio signal.

[0006] Background of the invention

[0007] A typical audio bitstream includes both audio data (e.g., encoded audio data) indicative of one or more channels of audio content, and metadata indicative of at least one characteristic of the audio data or audio content. A well-known format for generating an encoded audio bitstream is the MPEG-4 Advanced Audio Coding (AAC) format, described in the MPEG ISO/IEC 14496-3:2009 standard. In the MPEG-4 standard, AAC denotes "Advanced Audio Coding" and HE-AAC denotes "High Efficiency Advanced Audio Coding."

[0008] The MPEG-4 AAC standard defines several audio profiles, which determine which objects and encoding tools are present in a complaint encoder or decoder. Three of these audio profiles are (1) the AAC profile, (2) the HE-AAC profile, and (3) the HE-AAC v2 profile. The AAC profile includes the AAC Low Complexity object type (or "AAC-LC"). The AAC-LC object is the counterpart of the MPEG-2 AAC Low Complexity profile, with some adjustments, and does not include either the Spectral Band Replication ("SBR") object type or the Parametric Stereo ("PS") object type. The HE-AAC profile is a superset of the AAC profile and, in addition, includes the SBR object type. The HE-AAC v2 profile is a superset of the HE-AAC profile and, in addition, includes the PS object type.

[0009] The SBR object type contains the Spectral Band Replication tool, which is an important high-frequency reconstruction (HFR) coding tool that significantly improves the compression efficiency of perceptual audio codecs. SBR reconstructs the high-frequency components of an audio signal at the receiver side (e.g., at the decoder). Therefore, the encoder only needs to encode and transmit low-frequency components, enabling much higher audio quality at lower data rates. SBR relies on replicating harmonic sequences, previously truncated to reduce the data rate, from the limited bandwidth available signal and control data obtained from the encoder. The ratio between tonal and noise-like components is maintained by adaptive inverse filtering, as well as the optional addition of noise and sinusoidal components. In the AAC MPEG-4 standard, the SBR tool performs spectral patching (also called linear translation or spectral translation), where a series of consecutive quadrature mirror filter (QMF) subbands are copied (or "patched") from a transmitted low-band portion of an audio signal to a high-band portion of the audio signal, which is generated in the decoder.

[0010] Spectral patching or linear translation may not be ideal for certain types of audio, such as musical content with relatively low crossover frequencies. Therefore, techniques to improve spectral band replication are needed.

[0011] WO2016149015A1 relates to an audio processing unit comprising a buffer, a bitstream payload deformatizer, and a decoding subsystem. The buffer stores at least one block of an encoded audio bitstream. The block includes a padding element beginning with an identifier followed by padding data. The padding data includes at least one flag identifying whether enhanced spectral band replication (eSBR) processing is to be performed on the audio content of the block. A corresponding method for decoding an encoded audio bitstream is also provided.

[0012] Brief detailed description of embodiments of the invention

[0013] The present disclosure provides a method as detailed in claim 1. A non-transient computer-readable medium according to claim 6 and an audio processing unit according to claim 7 are also provided. Advantageous features are provided in the dependent claims.

[0014] Brief description of the drawings

[0015] Figure 1 is a block diagram of an embodiment of a system that can be configured to perform an embodiment of the method of the invention.

[0016] Figure 2 is a block diagram of an encoder that is an embodiment of the audio processing unit of the invention.

[0017] Figure 3 is a block diagram of a system that includes a decoder that is an embodiment of the audio processing unit of the invention, and optionally also a post-processor coupled thereto.

[0018] Figure 4 is a block diagram of a decoder that is an embodiment of the audio processing unit of the invention.

[0019] Figure 5 is a block diagram of a decoder which is another embodiment of the audio processing unit of the invention.

[0020] Figure 6 is a block diagram of another embodiment of the audio processing unit of the invention. Figure 7 is a block diagram of an AAC MPEG-4 bitstream, including the segments into which it is divided.

[0021] Notation and nomenclature

[0022] Throughout this disclosure, including in the claims, the expression performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying gain to the signal or data) is used in a broad sense to denote performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone filtering or preliminary preprocessing before performing the operation on it).

[0023] Throughout this disclosure, including in the claims, the term "audio processing unit" or "audio processor" is used in a broad sense to mean a system, device, or apparatus configured to process audio data. Examples of audio processing units include, but are not limited to, encoders, transcoders, decoders, codecs, preprocessing systems, postprocessing systems, and bitstream processing systems (sometimes called bitstream processing tools). Virtually all consumer electronics, such as mobile phones, televisions, laptops, and tablets, contain an audio processing unit or audio processor.

[0024] Throughout this disclosure, including in the claims, the term “couples” or “coupled” is used in a broad sense to mean a direct or indirect connection. Therefore, if a first device couples to a second device, that connection may be through a direct connection, or through an indirect connection via other devices and connections. Furthermore, components that are integrated into or with other components are also coupled to each other.

[0025] Detailed description of the embodiments of the invention

[0026] The AAC MPEG-4 standard stipulates that an encoded AAC MPEG-4 bitstream includes metadata indicative of each type of high-frequency reconstruction ("HFR") processing to be applied (if any) by a decoder to decode audio content from the bitstream, and/or that controls such HFR processing, and/or is indicative of at least one feature or parameter of at least one HFR tool to be used to decode audio content from the bitstream. In this document, the term "SBR metadata" is used to denote such metadata as described or referenced in the AAC MPEG-4 standard for use with spectral band replication ("SBR"). As someone skilled in the art will appreciate, SBR is a form of HFR.

[0027] SBR is preferably used as a dual-rate system, with the underlying codec operating at half the original sampling rate, while SBR operates at the original sampling rate. The SBR encoder operates in parallel with the underlying core codec, albeit at a higher sampling rate. Although SBR is primarily a post-processing step in the decoder, important parameters are extracted in the encoder to ensure the most accurate high-frequency reconstruction in the decoder. The encoder estimates the spectral envelope of the SBR range for a time and frequency range/resolution suitable for the characteristics of the current input signal segments. The spectral envelope is estimated by complex QMF analysis and post-power calculation. The time and frequency resolutions of the spectral envelopes can be chosen with a high degree of freedom to ensure the most suitable time-frequency resolution for the given input segment. The envelope estimation must consider that a transient in the original, located primarily in the high-frequency region (e.g., a high-pitched sound), will be present to a lesser extent in the high band generated by SBR before envelope fitting, since the high band in the decoder is based on the low band where the transient is much less pronounced compared to the high band. This aspect imposes different requirements on the time-frequency resolution of the spectral envelope data, compared to ordinary spectral envelope estimation as used in other audio coding algorithms.

[0029] In addition to the spectral envelope, several additional parameters are extracted that represent the spectral characteristics of the input signal for different time and frequency regions. Since the encoder naturally has access to the original signal, as well as information about how the SBR unit in the decoder will create the high band, given the specific set of control parameters, the system can handle situations where the low band consists of a strong harmonic series and the high band to be recreated consists mainly of random signal components, as well as situations where strong tonal components are present in the original high band without counterparts in the low band on which the high band region is based. Furthermore, the SBR encoder works closely with the underlying core codec to assess which frequency range should be covered by SBR at any given time. The SBR data is efficiently encoded before transmission by leveraging entropy coding, as well as the channel dependencies of the control data, in the case of stereo signals.

[0031] Control parameter extraction algorithms typically need to be carefully tuned to the underlying codec at a given bit rate and sampling rate. This is because a lower bit rate generally implies a larger SBR range compared to a higher bit rate, and different sampling rates correspond to different time resolutions of the SBR frames.

[0033] An SBR decoder typically includes several different parts. It comprises a bitstream decoding module, a high-frequency (HFR) reconstruction module, an additional high-frequency component module, and an envelope tuning module. The system is based on either a complex-value QMF filter bank (for high-quality SBR) or a real-value QMF filter bank (for low-power SBR). The embodiments of the invention are applicable to both high-quality and low-power SBR. In the bitstream extraction module, the control data is read from the bitstream and decoded. The time-frequency grid is obtained for the current frame before the envelope data is read from the bitstream. The underlying core decoder decodes the audio signal from the current frame (albeit at the lowest sampling rate) to produce audio samples in the time domain. The resulting audio data frame is used for high-frequency reconstruction by the HFR module. The decoded low-band signal is then analyzed using a QMF filter bank. High-frequency reconstruction and envelope tuning are subsequently performed on subband samples from the QMF filter bank. The high frequencies are flexibly reconstructed from the low band based on given control parameters. Furthermore, the reconstructed high band is adaptively filtered on a subband channel basis according to the control data to ensure the appropriate spectral characteristics for the given time/frequency region.

[0035] The top level of an AAC MPEG-4 bitstream is a sequence of data blocks ("raw_data_block" elements), each of which is a data segment (referred to herein as a "block") containing audio data (typically over a period of 1024 or 960 samples) and related information and/or other data. Herein, the term "block" is used to denote a segment of an AAC MPEG-4 bitstream comprising audio data (and corresponding metadata and optionally also other related data) that determines or is indicative of one (but no more than one) "raw_data_block" element.

[0037] Each block of an AAC MPEG-4 bitstream can include a number of syntactic elements (each of which is also materialized in the bitstream as a data segment). Seven types of such syntactic elements are defined in the AAC MPEG-4 standard. Each syntactic element is identified by a different value of the "id_syn_ele" data element. Examples of syntactic elements include a "single_channel_element()", a "channel_pair_element()", and a "fill_element()". A single-channel element is a container that includes audio data from a single audio channel (a monophonic audio signal). A channel-pair element includes audio data from two audio channels (i.e., a stereo audio signal).

[0039] A padding element is an information container that includes an identifier (for example, the value of the "id_syn_ele" element mentioned above) followed by data, which is called "padding data". Padding elements have historically been used to adjust the instantaneous bit rate of bit streams to be transmitted over a constant-rate channel. By adding the appropriate amount of padding data to each block, a constant data rate can be achieved.

[0041] According to the invention, padding data includes one or more extension payloads that extend the type of data (e.g., metadata) that can be transmitted in a bitstream. A decoder that receives bitstreams with padding data containing a new data type can optionally be used by a device receiving the bitstream (e.g., a decoder) to extend the device's functionality. Therefore, as one skilled in the art can appreciate, padding elements are a special type of data structure and are different from the data structures typically used to transmit audio data (e.g., audio payloads containing channel data).

[0042] In some embodiments of the invention, the identifier used to identify a padding element may consist of a three-bit unsigned integer transmitted first in the most significant bit ("uimsbf") having a value of 0x6. In a block, several instances of the same type of syntactic element (e.g., several padding elements) may occur.

[0043] Another standard for encoding audio bitstreams is the Unified Audio and Speech Coding (USAC) MPEG standard (ISO/IEC 23003-3:2012). The USAC MPEG standard describes the encoding and decoding of audio content using spectral band replication processing (including SBR processing as described in the AAC MPEG-4 standard, and also including other enhanced forms of spectral band replication processing). This processing applies spectral band replication tools (sometimes referred to herein as "enhanced SBR tools" or "eSBR tools") from an expanded and improved version of the SBR toolset described in the AAC MPEG-4 standard. Therefore, eSBR (as defined in the USAC standard) is an enhancement of SBR (as defined in the AAC MPEG-4 standard).

[0044] In this document, the term "enhanced SBR processing" (or "eSBR processing") is used to indicate spectral band replication processing using at least one eSBR tool (for example, at least one eSBR tool that is described or mentioned in the USAC MPEG standard) that is not described or mentioned in the AAC MPEG-4 standard. Examples of such eSBR tools are harmonic transposition and additional QMF patch preprocessing or "pre-flattening".

[0045] An integer-order harmonic transponder T maps a sinusoid with frequency ω onto a sinusoid with frequency Tω, while preserving the signal duration. Typically, three orders, T = 2, 3, 4, are used sequentially to produce each part of the desired output frequency range using the smallest possible transposition order. If an output above the fourth-order transposition range is required, it can be generated by frequency shifts. Where possible, nearly critically sampled baseband time domains are created for processing to minimize computational complexity.

[0046] The harmonic transponder can be based on QMF or DFT. When using the QMF-based harmonic transponder, the bandwidth stretching of the core encoder's time-domain signal is performed entirely in the QMF domain, using a modified phase vocoder structure, performing decimation followed by time stretching for each QMF subband. Transposition using various transposition factors (e.g., T = 2, 3, 4) is carried out in a common QMF synthesis/analysis transformation stage. Since the QMF-based harmonic transponder does not feature adaptive frequency-domain signal oversampling, the corresponding flag in the bitstream (sbrOversamplingFlag[ch]) can be ignored.

[0047] When using the DFT-based harmonic transponder, the factor 3 and 4 transponders (third- and fourth-order transponders) are preferably integrated into the factor 2 transponder (second-order transponder) by means of interpolation to reduce complexity. For each frame (corresponding to coreCoderFrameLength samples), the nominal "full-size" transform size of the transponder is first determined by the signal adaptive frequency-domain oversampling flag (sbrOversamplingFlag[ch]) in the bitstream.

[0048] When sbrPatchingMode==1, indicating that linear transposition will be used to generate the high band, an additional stage is introduced to prevent discontinuities in the shape of the spectral envelope of the high-frequency signal being fed into the downstream envelope adjuster. This improves the performance of the downstream envelope adjustment stage, resulting in a high-band signal that is perceived as more stable. The operation of the additional preprocessing is beneficial for signal types where the coarse spectral envelope of the low-band signal used for high-frequency reconstruction exhibits large level variations. However, the value of the bitstream element can be determined at the encoder by applying any type of signal-dependent sorting. The additional preprocessing is activated via a one-bit bitstream element, bs_sbr_preprocessing. When bs_sbr_preprocessing is set to one, the additional processing is enabled. When bs_sbr_preprocessing is set to zero, further preprocessing is disabled. The preferred additional processing uses a pregain curve that is used by the high-frequency generator to scale the low band, X<low>, for each patch. For example, the pregain curve can be calculated as follows:

[0050]

[0052] where k<0> is the first QMF subband in the master frequency band table and ylowEnvSlope is calculated using a function that computes the coefficients of a best-fit polynomial (in a least-squares sense), such as polyfit(). For example,

[0054]

[0056] can be employed (using a third-degree polynomial) and where

[0059]

[0061] where x_lowband (k)=[0... k<0>-1], numTimeSlot is the number of SBR envelope time slots that exist within a frame, SPEED is a constant indicating the number of QMF subband samples per time slot (e.g., 2), φ<k> is a linear prediction filter coefficient (possibly obtained from the covariance method) and where

[0064]

[0066] A bitstream generated according to the USAC MPEG standard (sometimes referred to herein as a "USAC bitstream") includes encoded audio content and normally includes metadata indicative of each type of spectral band replication processing to be applied by a decoder to decode audio content from the USAC bitstream, and/or metadata controlling such spectral band replication processing and/or indicative of at least one feature or parameter of at least one SBR tool and/or eSBR tool to be employed to decode audio content from the USAC bitstream. In this document, the term "enhanced SBR metadata" (or "eSBR metadata") is used to refer to metadata indicative of each type of spectral band replication processing that a decoder will apply to decode audio content from an encoded audio bitstream (for example, a USAC bitstream) and/or that controls such spectral band replication processing, and/or indicative of at least one feature or parameter of at least one SBR tool and/or eSBR tool that will be used to decode such audio content, but which is not described or referenced in the AAC MPEG-4 standard. An example of eSBR metadata is the metadata (indicative of, or for controlling, spectral band replication processing) that is described or referenced in the USAC MPEG standard but not in the AAC MPEG-4 standard. Therefore, eSBR metadata herein refers to metadata that is not SBR metadata, and SBR metadata herein refers to metadata that is not eSBR metadata. A USAC bitstream may include both SBR metadata and eSBR metadata. More specifically, a USAC bitstream may include eSBR metadata that controls the performance of eSBR processing by a decoder, and SBR metadata that controls the performance of SBR processing by the decoder. According to typical embodiments of the present invention, eSBR metadata (e.g., eSBR-specific configuration data) is included (according to the present invention) in an AAC MPEG-4 bitstream (e.g., in the sbr_extension() container at the end of an SBR payload).

[0067] The performance of eSBR processing, during the decoding of a bitstream encoded using an eSBR toolset (comprising at least one eSBR tool), by means of a decoder, regenerates the high-frequency band of the audio signal, based on the replication of harmonic sequences that were truncated during encoding. Such eSBR processing typically adjusts the spectral envelope of the generated high-frequency band and applies inverse filtering, and adds noise and sinusoidal components in order to recreate the spectral characteristics of the original audio signal.

[0068] According to typical embodiments of the invention, eSBR metadata (for example, a small number of control bits that are eSBR metadata) is included in one or more metadata segments of an encoded audio bitstream (for example, an AAC MPEG-4 bitstream) that also includes audio data encoded in other segments (audio data segments). Typically, at least one of these metadata segments in each block of the bitstream is (or includes) a padding element (which includes an identifier indicating the start of the padding element), and the eSBR metadata is included in the padding element after the identifier.

[0069] Figure 1 is a block diagram of an example audio processing chain (an audio data processing system), wherein one or more of the system elements can be configured according to an embodiment of the present invention. The system includes the following elements, coupled together as shown: encoder 1, supply subsystem 2, decoder 3, and post-processing unit 4. In variations of the system shown, one or more of the elements are omitted, or additional audio data processing units are included.

[0070] In some implementations, the encoder 1 (which optionally includes a preprocessing unit) is configured to accept PCM (time-domain) samples comprising audio content as input, and to output an encoded audio bitstream (which is formatted to comply with the AAC MPEG-4 standard) that is indicative of the audio content. The bitstream data that is indicative of the audio content is sometimes referred to herein as "audio data" or "encoded audio data." If the encoder is configured according to a typical embodiment of the present invention, the audio bitstream output from the encoder includes eSBR metadata (and normally also other metadata) as well as audio data.

[0071] One or more encoded audio bitstreams emitted from encoder 1 may be asserted to the encoded audio delivery subsystem 2. Subsystem 2 is configured to store and/or deliver each encoded bitstream output from encoder 1. An encoded audio bitstream output from encoder 1 may be stored by subsystem 2 (for example, in the form of a DVD or Blu-ray disc), or transmitted by subsystem 2 (which may implement a transmission link or network), or it may be both stored and transmitted by subsystem 2.

[0072] Decoder 3 is configured to decode an encoded MPEG-4 AAC audio bitstream (generated by Encoder 1) received via Subsystem 2. In some implementations, Decoder 3 is configured to extract eSBR metadata from each block of the bitstream and to decode the bitstream (including performing eSBR processing using the extracted eSBR metadata) to generate decoded audio data (e.g., decoded PCM audio sample streams). In some implementations, Decoder 3 is configured to extract SBR metadata from the bitstream (but to ignore eSBR metadata included in the bitstream) and to decode the bitstream (including performing SBR processing using the extracted SBR metadata) to generate decoded audio data (e.g., decoded PCM audio sample streams). Typically, decoder 3 includes a buffer that stores (for example, non-transiently) segments of the encoded audio bitstream received from subsystem 2.

[0073] The post-processing unit 4 in Figure 1 is configured to accept a stream of decoded audio data from decoder 3 (e.g., decoded PCM audio samples) and to perform post-processing on it. The post-processing unit can also be configured to play back the post-processed audio content (or the decoded audio received from decoder 3) for playback through one or more speakers.

[0074] Figure 2 is a block diagram of an encoder (100) that is an embodiment of the audio processing unit of the invention. Any of the components or elements of the encoder 100 may be implemented as one or more processes and/or one or more circuits (e.g., ASIC, FPGA, or other integrated circuits), in hardware, software, or a combination of hardware and software. The encoder 100 includes the encoder 105, the padding/formatting stage 107, the metadata generation stage 106, and the buffer 109, connected as shown. Typically, the encoder 100 also includes other processing elements (not shown). The encoder 100 is configured to convert an input audio bitstream into an encoded AAC MPEG-4 output bitstream.

[0075] Metadata generator 106 is coupled and configured to generate (and/or pass through stage 107) metadata (including eSBR metadata and SBR metadata) to be included in stage 107 in the encoded bitstream to be emitted from encoder 100.

[0076] Encoder 105 is coupled and configured to encode (for example, by performing compression thereon) the input audio data, and to affirm the resulting encoded audio to stage 107 for inclusion in the encoded bitstream to be output from stage 107.

[0077] Stage 107 is configured to multiplex the encoded audio from encoder 105 and the metadata (including eSBR metadata and SBR metadata) from generator 106 to generate the encoded bitstream to be output from stage 107, preferably so that the encoded bitstream is in the format specified by one of the embodiments of the present invention.

[0078] Buffer 109 is configured to store (for example, non-transiently) at least one block of the encoded audio bitstream emitted from stage 107, and a sequence of the encoded audio bitstream blocks is then claimed from buffer 109 as output from encoder 100 to a supply system.

[0079] Figure 3 is a block diagram of a system that includes a decoder (200) that is an embodiment of the audio processing unit of the invention, and optionally also a post-processor (300) coupled thereto. Any of the components or elements of the decoder 200 and the post-processor 300 may be implemented as one or more processes and/or one or more circuits (for example, ASICs, FPGAs, or other integrated circuits), in hardware, software, or a combination of hardware and software. The decoder 200 comprises the buffer 201, the bitstream payload deformatizer (analyzer) 205, the audio decoding subsystem 202 (sometimes referred to as the “core” decoding stage or “core” decoding subsystem), the eSBR processing stage 203, and the control bit generation stage 204, connected as shown. Typically, the decoder 200 also includes other processing elements (not shown).

[0081] The buffer 201 stores (for example, non-transiently) at least one block of an MPEG-4 encoded AAC audio bitstream received by the decoder 200. In the operation of the decoder 200, a sequence of the bitstream blocks is asserted from buffer 201 to the deformatter 205.

[0083] In variations of the embodiment of Figure 3 (or the embodiment of Figure 4 to be described), an APU that is not a decoder (e.g., APU 500 of Figure 6) includes a buffer (e.g., a buffer identical to buffer 201) that stores (e.g., non-transiently) at least one block of an encoded audio bitstream (e.g., an MPEG-4 AAC audio bitstream) of the same type received by buffer 201 of Figure 3 or Figure 4 (i.e., an encoded audio bitstream that includes eSBR metadata).

[0085] Referring again to Figure 3, the deformatter 205 is coupled and configured to demultiplex each block of the bitstream to extract SBR metadata (including quantized envelope data) and eSBR metadata (and typically other metadata as well) from them, to assert at least the eSBR metadata and the SBR metadata to the eSBR processing stage 203, and typically also to assert other extracted metadata to the decoding subsystem 202 (and optionally also to control the bit generator 204). The formatter 205 is also coupled and configured to extract audio data from each block of the bitstream, and to assert the extracted audio data to the decoding subsystem (decoding stage) 202.

[0087] The system in Figure 3 also optionally includes a post-processor 300. The post-processor 300 includes a buffer 301 and other processing elements (not shown) including at least one processing element coupled to the buffer 301. The buffer 301 stores (for example, non-transiently) at least one block (or frame) of the decoded audio data received by the post-processor 300 from the decoder 200. The processing elements of the post-processor 300 are coupled and configured to adaptively receive and process a sequence of the blocks (or frames) of the decoded audio output from the buffer 301, using metadata emitted from the decoding subsystem 202 (and/or the deformatter 205) and/or control bits emitted from stage 204 of the decoder 200.

[0089] The audio decoding subsystem 202 of decoder 200 is configured to decode the audio data extracted by analyzer 205 (such decoding may be referred to as the "core" decoding operation) to generate decoded audio data and to assert the decoded audio data to the eSBR processing stage 203. The decoding is performed in the frequency domain and typically includes inverse quantization followed by spectral processing. Typically, a final processing stage in subsystem 202 applies a frequency-domain to time-domain transform to the decoded frequency-domain audio data, so that the output of the subsystem is decoded time-domain audio data. Stage 203 is configured to apply SBR and eSBR tools indicated by the eSBR metadata and the eSBR (extracted by analyzer 205) to the decoded audio data (i.e., to perform SBR and eSBR processing on the output of decoding subsystem 202 using the SBR and eSBR metadata) to generate the fully decoded audio data that is output (e.g., to post-processor 300) from decoder 200. Typically, decoder 200 includes a memory (accessible by subsystem 202 and stage 203) that stores the unformatted audio data and metadata output from deformatter 205, and stage 203 is configured to access the audio data and metadata (including SBR metadata and eSBR metadata) as needed during SBR processing. and eSBR. The SBR and eSBR processing in stage 203 can be considered as post-processing at the output of the core decoding subsystem 202. Optionally, the decoder 200 also includes a final upmix subsystem (which can apply parametric stereo tools ("PS") defined in the AAC MPEG-4 standard, using PS metadata extracted by the deformatter 205 and/or control bits generated in subsystem 204) that is coupled and configured to perform upmixing at the output of stage 203 to generate fully decoded and upmixed audio that is output from the decoder 200. Alternatively, the post-processor 300 is configured to perform upmixing at the output of the decoder 200 (e.g., using PS metadata extracted by the deformatter 205 and/or control bits generated in subsystem 204).

[0091] In response to metadata extracted by deformatter 205, control bit generator 204 can generate control data, and the control data can be used within decoder 200 (for example, in a final upmix subsystem) and/or asserted as output of decoder 200 (for example, to postprocessor 300 for use in post-processing). In response to metadata extracted from the input bitstream (and optionally also in response to control data), stage 204 can generate (and assert to postprocessor 300) control bits indicating that the decoded audio data output from eSBR processing stage 203 should undergo a specific type of post-processing. In some implementations, decoder 200 is configured to assert the metadata extracted by deformatter 205 from the input bitstream to postprocessor 300, and postprocessor 300 is configured to perform post-processing on the output of audio data decoded by decoder 200 using the metadata.

[0093] Figure 4 is a block diagram of an audio processing unit ("APU") (210) that is another embodiment of the audio processing unit of the invention. The APU 210 is a legacy decoder that is not configured to perform eSBR processing. Any of the components or elements of the APU 210 may be implemented as one or more processes and/or one or more circuits (e.g., ASIC, FPGA, or other integrated circuits), in hardware, software, or a combination of hardware and software. The APU 210 comprises a buffer 201, a bitstream payload deformattor (analyzer) 215, an audio decoding subsystem 202 (sometimes referred to as a "core" decoding stage or "core" decoding subsystem), and an SBR processing stage 213, connected as shown. Typically, the APU 210 also includes other processing elements (not shown). The APU 210 might represent, for example, an audio encoder, decoder, or transcoder.

[0095] Elements 201 and 202 of APU 210 are identical to the identically numbered elements of decoder 200 (in Figure 3), and their description above will not be repeated. In the operation of APU 210, a sequence of blocks from an encoded audio bitstream (an AAC MPEG-4 bitstream) received by APU 210 is asserted from buffer 201 to deformatter 215.

[0097] The formatter 215 is coupled and configured to demultiplex each block of the bitstream to extract SBR metadata (including quantized envelope data) and typically other metadata therefrom, but to ignore eSBR metadata that may be included in the bitstream according to any embodiment of the present invention. The formatter 215 is configured to assert at least the SBR metadata to the SBR processing stage 213. The formatter 215 is also coupled and configured to extract audio data from each block of the bitstream and to assert the extracted audio data to the decoding subsystem (decoding stage) 202.

[0099] The audio decoding subsystem 202 of decoder 200 is configured to decode the audio data extracted by the deformatter 215 (such decoding may be referred to as the "core" decoding operation) to generate decoded audio data and to assert the decoded audio data to the processing stage of SBR 213. The decoding is performed in the frequency domain. Typically, a final processing stage in subsystem 202 applies a frequency-domain to time-domain transform to the decoded frequency-domain audio data, so that the output of the subsystem is decoded time-domain audio data. Stage 213 is configured to apply SBR tools (but not eSBR tools) indicated by the SBR metadata (extracted by deformatter 215) to the decoded audio data (i.e., to perform SBR processing on the output of decoding subsystem 202 using the SBR metadata) to generate the fully decoded audio data that is output (e.g., to postprocessor 300) from APU 210. Typically, APU 210 includes a memory (accessible by subsystem 202 and stage 213) that stores the deformatted audio data and metadata output from deformatter 215, and stage 213 is configured to access the audio data and metadata (including SBR metadata) as needed during SBR processing. The SBR processing in stage 213 can be considered as post-processing at the output of the core decoding subsystem 202. Optionally, APU 210 also includes a final upmix subsystem (which can apply parametric stereo tools ("PS") defined in the AAC MPEG-4 standard, using PS metadata extracted by deformatter 215) that is coupled and configured to perform an upmix at the output of stage 213 to generate fully decoded and upmixed audio that is output from APU 210. Alternatively, a post-processor is configured to perform an upmix at the output of APU 210 (e.g., using PS metadata extracted by deformatter 215 and/or control bits generated in APU 210).

[0101] Several implementations of the encoder 100, decoder 200, and APU 210 are configured to perform different embodiments of the method of the invention.

[0103] According to some implementations, eSBR metadata (e.g., a small number of control bits that are eSBR metadata) is included in an encoded audio bitstream (e.g., an AAC MPEG-4 bitstream) so that legacy decoders (which are not configured to parse the eSBR metadata, or to use any eSBR tool to which the eSBR metadata belongs) can ignore the eSBR metadata but still decode the bitstream as far as possible without the use of the eSBR metadata or any eSBR tool to which the eSBR metadata belongs, typically without any significant penalty to the quality of the decoded audio. However, eSBR decoders configured to parse the bitstream to identify eSBR metadata and to use at least one eSBR tool in response to the eSBR metadata will benefit from using at least one such eSBR tool. Therefore, the embodiments of the invention provide a means for efficiently transmitting enhanced spectral band replication (eSBR) control data or metadata in a backward-compatible manner.

[0104] Typically, eSBR metadata in the bitstream is indicative of (for example, is indicative of at least one feature or parameter of) one or more of the following eSBR tools (which are described in the USAC MPEG standard, and which may or may not have been applied by an encoder during bitstream generation):

[0105] • Harmonic transposition; and

[0106] • Additional QMF patching preprocessing (pre-flattening).

[0107] For example, the eSBR metadata included in the bitstream may be indicative of parameter values (described in the USAC MPEG standard and in this disclosure): sbrPatchingMode[ch], sbrOversamplingFlag[ch], sbrPitchInBins[ch], sbrPitchInBins[ch] and bs_sbr_preprocessing.

[0108] In this document, the notation X[ch], where X is some parameter, denotes that the parameter belongs to the channel ("ch") of the audio content of an encoded bitstream to be decoded. For simplicity, the expression [ch] is sometimes omitted and it is assumed that the relevant parameter belongs to an audio content channel.

[0109] In this document, the notation X[ch][env], where X is some parameter, denotes that the parameter belongs to the SBR envelope ("env") of the audio content channel ("ch") of an encoded bitstream to be decoded. For simplicity, the expressions [env] and [ch] are sometimes omitted, and it is assumed that the relevant parameter belongs to an SBR envelope of an audio content channel.

[0110] During the decoding of an encoded bitstream, the realization of harmonic transposition during an eSBR processing stage of the decoding (for each channel, "ch", of the audio content indicated by the bitstream) is controlled by the following eSBR metadata parameters: sbrPatchingMode[ch]: sbrOversamplingFlag[ch]; sbrPitchInBinsFlag[ch]; and sbrPitchinBins[ch].

[0111] The value "sbrPatchingMode[ch]" indicates the type of transponder used in eSBR: sbrPatchingMode[ch] = 1 indicates linear transposition patching as described in Section 4.6.18 of the AAC MPEG-4 standard (as used with high-quality SBR or low-power SBR); sbrPatchingMode[ch] = 0 indicates harmonic SBR patching as described in Section 7.5.3 or 7.5.4 of the USAC MPEG standard.

[0112] The value "sbrOversamplingFlag[ch]" indicates the use of adaptive frequency-domain signal oversampling in eSBR in combination with DFT-based harmonic SBR patching as described in Section 7.5.3 of the USAC MPEG standard. This flag controls the size of DFTs used in the transponder: 1 indicates adaptive frequency-domain signal oversampling enabled as described in Section 7.5.3.1 of the USAC MPEG standard; 0 indicates adaptive frequency-domain signal oversampling disabled as described in Section 7.5.3.1 of the USAC MPEG standard.

[0113] The value "sbrPitchInBinsFlag[ch]" controls the interpretation of the parameter sbrPitchInBins[ch]: 1 indicates that the value in sbrPitchInBins[ch] is valid and greater than zero; 0 indicates that the value of sbrPitchInBins[ch] is set to zero. The value "sbrPitchInBins[ch]" controls the addition of cross-product terms in the SBR harmonic transponder. The value sbrPitchinBins[ch] is an integer in the range [0,127] and represents the distance measured in frequency intervals for a 1536-line DFT acting on the core encoder's sampling frequency.

[0114] In the case where an AAC MPEG-4 bitstream is indicative of a pair of SBR channels whose channels are not coupled (rather than a single SBR channel), the bitstream is indicative of two instances of the syntax above (for harmonic or non-harmonic transposition), one for each channel of the sbr_channel_pair_element(). Harmonic transposition by the eSBR tool typically improves the quality of music signals decoded at relatively low crossover frequencies. Non-harmonic transposition (i.e., legacy spectral patching) typically improves speech signals. Therefore, a starting point in deciding which type of transposition is preferable for encoding specific audio content is to select the transposition method based on the harmonic transposition used for speech/music detection in music content and the spectral patching used in velocity content.

[0115] The performance of pre-flattening during eSBR processing is controlled by the value of a one-bit eSBR metadata parameter known as "bs_sbr_preprocessing," meaning that pre-flattening is performed or not depending on the value of this single bit. When using the QMF SBR patching algorithm, as described in Section 4.6.18.6.3 of the AAC MPEG-4 standard, the pre-flattening stage (when indicated by the "bs_sbr_preprocessing" parameter) may be performed in an effort to avoid discontinuities in the shape of the spectral envelope of a high-frequency signal being fed into a downstream envelope tuner (the envelope tuner performs another stage of eSBR processing). Pre-flattening typically improves the performance of the downstream envelope tuning stage, resulting in a high-band signal that is perceived as more stable.

[0116] The general bit rate requirement for including eSBR metadata indicative of the aforementioned eSBR tools (harmonic transposition and pre-flattening) in an AAC MPEG-4 bitstream is expected to be on the order of a few hundred bits per second because only the differential control data necessary to perform eSBR processing according to some embodiments of the invention is transmitted. Legacy decoders can ignore this information because it is included in a backward-compatible manner (as will be explained later). Therefore, the detrimental effect on bit rate associated with the inclusion of eSBR metadata is negligible, for a number of reasons, including the following:

[0117] • The bit rate penalty (due to the inclusion of eSBR metadata) is a very small fraction of the total bit rate because only the differential control data necessary to perform eSBR processing is transmitted (and not a simultaneous transmission of the SBR control data); and

[0118] • The adjustment of control information related to SBR normally does not depend on the details of the transposition. Examples of when control data depends on the operation of the transponder are discussed later in this application.

[0119] Therefore, the embodiments of the invention provide a means for efficiently transmitting enhanced spectral band replication (eSBR) control data or metadata in a backward-compatible manner. This efficient transmission of eSBR control data reduces the memory requirements in decoders, encoders, and transcoders employing aspects of the invention, while having no tangible adverse effect on the bit rate. Furthermore, the complexity and processing requirements associated with the realization of eSBR according to the embodiments of the invention are also reduced because the SBR data only needs to be processed once and not in simultaneous transmission, which would be the case if eSBR were treated as a completely separate object type in AAC MPEG-4 instead of being integrated into the AAC MPEG-4 codec in a backward-compatible manner.

[0120] The following describes, with reference to Figure 7, elements of a block ("raw_data_block") of an AAC MPEG-4 bitstream containing eSBR metadata according to some embodiments of the present invention. Figure 7 is a diagram of a block ("raw_data_block") of the AAC MPEG-4 bitstream, showing some of its segments.

[0121] A block of an AAC MPEG-4 bitstream may include at least one "single_channel_element()" (e.g., the single-channel element shown in Figure 7), and/or at least one "channel_pair_element()" (not specifically shown in Figure 7, although it may be present), which contain audio data for an audio program. The block may also include a number of "fill_elements" (e.g., Fill Element 1 and/or Fill Element 2 in Figure 7) which contain data (e.g., metadata) related to the program. Each "single_channel_element()" includes an identifier (e.g., "ID1" in Figure 7) that indicates the start of a single-channel element, and may contain audio data indicative of a different channel in a multichannel audio program. Each "channel_pair_element" includes an identifier (not shown in Figure 7) that indicates the start of a channel pair element and may include audio data indicative of two program channels. A fill_element (referred to herein as a fill element) of an AAC MPEG-4 bitstream includes an identifier ("ID2" in Figure 7) that indicates the start of a fill element, and fill data following the identifier. The identifier ID2 consists of a three-bit unsigned integer, transmitted most significant bit first ("uimsbf"), which has a value of 0x6. The fill data includes an extension_payload() element (sometimes referred to herein as an extension payload), the syntax of which is shown in Table 4.57 of the AAC MPEG-4 standard. There are several types of extension payloads, and they are identified by the "extension_type" parameter, which is a four-bit unsigned integer, transmitted most significant bit first ("uimsbf").

[0122] Padding data (for example, an extension payload thereof) includes a header or identifier (for example, "header1" in Figure 7) that indicates a segment of padding data that is indicative of an SBR object (i.e., the header initializes an "SBR object" type, named sbr_extension_data() in the AAC MPEG-4 standard). A Spectral Band Replication (SBR) extension payload is identified by the value '1101' or '1110' for the extension_type field in the header, with the identifier '1101' identifying an extension payload with SBR data and '1110' identifying an extension payload with SBR data that has a cyclic redundancy check (CRC) to verify the correctness of the SBR data.

[0123] When the header (for example, the extension_type field) initializes an SBR object type, the SBR metadata (sometimes referred to herein as "spectral band replication data" and denoted sbr_data() in the AAC MPEG-4 standard) follows the header, and at least one spectral band replication extension element (for example, the "SBR extension element" of padding element 1 in Figure 7) may follow the SBR metadata. Such a spectral band replication extension element (a segment of the bitstream) is called the "sbr_extension()" container in the AAC MPEG-4 standard. A spectral band replication extension element optionally includes a header (for example, the "SBR extension header" of padding element 1 in Figure 7).

[0125] The AAC MPEG-4 standard stipulates that a spectral band replication extension element may include PS (parametric stereo) data for program audio data. The AAC MPEG-4 standard stipulates that when the header of a filler element (e.g., of an extension payload of the same) initializes an SBR object type (as does "header1" in Figure 7) and a spectral band replication extension element of the filler element includes PS data, the filler element (e.g., the extension payload of the same) includes spectral band replication data and a parameter "bs_extension_id" whose value (i.e., bs_extension_id = 2) indicates that PS data is included in a spectral band replication extension element of the filler element.

[0127] According to some embodiments of the present invention, eSBR metadata (e.g., an indicator indicating whether enhanced spectral band replication (eSBR) processing should be performed on block audio content) is included in a spectral band replication extension element of a filler element. For example, such an indicator is indicated in filler element 1 of Figure 7, where the indicator occurs after the header (the "SBR extension header" of filler element 1) of the "SBR extension element" of filler element 1. Optionally, such an indicator and additional eSBR metadata are included in a spectral band replication extension element after the header of the spectral band replication extension element (e.g., in the SBR extension element of filler element 1 in Figure 7, after the SBR extension header). According to some embodiments of the present invention, a padding element that includes eSBR metadata also includes a parameter "bs_extension_id" whose value (e.g., bs_extension_id = 3) indicates that the eSBR metadata is included in the padding element and that the eSBR processing should be performed on the relevant block's audio content.

[0129] According to some embodiments of the invention, the eSBR metadata is included in a padding element (e.g., padding element 2 in Figure 7) of an AAC MPEG-4 bitstream distinct from a spectral band replication extension element (SBR extension element) of the padding element. This is because padding elements containing an extension_payload() with SBR data or SBR data with CRC do not contain any other extension payload of any other type of extension. Therefore, in embodiments where the eSBR metadata is stored with its own extension payload, a separate padding element is used to store the eSBR metadata. Such a padding element includes an identifier (e.g., "ID2" in Figure 7) that indicates the start of a padding element, and padding data following the identifier. Padding data may include an extension_payload() element (sometimes referred to herein as an extension payload) whose syntax is shown in Table 4.57 of the AAC MPEG-4 standard. Padding data (for example, an extension payload thereof) includes a header (for example, "header2" of padding element 2 in Figure 7) that is indicative of an eSBR object (i.e., the header initializes an Enhanced Spectral Band Replication (eSBR) object type), and the padding data (for example, an extension payload thereof) includes eSBR metadata after the header. For example, padding element 2 in Figure 7 includes such a header ("header2") and also includes, after the header, eSBR metadata (i.e., the "flag" in padding element 2, which is indicative of whether Enhanced Spectral Band Replication (eSBR) processing should be performed on the block's audio content). Optionally, additional eSBR metadata is also included in the padding data of padding element 2 in Figure 7, after header2. In the embodiments described in this paragraph, the header (e.g., header2 in Figure 7) has an identification value that is not one of the conventional values specified in Table 4.57 of the AAC MPEG-4 standard, and is instead indicative of an eSBR extension payload (so that the header's extension_type field indicates that the padding data includes eSBR metadata).

[0131] In a first class of embodiments, the invention is an audio processing unit (for example, a decoder), comprising:

[0133] a memory (for example, buffer 201 in Figure 3 or 4) configured to store at least one block of an encoded audio bitstream (for example, at least one block of an AAC MPEG-4 bitstream);

[0135] a bitstream payload deformatter (for example, item 205 in Figure 3 or item 215 in Figure 4) coupled to memory and configured to demultiplex at least a portion of said bitstream block; and

[0136] a decoding subsystem (for example, elements 202 and 203 in Figure 3, or elements 202 and 213 in Figure 4), coupled and configured to decode at least a portion of the audio content of said bitstream block, wherein the block includes:

[0137] a padding element, which includes an identifier indicating the start of the padding element (for example, the identifier "id_syn_ele" having a value of 0x6, from Table 4.85 of the AAC MPEG-4 standard), and padding data after the identifier, wherein the padding data includes:

[0138] at least one indicator that identifies whether enhanced spectral band replication (eSBR) processing should be performed on the block's audio content (e.g., using spectral band replication data and eSBR metadata included in the block).

[0139] The indicator is eSBR metadata, and an example of the indicator is the sbrPatchingMode indicator. Another example of the indicator is the harmonicSBR indicator. Both indicators indicate whether a basic form of spectral band replication or an enhanced form of spectral band replication should be performed on the block's audio data. The basic form of spectral replication is spectral patching, and the enhanced form of spectral band replication is harmonic transposition.

[0140] In some realizations, the fill data also includes additional eSBR metadata (i.e., eSBR metadata other than the indicator).

[0141] The memory may be a buffer (e.g., an implementation of buffer 201 in Figure 4) that stores (e.g., non-transiently) at least one block of the encoded audio bitstream. The estimated performance complexity of eSBR processing (using eSBR harmonic transposition and pre-flattening) by an eSBR decoder during the decoding of an AAC MPEG-4 bitstream that includes eSBR metadata (indicative of these eSBR tools) would be as follows (for typical decoding with the indicated parameters):

[0142] • Harmonic transposition (16 kbps, 14400/28800 Hz)

[0143] ∘ Based on DFT: 3.68 WMOPS (million weighted operations per second);

[0144] ∘ Based on QMF: 0.98 WMOPS;

[0145] • QMF patch preprocessing (pre-flattening): 0.1WMOPS.

[0146] DFT-based transposition is generally known to work better than QMF-based transposition for transients.

[0147] According to some embodiments of the present invention, a padding element (of an encoded audio bitstream) that includes eSBR metadata also includes a parameter (e.g., a "bs_extension_id" parameter) whose value (e.g., bs_extension_id = 3) indicates that eSBR metadata is included in the padding element and that eSBR processing is to be performed on the relevant block's audio content, and/or a parameter (e.g., the same "bs_extension_id" parameter) whose value (e.g., bs_extension_id = 2) indicates that an sbr_extension() container of the padding element includes PS data. For example, as shown in Table 1 below, a parameter with the value bs_extension_id = 2 can indicate that a padding element's sbr_extension() container includes PS data, and a parameter with the value bs_extension_id = 3 can indicate that a padding element's sbr_extension() container includes eSBR metadata:

[0148] Table 1

[0151]

[0154] According to some embodiments of the invention, the syntax of each spectral band replication extension element that includes eSBR metadata and/or PS data is as indicated in Table 2 below (where "sbr_extension()" indicates a container that is the spectral band replication extension element, "bs_extension_id" is as described in Table 1 above, "ps_data" indicates PS data and "esbr_data" indicates eSBR metadata):

[0155] Table 2

[0158]

[0160] In an example implementation, esbr_data() referenced in Table 2 above is indicative of values for the following metadata parameters:

[0161] 1. the one-bit metadata parameter, "bs_sbr_preprocessing"; and

[0162] 2. For each channel ("ch") of audio content in the encoded bitstream to be decoded, each of the parameters described above: "sbrPatchingMode[ch]"; "sbrOversamplingFlag[ch]"; "sbrPitchInBinsFlag[ch]"; and "sbrPitchInBins[ch]".

[0163] For example, in some implementations, esbr_data() may have the syntax indicated in Table 3, to indicate these metadata parameters:

[0164] Table 3

[0166]

[0167]

[0170] The syntax above allows for the efficient implementation of an enhanced form of spectral band replication, such as harmonic transposition, as an extension of a legacy decoder. Specifically, the eSBR data in Table 3 includes only those parameters necessary to perform the enhanced form of spectral band replication that are not already supported in the bitstream or directly derivable from parameters already supported in the bitstream. All other parameters and processing data required to perform the enhanced form of spectral band replication are extracted from pre-existing parameters at predefined locations in the bitstream.

[0172] For example, an MPEG-4 HE-AAC or HE-AAC v2 compliant decoder can be extended to include an enhanced form of spectral band replication, such as harmonic transposition. This enhanced form of spectral band replication is in addition to the base form of spectral band replication already supported by the decoder. In the context of an MPEG-4 HE-AAC or HE-AAC v2 compliant decoder, this base form of spectral band replication is the QMF spectral patching SBR tool as defined in Section 4.6.18 of the MPEG-4 AAC standard.

[0174] When the enhanced form of spectral band replication is performed, an extended HE-AAC decoder can reuse many of the bitstream parameters already included in the bitstream SBR extension payload. Specific parameters that can be reused include, for example, the various parameters that determine the master frequency band table. These parameters include bs_start_freq (the parameter that determines the start of the master frequency table), bs_stop_freq (the parameter that determines the stop of the master frequency table), bs_freq_scale (the parameter that determines the number of frequency bands per octave), and bs_alter_scale (the parameter that alters the scale of the frequency bands). Parameters that can be reused also include parameters that determine the noise band table (bs_noise_bands) and the limiter band table (bs_limiter_bands). Therefore, in various implementations, at least some of the equivalent parameters specified in the USAC standard are omitted from the bitstream, thus reducing the control overhead in the bitstream. Typically, when a parameter specified in the AAC standard has an equivalent parameter specified in the USAC standard, the equivalent parameter specified in the USAC standard has the same name as the parameter specified in the AAC standard, for example, the envelope scaling factor E<OrigMapped>. However, the equivalent parameter specified in the USAC standard usually has a different value, which is "tuned" for the enhanced SBR processing defined in the USAC standard rather than the SBR processing defined in the AAC standard.

[0176] To improve the subjective quality of audio content with a strong harmonic frequency structure and tonal characteristics, particularly at low bitrates, the activation of enhanced SBR is recommended. The values of the corresponding bitstream element (i.e., esbr_data()), which control these tools, can be determined in the encoder by applying a signal-dependent sorting mechanism. In general, the use of the harmonic patching method (sbrPatchingMode = 1) is preferable for encoding music signals at very low bitrates, where the core codec may be considerably limited in audio bandwidth. This is especially true if these signals include a pronounced harmonic structure. Conversely, the use of the regular SBR patching method is preferred for speech and mixed signals, as it provides better preservation of temporal structure in speech.

[0178] To improve the performance of the harmonic transponder, a preprocessing stage (bs_sbr_preprocessing == 1) can be activated. This stage attempts to prevent the introduction of spectral discontinuities in the signal entering the subsequent envelope adjuster. This feature is beneficial for signal types where the thick spectral envelope of the low-band signal used for high-frequency reconstruction exhibits large level variations.

[0180] In order to improve the transient response of the harmonic SBR patch, an adaptive frequency-domain signal oversampling (sbrOversamplingFlag= = 1) can be applied. Since adaptive frequency-domain signal oversampling increases the computational complexity of the transponder, but only provides benefits for frames containing transients, the use of this tool is controlled by the bitstream element, which is transmitted once per frame and per independent SBR channel.

[0182] A decoder operating in the proposed enhanced SBR mode typically needs to be able to switch between legacy and enhanced SBR patching. Therefore, a delay may be introduced that can be as long as the duration of a core audio frame, depending on the decoder configuration. Typically, the delay for both legacy and enhanced SBR patching will be similar.

[0184] In addition to the numerous parameters, other data elements can also be reused by an enhanced HE-AAC decoder when performing an improved form of spectral band replication according to the embodiments of the invention. For example, envelope data and background noise data can also be extracted from the bs_data_env (envelope scaling factors) and bs_noise_env (background noise scaling factors) data and used during the enhanced form of spectral band replication.

[0186] In essence, these implementations leverage the configuration parameters and envelope data already supported by a legacy HE-AAC or HE-AAC v2 decoder in the SBR extension payload to enable an enhanced form of spectral band replication that requires the least amount of additional transmitted data. The metadata was originally tuned for a base form of HFR (e.g., SBR's spectral translation operation), but according to implementations, it is used for an enhanced form of HFR (e.g., eSBR's harmonic transposition). As discussed above, the metadata typically represents operating parameters (e.g., envelope scaling factors, background noise scaling factors, time/frequency grid parameters, sinusoid addition information, variable crossover frequency/band, inverse filtering mode, envelope resolution, smoothing mode, frequency interpolation mode) tuned and intended for use with the base form of HFR (e.g., linear spectral translation). However, this metadata, combined with additional metadata parameters specific to the enhanced form of HFR (e.g., harmonic transposition), can be used to efficiently and effectively process audio data using the enhanced form of HFR.

[0187] Consequently, extended decoders that support an enhanced form of spectral band replication can be created very efficiently by relying on already defined bitstream elements (for example, those in the SBR extension payload) and adding only the parameters necessary to support the enhanced form of spectral band replication (in a padding element extension payload). This data reduction feature, combined with placing the newly added parameters in a reserved data field, such as an extension container, substantially lowers the barriers to creating a decoder that supports an enhanced form of spectral band replication while ensuring that the bitstream is backward compatible with the legacy decoder that does not support the enhanced form of spectral band replication. It will be appreciated that the reserved data field is a backward-compatible data field; that is, it is a data field that is already compatible with earlier decoders, such as legacy HE-AAC or HE-AAC v2 decoders. Similarly, the extension container is backward compatible, meaning that it is an extension container that is already compatible with previous decoders, such as legacy HE-AAC or HE-AAC v2 decoders.

[0188] In Table 3, the number in the right-hand column indicates the number of bits of the corresponding parameter in the left-hand column.

[0189] In some implementations, the SBR object type defined in AAC MPEG-4 is updated to contain the SBR tool and aspects of the enhanced SBR tool (eSBR) as indicated in the SBR extension element (bs_extension_id== EXTENSION_ID_ESBR). If a decoder detects this SBR extension element, the decoder uses the signaled aspects of the enhanced SBR tool.

[0190] In some embodiments, the invention is a method that includes an audio data encoding step to generate an encoded bitstream (e.g., an AAC MPEG-4 bitstream), which includes embedding eSBR metadata in at least one segment of at least one block of the encoded bitstream and audio data in at least one other segment of the block. In typical embodiments, the method includes a multiplexing step of the audio data with the eSBR metadata in each block of the encoded bitstream. In the typical decoding of the encoded bitstream in an eSBR decoder, the decoder extracts the eSBR metadata from the bitstream (which includes parsing and demultiplexing the eSBR metadata and the audio data) and uses the eSBR metadata to process the audio data to generate a decoded audio data stream.

[0191] Another aspect of the invention is an eSBR decoder configured to perform eSBR processing (e.g., using at least one of the eSBR tools known as harmonic transposition or pre-flattening) during the decoding of an encoded audio bitstream (e.g., an AAC MPEG-4 bitstream) that does not include eSBR metadata. An example of such a decoder will be described with reference to Figure 5.

[0192] The eSBR decoder (400) in Figure 5 includes a buffer 201 (which is identical to the buffer 201 in Figures 3 and 4), a bitstream payload deformatter 215 (which is identical to the deformatter 215 in Figure 4), an audio decoding subsystem 202 (sometimes called the "core" decoding stage or "core" decoding subsystem, and which is identical to the core decoding subsystem 202 in Figure 3), an eSBR control data generation subsystem 401, and an eSBR processing stage 203 (which is identical to stage 203 in Figure 3), connected as shown. Typically, the decoder 400 also includes other processing elements (not shown).

[0193] In the operation of decoder 400, a sequence of blocks from an encoded audio bitstream (an AAC MPEG-4 bitstream) received by decoder 400 is asserted from buffer 201 to deformattor 215. Formatter 215 is coupled and configured to demultiplex each block of the bitstream to extract SBR metadata (including quantized envelope data) and typically other metadata as well. Formatter 215 is configured to assert at least the SBR metadata to the eSBR processing stage 203. Formatter 215 is also coupled and configured to extract audio data from each block of the bitstream and to assert the extracted audio data to the decoding subsystem (decoding stage) 202.

[0194] The audio decoding subsystem 202 of the decoder 400 is configured to decode the audio data extracted by the deformatter 215 (such decoding may be referred to as the "core" decoding operation) to generate decoded audio data and to assert the decoded audio data in the eSBR processing stage 203. The decoding is performed in the frequency domain. Typically, a final processing stage in subsystem 202 applies a frequency-domain to time-domain transform to the decoded frequency-domain audio data, so that the output of the subsystem is decoded time-domain audio data. Stage 203 is configured to apply SBR tools (and eSBR tools) indicated by the SBR metadata (extracted by deformatter 215) and by the eSBR metadata generated in subsystem 401, to the decoded audio data (i.e., to perform SBR and eSBR processing on the output of decoding subsystem 202 using the SBR and eSBR metadata) to generate the fully decoded audio data that is output from decoder 400. Typically, decoder 400 includes a memory (accessible by subsystem 202 and stage 203) that stores the deformatted audio data and metadata output from deformatter 215 (and optionally also subsystem 401), and stage 203 is configured to access the audio data and metadata as needed during SBR and eSBR processing. SBR processing in stage 203 can be considered as post-processing at the output of the core decoding subsystem 202. Optionally, the decoder 400 also includes a final upmix subsystem (which can apply parametric stereo tools ("PS") defined in the AAC MPEG-4 standard, using PS metadata extracted by the deformatter 215) that is coupled and configured to perform upmixing at the output of stage 203 to generate fully decoded and upmixed audio that is output from the APU 210.

[0196] Parametric stereo is an encoding tool that represents a stereo signal using a linear downmix of the left and right channels of the stereo signal and sets of spatial parameters that describe the stereo image. Parametric stereo typically employs three types of spatial parameters: (1) inter-channel intensity differences (IID), which describe the intensity differences between the channels; (2) inter-channel phase differences (IPD), which describe the phase differences between the channels; and (3) inter-channel coherence (ICC), which describes the coherence (or similarity) between the channels. Coherence can be measured as the maximum cross-correlation as a function of time or phase. These three parameters generally allow for high-quality reconstruction of the stereo image. However, IPD parameters only specify the relative phase differences between the channels of the stereo input signal and do not indicate the distribution of these phase differences across the left and right channels. Therefore, a fourth type of parameter, which describes an overall phase shift or overall phase difference (OPD), can be used in addition. In the stereo reconstruction process, consecutive window segments of both the received downmix signal, s[n], and a decorrelated version of the received downmix, d[n], are processed together with the spatial parameters to generate the reconstructed left (l<k>(n)) and right (r<k>(n)) signals according to:

[0199]

[0202] where H<11>, H<12>, H<21> and H<22> are defined by the stereo parameters. The signals l<k>(n) and r<k>(n) are finally transformed back to the time domain by means of a frequency-to-time transform.

[0204] The control data generation subsystem 401 of Fig. 5 is coupled and configured to detect at least one property of the encoded audio bitstream to be decoded, and to generate eSBR control data (which may be or may include eSBR metadata of any of the types included in encoded audio bitstreams according to other embodiments of the invention) in response to at least one result of the detection step. The eSBR control data is asserted in step 203 to trigger the application of individual eSBR tools or combinations of eSBR tools upon detecting a specific property (or combination of properties) of the bitstream, and/or to control the application of such eSBR tools. For example, to control the performance of eSBR processing using harmonic transposition, some implementations of the control data generation subsystem 401 would include: a music detector (for example, a simplified version of a conventional music detector) to set the parameter sbrPatchingMode[ch] (and assert the parameter set in stage 203) in response to detecting whether the bitstream is indicative of music or not; a transient detector to set the parameter sbrOversamplingFlag[ch] (and assert the parameter set in stage 203) in response to detecting the presence or absence of transients in the audio content indicated by the bitstream; and/or a pitch detector to set the parameters sbrPitchInBinsFlag[ch] and sbrPitchInBins[ch] (and assert the parameters set in stage 203) in response to detecting the pitch of the audio content indicated by the bitstream. Other aspects of the invention are audio bitstream decoding methods performed by any embodiment of the decoder of the invention described in this and the preceding paragraph.

[0206] Aspects of the invention include a method of encoding or decoding of the type that any embodiment of the APU, system, or device of the invention is configured (e.g., programmed) to perform. Other aspects of the invention include a system or device configured (e.g., programmed) to perform any embodiment of the method of the invention, and a computer-readable medium (e.g., a disk) that stores code (e.g., non-transiently) for implementing any embodiment of the method of the invention or steps thereof. For example, the system of the invention may be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor, programmed with software or firmware and/or otherwise configured to perform any of a variety of operations on data, including an embodiment of the method of the invention or steps thereof. Said general-purpose processor may be or include a computer system that includes an input device, memory, and processing circuits programmed (and/or otherwise configured) to perform an embodiment of the method of the invention (or steps thereof) in response to data stated therein.

[0208] The embodiments of the present invention can be implemented in hardware, firmware, or software, or a combination thereof (e.g., as a programmable logic array). Unless otherwise specified, the algorithms or processes included as part of the invention are not inherently related to any particular computer or other apparatus. In particular, various general-purpose machines may be used with programs written according to the teachings of this document, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to carry out the required steps of the method. Therefore, the invention can be implemented in one or more computer programs running on one or more programmable computer systems (for example, an implementation of any of the elements in Figure 1, or the encoder 100 of Figure 2 (or an element thereof), or the decoder 200 of Figure 3 (or an element thereof), or the decoder 210 of Figure 4 (or an element thereof), or the decoder 400 of Figure 5 (or an element thereof)), each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. The program code is applied to the input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

[0210] Each of these programs can be implemented in any desired computer language (including machine, assembly, or high-level procedural, logic, or object-oriented languages) to communicate with a computer system. In any case, the language can be either compiled or interpreted.

[0212] For example, when implemented by means of computer software instruction sequences, various functions and stages of embodiments of the invention may be implemented by means of multi-threaded software instruction sequences running on suitable digital signal processing hardware, in which case the various devices, stages, and functions of the embodiments may correspond to parts of the software instructions.

[0214] Each of these computer programs is preferably stored on or downloaded to a storage medium or device (e.g., memory or solid-state medium, or magnetic or optical medium) readable by a general-purpose or special-purpose programmable computer, to configure and operate the computer when the storage device or medium is read by the computer system to perform the procedures described herein. The system of the invention may also be implemented as a computer-readable storage medium configured with (i.e., storing) a computer program, wherein the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein.

[0216] Several embodiments of the invention have been described. However, it is understood that various modifications are possible. Numerous modifications and variations of the present invention are possible in view of prior teachings. For example, in order to facilitate efficient implementations, phase shifts may be used in combination with complex QMF analysis and synthesis filter banks. The analysis filter bank is responsible for filtering the low-band time-domain signal generated by the core decoder into a plurality of subbands (e.g., QMF subbands). The synthesis filter bank is responsible for combining the regenerated highband produced by the selected HFR technique (as indicated by the received sbrPatchingMode parameter) with the decoded lowband to produce a wideband output audio signal. However, a given filter bank implementation operating in a certain sample rate mode, e.g., normal double-rate operation or down-sampled SBR mode, must not have bitstream-dependent phase shifts. The QMF banks used in SBR are a complex-exponential extension of the theory of cosine-modulated filter banks. It can be shown that the alias cancellation restrictions become obsolete when the cosine-modulated filter bank is extended with complex-exponential modulation. Therefore, for the QMF banks of SBR, both the analysis filters, h<k>(n), and the synthesis filters, f<k>(n), can be defined by:

[0219]

[0222] where p<0>(n) is a real-valued symmetric or asymmetric prototype filter (typically a low-pass prototype filter), M denotes the number of channels, and N is the order of the prototype filter. The number of channels used in the analysis filter bank may differ from the number of channels used in the synthesis filter bank. For example, the analysis filter bank may have 32 channels, and the synthesis filter bank may have 64 channels. When the synthesis filter bank is operated in downsampling mode, it may have only 32 channels. Since the subband samples in the filter bank have a complex value, an additive, possibly channel-dependent, phase-shift stage may be added to the analysis filter bank. These additional phase shifts must be compensated before the synthesis filter bank. While the phase-shift terms can, in principle, be of arbitrary values without disrupting the operation of the QMF analysis/synthesis chain, they can also be restricted to certain values for compliance verification. The SBR signal will be affected by the choice of phase factors, while the low-pass signal from the core decoder will not. The audio quality of the output signal is unaffected.

[0223] The prototype filter coefficients, p<0>(n), can be defined with a length, L, of 640, as shown in Table 4 below.

[0224] Table 4

[0227]

[0228]

[0229]

[0230]

[0231]

[0234] The prototype filter, p<0>(n), can also be derived from Table 4 by one or more mathematical operations such as rounding, subsampling, interpolation, and decimation.

[0236] Although the tuning of SBR-related control information does not typically depend on the transposition details (as discussed above), in some implementations certain elements of the control data can be simultaneously broadcast in the eSBR extension container (bs_extension_id ==EXTENSION_ID_ESBR) to improve the quality of the regenerated signal. Some of the simultaneously broadcast elements may include background noise data (e.g., background noise scaling factors and a parameter indicating the direction, either frequency or time, of delta encoding for each background noise), inverse filtering data (e.g., a parameter indicating the selected inverse filtering mode: non-inverse filtering, low inverse filtering, intermediate inverse filtering, and strong inverse filtering), and missing harmonic data (e.g., a parameter indicating whether a sinusoid should be added to a specific frequency band of the regenerated high band). All these elements are based on a synthesized emulation of the decoder's transponder performed in the encoder and, therefore, if tuned correctly for the selected transponder, it can increase the quality of the regenerated signal.

[0238] Specifically, in some implementations, the missing harmonics and inverse filter control data are transmitted in the eSBR extension container (along with the other bitstream parameters in Table 3) and tuned for the eSBR harmonic transponder. The additional bit rate required to transmit these two types of metadata for the eSBR harmonic transponder is relatively low. Therefore, sending tuned inverse filter control data and/or missing harmonics in the eSBR extension container will increase the quality of the audio produced by the transponder while only minimally affecting the bit rate. To ensure backward compatibility with legacy decoders, the tuned parameters for the SBR spectral translation operation can also be sent in the bitstream as part of the SBR control data using implicit or explicit signaling.

Claims (7)

1. CLAIMS 1. Method for performing high-frequency reconstruction of an audio signal, the method comprising: receive an encoded audio bitstream, including the encoded audio bitstream audio data representing a low-band portion of the audio signal and high-frequency reconstruction metadata; decode the audio data to generate a decoded low-band audio signal; extract from the encoded audio bitstream the high-frequency reconstruction metadata, including the high-frequency reconstruction metadata operating parameters for a high-frequency reconstruction process, including the operating parameters a patch mode parameter located in a backward-compatible extension container of the encoded audio bitstream, wherein a first patch mode parameter value indicates spectral translation and a second patch mode parameter value indicates harmonic transposition by voice coder frequency propagation by phase; filter the decoded low-band audio signal to generate a filtered low-band audio signal; regenerate a high-band portion of the audio signal using the filtered low-band audio signal and the high-frequency reconstruction metadata, wherein the regeneration includes spectral translation if the patch mode parameter is the first value and the regeneration includes harmonic transposition by means of phase-encoder frequency propagation if the patch mode parameter is the second value; and combine the filtered low-band audio signal with the regenerated high-band portion to form a wideband audio signal; characterized by The encoded audio bitstream further includes a padding element with an identifier indicating the start of the padding element and padding data following the identifier, wherein the identifier is a three-bit unsigned integer transmitted first as the most significant bit and having a value of 0x6, wherein the padding data includes the backward-compatible extension container, wherein the padding data includes an extension payload, the extension payload includes spectral band replication extension data, and the extension payload is identified by a four-bit unsigned integer transmitted first as the most significant bit and having a value of '1101' or '1110', and wherein the backward-compatible extension container further includes a flag indicating whether additional preprocessing is used to avoid discontinuities in the form of a spectral envelope of the high-band portion when the patch mode parameter is equal to the first value, wherein a first value of the flag enables additional preprocessing and a second value of the flag disables additional preprocessing. 2. Method according to claim 1, wherein the filtering is performed by means of a bank of analysis filters that includes analysis filters, h<k>(n), which are modulated versions of a prototype filter, p<0>(n), according to: where p<0>(n) is a real-valued symmetric or asymmetric prototype filter, M is a number of channels in the analysis filter bank, and N is an order of the prototype filter. 3. Method according to claim 2, wherein the prototype filter, p<0>(n), is derived from coefficients in the table: 4. Method according to claim 2, wherein the prototype filter, p<0>(n), is derived from the coefficients of the table defined in claim 3 by means of one or more mathematical operations selected from the group consisting of rounding, subsampling, interpolation or decimation. 5. Method according to any of claims 1-4, wherein a phase shift is added to the filtered low-band audio signal after filtering and compensated before combining to reduce the complexity of the method. 6. A non-transient, computer-readable medium containing instructions that, when executed by a processor, carry out the method according to any one of claims 1 to 5. 7. Audio processing unit for performing high-frequency reconstruction of an audio signal, the audio processing unit comprising: an input interface for receiving an encoded audio bitstream, including the encoded audio bitstream audio data representing a low-band portion of the audio signal and high-frequency reconstruction metadata; a core audio decoder (202) to decode the audio data to generate a decoded low-band audio signal; a deformatter (205, 215) extract from the encoded audio bitstream the high-frequency reconstruction metadata, including the high-frequency reconstruction metadata operating parameters for a high-frequency reconstruction process, including the operating parameters a patch mode parameter located in a backward-compatible extension container of the encoded audio bitstream, wherein a first patch mode parameter value indicates spectral translation and a second patch mode parameter value indicates harmonic transposition by voice coder frequency propagation by phase; an analysis filter bank (203, 213) to filter the decoded low-band audio signal to generate a filtered low-band audio signal; a high-frequency regenerator (203, 213) for reconstructing a high-band portion of the audio signal using the filtered low-band audio signal and the high-frequency reconstruction metadata, wherein the reconstruction includes a spectral translation if the patch mode parameter is the first value and the reconstruction includes harmonic transposition by means of phase-encoder frequency propagation if the patch mode parameter is the second value; and a bank of synthesis filters (203, 213) to combine the filtered low-band audio signal with the regenerated high-band portion to form a wideband audio signal; characterized by The encoded audio bitstream further includes a padding element with an identifier indicating the start of the padding element and padding data following the identifier, wherein the identifier is a three-bit unsigned integer transmitted first as the most significant bit and having a value of 0x6, wherein the padding data includes the backward-compatible extension container, wherein the padding data includes an extension payload, the extension payload includes spectral band replication extension data, and the extension payload is identified by a four-bit unsigned integer transmitted first as the most significant bit and having a value of '1101' or '1110', and because the backward-compatible extension container further includes a flag indicating whether additional preprocessing is used to avoid discontinuities in the form of a spectral envelope of the high-band portion when the patch mode parameter is equal to the first value, wherein a first value of the flag enables additional preprocessing and a second value of the flag disables additional preprocessing.