EP2346031B1 - Raten-Verzerrungsoptimierung für eine erweiterte Audiokodierung - Google Patents

Raten-Verzerrungsoptimierung für eine erweiterte Audiokodierung Download PDF

Info

Publication number
EP2346031B1
EP2346031B1 EP09177267.3A EP09177267A EP2346031B1 EP 2346031 B1 EP2346031 B1 EP 2346031B1 EP 09177267 A EP09177267 A EP 09177267A EP 2346031 B1 EP2346031 B1 EP 2346031B1
Authority
EP
European Patent Office
Prior art keywords
sequence
spectral coefficient
quantized spectral
scale factor
stage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP09177267.3A
Other languages
English (en)
French (fr)
Other versions
EP2346031A1 (de
Inventor
Guixing Wu
En-Hui Yang
Longji Wang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BlackBerry Ltd
Original Assignee
BlackBerry Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BlackBerry Ltd filed Critical BlackBerry Ltd
Priority to EP09177267.3A priority Critical patent/EP2346031B1/de
Priority to CA2718865A priority patent/CA2718865C/en
Publication of EP2346031A1 publication Critical patent/EP2346031A1/de
Application granted granted Critical
Publication of EP2346031B1 publication Critical patent/EP2346031B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/032—Quantisation or dequantisation of spectral components
    • G10L19/035—Scalar quantisation

Definitions

  • Example embodiments herein relate to audio signal encoding, and in particular to rate-distortion optimization for Advanced Audio Coding (AAC).
  • AAC Advanced Audio Coding
  • AAC Advanced Audio Coding
  • MP3 MPEG-1/2 Layer-3 format
  • AAC was first specified in the standard MPEG-2 Part 7, and later updated in MPEG-4 Part 3.
  • AAC has found applications in digital audio broadcasting and storage applications such as in portable digital audio devices, the Internet and wireless communications.
  • the decoding algorithms are predetermined and fixed. However, there may be opportunities to manipulate the encoding algorithm while maintaining full decoder compatibility.
  • AAC and MP3 Some differences between AAC and MP3 include the AAC standard providing for the selection of quantization step sizes (which are differentially coded), and selection of Huffman codebooks from a set of 12 Huffman codebooks. Some conventional encoding algorithms are limited to optimization of these two parameters for optimization of rate-distortion in AAC encoding. These two parameters may thereafter be used to configure an encoder.
  • the present application provides for the optimization of rate-distortion for AAC encoding based on quantized spectral coefficient sequences.
  • the present application provides for joint optimization of scale factors, Huffman codebooks and quantized spectral coefficient sequences for optimization of rate-distortion.
  • the present application provides a method having an iterative rate-distortion optimization algorithm for AAC encoding based on a method of Lagrangian multipliers.
  • the method first finds the optimal values of scale factors and quantized spectral coefficients when Huffman codebooks are fixed, and then updates the values of Huffman codebooks and quantized spectral coefficients given the optimized scale factors. The iterations may be applied until a predetermined threshold is attained.
  • the present application provides a method for optimizing performance of Advanced Audio Coding of an audio source sequence, the Advanced Audio Coding being dependent on a quantized spectral coefficient sequence, wherein the quantized spectral coefficient sequence is a quantized sequence of the audio source sequence.
  • the method includes determining values of the quantized spectral coefficient sequence which minimize a cost function of an encoding of the audio source sequence within a predetermined threshold, by using soft decision quantization, the cost function being dependent on the quantized spectral coefficient sequence, and performing Advanced Audio Coding of the audio source sequence using the determined quantized spectral coefficient sequence.
  • the present application provides a method for optimizing performance of Advanced Audio Coding of an audio source sequence, the Advanced Audio Coding being dependent on a quantized spectral coefficient sequence, on a scale factor sequence, and on Huffman codebooks, wherein the quantized spectral coefficient sequence is a quantized sequence of the audio source sequence, the scale factor sequence corresponds to quantization step sizes of the quantized spectral coefficient sequence, and the Huffman codebooks are from a set of selectable Huffman codebooks.
  • the method includes determining values of the quantized spectral coefficient sequence, the scale factor sequence, and the Huffman codebooks which minimize a cost function of an encoding of the audio source sequence within a predetermined threshold, the cost function being dependent on the quantized spectral coefficient sequence, the scale factor sequence, and the Huffman codebooks, and performing Advanced Audio Coding of the audio source sequence using the determined quantized spectral coefficient sequence, the determined scale factor sequence, and the determined Huffman codebooks.
  • the present application provides an encoder for optimizing performance of Advanced Audio Coding of an audio source sequence, the Advanced Audio Coding being dependent on a quantized spectral coefficient sequence, wherein the quantized spectral coefficient sequence is a quantized sequence of the audio source sequence.
  • the encoder includes a controller, a memory accessible by the controller, and a predetermined threshold stored in the memory.
  • the controller is configured to: access the predetermined threshold from memory, determine values of the quantized spectral coefficient sequence which minimize a cost function within the predetermined threshold, by using soft decision quantization, the cost function being dependent on the quantized spectral coefficient sequence, and store the determined quantized spectral coefficient sequence in memory for Advanced Audio Coding of the audio source sequence.
  • the present application provides an encoder for optimizing performance of Advanced Audio Coding of an audio source sequence, the Advanced Audio Coding being dependent on a quantized spectral coefficient sequence, a scale factor sequence, and Huffman codebooks, wherein the quantized spectral coefficient sequence is a quantized sequence of the audio source sequence, the scale factor sequence corresponds to quantization step sizes of the quantized spectral coefficient sequence, and the Huffman codebooks are from a set of selectable Huffman codebooks.
  • the encoder includes a controller, a memory accessible by the controller; and a predetermined threshold stored in the memory.
  • the controller is configured to: access the predetermined threshold from memory, determine values of the quantized spectral coefficient sequence, the scale factor sequence, and the Huffman codebooks which minimize a cost function of an encoding of the audio source sequence within the predetermined threshold, the cost function being dependent on the quantized spectral coefficient sequence, the scale factor sequence, and the Huffman codebooks, and store the determined quantized spectral coefficient sequence, the scale factor sequence, and the Huffman codebooks in memory for Advanced Audio Coding of the audio source sequence.
  • FIG. 1 shows an AAC process 20 to which example embodiments may be applied.
  • the AAC process 20 may for example be implemented by a suitably configured encoder, for example by a computer having a memory with suitable instructions stored thereon.
  • the AAC process generally processes digital audio and produces an encoded or compressed bit stream for storage and transmission.
  • the continuous lines denote the time or spectral domain signal flow
  • the dash lines denote the control information flow.
  • the AAC process 20 includes audio input 22 for input to a time/frequency (T/F) mapping module 24 and a psychoacoustic model module 26.
  • a quantization and entropy coding module 28 and a frame packing module 30 are also shown.
  • the AAC process 20 results in an encoded output 32 of the audio input 22, for example for sending to a decoder for subsequent decoding.
  • the audio input 22 may for example be time domain audio samples which are first preprocessed (as is known in the art; not shown) and sent into the T/F mapping module 24 which converts the audio input 22 into spectral coefficients.
  • the T/F mapping module 24 shown is for example a time-variant modified discrete cosine transform (MDCT).
  • MDCT time-variant modified discrete cosine transform
  • the transform length could be set to 1024 (long block) or 128 (short block) time samples.
  • the long block is used to address stationary audio signals. This may ensure a higher frequency resolution, but may also cause quantization errors spreading over the 1024 time samples in the process of quantization.
  • the short block is used to reduce temporal noise to spread for the signals containing transients/attacks.
  • two transition blocks long-short (start) and short-long (stop), which have the same size as a long block, may be employed.
  • the time-variant MDCT is used to generate a frame of 1024 spectral coefficients.
  • One spectral frame may contain one long block sequence (including long-short and short-long) and eight short block sequences.
  • the psychoacoustic model module 26 is generally used to generate control information for the T/F mapping module 24 and the quantization and entropy coding module 28. Based on the control information from the psychoacoustic model module 26, spectral coefficients received from the T/F mapping module 24 are sent to the quantization and entropy coding module 28, and are quantized and entropy coded, resulting in quantized spectral coefficients. These encoded bit streams are packed up along with format information, control information and other auxiliary data in AAC frames, and are sent as encoded output 32.
  • the AAC syntax leaves the selection of quantization step sizes and Huffman codebooks to the encoder implementing the AAC process 20.
  • the spectral coefficients received at the quantization and entropy coding module 28 are first quantized using the selected quantization step sizes and then further encoded using Huffman codebooks from a set of selectable Huffman codebooks.
  • the AAC syntax for example specifies twelve fixed Huffman codebooks.
  • the indices of scale factors (SFs) and Huffman codebooks are coded and transmitted as side information.
  • the SFs are differentially coded relative to the previous SF, and then Huffman coded using a fixed Huffman codebook.
  • the indices of Huffman codebooks used for the encoding of the quantized spectral coefficients are coded by run-length codes.
  • TNLS nested loop search
  • the AAC quantization and entropy coding module 28 first groups an entire frame of 1024 spectral coefficients into a number of scale factor bands.
  • global_gain is usually set to be equal to scale_factor[0] .
  • the formulaic calculation of y i nint ⁇ x ⁇ r
  • a noise shaping method needs to be applied to find the proper global quantization step size global_gain and scale factors before the actual quantization.
  • Some conventional algorithms use the TNLS algorithm to jointly control the bit rate and distortion.
  • the TNLS algorithm may require quantization step sizes so small to obtain the best perceptual quality.
  • it has to increase to the quantization step sizes to enable coding at the required bit-rate.
  • quantized spectral coefficients it is identified to use quantized spectral coefficients as another free parameter to which an AAC encoder can optimize.
  • a method is provided to jointly optimize the quantized coefficients, quantization step sizes and Huffman codebooks. The method may for example be based on the method of Lagrangian multipliers, as can be implemented by those skilled in the art.
  • one purpose is to achieve the minimum perceptual distortion for a given encoding rate.
  • the following minimization problem is to be solved: ⁇ min y , s , h ⁇ D w xr rxr , subject to R s + R h + R y ⁇ R 1
  • xr is the original spectral signal sequence
  • rxr is the reconstructed signal sequence
  • y is the quantized spectral coefficient sequence
  • h is the Huffman codebook index sequence ("Huffman codebooks")
  • R ( s ), R ( y ) and R ( h ) are the bit rates for transmitting s , y and h respectively
  • R 1 is the rate constraint
  • D w ( xr , rxr ) denotes the weighted distortion measure between xr and rxr .
  • ANMR average noise-to mask ratio
  • NMR noise-to mask ratio
  • NMR the ratio of the quantization noise to the masking threshold
  • N the mostly widely used objective measure for the evaluation of an audio signal.
  • N is the number of scale factor bands
  • w [sb] is the inverse of the masking threshold for scale factor band sb
  • d [sb] is the quantization distortion, mean squared quantization error for scale factor band sb .
  • AAC employs differential coding of scale factors and runlength coding of Huffman codebook indices, this may introduce significant inter-band dependencies in coding of the side information.
  • the absolute difference between the scale factor values of two neighboring scale factor bands should be restricted within a dynamic range of 60, and the scale factor value is differentially encoded relative to the one of the preceding band (or the global gain for the first band) by a fixed Huffman codebook.
  • the whole quantized spectrum is segmented into sections whose boundaries are aligned with those of scale factor bands, such that a single Huffman codebook is used to code each section.
  • the indices of Huffman codebooks are coded by runlength codes.
  • N denotes the total number of scale factor bands of one spectral frame
  • R s determines the number of side information bits needed to encode the scale factor s i of band i as a function of s i and s i - 1
  • R h represents the number of bits to encode Huffman codebook index h i for band i as a function of h i and the length of h i
  • run(h ⁇ ) run(h ⁇ )
  • the summation in (3.5) is over all pairs of ( h i , run(h ⁇ )) along with the Huffman codebook index sequence.
  • s -1 is equal to global_gain.
  • bit rates to transmit the scale factors, R ( s ) and Huffman codebook indices R( h ), depend on the actual scale factors and Huffman codebook indices transmitted, and the bit rate to transmit the quantized coefficients R( y ) is determined by the actual Huffman codebook.
  • Some conventional systems have limited the optimization algorithms to the two above-mentioned parameters of scale factors and Huffman codebooks.
  • some of the methods described herein also consider the optimization of the quantized spectral coefficient sequence y . This may be referred to herein as "soft-decision quantization" (rather than hard decision quantization), such that y is chosen as a parameter to minimize the rate-distortion cost (3.3).
  • Figure 2 shows an optimization process 50 in accordance with an example embodiment
  • Figure 3 shows a detail of an example Trellis process 66 to be used in the optimization process 50 of Figure 2
  • Figure 4 shows a detail of another example Trellis process 68 to be used in the optimization process 50 of Figure 2
  • the Trellis process 66 is an example Trellis-based implementation of step 56 of the optimization process 50
  • the Trellis process 68 is an example Trellis-based implementation of step 58 of the optimization process 50.
  • the optimization process 50 includes an alternating minimization procedure to optimize the scale factors s and Huffman codebooks h alternatively to minimize the Lagrangian cost. The exact order of steps may vary from those shown in Figures 2 and 3 in different applications and embodiments. It can also be appreciated that some steps may not be required in some embodiments.
  • h t is fixed or given for any t ⁇ 0.
  • This step 58 may for example be implemented by a Trellis process 68 in a similar manner as Trellis process 66. Compute J ⁇ ( y t+1 , s t+1 , h t+1 ), and denote is as J ⁇ t + 1 .
  • the final y , s and h may thereafter be provided for AAC coding of xr .
  • Steps 56 and 58 will now be explained in greater detail, which may for example be solved by applying dynamic programming for the soft decision quantization.
  • Reference is now made to Figure 3 which shows the Trellis process 66 to be used for step 56.
  • the number of states at each stage is N s (or any suitable N x , depending on the parameter used for minimization).
  • Each state at the i th stage represents an SF candidate (i.e., s) for the i th SFB.
  • ⁇ k,i where 0 ⁇ k ⁇ N s and 0 ⁇ i ⁇ N.
  • J k,i as the minimum accumulative cost from stage 0 to ⁇ k,i .
  • the state transition cost from ⁇ l , i -1 to ⁇ k , i is ⁇ ⁇ R s ( s i -s i-1 ).
  • the optimization procedure for the Trellis process 66 (step 56) is described as follows:
  • the optimal path can be extracted by tracing backward from the state with the minimum Lagrangian cost at the last stage.
  • the optimal quantized spectral coefficient sequence y and SFs s for all SFBs that minimize the Lagrangian cost are determined.
  • FIG. 4 shows the Trellis process 68 to be used for step 58.
  • the Trellis process 68 follows a similar procedure to Trellis process 66. It is used to attain a solution for step 58 for the optimal quantized spectral coefficient sequence y and Huffman codebooks h for a fixed or given s .
  • Each state at the i th stage represents a Huffman codebook candidate (i.e., h) for the i th SFB. Denote these states as ⁇ k,i where 0 ⁇ k ⁇ N h and 0 ⁇ i ⁇ N .
  • Trellis process 66 Denote J k , i as the minimum accumulative cost from stage 0 to ⁇ k,i .
  • Trellis process 66 there are transition paths between any of two states in neighboring stages.
  • transition paths between any of two states which have identical state numbers There two states are not restricted within neighboring stages.
  • the optimization procedure for the Trellis process 68 (step 58) is described as follows:
  • the optimal path can be extracted by tracing backward from the state with the minimum Lagrangian cost at the last stage.
  • the optimal quantized spectral coefficient sequence y and Huffman codebooks for all SFBs that minimize the Lagrangian cost are determined.
  • the extra weighted distortion introduced by y s is 0.00402, based on the de-quantizer/decoder defined in the standard. This brings a rate reduction of 1 bit. For ⁇ >0.00402, this directly leads to a better rate-distortion tradeoff defined by (3.3).
  • Figures 5 and 6 show graphs 80, 90 of comparative performance characteristics of an example embodiment using the above-described optimization process using a specified configuration for encoding of audio files Waltz.wav and Violin.wav, respectively.
  • Figures 7 and 8 show graphs 100, 110 of performance characteristics, having alternate configurations, for encoding of audio file Waltz.wav.
  • ⁇ final R c 1 ⁇ 10 c 2 ⁇ PE - c 3 ⁇ R
  • PE Perceptual Entropy of an encoded frame
  • R the encoding rate
  • the simulations may for example be implemented by a FAAC encoder, which is an open source simulation tool for implementing AAC.
  • Faac_src_26102001 is used, which adopts ISO perceptual model.
  • the optimization process 50 also uses the original FAAC encoder output as the initial point.
  • the optimization process 50 is implemented as explained above.
  • the search range for y j is set to [yh j -2, yh j +2], where yh j is the jth quantized coefficient from hard decision quantization (e.g., using (2.1)).
  • the number of possible SFs for each Trellis stage is set to 60. For each case, the perceptual model, joint stereo encoding mode and window switching decision are kept intact, as can be implemented by those skilled in the art.
  • Figure 5 depicts a graph 80 showing the rate-distortion performance for the audio test file Waltz.wav.
  • the test file may for example be configured at 48khz, 2 channel, 16 bits/sample, 30 seconds.
  • FAAC 82 represents the results obtained by using the FAAC encoder
  • Trellis 84 represents the conventional Trellis-based optimized AAC encoder using hard-decision quantization
  • Trellis+SQ 86 represents the results from the optimization process 50 ( Figure 2 ) using soft-decision quantization, as described above.
  • the vertical axes denote the average noise to mask ratio (i.e., distortion) over all audio frames, while the horizontal axes denote the rate in kbps.
  • the optimization process 50 achieves a performance gain over the FAAC reference encoder.
  • the proposed optimization algorithm achieves 1.858 dB and 0.67 dB ANMR gains over the FAAC reference encoder and Trellis-based optimized AAC encoder respectively, which is equivalent to 22.6% and 8% compression rate gains respectively.
  • Figure 6 shows a graph 90 of another simulation, performed in a similar manner as the simulation shown in Figure 5 , for the audio coding of test file Violin.wav.
  • the test file may for example be configured at 48khz, 2 channel, 16 bits/sample, 30 seconds. Improvements in rate-distortion are shown in the graph 90. Similar results may be achieved for other test music files.
  • the computational complexity has been reduced from O((N s ⁇ N h ) 2 N) to O((N s 2 + N h 2 ) ⁇ 3N ) .
  • the number of possible SFs at each stage is set to 60. In some example embodiments, further expansion of the search range for y j and SFs would not significantly improve the compression performance.
  • Table 1 lists the computation time in seconds on a Pentium PC, 2.16GHZ, 1G bytes of RAM to encode waltz.wav at different bit rates for three different encoders.
  • Figures 7 and 8 represent simulations configured to further improve the computation speed in two aspects.
  • the number of possible SFs could be reduced to 50. In some example embodiments, this does not contribute significantly to any performance loss.
  • Table 2 lists the computation time in seconds to encode Waltz.wav for the two optimized encoders after applying the above changes.
  • Fast Trellis refers to implementing the above two changes on conventional hard-decision quantization.
  • Figure 7 accordingly shows the performance for Fast Trellis versus Trellis (conventional hard-decision quantization).
  • Fast Trellis+SQ refers to implementing the above two changes on the optimization process 50 using soft-decision quantization.
  • Figure 8 accordingly shows the performance for Fast Trellis+SQ versus Trellis+SQ.
  • the computational complexity may be reduced significantly after reducing the number of possible scale factors.
  • the performance loss is relatively small.
  • the fast Trellis-based optimized AAC encoder may realize near real time throughput.
  • the two above-mentioned configurations for improving computational time may be implemented by other methods, and are not limited to the Fast Trellis and Fast Trellis+SQ simulations described herein.
  • FIG. 9 shows a method 200 for optimizing performance of AAC of a source sequence in accordance with an example embodiment.
  • the method 200 defines and initializes a quantized spectral coefficient sequence ( y ) as a quantized sequence of the source sequence to be determined, Huffman codebooks ( h ) from a set of selectable Huffman codebooks, and a scale factor sequence ( s ) corresponding to quantization step sizes of the quantized spectral coefficient sequence.
  • a cost function (J) based on distortion and bit rate transmission of an encoding of the source sequence, the cost function being dependent on the quantized spectral coefficient sequence ( y ), the scale factor sequence ( s ), and the Huffman codebooks ( h ).
  • a tolerance ⁇ is also specified as a tolerance for the cost function (J).
  • the method 200 determines the quantized spectral coefficient sequence ( y ) which minimizes the cost function (J) within the predetermined tolerance ⁇ . As shown, the method may also minimize the scale factor sequence ( s ) and the Huffman codebooks ( h ). At step 208, the method outputs y , s and h as parameters for performing of Advanced Audio Coding of the source sequence.
  • the encoder 300 may for example be implemented on a suitable configured computer device.
  • the encoder 300 includes a controller such as a microprocessor 302 that controls the overall operation of the encoder 300.
  • the microprocessor 302 may also interact with other subsystems (not shown) such as a communications subsystem, display, and one or more auxiliary input/output (I/O) subsystems or devices.
  • the encoder 300 includes a memory 304 accessible by the microprocessor 302.
  • Operating system software 306 and various software applications 308 used by the microprocessor 302 are, in some example embodiments, stored in memory 304 or similar storage element.
  • AAC software application 310 such as the FAAC encoder software described above, may be installed as one of the various software applications 308.
  • the microprocessor 302, in addition to its operating system functions, in example embodiments enables execution of software applications 308 on the device.
  • the encoder 300 may be used for optimizing performance of AAC of a source sequence. Specifically, the encoder 300 may enable the microprocessor 302 to determine a quantized spectral coefficient sequence as a quantized sequence of the source sequence.
  • the memory 304 may contain a cost function of an encoding of the source sequence, wherein the cost function is dependent on the quantized spectral coefficient sequence.
  • the memory 304 may also contain a predetermined threshold of the cost function stored in the memory 304. Instructions residing in memory 304 enable the microprocessor 302 to access the cost function and predetermined threshold from memory 304, determine the quantized spectral coefficient sequence which minimizes the cost function within the predetermined threshold, and store the determined quantized spectral coefficient sequence in memory 304 for AAC of the source sequence.
  • AAC software application 310 may be used to perform AAC using the determined quantized spectral coefficient sequence.
  • the encoder 300 may be configured for optimizing of quantized spectral coefficient sequences, in a manner similar to the example methods described above.
  • the encoder 300 may further be configured for jointly optimizing performance of scale factors, Huffman codebooks and quantized spectral coefficient sequences, in a manner similar to the example methods described above.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Claims (14)

  1. Verfahren zum Optimieren einer Leistungsfähigkeit von Advanced Audio Coding einer Audioquellsequenz (22), wobei das Advanced Audio Coding von einer quantisierten Spektralkoeffizientensequenz, von einer Skalierfaktorsequenz und von Huffman-Codebüchern abhängt, wobei die quantisierte Spektralkoeffizientensequenz eine quantisierte Sequenz der Audioquellsequenz ist, wobei die Skalierfaktorsequenz den Quantisierungsschrittgrößen der quantisierten Spektralkoeffizientensequenz entspricht und die Huffman-Codebücher aus einer Menge von auswählbaren Huffman-Codebüchern stammen, wobei das Verfahren Folgendes aufweist:
    Bestimmen (56, 58) von Werten der quantisierten Spektralkoeffizientensequenz, der Skalierfaktorsequenz und der Huffman-Codebücher, die eine Kostenfunktion eines Codierens der Audioquellsequenz minimieren, innerhalb eines vorbestimmten Schwellenwerts (60), wobei die Kostenfunktion von der quantisierten Spektralkoeffizientensequenz, der Skalierfaktorsequenz und den Huffman-Codebüchern abhängt, und
    Durchführen von Advanced Audio Coding der Audioquellsequenz (22) unter Verwendung der bestimmten quantisierten Spektralkoeffizientensequenz, der bestimmten Skalierfaktorsequenz und der bestimmten Huffman-Codebücher,
    wobei das Bestimmen ein Initialisieren (54) fester Werte eines aus der quantisierten Spektralkoeffizientensequenz, der Skalierfaktorsequenz und den Huffman-Codebüchern und iteratives Durchführen von Folgendem aufweist:
    Bestimmen von Werten der anderen beiden aus der quantisierten Spektralkoeffizientensequenz, der Skalierfaktorsequenz und den Huffman-Codebüchen, welche die Kostenfunktion minimieren, für die festen Werte des einen aus der quantisierten Spektralkoeffizientensequenz, der Skalierfaktorsequenz und den Huffman-Codebüchern,
    Bestimmen von Werten der verbleibenden beiden aus der quantisierten Spektralkoeffizientensequenz, der Skalierfaktorsequenz und den Huffman-Codebüchen, welche die Kostenfunktion minimieren, für einen der bestimmten Werte der anderen beiden, und Festsetzen der bestimmten Werte der verbleibenden beiden aus der quantisierten Spektralkoeffizientensequenz, der Skalierfaktorsequenz und den Huffman-Codebüchern und
    Bestimmen, ob die Kostenfunktion innerhalb eines vorbestimmten Schwellenwerts liegt, und wennja, Beenden des iterativen Durchführens.
  2. Verfahren nach Anspruch 1, wobei die Kostenfunktion von einer Verzerrung von und einer Übertragungs-Bit-Rate von einem Codieren der Audioquellsequenz abhängt.
  3. Verfahren nach Anspruch 1, wobei das Bestimmen ein Initialisieren fester Werte der Huffman-Codebücher aufweist und das iterative Durchführen Folgendes aufweist:
    Bestimmen von Werten der quantisierten Spektralkoeffizientensequenz und der Skalierfaktorsequenz, welche die Kostenfunktion minimieren, für die festen Werte der Huffman-Codebücher,
    Bestimmen von Werten der quantisierten Spektralkoeffizientensequenz und den Huffman-Codebüchern, welche die Kostenfunktion minimieren, für die bestimmten Werte der Skalierfaktorsequenz und Festsetzen der bestimmten Werte der quantisierten Spektralkoeffizientensequenz und der Huffman-Codebücherund
    Bestimmen, ob die Kostenfunktion innerhalb des vorbestimmten Schwellenwerts liegt, und wennja, Beenden des iterativen Durchführens.
  4. Verfahren nach Anspruch 3, wobei mindestens eines aus dem Bestimmen ein Implementieren eines Trellis-basierten Vorgangs (66, 68) zum Minimieren aufweist.
  5. Verfahren nach Anspruch 3, wobei das Minimieren der Kostenfunktion in Bezug auf die quantisierte Spektralkoeffizientensequenz und die Skalierfaktorsequenz ein Implementieren eines Trellis-basierten Vorgangs (66) aufweist, der Folgendes aufweist:
    Bereitstellen einer Trellis-Struktur, die N Stufen hat, wobei jede Stufe Ns Zustände hat, wobei die Zustände einem Bereich von Skalierfaktoren entsprechen,
    Verknüpfen jedes Zustands an jeder Stufe der Trellisstruktur mit jeweilig minimalen inkrementellen Kosten der quantisierten Spektralkoeffizientensequenz,
    Initialisieren einer Trellissuche von allen k Zuständen an einer Anfangsstufe i=0;
    Herausfinden minimaler sich anhäufender Kosten, den k-ten Zustand an der i-ten Stufe anzunehmen, wobei o < i ≤ N-1, von der Anfangsstufe für jeden k-ten Zustand an der i-ten Stufe durch Prüfen von Zuständen an der (i-1)-ten Stufe, die zu dem k-ten Zustand an der i-ten Stufe führen, und
    Bestimmen eines optimalen Pfads durch Zurückverfolgen von dem Zustand mit den minimalen sich anhäufenden Kosten an der letzten Stufe i = N - 1.
  6. Verfahren nach Anspruch 3, wobei das Minimieren der Kostenfunktion in Bezug auf die quantisierte Spektralkoeffizientensequenz und die Huffman-Codebücher ein Implementieren eines Trellis-basierten Vorgangs (68) aufweist, der Folgendes aufweist:
    Bereitstellen einer Trellis-Struktur, die N Stufen hat, wobei jede Stufe Nh Zustände hat, wobei die Zustände einem Bereich von Huffman-Codebüchern entsprechen,
    Verknüpfen mit jedem Zustand an jeder Stufe der Trellisstruktur mit jeweilig minimalen inkrementellen Kosten der quantisierten Spektralkoeffizientensequenz,
    Initialisieren einer Trellissuche von allen k Zuständen an einer Anfangsstufe i = 0,
    Herausfinden minimaler sich anhäufender Kosten, den k-ten Zustand an der i-ten Stufe anzunehmen, wobei o < i ≤ N-1, von der Anfangsstufe für jeden k-ten Zustand an der i-ten Stufe durch Prüfen von Zuständen an der (i-1)-ten Stufe, die zu dem k-ten Zustand an der i-ten Stufe führen, und durch Prüfen des k-ten Zustands an der n-ten Stufe, wobei o ≤ n < i-1, die zu dem k-ten Zustand an der i-ten Stufe führen, und
    Bestimmen eines optimalen Pfads durch Zurückverfolgen von dem Zustand mit den minimalen sich anhäufenden Kosten an der letzten Stufe i = N - 1.
  7. Verfahren nach Anspruch 1, ferner mit Initialisieren der quantisierten Spektralkoeffizientensequenz durch Berechnen einer Funktion, die von der Skalierfaktorsequenz und der Audioquellsequenz abhängt, was zu einer initialisierten quantisierten Spektralkoeffizientensequenz führt.
  8. Verfahren nach Anspruch 7, ferner mit Begrenzen des Bestimmens der quantisierten Spektralkoeffizientensequenz auf innerhalb eines Suchbereichs, der von der initialisierten quantisierten Spektralkoeffizientensequenz abhängt.
  9. Verfahren nach Anspruch 8, wobei der Suchbereich [yh - α, yh + α] ist, wobei yh die initialisierte quantisierte Spektralkoeffizientensequenz ist und α eine feste Ganzzahl ist.
  10. Verfahren nach Anspruch 1, wobei die Skalierfaktorsequenz differenziell codiert ist, wobei das Verfahren ferner ein Begrenzen des Bestimmens der Skalierfaktorsequenz auf innerhalb der Suchbereichs aufweist.
  11. Verfahren nach Anspruch 10, ferner mit Begrenzen des Bereichs von Skalierfaktorsequenzen auf innerhalb des Suchbereichs in einer ersten Iteration des Bestimmens und ferner Begrenzen des Suchbereichs von Skalierfaktorsequenzen in darauffolgenden Iterationen des Bestimmens.
  12. Verfahren nach Anspruch 1, wobei die Kostenfunktion von einem festen Wert eines Lagrange 'schen Faktors λ als Parameter abhängt, der den Kompromiss der Übertragungsbitrate zur Verzerrung darstellt.
  13. Verfahren nach Anspruch 12, wobei λ wie folgt berechnet wird: λ final R = c 1 × 10 c 2 ⁢ PE - c 3 ⁢ R
    Figure imgb0022
    wobei PE eine Wahrnehmungsentropie eines codierten Rahmens ist, R eine Codierrate ist und c1, c2 und c3 unter Verwendung eines Kleinste-Quadrate-Kriteriums bestimmt werden.
  14. Codierer (300) zum Optimieren einer Leistungsfähigkeit von Advanced Audio Coding einer Audioquellsequenz (22), wobei der Codierer (300) dazu konfiguriert ist, das Verfahren nach einem der Ansprüche 1 bis 13 durchzuführen.
EP09177267.3A 2009-11-26 2009-11-26 Raten-Verzerrungsoptimierung für eine erweiterte Audiokodierung Active EP2346031B1 (de)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP09177267.3A EP2346031B1 (de) 2009-11-26 2009-11-26 Raten-Verzerrungsoptimierung für eine erweiterte Audiokodierung
CA2718865A CA2718865C (en) 2009-11-26 2010-10-26 Rate-distortion optimization for advanced audio coding

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP09177267.3A EP2346031B1 (de) 2009-11-26 2009-11-26 Raten-Verzerrungsoptimierung für eine erweiterte Audiokodierung

Publications (2)

Publication Number Publication Date
EP2346031A1 EP2346031A1 (de) 2011-07-20
EP2346031B1 true EP2346031B1 (de) 2015-09-30

Family

ID=42083941

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09177267.3A Active EP2346031B1 (de) 2009-11-26 2009-11-26 Raten-Verzerrungsoptimierung für eine erweiterte Audiokodierung

Country Status (2)

Country Link
EP (1) EP2346031B1 (de)
CA (1) CA2718865C (de)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109889837B (zh) * 2019-03-12 2022-05-27 电子科技大学 最优拉格朗日乘子基准定点标定方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7272566B2 (en) * 2003-01-02 2007-09-18 Dolby Laboratories Licensing Corporation Reducing scale factor transmission cost for MPEG-2 advanced audio coding (AAC) using a lattice based post processing technique

Also Published As

Publication number Publication date
CA2718865C (en) 2015-10-20
CA2718865A1 (en) 2011-05-26
EP2346031A1 (de) 2011-07-20

Similar Documents

Publication Publication Date Title
US8380524B2 (en) Rate-distortion optimization for advanced audio coding
EP1905000B1 (de) Selektive verwendung mehrerer entropiemodelle bei der adaptiven kodierung und dekodierung
US7693709B2 (en) Reordering coefficients for waveform coding or decoding
EP1904999B1 (de) Frequenzsegmentierung zur gewinnung von bändern zur effizienten kodierung digitaler medien
EP1400954B1 (de) Entropische Kodierung mittels Anpassung des Kodierungsmodus zwischen Niveau- und Lauflängenniveau-Modus
EP4407614B1 (de) Verfahren zur decodierung eines audiosignals und computerprogramm
CN101601087B (zh) 用于编码和解码的设备
US7684981B2 (en) Prediction of spectral coefficients in waveform coding and decoding
US7383180B2 (en) Constant bitrate media encoding techniques
US8457957B2 (en) Optimization of MP3 audio encoding by scale factors and global quantization step size
US20070094035A1 (en) Audio coding
US6593872B2 (en) Signal processing apparatus and method, signal coding apparatus and method, and signal decoding apparatus and method
EP2439736A1 (de) Abwärtsmischvorrichtung, encoder und verfahren dafür
CN109300480B (zh) 立体声信号的编解码方法和编解码装置
CN101432803A (zh) 编码方法和编码装置
KR20060121973A (ko) 양자화 스텝 사이즈 결정 장치 및 방법
WO2005027096A1 (en) Method and apparatus for encoding audio
EP2346031B1 (de) Raten-Verzerrungsoptimierung für eine erweiterte Audiokodierung
US6678648B1 (en) Fast loop iteration and bitstream formatting method for MPEG audio encoding
EP2192577B1 (de) Optimierung von MP3-Kodierung mit vollständiger Dekodiererkompatibilität
TWI907896B (zh) 用於音訊編碼/解碼的錯誤恢復工具
KR101783967B1 (ko) 멀티 채널 신호의 부호화/복호화 장치 및 방법
KR101635099B1 (ko) 멀티 채널 신호의 부호화/복호화 장치 및 방법
HK1127434B (en) Device and method of emitting an estimated value

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20091126

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA RS

17Q First examination report despatched

Effective date: 20110901

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: BLACKBERRY LIMITED

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: BLACKBERRY LIMITED

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602009033897

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G10L0019020000

Ipc: G10L0019035000

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 19/035 20130101AFI20150319BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20150505

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 752837

Country of ref document: AT

Kind code of ref document: T

Effective date: 20151015

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602009033897

Country of ref document: DE

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20151230

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20151231

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20150930

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 752837

Country of ref document: AT

Kind code of ref document: T

Effective date: 20150930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160130

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160201

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: LU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20151126

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602009033897

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20151130

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20151130

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

26N No opposition filed

Effective date: 20160701

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20151126

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20091126

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 9

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150930

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602009033897

Country of ref document: DE

Ref country code: DE

Ref legal event code: R081

Ref document number: 602009033897

Country of ref document: DE

Owner name: MALIKIE INNOVATIONS LTD., IE

Free format text: FORMER OWNER: BLACKBERRY LTD., WATERLOO, ONTARIO, CA

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20241128

Year of fee payment: 16

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20241126

Year of fee payment: 16

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20241126

Year of fee payment: 16