Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
  • G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
  • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
  • G10L19/005—Correction of errors induced by the transmission channel, if related to the coding algorithm
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04S—STEREOPHONIC SYSTEMS 
  • H04S3/00—Systems employing more than two channels, e.g. quadraphonic
  • H04S3/02—Systems employing more than two channels, e.g. quadraphonic of the matrix type, i.e. in which input signals are combined algebraically, e.g. after having been phase shifted with respect to each other
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
  • H04R2420/00—Details of connection covered by H04R, not provided for in its groups
  • H04R2420/03—Connection circuits to selectively connect loudspeakers or headphones to amplifiers

Definitions

• the invention relates to the concealment of defective spatialisation data for the reconstruction of muiticanai audio data.
• the multi-channel audio data is typically reconstructed from at least spatialization data and audio data over a restricted number of channels, for example, single channel data.
• the muiticanai audio data is typically for a plurality of respective audio tracks.
• Several different sound sources can be used to help give the listener the illusion of sound immersion.
• the muiticanai audio data may for example comprise stereo data on two channels, or even 5.1 data on six channels, in particular for home theater applications.
• the invention can also find an application in the field of spatialized audio conferencing, where the data corresponding to a speaker undergo a spatialization process in order to give the listener the illusion that the speaker's voice comes from a particular position from space.
• Spatialization data is used to obtain muiticanai data from data on a smaller number of channels, for example single channel data.
• These spatialization data may for example comprise differences in inter-channel level or ILD (of the English "Interchannel Level Difference"), inter-channel correlations or ICC
• Interchannel Cross Correlation Interchannel Time Difference (ITD), Interchannel Phase Difference (IPD), or Inter-channel Phase Difference (IPD). Or other.
• received audio data including at least the single channel data and the spatialization data, is defective, that is, some data is missing or erroneous.
• the detection of this defective transmission can be carried out by means of a CRC type code (of the English "Cyclic Redundancy Check").
• prediction models are known. For example, an arbitrary value, a previous value or a value determined from the previously received audio data, for example linear prediction or other methods, are chosen as predicted value.
• Brutal variations of spatialisation data over time translate for the listener into the sensation of sudden displacements of the sound sources. For example, if defective values are replaced by an arbitrary value corresponding to an absence of spatialization, the feeling of returning to a single-channel sound can be disturbing for the listener, especially in the case of binaural signals. Indeed, the binaural signals, that is to say, allowing a faithful reproduction of the 3D space at the level of the ears, often correspond to relatively fixed virtual sound sources in the space.
• the subject of the invention is a sound data processing method for reconstructing audio data. multichannel from at least data on a restricted number of channels and spatialization data, this method comprising a step of testing the validity of spatialization data of a received frame.
• this test shows that these spatialization data are valid: a / by a respective model of a plurality of prediction models, according to this model, a spatialization value is predicted, b / a prediction model is chosen, based on spatialization values thus predicted and from the spatialization data actually received, so as to be able, in the event of subsequent reception of defective spatialization data, to predict, according to this chosen model, a spatialization value, and to use this predicted spatialization value for the spatialization value; reconstruction of multichannel audio data.
• a restricted number of channels is meant a number of channels less than the number of channels of the multichannel data.
• data on a restricted number of channels may include single channel data.
• the spatialization data can come from a transmission channel.
• this data can be received over the Internet.
• the received audio data can be read on a storage medium, for example a DVD (Digital Versatile Disk), or other.
• the invention is in no way limited by the origin of the audio data received.
• the received audio data may comprise a coded signal, a demultiplexed and / or decoded signal, digital values, or the like.
• the steps a / and b / can be performed systematically following the reception of a frame considered valid. The treatments are thus distributed over time.
• steps a / and b / are performed for each valid frame, it is possible to write an identifier of the prediction model chosen in memory in order to be able, in the event of subsequent reception of defective spatialization data. , quickly find the model of prediction to apply.
• steps a / and / or b / may be subject to the fulfillment of certain conditions, which may make it possible to avoid performing unnecessary calculations.
• the spatialization data is stored in a memory, at least temporarily.
• the steps a / and b / are performed (from the data thus stored), only in the event of subsequent reception of spatialisation data considered to be defective. This avoids, in particular, making the predictions of step a / when this is not necessary.
• step a / it is possible to make the predictions of step a / systematically following the reception of a frame considered valid, while step b / is performed (from the spatialization data of the previous frame (s), stored in memory) in case of reception of a defective frame.
• each predicted spatialization value is compared with an estimated value from the spatialization data received.
• the estimated value may be one of the spatialization data, for example the estimated value may include an ILD.
• the estimated value can derive only spatialization data.
• the estimated value may include a gain from the ILDs for a given frame and frequency band, a delay, or the like. In this case, it is possible during step b to compare the predicted spatialization values with values obtained from received spatialization data.
• the previously predicted spatialisation values are compared with corresponding estimated values.
• the choice of the prediction model most in line with the content can be made more accurately.
• the spatialization data received over several frames can be used, and the predicted values and the estimated values can be compared for several frames.
• a spatialization value per frame of a sequence of received frames, and for at least one model, it is possible to predict, according to this model, a spatialization value, so that a sequence of spatialization values is predicted.
• the resemblance value can be calculated from a part of this sequence of predicted spatialization values, and from a sequence of values estimated from the data of the frame sequence.
• one will abstain from using defective spatialization data during the step of choosing the prediction model, in order to avoid falsifying this choice.
• the current spatialization data received for example in the same frame, for the choice of the prediction model.
• the data may be defective due to degradations introduced during the transmission, or degradations of a data storage medium.
• the invention is not limited to this origin of defects.
• data may be missing. among the spatialization data received.
• the defectiveness of the spatialization data can be detected according to known methods, for example by means of a CRC type code.
• the invention is in no way limited by the form of the writing in memory of the identifier of the prediction model chosen. For example, it is possible to copy in a program memory all the instructions of a program corresponding to this model, or simply to memorize a model name in a possibly volatile memory.
• step a / the prediction of the spatialization value is performed according to a prediction model, that is to say in particular that the data used for the prediction can vary according to the model. For example, for a model that assigns an arbitrary value to the spatialization value, no data is needed for the prediction. For a model that consists of taking a previous spatialization value, and / or weighting a previous spatialization value, this previous spatialization value is used during the prediction.
• step a / is performed for spatialization data corresponding to a given frequency band. Thus several predictions can be carried out in parallel, in different frequency bands.
• the choice of the most accurate prediction model can be linked to the frequency: according to the frequency band considered, it may be necessary to choose different prediction models.
• the subject of the invention is a computer program comprising instructions for implementing the method explained above, when these instructions are executed by a processor.
• the invention has the aspect of a device for concealing defective spatialization data.
• the device includes a memory unit, which may include one or more memories, for storing a plurality of instruction sets, each instruction set corresponding to a prediction model.
• This device further comprises receiving means for receiving spatialization data.
• a test module makes it possible to test the validity of the spatialization data received by the reception means.
• an estimation module makes it possible, by instruction set stored in the memory unit, to execute this set of instructions so as to predict a spatialization value.
• a selection module makes it possible to choose a prediction model, based on the spatialization values predicted by the estimation module and on the spatialization data received by the reception means.
• the concealment device further comprises a prediction module arranged for, in the event of reception of spatialization data considered as defective by the detection module, to predict, according to the model chosen by the selection module, a spatialization value.
• the subject of the invention is a device for reconstructing multichannel audio data.
• This apparatus comprises multichannel reconstruction means, for reconstructing multichannel audio data from at least data on a restricted number of channels, for example single channel data.
• This apparatus further comprises the concealment device described above.
• the prediction module is arranged, in the event of reception of spatialization data considered as defective by the detection module, to provide the predicted spatialization value to the multichannel reconstruction means for the reconstruction of the multichannel audio data.
• the multi-channel audio data reconstruction apparatus can be integrated into a processor, or else comprise a computer-type device, hi-fi system, or the like.
• the various components of the reconstruction apparatus for example the reconstruction means, the concealment device, the detection module, or the like, may be distinct or merged.
• FIG. 1 shows an example of a conversational coding device
• FIG. 2 shows an example of a decoding device comprising an exemplary reconstruction apparatus according to one embodiment of the invention
• FIG. 3 is an example of an algorithm of a method according to an embodiment of the invention. 'invention,
• FIG. 4 is a graph showing an example of possible evolution of the gain
• FIG. 5 shows a device capable of executing a computer program according to one aspect of the invention.
• the number of channels of the multichannel audio data is exactly two, but of course there may be more.
• the multichannel audio data may for example comprise 5.1 data on six channels.
• the invention can also find an application in the field of spatialized audio conferencing.
• the audio data is grouped by frames or packets, indexed n.
• Figure 1 shows an exemplary encoder, for which stereo information is transmitted in frequency bands and is applied in the frequency domain.
• the encoder integrates time-frequency transformation means 10, for example a DSP (of the "Digital Signal Processor") capable of producing a transform, for example a discrete Fourier transform or DFT (of the English “Discrete Fourier Transform"), an MDCT (Modified Discrete Cosine Transform), an MCLT (Modulated Complex Lapped Transform).
• a DSP of the "Digital Signal Processor”
• DFT discrete Fourier transform
• MDCT Modified Discrete Cosine Transform
• MCLT Modulated Complex Lapped Transform
• the single-channel signal M (k) is typically the half-sum of the left signals S ⁇ _ (k) and right SR (k).
• the residual signal E (k) may be equal to half the difference between the left signals S L (k) and the right S R (k).
• Matrices may be adaptive so that the single channel signal M (k) carries more information.
• the method implemented by the matrixing means 11 may change over time, so as to avoid the cancellation of components that would be in phase opposition between the left and right channels.
• Spatialization data estimation means 12 make it possible to estimate from the single-channel signal M (k) and from the residual signal
• E (k) spatialization data for example stereo parameters.
• stereo parameters can be known to those skilled in the art, and understand for example, inter-channel level differences (ILD), inter-channel correlations (ICC), and inter-channel delays or phase differences (IPD / ITD).
• ILD inter-channel level differences
• ICC inter-channel correlations
• IPD / ITD inter-channel delays or phase differences
• stereo parameters ILD (b) can be determined by frequency bands, indexed by the variable b. These bands can be constituted according to a frequency scale close to human perception.
• Quantization, coding and multiplexing means 13 make it possible to quantify and code the stereo parameters ILD (b) in order to allow transmission at a reduced rate.
• the single-channel signal M (k) is also quantized and coded by the means 13, in the transformed domain as shown in FIG. 1, or alternatively in the time domain.
• Standardized algorithms may be used to process this single-channel signal M (k), for example an ITU G.729.1 or G.718 type speech encoder. It may also be a generic audio encoder type MPEG-4 AAC or HE-AAC.
• the residual signal E (k) is optionally transmitted, also using a standard coding or a transmission technique specific to this signal in the frequency or time domain.
• the encoded signal S e nc obtained at the output of the quantization, coding and multiplexing means 13 is transmitted, for example by radio.
• the encoder leads to obtaining data on more than one monophonic channel, provided that the number of channels of the data obtained at the output of the encoder is less than the number of channels of the input data of the encoder.
• FIG. 2 shows an example of a decoder capable of receiving a signal S ' at c corresponding to the signal S enc transmitted.
• Decoding and demultiplexing means 29 make it possible to extract from the signal S ' at c received single-channel data M' (k), spatialization data ILD ' (b> , and possibly residual data E' (k).
• the decoder further comprises a reconstruction apparatus 26 for reconstructing multi-channel audio data S' ⁇ _ (k), S ' R (k), from the single-channel data M' (k), spatialization data ILD ' (b) , and any residual data E '(k).
• FIG. 3 shows an algorithm executable by the reconstruction apparatus 26 of FIG. 2. These two figures will therefore be commented simultaneously.
• the reconstruction apparatus 26 includes a concealment device 20 for providing replacement values in case of defective ILD ' (b) spatialization data, and multichannel reconstruction means 27 for the actual reconstruction.
• the multichannel reconstruction means 27 may, for example, perform, during a step 300, combinations of the type:
• M R (k) a signal in the frequency domain, obtained equivalently in step 301, for the right channel.
• E'L is a signal specific to the left channel, issued in a manner known to those skilled in the art from the residual data E '(k) optionally transmitted
• E'R a signal specific to the right channel, issued in a manner known to those skilled in the art residual data E '(k) optionally transmitted.
• the step of obtaining the data E ' L , E'R is not represented in FIG.
• W L and WR are the gains from spatialisation data ILD '(b, n) for the band b considered and the frame n.
• the gains W L and WR can for example be determined as follows, via values W ' L and WR, during a step 302:
• ILD '(b, n) is the spatialization data ILD' (b) received for the frame n.
• W L (b, n) a.W ' L (b, n) + (la) W L (b, n-1), where W L (b, nl) denotes the value obtained for the previous frame.
• W R (b, n) a.W'R (b, n) + (la) .W R (b, nl), where W R (b, n - l) denotes the value obtained for the previous frame.
• the concealment device 20 makes it possible to prevent possible losses of data ILD '(b, n), so that data W R and W L can despite everything to be determined.
• the concealment device 20 comprises unrepresented receiving means for receiving, during a step 305, the spatialization data ILD '(b, n), and possibly the single-channel data M' (k), and the residual data E '. (k). These receiving means may for example comprise an input port, pins input, or other.
• a test module 22 connected to these reception means makes it possible to test, during a step 306, the validity of the spatialization data ILD ' (b) .
• This test module can implement a verification of a CRC-type encoding, for example to verify that the transmission did not lead to degradation of the spatialization data.
• the test module 22 can also read certain values (not shown) extracted from the signal S enc received, these values indicating possible transmitted data layers deletions. Indeed, it can be expected that some elements of the transmission network abstain from transmitting, particularly in the event of congestion of the network, or reduction of the bandwidth of the transmission channel, such or such a set of data. Non-transmitted data sets may be sound details, for example. When the test module 22 reads a value indicating a deletion of certain data, these data are considered as missing.
• the concealment device 20 comprises a memory unit 21 storing several sets of instructions, each set of instructions corresponding to a prediction model. For example, according to a first prediction model, when ILD '(b, n) spatialization data is defective for a given frame n and a given frequency band b, one chooses
• W ⁇ ⁇ b, n) W R (b, n - ⁇ )
• the corresponding instructions then consist in copying the values W R ⁇ , n-1), W L ⁇ , n- 1 obtained for the previous frame.
• W ⁇ , n) 2W R ⁇ , n1) -W R ⁇ , n-2).
• W ⁇ ⁇ , n ⁇ .W R ⁇ , nl) -W ⁇ + R ⁇ n-2).
• W ⁇ 5) ⁇ , n) Median (W R ⁇ , n1), W R ⁇ , n-2), ).
• W L ⁇ , ni) and W R ⁇ , ni) respectively attenuated values, for example 0.9.W L ⁇ , ni) and 0.9 will be used.
• W R ⁇ , n - / It can be expected to keep in the memory unit these attenuated values, for use directly by applying one of the models described above.
• model examples lead to the prediction of WL and WR values.
• the models can be used to predict values of the ILD '(b, n), W' L and W ' R , or other variables.
• ILD' (b, n) ILD '(b, n-1).
• the corresponding instruction then consists in copying this value ILD '(b, n-1) obtained for the previous frame.
• An estimation module 23 makes it possible to execute the instructions of the different instruction sets. This module 23 is activated for example for each frame such that the corresponding spatialization data ILD '(b, n) are considered valid by the test module 22, or even only for the frames considered valid and which precede a frame considered defective.
• this module 23 When this module 23 is activated, all the stored instruction sets are executed, during repeated steps 307 in a loop traversing the instruction sets, with the conventional steps of initialization, testing and incrementation, so that to obtain a set of values Ji ⁇ '"', W ⁇ " 0 J, m indicating the model used.
• a selection module 24 makes it possible to choose one of these models by comparing the predicted spatialization values jw ⁇ " 0 , ⁇ " 0 ⁇ with estimated spatialization values W L , W R from the spatialization data actually received ILD '( b, n).
• ⁇ L 2 m E [(W L (b, n) E representing the expected value, according to for example:
• a sequence of N received frames is used to determine N values W [ m) (b, n) and compare them with N estimated values W L (b, n).
• the prediction model for which the resemblance value indicates a greater adequacy between predicted values and estimated values. For example, we determine the index m * of the model giving the best concealment: it will be the index that will minimize ⁇ i or maximize p m in another embodiment. For the sake of simplicity, it is possible to choose the index which will minimize ⁇ 1 on only one of the channels, for example the left channel.
• This value m constitutes an identifier of the prediction model chosen and is stored in the memory unit 21 during a step 309. It is clear that the steps 307 can be executed before the steps 302, 304, or else in parallel . Each step 308 here puts into play values obtained during step 304, and is therefore executed after this step 304.
• the concealment device 20 further comprises a prediction module 25, for, in the event of reception of spatialization data considered to be defective, to predict during a step 310 according to the model identified by the value m of spatialization values W i ( b, n) and
• This value is supplied to the multichannel reconstruction means 27, which are then able to reconstruct in step 300 the multichannel data S'.sub.i (k), S'R (k) despite the defects of the spatialization data.
• Frequency-time transformation means 28 make it possible to retrieve temporal audio data S'L ( ⁇ ), S'R ( ⁇ ) from the multi-channel data S' ⁇ _ (k), S'R ( k) reconstructed.
• the values of W L (1, n) correspond to a signal located on the left, while for the C portion, the values of W L (1, n) correspond to a signal located on the right.
• the values of W L (1, n) correspond to a plurality of sound sources located at various locations.
• the best prediction model chosen may vary depending on the type of gain variation.
• the model consisting of repeating the value obtained for the previous frame would lead to erroneously repeat the peaks of values of W L (1, n).
• a more judicious model would be to choose an arbitrary value corresponding to a single channel signal, or to weight the gain obtained for the previous frame so as to approach gradually a gain of 1.
• the most judicious approach may be to repeat the gain value obtained for the previous frame.
• the most judicious model can change according to the type of variations of the gain from one frame to another.
• the method of Figure 3 allows to select, without human intervention, the most suitable prediction model.
• Figure 5 shows a computer comprising a screen 502, a keyboard, and a CPU.
• This central unit has a memory 500 for storing a computer program comprising instructions corresponding to the steps of the method described above.
• This central unit further comprises a processor 501 connected to the memory 500, to execute these instructions.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Acoustics & Sound (AREA)
• Signal Processing (AREA)
• Multimedia (AREA)
• Health & Medical Sciences (AREA)
• Audiology, Speech & Language Pathology (AREA)
• Human Computer Interaction (AREA)
• Mathematical Physics (AREA)
• Computational Linguistics (AREA)
• Algebra (AREA)
• General Physics & Mathematics (AREA)
• Mathematical Analysis (AREA)
• Mathematical Optimization (AREA)
• Pure & Applied Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Compression, Expansion, Code Conversion, And Decoders (AREA)
• Stereophonic System (AREA)

Applications Claiming Priority (2)

Publications (2)

Family

ID=40276118

Family Applications (1)

Country Status (8)

Families Citing this family (11)

Family Cites Families (11)

• 2009
  • 2009-07-03 KR KR1020117004404A patent/KR101590919B1/ko active Active
  • 2009-07-03 WO PCT/FR2009/051304 patent/WO2010012927A1/fr not_active Ceased
  • 2009-07-03 AT AT09802568T patent/ATE557387T1/de active
  • 2009-07-03 ES ES09802568T patent/ES2387869T3/es active Active
  • 2009-07-03 EP EP09802568A patent/EP2319037B1/de active Active
  • 2009-07-03 CN CN200980134855.0A patent/CN102138177B/zh active Active
  • 2009-07-03 US US13/056,169 patent/US8867752B2/en active Active
  • 2009-07-03 JP JP2011520560A patent/JP5421367B2/ja active Active

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events