EP3635718B1 - Verarbeitung von klangdaten zur trennung von klangquellen in einem mehrkanalsignal - Google Patents

Verarbeitung von klangdaten zur trennung von klangquellen in einem mehrkanalsignal Download PDF

Info

Publication number
EP3635718B1
EP3635718B1 EP18737650.4A EP18737650A EP3635718B1 EP 3635718 B1 EP3635718 B1 EP 3635718B1 EP 18737650 A EP18737650 A EP 18737650A EP 3635718 B1 EP3635718 B1 EP 3635718B1
Authority
EP
European Patent Office
Prior art keywords
components
sources
descriptors
direct
sound
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP18737650.4A
Other languages
English (en)
French (fr)
Other versions
EP3635718A1 (de
Inventor
Mathieu BAQUÉ
Alexandre Guerin
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Orange SA
Original Assignee
Orange SA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Orange SA filed Critical Orange SA
Publication of EP3635718A1 publication Critical patent/EP3635718A1/de
Application granted granted Critical
Publication of EP3635718B1 publication Critical patent/EP3635718B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0272—Voice signal separating
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0272—Voice signal separating
    • G10L21/0308—Voice signal separating characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/78—Detection of presence or absence of voice signals
    • G10L25/84—Detection of presence or absence of voice signals for discriminating voice from noise
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R5/00—Stereophonic arrangements
    • H04R5/02—Spatial or constructional arrangements of loudspeakers
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L2021/02082—Noise filtering the noise being echo, reverberation of the speech
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
    • G10L2021/02166—Microphone arrays; Beamforming

Definitions

  • the present invention relates to the field of audio or acoustic signal processing and more particularly to the processing of real multi-channel sound content to separate the sound sources.
  • a blind separation of the sources consists, from a number M of observations resulting from sensors distributed in this space E, in counting and extracting the number N of sources.
  • each observation is obtained using a sensor which records the signal reaching a point in space where the sensor is located.
  • the recorded signal then results from the mixing and propagation in the space E of the signals s i and is therefore affected by various disturbances specific to the environment crossed, such as noise, reverberation, interference, etc.
  • x is the vector of the M recorded channels, s the vector of the N sources and A a so-called “mixing matrix” of dimension M ⁇ N containing the contributions from each source to each observation, and the abbreviation * symbolizes linear convolution.
  • US 2005/060142 A1 describes a step of source separation of a multi-channel signal followed by an identification step. This document uses independent component analysis, ACl, based on a separation matrix, and can identify a dominant signal by two separate signals.
  • US 2010/111290 A1 describes a step for separating sources of a signal that can be carried out using an analysis technique of independent components, AC1, as well as a step for estimating a type of sound after separation. The type of sound estimated is stationary or non-stationary. Obtaining a separation matrix seems implicit in view of the separation methods indicated.
  • An example of beamforming to extract three sources positioned respectively at 0°, 90° and -120° azimuth is shown in figure 1 .
  • Each of the directivities formed corresponds to the extraction of one of the sources of s.
  • the total acoustic field can be modeled as the sum of the direct field of the sources of interest (represented in 1 on the picture 2 ), first reflections (secondary sources, represented in 2 on the picture 2 ) and a diffuse field (represented in 3 on the figure 2 ).
  • the covariance matrix of the observations is then of full rank, whatever the actual number of sources active in the mixture: this means that the rank of Co can no longer be used to estimate the number of sources.
  • a couple ( a i , t i ) corresponding to the active source i is estimated as follows:
  • a representation in space of all the pairs ( a i , t i ) is performed in the form of a histogram, the “ clustering” is then performed on the histogram by maximum likelihood, depending on the position of the zone and the assumed position of the associated source, assuming a Gaussian distribution of the estimated positions of each zone around the actual position of the sources.
  • the calculation of a bivariate descriptor includes the calculation of a consistency score between two components. This descriptor calculation makes it possible to know in a relevant way whether a pair of components corresponds to two direct components (2 sources) or whether at least one of the components comes from a reverberant effect.
  • the calculation of a bivariate descriptor includes the determination of a delay between the two components of the couple. This determination of the delay and of the sign associated with this delay makes it possible to determine, for a pair of components, which component more probably corresponds to the direct signal and which component more probably corresponds to the reverberated signal.
  • the delay between two components is determined by taking into account the delay maximizing an inter-correlation function between the two components of the couple. This method of obtaining the delay offers a reliable determination of a bivariate descriptor.
  • the determination of the delay between two components of a couple is associated with a reliability indicator of the sign of the delay, a function of the coherence between the components of the couple.
  • the determination of the delay between two components of a couple is associated with a reliability indicator of the sign of the delay, a function of the ratio of the maximum of an inter-correlation function for delays of opposite sign.
  • the calculation of a univariate descriptor is a function of a matching between the mixing coefficients of a mixing matrix estimated from the source separation step and the characteristics of encoding of a plane wave type source. This descriptor calculation makes it possible, for a single component, to estimate the probability that the component is direct or reverberated.
  • the classification of the components of the set of M components is carried out by taking into account all of the M components, and by calculating the most probable combination of the classifications of the M components.
  • the calculation of the most probable combination is carried out by determining a maximum of the likelihood values expressed as the product of the conditional probabilities associated with the descriptors, for the possible combinations of classification of the M components.
  • a step of pre-selection of the possible combinations is carried out on the basis of the univariate descriptors alone before the step of calculating the most probable combination.
  • a step of pre-selection of the components is carried out on the basis of the univariate descriptors alone before the step of calculating the bivariate descriptors.
  • the multichannel signal is an Ambisonic signal.
  • the invention also applies to a computer program comprising code instructions for implementing the steps of the processing method as described previously, when these instructions are executed by a processor and to a storage medium, readable by a processor, on which is recorded a computer program comprising code instructions for the execution of the steps of the processing method as described.
  • the device, program and storage medium have the same advantages as the method described previously, which they implement.
  • FIG. 3 illustrates the main steps of a sound data processing method for a separation of N sound sources of a multi-channel sound signal picked up in a real environment in one embodiment of the invention.
  • the method implements a step E310 of blind separation of sound sources (SAS). It is assumed here, in this embodiment, that the number of observations is equal to or greater than the number of active sources.
  • the blind source separation step can be implemented, for example by using an independent component analysis (or “ICA”) algorithm, or even a component analysis algorithm main.
  • ICA independent component analysis
  • Ambisonia consists of a projection of the acoustic field on a basis of spherical harmonic functions, to obtain a spatialized representation of the sound stage.
  • the Ambisonic formalism initially limited to the representation of spherical harmonic functions of order 1, was subsequently extended to higher orders.
  • the Ambisonic formalism with a larger number of components is commonly referred to as “ Higher Order Ambisonics ” (or “HOA” hereafter).
  • a content of order m contains a total of (m+1) 2 channels (4 channels at order 1, 9 channels at order 2, 16 channels at order 3, and so on).
  • step E310 The blind separation of sources is therefore carried out in step E310 as explained previously.
  • the components obtained at the output of the source separation step can be classified according to two classes of components: a first class of so-called direct components corresponding to the direct sound sources and a second class of so-called reverberated components corresponding to the reflections of the sources.
  • step E320 a calculation of descriptors of the M components (s 1 , s 2 , ...s M ) resulting from the source separation step is implemented, descriptors which will make it possible to associate with each component extracts the class that corresponds to it: direct component or reverberant component.
  • bi-varied descriptors which involve pairs of components (s j , s i ) and uni-varied descriptors calculated for a component s i .
  • a set of first bivariate descriptors is calculated. These descriptors are representative of statistical relationships between the components of the pairs of the set of M components obtained.
  • each direct component is mainly made up of the direct field of a source, comparable to a plane wave, to which is added a residual reverberation whose energy contribution is lower than that of the direct field.
  • the sources being by nature statistically independent, there is therefore a weak correlation between the direct components extracted.
  • each reverberant component is made up of early reflections, delayed and filtered versions of the direct field(s), and late reverberation.
  • the reverberated components show a significant correlation with the direct components, and generally an identifiable group delay compared to the direct components.
  • Coherence is ideally zero when s j and si are the direct fields of independent sources but it takes on a high value when s j and si are two contributions from the same source: the direct field and a first reflection or else two reflections.
  • Such a coherence function therefore indicates a probability of having two direct components or two contributions from the same source (direct/reverberated or first reflection/subsequent reflections).
  • the coherence value d ⁇ is less than 0.3 while in the second case d ⁇ reaches 0.7 in the presence of a single active source.
  • the determination of a probability of belonging to the same class or to a different class for a pair of components can depend on the number of a priori active sources.
  • this parameter may be taken into account in a particular embodiment.
  • the probability densities of the figure 5 And 7 described below, and more generally all the probability densities of the descriptors, are learned statistically on databases including various acoustic conditions (reverberant/mass) and different sources (male/female voice, French languages/ English/).
  • the components are classified in an informed manner: each source is associated with the closest extracted component spatially, the remaining being classified as reverberated components.
  • To calculate the position of the component we use the first 4 coefficients of its mixing vector from the matrix A (i.e. order 1), the inverse of the separation matrix B.
  • the coherence estimators deteriorate, whether they are the direct/reverberated or reverberated/reverberated pairs (in the presence of a single source, the direct/direct pair does not exist) .
  • the probability densities strongly depend on the number of sources in the mixture, and the number of sensors available.
  • This descriptor is therefore relevant for detecting whether a pair of extracted components corresponds to two direct components (2 true sources) or whether at least one of the two components comes from the room effect.
  • step E320 another type of bivariate descriptor is calculated in step E320. Either this descriptor is calculated instead of the consistency type descriptor described above, or in addition to it.
  • This descriptor will make it possible to determine, for a couple (direct/reverberated) which component is more likely the direct signal and which corresponds to the reverberated signal, based on the simple assumption that the first reflections are delayed and attenuated versions of the signal direct.
  • This descriptor is based on another statistical relationship between the components, the delay between the two components of the couple.
  • the relative value of the cross-correlation peak ⁇ jl,max to the other values of the cross-correlation function r jl ( ⁇ ) also provides information on the reliability of the group delay.
  • FIG 6 illustrates the emergent nature of the autocorrelation peak between a direct component and a reverberated component.
  • the cross-correlation maximum emerges clearly from the rest of the cross-correlation, reliably indicating that one of the components lags the other. It emerges in particular with respect to the values of the autocorrelation function for signs opposite to that of ⁇ jl,max (that of positive ⁇ on the figure 6 ) which are very small, whatever the value of ⁇ .
  • This ratio which is called emergence, is an ad hoc criterion whose relevance is verified in practice: it takes values close to 1 for independent signals, i.e. 2 direct components, and higher values for correlated signals such as a direct component and a reverberant component. In the aforementioned case of the curve (1) of the figure 6 , the emergence value is 4.
  • descriptor d ⁇ which determines, for each assumed direct/reverberated pair, the probability for each component of the pair of being the direct component or the reverberated component.
  • This descriptor is a function of the sign of ⁇ max, of the average coherence between the components and of the emergence of the maximum of intercorrelation.
  • this descriptor is sensitive to noise, and in particular to the presence of several simultaneous sources, as illustrated on curve (2) of the figure 6 : in the presence of 2 sources, even if the maximum correlation always emerges, its relative value - 2.6 - is lower due to the presence of an interfering source which reduces the correlation between the extracted components.
  • the reliability of the sign of the delay will be measured as a function of the value of the emergence, which will be weighted by the a priori number of sources to be detected.
  • step E330 a probability of belonging to a first class of direct components or a second class of reverberated components is calculated for a pair of components.
  • s j identified as being ahead of si we estimate the probability that s j is direct and si reverberated by a two-dimensional law.
  • the sign of lag is a reliable indicator when both coherence and emergence have medium or high values. A weak emergence or a weak coherence will make the direct/reverberant or reverberant/direct pairs equally probable.
  • a set of second so-called univariate descriptors representative of encoding characteristics of the components of the set of M components obtained is also calculated.
  • the encoding of a source coming from a given direction is carried out with mixing coefficients depending, among other things, on the directivity of the sensors.
  • the source can be considered as a point and where the wavelengths are large compared to the size of the antenna, the source can be considered as a plane wave. This assumption is generally verified in the case of an ambisonic microphone which is small, provided that the source is far enough from the microphone (in practice, one meter is enough).
  • the j th column of the estimated mixing matrix A obtained by inverting the separation matrix B, will contain the mixing coefficients associated with it. If this component is direct, that is to say it corresponds to a single source, the mixing coefficients of the column Aj will tend towards the characteristics of the microphone encoding for a plane wave. In the case of a reverberated component, sum of several reflections and of a diffuse field, the estimated mixing coefficients will be more random and will not correspond to the encoding of a single source with a precise direction of arrival.
  • plane wave criterion 3 To 1 I 2 To 2 I 2 + To 3 I 2 + To 4 I 2
  • the criterion c op is by definition equal to 1 in the case of a plane wave. In the presence of a correctly identified direct field, the plane wave criterion will remain very close to the value 1. Conversely, in the case of a reverberated component, the multitude of contributions (early reflections and late reverberation) with levels equivalent energy will generally move the plane wave criterion away from its ideal value.
  • the probability laws (probability density) associated with this descriptor depending on the number of simultaneously active sources (1 or 2) and the ambisonic order of the analyzed content (orders 1 to 2).
  • the value of the plane wave criterion is concentrated around the value 1 for the direct components.
  • the distribution is more uniform, with however a slightly asymmetrical shape, because of the descriptor itself which is asymmetrical, with a 1/x shape.
  • the distance between the distributions of the two classes allows a fairly reliable discrimination between the plane wave type components and the more diffuse ones.
  • the descriptors calculated in step E320 and presented here are based both on the statistics of the extracted components (mean coherence and group delay) and on the estimated mixing matrix (plane wave criterion). These make it possible to determine conditional probabilities of belonging of a component to one of the two classes C d or C r .
  • step E340 determines a classification of the components of the set of M components, according to the two classes.
  • C j the corresponding class.
  • the problem finally comes down to choosing among a total of 2 M potential configurations assumed to be equiprobable.
  • the approach chosen can be exhaustive and then consists in estimating the likelihood of all the possible configurations, from the descriptors determined in step E320 and the distributions associated with them and which are calculated in step E330.
  • a pre-selection of the configurations can be carried out to reduce the number of configurations to be tested, and therefore the complexity of the implementation of the solution.
  • This pre-selection can be done for example according to the plane wave criterion alone by classifying certain components in the category C r , when the value of their c op criterion deviates too much from the theoretical value of a plane wave 1: in the case of ambisonic signals, one can see on the distributions of the figure 7 that we can, whatever the configuration (order or number of sources) and a priori without loss of robustness, classify in the category C r the components whose c op verifies one of the following inequalities: ⁇ vs op ⁇ 0.7 vs op > 1.5
  • This pre-selection makes it possible to reduce the number of configurations to be tested by pre-classifying certain components, by excluding the configurations which impose class C d on these pre-classified components.
  • Another possibility to further reduce the complexity is to exclude the pre-classified components from the calculation of the bivariate descriptors and the calculation of the likelihood, which reduces the number of bivariate criteria to be calculated and therefore even more the complexity. treatment.
  • the likelihood is expressed as the product of the conditional probabilities associated with each of the K descriptors, assuming they are independent: where d is the vector of descriptors and C a vector representing a configuration (ie the combination of the supposed classes of the M components), as defined above.
  • This equation is the one ultimately used to determine the most likely configuration in the Bayesian classifier described here for this embodiment.
  • Bayesian classifier presented here is only an example of an implementation, it could be replaced, among other things, by a support vector machine or a neural network.
  • the configuration presenting the maximum likelihood is retained, indicating the direct or reverberated class associated with each of the M components C (C 1 , ..., C i , ..., C M ).
  • the processing described here is performed in the time domain, but can also be, in a variant embodiment, applied in a transformed domain.
  • the useful bandwidth can be reduced depending on the potential imperfections of the capture system, at high frequencies (presence of spatial folding) or at low frequencies (impossibility of finding the theoretical directivities of the microphone encoding).
  • FIG 8 here represents an embodiment of a processing device (DIS) according to one embodiment of the invention.
  • Sensors Ca 1 to Ca M represented here in the form of a spherical microphone MIC make it possible to acquire, in a real medium, therefore reverberant, M mixing signals x ( x 1 , ..., x i , ... , x M ), from a multichannel signal.
  • microphones or sensors can be provided. These sensors can be integrated into the DIS device or outside the device, the resulting signals then being transmitted to the processing device which receives them via its input interface 840. In a variant, these signals can simply be obtained beforehand and imported in memory of the DIS device.
  • This memory can comprise a computer program comprising the code instructions for the implementation of the stages of the treatment method as described for example with reference to the picture 3 and in particular the steps of applying source separation processing to the captured multi-channel signal and obtaining a set of M sound components, with M ⁇ N, of calculating a set of first so-called bi-varied descriptors, representative of statistical relations between the components of the pairs of the set of M components obtained and of a set of second descriptors called univariate representative of encoding characteristics of the components of the set of M components obtained and classification of the components of the set of M components, according to two classes of components, a first class of N so-called direct components corresponding to the N direct sound sources and a second class of MN so-called reverberated components, by calculating the probability of belonging to one of the two classes, a function of the sets of first and second descriptors.
  • the device comprises a source separation processing module 810 applied to the picked up multi-channel signal to obtain a set of M sound components s (s 1 , ..., s i , .. s M ), with M ⁇ N.
  • the M components are supplied as input to a computer 820 capable of calculating a set of first so-called bi-varied descriptors, representative of statistical relationships between the components of the pairs of the set of M components obtained and a set of second so-called univariate descriptors -varied representative of encoding characteristics of the components of the set of M components obtained.
  • a classification module 830 or classifier capable of classifying components of the set of M components, according to two classes of components, a first class of N so-called direct components corresponding to the N direct sound sources and a second class of M-N so-called reverberated components.
  • the classification module includes a module 831 for calculating the probability of belonging to one of the two classes of the components of the set M, a function of the sets of first and second descriptors.
  • the classifier uses descriptors related to the correlation between the components to determine which are direct signals (ie true sources) and which are reverberation residues. It also uses descriptors related to the mixing coefficients estimated by SAS, to assess the conformity between the theoretical encoding of a single source and the estimated encoding of each component. Some of the descriptors are therefore a function of a couple of components (for the correlation), and others are functions of a single component (for the conformity of the estimated microphone encoding).
  • a likelihood calculation module 832 makes it possible to determine, in one embodiment, the most probable combination of the classifications of the M components by calculating likelihood values as a function of the probabilities calculated in module 831 and for the possible combinations.
  • the device comprises an output interface 850 to deliver the component classification information, for example to another processing device which can use this information to enhance the sound of the discriminated sources, to denoise them or to perform a mixing from several discriminating sources.
  • Another possible processing can also be to analyze or locate the sources to optimize the processing of a voice command.
  • the device DIS can be integrated into a microphone antenna to effect, for example, recordings of sound scenes or for voice command sound recording.
  • the device can also be integrated into a communication terminal capable of. process signals picked up by a plurality of sensors integrated or remote from the terminal.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
  • Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)

Claims (14)

  1. Verfahren zur Verarbeitung von Klangdaten zur Trennung von N Klangquellen eines Mehrkanalklangsignals, das in einer realen Umgebung erfasst wird, wobei das Verfahren die folgenden Schritte umfasst:
    - Anwenden (E310) einer Verarbeitung zur Trennung von Quellen auf das erfasste Mehrkanalsignal und Erhalten einer Trennungsmatrix und eines Satzes von M Klanganteilen, mit M ≥ N;
    - Berechnen (E320) eines Satzes erster sogenannter bivariater Deskriptoren, die für ein Korrelationsmaß zwischen den Anteilen der Paare des erhaltenen Satzes der M Anteile repräsentativ sind;
    - Berechnen (E320) eines Satzes zweiter sogenannter univariater Deskriptoren, die für Codierungseigenschaften der Anteile des erhaltenen Satzes der M Anteile repräsentativ sind, wobei das Berechnen von einem Abgleich zwischen den geschätzten und aus einer zu der Trennungsmatrix inversen Matrix stammenden Codierungseigenschaften und theoretischen Codierungseigenschaften einer Quelle vom Typ ebene Welle abhängt;
    - Klassifizieren (E340) der Anteile des Satzes der M Anteile in zwei Klassen von Anteilen, nämlich eine erste Klasse mit N sogenannten direkten Anteilen, die den N direkten Klangquellen entsprechen, und eine zweite Klasse mit M-N sogenannten Nachhallanteilen, durch Berechnen (E330) einer Zugehörigkeitswahrscheinlichkeit zu einer der zwei Klassen, die von den Sätzen erster und zweiter Deskriptoren abhängt.
  2. Verfahren nach Anspruch 1, wobei das Berechnen eines bivariaten Deskriptors das Berechnen eines Kohärenz-Scores zwischen zwei Anteilen umfasst.
  3. Verfahren nach einem der Ansprüche 1 bis 2, wobei das Berechnen eines bivariaten Deskriptors das Bestimmen einer Verzögerung zwischen den zwei Anteilen des Paares umfasst.
  4. Verfahren nach Anspruch 3, wobei die Verzögerung zwischen zwei Anteilen durch das Berücksichtigen der Verzögerung, die eine Kreuzkorrelationsfunktion zwischen den zwei Anteilen des Paares maximiert, bestimmt wird.
  5. Verfahren nach einem der Ansprüche 3 oder 4, wobei das Bestimmen der Verzögerung zwischen zwei Anteilen eines Paares mit einem Indikator für die Zuverlässigkeit des Vorzeichens der Verzögerung assoziiert ist, der von der Kohärenz zwischen den Anteilen des Paares abhängt.
  6. Verfahren nach einem der Ansprüche 3 oder 5, wobei das Bestimmen der Verzögerung zwischen zwei Anteilen eines Paares mit einem Indikator für die Zuverlässigkeit des Vorzeichens der Verzögerung assoziiert ist, der von dem Verhältnis des Maximums einer Kreuzkorrelationsfunktion für Verzögerungen mit umgekehrtem Vorzeichen abhängt.
  7. Verfahren nach einem der Ansprüche 1 bis 6, wobei das Klassifizieren der Anteile des Satzes der M Anteile durch das Berücksichtigen des Satzes der M Anteile und durch das Berechnen der wahrscheinlichsten Kombination der Klassifizierungen der M Anteile erfolgt.
  8. Verfahren nach Anspruch 7, wobei das Berechnen der wahrscheinlichsten Kombination durch das Bestimmen eines Maximums der Plausibilitätswerte, die als das Produkt aus den mit den Deskriptoren assoziierten bedingten Wahrscheinlichkeiten ausgedrückt werden, für die möglichen Klassifizierungskombinationen der M Anteile erfolgt.
  9. Verfahren nach Anspruch 7, wobei vor dem Schritt des Berechnens der wahrscheinlichsten Kombination ein Schritt des Vorauswählens der möglichen Kombinationen nur auf Basis der univariaten Deskriptoren erfolgt.
  10. Verfahren nach einem der vorhergehenden Ansprüche, wobei vor dem Schritt des Berechnens der bivariaten Deskriptoren ein Schritt des Vorauswählens der Anteile nur auf Basis der univariaten Deskriptoren erfolgt.
  11. Verfahren nach einem der vorhergehenden Ansprüche, wobei das Mehrkanalsignal ein Ambisonics-Signal ist.
  12. Vorrichtung zur Verarbeitung von Klangdaten, die eingesetzt wird, um eine Verarbeitung zur Trennung von N Klangquellen eines Mehrkanalklangsignals, das von einer Vielzahl von Sensoren in einer realen Umgebung erfasst wird, durchzuführen, wobei die Vorrichtung Folgendes umfasst:
    - eine Eingangsschnittstelle, um die durch eine Vielzahl von Sensoren erfassten Signale des Mehrkanalklangsignals zu empfangen;
    - eine Verarbeitungsschaltung, die einen Prozessor umfasst und dazu fähig ist, Folgendes zu steuern:
    o ein Modul für eine Verarbeitung zur Trennung von Quellen, die auf das erfasste Mehrkanalsignal angewendet wird, um eine Trennungsmatrix und einen Satz von M Klanganteilen zu erhalten, mit M ≥ N;
    o eine Recheneinheit, die dazu fähig ist, einen Satz erster sogenannter bivariater Deskriptoren, die für ein Korrelationsmaß zwischen den Anteilen der Paare des erhaltenen Satzes der M Anteile repräsentativ sind, und einen Satz zweiter sogenannter univariater Deskriptoren, die für Mikrofoncodierungseigenschaften der Anteile des erhaltenen Satzes der M Anteile repräsentativ sind, zu berechnen, wobei das Berechnen von einem Abgleich zwischen den geschätzten und aus einer zu der Trennungsmatrix inversen Matrix stammenden Codierungseigenschaften und theoretischen Codierungseigenschaften einer Quelle vom Typ ebene Welle abhängt;
    o ein Modul zum Klassifizieren der Anteile des Satzes der M Anteile in zwei Klassen von Anteilen, nämlich eine erste Klasse mit N sogenannten direkten Anteilen, die den N direkten Klangquellen entsprechen, und eine zweite Klasse mit M-N sogenannten Nachhallanteilen, durch Berechnen einer Zugehörigkeitswahrscheinlichkeit zu einer der zwei Klassen, die von den Sätzen erster und zweiter Deskriptoren abhängt;
    - eine Ausgangsschnittstelle, um die Klassifizierungsinformation der Anteile zu übermitteln.
  13. Computerprogramm, das Codeanweisungen zur Umsetzung der Schritte des Verarbeitungsverfahrens nach einem der Ansprüche 1 bis 11 umfasst, wenn diese Anweisungen von einem Prozessor ausgeführt werden.
  14. Speicherungsmedium, das von einem Prozessor gelesen werden kann und auf dem ein Computerprogramm aufgezeichnet ist, das Codeanweisungen zur Ausführung der Schritte des Verarbeitungsverfahrens nach einem der Ansprüche 1 bis 11 beinhaltet.
EP18737650.4A 2017-06-09 2018-05-24 Verarbeitung von klangdaten zur trennung von klangquellen in einem mehrkanalsignal Active EP3635718B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FR1755183A FR3067511A1 (fr) 2017-06-09 2017-06-09 Traitement de donnees sonores pour une separation de sources sonores dans un signal multicanal
PCT/FR2018/000139 WO2018224739A1 (fr) 2017-06-09 2018-05-24 Traitement de donnees sonores pour une separation de sources sonores dans un signal multicanal

Publications (2)

Publication Number Publication Date
EP3635718A1 EP3635718A1 (de) 2020-04-15
EP3635718B1 true EP3635718B1 (de) 2023-06-28

Family

ID=59746081

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18737650.4A Active EP3635718B1 (de) 2017-06-09 2018-05-24 Verarbeitung von klangdaten zur trennung von klangquellen in einem mehrkanalsignal

Country Status (5)

Country Link
US (1) US11081126B2 (de)
EP (1) EP3635718B1 (de)
CN (1) CN110709929B (de)
FR (1) FR3067511A1 (de)
WO (1) WO2018224739A1 (de)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110473566A (zh) * 2019-07-25 2019-11-19 深圳壹账通智能科技有限公司 音频分离方法、装置、电子设备及计算机可读存储介质
WO2021164001A1 (en) * 2020-02-21 2021-08-26 Harman International Industries, Incorporated Method and system to improve voice separation by eliminating overlap
JP7351401B2 (ja) * 2020-02-26 2023-09-27 日本電信電話株式会社 信号処理装置、信号処理方法、およびプログラム
CN113450823B (zh) * 2020-03-24 2022-10-28 海信视像科技股份有限公司 基于音频的场景识别方法、装置、设备及存储介质
WO2022079854A1 (ja) * 2020-10-15 2022-04-21 日本電信電話株式会社 音響信号強調装置、方法及びプログラム
FR3116348A1 (fr) * 2020-11-19 2022-05-20 Orange Localisation perfectionnée d’une source acoustique
CN112599144B (zh) * 2020-12-03 2023-06-06 Oppo（重庆）智能科技有限公司 音频数据处理方法、音频数据处理装置、介质与电子设备
JP7552742B2 (ja) * 2021-02-15 2024-09-18 日本電信電話株式会社 音源分離装置、音源分離方法、およびプログラム
KR20230153409A (ko) * 2021-03-11 2023-11-06 돌비 레버러토리즈 라이쎈싱 코오포레이션 미디어 유형에 기반한 잔향 제거
US20230057082A1 (en) * 2021-08-19 2023-02-23 Sony Group Corporation Electronic device, method and computer program
CN116229997B (zh) * 2022-12-27 2026-03-10 中国电信股份有限公司 多通道音频数据增强处理方法、装置、设备及介质

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6879952B2 (en) * 2000-04-26 2005-04-12 Microsoft Corporation Sound source separation using convolutional mixing and a priori sound source knowledge
US20040086130A1 (en) * 2002-05-03 2004-05-06 Eid Bradley F. Multi-channel sound processing systems
US7809145B2 (en) * 2006-05-04 2010-10-05 Sony Computer Entertainment Inc. Ultra small microphone array
US7099821B2 (en) * 2003-09-12 2006-08-29 Softmax, Inc. Separation of target acoustic signals in a multi-transducer arrangement
KR100647286B1 (ko) * 2004-08-14 2006-11-23 삼성전자주식회사 교차채널 간섭을 제거하기 위한 후처리장치 및 방법과이를 이용한 다채널 음원 분리장치 및 방법
WO2007029412A1 (ja) * 2005-09-01 2007-03-15 Matsushita Electric Industrial Co., Ltd. マルチチャンネル音響信号処理装置
WO2007103037A2 (en) * 2006-03-01 2007-09-13 Softmax, Inc. System and method for generating a separated signal
FR2899424A1 (fr) * 2006-03-28 2007-10-05 France Telecom Procede de synthese binaurale prenant en compte un effet de salle
FR2903562A1 (fr) * 2006-07-07 2008-01-11 France Telecom Spatialisation binaurale de donnees sonores encodees en compression.
CN101622669B (zh) * 2007-02-26 2013-03-13 高通股份有限公司 用于信号分离的系统、方法及设备
WO2008120933A1 (en) * 2007-03-30 2008-10-09 Electronics And Telecommunications Research Institute Apparatus and method for coding and decoding multi object audio signal with multi channel
US8131542B2 (en) * 2007-06-08 2012-03-06 Honda Motor Co., Ltd. Sound source separation system which converges a separation matrix using a dynamic update amount based on a cost function
GB0720473D0 (en) * 2007-10-19 2007-11-28 Univ Surrey Accoustic source separation
JP5195652B2 (ja) * 2008-06-11 2013-05-08 ソニー株式会社 信号処理装置、および信号処理方法、並びにプログラム
JP4816711B2 (ja) * 2008-11-04 2011-11-16 ソニー株式会社 通話音声処理装置および通話音声処理方法
US20110058676A1 (en) * 2009-09-07 2011-03-10 Qualcomm Incorporated Systems, methods, apparatus, and computer-readable media for dereverberation of multichannel signal
KR101567461B1 (ko) * 2009-11-16 2015-11-09 삼성전자주식회사 다채널 사운드 신호 생성 장치
US9165565B2 (en) * 2011-09-09 2015-10-20 Adobe Systems Incorporated Sound mixture recognition
US9654894B2 (en) * 2013-10-31 2017-05-16 Conexant Systems, Inc. Selective audio source enhancement

Also Published As

Publication number Publication date
US20200152222A1 (en) 2020-05-14
WO2018224739A1 (fr) 2018-12-13
FR3067511A1 (fr) 2018-12-14
EP3635718A1 (de) 2020-04-15
CN110709929A (zh) 2020-01-17
CN110709929B (zh) 2023-08-15
US11081126B2 (en) 2021-08-03

Similar Documents

Publication Publication Date Title
EP3635718B1 (de) Verarbeitung von klangdaten zur trennung von klangquellen in einem mehrkanalsignal
EP3807669B1 (de) Ortung von schallquellen in einer bestimmten akustischen umgebung
JP4406428B2 (ja) 信号分離装置、信号分離方法、信号分離プログラム及び記録媒体
EP2898707B1 (de) Optimierte kalibrierung eines klangwiedergabesystems mit mehreren lautsprechern
EP4046390B1 (de) Verbesserte ortung einer schallquelle
Wang et al. Over-determined source separation and localization using distributed microphones
US10078785B2 (en) Video-based sound source separation
May et al. Requirements for the evaluation of computational speech segregation systems
EP4248231A1 (de) Verbesserte ortung einer akustischen quelle
WO2016162515A1 (fr) Procédé de séparation de sources pour signaux parcimonieux
WO2011076696A1 (fr) Procede pour estimer le nombre de sources incidentes a un reseau de capteurs par estimation de la statistique du bruit
EP3559947B1 (de) Verarbeitung in subbändern eines aktuellen ambisonic-inhalts zur verbesserten dekodierung
EP3025342B1 (de) Verfahren zur unterdrückung des späten nachhalls eines akustischen signals
Zohny et al. Modelling interaural level and phase cues with Student's t-distribution for robust clustering in MESSL
EP4315328B1 (de) Schätzung einer optimierten maske zur verarbeitung von erfassten tondaten
Cobos et al. Two-microphone separation of speech mixtures based on interclass variance maximization
Hammond et al. Robust full-sphere binaural sound source localization
EP1359685A1 (de) Quellentrennung für zyklostationäre Signale
Li et al. Separation of Multiple Speech Sources in Reverberant Environments Based on Sparse Component Enhancement
Bentsen et al. The impact of exploiting spectro-temporal context in computational speech segregation
Hammond Methods for Robust Binaural Sound Source Localisation
US11835625B2 (en) Acoustic-environment mismatch and proximity detection with a novel set of acoustic relative features and adaptive filtering
RU2788939C1 (ru) Способ и устройство для определения глубокого фильтра
Hadadi et al. Blind Localization of Early Room Reflections with Arbitrary Microphone Array
Cheng et al. Using spatial audio cues from speech excitation for meeting speech segmentation

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20191210

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: ORANGE

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
RAP3 Party data changed (applicant data changed or rights of an application transferred)

Owner name: ORANGE

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20211006

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20230310

RIN1 Information on inventor provided before grant (corrected)

Inventor name: GUERIN, ALEXANDRE

Inventor name: BAQUE, MATHIEU

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1583444

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230715

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: FRENCH

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018052425

Country of ref document: DE

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230928

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20230628

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1583444

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230929

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231028

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231030

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20231028

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018052425

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

26N No opposition filed

Effective date: 20240402

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230628

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240524

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240531

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20240531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240531

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250423

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20250423

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180524

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20260317

Year of fee payment: 9