EP2187389A2 - Tonverarbeitungsvorrichtung - Google Patents

Tonverarbeitungsvorrichtung Download PDF

Info

Publication number
EP2187389A2
EP2187389A2 EP20090014232 EP09014232A EP2187389A2 EP 2187389 A2 EP2187389 A2 EP 2187389A2 EP 20090014232 EP20090014232 EP 20090014232 EP 09014232 A EP09014232 A EP 09014232A EP 2187389 A2 EP2187389 A2 EP 2187389A2
Authority
EP
European Patent Office
Prior art keywords
frequency
frequencies
observed
matrix
learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP20090014232
Other languages
English (en)
French (fr)
Other versions
EP2187389B1 (de
EP2187389A3 (de
Inventor
Makoto Yamada
Kazunobu Kondo
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yamaha Corp
Original Assignee
Yamaha Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yamaha Corp filed Critical Yamaha Corp
Publication of EP2187389A2 publication Critical patent/EP2187389A2/de
Publication of EP2187389A3 publication Critical patent/EP2187389A3/de
Application granted granted Critical
Publication of EP2187389B1 publication Critical patent/EP2187389B1/de
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0272—Voice signal separating
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/0208—Noise filtering
    • G10L21/0216—Noise filtering characterised by the method used for estimating noise
    • G10L2021/02161—Number of inputs available containing the signal or the noise to be suppressed
    • G10L2021/02165—Two microphones, one receiving mainly the noise signal and the other one mainly the speech signal
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00—Circuits for transducers
    • H04R3/005—Circuits for transducers for combining the signals of two or more microphones

Definitions

  • the present invention relates to a technology for emphasizing (typically, separating or extracting) or suppressing a specific sound in a mixture of sounds.
  • Each sound in a mixture of a plurality of sounds (voice or noise) emitted from separate sound sources is individually emphasized or suppressed by performing sound source separation on a plurality of observed signals that a plurality of sound receiving devices produce by receiving the mixture of the plurality of sounds.
  • Learning according to Independent Component Analysis (ICA) is used to calculate a separation matrix used for sound source separation of the observed signals.
  • FDICA Frequency-Domain Independent Component Analysis
  • FDICA requires a large-capacity storage unit that stores the time series of observed vectors of each of the plurality of frequencies.
  • terminating the learning of separation matrices of frequencies at which the accuracy of separation undergoes little change reduces the amount of calculation
  • the technology of Japanese Patent Application Publication No. 2006-84898 requires a large-capacity storage unit to store the time series of observed vectors for all frequencies since learning of the separation matrix is performed for every frequency when the learning is initiated.
  • an object of the invention is to reduce the capacity of storage required to generate (or learn) separation matrices.
  • a signal processing device processes a plurality of observed signals at a plurality of frequencies, the plurality of the observed signals being produced by a plurality of sound receiving devices which receive a mixture of a plurality of sounds (such as voice or (non-vocal) noise).
  • the inventive signal processing device comprises: a storage means that stores observed data of the plurality of the observed signals, the observed data representing a time series of magnitude (amplitude or power) of each frequency in each of the plurality of the observed signals; an index calculation means that calculates an index value from the observed data for each of the plurality of the frequencies, the index value indicating significance of learning of a separation matrix using the observed data of each frequency, the separation matrix being used for separation of the plurality of the sounds; a frequency selection means that selects at least one frequency from the plurality of the frequencies according to the index value of each frequency calculated by the index calculation means; and a learning processing means that determines the separation matrix by learning with a given initial separation matrix using the observed data of the frequency selected by the frequency selection means among the plurality of the observed data stored in the storage means.
  • the total number of bases in a distribution of observed vectors, each including, as elements, respective magnitudes of a corresponding frequency in the plurality of observed signals is preferably used as an index indicating the significance of learning using observed data. Therefore, in a preferred embodiment of the invention, the index calculation means calculates an index value representing a total number of bases in a distribution of observed vectors obtained from the observed data, each observed vector including, as elements, respective magnitudes of a corresponding frequency in the plurality of the observed signals, and the frequency selection means selects one or more frequency at which the total number of the bases represented by the index value is larger than total number of bases represented by index values at other frequencies.
  • a determinant or a number of conditions of a covariance matrix of the observed vector is preferably used as the index value indicating the total number of bases.
  • the index calculation means calculates a first determinant corresponding to product of a first number of diagonal elements (for example, n diagonal elements) among a plurality of diagonal elements of a singular value matrix specified through singular value decomposition of the covariance matrix of the observed vectors, and a second determinant corresponding to product of a second number of the diagonal elements (for example, n-1 diagonal elements), which are fewer in number than the first number of the diagonal elements, among the plurality of diagonal elements, and the frequency selection means sequentially performs frequency selection using the first determinant and frequency selection using the second determinant.
  • the index calculation means calculates an index value representing independency between the plurality of the observed signals at each frequency, and the frequency selection means selects one or more frequency at which the independency represented by the index value is higher than independencies calculated at other frequencies.
  • a correlation between the plurality of the observed signals or an amount of mutual information of the plurality of the observed signals is preferably used as the index value of the independency between the plurality of the observed signals.
  • the frequency selection means selects a frequency at which the trace of the covariance matrix of the plurality of observed signals is great.
  • an observed signal includes a greater number of sounds from a greater number of sound sources as the kurtosis of a frequence distribution of the magnitude of the observed signal decreases
  • the frequency selection means selects a frequency at which the kurtosis of the frequence distribution of the magnitude of the observed signal is lower than kurtoses at other frequencies.
  • the learning processing means generates the separation matrix of the frequency selected by the frequency selection means through learning using the initial separation matrix of the selected frequency as an initial value, and uses the initial separation matrix of a frequency not selected by the frequency selection means as a separation matrix of the frequency that is not selected. According to this configuration, it is possible to easily prepare separation matrices of unselected frequencies.
  • the signal processing device further comprises a direction estimation means that estimates a direction of a sound source of each of the plurality of the sounds from the separation matrix generated by the learning processing means; and a matrix supplementation means that generates a separation matrix of a frequency not selected by the frequency selection means from the direction estimated by the direction estimation means.
  • a direction estimation means that estimates a direction of a sound source of each of the plurality of the sounds from the separation matrix generated by the learning processing means
  • a matrix supplementation means that generates a separation matrix of a frequency not selected by the frequency selection means from the direction estimated by the direction estimation means.
  • the direction estimation means estimates a direction of a sound source of each of the plurality of the sounds from the separation matrix that is generated by the learning processing means for a frequency excluding at least one of a frequency at lower-band-side and a frequency at higher-band-side among the plurality of the frequencies.
  • the index calculation means sequentially calculates, for each unit interval of the sound signals, an index value of each of the plurality of the frequencies
  • the frequency selection means comprises: a first selection means that sequentially determines, for each unit interval, whether or not to select each of the plurality of the frequencies according to an index value of the unit interval; and a second selection means that selects the at least one frequency from results of the determination of the first selection means for a plurality of unit intervals.
  • frequencies are selected from the results of the determination of the first selection means for a plurality of unit intervals, whether or not to select frequencies is reliably determined even when observed data changes (for example, when noise is great), compared to the configuration in which frequencies are selected from the index value of only one unit interval. Accordingly, there is an advantage in that the separation matrix is accurately learned.
  • the first selection means sequentially generates, for each unit interval, a numerical value sequence indicating whether or not each of the plurality of the frequencies is selected, and the second selection means selects the at least one frequency based on a weighted sum of respective numerical value sequences of the plurality of the unit intervals.
  • frequencies are selected from a weighted sum of respective numerical value sequences of the plurality of unit intervals, there is an advantage in that whether or not to select frequencies can be determined preferentially taking into consideration the index value of a specific unit interval among the plurality of unit intervals (i.e., preferentially taking into consideration the results of determination of whether or not to select frequencies).
  • the signal processing device may not only be implemented by hardware (electronic circuitry) such as a Digital Signal Processor (DSP) dedicated to audio processing but may also be implemented through cooperation of a general arithmetic processing unit such as a Central Processing Unit (CPU) with a program.
  • a program is provided according to the invention for use in a computer having a processor for.processing a plurality of observed signals at a plurality of frequencies, the plurality of the observed signals being produced by a plurality of sound receiving'devices which receive a mixture of a plurality of sounds, and a storage that stores observed data of the plurality of the observed signals, the observed data representing a time series of magnitude of each frequency in each of the plurality of the observed signals.
  • the program is executed by the processor to perform: an index calculation process for calculating an index value from the observed data for each of the plurality of the frequencies, the index value indicating significance of learning of a separation matrix using the observed data of each frequency, the separation matrix being used for separation of the plurality of the sounds; a frequency selection process for selecting at least one frequency from the plurality of the frequencies according to the index value of each frequency calculated by the index calculation process; and a learning process for determining the separation matrix by learning with a given initial separation matrix using the observed data of the frequency selected by the frequency selection process among the plurality of the observed data stored in the storage.
  • This program achieves the same operations and advantages as those of the signal processing device according to the invention.
  • the program of the invention may be provided to a user through a computer machine readable recording medium storing the program and then installed on a computer and may also be provided from a server device to a user through distribution over a communication network and then installed on a computer.
  • FIG. 1 is a block diagram of a signal processing device associated with a first embodiment of the invention.
  • An n number of sound receiving devices M which are located at intervals in a plane PL are connected to a signal processing device 100, where n is a natural number equal to or greater than 2.
  • n is a natural number equal to or greater than 2.
  • An n number of sound sources S S1, S2 are provided at different positions around the sound receiving device M1 and the sound receiving device M2.
  • the sound source S1 is located in a direction at an angle of ⁇ 1 with respect to the normal Ln to the plane PL and the sound source S2 is located in a direction at an angle of ⁇ 2 ( ⁇ 2 ⁇ 1) with respect to the normal Ln.
  • a mixture of a sound SV1 emitted from the sound source S1 and a sound SV2 emitted from the sound source S2 arrives at the sound receiving device M1 and the sound receiving device M2.
  • the sound receiving device M1 and the sound receiving device M2 are microphones that generate observed signals V (V1, V2) representing a waveform of the mixture of the sound SV1 from the sound source S1 and the sound SV2 from the sound source S2.
  • the sound receiving device M1 generates the observed signal V1 and the sound receiving device M2 generates the observed signal V2.
  • the signal processing device 100 performs a filtering process (for sound source separation) on the observed signal V1 and the observed signal V2 to generate a separated signal U1 and a separated signal U2.
  • the separated signal U1 is an audio signal obtained by emphasizing the sound SV1 from the sound source S1 (i.e., obtained by suppressing the sound SV2 from the sound source S2) and the separated signal U2 is an audio signal obtained by emphasizing the sound SV2 from the sound source S2 (i.e., obtained by suppressing the sound SV1). That is, the signal processing device 100 performs sound source separation to separate the sound SV1 of the sound source S1 and the sound SV2 of the sound source S2 from each other (sound source separation).
  • the separated signal U1 and the separated signal U2 are provided to a sound emitting device (for example, speakers or headphones) to be reproduced as audio.
  • a sound emitting device for example, speakers or headphones
  • This embodiment may also employ a configuration in which only one of the separated signal U1 and the separated signal U2 is reproduced (for example, a configuration in which the separated signal U2 is discarded as noise).
  • An A/D converter that converts the observed signal V1 and the observed signal V2 into digital signals and a D/A converter that converts the separated signal U1 and the separated signal U2 into analog signals are not illustrated for the sake of convenience.
  • the signal processing device 100 is implemented as a computer system including an arithmetic processing unit 12 and a storage unit 14.
  • the storage unit 14 is a machine readable medium that stores a program and a variety of data for generating the separated signal U1 and the separated signal U2 from the observed signal V1 and the observed signal V2.
  • a known machine readable recording medium such as a semiconductor recording medium or a magnetic recording medium is arbitrarily employed as the storage unit 14.
  • the arithmetic processing unit 12 functions as a plurality of components (for example, a frequency analyzer 22, a signal processing unit 24, a signal synthesizer 26, and a separation matrix generator 40) by executing the program stored in the storage unit 14.
  • This embodiment may also employ a configuration in which an electronic circuit (DSP) dedicated to processing observed signals V implements each of the components of the arithmetic processing unit 12 or a configuration in which each of the components of the arithmetic processing unit 12 is mounted in a distributed manner on a plurality of integrated circuits.
  • DSP electronic circuit
  • the frequency analyzer 22 calculates frequency spectrums Q (i.e., a frequency spectrum Q1 of the observed signal V1 and a frequency spectrum Q2 of the observed signal V2) for each of a plurality of frames into which the observed signals V (V1, V2) are divided in time. For example, short-time Fourier transform may be used to calculate each frequency spectrum Q. As shown in FIG. 2 , the frequency spectrum Q1 of one frame identified by a number (time) t is calculated as a set of respective magnitudes x1 (t, f1) to x1(t, fK) of K frequencies f1 to fK set on the frequency axis. Similarly, the frequency spectrum Q2 is calculated as a set of respective magnitudes x2 (t, f1) to x2(t, fK) of the K frequencies f1 to fK.
  • the frequency analyzer 22 generates observed vectors X (t, f1) to X(t, fK)of each frame for the K frequencies f1 to fK.
  • the observed vectors X (t, f1) to X(t, fK) that the frequency analyzer 22 generates for each frame are stored in the storage unit 14.
  • the observed vectors X (t, f1) to X(t, fK) stored in the storage unit 14 are divided into observed data D(f1) to D(fK) of unit intervals TU, each including a predetermined number of (for example, 50) frames as shown in FIG. 2 .
  • the observed data D(fk) of the frequency fk is a time series of the observed vector X (t, fk) of the frequency fk calculated for each frame of the unit interval TU.
  • the signal processing unit 24 of FIG. 1 sequentially generates a magnitude u1(t, fk) and a magnitude u2(t, fk) for each frame by performing a filtering.process (or sound source separation) on the magnitude x1(t, fk) and the magnitude x2(t, fk) calculated by the frequency analyzer 22.
  • the signal synthesizer 26 converts the magnitudes u1(t, f1) to u1(t, fK) generated by the signal processing unit 24 into a time-domain signal and connects adjacent frames to generate a separated signal U1.
  • the signal synthesizer 26 converts the magnitudes u2(t, f1) to u2(t, fK) into a time-domain signal and connects adjacent frames to generate a separated signal U2.
  • FIG. 3 is a block diagram of the signal processing unit 24.
  • the signal processing unit 24 includes K processing units P1 to PK corresponding respectively to the K frequencies f1 to fK.
  • the processing unit Pk corresponding to the frequency fk includes a filter 32 that generates the magnitude u1 (t, fk) from the magnitude x1 (t, fk) and the magnitude x2 (t, fk) and a filter 34 that generates the magnitude u2(t, fk) from the magnitude x1 (t, fk) and the magnitude x2 (t, fk).
  • a Delay-Sum (DS) type beam-former is used for each of the filter 32 and the filter 34.
  • the filter 32 of the processing unit Pk includes a delay element 321 that adds delay according to a coefficient w11(fk) to the magnitude x1(t, fk), a delay element 323 that adds delay according to a coefficient w21(fk) to the magnitude x2(t, fk), and an adder 325 that sums an output of the delay element 321 and an output of the delay element 323 to generate the magnitude u1(t, fk) of the separated signal U1.
  • the filter 34 of the processing unit Pk includes a delay element 341 that adds delay according to a coefficient w12(fk) to the magnitude x1(t, fk), a delay element 343 that adds delay according to a coefficient w22(fk) to the magnitude x2(t, fk), and an adder 345 that sums an output of the delay element 341 and an output of the delay element 343 to generate the magnitude u2(t, fk) of the separated signal U2.
  • the separation matrix generator 40 shown in FIGS. 1 and 3 generates separation matrices W(f1) to W(fK) used by the signal processing unit 24.
  • the separation matrix W(fk) of the frequency fk is a matrix of 2 rows and 2 columns (n rows and n columns in general form) whose elements are the coefficients w11(fk) and w21(fk) applied to the filter 32 of the processing unit Pk and the coefficients w12(fk) and w22(fk) applied to the filter 34 of the processing unit Pk.
  • the separation matrix generator 40 generates the separation matrix W(fk) from the observed data D(fk) stored in the storage unit 14. That is, the separation matrix W(fk) is generated in each unit interval TU for each of the K frequencies f1 to fK.
  • FIG. 4 is a block diagram of the separation matrix generator 40.
  • the separation matrix generator 40 includes an initial value generator 42, a learning processing unit 44, an index calculator 52, and a frequency selector 54.
  • the initial value generator 42 generates respective initial separation matrices W0(f1) to W0(fK) for the K frequencies f1 to fK.
  • the initial separation matrix W0(fk) corresponding to the frequency fk is generated for each unit interval TU using the observed data D(fk) stored in the storage unit 14. Any known technology is used to generate the initial separation matrices W0(f1) to W0(fK).
  • this embodiment preferably uses a partial space method such as second-order static ICA or main component analysis described in K. Tachibana, et al., "Efficient Blind Source Separation Combining Closed-Form Second-Order ICA and Non-Closed-Form Higher-Order ICA," International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vol. 1, pp.45-48, April 2007 or an adaptive beam-former described in Patent No. 3949074 .
  • a partial space method such as second-order static ICA or main component analysis described in K. Tachibana, et al., "Efficient Blind Source Separation Combining Closed-Form Second-Order ICA and Non-Closed-Form Higher-Order ICA," International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vol. 1, pp.45-48, April 2007 or an adaptive beam-former described in Patent No. 3949074 .
  • This embodiment may also employ a method in which the initial separation matrices W0(f1) to W0(fK) are specified using a variety of beam-formers (for example, adaptive beam-formers) from the directions of sound sources S estimated using a minimum variance method, or a multiple signal classification (MUSIC) method or the initial separation matrices W0(f1) to W0(fK) are specified from canonical vectors specified using canonical correlation analysis or a factor vector specified using factor analysis.
  • a variety of beam-formers for example, adaptive beam-formers
  • MUSIC multiple signal classification
  • the learning processing unit 44 of FIG. 4 generates separation matrices W(fk) (W(f1) to W(fK)) by performing sequential learning on each of the K frequencies f1 to fK using the initial separation matrix W0(fk) as an initial value.
  • the observed data D(fk) of the frequency fk stored in the storage unit 14 is used to learn the separation matrix W(fk).
  • an independent component analysis for example, high-order ICA
  • the separation matrix W(fk) is repeatedly updated so that the separated signal U1 (which is a time series of the magnitude u1 in Equation (1a)) and the separated signal U2 (which is a time.series of the magnitude u2 in Equation (1b)), which are separated from the observed data D(fk) using the separation matrix W(fk), are statistically independent of each other is preferably used to generate the separation matrix W(fk).
  • ICA independent component analysis
  • the learning processing unit 44 performs learning of the separation matrix W(fk) using the observed data D(fk) for one or more frequencies fk, in which the significance and efficiency of learning of the separation matrix W(fk) using the observed data D(fk) is high (i.e., the degree of improvement of the accuracy of sound source separation through learning of the separation matrix W(fk), compared to when the initial separation matrix W0(fk) is used, is high), among the K frequencies f1 to fK.
  • the index calculator 52 of FIG. 4 calculates an index value that is used as a reference for selecting the frequencies (fk).
  • the index calculator 52 of the first embodiment calculates a determinant z1(fk) (z1(f1) to z1(fK)) of a covariance matrix Rxx(fk) of the observed data D(fk) (i.e., of the observed signal V1 and the observed signal V2) for each of the K frequencies f1 to fK.
  • the index calculator 52 includes a covariance matrix calculator 522 and a determinant calculator 524.
  • the covariance matrix calculator 522 calculates a covariance matrix Rxx(fk) (Rxx(f1) to Rxx(fK)) of the observed data D(fk) for each of the K frequencies f1 to fK.
  • the covariance matrix Rxx(fk) is a matrix whose elements are covariances of the observed vectors X(t, fk) in the observed data D(fk) (in the unit interval TU).
  • the covariance matrix Rxx(fk) is defined, for example, using the following Equation (2).
  • Equation (3) it is assumed that the sum of observed vectors X(t, fk) of all frames in the unit interval TU is a zero matrix (i.e., zero average) as represented by the following Equation (3).
  • Equations (2) and (3) denotes the expectation (or sum) and the symbol ⁇ _(t) denotes the sum (or average) over a plurality of (for example, 50) frames in the unit interval TU.
  • the covariance matrix Rxx(fk) is a matrix of n rows and n columns obtained by summing the products of the observed vectors X(t, fk) and the transposes of the observed vectors X(t, fk) over a plurality of observed vectors X(t, fk) in the unit interval TU (i.e., in the observed data D(fk)).
  • the determinant calculator 524 calculates respective determinants z1(fk) (z1(f1) to z1(fK)) for the K covariance matrices Rxx(f1) to Rxx(fK) calculated by the covariance matrix calculator 522.
  • this embodiment preferably employs, for example, the following method using singular value decomposition of the covariance matrix Rxx(fk).
  • Each covariance matrix Rxx(fk) is singular-value-decomposed as represented by the following Equation (4).
  • a matrix F in Equation (4) is an orthogonal matrix of n rows and n columns (2 rows and 2 columns in this embodiment) and a matrix D is a singular value matrix of n rows and n columns in which all elements other than diagonal elements d1, ..., dn are zero.
  • Rxx fk FDF H
  • Equation (5) the determinant zi(fk) of the covariance matrix Rxx(fk) is represented by the following Equation (5).
  • a relation (F H F I) that the product of the transpose F H of a matrix F and the matrix F is an n-order unit matrix and a relation that the determinant det (AB) of a matrix AB is equal to the determinant det (BA) of a matrix BA are used to derive Equation (5).
  • z ⁇ 1 fk det
  • the determinant z1(fk) of the covariance matrix Rxx(fk) corresponds to the product of the n diagonal elements (d1, ., dn) of the singular value matrix D specified through singular value decomposition of the covariance matrix Rxx(fk).
  • the determinant calculator 524 calculates determinants z1(f1) to z1(fK) by-performing the calculation.of Equation (5) for each of the K frequencies f1 to fK.
  • FIGS. 6(A) and 6(B) are scatter diagrams of observed vectors X (t, fk) in a unit interval TU.
  • the horizontal axis represents the magnitude x1(t, fk) and the vertical axis represents the magnitude x2(t, fk).
  • FIG. 6(A) is a scatter diagram when the determinant z1(fk) is great and FIG. 6(B) is a scatter diagram when the determinant z1(fk) is small.
  • an axis line (basis) of a region in which the observed vectors X(t, fk) are distributed is clearly discriminated for each sound source S when the determinant z1(fk) of the covariance matrix Rxx(fk) is great.
  • a region A2 in which observed vectors X(t, fk), where the sound SV2 from the sound source S2 is dominant are distributed along an axis line ⁇ 2 are clearly discriminated.
  • the determinant z1(fk) of the covariance matrix Rxx(fk) is small, the number of regions (or the number of axis lines) in which observed vectors X(t, fk) are distributed, which can be clearly discriminated in a scatter diagram, is less than the total number of actual sound sources S.
  • a definite region A2 (axis line ⁇ 2) corresponding to the sound SV2 from the sound source S2 is not present as shown in FIG. 6(B) .
  • the determinant z1(fk) of the covariance matrix Rxx(fk) serves as an index indicating the total number of bases of distributions of observed vectors X(t, fk) included in the observed data D(fk) (i.e., the total number of axis lines of regions in which the observed vectors X(t, fk) are distributed). That is, there is a tendency that the number of bases of a frequency fk increases as the determinant z1(fk) of the frequency fk increases. Only one independent basis is present at a frequency fk at which the determinant z1(fk) is zero.
  • independent component analysis applied to learning of the separation matrix W(fk) through the learning processing unit 44 is equivalent to a process for specifying the number of independent bases as same as the number of sound sources S, it can be considered that the significance of learning of observed data D(fk) (i.e., the degree of improvement of the accuracy of sound source separation through learning of the separation matrix W(fk)) is small at-a frequency fk, at which the determinant z1(fk) of the covariance matrix Rxx(fk) is small, among the K frequencies f1 to fK.
  • the separation matrix W(fk) is generated through learning, by the learning processing unit 44, of only frequencies fk at which the determinant z1(fk) is large among the K frequencies f1 to fK (i.e., when, for example, the initial separation matrix W0(fk) is used as the separation matrix W(fk) without learning at each frequency fk at which the determinant z1(fk) is small), it is possible to perform sound source separation with almost the same accuracy as when the separation matrices W(f1) to W(fK) are specified through learning of all observed data D(f1) to D(fK) of the K frequencies f1 to fK.
  • the determinant z1(fk) as an index value of the significance of learning of the separation matrix W(fk) using the observed data D(fk) of the frequency fk.
  • the frequency selector 54 of FIG. 4 selects one or more frequencies fk at which the determinant z1(fk) calculated by the index calculator 52 is large from the K frequencies f1 to fK. For example, the frequency selector 54 selects, from the K frequencies f1 to fK, a predetermined number of frequencies fk, which are located at higher positions when the K frequencies f1 to fK are arranged in descending order of the determinants z1(f1) to z1(fK) (i.e., in decreasing order of the determinants), or selects one or more frequencies fk whose determinant z1(fk) is greater than a predetermined threshold from the K frequencies f1 to fK.
  • FIG. 7 is a conceptual diagram illustrating a relation between selection through the frequency selector 54 and learning through the learning processing unit 44.
  • the learning processing unit 44 For each frequency fk (f1, f2, ..., fK-1 in FIG. 7 ) selected by the frequency selector 54, the learning processing unit 44 generates the separation matrix W(fk) by sequentially updating the initial separation matrix W0(fk) using the observed data D(fk) of the frequency fk.
  • the initial separation matrix W0(fk) specified by the initial value generator 42 is set as the separation matrix W(fk) without learning in the signal processing unit 24.
  • this embodiment has advantages in that the capacity of the storage unit 14 required to generate the separation matrices W(f1) to W(fK) is reduced and the load of processing through the learning processing unit 44 is also reduced.
  • FIG. 8 illustrates a relation between the number of frequencies fk that are subjected to learning by the learning processing unit 44 (when the total number of K frequencies is 512), Noise Reduction Rate (NRR), and the required capacity of the storage unit 14.
  • the capacity of the storage unit 14 is expressed; assuming that the capacity required for learning using the observed data D(fk) of all frequencies (f1-f512) is 100%.
  • the ratio of change of the capacity of the storage unit 14 to change of the number of frequencies fk that are subjected to learning is sufficiently high, compared to the ratio of change of the NRR to change of the number of frequencies fk.
  • the NRR is reduced by about 20% (14.37->11.5) while the capacity of the storage unit 14 is reduced by about 90%.
  • FIG. 9 is a flow chart of the operations of the index calculator 52 and the frequency selector 54. The procedure of FIG. 9 is performed for each unit interval TU.
  • the index calculator 52 initializes a variable N to n which is the total number of sound receiving devices M (i.e., the total number of sound sources S that are subjected to sound source separation) (step S1), and then calculates determinants z1(f1) to z1(fK) (step S2).
  • the determinant z1(fk) is calculated as the product of N diagonal elements (n diagonal elements d1, d2, ..., dn at the present step) of the singular value matrix D of the covariance matrix Rxx(fk).
  • the frequency selector 54 selects one or more frequencies fk at which the determinant z1(fk) that the index calculator 52 calculates at step S2 is great (step S3).
  • this embodiment preferably employs a configuration in which the frequency selector 54 selects, from the K frequencies f1 to fK, a predetermined number of frequencies fk, which are located at higher positions when the K frequencies f1 to fK are arranged in descending order of the determinants z1(f1) to z1(fK), or a configuration in which the frequency selector 54 selects one or more frequencies fk whose determinant z1(fk) is greater than a predetermined threshold from the K frequencies f1 to fK.
  • the frequency selector 54 determines whether or not the number of selected frequencies fk has reached a predetermined value (step S4). The procedure of FIG. 9 is terminated when the number of selected frequencies fk is equal to or greater than the predetermined value (YES at step S4).
  • the index calculator 52 subtracts 1 from the variable N (step S5) and calculates determinants z1(f1) to z1(fK) corresponding to the changed variable N (step S2). That is, the index calculator 52 calculates the determinant z1(fk) after removing one diagonal element from the n diagonal elements of the singular value matrix D of the covariance matrix Rxx(fk).
  • the frequency selector 54 selects a frequency fk, which does not overlap the previously selected frequencies fk, using determinants z1(f1) to z1(fK) newly calculated at step S1 (step S3).
  • the index calculator 52 and frequency selector 54 repeat the calculation of the determinant z1(fk) (step S2) and the selection of the frequency fk (step S3) while sequentially decrementing (the variable N indicating) the number of diagonal elements used to calculate the determinant z1(fk) among the n diagonal elements of the singular value matrix D of the covariance matrix Rxx(fk).
  • the process for reducing the number of diagonal elements of the singular value matrix D (step S5) is equivalent to the process for removing one basis in the distribution of the observed vectors X(t, fk).
  • the determinants z1(f1) to z1(fK) which are indicative of selection of frequencies fk is calculated while sequentially removing bases in the distribution of the observed vectors X(t, fk). Accordingly, it is possible to accurately select frequencies fk at which the significance of learning using the observed data D is high, when compared to the case where frequencies fk are selected using determinants z1(f1) to z1(fK) calculated as the product of n'diagonal elements of the singular value matrix D.
  • the number of conditions z2(fk) of the covariance matrix Rxx(fk) of the observed vectors X(t, fk) included in the observed data D(fk) is defined by the following Equation (6).
  • An operator ⁇ A ⁇ in Equation (6) represents a norm of a matrix A (i.e., the distance of the matrix).
  • the number of conditions z2(fk) is a numerical value which is small when an inverse matrix exists for the covariance matrix Rxx(fk) (i.e., when the covariance matrix Rxx(fk) is nonsingular) and which is large when no inverse matrix exists for the covariance matrix Rxx(fk).
  • z ⁇ 2 fk ⁇ Rxx fk ⁇ ⁇ ⁇ Rxx ⁇ fk - 1 ⁇
  • Equation (7a) The covariance matrix Rxx(fk) is decomposed into eigenvalues as represented by the following Equation (7a).
  • a matrix U is an eigenmatrix, whose elements are eigenvectors and a matrix ⁇ is a matrix in which eigenvalues are arranged in diagonal elements.
  • An inverse matrix of the covariance matrix Rxx(fk) is represented by the following Equation (7b) obtained by rearranging Equation (7a).
  • Equation (7b) An inverse matrix of the covariance matrix Rxx(fk) is represented by the following Equation (7b) obtained by rearranging Equation (7a).
  • Equation (7b) An inverse matrix of the covariance matrix Rxx(fk) is represented by the following Equation (7b) obtained by rearranging Equation (7a).
  • Rxx fk U ⁇ U H
  • Rxx ⁇ fk - 1 U ⁇ - 1 U H
  • the number of conditions z2(fk) of the covariance matrix Rxx(fk) increases as the total number of bases of the observed vectors X(t, fk) decreases (i.e., the number of conditions z2(fk) decreases as the total number of bases increases). That is, the number of conditions z2(fk) of the covariance matrix Rxx(fk) serves as an index of the total number of bases of the observed vectors X(t, fk), similar to the determinant z1(fk).
  • the number of conditions z2(fk) of the covariance matrix Rxx(fk) is used to select frequencies fk.
  • the index calculator 52 calculates the numbers of conditions z2(fk) (z2(f1) to z2(fK)) by performing the calculation of Equation (6) on respective covariance matrices Rxx(fk) of the K frequencies f1 to fK.
  • the frequency selector 54 selects one or more frequencies fk at which the number of conditions z2(fk) calculated by the index calculator 52 is small.
  • the frequency selector 54 selects, from the K frequencies f1 to fK, a predetermined number of frequencies fk, which are located at higher positions when the K frequencies f1 to fK are arranged in ascending order of the numbers of conditions z2(f1) to z2(fK) (i.e., in increasing order thereof), or selects one or more frequencies fk whose number of'conditions z2(fk) is less than a predetermined threshold from the K frequencies f1 to fK.
  • the operations of the initial value generator 42 and the learning processing unit 44 are similar to those of the first embodiment.
  • the significance of learning of the separation matrix W(fk) using the observed data D(fk) of a frequency fk increases as the statistical correlation between a time series of the magnitude x1 (t, fk) of the observed signal V1 and a time series of the magnitude x2 (t, fk) of the observed signal V2 decreases, since the separation matrix W(fk) is learned such that the separated signal U1 and the separated signal U2 obtained through sound source separation of the observed data D(fk) are statistically independent of each other. Therefore, in the fourth embodiment, an index value (correlation or amount of mutual information) corresponding to the degree of independency between the observed signal V1 and the observed signal V2 is used to select frequencies fk.
  • Equation (8) A correlation z3(fk) between the component of the frequency fk of the observed signal V1 and the component of the frequency fk of the observed signal V2 is represented by the following Equation (8).
  • a symbol E denotes the sum (or average) over a plurality of frames in the unit interval TU.
  • a symbol ⁇ 1 denotes a standard deviation of the magnitude x1(t, fk) in the unit interval TU and a symbol ⁇ 2 denotes a standard deviation of the magnitude x2(t, fk) in the unit interval TU.
  • the index calculator 52 calculates the correlations z3(fk) (z3(f1) to z3(fK)) by performing the calculation of Equation (8) for each of the K frequencies f1 to fK, and the frequency selector 54 selects one or more frequencies fk at which the correlation z3(fk) is low from the K frequencies f1 to fK.
  • the frequency selector 54 selects, from the K frequencies f1 to fK, a predetermined number of frequencies fk, which are located at higher positions when the K frequencies f1 to fK are arranged in ascending order of the correlations z3(f1) to z3(fK), or selects one or more frequencies fk whose correlation z3(fk) is less than a predetermined threshold from the K frequencies f1 to fK.
  • the operations of the initial value generator 42 and the learning processing unit 44 are similar to those of the first embodiment.
  • This embodiment preferably employs a configuration in which frequencies fk are selected using the amount of mutual information z4(fk) defined by the following Equation (9) instead of the correlation z3(fk).
  • the value of the amount of mutual information z4(fk) of a frequency fk decreases as the degree of independency between the observed signal V1 and the observed signal V2 increases (i.e., as the correlation therebetween decreases), similar to the correlation z3.
  • the frequency selector 54 selects one or more frequencies fk at which the amount of mutual information z4(fk) is low from the K frequencies f1 to fK.
  • z ⁇ 4 fk - 1 / 2 ⁇ log ⁇ 1 - z ⁇ 3 ⁇ fk 2
  • FIGS. 10(A) and 10(B) are scatter diagrams of observed vectors X(t, fk) in a unit interval TU.
  • FIG. 10 (A) is a scatter diagram when the trace z5(fk) is great and
  • FIG. 10(B) is a scatter diagram when the trace z5(fk) is small.
  • FIGS. 10(A) and 10(B) schematically show a region A1 in which observed vectors X(t, fk) where the sound SV1 from the sound source S1 is dominant are distributed and a region A2 in which observed vectors X(t, fk) where the sound SV2 from the sound source S2 is dominant are distributed.
  • the width of the distribution of the observed vectors X(t, fk) increases as the trace z5(fk) of the covariance matrix Rxx(fk) increases as is also understood from the fact that the trace z5(fk) is defined as the sum of the variance ⁇ 1 2 of the magnitude x1(t, fk) and the variance ⁇ 2 2 of the magnitude x2(t, fk). Accordingly, there is a tendency that, when the trace z5(fk) of the covariance matrix Rxx(fk) is large, regions (i.e., the regions A1 and A2) in which the observed vector X(t, fk) are distributed are clearly discriminated for each sound source S as shown in FIG.
  • the trace z5(fk) serves as an index value of the pattern (width) of the region in which the observed vectors X(t, fk) are distributed.
  • the traces z5(f1) to z5(fK) of the covariance matrices Rxx(f1) to Rxx(fK) are used to select frequencies fk.
  • the index calculator 52 calculates traces z5(fk) (z5(f1) to z5(fK)) by summing the diagonal elements of the covariance matrix Rxx(fk) of each of the K frequencies f1 to fK.
  • the frequency selector 54 selects one or more frequencies fk at which the trace z5(fk) calculated by the index calculator 52 is large.
  • the frequency selector 54 selects, from the K frequencies f1 to fK, a predetermined number of frequencies fk, which are located at higher positions when the K frequencies f1 to fK are arranged in descending order of the traces z5(f1) to z5(fK), or selects one or more frequencies fk whose trace z5(fk) is greater than a predetermined threshold from the K frequencies f1 to fK.
  • the operations of the initial value generator 42 and the learning processing unit 44 are similar to those of the first embodiment.
  • Equation 10 The kurtosis z6(fk) of a frequence distribution of the magnitude x1(t, fk) of the observed signal V1 is defined by the following Equation (10), where the frequence distribution is a distribution function whose random variable is the magnitude x1(t, fk).
  • Equation 10 the frequence distribution is a distribution function whose random variable is the magnitude x1(t, fk).
  • Equation (10) the symbol ⁇ 4(fk) denotes a 4th-order central moment defined by Equation (11a) and the symbol ⁇ 2(fk) denotes a 2nd-order central moment defined by Equation (11b).
  • a symbol m(fk) denotes the average of the magnitudes x1(t, fk) of a plurality of frames in a unit interval TU.
  • the kurtosis z6(fk) has a large value when only one of the sound SV1 of the sound source S1 and the sound SV2 of the sound source S2 is included (or dominant) in the elements of the frequency (fk) of the observed signal V1, and has a small value when both the sound SV1 of the sound source S1 and the sound SV2 of the sound source S2 are included with approximately equal magnitude in the elements of the frequency (fk) of the observed signal V1 (central limit theorem).
  • the kurtoses z6(fk) (z6(f1) to z6(fK)) of the frequence distribution of the magnitude x(t, fk) of the observed signal V1 are used to select frequencies fk.
  • the index calculator 52 calculates kurtoses z6(fk) (z6(f1) to z6(fK)) by performing the calculation of Equation (10) for each of the K frequencies f1 to fK.
  • the frequency selector 54 selects one or more frequencies fk at which the kurtosis z6(fk) is small from the K frequencies f1 to fK.
  • the frequency selector 54 selects, from the K frequencies f1 to fK, a predetermined number of frequencies fk, which are located at higher positions when the K frequencies f1 to fK are arranged in ascending order of the kurtoses z6(f1) to z6(fK), or selects one or more frequencies fk whose kurtosis z6(fk) is less than a predetermined threshold from the K frequencies f1 to fK.
  • the operations of the initial value generator 42 and the learning processing unit 44 are similar to those of the first embodiment.
  • the value of kurtosis of human vocal sound is within a range from about 40 to 70.
  • the kurtosis of human vocal sound is included in a range from about 20 to 80, which will hereinafter be referred to as a "vocal range”.
  • a frequency fk at which only normal noise such as air conditioner operating noise or crowd noise is present is highly likely to be selected by the frequency selector 54 since the kurtosis of the observed signal V1 has a sufficiently low value (for example, a value less than 20).
  • the significance of learning of the separation matrix W using the observed data D(fk) of the frequency fk of normal noise is low if the target sounds of sound source separation (SV1 and SV2) are human vocal sounds.
  • this embodiment preferably employs a configuration in which the kurtosis of Equation (10) is corrected so that frequencies fk of normal noise are excluded from frequencies to be selected by the frequency selector 54.
  • the index calculator 52 calculates, as the corrected kurtosis z6(fk), the product of the value defined by Equation (10), which will hereinafter be referred to as "uncorrected kurtosis", and a weight q.
  • the weight q is selected nonlinearly with respect to the uncorrected kurtosis as illustrated in FIG. 11 .
  • the weight q is selected variably according to the uncorrected kurtosis so that the kurtosis z6(fk) corrected through multiplication by the weight q exceeds the upper limit (for example, 80) of the vocal range.
  • the weight q is set to a predetermined value (for example, 1).
  • the weight q is set to the same predetermined value as when the uncorrected kurtosis is within the vocal range since the uncorrected kurtosis is sufficiently high (i.e., since the frequency fk is less likely to be selected). According to the above configurations, it is possible to generate a separation matrix W(fk) which can accurately separate a desired sound.
  • the initial separation matrix W0(fk) specified by the initial value generator 42 is applied as the separation matrix W(fk) to the signal processing unit 24.
  • the separation matrix W(fk) of the unselected frequency fk is generated (or supplemented) using the separation matrix W(fk) learned by the learning processing unit 44.
  • FIG. 12 is a block diagram of a separation matrix generator 40 in a signal processing device 100 of the seventh embodiment
  • FIG. 13 is a conceptual diagram illustrating a procedure performed by the separation matrix generator 40.
  • the separation matrix generator 40 of the seventh embodiment includes a direction estimator 72 and a matrix supplementation unit 74 in addition to the components of the separation matrix generator 40 of the first embodiment.
  • the separation matrix W(fk) that the learning processing unit 44 learns for each frequency fk selected by the frequency selector 54 is provided to the direction estimator 72.
  • the direction estimator 72 estimates a direction ⁇ 1 of the sound source S1 and a direction ⁇ 2 of the sound source S2 from each learned separation matrix W(fk). For example, the following methods are preferably used to estimate the direction 61 and the direction ⁇ 2.
  • the direction estimator 72 estimates the direction ⁇ 1(fk) of the sound source S1 and the direction ⁇ 2(fk) of the sound source S2 for each frequency fk selected by the frequency selector 54. More specifically, the direction estimator 72 specifies the direction ⁇ 1(fk) of the sound source S1 from a coefficient w11(fk) and a coefficient w21(fk) included in the separation matrix W(fk) learned by the learning processing unit 44 and specifies the direction ⁇ 2(fk) of the sound source S2 from the coefficient w12(fk) and the coefficient w22(fk).
  • the direction of a beam formed by a filter 32 of a processing unit pk when the coefficient w11(fk) and the coefficient w21(fk) are set is estimated as the direction ⁇ 1(fk) of the sound source S1 and the direction of a beam formed by a filter 34 of a processing unit pk when the coefficient w12(fk) and the coefficient w22(fk) are set is estimated as the direction ⁇ 2(fk) of the sound source S2.
  • a method described in H. Saruwatari, et. al., "Blind Source Separation Combining Independent Component Analysis and Beam-Forming," EURASIP Journal on Applied Signal Processing Vol. 2003, No. 11, pp. 1135-1146, 2003 is preferably used to specify the direction ⁇ 1(fk) and direction ⁇ 2(fk) using the separation matrix W(fk).
  • the direction estimator 72 estimates the direction ⁇ 1 of the sound source S1 and the direction ⁇ 2 of the sound source S2 from the direction ⁇ 1(fk) and the direction ⁇ 2(fk) of each frequency fk selected by the frequency selector 54.
  • the average or central value of the direction ⁇ 1(fk) estimated for each frequency fk is specified as the direction ⁇ 1 of the sound source S1 and the average or central value of the direction ⁇ 2(fk) estimated for each frequency fk is specified as the direction ⁇ 2 of the sound source S2.
  • the matrix supplementation unit 74 of FIG. 12 specifies the separation matrix W(fk) of each unselected frequency fk from the directions ⁇ 1 and ⁇ 2 estimated by the direction estimator 72 as shown in FIG. 13 . Specifically, for each unselected frequency fk, the matrix supplementation unit 74 generates a separation matrix W(fk) of 2 rows and 2 columns whose elements are the coefficients w11(fk) and w21(fk) calculated such that the filter 32 of the processing unit pk forms a beam in the direction ⁇ 1 and the coefficients w12(fk) and w22(fk) calculated such that the filter 34 of the processing unit pk forms a beam in the direction ⁇ 2. As shown in FIGS.
  • the separation matrix W(fk) learned by the learning processing unit 44 is used for the signal processing' unit 24 for each frequency fk selected by the frequency selector 54 and the separation matrix W(fk) generated by the matrix supplementation unit 74 is used for the signal processing unit 24 for each unselected frequency fk.
  • the sevent embodiment Since the separation matrix W(fk) learned for each frequency fk selected by the frequency selector 54 is used (i.e., the initial separation matrix W0(fk) of the unselected frequency fk is not used) to generate the separation matrix W(fk) of each unselected frequency fk, the sevent embodiment has an advantage in that accurate sound source separation is achieved not only for the frequency (fk) selected by the frequency selector 54 but also for the unselected frequency fk, regardless of the performance of sound source separation of the initial separation matrix W0(fk) of the unselected frequency fk.
  • this embodiment also preferably employs a configuration in which a direction ⁇ 1(fk) and a direction ⁇ 2(fk) corresponding to a specific frequency fk among the plurality of frequencies fk selected by the frequency selector 54 are used as a direction ⁇ 1 and a direction ⁇ 2 to be used for the matrix supplementation unit 74 to generate the separation matrix W(fk).
  • the direction estimator 72 estimates the direction ⁇ 1(fk) and the direction ⁇ 2(fk) using the separation matrices W(fk) of all frequencies fk selected by the frequency selector 54.
  • the direction ⁇ 1(fk) or the direction ⁇ 2(fk) cannot be accurately estimated from separation matrices W(fk) of frequencies fk at a lower band side or frequencies fk at a higher band side in the range of frequencies.
  • separation matrices W(fk) learned for frequencies fk excluding the frequencies fk at the lower side and the frequencies fk at the higher side among the plurality of frequencies fk selected by the frequency selector 54 are used to estimate the direction ⁇ 1(fk), and the direction ⁇ 2(fk) (thus to estimate the direction ⁇ 1 and the direction 62).
  • the direction estimator 72 estimates a direction ⁇ 1(fk) and a direction ⁇ 2(fk) from separation matrices W(fk) that the learning processing unit 44 has learned for frequencies fk that the frequency selector 54 has selected from frequencies f200 to f399 excluding the lower-band-side frequencies f1 to f199 and the higher-band-side frequencies f400 to f512.
  • the direction ⁇ 1 and the direction ⁇ 2 are accurately estimated, compared to when separation matrices W(fk) of all frequencies fk selected by the frequency selector 54 are used, since separation matrices W(fk) learned for frequencies fk excluding lower-band-side frequencies fk and higher-band-side frequencies fk are used to estimate the direction ⁇ 1 and the direction ⁇ 2. Accordingly, it is possible to generate separation matrices W(fk) which enable accurate sound source separation for unselected frequencies fk.
  • this embodiment may also employ a configuration in which either the lower-band-side frequencies fk and the higher-band-side frequencies fk are excluded to estimate the direction ⁇ 1(fk) and the direction ⁇ 2(fk).
  • a predetermined number of frequencies are selected using index values z(f1) to z(fK) (for example, the determinant z1(fk), the number of conditions z2(fk), the correlation z3(fk), the amount of mutual information z4(fk), the trace z5(fk), and the kurtosis z6(fk)) calculated for a single unit interval TU.
  • index values z(f1) to z(fK) of a plurality of unit intervals TU are used to select frequencies fk in one unit interval TU.
  • FIG. 14 is a block diagram of a frequency selector 54 in a separation matrix generator 40 of the ninth embodiment.
  • the frequency selector 54 includes a selector 541 and a selector 542.
  • Index values z(f1) to z(fK) that the index calculator 52 calculates from observed data D(f1) to D(fK) are provided to the selector 541 for each unit interval TU.
  • the index value z(fk) is a numerical value (for example, any of the determinant z1(fk), the number of conditions z2(fk), the correlation z3(fk), the amount of mutual information z4(fk), the trace z5(fk), and the kurtosis z6(fk)) that is used as a measure of the significance of learning of separation matrices W(fk) using observed data D(fk).
  • the selector 541 sequentially determines whether or not to select each of the K frequencies fl to fK according to the index values z(f1) to z(fK) of each unit interval TU. Specifically, for each unit interval TU, the selector 541 sequentially generates a series y(T) of K numerical values sA_l to sA_K representing whether or not to select each of the K frequencies f1 to fK. In the following, the series of numerical values will be referred to as a "numerical value sequence".
  • the numerical value sA_k of the numerical value sequence y(T) is set to different values when it is determined according to the index value z(fk) that the frequency fk is selected and when it is determined that the frequency fk is not selected. For example, the numerical value sA_k is set to "1" when the frequency fk is selected and is set to "0" when the frequency fk is not selected.
  • the selector 542 selects a plurality of frequencies fk from the results of determination that the selector 541 has made for a plurality of unit intervals TU (J+1 unit intervals TU).
  • the selector 542 includes a calculator 56 and a determinator 57.
  • the calculator 56 calculates a coefficient sequence Y(T) according to coefficient sequences y(T) to y(T-J) of J+1 unit intervals TU that are a unit interval TU of number T and J previous unit intervals TU.
  • the coefficient sequence Y(T) corresponds to, for example, a weighted sum of coefficient sequences y(T) to y(T-J) as defined by the following Equation (12).
  • the coefficient sequence Y(T) is a series of K numerical values sB_l to sB_K.
  • the numerical values sB_k are weights of the respective numerical values sA_k of coefficient sequences y(T) to y(T-J).
  • the numerical value sB_k of the coefficient sequence Y(T) corresponds to an index of the number of times the selector 541 has selected the frequency fk in J+1 unit intervals TU. That is, the numerical value sB_k of the coefficient sequence Y(T) increases as the number of times the selector 541 has selected the frequency fk in J+1 unit intervals TU increases.
  • the determinator 57 selects a predetermined number of frequencies fk using the coefficient sequence Y(T) calculated by the calculator 56. Specifically, the determinator 57 selects a predetermined number of frequencies fk corresponding to numerical values sB_k, which are located at higher positions among the K numerical values sB_l to sB_K of the coefficient sequence Y(T) when they are arranged in descending order. That is, the determinator 57 selects frequencies fk that the selector 541 has selected a large number of times in J+1 unit intervals TU. The selection of frequencies fk by the determinator 57 is performed sequentially for each unit interval TU.
  • the learning processing unit 44 generates separation matrices W(fk) by performing learning upon the initial separation matrix W0(fk) using the observed data D(fk) of each frequency fk that the determinator 57 has selected from the K frequencies f1 to fK.
  • a configuration in which the initial separation matrix W0(fk) is used as the separation matrix W(fk) (the first embodiment) or a configuration in which a separation matrix W(fk) that the matrix supplementation unit 74 generates from the learned separation matrix W(fk) is used (the seventh embodiment or the eighth embodiment) may be employed for unselected frequencies (i.e., for frequencies not selected by the determinator 57).
  • the results of determination of selection/unselection of frequencies fk is stable (or reliable) (i.e., the frequency of change of the determination results is low) even when the observed data D(fk) has suddenly changed, for example, due to noise since whether or not to select frequencies fk of each unit interval TU is determined taking into consideration the overall results of determination of selection/unselection of frequencies fk of a plurality of unit intervals TU (J+1 unit intervals TU). Accordingly, the ninth embodiment has an advantage in that it is possible to generate a separation matrix w(fk) which can accurately separate a desired sound.
  • FIG. 15 is a diagram illustrating measurement results of the Noise Reduction Rate (NRR).
  • NRRs of a configuration for example, the first embodiment in which frequencies fk that are targets of learning are selected from index values z(fk) of only one unit interval TU are illustrated as an example for comparison with the ninth embodiment.
  • NRRs were measured for angles ⁇ 2 (-90°, -45°, 45°, and 90°) of the sound source S2 obtained by sequentially changing the direction ⁇ 2 in intervals of 45°, starting from - 90°, with the direction ⁇ 1 of the sound source S1 fixed to 0°. It can be understood from FIG.
  • the configuration in which whether or not to select frequencies fk of each unit interval TU is determined taking into consideration the determination of selection/unselection of frequencies fk in a plurality of unit intervals TU (50 unit intervals TU in FIG. 15 ), increases the NRR (i.e., increases the accuracy of sound source separation).
  • this embodiment may also employ a configuration in which, for each of the K frequencies f1 to fK, the number of times the frequency is selected in J+1 unit intervals TU is counted and a predetermined number of frequencies fk which are selected a large number of times are selected as learning targets (i.e., a configuration in which a weighted sum of coefficient sequences y(T) to y(T-J) is not calculated).
  • this embodiment may also preferably employ a configuration in which the coefficient sequence Y(T) is calculated by simple summation of the coefficient sequences y(T) to y(T-J).
  • the configuration in which the weighted sum of the coefficient sequences y(T) to y(T-J) is calculated it is possible to determine whether or not to select frequencies fk, preferentially taking into consideration the results of determination of selection/unselection of frequencies fk in a specific unit interval TU among the J+1 unit intervals TU.
  • the method for selecting weights ⁇ 0 to ⁇ J is arbitrary.
  • a Delay-Sum (DS) type beam-former which emphasizes a sound arriving from a specific direction is applied to each processing unit Pk (the filter 32 and the filter 34) in each of the above embodiments
  • a blind control type (null) beam-former which suppresses a sound arriving from a specific direction (i.e., which forms a blind zone for sound reception) may also be applied to each processing unit pk.
  • the blind control type beam-former is implemented by changing the adder 325 of the filter 32 and the adder 345 of the filter 34 of the processing unit pk to subtractors.
  • the separation matrix generator 40 determines the coefficients (w11(fk) and w21(fk)) of the filter 32 so that a blind zone is formed in the direction ⁇ 1 and determines the coefficients (w12(fk) and w22(fk)) of the filter 34 so that a blind zone is formed in the direction ⁇ 2. Accordingly, the sound SV1 of the sound source S1 is suppressed (i.e., the sound SV2 is emphasized) in the separated signal U1 and the sound SV2 of the sound source S2 is suppressed (i.e., the sound SV1 is emphasized) in the separated signal U2.
  • the frequency analyzer 22, the signal processing unit 24, and the signal synthesizer 26 may be omitted from the signal processing device 100.
  • the invention may also be realized using a signal processing device 100 that includes a storage unit 14 that stores observed data D(fk) and a separation matrix generator 40 that generates separation matrices W(fk) from the observed data D(fk).
  • a separated signal U1 and a separated signal U2 are generated by providing the separation matrices w(fk) (W(f1) to W(fK)) generated by the separation matrix generator 40 to a signal processing unit 24 in a device separated from the signal processing device 100.
  • the initial value generator 42 may also employ a configuration in which a predetermined initial separation matrix W0 is commonly applied as an initial value for learning of the separation matrices W(f1) to W(fK) by the learning processing unit 44.
  • the configuration in which the initial separation matrix W0(fk) is generated from observed data D(fk) is not essential in the invention.
  • the invention may also employ a configuration in which initial separation matrices W0(f1) to W0(fK) which are previously generated and stored in the storage unit 14 are used as initial values for learning of the separation matrices W(f1) to W(fK) by the learning processing unit 44.
  • the initial value generator 42 may generate an initial separation matrix W0(fk) only for each frequency fk that the frequency selector 54 has selected from the K frequencies f1 to fK.
  • the index values (i.e., the determinant z1(fk), the number of conditions z2(fk), the correlation z3(fk), the amount of mutual information z4(fk), the trace z5(fk), and the kurtosis z6(fk))) which are each used as a reference for selection of frequencies fk in each of the above embodiments are merely examples of a measure (or indicator) of the significance of learning of the separation matrices w(fk) using the observed data D(fk) of the frequencies fk.
  • a configuration in which index values different from the above examples are used as a reference for selection of frequencies fk is also included in the scope of the invention.
  • a combination of two or more index values arbitrarily selected from the above examples may also be preferably used as a reference for selection of frequencies fk.
  • the invention may employ a configuration in which frequencies fk at which a weighted sum of the determinant z1 and the trace z5 is great are selected or a configuration in which frequencies fk at which a weighted sum of the reciprocal of the determinant z1 and the kurtosis z6 is small are selected. In both of these configurations, frequencies fk with high learning effect are selected.
  • the invention may employ not only the method of the first embodiment in which singular value decomposition of the covariance matrix Rxx(fk) is used but also a method in which the variance ⁇ 1 2 of the magnitude x1(r, fk) of the observed signal V1, the variance ⁇ 2 2 of the magnitude x2(r, fk) of the observed signal V2, and the correlation z3(fk) of Equation (8) are substituted into the following Equation (13).
  • the invention is also applicable to the case of separation of a sound from three or more sound sources S.
  • n or more sound receiving devices M are required when the number of sound sources S; which are targets of sound source separation, is n.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
EP09014232.4A 2008-11-14 2009-11-13 Tonverarbeitungsvorrichtung Not-in-force EP2187389B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2008292169A JP5277887B2 (ja) 2008-11-14 2008-11-14 信号処理装置およびプログラム

Publications (3)

Publication Number Publication Date
EP2187389A2 true EP2187389A2 (de) 2010-05-19
EP2187389A3 EP2187389A3 (de) 2014-03-26
EP2187389B1 EP2187389B1 (de) 2016-10-19

Family

ID=41622008

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09014232.4A Not-in-force EP2187389B1 (de) 2008-11-14 2009-11-13 Tonverarbeitungsvorrichtung

Country Status (3)

Country Link
US (1) US9123348B2 (de)
EP (1) EP2187389B1 (de)
JP (1) JP5277887B2 (de)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015070918A1 (en) * 2013-11-15 2015-05-21 Huawei Technologies Co., Ltd. Apparatus and method for improving a perception of a sound signal

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6303385B2 (ja) * 2013-10-16 2018-04-04 ヤマハ株式会社 収音解析装置および収音解析方法
CN105898667A (zh) 2014-12-22 2016-08-24 杜比实验室特许公司 从音频内容基于投影提取音频对象
CN105989852A (zh) 2015-02-16 2016-10-05 杜比实验室特许公司 分离音频源
CN108701468B (zh) * 2016-02-16 2023-06-02 日本电信电话株式会社 掩码估计装置、掩码估计方法以及记录介质
EP3324407A1 (de) 2016-11-17 2018-05-23 Fraunhofer Gesellschaft zur Förderung der Angewand Vorrichtung und verfahren zur dekomposition eines audiosignals unter verwendung eines verhältnisses als eine eigenschaftscharakteristik
EP3324406A1 (de) * 2016-11-17 2018-05-23 Fraunhofer Gesellschaft zur Förderung der Angewand Vorrichtung und verfahren zur zerlegung eines audiosignals mithilfe eines variablen schwellenwerts
EP3742185B1 (de) * 2019-05-20 2023-08-09 Nokia Technologies Oy Vorrichtung und zugehörige verfahren zur erfassung von raumklang
CN115280413A (zh) * 2020-02-28 2022-11-01 东京都公立大学法人 音源分离程序、音源分离方法以及音源分离装置
AU2021341939A1 (en) * 2020-09-09 2023-03-23 Dolby International Ab Processing parametrically coded audio
CN117597733A (zh) * 2021-06-30 2024-02-23 西北工业大学 使用深度神经网络从单输入生成高清晰度双耳语音信号的系统和方法
US12380871B2 (en) 2022-01-21 2025-08-05 Band Industries Holding SAL System, apparatus, and method for recording sound

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006084898A (ja) 2004-09-17 2006-03-30 Nissan Motor Co Ltd 音声入力装置

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1999046659A2 (en) * 1998-03-10 1999-09-16 Management Dynamics, Inc. Statistical comparator interface
US20010044719A1 (en) * 1999-07-02 2001-11-22 Mitsubishi Electric Research Laboratories, Inc. Method and system for recognizing, indexing, and searching acoustic signals
JP3887192B2 (ja) * 2001-09-14 2007-02-28 日本電信電話株式会社 独立成分分析方法及び装置並びに独立成分分析プログラム及びそのプログラムを記録した記録媒体
EP1473964A3 (de) * 2003-05-02 2006-08-09 Samsung Electronics Co., Ltd. Mikrofonvorrichtung, Verfahren zur Verarbeitung von Signalen von dieser Mikrofonvorrichtung und dieses benutzende Spracherkennungsverfahren und Spracherkennungssystem
US7496482B2 (en) * 2003-09-02 2009-02-24 Nippon Telegraph And Telephone Corporation Signal separation method, signal separation device and recording medium
US20060031067A1 (en) * 2004-08-05 2006-02-09 Nissan Motor Co., Ltd. Sound input device
JP4529611B2 (ja) * 2004-09-17 2010-08-25 日産自動車株式会社 音声入力装置
JP4896449B2 (ja) * 2005-06-29 2012-03-14 株式会社東芝 音響信号処理方法、装置及びプログラム
JP2007034184A (ja) * 2005-07-29 2007-02-08 Kobe Steel Ltd 音源分離装置，音源分離プログラム及び音源分離方法
US20070083365A1 (en) * 2005-10-06 2007-04-12 Dts, Inc. Neural network classifier for separating audio sources from a monophonic audio signal
JP2007156300A (ja) * 2005-12-08 2007-06-21 Kobe Steel Ltd 音源分離装置、音源分離プログラム及び音源分離方法
US20070133819A1 (en) * 2005-12-12 2007-06-14 Laurent Benaroya Method for establishing the separation signals relating to sources based on a signal from the mix of those signals
JP4556875B2 (ja) * 2006-01-18 2010-10-06 ソニー株式会社 音声信号分離装置及び方法
JP4920270B2 (ja) * 2006-03-06 2012-04-18 Ｋｄｄｉ株式会社 信号到来方向推定装置及び方法、並びに信号分離装置及び方法、コンピュータプログラム
JP2007282177A (ja) * 2006-03-17 2007-10-25 Kobe Steel Ltd 音源分離装置、音源分離プログラム及び音源分離方法
JP4672611B2 (ja) * 2006-07-28 2011-04-20 株式会社神戸製鋼所 音源分離装置、音源分離方法及び音源分離プログラム
US20080228470A1 (en) * 2007-02-21 2008-09-18 Atsuo Hiroe Signal separating device, signal separating method, and computer program
US20080212666A1 (en) * 2007-03-01 2008-09-04 Nokia Corporation Interference rejection in radio receiver
US8660841B2 (en) * 2007-04-06 2014-02-25 Technion Research & Development Foundation Limited Method and apparatus for the use of cross modal association to isolate individual media sources
US8126829B2 (en) * 2007-06-28 2012-02-28 Microsoft Corporation Source segmentation using Q-clustering
US20100324708A1 (en) * 2007-11-27 2010-12-23 Nokia Corporation encoder
US8144896B2 (en) * 2008-02-22 2012-03-27 Microsoft Corporation Speech separation with microphone arrays
JP5195652B2 (ja) * 2008-06-11 2013-05-08 ソニー株式会社 信号処理装置、および信号処理方法、並びにプログラム

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006084898A (ja) 2004-09-17 2006-03-30 Nissan Motor Co Ltd 音声入力装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
H. SARUWATARI: "Blind Source Separation Combining Independent Component Analysis and Beam-Forming", URASIP JOURNAL ON APPLIED SIGNAL PROCESSING, vol. 2003, no. 11, 2003, pages 1135 - 1146
K. TACHIBANA ET AL.: "Efficient Blind Source Separation Combining Closed-Form Second-Order ICA and Non-Closed-Form Higher-Order ICA", INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, vol. 1, April 2007 (2007-04-01), pages 45 - 48

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015070918A1 (en) * 2013-11-15 2015-05-21 Huawei Technologies Co., Ltd. Apparatus and method for improving a perception of a sound signal

Also Published As

Publication number Publication date
US9123348B2 (en) 2015-09-01
EP2187389B1 (de) 2016-10-19
JP2010117653A (ja) 2010-05-27
JP5277887B2 (ja) 2013-08-28
EP2187389A3 (de) 2014-03-26
US20100125352A1 (en) 2010-05-20

Similar Documents

Publication Publication Date Title
EP2187389B1 (de) Tonverarbeitungsvorrichtung
KR100486736B1 (ko) 두개의 센서를 이용한 목적원별 신호 분리방법 및 장치
US8693287B2 (en) Sound direction estimation apparatus and sound direction estimation method
EP2786593B1 (de) Vorrichtung und verfahren zur mikrofonpositionierung basierend auf räumlicher leistungsdichte
US7720679B2 (en) Speech recognition apparatus, speech recognition apparatus and program thereof
EP3370232B1 (de) Schallquellensondierungsvorrichtung, schallquellensondierungsverfahren und speichermedium zur speicherung eines programms dafür
EP2254113A1 (de) Geräuschunterdrückungsvorrichtung und -programm
KR102236471B1 (ko) 재귀적 최소 제곱 기법을 이용한 온라인 cgmm에 기반한 방향 벡터 추정을 이용한 음원 방향 추정 방법
US20080228470A1 (en) Signal separating device, signal separating method, and computer program
CN113687305B (zh) 声源方位的定位方法、装置、设备及计算机可读存储介质
EP2544180A1 (de) Tonverarbeitungsvorrichtung
EP2884491A1 (de) Extraktion von Wiederhall-Tonsignalen mittels Mikrofonanordnungen
JP5516169B2 (ja) 音響処理装置およびプログラム
CN110709929A (zh) 处理声音数据以分离多声道信号中的声源
EP3113508A1 (de) Signalverarbeitungsvorrichtung, -verfahren und -programm
JP5387442B2 (ja) 信号処理装置
US20190189114A1 (en) Method for beamforming by using maximum likelihood estimation for a speech recognition apparatus
US10657958B2 (en) Online target-speech extraction method for robust automatic speech recognition
US20130311183A1 (en) Voiced sound interval detection device, voiced sound interval detection method and voiced sound interval detection program
JP5233772B2 (ja) 信号処理装置およびプログラム
JP7014682B2 (ja) 音源分離の評価装置および音源分離装置
JP4422662B2 (ja) 音源位置・受音位置推定方法、その装置、そのプログラム、およびその記録媒体
JP5263020B2 (ja) 信号処理装置
JP4095348B2 (ja) 雑音除去システムおよびプログラム
US7885421B2 (en) Method and system for noise measurement with combinable subroutines for the measurement, identification and removal of sinusoidal interference signals in a noise signal

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA RS

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA RS

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 21/02 20130101AFI20140220BHEP

17P Request for examination filed

Effective date: 20140924

RBV Designated contracting states (corrected)

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

17Q First examination report despatched

Effective date: 20150122

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: G10L 21/0272 20130101ALI20160331BHEP

Ipc: G10L 21/02 20130101AFI20160331BHEP

Ipc: H04R 3/00 20060101ALN20160331BHEP

INTG Intention to grant announced

Effective date: 20160425

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 838935

Country of ref document: AT

Kind code of ref document: T

Effective date: 20161115

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602009041785

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20161019

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161130

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 838935

Country of ref document: AT

Kind code of ref document: T

Effective date: 20161019

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170120

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170119

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170219

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170220

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602009041785

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161130

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161130

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20170731

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170119

26N No opposition filed

Effective date: 20170720

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161219

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161113

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20171108

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20171108

Year of fee payment: 9

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20091113

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161019

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20161113

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602009041785

Country of ref document: DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20181113

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190601

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181113