RS49859B - SYSTEM AND PROCEDURE FOR LOCATION OF SPEAKERS BY MICROPHONE - Google Patents

Abstract

A system for locating a speaker by means of a microphone array, characterized in that it comprises: a microphone array of M microphones with respect to whose symmetry determines the azimuth angle or position of the speaker in a horizontal plane; block for preprocessing of microphone signals and conversion to digital form and frequency domain; block for cross-correlation analysis of microphone signals and its optimization based on phase transformation; a block for determining a filter function based on the prosodic characteristics of the speech signal, which optimizes the cross-correlation PHAT analysis; a speech-based activity detection unit (VAD) based on a superdirectional router (SD-BF) that provides spatial filtering for speakers; block for azimuth angle estimation based on maximum of interpolated cross-correlation functions.

Description

TECHNICAL FIELD TO WHICH THE INVENTION RELETS

The invention belongs to the field of acoustic signal processing, or more precisely, methods of locating speakers using a microphone array in an acoustic environment with noise and reverberation.

TECHNICAL PROBLEM

Localization of speakers in space is a very important technical problem in systems based on human-human or human-machine voice communication. It arises as a need for communication to be as intelligible as possible despite the many disturbances that may appear in space and that mask the speech signal. In systems such as teleconference systems or speakerphones in a room or in a car, in addition to intelligibility, the quality of communication is also of primary importance. The same attributes of speech communication are important in communication, for example, between a human and a robot, where the robot must correctly recognize a spoken command. A slightly different problem appears with the control of the video camera, where the camera should be directed towards the current speaker, abstracting other sound sources in the given environment. Therefore, the problem is not the intelligibility of speech, but the separation of the speech signal and other acoustic signals.

The given examples of speech communication in a space, or room, define the basic problem in the form of localization of the current speaker, that is, directing the microphone system towards him. A microphone system can be a directional microphone or multiple microphones in a suitable physical arrangement. Since in the acoustic environment, next to the source of the useful signal, there are disturbances of very different origins, whose sources can be arbitrarily distributed in space, the microphone system must have a directional sensitivity characteristic and must be directed towards the desired signal source, i.e. current speaker. In other words, the microphone system must locate the speaker in the horizontal plane and determine the azimuth angle relative to its coordinates in space.

A technical problem arises when a large number of interference sources appear in the environment, when these interferences are non-stationary, when the interference sources move in space or when the current speaker moves in space. The procedure for determining the azimuth angle of the speaker must solve three basic problems: (1) detection of the speaking activity of the current speaker, while keeping in mind that a larger number of speakers may appear in the observed space, (2) separation of the current speaker in relation to all other sources of interference, which implies suppression of the interference signal and emphasis of the useful signal, and (3) adaptive tracking of the current speaker in motion, where it must be taken into account that other sound sources may also be moving. This third problem concerns the correct orientation of the system (robot, camera) towards the current speaker.

Additional problems arise with speaker localization in a room with pronounced reverberation. The signals of reflections from the walls or objects in the room can be, depending on the position of the speaker, the source of interference and the microphone system, significantly stronger than the direct sound wave of the current speaker.

It can be seen from the above that the technical problems in the solution of speaker localization in space are very complex and require a complex approach in the optimization of the solution, especially when considering the operation of the system in real time based on a commercial digital signal processor (DSP) platform.

STATE OF THE ART

Locating speakers in the presence of acoustic disturbances and room reverberation is a complex problem. In conditions when the spectrum of the useful speech signal overlaps with the spectrum of the present disturbances, locating the speaker can be solved in a reliable way by using a microphone array and appropriate signal processing that takes into account the specifics of the application conditions of the locating system. Theoretical foundations in the application of microphone arrays for speaker localization are given in M.S. Brandstein, D.B. Ward (Eds.), Microphone Arrays: Signal Processing Techniques and Applications, Springer, Berlin 2001; and in Y. Huang, J. Benestv, Audio signal processing for next generation multimedia communication systems, Kluwer Academic Publishers Publ., 2004.

There are a number of patented solutions based on microphone arrays such as: U.S. published patent application 2003/0051532 Al, filed on August 15, 2002, with the title "Robust talker localization in reverberant environment", provides a solution for speaker localization in a reverberant room based on a circular configuration microphone system and on the basis of direct sound wave energy detection; then the U.S. patent 6,970,796 B2, filed on March 1, 2004, with the title "System and method for improving the precision of localization estimates", provides a solution that, in addition to conventional DO A determination with a microphone array, has a post-processing system based on statistical clustering of initial location estimates and obtaining a final location estimate with increased precision and reliability; then the U.S. patent 6,999,593 B2, filed May 28, 2003, entitled "System and process for robust sound source localization", provides a solution for microphone array-based speaker localization by combining weighted cross-correlation and tuned directional characteristics of microphone array pairs; then the U.S. published patent application 2005/0080619 Al, filed on October 13, 2004, entitled "Method and apparatus for robust speaker localization and automatic camera steering system employing the same", provides a solution that determines speaker localization using MUSIC technology.

In general, the methods of estimating the direction of locating the speaker (sound source) can be divided into three basic groups: methods based on the superdirective characteristic of the directionality of the microphone array, methods based on the complex estimation of the spectrum of microphone signals and methods based on the time delay of sound waves to the microphone array TDOA{Time Delay of Arrival).The methods from the first and second groups are sensitive to the frequency characteristics of all sound sources in the analyzed room and without prior knowledge do not provide satisfactory accuracy. TDOA methods are most often used in practice (P. Julian et al., A comparative study of sound localization algorithms for energy aware sensor network nodes, ZE££ Trans. Circuits and Systems, Vol. 51, No. 4, pp. 640-648, Apr. 2004.). In the conventional procedure, in the first step TDOA is estimated for each pair of microphones in the microphone array. The estimation is based on cross-correlation analysis, which in the second step is weighted by the PHAT (Phase Trans/orni) weighting function, which increases the robustness of the algorithm for estimating incoming directions for the presence of noise and reverberation in the room. However, room reverberation, the presence of multiple sound sources and noise are still a major problem for these speaker localization methods.

DISCLOSURE OF THE ESSENCE OF THE INVENTION

The subject of this invention is a system and procedure for locating a speaker using a microphone array in a complex acoustic environment which, in addition to the actual speaker, contains many interference signals such as: ambient noise, sources of acoustic interference, room reverberation and other speakers. As such, the system can find wide application in speech communication systems as well as in voice control and management systems.

The system, which is the subject of the invention, contains M microphones arranged in a linear structure and at equal distances, a block for preprocessing and digitization of microphone signals and a block for digital processing of microphone signals. The system is placed in the horizontal plane and determines the azimuth angle of the speaker in relation to the bisector of the system. The system can be connected to, or be part of, a conferencing system, or a management or control system, such as a robot or video camera.

The essence of the invention lies in the specific processing of the speech signal that is recorded in the acoustic environment of the room where the system and the speaker are located. The microphone array records all the signals in the room: the useful signal as a direct wave arriving from the speaker to the microphone and interference signals that can be varied. Disturbance signals include direct waves from one or more noise sources or sources of other disturbances that can be found in the room and all reflected waves (room echoes) originating from all sound sources, including the actual speaker, which are caused by room reverberation. It should be emphasized that the sources of sounds in the room can be stationary or non-stationary, which is the most common case, both according to their characteristics and their location in the room (moving sources of sounds).

Microphone signals from the microphone array are processed in digital form in the frequency domain. This domain provides certain advantages in terms of processing speed and the number of computational operations, which is very important for the realization of real-time systems.

A specific aspect of the invention is in the optimization of the cross-correlation analysis of microphone signals through two aspects: first, by generalizing the cross-correlation, which is referred to in the literature as phase transformation PHAT (Phase Transform), which implies the normalization of the cross-correlation to its modulus when the information about the signal strength is lost, and only the information about the phase that contains the relative time delay of the signal remains, and second, by weighting the PHAT transformation with a filter function W(n) that contains the basic prosodic characteristics of the speech signal, before above all, the energetic dynamics of the vocal formant structures.

The next specificity of the invention is the determination of the filter function W(n) based on the analysis of microphone signals in three domains: energy, frequency and time. The goal of this analysis is that the generalized cross-correlation analysis takes place under the control of the prosodic characteristics of the speech signal, which represents in a specific way the separation of the speech signal in relation to other signals of ambient disturbances and reverberation, and which ultimately provides a more reliable estimate of the speaker's location.

The specificity of the invention is the implementation of a speech activity detector (VAD) based on a superdirective router (SD-BF), which provides a higher directivity index of the microphone array and thus more efficient spatial filtering of the speech signal in relation to ambient interference.

Inventiveness in this invention is found in the way of realization of each of the mentioned specificities, but also in the process of integrating all algorithms into a unique unit that functions stably and with quality. Algorithmic procedures are optimized by using common resources, especially if one considers the realization in the spectral and multidimensional domain (multimicrophone system).

These and other aspects, specificities and benefits of the present invention will be more apparent upon review of the detailed description of the invention, patent claims and accompanying drawings.

BRIEF DESCRIPTION OF THE IMAGES AND DRAWINGS

Figure 1 - shows the ambient conditions of application of the system for locating speakers using a microphone array.

Figure 2- shows the basic block diagram of the system for locating speakers.

Figure 3 - shows the block diagram of the filter function estimation subsystem

W(n).

Figure 4- shows the block diagram of the azimuth angle estimation subsystem0.

Figure 5- shows a block diagram of the VAD subsystem for detecting the speech activity of the current speaker.

DETAILED DESCRIPTION OF THE INVENTION

This invention describes a system and method for localizing a speaker using a microphone array in an acoustic environment such as a room, with stationary and/or non-stationary disturbances present.

Figure 1 shows the ambient conditions in which the system, which is the subject of the present invention, can be found. Namely, in the room 100 there is the current speaker 101 in the horizontal plane in the direction 102 at an angle 9 in relation to the bisector of the microphone array 103. The microphone array contains M microphones that transmit the recorded signals to the block 104 where the signal is processed in order to determine the estimation of the azimuth angle 6. Information about the estimated azimuth angle can be used, for example, to control a robot 105 or a video camera 106, or for communication needs such as voice communication over the Internet107 or through a teleconference system108. In the second case, the azimuth angle is controlled by the directionality characteristic of the microphone array 103, which is directed towards the current speaker.

The main problem in the estimation of the azimuth angle is the interference in the room that directly affects the accuracy and precision of the estimation. The main source of disturbance can be a source of noise, speech, music, etc., 109, with a direct sound wave110, but also reflected sound waves on the walls of the room, such as wave 110a. If the microphone array 103 is used for communication purposes, cases 107 and 108, in the so-called "hands-free" communication, then a very serious disturbance appears in the form of an acoustic echo111, which has its own acoustic reflection elements.

Therefore, the accuracy and precision of determining the azimuth angle largely depends on the ambient conditions in which the system, which is the subject of this patent, is used. An additional problem arises if the current speaker or the sources of interference move in the room, which requires adaptive tracking of the position of the current speaker.

Figure 2 shows a block diagram of a speaker localization system using a microphone array. Signals from the microphone and the microphone array 103 enter the block 201 in which their pre-processing is performed, i.e. amplification, filtering, digitization and conversion into the frequency domain using discrete Fourier transformation (DFT). Preprocessing is performed at the level of segments of length N measurements, with an overlap of 50% and with an applied Hamming window and FFT of order N.

The output of block 201 are Fourier transformations X/doXM. Cross-correlation analysis of the first microphone with all other microphones is performed on these signals. At the output of block 201, estimates of the cross-correlation between the signal and all other microphones are obtained, G\ j( n)doGi,m(") by recursive averaging according to the relation:

Constants a+ and a. are chosen to satisfy the inequality 0.5 < a+ < a. < 1 and under that condition the influence of members Xi( t, f) Xk'( t, f) with a larger modulus is favored. Signals Gu(/i) doGiM( n) enter blocks 203 and 205.

In block 203 labeled PHA, generalized cross-correlation, often labeled as phase transformation in the literature, is implemented. Namely, by normalizing the cross-correlation to its modulus, the information about the signal strength is lost, and only the information about the phase in which the relative time delay of the signal is contained remains.

The filter function W(n) generated in block 204 takes part in the processing of generalized cross-correlation functions G12 P. The function W(n) is obtained by processing microphone signals X\doXu, which will be described in more detail later, and whose goal is to use the basic prosodic characteristics of the speech signal, primarily the energy dynamics of the vocal formant structures, for a more reliable assessment of the azimuth angle, i.e. the location of the speaker in the room.

In block 205, the estimation of the azimuth angle is determined based on the maximum of the generalized cross-correlation functions. The validity of the given estimate is controlled by block 206, labeled VAD, which detects the activity of the current speaker, and when the speaker is active, the current estimate of the azimuth angle is valid, otherwise the estimate obtained during his last activity is adopted.

Figure 3 shows the block diagram of the subsystem for determining the filter function W{n). The post-speech signal has a formant structure, which is why all frequency bins do not have the same power, it is necessary to select the bins with the highest power and use them to determine the cross-correlation function. To this end, block 301 calculates the mean power of microphone signals XjdoXupo each DFT bin within the block, ie. current channel strengths according to the relationship:

In block 302, the weighting function W(n) is determined, which favors the bins where there is an increase in the current signal strength. The reason for choosing such a solution is that the part of the signal with a sudden increase in power has a larger share of direct waves than the part with a drop in power, where wave reflections dominate, i.e. room reverberation. This approach is realized by the relation:

In block 303, further processing of channel current powers is performed by smoothing (smoothing, English) powerP(n) by frequency, powerP(n), and then averaging by time, powerP(ri). Smoothing of powerP(n) is performed by a non-causal IIR filter of the first order (zero phase delay is achieved by double forward and backward filtering), so that powerP(n) is obtained, while the averaging of this power by time is performed by nonlinear A first-order IIR filter with two averaging coefficients, one for increasing and the other for decreasing signal power. This nonlinear filter is described by the relations:

SizeP(n) is used to define a decision threshold for extracting bins with the highest power in block 304. The procedure consists of comparing sizeP(n) and cross-correlation functions Cn,2(«) to Gi,m(h) with a binary decision at the output for each bin. This means that at the output of block 304, M-l binary strings of length N are obtained.

By multiplying the binary output from block 304 and the weighting function W(n) from block 302, a filter function W{n) is obtained at the output of block 305, which weights the bins of the phase transformation Gl kPha,(n) in block 203, Figure 2. The phase-transformed cross-correlation functions are additionally filtered with an IIR filter in time in order to reduce the variance of the estimation of the correlation functions. This is described by the relation:

Figure 4 shows a detailed block diagram of block 205 from Figure 2, in which the estimation of the azimuth angle is determined9. Phase-transformed cross-correlation functions

G, kPhats in block 401 are transformed from the frequency to the time domain into cross-correlations /?i,2(x) doR], m(x) using the inverse Fourier transform (IFFT). Before the IFFT transformation, a priori rejection of bins that are outside the range of interest is applied in block 402. The criterion for this rejection is the selection of a frequency range for which the power of the speech signal is high enough without aliasing occurring in the spatial domain for the highest frequency of the range.

In block 403, cross-correlation functions R\ j.(t) to Ri, m(x) are time-matched by applying appropriate interpolation factors, which are then averaged and at their mean valueR, r(T), the maximum is determined in block 404, whose abscissa represents the estimate of the time delay f.

In block 405, the time delay and incident angle 6R of the direct wave of the active speaker are recalculated. Incoming direction estimation makes sense when the speaker is active; when he is not active, the estimate obtained during his last activity is used for a valid estimate. For this purpose, the validity of the estimation 0R is definitively determined under the control of the VAD signal in block 406, so that the final value of the estimated azimuth angle 9 is obtained at the output.

In order to detect speaker activity, the following are used: a) information from block 301 about the mean power of microphone signals P(n), Figure 3, and b) information from Phys block 501, block SD-BF superdirective router, Figure 5. Based on this information, a decision is made in block 502 about the activity of a nearby speaker.

The formation of the superdirective spatial filter is performed in block 501. It provides a higher directivity index compared to a conventional spatial filter that contains only delay compensation and summation.

For a room with reverberation, a diffuse noise field model is usually adopted, which implies that the noise comes from all directions with approximately the same intensity. For such a noise field model, it is shown that the coherence between two microphones is a real number equal to:

where /u is the frequency, dtj is the microphone distance and ij, and c is the sound speed. Coherences of microphone pairs (f) form the coherence matrix Td. Using the coherence matrix Td defined in this way, the coefficients of the superdirective microphone array are determined in block 504 according to the relation: where Ce is the direction vector to the direction of the selected speaker defined by the estimated azimuth angle 9. This vector is determined in block 503 according to the relation:

Sized is the distance between two adjacent microphones.

At the output of block 501, an estimate of the actual speaker's speech is obtained based on the relation:

Therefore, two pieces of information arrive in block 502: the mean power of the microphone signals P(n), which, in addition to the current speaker, also contains all noise signals in the room, and the speech estimation signal b of the actual speaker in the direction of the estimated azimuth angle 9. In block 502, a binary decision is made about the activity of the current speaker based on a comparative analysis of the received information and the binary signal VAD decides on the output value of the azimuth angle 6, that is, the current estimate is sent to the output of the system incoming direction if the current speaker is active, otherwise the last valid direction estimate is passed.

This invention describes the process of processing acoustic and speech signals in order to locate the speaker in space in relation to the speaker location system designed on the basis of a microphone array. The described system can locate the speaker in a closed or open space, and the system can be applied in the control and management of robots, video cameras or processes that require interactive information about the location of the speaker, or in "hands-free" communication systems such as teleconference systems, video conference systems, speakerphones, etc.

The procedures and techniques for processing acoustic and speech signals in this invention are independent of the number of microphones in the array and are under the control of a number of parameters that enable the optimization of solutions for different applications.

The acoustic and speech signal processing methods and techniques of the present invention can be implemented in a variety of ways. For example, these techniques can be implemented in hardware, software, or a combination. A hardware implementation may use specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic circuits (PLDs or FPGAs), and other electronic circuits designed to perform the functions described in this invention.

The acoustic and speech signal processing methods and techniques of the present invention can also be implemented in software, so that program codes can be stored in memory units and executed by processors such as PCs, PDAs, DSPs, etc.

The details of the invention described herein allow any person skilled in the art to implement the generic principles of the invention in other systems without departing from the scope of the invention.

Claims (29)

1. A system for locating the speaker using a microphone array, characterized by the fact that it contains: a microphone array of M microphones in relation to whose bisector the angle is determined azimuth, that is, the position of the speaker in the horizontal plane; block for preprocessing of microphone signals and conversion into digital form i frequency domain; block for cross-correlation analysis of microphone signals and its optimization at based on phase transformation; block for determining the phdter function on the basis of prosodic characteristics speech signal, which is used to optimize the cross-correlation PHAT analysis; superdirective-based speech activity detection (VAD) block the router (SD-BF) which provides spatial filtering of speakers; block for estimation of the azimuth angle based on the interpolated maxima cross-correlation functions. 2. The system according to claim 1, characterized by the fact that it contains blocks that detect the speech activity of the current speaker, that perform the separation of the current speaker in relation to all other sources of interference and that perform adaptive monitoring of the current speaker in motion. 3. The system according to claim 1, characterized by the fact that it contains a microphone array of M microphones and that the number of microphones in the array is not a limiting factor. 4. The system according to claim 2, characterized by the fact that the microphone array is located in a horizontal plane and that the location of the speaker is determined by the azimuth angle in relation to the bisector of the microphone array. 5. The system according to claim 1, characterized by the fact that signal processing takes place in the frequency domain and that it can be realized in real time. 6. The system according to any of the previous requirements, characterized by the fact that it contains a block of cross-correlation analysis of microphone signals that determines the time delay of sound waves from the sound source to the microphone array (TDOA) and performs its optimization by cross-correlation analysis based on phase transformation (PHAT). 7. The system according to claim 6, characterized by the fact that it contains a block for determining the filter function based on the prosodic characteristics of the speech signal, which enables the optimization of cross-correlation PHAT analysis adapted to the characteristics of the speech signal. 8. The system according to claims 1 to 5, characterized by the fact that it contains a speech activity detection block (VAD) in conditions of ambient disturbances and reverberation, which decides on the final value of the azimuth angle. 9. The system according to claim 8, characterized by the fact that the basis of the speech activity detection block (VAD) is a superdirective router (SD-BF) that ensures the separation of the current speaker from other sound sources in the room based on spatial filtering. 10. The system according to claims 6 to 7, characterized by the fact that it contains a block for estimating the azimuth angle based on the maximum of the interpolated and averaged M-1 cross-correlation functions. 11. A system according to any of the previous requirements, characterized by the fact that it can be applied to control devices, systems or processes by voice. 12. The system according to any of the previous requirements, characterized by the fact that it can be applied in "hands-free" communication systems for free speech communication in order to improve the quality and intelligibility of communication in an acoustic environment. 13. A method for locating a speaker using a microphone array of characteristics, which includes: cross-correlation analysis that analyzes the time delay of sound waves from the sound source to the microphone array; generalization of cross-correlation analysis, i.e. its phase transformation (PHAT), with the application of adaptive weighting by the filter function W ( n); adaptive determination of the filter function W( ri) based on prosodic speech signal characteristic; adaptive speaker activity detection (VAD) based on superdirectivity router (SD-BF); interpolation of cross-correlation functions and determination of azimuth angle estimation. 14. The method according to claim 13, characterized by the fact that cross-correlation is performed between the first microphone signal and all other microphone signals, so that M-1 cross-correlation is performed. 15. The procedure according to claim 14, characterized by the fact that the generalization of the cross-correlation analysis is carried out by normalizing the cross-correlation to its modulus, whereby the information about the signal strength is lost, and only the information about the phase that contains the relative time delay between the analyzed signals remains. 16. The method according to claim 15, characterized by the fact that the phase-transformed cross-correlation functions are additionally adaptively filtered with an IIR filter in time in order to reduce the variances of the estimation of the correlation functions. 17. The method according to claim 16, characterized in that the additional filtering of the phase-transformed cross-correlation functions is performed by adaptive weighting of the phase transformation bins with the filter function W(ri). 18. The method according to claim 13, characterized by the fact that the filter function W{n) is determined based on the prosodic characteristics of the speech signal detected in the current strength of the microphone signals. 19. The procedure according to claim 18, characterized by the fact that bins with the highest power are selected by the filter function W(n) and used to determine the cross-correlation functions. 20. The method according to claim 18 and 19, characterized by the fact that in determining the filter function W(ri), the power trajectories of the microphone signals are calculated by averaging over frequency and over time. 21. The procedure according to claim 18 to 20, characterized by the fact that in the determination of the filter function W(n), the bins are favored where there is an increase in the current signal strength, due to the fact that the part of the signal with a sudden increase in power has a greater share of direct waves than the part with a drop in power, where wave reflections dominate, i.e. room reverberation. 22. The method according to claim 13, characterized by the fact that a decision is made in the speech activity detector (VAD) about the activity of a nearby speaker. 23. The method according to claim 13 and 22, characterized by the fact that the VAD is based on a superdirective router (SD-BF) which, by processing microphone signals, ensures the directional characteristic of the sensitivity of the microphone array. 24. The method according to claim 23, characterized by the fact that the superdirective router performs spatial filtering, which emphasizes the signal of the current speaker and suppresses signals of ambient interference. 25. The method according to claims 23 and 24, characterized in that the directionality characteristic of the superdirective beam (SD-BF) is controlled by the estimated azimuth angle9. 26. The method according to claim 13, characterized by the fact that the estimation of the azimuth angle is determined based on the maximum of the matched and averaged M-1 cross-correlation functions. 27. The method according to claim 26, characterized in that the matching of cross-correlation functions is performed by an interpolation method. 28. The procedure according to claims 13, 26 and 27, characterized by the fact that the estimation of the azimuth angle is obtained by recalculating the time delay f at which the maximum of the matched and averaged M-l cross-correlation functions is located in the angle of incidence of the direct wave of the current speaker. 29. The procedure according to claims 13 and 28, characterized by the fact that the azimuth angle6 is determined based on the activity of the current speaker, so the current estimate of the incoming direction9 is sent to the system output in the case of the activity of the current speaker, otherwise when it is not active, the last valid estimate of the azimuth6 is sent.