RS49875B - SYSTEM AND PROCEDURE FOR FREE SPEECH COMMUNICATION WITH A MICROPHONE STRIP - Google Patents

Abstract

Microphone free voice communication system comprising a digital TV receiver for audio and video communication in full duplex, characterized in that the digital TV receiver (100) has stereo audio playback (102) for reproducing stereo TV programs and mono incoming voice signals in video telephony communication, having a built-in motion video camera (l04) for recording speakers in a room and reproducing, at a portion of its screen, an image of an interlocutor from the far end (105); comprising a microphone system (103) built into the TV receiver (100) for the purpose of recording the speaker's speech at the near end and other ambient sounds and for locating the speaker in the room and operating the video camera (104).

Description

TECHNICAL FIELD TO WHICH THE INVENTION RELETS

The invention belongs to the field of acoustic signal processing, or more specifically, methods of acoustic echo cancellation, spatial selection and location of speakers in a reverberant acoustic environment, and noise suppression using a microphone array.

TECHNICAL PROBLEM

Hands-free, full-duplex voice communication systems are used in many applications such as: video-telephone systems, teleconferencing systems, speakerphones in rooms or cars, human-computer voice communication, etc. "Hands-free" voice communication implies that the speaker is in an acoustic environment at a certain distance from the interface elements of the system - microphone and speaker. Such conditions of speech communication generate several technical problems that need to be solved in order to maintain the quality of communication at an acceptable level.

The main problem is the acoustic echo, which is caused by the transfer of part of the acoustic energy from the speaker to the microphone, so that the interlocutor at the far end hears his own voice as a disturbance. Conventionally, cancellation of the echo signal is performed by an adaptive filter by estimating the transfer function of the acoustic path between the loudspeaker and the microphone, so that its output produces approximately the same signal as the acoustic echo signal. Subtraction of these two signals cancels the acoustic echo. However, echo cancellation cannot be ideal due to the non-linearity of the system and the non-stationarity of the acoustic environment. As a result, a residual echo signal appears. At the same time, the basic requirement remains that the recorded voice signal at the near end is not distorted by the application of the echo suppression procedure.

In the acoustic environment, acoustic disturbances can be of different nature and causes. They can be stationary or non-stationary (for example, computer noise or car noise) and originate from multiple sources located at different positions in the space where the speaker is located. In addition, in closed spaces (workrooms, halls, car cabins) the effect of reverberation appears, which manifests itself as a diffuse disturbance. Since the speaker is most often found in such an environment, his separation from other sources of interference must be carried out in order to allow only him to be recorded. Conventionally, this problem is solved by using a microphone array consisting of several microphones arranged at a minimum distance from each other. A certain configuration of the microphone makes it possible to obtain a system with a directional sensitivity characteristic. This type of microphone system has a sufficiently narrow directional characteristic that in a simple environment it can record only the selected speaker, while other sources of interference located in other positions (locations) can be suppressed and thereby achieve a gain in the relationship between the selected speaker and other interference. The size of this gain depends on: the characteristics of the directionality of the microphone array (width of the basic loop), the size of the side loops, the separability of the speaker and the source of interference (that they are not too close), the size of the reverberation, the non-stationarity of all signal sources, etc.

Determining the direction in the space where the selected speaker is located and directing the directionality characteristic of the microphone array towards him is an important problem in "hands-free" communication systems. Direction determination procedures are very sensitive to all disturbances present in the environment and especially: to the non-stationarity of the chosen speaker (when he moves in the environment) and when there are several speakers speaking at the same time in the given environment (cocktail-party effect). Determining the direction of the current speaker in relation to the microphone array in the horizontal plane is very important in video-telephone and teleconferencing systems, because it is necessary to determine the coordinates for controlling the video camera.

When recording speech in an acoustic environment, there is always the problem of additive stationary and/or non-stationary noise as well as residual noise in the processing of the acoustic signal. These noises degrade the quality of the recorded speech signal, and if they are intense enough, they can cause a violation of its intelligibility. There are many algorithms for noise suppression, optimized for certain types of noise, but there is always a requirement to achieve a certain gain in improving the signal/noise ratio, provided that no distortions are introduced into the speech signal and thus its intelligibility is not further impaired.

Variable ambient conditions and, in particular, the variable distance of the speaker-microphone array, require automatic system gain control so that the speaker's voice level is as stable and pleasant as possible for the listener at the far end of the telecommunications channel. Automatic gain control in full-duplex systems requires additional information from near-end speech activity detectors, far-end speech activity detectors, and acoustic echo suppressors.

It can be seen from the above that the technical problems in the solution of a free, "hands-free" communication system for the transmission of voice signals in full duplex and its application in video-telephone and/or teleconferencing systems are very complex and require an integral approach in optimizing the solution, especially when considering the operation of the system in real time based on a commercial platform of a digital signal processor

(DSP).

STATE OF THE ART

High-quality speech recording in the presence of acoustic disturbances and room reverberation is a complex problem. In conditions where the spectrum of the useful speech signal overlaps with the spectrum of the present disturbances, it is not possible to achieve a significant improvement in the quality of the speech signal with single-channel processing methods. With the development of digital signal processing and the achievement of sufficiently large DSP computing power, the way is open for the application of multi-microphone acoustic signal processing procedures. The advantage of microphone arrays over single-channel processing methods is their ability to adapt their simple reception characteristic (directivity characteristic) to the current spatial arrangement of the selected speaker and interference. At the same time, they achieve the maximum suppression of the present disturbances while at the same time highlighting the selected speaker. The basic problems encountered in the application of microphone arrays are as follows (M.S. Brandstein, D.B. Ward (Eds.), Microphone Arrays: Signal Processing Techniques and Applications, Springer, Berlin 2001; Y. Huang, J. Benestv, Audio signal processing far next generation multimedia communication systems, Kluvver Academic Publishers Publ., 2004.): not knowing the exact location of the selected speaker, not knowing the number and spatial arrangement of the interference present, multiple reflections of the useful source and interference on the walls of the room and the non-stationarity of the source of acoustic interference and the selected speaker.

When a microphone array is used in videophone or teleconferencing systems that operate in full duplex, then the number of problems increases. The biggest problem is the appearance of acoustic echo, then the need for automatic gain regulation

(AGC) of the transmitting part of the system, as well as the possible occurrence of system instability, the so-called microphonics. An additional problem that this patent considers is the existence of a TV program signal that appears as an additive acoustic echo at the input of the microphone array.

A large number of the mentioned problems have generated very different solutions which have been patented and which solve either individual problems or integrally several problems. For example: U.S. published patent application 2006/0153360 Al, filed on Sep. 2, 2005, entitled "Speech signal processing with combined noise reduction and echo compensation", provides an integral echo canceller and noise canceller solution, then U.S. Pat. patent 7,035,415 B2, filed on May 15, 2001, entitled "Method and device for acoustic echo cancellation combined with adaptive beamforming", which provides an integral echo canceller solution and a solution for forming the directional characteristic of a microphone array, then EP published patent application 1 633 121 Al, filed on September 3, 2004, entitled "Speech signal processing with combined adaptive noise reduction and adaptive echo compensation", provides an integral solution of residual echo suppressors and noise suppressors, then EP published patent application l 571 875 A2, filed on February 23, 2005, with the title "A system and method for beamforming using a microphone array", which provides a solution only for forming the directional characteristic of a microphone array, then EP published patent application 1 581 026 Al, filed on March 17, 2004, with the title "Method for detecting and reducing noise from a microphone array", provides a solution only for noise suppression in a microphone array, as well as EP published patent application 1 286 175 A2, filed on August 1, 2002, entitled "Robust talker localization in reverberant environment", provides a solution only for speaker localization in a reverberant room.

The integral solution to all the indicated problems, presented in this patent, combines the positive features of individual signal processing procedures in the solution of each of the indicated problems, solves them integrally in the frequency domain, optimizing computer resources and provides a solution that in real time ensures high-quality free voice communication in video-telephone and/or teleconferencing systems.

DISCLOSURE OF THE ESSENCE OF THE INVENTION

The subject of this invention is a system for free speech communication in video-telephone or teleconferencing applications that uses a microphone array and complex acoustic signal processing in order to ensure the quality and intelligibility of the speech signal in a complex acoustic environment and in which many of the previously listed shortcomings are individually or integrally eliminated.

The system, which is the subject of the invention, transmits speech and digital television is used as a transmission medium. A microphone array and speakers are used to record and reproduce the voice signal, respectively, which are integral elements of the TV receiver. Since we are talking about video-telephone and teleconferencing applications, a digital camera and a digital TV receiver are used to record and reproduce the image, respectively.

The essence of the invention lies in the specific processing of the speech signal that is recorded in the acoustic environment of the room where the system and the speaker are located. To record the speaker in the room, which is located at a certain distance (up to several meters) from the TV receiver, the system uses a microphone array of N microphones. The microphone array records all the signals in the room: the useful signal as a direct wave arriving from the speaker to the microphone and interference signals that can be varied. The following interference signals appear: acoustic echo as a direct sound wave from the speakers through which the speaker's voice is emitted from the far end of the communication channel, acoustic echo as a direct sound wave from the speakers through which a stereo TV program is broadcast, direct waves from one or more sources of noise or sources of other disturbances that can be found in the room and all reflected waves (room echoes) originating from all sound sources, including the speaker, which arise as a result of room reverberation. It should be emphasized that the sources of sounds in the room can be stationary or non-stationary, which is the most common case, both according to their characteristics and their location in the room (moving sources of sounds).

Different interferences require different techniques for their elimination and the essence of the invention is in the optimal design of algorithms that should maximally eliminate interferences and ensure the best quality of the voice signal that is transmitted to the speaker at the far end of the communication channel.

The microphone signals from the microphone array are processed in digital form in the DSP, completely in the frequency domain. This domain provides certain advantages in terms of processing speed and number of computational operations, which is very important for DSP and real-time work. In order to suppress the acoustic echo, it is necessary to introduce the signals from the speakers into the DSP.

Several complex algorithms are executed in the DSP: an algorithm for suppressing acoustic echo signals (AEC-Acoustic Echo Cancelling), an algorithm for processing microphone signals in order to form an adaptive characteristic of the directionality of the microphone array

(ABF - Adaptive Beam Forming), an algorithm for estimating the direction of arrival of the useful signal (DOA - Direction of Arrival), that is, locating the speaker in the room, an algorithm for suppressing stationary and non-stationary noise and residual echo (NR- Noise Reduction) and an algorithm for automatic system gain control (AGC - Automatic Gain Control) to compensate for the different distance of the speaker from the microphone array. In addition to these basic algorithms, several other algorithms are executed in the DSP, such as: speech activity detector (VAD - Voice Activated Detector) at the near end, VAD at the far end, simultaneous speech activity detector at both ends (DTD - Double Talk Detector), additional filtering for noise reduction (PF - Post Filtering), etc. The goal of all the mentioned algorithms is the maximum reduction of all interference with the minimum degradation of the voice signal and thereby ensuring the maximum quality of the transmitted voice signal.

A specific aspect of the invention resides in adaptive acoustic echo suppression using adaptive filters that model the transmission characteristic of the acoustic path from the speaker to the microphone. The transmission characteristic is complex because it is a transmission path from 2 (stereo) speakers to N microphones in a microphone array, which is why each microphone signal is filtered by its own adaptive filter. The operation of the adaptive filters is controlled by the speech activity detector at both ends.

The next specificity of the invention is the adaptive characteristic of the directionality of the microphone array, which enables spatial filtering, that is, the selection of the direction in the space where the speaker is located and where the useful signal is maximally amplified compared to signals from other directions that are weakened. The directional characteristic of the microphone array is achieved by adaptive weighting and summation of microphone signals, which ensures a stable directionality index in the frequency domain and greater robustness of the system for free speech communication in a reverberant acoustic environment.

Determining the incoming direction of the direct acoustic wave from the speaker is a further specificity of the invention. This function in a free speech communication system is necessary to control and operate the directional feature of the microphone array in azimuth, and it can also be used to control and operate a video camera. It uses microphone signals after acoustic echo suppression. After determining the generalized cross-correlation of microphone signals and their phase transformations, the incoming direction of the speaker's direct acoustic wave is estimated. This function is under the direct control of the speech activity detector.

The next specificity of the invention is the process of adaptive suppression of stationary and non-stationary noise. The procedure was implemented on the basis of a non-linear compressor of estimated noise, which is determined in several sub-bands. Two noise estimations are used, which provide a suppression result optimized according to the characteristics of the speech signal. This is done due to the need that the process of adaptive noise suppression must not degrade the speech signal. The filtering process is completed with an adaptive Wiener post-filter.

A specific aspect of the invention is the automatic control of the amplification of the voice signal before transmission to the remote interlocutor. This specificity is an important integral element of the system for free speech communication. The system provides compensation of different intensities of the speech signal, as individual characteristics of the speaker, and also different intensities of speech depending on whether the speaker is closer or further away from the microphone array. The solution makes a difference whether the speaker is active or whether the following signal appears in the useful signal: a pause, a residual echo, an acoustic disturbance or a speech signal from the far end; therefore, the solution uses more information previously detected in the system. The analysis of the possible scenario must be reliable, otherwise there may be a negative effect of weakening the useful speech signal.

The inventiveness of this invention is found in the improvement of each of the aforementioned specificities, but also in the process of integrating all the algorithms into a single entity that functions stably and with quality. Algorithmic procedures are optimized using shared resources.

These and other aspects, specificities and benefits of the present invention will be more apparent upon review of the detailed description of the invention, patent claims and accompanying drawings.

BRIEF DESCRIPTION OF THE IMAGES AND DRAWINGS

Figure 1 - shows the elements of a system for free video-telephone communication using a microphone array and digital television.

Figure 2 - shows the ambient conditions of application of the system for free video-telephone communication using a microphone array.

Figure 3 - shows a block diagram of the audio signal processing subsystem within the system for free video-telephone communication; it contains a microphone array with adaptive directivity (SD-BF), a speaker localization (DOA) block, an echo cancellation (AEC) block, a noise cancellation (NR) block and an automatic gain control (AGC) block.

Figure 4- shows a block diagram for Acoustic Echo Cancellation (AEC).

Figure 5 - shows a block diagram for adaptive determination of the direction of a nearby speaker horizontally (DOA-azimuth).

Figure 6- shows the block diagram for spatial filtering (SD-BF).

Figure 7 - shows the noise reduction (NR) block diagram.

Figure 8 - shows the block diagram for automatic gain control (AGC).

DETAILED DESCRIPTION OF THE INVENTION

This invention describes an acoustic signal processing system and method for free speech communication using a microphone array.

Figure 1 shows the elements of a system for free video-telephone communication using a microphone array and digital television. Digital television 100, which normally serves the user to monitor TV programs, in the system for free video-telephone communication is used as a video monitor for video communication with the interlocutor and as an audio terminal for audio communication. Namely, when a call is received through the communication channel 101 and a connection is established with the interlocutor, then the television 100 is used as a multimedia interface where the interlocutor is heard through the speaker 102 and the image of the interlocutor is shown on the screen 105 of the television 100. At the same time, at the far end of the communication channel, the interlocutor on a similar TV receiver sees the speaker from the near end, who is recorded by camera 104 and microphone array 103. Camera 104 is mobile and is controlled based on coordinates obtained by processing microphone signals from microphone array 103.

Analog signals from the microphones in the microphone array 103 are amplified by the amplifier 106 and together with the stereo signals from the loudspeakers 102 are introduced into the acquisition module 107, where they are digitized and thus digitized are handed over to the DSP 108 for further processing. The processed speech signal of the speaker at the near end using the DSP 108 is transmitted via the communication channel 101 to the speaker at the far end. By processing the acoustic signals in the DSP108, simple coordinates for locating the speaker in the room where the free communication system is located are obtained, with the help of which the DSP108 manages the moving camera 104 directing it towards the speaker. In this way, completely free audio and video communication between two speakers is achieved through the digital television system.

Figure 2 schematically shows the ambient conditions of application of the system for free video-telephone communication using a microphone array; only part of the system related to acoustic signal processing is shown. In the room 201 there are a system for free video-telephone communication, a speaker 202 and a noise source 203, which is common for any acoustic environment. Through the speaker 102 of the digital television stereo audio system, the speaker 202 listens to the incoming speech signal 204 of the speaker from the far end, usually as a mono signal. Sound in the room environment 201 is recorded by a microphone array 103 composed of N microphones. After the complex processing of microphone signals in block 207, the speech signal of the speaker 202 is transmitted via block 208 to the remote speaker as a mono signal.

The ambient conditions for speech communication in room 201 are very complex. During free video-telephone communication in the room 201, there are at least three sound sources: stereo speakers 102 that broadcast the speech of the remote speaker and the TV program, speaker 202 and at least one source of noise 203. There can be several sources of noise in the room: the noise of the computer, the noise of the air conditioning system, noise from the street that penetrates into the room through the windows, noise from neighboring rooms, vibrations of the building, or another speaker, multiple speakers, music source, etc. Thus, a very complex acoustic picture appears in the room. The microphone array 103 records, as a sensor system, all sounds in the room, it records direct sound waves from each source, but also all reflections from the walls of the room and other objects in it. For example, from the loudspeaker 102 to the microphone array 103, a direct wave 209 and many reflected waves arrive, only one of which 210 is shown in Figure 2; from the speaker 202 comes a direct wave 211 and, in addition to the others, two reflected waves 212a and 212b, from the noise source 203 a direct wave 213 arrives and, in addition to the others, a reflected wave 214.

Of all the sounds that the microphone array records, only the direct wave 211 from the speaker 202 is a useful signal, all others are interference. Of all the disturbances, the biggest is the acoustic echo 209 that comes from the speaker 102. All other reflections collectively make up the reverberation of the room. The task of the audio signal processing block 207 is to suppress the acoustic echo signal, to select the useful signal 211 from all other interferences, to suppress the reverberation signals and to suppress the direct signals of the interference sources, which may be more than one source. The special task of block 211 is to adaptively monitor the non-stationarity of the acoustic scene in the room, whether the speaker is moving, or is in different positions in the room from conversation to conversation, or that noise sources are moving, are non-stationary or change their characteristics. In the following text, the solutions used in this invention will be described individually.

Figure 3 shows a block diagram of the complete audio signal processing procedure within the system for free video-telephone communication using a microphone array. All microphone signals 103, from Ml to M5, as well as stereo speaker signals 102, Zv-L and Zv-D, are digitized in the acquisition block 107, Figure 1, and converted to the frequency domain using the fast Fourier transform (FFT) 301 in signalex;doxi. It should be emphasized that the microphone array contains 5 microphones in the solution of this patent, but a larger number of microphones can be applied if a certain application requires it. In block 302, acoustic echo suppression is performed in all signalsmax\dox$, using signalsex&ix7 as reference. Signals with suppressed echoSaecidoSaecs are used in block 304 to determine the direction of the direct sound wave DOA (Direction Of Arrival) horizontally (azimuth0a) from the current speaker and thus enables his monitoring in the room. Based on the estimated angle in block 303, the weighting coefficients of the x-signal are optimized with the goal of forming the characteristic of the horizontal directionality of the microphone array with maximum reception in the direction 8a.

In block 303, the time compensation of the mutual delay of the acoustic signals from the speaker to the microphone is performed. By controlling this delay with the DOA (0a) signal from block 304, it is possible to control the directionality characteristic of the microphone array in azimuth. Also, in block 303, the directional feature of the microphone array, SD-BF (Superdirective Beamformer), is formed. This feature has a basic directional loop that is sufficiently narrow and directed in the desired direction, while the side loops are significantly less intense. This enables the microphone array to perform spatial filtering, i.e. horizontal separation of sound sources. The directional characteristic formed in this way is very important from the aspect of muting the signal of lateral interference in relation to the useful signal and from the aspect of reducing the effect of room reverberation. The directivity characteristic is formed by weighting the microphone signals and summing them into a single output signal.

The signal at the output of block 303 contains the useful speech signal and the interference signal consisting of the residual signal after acoustic echo suppression, suppressed ambient noise and suppressed reverberation signals. This signal enters the block NR (Noise Reduction) 305, where additional interference signal suppression is performed. The suppression process is adaptive considering the non-stationarity of the interference signal. Also, an important requirement in the realization of the NR block is that the noise suppression process must not affect the quality of the voice signal.

The final block of signal processing in the system for free voice communication in video-telephone or teleconferencing applications is block 306 for automatic gain control of the AGC (Automatic Gain Control) processed voice signal. In this block, more information from the entire system is used, which is important for defining the possible conditions in which the speech signal can be found and where it is necessary to carry out its amplitude correction in an appropriate manner. In this way, it is possible to ensure approximately the same level of transmission speech signal regardless of the distance of the current speaker from the microphone array and ensure its better quality at the far end of the communication channel.

At the output of the system, the result of the signal processing is transformed from the frequency domain to the time domain using the inverse FFT in block 307. The estimated speech signal at the near end (š) is transmitted through the channel to the remote interlocutor.

Figure 4 shows the block diagram of the acoustic echo suppressor (AEC) 302, which consists of two basic blocks: block 401, which consists of 5 adaptive NLMS (Normalized Least Mean Square) algorithms, and block 402, whose basic function is the detection of the speech activity of the near and far speaker DTD (Double Talk

Detection).

The NLMS algorithms, NLMS1 through NLMS6, process the signals from the microphones/dox and pass the processed signals on to blocks 303, 304, and 306, Figure 3. The function of the NLMS algorithms is to suppress echoes in each of the microphone signals. This function is enabled by the reference signals from the speaker 102 and the control signals from the DTD detector 402. The NLMS algorithm models the transfer function of the acoustic path from each speaker 102 to each microphone 103; for example, NLMS1 models the transfer functions from the speaker Zv-L to the microphone Ml and from the speaker Zv-D to the microphone Ml, etc. By passing the signal from the speaker through the NLMS filters, a replica of the signal on the microphones that came acoustically is obtained, and by subtracting these two signals, the suppression of the echo signal at the output of the NLMS algorithms is achieved. In order to better suppress the echo, as in the case of the RLS1AEC algorithm (RLS-Recursive Least Squares) which is described below, the DFT coefficients from the previous processing blocks are used. As the NLMS algorithm requires significantly less computer time compared to the RLS algorithm, the DFT coefficients from the previous 5 processing blocks are used in the implementation of the NLMS algorithms.

Block 403 labeled RLS1 AEC is the key algorithmic part of the dual speech activity detection procedure of block 402. RLS1AEC performs rough suppression of acoustic disturbances in the signal from the microphone Ml by applying the RLS algorithm. The RLS algorithm has fast convergence, which provides a good estimation of the speech signal as well as the estimation of the additive component of the echo signal. Given that the size of the applied DFT window of 1024 is not large enough to achieve maximum suppression of echo interference in a room with high reverberation, DFT coefficients from the 3 previous processing blocks are added to the regression vector. This achieves a double gain: maximum echo suppression and signal delay through the system is not increased because the DFT order remains unchanged.

The output from the RLS1AEC block is two signals. The first signal is the estimation of the speech of the nearby speaker on the microphone Ml. The second signal is the estimation of the additive component of the echo signal in the microphone signal Ml. Both of these signals are used for the detection of double speech activity, which is realized in block 402 labeled DTD. The signal from the DTD detector controls the operation of the NLMS algorithms in the sense that it prevents the adaptation of the algorithms NLMS 1 to NLMS 5 during the double speech activity, when the operation of the adaptive algorithms is disturbed. In block 405, the signal strength on the loudspeakers is averaged according to the relation:

On both signals, yiPrefse applies recursive averaging, so that the averaged strengths of the echo signal in the microphone Ml (2) and the signal at the speakers producing the echo (3) are obtained.

The estimation of the ratio of these two forces is determined by the quantity Cs:

and it is used to scale the power of speaker signals for the purposes of making a soft decision in block 408. In this block, the absence of a closer speaker in the microphone signal is determined on the basis of a soft decision defined by the relation: where:af- is a frequency-dependent constant that artificially favors the permission for convergence at higher frequencies, where the signal powers are smaller, and thus there is less possibility of divergence of NLMS algorithms. SizeX is the minimum ratio of the echo signal strength to the nearby speaker for which the soft decision is a positive number. In block 409, the control signal Dui is limited, which, in addition to the NLMS algorithms, is also fed into the DOA-azimuth block. Figure 5 shows a block diagram of the solution for determining the azimuth 304, i.e. the direction of arrival of the direct sound wave DOA-azimuth from the active speaker. The input signals to this block are channel signals from the AEC block SAecidoSaecs, and the output signal is the estimation of the arrival angle 6a. The algorithm is based on the cross-correlation analysis of the input signals to the SaecidoSaecsu block 501, at the output of which estimates of four cross-correlation functions are obtained G\ t2{ tJ)doG\$( tj) by recursive averaging according to the relation

The constants cu and a are chosen to satisfy the inequality 0.5 < a+ < a. < 1 and under that condition the influence of members Xx( t, f) X\ (r, /) with the largest modulus is favored.

In block 502 labeled PHAT, generalized cross-correlation, often labeled as phase transformation in the literature, is implemented. Namely, by normalizing the cross-correlation to its modulus, the information about the signal strength is lost, and only the information about the phase in which the relative time delay of the signal is contained remains. By inverse FFT transformation Glk(t, f) and finding the maximum, the relative time delay of the sound wave between two microphones is evaluated.

Since the speech signal has a formant structure, which is why all frequency bins do not have the same power, it is necessary to select the bins with the highest power and use them to determine the cross-correlation function. To this end, block 503 calculates the current power of each channel signal and calculates the average value of the power of all channels P(t, f). Block 504 determines the weighting function W(tJ) that favors bins with an increase in current signal power. The reason for choosing such a solution is that the part of the signal with a sudden increase in power has a larger share of direct waves than the part with a drop in power, where wave reflections or room reverberation dominate. In block 505, the mean power of the microphone signals averaged by time and frequency, P(t, f) is calculated. First, the bins are averaged by frequency with a non-causal HR filter of the first order (zero phase delay is achieved by double forward and backward filtering). Time averaging is performed by a first-order nonlinear IIR filter with two averaging coefficients, one for increasing and one for decreasing signal power. This nonlinear filter is described by the relations:

The size P(t, f) is used to define the decision threshold for extracting the bins with the highest power in block 506. Multiplying the binary output from block 506 and the weight vector W(t, J) yields a filter function W(t, f) that weights the phase transform bins in block 502. The phase transformed cross-correlation functions are additionally filtered with an IIR filter in time to reduce variance. estimation of correlation functions. This is described by the relation:

In addition to the selection of bins with the function W(t, f), an a priori rejection of bins that are outside the range of interest is also applied. In block 507, ranges that are not a priori of interest are defined and they are discarded before the inverse FFT (FFT<1>). In block 509, the time alignment of the cross-correlation functions is performed, which are then averaged, and at their mean value, the maximum is determined in block 510, whose abscissa represents the estimation of the time delay, .

Incoming direction estimation makes sense when a nearby speaker is active; when he is not active, the estimate obtained during his last activity is used for a valid estimate. In order to detect the activity of a nearby speaker, the following are used: a) information from block 513 about the average strength of microphone signals; b) information from the speaker double activity detector D?, from block 402, Figure 4; and c) informationsbfiz of block 303, SD-BF Figure 3. Based on this information in block 512, a decision is made about the activity of the close speaker. In the case of a decision that the estimation of the incoming direction is valid, that the closer speaker is active, the current estimation of the incoming direction is sent to the output of the DOA block 304; otherwise, the last valid direction estimate is passed.

Figure 6 shows a block diagram of the procedure for forming the superdirective spatial filter 303, Figure 3. Due to the problem of self-cancellation of the useful signal that occurs when the adaptive algorithm for suppressing acoustic interference is applied in a room with reverberation, a superdirective spatial filter 601 with fixed coefficients is often applied instead of the adaptive algorithm. A superdirective spatial filter provides a higher directivity index than a conventional spatial filter that only includes delay compensation and summation. The description of the procedure for obtaining the weighting coefficients that ensure the superdirective characteristic of the filter is given below.

For a room with reverberation, a diffuse noise field model is usually adopted, which implies that the noise comes from all directions with approximately the same intensity. For such a noise field model, it is shown that the coherence between two microphones is a real number equal to

where / is the frequency, dtJ is the microphone distance and ij, ac is the speed of sound. Coherences of pairs of microphones fJ(/) form the matrix of coherences Frf. Using the thus defined coherence matrix, the coefficients of the superdirective microphone array are determined in block 602 according to the relation: where C9 is the direction vector to the direction of the selected speaker defined by azimuth 0. This vector is determined in block 603 according to the relation:

Sized is the distance between two adjacent microphones. At the output of block 303, an estimate of the actual speaker's speech is obtained based on the relation:

Figure 7 shows the block for noise suppression 305 marked NR. SignalSb places the input signal in block 305 and it contains the estimated speech signal and residual interference signals originating from acoustic echo, acoustic room interference and room reverberation. SignalSbfse is introduced into block 701, marked with FWF"', in which IFFT is performed, then additional windowing of the time shape of the signal segment in order to "softly" cut off the ends of the segment, and finally returning to the frequency domain using FFT. The essence of this operation is as follows. In the process of previous signal processing, the equivalent time shape of the signal is expanded to the limits of the DFT window. By applying a new Wiener filtering operation, additional segment expansion and cyclic overlapping at the ends are performed segment, which creates impulse interference that manifests itself as a smooth "crack". The applied FWF" procedure completely eliminates the described problem and does not introduce any additional signal distortions.

In the next two blocks 702 and 703, noise estimation is performed based on the minimum input signal power. Since the current adaptation to the minimum power does not give good results, because the DFT coefficients on certain blocks have an extremely low power that disturbs the previous noise power estimation, the Noise estimation is realized in three processing blocks. In the first block 702, a slow estimation of the noise power N, low is performed, in the second 703, a fast estimation of the noise power N^, and in the third 704, based on the estimate N) lowiNfttll, an estimation of the current power is performed by means of a nonlinear transformation forestN.

Fast and slow noise power estimation is realized by the same procedure of recursive averaging by a first-order IIR filter with different adaptation factors for the rise and fall of the output value

where between constantUnM., ash„_, afas,+,<a>slaw_ there is a relation:

The fast and slow noise estimates are combined in block 704, which is designated as the nonlinear compressor. The final estimate of the noise level is obtained based on the following relationship:

where the parameter a (0.25<a<0.5) regulates the degree of compression of the noise estimation dynamics, and the parameter p defines the overestimation of the noise power. The meaning of the nonlinear transformation is as follows: in case Nfile>N! tow application of only fast estimation would result in excessive suppression of the speech signal, that is why compression of noise estimation dynamics was introduced. In the case of Nfmt < N, lowne compression is applied so that the noise estimate decreases as quickly as possible. This prevents the clipping of phoneme parts at the ends of words when, due to a rapid drop in signal strength, the high value of the previous noise estimate of the slow estimator cannot follow this change in dynamics.

Since the ratio of useful speech signal and noise is significantly less favorable at high frequencies, a set of parameters a and p is defined for 4 characteristic frequency ranges (0-2000Hz), (2000-2500Hz), (2500-3500Hz)H(3500-5012Hz), according to the expected signal/noise ratio. This set of parameters is stored in block 705 .

In block 706, Wiener filtering is performed using the following transfer function:

where the constant /?M has the function of estimating the original estimate of the noise power in order to achieve a compromise between the highest possible noise suppression and the minimum degradation of the useful speech signal. The transfer function can have an unacceptably long impulse response in the time domain, which produces distortions at the boundaries of the DFT blocks, and therefore a "soft" shortening of the impulse response is performed using the above-described FWF'' procedure. Finally, additional filtering of the output estimated speech signal is performed in block 707, in order to reject out-of-band spectral components.

speech signal, which may arise in previous signal processing processes, and which may affect the operation of the AGC block.

Figure 8 shows the block for automatic gain control (AGC) of the output signal of the system, block 306. The task of the AGC block is: (1) to amplify weak speech signals and to weaken excessively strong signals according to a predetermined signal dynamics compression characteristic, (2) to reduce the gain on parts of the input signal where there is only signal echo, stationary noise or competing speaker interference, in order to silence these interferences sufficiently and (3) to silence parts of the input signal where both the useful speech signal and interference are simultaneously present, while preserving speech intelligibility.

At the input of block 306 comes signals^ giz of block NR, figure 3 block 305, and passes through a compressor of signal dynamics with an adaptive slope of the compression characteristic, block 801. The output of block 801 is a signal which then passes through block 307, figure 3, where it is converted from frequency to time domain by inverse Fourier transformation FFT"<1> and as the final signal of speech signal estimation is transmitted to the remote speaker through a digital channel television.

Voice signal gain control is performed in block 801 based on the following relationship:

where: Aagc- amplification of the AGC block, Pnom- nominal power of the output signal, and -

constant which limits the maximum amplification to the level AagcamK= Vl/or (for the value a = 0.001 the maximum amplification is Aagc max =31.6 dB), Pin<=>Pa + P„<+>Pecka ( Pd

- power of useful voice signal, P„- power of diffuse ambient noise and Peko-

the power of the unsuppressed echo signal), iSLOPE = /[ P^ it)]- a quantity that represents the degree of compression of the signal dynamics and is a complex function of the peak power of the useful speech signal. In block 802, the value of SLOPE is calculated based on the analysis of the trajectory of the peak power of the useful speech signal and monitoring its convexity and growth trend.

In block 803, the peak power of the useful speech signal is calculated according to the following relations:

where Ođ is a constant value close to 1.

In block 804, the unsuppressed echo power estimate is determined according to the relation:

where echa is the echo suppression constant of block 402, Figure 4.

In block 805, the diffuse noise P„ is estimated as the average power of the input signals in block 303, Figure 3, and the power of the output signal in block 303.

Direct application of relations. Aagcza pre-fixed size SLOPE does not give good results, because it treats the residual interference and the useful signal equally. When only disturbances are present, they are amplified, which is not good. It is therefore necessary to detect and separate the following cases: (a) a pause in the useful speech signal, (b) a residual echo present, and (c) a competing speaker or acoustic disturbance present. When any of these cases are detected, the SLOPE variable is set to 1, thus preventing interference amplification.

A pause in a useful speech signal differs from a speech signal in its stationarity. The speech signal, no matter how weak it is, is non-stationary in time, while the slow-changing ambient noise is present during the pause. A linear trend of signal strength normalized to power is a good indicator of signal non-stationarity. To that should be added the indicator of convexity of the trajectory, which is negative at the local maximum.

This invention describes the process of processing acoustic and speech signals in a free speech communication system that operates in full duplex. This invention relates to free speech communication in a digital television system, but it can also be applied to other communication systems such as video-telephone systems, teleconferencing systems, speakerphones in a room or in a car, human-computer communication by voice, etc. The specificity of the solution in this invention is its integration into a standard digital TV receiver and its optimization for use in medium-sized rooms (acoustic environments) with a reverberation time of up to 600 ms.

The procedures and techniques of acoustic and speech signal processing in this invention can be generalized to N microphones in a microphone array in multi-channel recording and to M speakers in multi-channel playback.

The procedures and techniques of processing acoustic and speech signals in this invention are under the control of a number of parameters that enable the optimization of solutions for different applications.

The acoustic and speech signal processing methods and techniques of the present invention can be implemented in a variety of ways. For example, these techniques can be implemented in hardware, software, or a combination. A hardware implementation may use specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic circuits (PLDs or FPGAs), and other electronic circuits designed to perform the functions described in this invention.

The procedures and techniques for processing acoustic and speech signals in this invention can be implemented in software as a whole or by modules that perform certain functions described in this invention. Program codes can be stored in memory units and executed by processors such as PC, PDA, DSP, etc.

The details of the invention described herein enable any person skilled in the art to implement the generic principles of the invention in other speech communication systems without departing from the scope of the invention.

Claims (24)

1. A system for free speech communication using a microphone array containing a digital TV receiver that enables audio and video communication in full duplex, characterized by the fact that the digital TV receiver (100) has a stereo audio reproduction (102) for reproducing stereo TV programs and a mono incoming speech signal in videotelephone communication, which has a built-in moving video camera (104) for recording speakers in the room and which on part of its screen reproduces the image of the speaker from the far end (105); which contains a microphone system (103) built into the TV receiver (100) whose purpose is to record the speaker's speech at the near end as well as other ambient sounds and whose purpose is to locate the speaker in the room and control the video camera (104). 2. The system according to claim 1, characterized in that its audio transmission part (207) and (208) enables the suppression of the acoustic echo (209) generated by the speakers of the TV receiver (102), enables the suppression of ambient interference (213) and reverberation (210), (212) and (214), enables the location of speakers in the room, enables adaptive control of the signal level in the transmission and provides coordinates for controlling the video camera. 3. The system according to claim 2, characterized in that it contains a microphone array (103) of more than 2 microphones that provide microphone signals for further parallel processing, a module for adaptive acoustic echo suppression (AEC) (302) consisting of a set of adaptive filters, a module for estimating the incoming direction of the speaker's direct sound wave (DOA) (304) and managing the directionality characteristic of the microphone array, a module for forming the directionality characteristic of the microphone array with optimized ratio of main and side loops (SB-CBF) (303), module for adaptive suppression of all residual interference signals (NR) (305) and module for automatic system gain control (AGC) (306). 4. The system according to claim 3, characterized in that it contains a set of microphones (103) located in a horizontal plane at equal distance from each other and mounted on the upper edge of the digital TV receiver (100). 5. System according to claim 4, characterized in that it suppresses the acoustic echo (209) generated by the stereo speakers (102) and which consists of the stereo audio TV signal (205) and the mono speech signal originating from the remote speaker (204). 6. The system according to claim 5, characterized in that the echo suppression unit (302) and the ambient interference suppression unit (305) work even in low signal-to-noise ratio conditions. 7. A system according to any of the previous requirements, characterized by the fact that it enables adaptive locating and tracking of speakers in space by azimuth. 8. The system according to claim 7, characterized in that it enables adaptive determination of spatial coordinates for controlling the video camera. 9. The system according to claim 4, characterized in that its microphone array forms a narrow directional characteristic that enables spatial filtering and separation of the current speaker from other sources of interference in the room. 10. The system according to claim 9, characterized by the fact that its microphone array forms a narrow directional characteristic that enables suppression of echoes due to reflections in the room, i.e. reverberation signals. 11. A system according to any of the previous requirements, characterized by the fact that, through automatic system gain control, it maintains the average level of the transmitted speech signal within acceptable limits of normal speech dynamics, regardless of the distance and position of the speaker in relation to the microphone array. 12. A procedure for free speech communication using a microphone array, characterized by parallel processing of microphone signals from the microphone array and thereby achieving adaptive suppression of acoustic echo in microphone signals, which estimates the incoming direction of the direct sound wave of a nearby speaker, which forms a superdirective characteristic of the directionality of the microphone array and controls its spatial position in azimuth, which suppresses all interference signals found in microphone signals and which performs automatic maintenance of the level of the transmitted voice signal. 13. The method according to claim 12, characterized by the fact that the complete processing of all audio signals is performed in the frequency domain. 14. The method according to claim 12, characterized by the fact that the adaptive suppression of the acoustic echo is performed individually for each microphone signal and that the suppression includes both signals coming from the stereo speakers. 15. The method according to claim 14, characterized in that the adaptive acoustic echo suppression is performed for each microphone signal by means of NLMS algorithms controlled by means of a speech activity detector at both ends (DTD). 16. The method according to claim 14, characterized by the fact that the NLMS algorithms are controlled by means of a near-end speech activity detector implemented within the DTD and based on the RLS adaptive algorithm under specific conditions of the continuous presence of a TV audio program signal, which, in addition to speech, also contains a music signal. 17. The method according to claim 12, characterized in that the estimation of the incoming direction of the direct sound wave from the current speaker is performed on the basis of cross-correlation analysis of microphone signals after acoustic echo suppression. 18. The method according to claim 17, characterized in that the estimation of the incoming direction of the direct sound wave from the current speaker is performed under the control of the VAD speech detector at the near end. 19. The method according to claim 12, characterized by the fact that the directional characteristic of the microphone array is formed in the SB-CBF module as a superdirective characteristic based on the principle of weighting and summing the microphone signals after acoustic echo suppression and adaptive control according to azimuth. 20. The method according to claim 19, characterized in that the coefficients of the superdirective microphone array are determined using the coherence functions of the pairs of microphone signals and the direction vector to the direction of the selected speaker, defined by the azimuth angle. 21. The method according to claim 12, characterized in that the residual noise suppression function is realized by an adaptive Wiener filter. 22. The method according to claim 21, characterized in that the estimation of the residual noise in the noise suppressor is optimized according to the characteristics of the speech signal and realized on the basis of a non-linear compressor of the dynamics of the estimated noise, parametrically controlled and frequency dependent. 23. The method according to claims 12 to 22, characterized in that the module for automatic system gain control is based on a dynamics compressor with an adaptive slope of the compression characteristic. 24. The method according to claim 23, characterized in that the speech signal dynamics compressor is controlled by means of a residual acoustic echo presence detector, a speech signal pause detector, and a competing speaker and acoustic disturbance detector.