CN116830561A - Echo reference prioritization and selection - Google Patents

Abstract

Some embodiments involve obtaining a plurality of echo references, the plurality of echo references including at least one echo reference for each of a plurality of audio devices in an audio environment, each echo reference corresponding to a response provided by one of the plurality of audio devices. Audio data played back by one or more loudspeakers of an audio device. Some examples involve making importance estimates for each of multiple echo references. Making the importance estimate may involve determining the expected contribution of each echo reference to echo mitigation by at least one echo management system of at least one audio device of the audio environment. Some embodiments involve selecting one or more selected echo references based at least in part on an importance estimate and providing the one or more selected echo references to at least one echo management system.

Description

Echo reference prioritization and selection

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No. 63/147,573 filed on February 9, 2021, U.S. Provisional Application No. 63/201,939 filed on May 19, 2021, and European Application No. 21177382.5 filed on June 2, 2021, all of which are incorporated herein by reference in their entirety.

Technical Field

The present disclosure relates to devices, systems, and methods for implementing acoustic echo management.

Background Art

Audio devices with acoustic echo management systems have been widely deployed. An acoustic echo management system may include an acoustic echo canceller and/or an acoustic echo suppressor. Although existing devices, systems, and methods for acoustic echo management provide benefits, improved devices, systems, and methods would still be desirable.

Symbols and terminology

Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-producing transducer (or set of transducers). A typical set of headphones includes two speakers. Speakers may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feed(s) may undergo different processing in different circuit branches coupled to different transducers.

Throughout this disclosure, including in the claims, the expression “performing an operation “on” a signal or data” (e.g., filtering, scaling, transforming, or applying a gain to a signal or data) is used in a broad sense to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other X-M inputs are received from external sources) may also be referred to as a decoder system.

Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and/or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets.

Throughout this disclosure, including in the claims, the terms "couple" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

As used herein, a "smart device" is an electronic device that can operate interactively and/or autonomously to some extent, and is typically configured to communicate with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, near field communication, Wi-Fi, Light Fidelity (Li-Fi), 3G, 4G, 5G, etc. Some well-known types of smart devices are smart phones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, tablet phones and tablet computers, smart watches, smart bracelets, smart key chains, and smart audio devices. The term "smart device" may also refer to devices that exhibit some properties of ubiquitous computing such as artificial intelligence.

In this document, the expression "smart audio device" is used to represent a smart device, which is a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of the virtual assistant functionality). A single-purpose audio device is a device (e.g., a television (TV)) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and/or at least one camera) and is largely or primarily designed to implement a single purpose. For example, although a TV can generally play (and is considered to be able to play) audio from program material, in most instances, modern TVs run some kind of operating system on which applications (including applications for watching TV) run locally. In this sense, a single-purpose audio device with (multiple) speakers and (multiple) microphones is typically configured to run local applications and/or services to directly use the (multiple) speakers and (multiple) microphones. Some single-purpose audio devices can be configured to be combined together to enable audio to be played in a certain area or user-configured area.

A common type of multi-purpose audio device is an audio device that implements at least some aspects of the virtual assistant function, although other aspects of the virtual assistant function may be implemented by one or more other devices such as one or more servers, and the multi-purpose audio device is configured to communicate with the one or more servers. Such a multi-purpose audio device may be referred to as a "virtual assistant" in this article. A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and/or at least one camera). In some examples, a virtual assistant may provide the ability to use multiple devices (different from the virtual assistant) for applications that support the cloud in a sense or are otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant function (e.g., speech recognition function) may be (at least partially) implemented by one or more servers or other devices, and the virtual assistant may communicate with the one or more servers or other devices via a network (e.g., the Internet). Virtual assistants can sometimes work together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants can work together in the sense that one of them (e.g., the virtual assistant that is most confident that it has heard the wake-up word) responds to the wake-up word. In some implementations, the connected virtual assistants may form a constellation, which may be managed by a master application, which may be (or implement) the virtual assistant.

In this document, "wake-up word" is used in a broad sense to refer to any sound (e.g., a word spoken by a human or other sound) that the smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "waking up" means that the device enters a state of waiting (in other words, listening) for a voice command. In some instances, the so-called "wake-up word" in this document can include more than one word, for example, a phrase.

In this document, the expression "wake-up word detector" refers to a device configured to continuously search for alignment between real-time sound (e.g., speech) features and a trained model (or refers to software including instructions for configuring the device to continuously search for alignment between real-time sound features and a trained model). Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability of detecting a wake-up word exceeds a predefined threshold. For example, the threshold may be a predetermined threshold that is adjusted to give a reasonable compromise between a false acceptance rate and a false rejection rate. After a wake-up word event, the device may enter a state (which may be referred to as an "awake" state or an "attention" state) in which the device listens for commands and passes received commands to a larger, more computationally intensive recognizer.

As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some instances a collection of video signals, at least a portion of which is intended to be heard together. Examples include music selections, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from intelligent assistants, and the like. In some instances, a content stream may include multiple versions of at least a portion of an audio signal, e.g., the same conversation in more than one language. In such instances, only one version of the audio data or portion thereof is intended to be reproduced at a time (e.g., a version corresponding to a single language).

Summary of the invention

At least some aspects of the present disclosure may be implemented via one or more audio processing methods. The audio processing method manages echoes in an audio system. The audio system includes multiple audio devices in an audio environment. Each of the multiple audio devices includes one or more loudspeakers. In some instances, (multiple) methods may be implemented at least in part by a control system and/or via instructions (e.g., software) on one or more non-transitory media stored in a first device of the multiple audio devices of the audio system. The first device may include one or more microphones. Some such methods involve obtaining multiple echo references by the control system of the first device. The multiple echo references may include at least one echo reference for each of the multiple audio devices in the audio environment. Each echo reference may correspond to audio data played back by one or more loudspeakers of a corresponding audio device in the multiple audio devices. The multiple echo references include at least one echo reference for the first audio device.

The method may involve making, by the control system, an importance estimate for each of the plurality of echo references. In some examples, making the importance estimate may involve determining an expected contribution of each echo reference to echo mitigation performed by at least one echo management system of at least one audio device of the audio environment. The echo management system(s) may, for example, include an acoustic echo canceller (AEC) and/or an acoustic echo suppressor (AES).

The method may involve selecting, by the control system and based at least in part on the importance estimate, one or more echo references from the plurality of echo references. The selected echo references may be a subset of one or more echo references from the (entire) plurality of echo references. The method may involve providing, by the control system, the one or more selected echo references to the at least one echo management system. In some examples, the method may involve causing at least one echo management system to cancel or suppress echoes based at least in part on the one or more selected echo references.

According to some examples, the audio devices of the audio system may be communicatively coupled via a wired or wireless communication network. The plurality of echo references (e.g., non-local echo references of other audio devices different from the first audio device and/or an echo reference of the first audio device) may be obtained via the wired or wireless communication network.

According to some examples, obtaining the plurality of echo references may involve receiving a content stream including audio data and determining one or more echo references of the plurality of echo references based on the audio data.

In some embodiments, the control system may be or include an audio device control system of an audio device in the audio environment. In some such embodiments, the method may involve rendering, by the audio device control system, audio data for reproduction on an audio device, thereby generating a local speaker feed signal. In some such embodiments, the method may involve determining a local echo reference corresponding to the local speaker feed signal.

In some examples, obtaining a plurality of echo references may involve determining one or more non-local echo references based on the audio data. In some such examples, each non-local echo reference may correspond to a non-local speaker feed signal for playback on another audio device of the audio environment.

According to some examples, obtaining a plurality of echo references may involve receiving one or more non-local echo references. In some such examples, each non-local echo reference may correspond to a non-local speaker feed signal for playback on another audio device of the audio environment. In some examples, receiving one or more non-local echo references may involve receiving one or more non-local echo references from one or more other audio devices of the audio environment. In some examples, receiving one or more non-local echo references may involve receiving each of the one or more non-local echo references from a single other device of the audio environment.

In some examples, the method may involve cost determination. According to some examples, the cost determination may involve determining a cost of at least one echo reference of a plurality of echo references. In some examples, selecting the one or more selected echo references may be based at least in part on the cost determination.

According to some examples, the cost determination can be based on network bandwidth required for transmitting the at least one echo reference, encoding computational requirements for encoding the at least one echo reference, decoding computational requirements for decoding the at least one echo reference, echo management system computational requirements for using the at least one echo reference by the echo management system, or one or more combinations thereof.

In some examples, the cost determination may be based on a replica of the at least one echo reference in the time domain or the frequency domain, a downsampled version of the at least one echo reference, a lossy compression of the at least one echo reference, segment power information of the at least one echo reference, or one or more combinations thereof. According to some examples, the cost determination may be based on a method of compressing relatively more important echo references less than relatively less important echo references.

In some examples, the method may involve determining a current echo management system performance level.According to some examples, selecting the one or more selected echo references may be based at least in part on the current echo management system performance level.

According to some examples, making the importance estimate may involve determining an importance metric for the corresponding echo reference. In some such examples, determining the importance metric may involve determining a level of the corresponding echo reference, determining a uniqueness of the corresponding echo reference, determining a temporal duration of the corresponding echo reference, determining an audibility of the corresponding echo reference, or one or more combinations thereof.

In some examples, determining the importance metric can be based at least in part on data or metadata corresponding to an audio device layout, loudspeaker metadata, metadata corresponding to received audio data, an upmixing matrix, a loudspeaker activation matrix, or one or more combinations thereof.

According to some examples, determining the importance metric may be based at least in part on current listening goals, a current ambient noise estimate, an estimate of a current performance of the at least one echo management system, or one or more combinations thereof.

Some or all of the operations, functions, and/or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof.

The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and descriptions. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale.

BRIEF DESCRIPTION OF THE DRAWINGS

Like reference numbers and designations throughout the various drawings indicate like elements.

FIG. 1A is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

FIG. 1B shows an example of an audio environment.

1C and 1D show examples of how audio devices 110A- 110C may receive playback channels.

FIG. 1E shows another example of an audio environment.

2A presents a block diagram of an audio device capable of performing at least some disclosed embodiments.

2B and 2C illustrate additional examples of audio devices in an audio environment.

FIG3A presents a block diagram illustrating components of an audio device according to one example.

3B and 3C are graphs showing examples of expected echo management performance versus the number of echo references used for echo management.

FIG4 presents a block diagram illustrating components of an echo reference organizer according to one example.

FIG. 5A is a flow chart outlining one example of the disclosed method.

FIG. 5B is a flow chart outlining another example of the disclosed method.

FIG. 6 is a flow chart outlining one example of the disclosed method.

FIG. 7 shows an example of a floor plan of an audio environment, which in this example is a living space.

DETAILED DESCRIPTION

Fig. 1A is a block diagram showing an example of a component of a device that can implement various aspects of the present disclosure. As with other figures provided herein, the type and quantity of the elements shown in Fig. 1A are provided only as examples. Other embodiments may include more, less and/or different types and quantities of elements. According to some examples, device 50 may be configured to perform at least some of the methods disclosed herein. In some embodiments, device 50 may be or may include one or more components of an audio system. For example, in some embodiments, device 50 may be an audio device, such as an intelligent audio device. In other examples, device 50 may be a mobile device (such as a cellular phone), a laptop computer, a tablet computer device, a television, or other types of devices.

According to some alternative embodiments, the apparatus 50 may be or may include a server. In some such examples, the apparatus 50 may be or may include an encoder. Thus, in some instances, the apparatus 50 may be a device configured for use within an audio environment such as a home audio environment, whereas in other instances, the apparatus 50 may be a device configured for use in the "cloud", e.g., a server.

In this example, the device 50 includes an interface system 55 and a control system 60. In some embodiments, the interface system 55 can be configured to communicate with one or more other devices of the audio environment. In some examples, the audio environment can be a home audio environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 55 can be configured to exchange control information and associated data with the audio devices of the audio environment. In some examples, the control information and associated data can be related to one or more software applications being executed by the device 50.

In some embodiments, the interface system 55 may be configured to receive a content stream or to provide a content stream. The content stream may include audio data. The audio data may include but may not be limited to an audio signal. In some instances, the audio data may include spatial data such as channel data and/or spatial metadata. For example, metadata may be provided by a device that may be referred to as an "encoder" herein. In some examples, the content stream may include video data and audio data corresponding to the video data.

The interface system 55 may include one or more network interfaces and/or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 55 may include one or more wireless interfaces. The interface system 55 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and/or a gesture sensor system. In some examples, the interface system 55 may include one or more interfaces between the control system 60 and a memory system (such as the optional memory system 65 shown in Figure 1A). However, in some instances, the control system 60 may include a memory system. In some embodiments, the interface system 55 may be configured to receive input from one or more microphones in the environment.

For example, the control system 60 may include a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and/or discrete hardware components.

In some embodiments, the control system 60 may reside in more than one device. For example, in some embodiments, a portion of the control system 60 may reside in a device within one of the environments depicted herein, and another portion of the control system 60 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smart phone or a tablet computer), etc. In other examples, a portion of the control system 60 may reside in a device within one of the environments depicted herein, and another portion of the control system 60 may reside in one or more other devices of the environment. For example, the functionality of the control system may be distributed across multiple smart audio devices of the environment, or may be shared by an orchestration device (such as a device that may be referred to as a smart home hub herein) and one or more other devices of the environment. In other examples, a portion of the control system 60 may reside in a device (such as a server) that implements a cloud-based service, and another portion of the control system 60 may reside in another device (such as another server, a memory device, etc.) that implements a cloud-based service. In some examples, the interface system 55 may also reside in more than one device.

In some embodiments, the control system 60 may be configured to at least partially perform the method disclosed herein. According to some examples, the control system 60 may be configured to obtain multiple echo references. The multiple echo references may include at least one echo reference for each audio device in the multiple audio devices in the audio environment. Each echo reference may, for example, correspond to audio data played back by one or more loudspeakers of an audio device in the multiple audio devices.

In some embodiments, the control system 60 can be configured to make an importance estimate for each of the plurality of echo references. In some examples, making the importance estimate can involve determining an expected contribution of each echo reference to echo mitigation by at least one echo management system of at least one audio device of the audio environment. The echo management system(s) can include an acoustic echo canceller (AEC) and/or an acoustic echo suppressor (AES).

According to some examples, the control system 60 may be configured to select the one or more selected echo references based at least in part on the importance estimate.In some examples, the control system 60 may be configured to provide the one or more selected echo references to at least one echo management system.

Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transient media. Such non-transient media may include memory devices such as memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transient media may, for example, reside in the optional memory system 65 and/or control system 60 shown in Figure 1A. Therefore, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transient media on which software is stored. For example, the software may include instructions for controlling at least one device to perform some or all of the methods disclosed herein. For example, software may be executed by one or more components of a control system such as the control system 60 of Figure 1A.

In some examples, the device 50 may include an optional microphone system 70 shown in FIG. 1A. The optional microphone system 70 may include one or more microphones. According to some examples, the optional microphone system 70 may include a microphone array. In some examples, the microphone array may be configured to determine direction of arrival (DOA) and/or time of arrival (TOA) information, for example, according to instructions from the control system 60. In some instances, the microphone array may be configured to perform receiving side beamforming, for example, according to instructions from the control system 60. In some embodiments, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.). In some examples, the device 50 may not include the microphone system 70. However, in some such embodiments, the device 50 may still be configured to receive microphone data of one or more microphones in the audio environment via the interface system 60. In some such embodiments, the cloud-based implementation of the device 50 may be configured to receive microphone data or data corresponding to the microphone data from one or more microphones in the audio environment via the interface system 60.

According to some embodiments, the device 50 may include the optional loudspeaker system 75 shown in FIG. 1A. The optional loudspeaker system 75 may include one or more loudspeakers, which may also be referred to herein as "speakers," or more generally as "audio reproduction transducers." In some examples (e.g., cloud-based embodiments), the device 50 may not include the loudspeaker system 75.

In some embodiments, the device 50 may include the optional sensor system 80 shown in Figure 1A. The optional sensor system 80 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some embodiments, the optional sensor system 80 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of the optional sensor system 80 may reside in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of the optional sensor system 80 may reside in a television, a mobile phone, or a smart speaker. In some examples, the device 50 may not include the sensor system 80. However, in some such embodiments, the device 50 may still be configured to receive sensor data of one or more sensors in the audio environment via the interface system 60.

In some embodiments, the device 50 may include an optional display system 85 shown in Figure 1A. The optional display system 85 may include one or more displays, such as one or more light emitting diode (LED) displays. In some instances, the optional display system 85 may include one or more organic light emitting diode (OLED) displays. In some examples, the optional display system 85 may include one or more displays of a smart audio device. In other examples, the optional display system 85 may include a television display, a laptop computer display, a mobile device display, or another type of display. In some examples where the device 50 includes a display system 85, the sensor system 80 may include a touch sensor system and/or a gesture sensor system close to one or more displays of the display system 85. According to some such embodiments, the control system 60 may be configured to control the display system 85 to present one or more graphical user interfaces (GUIs).

According to some such examples, the apparatus 50 may be or may include a smart audio device. In some such implementations, the apparatus 50 may be or may include a wake-up word detector. For example, the apparatus 50 may be or may include a virtual assistant.

For stereo or mono playback media, it is traditionally rendered into an audio environment (e.g., living space, car, office space, etc.) via a pair of speakers connected to an audio player (e.g., CD/DVD player, television (TV), etc.) via physical cables. With the popularity of smart speakers, users typically have more than two audio devices (which may include but are not limited to smart speakers or other smart audio devices) configured for wireless communication that are capable of playing back audio in their homes (or other audio environments).

Smart speakers are typically configured to operate based on voice commands. Therefore, such smart speakers are typically configured to continuously listen for a wake word, which will typically be followed by a voice command. Any continuous listening tasks (such as waiting for the wake word or performing any type of "continuous calibration") will preferably continue to run during content playback (such as music playback, movie and TV show soundtrack playback, etc.) and when device interaction occurs (for example, during a phone call). Audio devices that need to listen during content playback typically need to employ some form of echo management, such as echo cancellation and/or echo suppression, to remove the "echo" (the content played by the device) from the microphone signal.

Fig. 1B shows an example of an audio environment. As with other figures provided herein, the types, quantities, and arrangements of the elements shown in Fig. 1B are provided only as examples. Other embodiments may include more, fewer, and/or different types, quantities, and/or arrangements of elements.

According to this example, the audio environment 100 includes audio devices 110A, 110B, and 110C. In this example, each of the audio devices 110A-110C is an instance of the apparatus 50 of FIG. 1A and includes an instance of the microphone system 70 and the loudspeaker system 75, but these are not shown in FIG. 1B. According to some examples, each audio device 110A-110C can be a smart audio device, such as a smart speaker.

In this example, audio devices 110A-110C play back audio content while person 130 is speaking. The microphone of audio device 110B detects not only the audio content played back by its own speaker, but also the voice sound 131 of person 130 and the audio content played back by audio devices 110A and 110C.

In order to utilize as many speakers as possible simultaneously, the typical approach is to have all audio devices in the audio environment play back the same content and use some kind of timing mechanism to keep the playback media synchronized. This has the advantage of making distribution simple, because all devices receive the same copy of the playback media, whether it is downloaded or streamed to each audio device, or broadcast by one device and multicast to all audio devices.

A major drawback of this approach is that spatial effects cannot be obtained. Spatial effects can be achieved by adding more playback channels (e.g., one for each speaker), for example, by upmixing. In some examples, spatial effects can be achieved via flexible rendering processes such as center of mass amplitude translation (CMAP), flexible virtualization (FV), or a combination of CMAP and FV. Related examples of CMAP, FV, and combinations thereof are described in International Patent Publication No. WO 2021/021707A1 (e.g., pages 25-41), which is hereby incorporated by reference.

1C and 1D illustrate additional examples of audio devices in an audio environment. According to these examples, the audio environment 100 includes a smart home hub 105 and audio devices 110A, 110B, and 110C. In these examples, the smart home hub 105 and the audio devices 110A-110C are instances of the apparatus 50 of FIG. 1A. According to these examples, each of the audio devices 110A-110C includes a corresponding one of the loudspeakers 121A, 121B, and 121C. According to some examples, each of the audio devices 110A-110C can be a smart audio device, such as a smart speaker.

Fig. 1C and Fig. 1D show examples of how audio devices 110A-110C can receive playback channels. In Fig. 1C, the encoded audio bitstream is multicast to all audio devices 110A-110C. In Fig. 1D, each of the audio devices 110A-110C only receives the channels required for playback by the audio device. The selection of bitstream distribution can change according to a separate implementation, and can be based on available system bandwidth, the codec efficiency of the audio codec used, the ability of audio devices 110A-110C and/or other factors. The exact topology of the audio environment shown in Fig. 1C and Fig. 1D is not important. However, these examples illustrate the fact that audio channels will be distributed to device audio devices and some costs will be generated. Cost can be evaluated from aspects such as required network bandwidth, the increased computational cost of encoding and decoding the audio channels.

FIG. 1E shows another example of an audio environment. According to this example, the audio environment 100 includes audio devices 110A, 110B, 110C, and 110D. In this example, each of the audio devices 110A-110D is an instance of the apparatus 50 of FIG. 1A and includes at least one microphone (see microphones 120A, 120B, 120C, and 120D), at least one loudspeaker (see loudspeakers 121A, 121B, 121C, and 121D). According to some examples, each audio device 110A-110D can be an intelligent audio device, such as an intelligent speaker.

In this example, the audio devices 110A-110D render content 122A, 122B, 122C, and 122D via loudspeakers 121A-121D. Each of the microphones 120A-120D detects an "echo" corresponding to the content 122A-122D played back by each of the audio devices 110A-110D. In this example, the audio devices 110A-110D are configured to listen for a command or wake-up word in the voice 131 from a person 130 within the audio environment 100.

Fig. 2A presents a block diagram of an audio device that can perform at least some disclosed embodiments. As with other figures provided herein, the type, quantity and arrangement of the elements shown in Fig. 2A are provided only as examples. Other embodiments may include elements of more, less and/or different types, quantities and/or arrangements. In this example, audio device 110A is an example of the audio device 110A of Fig. 1E. Here, audio device 110A includes control system 60a, which is an example of the control system 60 of Fig. 1A. According to this embodiment, control system 60 can listen to the voice 131 of a person 130 in the presence of an echo corresponding to the content 122A, 122B, 122C and 122D played back by each audio device in audio environment 100.

According to this example, the control system 60 implements a renderer 201A, a multi-channel acoustic echo management system (MC-EMS) 203A, and a voice processing block 240A. The MC-EMS 203A may include an acoustic echo canceller (AEC), an acoustic echo suppressor (AES), or both AEC and AES, depending on the specific implementation. According to this example, the voice processing block 240A is configured to detect the user's wake-up words and commands. In some embodiments, the voice processing block 240A may be configured to support communication sessions, such as telephone calls.

In this embodiment, the renderer 201A is configured to provide a local echo reference 220A to the MC-EMS 203A. The local echo reference 220A corresponds to (and is equivalent to in this example) a speaker feed signal provided to the loudspeaker 121A for playback by the audio device 110A. According to this example, the renderer 201A is also configured to provide a non-local echo reference 221A (corresponding to content 122B, 122C, and 122D played back by other audio devices in the audio environment 100) to the MC-EMS 203A.

According to some examples, the audio device 110A receives a combined bitstream of audio data including all audio devices 110A-110D of FIG. 1E (e.g., as shown in FIG. 1C ). In some such examples, the renderer 201A may be configured to separate the local echo reference 220A from the non-local echo reference 221A to provide the local echo reference 220A to the loudspeaker 121A, and to provide the local echo reference 220A and the non-local echo reference 221A to the MC-EMS 203A. In some alternative examples, the audio device 110A may receive a bitstream intended only for playback on the audio device 110A, for example, as shown in FIG. 1D . In some such examples, the smart home hub 105 (or other audio devices 110B-D) may provide the non-local echo reference 221A to the audio device 110A, as shown by the dashed arrow next to the reference numeral 221A in FIG. 2A .

In some instances, the local echo reference 220A and/or the non-local echo reference 221A may be a full-fidelity replica of the speaker feed signals provided to the loudspeakers 121A-121D for playback. In some alternative examples, the local echo reference 220A and/or the non-local echo reference 221A may be a lower fidelity representation of the speaker feed signals provided to the loudspeakers 121A-121D for playback. In some such examples, the non-local echo reference 221A may be a downsampled version of the speaker feed signals provided to the loudspeakers 121B-121D for playback. According to some examples, the non-local echo reference 221A may be a lossy compression of the speaker feed signals provided to the loudspeakers 121B-121D for playback. In some examples, the non-local echo reference 221A may be banded power information corresponding to the speaker feed signals provided to the loudspeakers 121B-121D for playback.

According to this embodiment, MC-EMS 203A is configured to use local echo reference 220A and non-local echo reference 221A to predict and eliminate and/or suppress echo from microphone signal 223A, thereby generating residual signal 224A, in which the speech echo ratio (SER) may have been improved relative to microphone signal 223A. The residual signal 224A can enable the voice processing block 240A to detect user wake-up words and commands. In some embodiments, the voice processing block 240A can be configured to support a communication session, such as a telephone call.

Some aspects of the present disclosure involve making an importance estimate for each echo reference in a plurality of echo references (e.g., for local echo reference 220A and non-local echo reference 221A). Making the importance estimate may involve determining an expected contribution of each echo reference to echo mitigation performed by at least one echo management system of at least one audio device of an audio environment (e.g., echo mitigation performed by MC-EMS 203A of audio device 110A). Various examples are provided below.

In the context of distributed and orchestrated devices, for purposes of echo management, according to some examples, each audio device may, in addition to its own echo reference, also obtain an echo reference corresponding to content played back by one or more other audio devices in the audio environment. The impact of including a particular echo reference in a local echo management system or "EMS" (e.g., MC-EMS 203A of audio device 110A) may vary depending on a number of parameters, such as the diversity of the audio content being played, the network bandwidth required for transmitting the echo reference, the encoding computational requirements for encoding the echo reference in the case of transmitting an encoded echo reference, the decoding computational requirements for decoding the echo reference, the echo management system computational requirements for use of the echo reference by the echo management system, the relative audibility of the audio devices, etc.

For example, if each audio device is rendering the same content (in other words, if mono audio is being played back), there is little (although non-zero) benefit to providing additional references to the EMS. Furthermore, due to practical limitations (such as bandwidth-constrained networks), it may not be desirable for all devices to share a copy of their local echo reference. Therefore, some embodiments may provide a distributed and orchestrated EMS (DOEMS) in which echo references are prioritized and transmitted (or not) accordingly. Some such examples may implement a trade-off between the cost of each additional echo reference (e.g., the required network bandwidth and/or the required computational overhead) and the benefit (e.g., the expected echo mitigation improvement, which may be measured in terms of signal-to-echo ratio (SER) and/or echo loss enhancement (ERLE)).

2B and 2C show additional examples of audio devices in an audio environment. According to these examples, the audio environment 100 includes a smart home hub 105 and audio devices 110A, 110B, and 110C. In these examples, the smart home hub 105 and the audio devices 110A-110C are instances of the apparatus 50 of FIG. 1A. According to these examples, each of the audio devices 110A-110C includes a corresponding microphone 120A, 120B, and 120C and a corresponding loudspeaker 121A, 121B, and 121C. According to some examples, each audio device 110A-110C can be a smart audio device, such as a smart speaker.

In Figure 2B, the smart home hub 105 sends the same encoded audio bitstream to all audio devices 110A-110C. In Figure 2C, the smart home hub 105 sends only the audio channels required by each audio device 110A-110C for playback. In both examples, audio channel 0 is intended for playback on audio device 110A, audio channel 1 is intended for playback on audio device 110B, and audio channel 2 is intended for playback on audio device 110C.

2B and 2C show examples of sharing echo reference data on a local network. In these examples, audio device 110A sends echo reference 220A' to audio devices 110B and 110C via a local network, which is an echo reference corresponding to the loudspeaker playback of audio device 110A. In these examples, echo reference 220A' is different from the channel 0 audio found in the bitstream. In some instances, echo reference 220A' may be different from channel 0 audio because playback post-processing is implemented on audio device 110A. In the example shown in FIG. 2C, instead of providing a combined bitstream to all audio devices 110A-110C, another device (such as audio device 110A or smart home hub 105) provides echo reference 220A'. In the scenario depicted in FIG. 2B, even if the combined bitstream is provided to all audio devices 110A-110C, in some such instances, it may still be necessary to transmit echo reference 220A'.

In other examples, the echo reference 220A' may differ from the channel 0 audio because the echo reference 220A' may not be a full-fidelity replica of the audio data played back on the audio device 110A. In some such examples, the echo reference 220A' may correspond to the audio data played back on the audio device 110A, but may require relatively less data than a full replica and, therefore, may consume relatively less local network bandwidth when transmitting the echo reference 220A'.

According to some such examples, audio device 110A may be configured to generate a downsampled version of local echo reference 220A described above with reference to FIG. 2A. In some such examples, echo reference 220A' may be or may include the downsampled version.

In some examples, the audio device 110A may be configured to lossily compress the local echo reference 220A. In such an instance, the echo reference 220A' may be the result of the control system 60a applying a lossy compression algorithm to the local echo reference 220A.

According to some examples, audio device 110A may be configured to provide audio devices 110B and 110C with segment power information corresponding to local echo reference 220A. In some such examples, instead of transmitting a full-fidelity replica of the audio data played back on audio device 110A, control system 60a may be configured to determine the power level in each of a plurality of frequency bands of the audio data played back on audio device 110A and transmit the corresponding segment power information to audio devices 110B and 110C. In some such examples, echo reference 220A' may be or may include the segment power information.

Fig. 3 A presents the block diagram showing the parts of the audio equipment according to an example.As with other figures provided herein, the type, quantity and arrangement of the key element shown in Fig. 3 A are provided only as examples.Other embodiments may include more, less and/or key elements of different types, quantities and/or arrangements.For example, some embodiments may be configured to send and/or receive a "original" echo reference (which may be a complete full-fidelity replica of the audio reproduced on the audio equipment), a low-fidelity version or representation of the audio reproduced on the audio equipment (such as a down-sampled version, a version produced by lossy compression or the segmented power information corresponding to the audio reproduced on the audio equipment), but do not send and/or receive the original version and the low-fidelity version simultaneously.

In this example, audio device 110A is an instance of audio device 110A of FIG. 1E and includes control system 60a, which is an instance of control system 60 of FIG. 1A. According to this example, control system 60a is configured to implement renderer 201A, multi-channel acoustic echo management system (MC-EMS) 203A, speech processing block 240A, echo reference arranger 302A, decoder 303A and noise estimator 304A. The reader may assume that MC-EMS 203A and speech processing block 240A function as described above with reference to FIG. 2A, unless otherwise indicated by the following description of FIG. 3A. In this example, network interface 301A is an instance of interface system 55 described above with reference to FIG. 1A.

In this example, the elements of FIG. 3A are as follows:

110A: Audio equipment;

120A: Representative microphone. In some embodiments, the audio device 110A may have more than one microphone;

121A: Representative loudspeaker. In some embodiments, the audio device 110A may have more than one loudspeaker;

201A: A renderer that generates a reference for local playback and an echo reference that simulates audio played back by other audio devices in the audio environment;

203A: A multi-channel acoustic echo management system (MC-EMS), which may include an acoustic echo canceller (AEC) and/or an acoustic echo suppressor (AES);

220A: Local echo reference for playback and cancellation;

221A: A locally generated copy of an echo reference being played by one or more non-local audio devices (one or more other audio devices in the audio environment);

223A: Multiple microphone signals;

224A: multiple residual signals (microphone signals after MC-EMS203A eliminates and/or suppresses the predicted echo);

240A: a voice processing block configured for wake-up word detection, voice command detection and/or providing telephone communication;

301A: A network interface configured for communication between audio devices, which may also be configured for communication via the Internet and/or via one or more cellular networks;

302A: An echo reference orchestrator configured to rank the echo references and select an appropriate set of one or more echo references;

303A: audio decoder block;

304A: Noise estimator block;

310A: one or more decoded echo references received by the audio device 110A from one or more other devices in the audio environment;

311A: Sending a request for an echo reference from one or more other devices (such as a smart home hub or one or more of the audio devices 110B-110D) via a local network;

312A: Metadata, which may be or may include metadata corresponding to an audio device layout, loudspeaker metadata, metadata corresponding to received audio data, an upmix matrix, and/or a loudspeaker activation matrix;

313A: echo reference selected by echo reference arranger 302A;

314A: echo reference received by device 110A from one or more other devices;

315A: echo reference sent from device 110A to other devices;

316A: raw echo reference received by device 110A from one or more other devices in the audio environment;

317A: a low-fidelity (eg, decoded) version of an echo reference received by device 110A from one or more other devices of the audio environment;

318A: Audio environment noise estimation;

350A: One or more indicators indicating the current performance of the MC-EMS 203A, which may be or may include adaptive filter coefficient data or other AEC statistics, speech echo (SER) ratio data, etc.

Depending on the particular implementation, the echo reference organizer 302A can function in various ways. Many examples are disclosed herein. In some examples, the echo reference organizer 302A can be configured to make an importance estimate for each of the plurality of echo references (e.g., for the local echo reference 220A and the non-local echo reference 221A). Making the importance estimate can involve determining the expected contribution of each echo reference to the echo mitigation performed by at least one echo management system of at least one audio device of the audio environment (e.g., the echo mitigation performed by the MC-EMS 203A of the audio device 110A).

Some examples of making importance estimates may involve determining an importance metric. In some such examples, the importance metric may be based at least in part on one or more characteristics of each echo reference, such as level, uniqueness, temporal persistence, audibility, or one or more combinations thereof. In some examples, the importance metric may be based at least in part on metadata (e.g., metadata 312A), such as metadata corresponding to an audio device layout, loudspeaker metadata, metadata corresponding to received audio data, an upmix matrix, a loudspeaker activation matrix, or one or more combinations thereof. In some examples, the importance metric may be based at least in part on current listening objectives, current ambient noise estimates, estimates of current performance of at least one echo management system, or one or more combinations thereof.

According to some examples, the echo reference scheduler 302A can be configured to select a set of one or more echo references based at least in part on a cost determination. In some examples, the echo reference scheduler 302A can be configured to perform the cost determination, while in other examples, another block of the control system 60a can be configured to perform the cost determination. In some instances, the cost determination can involve determining the cost of at least one echo reference in a plurality of echo references, or in some cases determining the cost of each of a plurality of echo references. In some examples, the cost determination can be based on the network bandwidth required for transmitting the echo references, the encoding computational requirements for encoding the at least one echo reference, the decoding computational requirements for decoding the at least one echo reference, the downsampling cost of making a downsampled version of the echo reference, the echo management system computational requirements for using the at least one echo reference by the echo management system, or one or more combinations thereof.

According to some examples, the cost determination can be based on a replica of the at least one echo reference in the time domain or frequency domain, a downsampled version of the at least one echo reference, a lossy compression of the at least one echo reference, segmented power information of the at least one echo reference, or one or more combinations thereof. In some instances, the cost determination can be based on a method of compressing relatively more important echo references less than relatively less important echo references. In some embodiments, the echo reference organizer 302A (or another block of the control system 60a) can be configured to determine a current echo management system performance level (e.g., based at least in part on the indicator(s) 350A). In some such examples, selecting one or more selected echo references can be based at least in part on the current echo management system performance level.

Depending on the distributed audio device system, its configuration and the type of audio session (e.g., communication or listening to music) and/or the nature of the rendered content, the rate at which the importance of each echo reference is estimated and the rate at which the set of echo references is evaluated may be different. Furthermore, the rate at which the importance is estimated need not be equal to the rate at which the echo reference selection process makes decisions. If the two are not synchronized, then in some examples, the importance calculations will be more frequent. In some instances, echo reference selection can be a discrete process in which a binary decision is made to include or not include a particular echo reference.

3B and 3C are graphs showing examples of expected echo management performance versus the number of echo references used for echo management. In FIG. 3B , it can be seen that as additional references are added, the expected echo performance also improves. However, in this example, it can be seen that there are only a few discrete points at which the system can operate. In some examples, the points shown in FIG. 3B may correspond to processing a complete, full-fidelity replica of each echo reference. For example, point 301 may correspond to an instance of processing a local echo reference (e.g., local reference 220A of FIG. 2A or FIG. 3A ), and point 310 may correspond to receiving a complete replica of a first non-local echo reference (e.g., a full-fidelity version of one of the received echo references 314A of FIG. 3A , which may have been selected as the most important non-local echo reference) and processing both the local echo reference and the complete replica of the first non-local echo reference.

FIG3C illustrates an example of operating between any two of the discrete operating points shown in FIG3B . The line connecting the points in FIG3B may, for example, correspond to a range of echo reference fidelity, including lower fidelity versions or representations of each echo reference. For example, points 303, 305, and 307 may correspond to copies or representations of the first non-local echo reference with increased fidelity levels, wherein point 303 corresponds to the lowest fidelity representation and point 307 corresponds to the highest fidelity representation other than the full fidelity replica. In some examples, point 303 may correspond to segmented power information of the first non-local echo reference. According to some examples, points 305 and 307 may correspond to relatively high lossy compression of the first non-local echo reference and relatively less lossy compression of the first non-local echo reference, respectively.

The fidelity of a copy or representation of an echo reference is generally inversely proportional to the number of bits required for each such copy or representation. Thus, the fidelity of a copy or representation of an echo reference provides an indication of the tradeoff between network cost (due to the different number of bits required for transmission) and expected echo management performance (since performance should improve with increasing fidelity). Note that the straight line connecting the points in FIG. 3C represents only one of many different possible trajectories, in part because the incremental change from one echo reference to the next depends on which echo reference will be selected as the next echo reference, and in part because there may not be a linear relationship between expected echo management performance and fidelity.

Fig. 4 presents a block diagram showing the components of an echo reference organizer according to an example. As with other figures provided herein, the types, quantities and arrangements of the elements shown in Fig. 4 are provided only as examples. Other embodiments may include more, fewer and/or elements of different types, quantities and/or arrangements. For example, some embodiments may be configured to send and/or receive a "original" echo reference (which may be a full-fidelity replica of the audio reproduced on an audio device), a low-fidelity version or representation of the audio reproduced on the audio device (such as a downsampled version, a version produced by lossy compression, or segmented power information corresponding to the audio reproduced on the audio device), but do not send and/or receive the original version and the low-fidelity version simultaneously.

In this example, the echo reference organizer 302A is an instance of the echo reference organizer 302A of Figure 3A and is implemented by an instance of the control system 60a of Figure 3A. According to this example, the elements of Figure 4 are as follows:

220A: Local echo reference for playback and cancellation;

221A: A locally generated copy of a non-local echo reference being played by another audio device of the audio environment;

302A: an echo reference orchestrator, a module configured to rank and select a set of one or more echo references;

310A: one or more decoded echo references received by the audio device 110A from one or more other devices in the audio environment;

311A: Sending a request for an echo reference from one or more other devices in the audio environment via a local network;

312A: Metadata, which may be or may include metadata corresponding to an audio device layout, loudspeaker metadata, metadata corresponding to received audio data, an upmix matrix, and/or a loudspeaker activation matrix;

313A: in this example, a set of one or more echo references selected by the echo reference scheduler 302A and sent to the MC-EMS 203A;

316A: raw echo reference received by device 110A from one or more other devices in the audio environment;

317A: a low-fidelity (eg, decoded) version of an echo reference received by device 110A from one or more other devices of the audio environment;

318A: Audio environment noise estimation;

350A: One or more indicators indicating the current performance of the MC-EMS 203A, which may be or may include adaptive filter coefficient data or other AEC statistics, speech echo (SER) ratio data, etc.

401A: an echo reference importance estimator configured to estimate the expected importance of each echo reference and, in this example, generate a corresponding importance measure 420A;

402: an echo reference selector configured to select an echo reference set 313A based at least in part on a current listening goal (as shown in 421A), a cost of each echo reference (as shown in 422A), a current state/performance of the EMS (as shown in 350A), and an estimated importance of each candidate echo reference (as shown in importance measure 420A) in this example;

403A: A cost estimation module configured to determine cost(s) (eg, computational and/or network costs) of including an echo reference in the echo reference set 313A;

404A: Optional module, used to determine or estimate the current listening target of the audio device 110A;

405A: A module configured to implement one or more MC-EMS performance models, which in some examples may generate data such as shown in FIG. 3B or FIG. 3C ;

420A: Importance measure 420A generated by echo reference importance estimator 401A;

421A: Information indicating the current listening target;

422A: Information indicating the cost(s) of including the echo reference in the echo reference set 313A; and

423A: Information generated by the MC-EMS performance model 405A, which in some examples may be or include data such as shown in FIG. 3B or FIG. 3C .

Depending on the particular implementation, the echo reference importance estimator 401A may function in various ways. Various examples are provided in the present disclosure. In some examples, the echo reference importance estimator 401A may be configured to make an importance estimate for each of the plurality of echo references (e.g., for the local echo reference 220A and the non-local echo reference 221A). Making the importance estimate may involve determining an expected contribution of each echo reference to echo mitigation performed by at least one echo management system of at least one audio device of the audio environment (e.g., echo mitigation performed by the MC-EMS 203A of the audio device 110A).

In this example, making an importance estimate involves determining an importance metric 420A. The importance metric 420A may be based at least in part on one or more characteristics of each echo reference, such as level, uniqueness, time duration, audibility, or one or more combinations thereof. In some examples, the importance metric may be based at least in part on metadata (e.g., metadata 312A), which may include metadata corresponding to an audio device layout, loudspeaker metadata (e.g., sound pressure level (SPL) rating, frequency range, whether the loudspeaker is an upward-firing loudspeaker, etc.), metadata corresponding to received audio data (e.g., position metadata, metadata indicating a human voice or other speech, etc.), an upmix matrix, a loudspeaker activation matrix, or one or more combinations thereof. In some instances, as shown by dashed arrow 420A, the echo reference importance estimator 401A may provide the importance metric 420A to the MC-EMS performance model 405A.

According to this example, importance metric 420A is based at least in part on the current listening goal, as indicated by information 421 A. As described in more detail below, the current listening goal can significantly change how factors such as level, uniqueness, temporal duration, audibility, etc. are evaluated. For example, importance analysis during a phone call may be very different than when waiting for a wake word.

In this example, the importance metric 420A is based at least in part on the current ambient noise estimate 318A, the (multiple) indicators 350A indicating the current performance of the MC-EMS 203A, the information 423A generated by the MC-EMS performance model 405A, or one or more combinations thereof. In some embodiments, the echo reference importance estimator 401A can determine that if the room noise level is relatively high (as indicated by the current ambient noise estimate 318A), then adding the echo reference will not likely help significantly mitigate the echo. As described above, the information 423A can correspond to the type of information described above with reference to Figures 3B and 3C, which can provide a direct correlation between the use of the echo reference and the expected performance increase of the MC-EMS 203A. As described in more detail below, the performance of the EMS can be based in part on the robustness of the EMS when it is disturbed by noise in the audio environment.

According to this embodiment, the echo reference selector 402 selects a set of one or more echo references based at least in part on: one or more indicators 350A indicating the current performance of the MC-EMS 203A, the importance metric 420A, the current listening goal 421A, information 422A indicating the cost(s) of including the echo reference in the echo reference set 313A, and information 423A generated by the MC-EMS performance model 405A. Some detailed examples of how the echo reference selector 402 can select echo references are provided below.

In this example, the cost estimation module 403A is configured to determine the computational and/or network cost of including the echo reference in the echo reference set 313A. The computational cost may, for example, include the additional computational cost of using a particular echo reference by the MC-EMS 203A. The computational cost may in turn depend on the number of bits required to represent the echo reference. In some examples, the computational cost may include the computational cost of a lossy echo reference encoding process and/or the computational cost of a corresponding echo reference decoding process. Determining the network cost may involve determining the amount of data required to send a complete replica of the echo reference or a copy or representation of the echo reference across a local data network (e.g., a local wireless data network).

In some instances, the echo reference selection block 402A may generate and transmit a request 311A for another device in the audio environment to send one or more echo references to it over the network. (Element 314A of FIG. 3A indicates reception of one or more echo references by the audio device 110A, which in some instances may have responded to the request 311A). In some examples, the request 311A may specify the fidelity of the requested echo reference, such as whether a "raw" copy (full-fidelity replica) of the echo reference should be sent, whether an encoded version of the echo reference should be sent, whether a relatively more or less lossy compression algorithm should be applied to the echo reference if an encoded version of the echo reference should be sent, whether segment power information corresponding to the echo reference should be sent, etc.

One may note that the request for a coded echo reference not only introduces a network cost due to sending the request and reference, but also adds the computational cost of the responding device(s) (e.g., the smart home hub 105 or one or more of the audio devices 110B-110D) having to encode the reference, and the computational cost of the audio device 110A decoding the received reference. However, this encoding cost may be a one-time cost. Thus, sending a request for an encoded reference over the network from one audio device to another changes the potential performance/cost tradeoffs performed in other devices (e.g., in the audio devices 402C and 402D).

In some embodiments, one or more blocks of echo reference scheduler 302A may be performed by a scheduler device (e.g., smart home hub 105 or one of audio devices 110A-110D). According to some such embodiments, at least some functions of echo reference importance estimator 401A and/or echo reference selection block 402A may be performed by a scheduler device. Some such embodiments may be able to determine the cost/benefit tradeoff of the entire system taking into account the performance enhancement of all instances of MC-EMS in the audio environment, the overall computational requirements of all instances of MC-EMS, the overall requirements of the local network, and/or the overall computational requirements of all encoders and decoders.

Examples of various indicators and weights

Importance Metrics

In short, the importance metric (which may be referred to herein as "importance" or "I") may be a measure of the expected improvement in EMS performance due to the inclusion of a specific echo reference. In some embodiments, importance may depend on the current state of the EMS, particularly on the echo reference set already in use and at what fidelity level they are being received. Depending on the specific implementation, importance may be obtained on different time scales. In an extreme case, importance may be implemented frame by frame (e.g., according to an importance signal for each frame). In other examples, importance may be implemented as a constant value for the duration of a content segment, or as a constant value for the time of a specific configuration of an audio device being used. The configuration of an audio device may correspond to an audio device position and/or an audio device orientation.

Thus, the importance metric may be calculated at various time scales depending on the particular implementation, for example:

Real-time, for example, analyzing the current audio content based on events in the audio environment (such as an incoming call);

on a longer time scale, e.g., track by track, where a track corresponds to a piece of content such as a song or other piece of musical content that may last, e.g., on a time scale of several minutes; or

Only once, for example, when the audio system is initially configured or reconfigured.

Decisions about which echo references to select for echo management purposes may be made on a time scale similar to (or slower than) that of evaluating the importance metric. For example, a device or system might estimate importance every 30 seconds and make decisions about changing the selected echo reference every few minutes.

According to some examples, the control system can be configured to determine an importance matrix that can include all importance information for the current audio device system. In some such examples, the importance matrix can have dimensions N×M, including an entry for each audio device and an entry for each potential echo reference channel. In some such examples, N represents the number of audio devices and M represents the number of potential echo references. Since some audio devices may play back more than one channel, this type of importance matrix is not always square.

In some implementations, the importance metric I may be based on one or more of the following:

L: level of echo reference;

U: uniqueness of echo reference;

P: the temporal persistence of the echo reference, and/or

A: The audibility of the device rendering the echo reference.

As used herein, the acronym "LUPA" generally refers to echo reference characteristics from which importance metrics may be determined, including but not limited to one or more of L, U, P, and/or A.

L or "Horizontal" aspect

This aspect describes the level or loudness of the echo reference. Under the same other conditions, it is well known that the louder the playback signal, the greater the impact on EMS performance. As used herein, the term "level" refers to the level within the digital representation of the audio signal, and does not necessarily refer to the actual sound pressure level of the audio signal after reproduction via a loudspeaker. In some examples, the loudness of a single channel of the echo reference can be based on a root mean square (RMS) index or a LKFS (k-weighted loudness relative to full scale) index. Such an index is easy to calculate in real time on the echo reference, or can exist as metadata in a bitstream. According to some embodiments, L can be determined based on a volume setting, such as an audio system volume setting or a volume setting in a media application.

U or the "uniqueness" aspect

The uniqueness aspect aims to capture the amount of new information about the overall audio presentation that a particular echo reference provides. From a statistical point of view, multi-channel audio presentations often contain redundancy across channels. For example, this redundancy may arise due to instruments and other sound sources being replicated on channels on the left and right sides of the room, or the signal being panned and thus further replicated in multiple active loudspeakers simultaneously. Although such scenarios lead to EMSs that need to address overshoot issues (where the echo filter may infer observations from multiple echo paths), some benefits and higher performance can still be observed in practice.

U can be calculated or estimated in various ways. In some examples, U can be based at least in part on the correlation coefficient between each echo reference. In one such example, U can be estimated as follows:

Wherein the subscript "r" corresponds to the specific echo reference being evaluated, N represents the total number of audio devices in the audio environment, n represents a single audio device, M represents the total number of potential echo references in the audio environment, and m represents a single echo reference.

Alternatively or additionally, in some examples, U may be based at least in part on decomposing the audio signal to find redundancy. Some such examples may involve instantaneous frequency estimation, fundamental frequency (F0) estimation, spectrogram inversion, and/or non-negative matrix factorization (NMF).

According to some examples, U can be based at least in part on data for matrix decoding. Matrix decoding is an audio technique in which a small number of discrete audio channels (e.g., 2) are decoded into a large number of channels (e.g., 4 or 5) when played back. Channels are typically arranged for transmission or recording by an encoder and decoded by a decoder for playback. Matrix decoding allows multi-channel audio (such as surround sound) to be encoded as a stereo signal, played back as stereo on a stereo device, and played back as surround sound on a surround sound device. In one such example, if a Dolby 5.1 system is receiving a stereo audio data stream, a static up-mixing matrix can be applied to the stereo audio data to provide correctly rendered audio for each loudspeaker in the Dolby 5.1 system. According to some examples, U can be based at least in part on the coefficients of an up-mixing or down-mixing matrix for allocating audio to each loudspeaker (e.g., each of audio devices 110A-110D) in an audio environment.

In some examples, U can be based at least in part on a standard canonical loudspeaker layout used in an audio environment (e.g., Dolby 5.1, Dolby 7.1, etc.). Some such examples can involve utilizing the way media content is traditionally mixed and presented in such a canonical loudspeaker layout. For example, in a Dolby 5.1 or Dolby 7.1 system, artists typically place vocals in a center channel rather than surround channels. As described above, audio corresponding to musical instruments and other sound sources is typically reproduced on channels on the left and right sides of a room. In some instances, sounds, dialogues, instrumentals, etc. can be identified via metadata received with the corresponding audio data.

P or "persistence" aspect

The persistence metric is intended to capture the aspect that different types of playback media may have a wide range of temporal persistence, with different types of content having different degrees of silence and loudspeaker activation. A spectrally dense continuous stream of content (such as music or the audio output of a video game console) may have a high level of temporal persistence, while a podcast may have a lower level of temporal persistence. Infrequent system notifications will have a very low level of temporal persistence. Depending on the specific list task at hand, echo references corresponding to media with a lower degree of persistence may be less important to the EMS. For example, an occasional system notification is unlikely to conflict with a wake word or barge request, so the relative importance of managing that echo is lower.

The following are examples of metrics that can be used to measure or estimate persistence:

The percentage of time in the recent history window that the playback signal was above a certain numerical loudness threshold;

A metadata tag or media classification indication that the content corresponds to music, radio content, podcast, or system sound; and/or

The percentage of time during the recent history window that the playback signal was in the typical frequency range of human voice (e.g., 100 Hz to 3 KHz).

According to some examples, the audio content type may affect the estimation of L, U and/or P. For example, knowing that the audio content is stereo music would allow all echo references to be ranked using only the channel assignments described above. Alternatively, if the control system does not analyze the audio content, but relies on the channel assignments, knowing that the audio content is Atmos may change the default L, U and/or P assumptions.

A or "audibility" aspect

Audibility metrics address the fact that audio devices have different playback characteristics and that in any given audio environment, the distances between audio devices may vary. The following are examples of metrics that can be used to measure or estimate the audibility of audio devices:

Direct measurements of the audibility of audio equipment;

Refers to a data structure that includes characteristics of one or more loudspeakers of an audio device, such as rated SPL, frequency response, and directivity (e.g., whether the loudspeaker is omnidirectional, forward-firing, upward-firing, etc.);

Based on an estimate of the distance to the audio device; and/or

Any combination of the above.

Other factors may be evaluated for estimating importance, and in some instances for determining an importance metric.

Listening Target

Listening goals can define the context and desired performance characteristics of the EMS. In some examples, listening goals can modify the parameters and/or domains evaluated by LUPA. The following discussion will consider three potential contexts in which listening goals change. In these different contexts, we will see how probability and criticality can affect LUPA.

1. Interruption (e.g., detecting instances of wake-up words)

When waiting to be interrupted, there is no immediate urgency: it is generally assumed that the probability of the user saying the wake word is the same at all future time intervals. Furthermore, the wake word detector is probably the most robust element in a voice assistant, and the impact of echo leakage is less critical.

2. Command

After a person says the wake word, the probability that the person will say a command immediately is very high. Therefore, the probability of a collision with the echo in the near future is high. In addition, because the command recognition module may be relatively less robust than the wake word detector, the criticality of the echo leakage is usually high.

3. Communication

During a voice call, the probability that any of the participants (the person(s) in the audio environment and the person(s) at the far end) are talking to each other is deterministic. In other words, the probability that an echo collides with a user's voice is essentially 1. However, since the person(s) at the far end are human and cope well with background noise, this is less critical since they are unlikely to be bothered by echo leakage.

In the context of these different listening goals, the way LUPA is assessed may change in some examples.

1. Interject

There may be no temporal distinction, since the probability of saying the wake word at all future time intervals is assumed to be the same. Therefore, the time range over which the control system evaluates LUPA may be quite long in order to obtain better estimates of these parameters. In some such examples, the time interval over which the control system evaluates LUPA may be set to look relatively far into the future (e.g., within a time range of several minutes).

2. Command

The time interval immediately after the wake word is spoken is likely to be when a command is spoken. Therefore, after the wake word is detected, in some embodiments, LUPA can be evaluated on a much shorter time scale (e.g., on the order of seconds) than in the context of interjection. In some examples, references that are sparse in time and have content playing in the next few seconds after the wake word detection will be considered more important during this time interval because the likelihood of collision is high.

Fig. 5A is a flow chart summarizing an example of the disclosed method. As with other methods described herein, the blocks of method 500 need not be executed in the order indicated. In some examples, one or more blocks may be executed simultaneously. In addition, such a method may include more or fewer blocks than those shown and/or described. For example, some embodiments may not include block 501.

In this example, method 500 is an echo reference selection method. The blocks of method 500 may be performed, for example, by a control system (such as control system 60a of FIG. 2A or FIG. 3A). In some examples, the blocks of method 500 may be performed by an echo reference selector module (such as echo reference selector 402A described above with reference to FIG. 4).

The reference selection method of FIG. 5A is an example of what may be referred to herein as a "greedy" echo reference selection method, which involves evaluating the cost and expected performance improvement only at the current operating point of the MC-EMS (in other words, how many references are currently being used by the MC-EMS, including the selected echo references), and evaluating the results of adding each additional echo reference, for example, in descending order of importance. Accordingly, the example involves a process for determining whether to add a new echo reference. In some embodiments, the echo references evaluated in method 500 may have been ranked according to estimated importance (e.g., by echo reference importance estimator 401A). If a more complex technique (such as a tree search method) is employed, there may be a more optimized type of solution in terms of cost and performance. Alternative examples may involve other search and/or optimization routines, including brute force methods. Some alternative embodiments may involve determining whether to throw away or discard a previously selected echo reference.

In this example, block 501 involves determining whether the current performance level of the EMS is greater than or equal to the desired performance level. If so, the process terminates (block 510). However, if it is determined that the current performance level is lower than the desired performance level, in this example, the process continues to block 502. According to this example, the determination of block 501 is based at least in part on one or more indicators indicating the current performance of the EMS, such as adaptive filter coefficient data or other AEC statistics, speech echo (SER) ratio data, etc. In some examples where the determination of block 501 is made by the echo reference arranger 302A, this determination may be based at least in part on one or more indicators 350A from the MC-EMS 203A. As described above, some embodiments may not include block 501.

According to this example, block 502 involves ranking the remaining unselected echo references by importance and estimating the potential EMS performance improvement obtained by including the most important echo references that have not yet been used by the EMS. In some examples where the process of block 502 is performed by the echo reference scheduler 302A, the process can be based at least in part on information 423A generated by the MC-EMS performance model 405A, which in some examples can be or include data as shown in FIG. 3B or FIG. 3C. In some embodiments, the above-described ranking and prediction process can be performed at an earlier stage of the method 500, for example, when evaluating previous echo references. In some examples, the above-described ranking and prediction process can be performed before performing the method 500. In some embodiments where the above-described ranking and prediction process has been previously performed, block 502 can simply involve selecting the highest ranked unselected echo reference determined by such a previous process.

In this example, block 503 involves comparing the performance and cost of adding the echo reference selected in block 502. In some examples where the process of block 503 is performed by echo reference organizer 302A, block 503 may be based at least in part on information 422A from cost estimation module 403A indicating the cost(s) of including the echo reference in echo reference set 313A.

Because performance and cost may be variables with different ranges and/or domains, it may be challenging to directly compare these variables. Therefore, in some embodiments, the evaluation of block 503 may be facilitated by mapping performance and cost, which may be variables, to a similar scale (such as a range between a predefined minimum and maximum value).

In some embodiments, if adding an echo reference does not result in exceeding a predetermined network bandwidth and/or computational cost budget, the cost of adding an evaluated echo reference can simply be set to zero. In some such examples, if adding an echo reference will result in exceeding a predetermined network bandwidth and/or computational cost budget, the cost of adding an evaluated echo reference can be set to infinity. Such an example has the benefit of simplicity and efficiency. In this way, the control system can simply add the maximum number of echo references allowed by the predetermined network bandwidth and/or computational cost budget.

According to some examples, if the estimated performance improvement corresponding to adding the echo reference is not above a predetermined threshold (e.g., 1%, 2%, 3%, 4%, 5%, 6%, 7%, 8%, 9%, 10%, etc.), then the estimated performance improvement can be set to zero. Such an approach can prevent consuming network bandwidth and/or computational overhead by including an echo reference that only adds a negligible performance improvement. Some detailed alternative examples of cost determination are described below.

In this example, block 504 involves determining whether a new echo reference will be added given the performance/cost evaluation of block 503. In some examples, blocks 503 and 504 may be combined into a single block. According to this example, block 504 involves determining whether the cost of adding the evaluated echo reference will be less than the estimated EMS performance improvement caused by adding the echo reference. In this example, if the estimated cost is not less than the estimated performance improvement, the process continues to block 511 and the method 500 terminates. However, in this embodiment, if the estimated cost is less than the estimated performance improvement, the process continues to block 505.

According to this example, block 505 involves adding the new echo reference to the selected echo reference set. In some instances, block 505 may include notifying the renderer 202 to output the relevant echo reference. According to some examples, block 505 may involve sending the echo reference over a local network or sending a command 311 to another device to send the echo reference over a local network.

The echo reference evaluated in method 500 may be a local echo reference or a non-local echo reference, which may be determined locally (e.g., by a local renderer as described above) or received over a local network. Thus, cost estimation of some echo references may involve evaluating both computational costs and network costs.

According to some examples, to evaluate the next echo reference after block 505, the control system may simply reset the selected and unselected echo references and revert to a previous block of FIG. 5A , such as block 501, block 502, or block 503. However, a more complex approach may also involve evaluating the references that have been selected, for example, ranking all references that have been selected and deciding whether to discard the echo reference with the lowest estimated importance.

Alternative echo reference form

The echo reference may be transmitted (or used locally within a device, such as a device that generates all echo references) in a variety of forms or variants, which may change the cost/benefit ratio of that particular echo reference. For example, if we transform the echo reference into a segmented power form (in other words, determine the power in each of a plurality of frequency bands and transmit segmented power information about the power in each frequency band), it is possible to reduce the cost of sending the echo reference over the local network. However, the potential improvement that can be obtained by an EMS using a low-fidelity variant of the echo reference will also generally be lower. The choice of making any particular variant of the echo reference available can be interpreted as making it a potential candidate for selection.

In some embodiments, the echo reference may be in one of the following forms listed below (the first four are listed in descending order of estimated performance):

Full fidelity (original, exact) echo reference, which incurs full computational cost and network cost (if transmitted over the network)

Downsample the echo reference, whose computational cost and network cost will be reduced proportionally to the downsampling factor, but incur the computational cost of the downsampling process;

The coded echo reference produced by the lossy coding process can reduce the network cost according to the compression ratio of the coding scheme, but incurs encoding and decoding computational costs;

Segmented power information corresponding to the echo reference, which can have significantly lower network cost because the number of frequency bands can be much lower than the number of sub-bands of the full-fidelity echo reference, and which can have significantly lower computational cost because the cost of implementing segmented AES is much lower than the cost of implementing sub-band AEC; or

Any other form of reducing fidelity in exchange for a reduction in some cost (whether computational, network, or other cost such as memory).

Fig. 5B is a flow chart outlining another example of the disclosed method. As with other methods described herein, the blocks of method 550 need not be executed in the order indicated. In some examples, one or more blocks may be executed simultaneously. In addition, such a method may include more or fewer blocks than those shown and/or described.

The blocks of method 550 may be performed, for example, by a control system such as control system 60a of FIG. 2A or FIG. 3A. In some examples, the blocks of method 550 may be performed by an echo reference selector module such as echo reference selector 402A described above with reference to FIG. 4.

Method 550 takes into account the fact that the echo reference is not necessarily transmitted or used in full fidelity form, but may be transmitted or used in one of the alternative partial fidelity forms described above. Thus, in method 550, the evaluation of performance and cost does not involve a binary decision as to whether the echo reference will be used or not in full fidelity form. Rather, method 550 involves determining whether to include one or more low-fidelity versions of the echo reference, which may involve and potentially less EMS performance improvement, but at a lower cost. Methods such as method 550 provide additional flexibility for the potential set of echo references to be used by the echo management system.

In this example, method 550 is an extension of the echo reference selection method 500 described above with reference to FIG. 5A. Therefore, blocks 501 (if included), 502, 503, 504, and 505 may be performed as above with reference to FIG. 5A, unless otherwise specified below. Method 550 adds a potential iteration loop including blocks 506 and 507 to method 500. According to this example, if it is determined (here, in block 504) that the estimated cost of adding a version of the echo reference will not be less than the estimated EMS performance improvement, then in block 506, it is determined whether there is another version of the echo reference. In some examples, the full-fidelity version of the echo reference may be evaluated before a lower-fidelity version (if any version is available). According to this embodiment, if it is determined in block 506 that another version of the echo reference is available, then in block 507, another version of the echo reference (e.g., the highest fidelity version that is not the full-fidelity version) will be selected and evaluated in block 503.

Thus, method 550 involves evaluating a lower fidelity version of the echo reference, if any is available. Such a lower fidelity version may include a downsampled version of the echo reference, an encoded version of the echo reference generated via a lossy encoding process, and/or segment power information corresponding to the echo reference.

Cost Model

The "cost" of an echo reference refers to the resources required to perform echo management using that reference, whether using AEC or AES. Some disclosed embodiments may involve estimating one or more of the following types of costs:

Computational cost, which may be determined with reference to the use of a limited amount of processing power available on one or more devices in the audio environment. Computational cost may refer to one or more of the following:

o the cost of using that reference to perform echo management on a particular listening device. This may refer to using that reference in AEC or AES. One will note that AEC operates on bins or subbands (which are complex numbers) and requires many more CPU operations than AES which operates on bands (which are few in number compared to the bins/subbands used by AES, and band powers are real numbers, not complex numbers);

o the cost required to encode or decode the echo reference when using a coded reference;

o the cost required to segment the signal (in other words, transform the signal from a simple linear frequency domain representation to a segmented frequency domain representation); and/or

ο The cost required to generate the echo reference (e.g., by a renderer).

• Network cost, which refers to the use of a limited amount of network resources, such as the bandwidth available in a local network (e.g., a local wireless network in an audio environment) for sharing an echo reference between devices.

The total cost of a particular set of echo references may be determined as the sum of the costs of each echo reference in the set. Some disclosed examples involve combining network costs and computational costs. According to some examples, the total cost C _total may be determined as follows:

In the above equation, R _comp represents the total amount of computing resources available for echo management, and R _network represents the total amount of network resources available for echo management; represents the computational cost associated with using the mth reference, and represents the network cost associated with using the mth reference (where a total of M references are used in the EMS). One may note that this definition implies

0≤C _total ≤1,

And _Ctotal includes only the cost components that are closest to the cost that becomes limited by the available resources of the system.

performance

The “performance” of an echo management system (EMS) can refer to the following:

The amount of echo removed from the microphone feed, which can be measured in terms of echo loss enhancement (ERLE), which is measured in decibels and is the ratio of the transmitted power to the power of the residual signal. This metric can be normalized, for example, based on an application-based metric such as the minimum ERLE required to enable an automatic speech recognition (ASR) processor to perform a wake-up word detection task of detecting a specific keyword spoken in the presence of echo;

Robustness of the EMS when subject to interference from room noise sources, nonlinearities of the local audio system, double-ended conversations, etc.

Robustness of the EMS when using less than full fidelity echo references;

· The ability of the EMS to track system changes, including the ability of the EMS to initially converge; and/or

• The ability of the EMS to track changes in the rendered audio scene. This may refer, for example, to shifts in the echo reference covariance matrix and the robustness of the EMS to non-stationary non-uniqueness issues.

Some examples may involve determining a single performance metric P. Some such examples use ERLE and robustness estimated from adaptive filter coefficient data or other AEC statistics obtained from EMS. According to some such examples, the performance robustness metric _Prob may be determined using "microphone probabilities" extracted from the AEC, such as as follows:

P _Rob = 1-M_prob

In the above equation, 0≤P _R o _b ≤1, 0≤M_prob≤1, and M_prob represents the microphone probability, which is the proportion of the number of subband adaptive filters in the AEC that produce poor echo predictions that do not provide substantial (or any) echo cancellation in the respective subband.

The performance of the wake-up word (WW) detector depends largely on the speech echo ratio (SER), which can be improved proportionally by the ERLE of the EMS. When the SER is too low, the WW detector is more likely to trigger falsely (false alarms) and miss keywords spoken by the user (missed detections) because the echo corrupts the microphone signal and reduces the accuracy of the system. The SER of the residual signal (e.g., residual signal 224A of FIG. 2A) consumed by the ASR processor (e.g., speech processing block 240A of FIG. 2A) is improved by the EMS in proportion to the ERLE of the EMS, thereby improving the performance of the WW detector.

Thus, some disclosed examples involve mapping a desired WW performance level to a nominal SER level, which in turn, combined with knowledge of typical playback levels of devices in the system, allows the control system to map such desired WW performance level directly to a nominal ERLE. In some examples, the method can be extended to map the WW performance of the system at various SER levels to an ERLE. In some such embodiments, input data having a range of SER values can be used to generate a receiver operating characteristic (ROC) curve for a particular WW detector. Some examples involve selecting a particular false alarm rate (FAR) of interest and basing our application on the accuracy of the WW detector as a function of SER for that particular FAR. In some such examples,

Acc(SER _res )=ROC(SER _res ,FAR _l )

In the above equation, Acc(SER _res ) represents the accuracy of the WW detector as a function of SER _res representing the SER of the residual signal output by the EMS. ROC() represents a collection of ROC curves for multiple SERs, and FAR _I represents the false alarm rate of interest, typical values of which may be 3 per 24 hours and 1 per 10 hours. The accuracy Acc(SER _res ) may be expressed as a percentage or normalized so that it is in the range of 0 to 1, which may be expressed as follows:

0≤Acc(SER _res )≤1

With knowledge of the playback capabilities of the audio devices in the audio environment, a typical SER value in a microphone signal (e.g., microphone signal 223A of FIG. 2A ) may be determined using, for example, the LUPA component of the actual echo level and the typical speech level in the target audio environment, for example, as follows:

In the above equation, Speech_pwr and Echo_pwr represent the expected baseline speech power level and echo power level of the target audio environment, respectively. Through EMS, the SER _mic can be improved to SER _res proportional to ERLE, for example as follows:

In the above equations, the superscript dB indicates that the variable is in decibels in this example. For completeness, some embodiments may define the ERLE of the EMS as follows:

Using the aforementioned equation, some embodiments may define the EMS performance metric based on the WW application as follows:

in, Represents the SER in the target environment. In some examples, can be a static default number, while in other examples, It can be estimated as a function of one or more LUPA components, for example. Some embodiments may involve defining the net performance indicator P as a vector containing each element, for example as follows:

P＝[P _ww ,P _Rob ]

In some examples, one or more additional performance components may be added by increasing the size of the net performance vector. In some alternative examples, one or more additional performance components may be combined into a single scalar metric by weighting them, such as as follows:

P＝(1-K) _Pww + _KPRob

In the above equation, K represents a weighting factor selected by the system designer that is used to determine how much each component contributes to the net performance. Some alternative examples may use another approach, such as simply averaging the individual performance indicators. However, it may be advantageous to combine the individual performance indicators into a single scalar indicator.

Cost and performance trade-off

When comparing the estimated cost of the echo reference and the estimated EMS performance enhancement, a method is needed to somehow compare these two parameters that are usually not in the same domain. One such method involves evaluating the cost estimate and the performance estimate separately and adopting the solution with the lowest cost and meeting a predefined minimum performance standard _Pmin . This predefined EMS performance standard can be determined, for example, according to the requirements of a specific downstream application (e.g., providing phone calls, music playback, waiting for WW, etc.).

For example, in an embodiment where the application is WW detection, the performance may be related to a WW performance indicator P _WW . In some such examples, there may be a certain minimum level of WW detector accuracy that is considered sufficient (e.g., 80% level of WW detector accuracy, 85% level of WW detector accuracy, 90% level of WW detector accuracy, 95% level of WW detector accuracy, etc.), which will have a corresponding ERLE ^dB according to the previous section. In some such examples, an EMS performance model (e.g., MC-EMS performance model 405 of FIG. 4) may be used to estimate the ERLE of the EMS. Therefore, if the goal is simply to find the lowest cost solution (e.g., in terms of total cost C _total ), such an embodiment does not require a direct trade-off between cost and performance.

As an alternative to meeting some minimum performance metric, some embodiments may involve using a performance metric P and a cost metric C. Some such examples may involve using a trade-off parameter λ (e.g., a Lagrange multiplier) and formulating the cost/performance evaluation process as an optimization problem that seeks to maximize some quantity, such as the variable F in the following expression:

F＝P-λC _total

It can be observed that in the above equation, a relatively large value of F corresponds to a relatively large difference between the performance indicator P and the product of λ and the total cost C _total . The trade-off parameter λ can be selected (e.g., by a system designer) to directly trade off cost and performance. An optimization algorithm can then be used to find a solution for the echo reference set used by the EMS, where the echo reference set (which can include all available echo reference fidelity levels) determines the search space.

Fig. 6 is a flow chart summarizing an example of the disclosed method. As with other methods described herein, the blocks of method 600 need not be performed in the order indicated. In addition, such a method may include more or fewer blocks than those shown and/or described. In some examples, two or more blocks may be performed simultaneously. In this example, method 600 is an audio processing method.

Method 600 may be performed by an apparatus or system such as apparatus 50 shown in FIG. 1A and described above. In some examples, the blocks of method 600 may be performed by one or more devices within an audio environment, for example, by an audio system controller (such as a device referred to herein as a smart home hub) or by another component of an audio system, such as a smart speaker, a television, a television control module, a laptop computer, a mobile device (such as a cellular phone), etc. In some embodiments, the audio environment may include one or more rooms of a home environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative embodiments, at least some blocks of method 600 may be performed by a device (such as a server) that implements a cloud-based service.

In this embodiment, block 605 involves obtaining, by the control system, a plurality of echo references. In this example, the plurality of echo references includes at least one echo reference for each of a plurality of audio devices in the audio environment. Here, each echo reference corresponds to audio data played back by one or more loudspeakers of an audio device in the plurality of audio devices.

In this example, block 610 involves making, by the control system, an importance estimate for each of the plurality of echo references. According to this example, making the importance estimate involves determining an expected contribution of each echo reference to echo mitigation by at least one echo management system of at least one audio device of the audio environment. In this example, the at least one echo management system includes an acoustic echo canceller (AEC) and/or an acoustic echo suppressor (AES).

In this embodiment, block 615 involves selecting, by the control system and based at least in part on the importance estimates, one or more selected echo references. In this example, block 620 involves providing, by the control system, the one or more selected echo references to at least one echo management system. In some embodiments, method 600 may involve causing at least one echo management system to cancel or suppress echoes based at least in part on the one or more selected echo references.

In some examples, obtaining the plurality of echo references may involve receiving a content stream including audio data and determining one or more of the plurality of echo references based on the audio data.Some examples are described above with reference to the renderer 201A of FIG. 2A .

In some implementations, the control system may include an audio device control system for an audio device in a plurality of audio devices in the audio environment. In some such examples, the method may involve rendering, by the audio device control system, audio data for reproduction on the audio device, thereby generating a local speaker feed signal. In some such examples, the method may involve determining a local echo reference corresponding to the local speaker feed signal.

In some examples, obtaining multiple echo references can involve determining one or more non-local echo references based on the audio data.For example, each non-local echo reference can correspond to a non-local speaker feed signal for playback on another audio device in the audio environment.

According to some examples, obtaining multiple echo references may involve receiving one or more non-local echo references. For example, each non-local echo reference may correspond to a non-local speaker feed signal for playback on another audio device of the audio environment. In some examples, receiving one or more non-local echo references may involve receiving one or more non-local echo references from one or more other audio devices of the audio environment. In some examples, receiving one or more non-local echo references may involve receiving each of the one or more non-local echo references from a single other device of the audio environment.

In some examples, the method may involve cost determination. According to some such examples, the cost determination may involve determining a cost of at least one echo reference of a plurality of echo references. In some such examples, selecting one or more selected echo references may be based at least in part on the cost determination. According to some such examples, the cost determination may be based at least in part on a network bandwidth required for transmitting at least one echo reference, an encoding computational requirement for encoding at least one echo reference, a decoding computational requirement for decoding at least one echo reference, an echo management system computational requirement for using at least one echo reference by an echo management system, or one or more combinations thereof. In some examples, the cost determination may be based at least in part on a full-fidelity replica of at least one echo reference in a time domain or a frequency domain, a downsampled version of at least one echo reference, a lossy compression of at least one echo reference, segmented power information of at least one echo reference, or one or more combinations thereof. In some examples, the cost determination may be based at least in part on a method of compressing relatively more important echo references less than relatively less important echo references.

According to some examples, the method may involve determining a current echo management system performance level.In some such examples, selecting one or more selected echo references may be based at least in part on the current echo management system performance level.

In some examples, making an importance estimate may involve determining an importance metric for a corresponding echo reference. In some examples, determining the importance metric may involve determining a level of a corresponding echo reference, determining a uniqueness of a corresponding echo reference, determining a temporal duration of a corresponding echo reference, determining an audibility of a corresponding echo reference, or one or more combinations thereof. According to some examples, determining the importance metric may be based at least in part on metadata corresponding to an audio device layout, loudspeaker metadata, metadata corresponding to received audio data, an upmix matrix, a loudspeaker activation matrix, or one or more combinations thereof. In some examples, determining the importance metric may be based at least in part on a current listening objective, a current ambient noise estimate, an estimate of a current performance of at least one echo management system, or one or more combinations thereof.

Fig. 7 shows an example of a floor plan of an audio environment, which in this example is a living space. As with other figures provided herein, the types and quantities of the elements shown in Fig. 7 are provided only as examples. Other embodiments may include more, fewer and/or different types and quantities of elements.

According to this example, the environment 700 includes a living room 710 at the upper left, a kitchen 715 at the lower center, and a bedroom 722 at the lower right. The boxes and circles distributed across the living space represent a group of loudspeakers 705a-705h, at least some of which may be smart speakers in some embodiments, placed in a convenient position for the space, but not following any standard prescribed layout (arbitrarily placed). In some examples, the television 730 may be configured to at least partially implement one or more disclosed embodiments. In this example, the environment 700 includes cameras 711a-711e distributed throughout the environment. In some embodiments, one or more smart audio devices in the environment 700 may also include one or more cameras. The one or more smart audio devices may be single-purpose audio devices or virtual assistants. In some such examples, one or more cameras of the optional sensor system 130 may reside in or on the television 730, in a mobile phone, or in a smart speaker (such as one or more of the loudspeakers 705b, 705d, 705e, or 705h). Although cameras 711a - 711e are not shown in each depiction of an audio environment presented in this disclosure, in some implementations, each audio environment may still include one or more cameras.

Some aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor that is programmed with software or firmware and/or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and/or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP), which is configured (e.g., programmed and otherwise configured) to perform the required processing on (multiple) audio signals, including the execution of one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed with software or firmware and/or otherwise configured to perform any of various operations, including one or more examples of the disclosed method. Alternatively, the elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more loudspeakers and/or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and/or keyboard), a memory, and a display device.

Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code for executing one or more examples of the disclosed method or its steps (e.g., an encoder executable to execute one or more examples of the disclosed method or its steps).

Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many changes may be made to the embodiments and applications described herein without departing from the scope of the present disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Various aspects of the present invention may be understood from the following enumerated example embodiments (EEE):

1. An audio processing method, comprising:

obtaining, by a control system, a plurality of echo references, the plurality of echo references comprising at least one echo reference for each of a plurality of audio devices in an audio environment, each echo reference corresponding to audio data played back by one or more loudspeakers of one of the plurality of audio devices;

making, by the control system, an importance estimate for each of the plurality of echo references, wherein making the importance estimate involves determining an expected contribution of each echo reference to echo mitigation performed by at least one echo management system of at least one audio device of the audio environment, the at least one echo management system comprising an acoustic echo canceller (AEC), an acoustic echo suppressor (AES), or both an AEC and an AES;

selecting, by the control system and based at least in part on the importance estimate, one or more selected echo references; and

The one or more selected echo references are provided by the control system to the at least one echo management system.

2. The audio processing method of EEE 1, further comprising causing at least one echo management system to cancel or suppress echo based at least in part on the one or more selected echo references.

3. The audio processing method according to EEE 1 or EEE 2, wherein obtaining the plurality of echo references involves:

receiving a content stream including audio data; and

One or more echo references of the plurality of echo references are determined based on the audio data.

4. The audio processing method according to EEE 3, wherein the control system comprises an audio device control system of an audio device among the plurality of audio devices in the audio environment, and the audio processing method further comprises:

rendering, by the audio device control system, the audio data for reproduction on the audio device to produce local speaker feed signals; and

A local echo reference corresponding to the local loudspeaker feed signal is determined.

5. The audio processing method of EEE 4, wherein obtaining the plurality of echo references involves determining one or more non-local echo references based on the audio data, each of the non-local echo references corresponding to a non-local speaker feed signal for playback on another audio device in the audio environment.

6. The audio processing method of EEE 4, wherein obtaining the plurality of echo references involves receiving one or more non-local echo references, each of the non-local echo references corresponding to a non-local speaker feed signal for playback on another audio device in the audio environment.

7. The audio processing method of EEE 6, wherein receiving the one or more non-local echo references involves receiving the one or more non-local echo references from one or more other audio devices of the audio environment.

8. The audio processing method of EEE 6, wherein receiving the one or more non-local echo references involves receiving each of the one or more non-local echo references from a single other device of the audio environment.

9. The audio processing method as described in any one of EEE 1-8 further includes cost determination, which involves determining the cost of at least one echo reference of the multiple echo references, wherein selecting the one or more selected echo references is at least partially based on the cost determination.

10. The audio processing method of EEE 9, wherein the cost determination is based on network bandwidth required for transmitting the at least one echo reference, encoding computational requirements for encoding the at least one echo reference, decoding computational requirements for decoding the at least one echo reference, echo management system computational requirements for using the at least one echo reference by the echo management system, or a combination thereof.

11. The audio processing method as described in EEE 9 or EEE 10, wherein the cost determination is based on a replica of the at least one echo reference in the time domain or the frequency domain, a downsampled version of the at least one echo reference, a lossy compression of the at least one echo reference, segmented power information of the at least one echo reference, or a combination thereof.

12. The audio processing method of any one of EEEs 9-11, wherein the cost determination is based on a method of compressing relatively more important echo references less than relatively less important echo references.

13. The audio processing method of any one of EEEs 1-12, further comprising determining a current echo management system performance level, wherein selecting the one or more selected echo references is based at least in part on the current echo management system performance level.

14. The audio processing method as described in any one of EEE 1-13, wherein making the importance estimate involves determining an importance measure of the corresponding echo reference.

15. The audio processing method of EEE 14, wherein determining the importance metric involves determining a level of the corresponding echo reference, determining a uniqueness of the corresponding echo reference, determining a temporal duration of the corresponding echo reference, determining an audibility of the corresponding echo reference, or a combination thereof.

16. An audio processing method as described in EEE 14 or EEE 15, wherein determining the importance metric is at least partially based on metadata corresponding to an audio device layout, loudspeaker metadata, metadata corresponding to received audio data, an upmixing matrix, a loudspeaker activation matrix, or a combination thereof.

17. The audio processing method of any one of EEE 14-16, wherein determining the importance metric is based at least in part on current listening goals, current ambient noise estimates, estimates of current performance of the at least one echo management system, or a combination thereof.

18. An apparatus configured to perform the method as described in any one of EEE 1-17.

19. A system configured to perform the method as described in any one of EEE 1-17.

20. One or more non-transitory media having software stored thereon, the software comprising instructions for controlling one or more devices to perform the method of any one of EEE 1-17.

Claims (20)

1. An audio processing method for managing echoes of a first audio device of a plurality of audio devices of an audio system, wherein each audio device of the plurality of audio devices includes one or more loudspeakers, Wherein, the first audio device further includes a control system, wherein the control system includes an echo management system, the echo management system includes an acoustic echo canceller (AEC), an acoustic echo suppressor (AES), or an AEC and an AES. For both, the methods include: A plurality of echo references are obtained by the control system of the first audio device, the plurality of echo references including at least one echo reference for each of the plurality of audio devices, each echo reference corresponding to Audio data played back by the one or more loudspeakers of the corresponding audio device; An importance estimate is made by the control system for each of the plurality of echo references, wherein making the importance estimate involves determining, by the echo management system of the first audio device, each The expected contribution of echo references to echo mitigation; Selecting, by the control system and based at least in part on the significance estimate, one or more echo references from the plurality of echo references; providing the one or more selected echo references to the echo management system by the control system; and Echo is suppressed or eliminated by the echo management system of the first audio device based at least in part on the one or more selected echo references. 2. The audio processing method of claim 1, wherein obtaining the plurality of echo references involves: receiving a content stream including audio data; and One or more echo references of the plurality of echo references are determined based on the audio data. 3. The audio processing method as claimed in claim 2, further comprising: The audio data is rendered by the control system for reproduction on the first audio device to produce a local speaker feed signal; and A local echo reference corresponding to the local loudspeaker feed signal is determined. 4. The audio processing method of claim 3, wherein obtaining the plurality of echo references involves determining one or more non-local echo references based on the audio data, each of the non-local echo references corresponding to Non-native speaker feeds for playback on another audio device in the audio environment. 5. The audio processing method of claim 3, wherein obtaining the plurality of echo references involves receiving one or more non-local echo references, each of the non-local echo references corresponding to the A non-local speaker feed that is played back on another audio device in the audio environment. 6. The audio processing method of claim 5, wherein receiving the one or more non-local echo references involves receiving the one or more non-local echoes from one or more other audio devices of the audio environment refer to. 7. The audio processing method of claim 5, wherein receiving the one or more non-local echo references involves receiving each of the one or more non-local echo references from a single other device of the audio environment. one. 8. The audio processing method of any one of claims 1 to 7, further comprising a cost determination involving determining a cost of at least one of the plurality of echo references, wherein selecting the One or more selected echo references are determined based at least in part on the cost. 9. The audio processing method of claim 8, wherein the cost determination is based on network bandwidth required for transmitting the at least one echo reference, encoding computational requirements for encoding the at least one echo reference, Decoding computational requirements for decoding the at least one echo reference, echo management system computational requirements for using the at least one echo reference by the echo management system, or a combination thereof. 10. An audio processing method as claimed in claim 8 or claim 9, wherein the cost determination is based on a replica of the at least one echo reference in the time or frequency domain, a downsampling of the at least one echo reference version, lossy compression of the at least one echo reference, segmented power information of the at least one echo reference, a method of compressing relatively more important echo references less than relatively less important echo references, or its combination. 11. An audio processing method as claimed in any one of claims 8 to 10, wherein the cost determination is based on a method of compressing relatively more important echo references less than relatively less important echo references. 12. The audio processing method of any one of claims 1 to 11, further comprising determining a current echo management system performance level, wherein selecting the one or more selected echo references is based at least in part on the current echo Manage system performance levels. 13. An audio processing method as claimed in any one of claims 1 to 12, wherein making the importance estimate involves determining an importance measure for a corresponding echo reference. 14. The audio processing method of claim 13, wherein determining the importance metric is based at least in part on a level of the corresponding echo reference, a uniqueness of the corresponding echo reference, a temporal duration of the corresponding echo reference properties, the audibility of the corresponding echo reference, or a combination thereof. 15. The audio processing method of claim 13 or claim 14, wherein determining the importance metric is based at least in part on metadata corresponding to audio device layout, loudspeaker metadata, and received audio. The data corresponds to metadata, an upmix matrix, a loudspeaker activation matrix, or a combination thereof. 16. The audio processing method of any one of claims 13 to 15, wherein determining the importance metric is based at least in part on current listening goals, current ambient noise estimates, and estimates of current performance of the echo management system , or a combination thereof. 17. The audio processing method of any one of the preceding claims, wherein the audio devices of the audio system are communicatively coupled via a wired or wireless communications network, and wherein the plurality of echo references are generated via the obtained from wired or wireless communication networks. 18. An apparatus configured to perform the method of any one of claims 1 to 17. 19. A system configured to perform the method of any one of claims 1 to 17. 20. One or more non-transitory media having software stored thereon, the software comprising instructions for controlling one or more devices to perform the method of any one of claims 1 to 17.