CN116830604A

CN116830604A - Progressive calculation and application of dynamically applied rendering configurations

Info

Publication number: CN116830604A
Application number: CN202180092766.5A
Authority: CN
Inventors: J·B·兰多; A·J·西斐德
Original assignee: Dolby Laboratories Licensing Corp
Current assignee: Dolby Laboratories Licensing Corp
Priority date: 2020-12-03
Filing date: 2021-12-02
Publication date: 2023-09-29

Abstract

Some examples involve rendering received audio data by: determining a first relative activation of a set of loudspeakers in the environment based on a first rendering configuration corresponding to activation of a first set of loudspeakers; receiving an indication from the first rendering configuration A first rendering transition indication of a transition to a second rendering configuration; and determining a second set of speaker activations corresponding to a simplified version of the second rendering configuration. Some examples involve: performing a first transition from a first set of speaker activations to a second set of speaker activations; determining a third set of speaker activations that corresponds to a complete version of the second rendering configuration; and performing a third set of speaker activations to a third set of speaker activations. Second transformation without completing the first transformation.

Description

Progressive computation and application of rendering configuration for dynamic applications

The inventors: joshua B.Lando and Alan J.Seefeldt

Cross Reference to Related Applications

The present application claims priority from the following applications: U.S. provisional application 63/121,108 filed on month 12 and 3 of 2020 and U.S. provisional application 63/202,003 filed on month 5 and 21 of 2021, each of which is incorporated herein by reference in its entirety.

Technical Field

The present disclosure relates to systems and methods for rendering audio for playback by some or all of a set of speakers (e.g., each active speaker).

Background

Audio devices, including but not limited to smart audio devices, have been widely deployed and are becoming a common feature for many households. While existing systems and methods for controlling audio devices provide benefits, improved systems and methods would still be desirable.

Symbols and terms

Throughout this disclosure, including in the claims, the terms "speaker (speaker)", "loudspeaker (loudspecker)" and "audio reproduction transducer" are synonymously used to denote any sound producing transducer (or set of transducers). A typical earphone set includes two speakers. The speaker may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feed(s) may undergo different processing in different circuit branches coupled to different transducers.

Throughout this disclosure, including in the claims, the expression "performing an operation on" a signal or data (e.g., filtering, scaling, transforming, or applying gain to a signal or data) is used in a broad sense to mean performing an operation directly on a signal or data or on a processed version of a signal or data (e.g., a version of a signal that has undergone preliminary filtering or preprocessing prior to performing an operation thereon).

Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to denote a device, system, or subsystem. For example, a subsystem implementing a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other X-M inputs are received from external sources) may also be referred to as a decoder system.

Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to mean a system or device that is programmable or otherwise configurable (e.g., in software or firmware) to perform operations on data (e.g., audio, video, or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chip sets), digital signal processors programmed and/or otherwise configured to perform pipelined processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chip sets.

Throughout this disclosure, including in the claims, the term "coupled" or "coupled" is used to mean a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

As used herein, a "smart device" is an electronic device that may operate interactively and/or autonomously to some extent, typically configured to communicate with one or more other devices (or networks) via various wireless protocols such as bluetooth, zigbee, near field communication, wi-Fi, light fidelity (Li-Fi), 3G, 4G, 5G, and the like. Several notable smart device types are smart phones, smart cars, smart thermostats, smart doorbell, smart locks, smart refrigerators, tablet phones and tablet computers, smart watches, smart bracelets, smart key chains, and smart audio devices. The term "smart device" may also refer to a device that exhibits some properties of pervasive computing such as artificial intelligence.

The expression "smart audio device" is used herein to denote a smart device that is a single-purpose audio device or a multi-purpose audio device (e.g., an audio device implementing at least some aspects of the virtual assistant functionality). A single-use audio device is a device that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and/or at least one camera) and is designed largely or primarily to achieve a single use, such as a Television (TV). For example, while a TV may generally play (and be considered capable of playing) audio from program material, in most instances, modern TVs run some operating system on which applications (including television-watching applications) run locally. In this sense, single-use audio devices having speaker(s) and microphone(s) are typically configured to run local applications and/or services to directly use the speaker(s) and microphone(s). Some single-use audio devices may be configured to be combined together to enable playback of audio over a zone or user-configured area.

One common type of multi-purpose audio device is an audio device that implements at least some aspects of the virtual assistant functionality, although other aspects of the virtual assistant functionality may be implemented by one or more other devices, such as one or more servers, with which the multi-purpose audio device is configured to communicate. Such multi-purpose audio devices may be referred to herein as "virtual assistants. A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and/or at least one camera). In some examples, the virtual assistant may provide the ability to use multiple devices (other than the virtual assistant) for applications that in a sense support the cloud or that are otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality (e.g., speech recognition functionality) may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant may communicate via a network (such as the internet). Virtual assistants can sometimes work together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants may work together in the sense that one of them (e.g., the virtual assistant that is most confident that the wake word has been heard) responds to the wake word. In some implementations, the connected virtual assistants may form a constellation that may be managed by a host application, which may be (or implement) the virtual assistant.

In this document, the "wake word" is used in a broad sense to mean any sound (e.g., a word spoken by a human or other sound), wherein the smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "wake-up" means a state in which the device enters a waiting (in other words, listening) for a voice command. In some examples, a so-called "wake word" herein may include more than one word, e.g., a phrase.

Herein, the expression "wake word detector" means a device (or means including software for configuring the device to continuously search for an alignment between real-time sound (e.g., speech) features and a training model). Typically, a wake word event is triggered whenever the wake word detector determines that the probability of detecting a wake word exceeds a predefined threshold. For example, the threshold may be a predetermined threshold that is adjusted to give a reasonable tradeoff between false acceptance rate and false rejection rate. After the wake word event, the device may enter a state (which may be referred to as a "wake" state or an "attention" state) in which the device listens for commands and passes the received commands to a larger, more computationally intensive recognizer.

As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some instances, a collection of video signals, at least portions of which are intended to be heard together. Examples include selection of music, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, real-time voice calls, synthesized voice responses from intelligent assistants, and the like. In some examples, the content stream may include multiple versions of at least a portion of the audio signal, e.g., the same dialog in more than one language. In such instances, only one version of the audio data or portion thereof (e.g., a version corresponding to a single language) is intended to be reproduced at a time.

Disclosure of Invention

At least some aspects of the present disclosure may be implemented via a method. Some such methods may involve audio processing. For example, some methods may involve receiving, by a control system and via an interface system, audio data. The audio data may include one or more audio signals and associated spatial data. The spatial data may indicate an expected perceived spatial location corresponding to the audio signal. In some examples, the spatial data may be or may include location metadata. According to some examples, the spatial data may be, may include, or may correspond to channels of a channel-based audio format.

In some examples, a method may involve rendering, by a control system, audio data for reproduction via a set of loudspeakers of an environment to produce a first rendered audio signal. In some such examples, rendering the audio data for reproduction may involve determining a first relative activation of a set of loudspeakers in the environment according to a first rendering configuration. The first rendering configuration may correspond to a first set of speaker activations. In some examples, the method may involve providing, via the interface system, the first rendered audio signal to at least some of a set of loudspeakers of the environment.

According to some examples, a method may involve receiving, by a control system and via an interface system, a first rendering transition indication. The first rendering transition indication may, for example, indicate a transition from a first rendering configuration to a second rendering configuration.

In some examples, the method may involve determining, by the control system, a second set of speaker activations. According to this example, the second set of speakers activates a simplified version corresponding to the second rendering configuration. However, in other examples, the second set of speaker activations may correspond to a full, full-fidelity version of the second rendering configuration.

According to some examples, the method may involve performing, by the control system, a first transition from a first set of speaker activations to a second set of speaker activations. In some examples, the method may involve determining, by the control system, a third set of speaker activations. According to this example, the third set of speakers activates a full version corresponding to the second rendering configuration. In some examples, the method may involve performing, by the control system, a second transition to a third set of speaker activations without completing the first transition. In some examples, a single renderer instance may render the audio data for rendering.

In some examples, the first set of speaker activations, the second set of speaker activations, and the third set of speaker activations may be frequency dependent speaker activations. According to some such examples, the frequency-dependent speaker activations may correspond to and/or be generated by applying a model of the perceived spatial location in at least the first frequency band, the model generating binaural responses corresponding to the audio object locations at the left and right ears of the listener.

In some examples, the frequency-dependent speaker activation may correspond to and/or be generated by applying a perceptual spatial location model in at least a second frequency band, the model placing the perceptual spatial location of an audio signal played from a set of loudspeakers at a centroid of locations of the set of loudspeakers weighted by associated activation gains of the loudspeakers.

According to some examples, the first set of speaker activations, the second set of speaker activations, and/or the third set of speaker activations may be based at least in part on a cost function. In some such examples, the first set of speaker activations, the second set of speaker activations, and/or the third set of speaker activations may be the result of optimizing costs as a function of: a model of perceived spatial location of an audio signal that is played when played back over a set of loudspeakers in an environment; a measure of the proximity of the intended perceived spatial position of the audio signal to the position of each loudspeaker in the set of loudspeakers; and/or one or more additional dynamically configurable functions. In some such examples, the one or more additional dynamically configurable functions may be based on one or more of the following: proximity of the loudspeakers to one or more listeners; proximity of the loudspeaker to the attractive force location (where attractive force may be a factor that favors relatively higher loudspeaker activation closer to the attractive force location); proximity of the loudspeaker to the repulsive force location (wherein repulsive force may be a factor in favor of relatively lower loudspeaker activation closer to the repulsive force location); the ability of each loudspeaker to be relative to other loudspeakers in the environment; synchronization of the loudspeakers with respect to the other loudspeakers; wake word performance; and/or echo canceller performance.

According to some examples, the method may involve receiving, by the control system and via the interface system, a second rendering transition indication. According to some such examples, the second rendering transition indication may indicate a transition to a third rendering configuration. In some such examples, the method may involve determining, by the control system, a fourth set of speaker activations corresponding to the third rendering configuration. In some such examples, the method may involve performing, by the control system, a third transition to a fourth set of speaker activations without completing the first transition or the second transition. In some examples, the method may involve receiving, by the control system and via the interface system, a third rendering transition indication. In some such examples, the third rendering transition indication may indicate a transition to a fourth rendering configuration. In some such examples, the method may involve determining, by the control system, a fifth set of speaker activations corresponding to the fourth rendering configuration. In some such examples, the method may involve performing, by the control system, a fourth transition to a fifth set of speaker activations without completing the first transition, the second transition, or the third transition.

In some examples, the method may involve sequentially receiving, by the control system and via the interface system, second through (N) th rendering transition indications. In some such examples, the method may involve determining, by the control system, fourth through (n+2) -th group speaker activations corresponding to the second through (N) -th rendering transition indications. In some such examples, the method may involve sequentially performing, by the control system, a third through (N) th transition from the fourth set of speaker activations to the (n+1) th set of speaker activations. In some such examples, the method may involve performing, by the control system, an (n+1) th transition to an (n+2) th group of speaker activations without completing any of the first through (N) th transitions.

According to some examples, the method may involve receiving, by the control system and via the interface system, a second rendering transition indication. In some examples, the second rendering transition indication may indicate a transition to a third rendering configuration. In some such examples, the method may involve determining, by the control system, a fourth set of speaker activations corresponding to the reduced version of the third rendering configuration. In some such examples, the method may involve performing, by the control system, a third transition from a third set of speaker activations to a fourth set of speaker activations. In some such examples, the method may involve determining, by the control system, a fifth set of speaker activations corresponding to the full version of the third rendering configuration. In some such examples, the method may involve performing, by the control system, a fourth transition to a fifth set of speaker activations without completing the first transition, the second transition, or the third transition.

In some examples, the method may involve sequentially receiving, by the control system and via the interface system, second through (N) th rendering transition indications. In some such examples, the method may involve determining, by the control system, a first set of speaker activations and a second set of speaker activations for each of the second through (N) th rendering transition indications. In some such examples, the first set of speaker activations may correspond to a simplified version of the rendering configuration, while the second set of speaker activations may correspond to a complete version of the rendering configuration for each of the second through (N) th rendering transition indications. In some such examples, the method may involve sequentially performing, by the control system, a third through (2N-1) th transition from a fourth set of speaker activations to a (2N) th set of speaker activations. In some such examples, the method may involve performing, by the control system, a (2N) th transition to (2n+1) th group speaker activation without completing any of the first through (2N) th transitions.

According to some examples, rendering audio data for rendering may involve determining a single set of interpolation activations according to a rendering configuration and applying the single set of interpolation activations to produce a single set of rendered audio signals. In some such examples, a single set of rendered audio signals may be fed into a set of loudspeaker delay lines. In some such examples, the set of loudspeaker delay lines may include one loudspeaker delay line for each of the plurality of loudspeakers.

In some examples, rendering the audio data for reproduction may be performed in the frequency domain. In some such examples, rendering the audio data for reproduction may involve determining and implementing loudspeaker delays in the frequency domain. In some such examples, determining and implementing speaker delays in the frequency domain may involve determining and implementing a combination of transform block delays and sub-block delays applied by frequency domain filter coefficients. In some such examples, the sub-block delay may be a residual phase term that allows a delay that is not an exact multiple of the frequency domain transform block size. In some examples, rendering audio data for reproduction may involve implementing a set of transform block delay lines with separate read offsets.

In some examples, rendering audio data for rendering may involve implementing sub-block delay filtering. In some such examples, implementing sub-block delay filtering may involve implementing a multi-tap filter across blocks of the frequency domain transform.

According to some examples, rendering audio data for rendering may involve determining and applying interpolation speaker activation and cross-fade windows for each rendering configuration. In some such examples, rendering the audio data for reproduction may involve implementing a set of transform block delay lines with separate delay line read offsets. In some such examples, the crossfade window selection may be based at least in part on the delay line read offset. In some such examples, the crossfade window may be designed to have a unit power sum if the delay line read offsets are not the same.

In some examples, a first set of speaker activations may be used for each of a corresponding plurality of locations in three-dimensional space. However, according to some examples, the first set of speaker activations may correspond to a channel-based audio format. In some such examples, the desired perceived spatial location may correspond to a channel of a channel-based audio format.

Some or all of the operations, functions, and/or methods described herein may be performed by one or more devices in accordance with instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to Random Access Memory (RAM) devices, read Only Memory (ROM) devices, and the like. Thus, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon.

At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some embodiments, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single or multi-chip processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or a combination thereof. In some examples, the apparatus may be one of the audio devices referenced above. However, in some embodiments, the apparatus may be another type of device, such as a mobile device, a laptop, a server, or the like.

The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.

Drawings

Fig. 1A is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

FIG. 1B is a block diagram of a minimum version of an embodiment.

Fig. 2A depicts another (more capable) embodiment with additional features.

Fig. 2B is a flowchart outlining one example of a method that may be performed by a device or system such as the device or system shown in fig. 1A, 1B, or 2A.

Fig. 2C and 2D are diagrams illustrating an example set of speaker activations and object rendering positions.

FIG. 2E is a flowchart outlining one example of a method that may be performed by a device or system, such as the device or system shown in FIG. 1A.

Fig. 2F is a diagram of speaker activation in an example embodiment.

FIG. 2G is a diagram of object rendering locations in an example embodiment.

Fig. 2H is a diagram of speaker activation in an example embodiment.

FIG. 2I is a diagram of an object rendering location in an example embodiment.

Fig. 2J is a diagram of speaker activation in an example embodiment.

Fig. 2H is a diagram of speaker activation in an example embodiment.

FIG. 2I is a diagram of an object rendering location in an example embodiment.

Fig. 2J is a diagram of speaker activation in an example embodiment.

FIG. 2K is a diagram of object rendering locations in an example embodiment.

Fig. 3A and 3B show plan view examples of connected living spaces.

Fig. 4A and 4B illustrate examples of multi-stream renderers that provide simultaneous playback of spatial music mixes and voice assistant responses.

Fig. 5A, 5B, and 5C illustrate a third example use case of the disclosed multi-stream renderer.

Fig. 6 shows a frequency domain/transform domain example of the multi-stream renderer shown in fig. 1B.

Fig. 7 shows a frequency domain/transform domain example of the multi-stream renderer shown in fig. 2A.

Fig. 8 illustrates an embodiment of a multi-stream rendering system with an audio stream loudness estimator.

Fig. 9A illustrates an example of a multi-stream rendering system configured for crossfading of multiple rendered streams.

Fig. 9B is a diagram indicating points of speaker activation in an example embodiment.

Fig. 10 is a diagram of tri-linear interpolation between points indicating speaker activation, according to one example.

Fig. 11A and 11B illustrate examples of providing a rendering configuration computing service.

Fig. 12A and 12B illustrate examples of rendering configuration transitions.

FIG. 13 presents blocks corresponding to an alternative embodiment for managing transitions between rendering configurations.

FIG. 14 presents blocks corresponding to another embodiment for managing transitions between rendering configurations.

FIG. 15 presents blocks corresponding to a frequency domain renderer according to one example.

FIG. 16 presents blocks corresponding to another embodiment for managing transitions between rendering configurations.

Fig. 17 shows an example of a cross-fade window pair with a unit power sum.

Fig. 18A and 18B present examples of cross-fade window pairs with unit sums.

Fig. 19 presents blocks corresponding to another embodiment for managing transitions between first through L-th set of speaker activations.

Fig. 20A shows an example of a cross-fade window pair having a unit power sum.

Fig. 20B shows an example of a cross-fade window pair with a unit sum.

Fig. 21, 22, 23, 24, and 25 are diagrams presenting examples of crossfade windows without, with some, or all of the matched read offsets.

Fig. 26, 27, 28, 29 and 30 illustrate the same case as fig. 21 to 25, but limit the 3 activation rendering configurations.

Fig. 31 is a flowchart outlining an example of a method.

Fig. 32 is a flow chart summarizing an example of another method.

Fig. 33 depicts a plan view of a listening environment, which in this example is a living space.

Detailed Description

Flexible rendering is a technique for rendering spatial audio over any number of arbitrarily placed speakers. With the widespread deployment of smart audio devices (e.g., smart speakers) in homes, there is a need to implement flexible rendering techniques that allow consumers to use smart audio devices to perform flexible rendering of audio and playback of the audio so rendered.

Several techniques have been developed to implement flexible rendering, including: centroid amplitude translation (CMAP) and Flexible Virtualization (FV). Both techniques treat the rendering problem as one of the cost function minimizations, where the cost function consists of two terms: a first term simulating the desired spatial impression that the renderer is attempting to achieve, and a second term allocating costs for activating the speakers. To date, this second term has focused on creating a sparse solution in which only speakers are activated that are very close to the desired spatial location of the audio being rendered.

Some embodiments of the present disclosure are methods for managing playback of multiple audio streams by at least one (e.g., all or some) of a set of intelligent audio devices (or by at least one (e.g., all or some) of another set of speakers).

One class of embodiments relates to a method for managing playback by at least one (e.g., all or some) of a plurality of coordinated (orchestrated) intelligent audio devices. For example, a set of smart audio devices present in (in a system of) a user's home may be orchestrated to handle various simultaneous use cases, including flexible rendering of audio for playback by all or some of the smart audio devices (i.e., by speaker(s) of all or some of the smart audio devices).

Orchestrating intelligent audio devices (e.g., handling various simultaneous use cases at home) may involve simultaneous playback of one or more audio program streams on a set of interconnected speakers. For example, a user may be listening to a movie panoramic sound track (or other object-based audio program) on a set of speakers, but then the user may speak a command to an associated intelligent assistant (or other intelligent audio device). In this case, the audio played back by the system (according to some embodiments) may be modified to warp the spatial presentation of the panoramic sound mix away from the location of the speaker (the speaking user) and away from the nearest smart audio device, while simultaneously warping the corresponding response of the smart audio device (of the voice assistant) toward the location of the speaker. This may provide important benefits compared to merely reducing the playback volume of the audio program content in response to detection of a command (or corresponding wake-up word). Similarly, a user may want to use a speaker to obtain a cooking tip in a kitchen when playing the same panoramic sound track in an adjacent open living space. In this case, according to some examples, the loudness of the one or more rendered signals of the panoramic sound track may be distorted away from the kitchen and/or the panoramic sound track may be modified in response to the loudness of the one or more rendered signals of the cooking tip sound track. Additionally, in some embodiments, the cooking cues played in the kitchen may be dynamically adjusted to be heard by people in the kitchen with sounds louder than any panoramic sound track that may penetrate from the living space.

Some embodiments relate to a multi-stream rendering system configured to implement the above example use cases, as well as many other example use cases contemplated. In one class of embodiments, the audio rendering system may be configured to simultaneously play a plurality of audio program streams on a plurality of arbitrarily placed loudspeakers, wherein at least one of the program streams is a spatial mix, and the rendering of the spatial mix is dynamically modified in response to (or in conjunction with) simultaneous playback of one or more additional program streams.

In some embodiments, the multi-stream renderer may be configured to implement the above-described scenarios as well as many other situations in which simultaneous playback of multiple audio program streams must be managed. Some embodiments of the multi-stream rendering system may be configured to perform one or more of the following operations:

● Simultaneously rendering and playing back a plurality of audio program streams on a plurality of arbitrarily placed loudspeakers, wherein at least one of the program streams is a spatial mix.

The term "program stream" refers to a collection of one or more audio signals intended to be listened to as a whole. Examples include music selections, movie soundtracks, podcasts, live voice calls, synthesized voice responses from intelligent assistants, and the like.

"spatial mixing" is a program stream intended to deliver different signals (not just mono) at the listener's left and right ears. Examples of audio formats for spatial mixing include stereo, 5.1 and 7.1 surround sound, object audio formats such as Dolby panoramic sound (Dolby Atmos) and Ambisonics (Ambisonics).

"rendering" a program stream refers to the process of actively distributing an associated audio signal or signals across a plurality of loudspeakers to achieve a particular perceived impression.

● Dynamically modifying the rendering of the at least one spatial mix based on the rendering of one or more of the additional program streams. Examples of such modifications to the rendering of the spatial mix include, but are not limited to

Correcting the relative activation of the plurality of loudspeakers in dependence on the relative activation of the loudspeakers associated with the rendering of at least one of the one or more additional program streams.

The expected spatial balance of the spatial mixing is distorted according to the rendered spatial properties of at least one of the one or more additional program streams.

Modifying the loudness or audibility of the spatial mix according to the loudness or audibility of at least one of the one or more additional program streams.

Fig. 1A is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure. According to some examples, the apparatus 100 may be or may include a smart audio device configured to perform at least some of the methods disclosed herein. In other implementations, the apparatus 100 may be or may include another device, such as a laptop computer, a cellular telephone, a tablet device, a smart home hub, or the like, configured to perform at least some of the methods disclosed herein. In some such embodiments, the apparatus 100 may be or may include a server. In some implementations, the apparatus 100 may be configured to implement a device that may be referred to herein as an "audio session manager".

In this example, the apparatus 100 includes an interface system 105 and a control system 110. In some implementations, the interface system 105 may be configured to communicate with one or more devices that are executing or are configured to execute software applications. Such software applications may sometimes be referred to herein as "applications" or simply "apps". In some implementations, the interface system 105 may be configured to exchange control information and associated data related to the application. In some implementations, the interface system 105 may be configured to communicate with one or more other devices of the audio environment. In some examples, the audio environment may be a home audio environment. In some implementations, the interface system 105 may be configured to exchange control information and associated data with an audio device of an audio environment. In some examples, the control information and associated data may relate to one or more applications with which the apparatus 100 is configured to communicate.

In some implementations, the interface system 105 may be configured to receive an audio program stream. The audio program stream may comprise audio signals arranged to be reproduced by at least some speakers of the environment. The audio program stream may include spatial data such as channel data and/or spatial metadata. In some implementations, the interface system 105 may be configured to receive input from one or more microphones in an environment.

The interface system 105 may include one or more network interfaces and/or one or more external device interfaces (e.g., one or more Universal Serial Bus (USB) interfaces). According to some embodiments, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and/or a gesture sensor system. In some examples, interface system 105 may include one or more interfaces between control system 110 and a memory system (such as optional memory system 115 shown in fig. 1A). However, in some examples, control system 110 may include a memory system.

For example, control system 110 may include a general purpose single or multi-chip processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, and/or discrete hardware components.

In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within one of the environments depicted herein, and another portion of the control system 110 may reside in a device outside of the environment, such as a server, mobile device (e.g., smart phone or tablet), etc. In other examples, a portion of the control system 110 may reside in a device within one of the environments depicted herein, and another portion of the control system 110 may reside in one or more other devices of the environments. For example, control system functionality may be distributed across multiple intelligent audio devices of an environment, or may be shared by orchestration devices (as may be referred to herein as devices of an intelligent home hub) and one or more other devices of the environment. In some such examples, the interface system 105 may also reside in more than one device.

In some implementations, the control system 110 may be configured to at least partially perform the methods disclosed herein. According to some examples, control system 110 may be configured to implement a method of managing playback of multiple audio streams on multiple speakers.

Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to Random Access Memory (RAM) devices, read Only Memory (ROM) devices, and the like. The one or more non-transitory media may reside, for example, in the optional memory system 115 and/or the control system 110 shown in fig. 1A. Accordingly, various innovative aspects of the subject matter described in the present disclosure can be implemented in one or more non-transitory media having software stored thereon. For example, the software may include instructions for controlling at least one device to process audio data. For example, the software may be executed by one or more components of a control system, such as control system 110 of FIG. 1A.

In some examples, the apparatus 100 may include an optional microphone system 120 shown in fig. 1A. The optional microphone system 120 may include one or more microphones. In some implementations, one or more microphones may be part of or associated with another device (e.g., a speaker of a speaker system, a smart audio device, etc.). In some examples, the apparatus 100 may not include the microphone system 120. However, in some such embodiments, the apparatus 100 may still be configured to receive microphone data for one or more microphones in an audio environment via the interface system 110.

According to some embodiments, the apparatus 100 may include an optional loudspeaker system 125 shown in fig. 1A. The optional microphone system 125 may include one or more microphones, which may also be referred to herein as "speakers. In some examples, at least some of the loudspeakers of the optional loudspeaker system 125 may be arbitrarily positioned. For example, at least some of the speakers of the optional loudspeaker system 125 may be placed in locations that do not correspond to any standard specified loudspeaker layout, such as dolby 5.1, dolby 5.1.2, dolby 7.1, dolby 7.1.4, dolby 9.1, kawasaki 22.2, and so forth. In some such examples, at least some of the loudspeakers of the optional speaker system 125 may be placed in locations convenient to space (e.g., where there is space to accommodate the loudspeakers), but not in any standard specified loudspeaker layout. In some examples, the apparatus 100 may not include the loudspeaker system 125.

In some embodiments, the apparatus 100 may include an optional sensor system 129 shown in fig. 1A. The optional sensor system 129 may include one or more cameras, touch sensors, gesture sensors, motion detectors, and the like. According to some embodiments, optional sensor system 129 may include one or more cameras. In some implementations, the camera may be a standalone camera. In some examples, one or more cameras of optional sensor system 129 may reside in a smart audio device, which may be a single-use audio device or a virtual assistant. In some such examples, one or more cameras of optional sensor system 129 may reside in a TV, mobile phone, or smart speaker. In some examples, the apparatus 100 may not include the sensor system 129. However, in some such embodiments, the apparatus 100 may still be configured to receive sensor data for one or more sensors in the audio environment via the interface system 110.

In some implementations, the apparatus 100 may include an optional display system 135 shown in fig. 1A. The optional display system 135 may include one or more displays, such as one or more Light Emitting Diode (LED) displays. In some examples, optional display system 135 may include one or more Organic Light Emitting Diode (OLED) displays. In some examples where apparatus 100 includes display system 135, sensor system 129 may include a touch sensor system and/or a gesture sensor system proximate to one or more displays of display system 135. According to some such embodiments, the control system 110 may be configured to control the display system 135 to present one or more Graphical User Interfaces (GUIs).

According to some such examples, apparatus 100 may be or may include a smart audio device. In some such embodiments, the apparatus 100 may be or may include a wake-up word detector. For example, the apparatus 100 may be or may include a virtual assistant.

FIG. 1B is a block diagram of a minimum version of an embodiment. N program streams (N.gtoreq.2) are depicted, with the first one explicitly labeled space, with its corresponding set of audio signals fed through a corresponding renderer, each individually configured to play back its corresponding program stream through a common set of M arbitrarily spaced apart loudspeakers (M.gtoreq.2). The renderer may also be referred to herein as a "rendering module". The rendering module and mixer 130a may be implemented via software, hardware, firmware, or some combination thereof. In this example, the rendering module and mixer 130a are implemented via a control system 110a, which control system 110a is an example of the control system 110 described above with reference to fig. 1A. Each of the N renderers outputs a set of M loudspeaker feeds that are summed across all N renderers for simultaneous playback on the M loudspeakers. According to this embodiment, information about the layout of the M loudspeakers within the listening environment is provided to all the renderers, indicated by the dashed lines fed back from the loudspeaker box, so that the renderers can be properly configured for playback through the loudspeakers. The layout information may or may not be sent from one or more speakers themselves, depending on the particular implementation. According to some examples, the layout information may be provided by one or more intelligent speakers configured to determine a relative position of each of the M loudspeakers in the listening environment. Some such automatic positioning methods may be based on a direction of arrival (DOA) method or a time of arrival (TOA) method. In other examples, the layout information may be determined by another device and/or entered by a user. In some examples, loudspeaker specification information regarding the capabilities of at least some of the M loudspeakers within the listening environment may be provided to all of the renderers. Such microphone specification information may include impedance, frequency response, sensitivity, power rating, number and location of individual drivers, etc. According to this example, information from the rendering of one or more of the additional program streams is fed into the renderer of the main spatial stream, so that the rendering can be dynamically modified according to this information. This information is represented by the dashed line from rendering box 2 to rendering box N back up to rendering box 1.

Fig. 2A depicts another (more capable) embodiment with additional features. In this example, the rendering module and mixer 130b are implemented via a control system 110b, which control system 110b is an example of the control system 110 described above with reference to fig. 1A. In this version, the dashed line running up and down between all N renderers represents the idea that any one of the N renderers can contribute to the dynamic correction of any one of the remaining N-1 renderers. In other words, the rendering of any one of the N program streams may be dynamically modified based on a combination of one or more renderings of any of the remaining N-1 program streams. Additionally, any one or more of the program streams may be spatial mixes and the rendering of any program stream, whether or not it is spatial, may be dynamically modified based on any of the other program streams. For example, as described above, the loudspeaker layout information may be provided to the N renderers. In some examples, loudspeaker specification information may be provided to N renderers. In some implementations, the microphone system 120a may include a set of K microphones (K.gtoreq.1) within the listening environment. In some examples, the microphone(s) may be attached to or associated with one or more of the loudspeakers. These microphones may feed both the audio signal they capture (represented by solid lines) and additional configuration information (e.g., their location) (represented by dashed lines) back into a set of N renderers. Any of the N renderers may then make dynamic corrections based on the additional microphone input. Various examples are provided herein.

Examples of information derived from microphone input and subsequently used to dynamically modify any of the N renderers include, but are not limited to:

● Detection of a particular word or phrase utterance by a system user.

● An estimate of the location of one or more users of the system.

● An estimate of the loudness of any combination of N program streams at a particular location in a listening space.

● An estimate of the loudness of other ambient sounds in the listening environment, such as background noise.

Fig. 2B is a flowchart outlining one example of a method that may be performed by a device or system such as the device or system shown in fig. 1A, 1B, or 2A. As with other methods described herein, the blocks of method 200 need not be performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described. The blocks of method 200 may be performed by (or may include) a control system, such as one of the control systems 110, 110a, or 110B shown in fig. 1A, 1B, and 2A and described above, or other disclosed examples of control systems.

In this embodiment, block 205 involves receiving a first audio program stream via an interface system. In this example, the first audio program stream includes first audio signals arranged to be reproduced by at least some speakers of the environment. Here, the first audio program stream includes first spatial data. According to this example, the first spatial data includes channel data and/or spatial metadata. In some examples, block 205 relates to a first rendering module of a control system that receives a first audio program stream via an interface system.

According to this example, block 210 involves rendering a first audio signal for reproduction via speakers of an environment, resulting in a first rendered audio signal. For example, as described above, some examples of the method 200 involve receiving loudspeaker layout information. For example, as described above, some examples of the method 200 involve receiving loudspeaker specification information. In some examples, the first rendering module may generate the first rendered audio signal based at least in part on the loudspeaker layout information and/or the loudspeaker specification information.

In this example, block 215 involves receiving a second audio program stream via the interface system. In this embodiment, the second audio program stream comprises second audio signals arranged to be reproduced by at least some speakers of the environment. According to this example, the second audio program stream includes second spatial data. The second spatial data includes channel data and/or spatial metadata. In some examples, block 215 relates to a second rendering module of the control system receiving the second audio program stream via the interface system.

According to this embodiment, block 220 involves rendering the second audio signal for reproduction via speakers of the environment, thereby producing a second rendered audio signal. In some examples, the second rendering module may generate the second rendered audio signal based at least in part on the received loudspeaker layout information and/or the received loudspeaker specification information.

In some instances, some or all of the speakers in the environment may be arbitrarily positioned. For example, at least some of the speakers of the environment may be placed in locations that do not correspond to any standard prescribed speaker layout (e.g., dolby 5.1, dolby 7.1, kawasaki 22.2, etc.). In some such examples, at least some of the speakers of the environment may be placed in a convenient location (e.g., where there is room to accommodate the speakers) relative to furniture, walls, etc. of the environment, but without employing any standard prescribed speaker layout.

Accordingly, some embodiments block 210 or 220 may involve flexible rendering to arbitrarily positioned speakers. Some such implementations may involve centroid amplitude translation (CMAP), flexible Virtualization (FV), or a combination of both. At a high level, both techniques render a set of one or more audio signals, each audio signal having an associated desired perceived spatial location, for playback on a set of two or more speakers, wherein the relative activation of the set of speakers is a function of a model of the perceived spatial location of the audio signal for playback through the speakers and the proximity of the desired perceived spatial location of the audio signal to the speaker location. The model ensures that the listener hears the audio signal near its intended spatial location and that the proximity item controls which speakers are used to achieve the spatial impression. In particular, the proximity item facilitates activation of speakers near a desired perceived spatial location of the audio signal. For both CMAP and FV, the functional relationship can be conveniently derived from a cost function written as the sum of two terms, one term for spatial aspects and one term for proximity:

Here, the collectionRepresents the position of a group of M loudspeakers, < >>Represents the desired perceived spatial position of the audio signal and g represents the M-dimensional vector of speaker activation. For CMAP, each speaker activation in the vector represents the gain of each speaker, while for FV, each speaker activation represents a filter (in the second case, g can equivalently be considered a vector of complex values at a particular frequency, and different g is calculated across multiple frequencies to form a filter). The best vector for speaker activation can be found by minimizing the cost function across activations:

under certain definitions of the cost function, it may be difficult to control the absolute level of optimal activation resulting from the above minimization, although g _opt The relative level between the components of (c) is appropriate. To solve this problem, g may be performed _opt In order to control the absolute level of activation. For example, it may be desirable to normalize the vector to have a unit length, which complies with the usual constant power translation rules:

in some examples, the exact behavior of the flexible rendering algorithm may depend on C of the cost function _spatial And C _proximity Specific construction of these two items. For CMAP, C _spatial Can be derived from a model that will play audio messages from a set of loudspeakers The perceived spatial position of the sign is placed at the associated activation gain g by the loudspeaker _i The centroid of the positions of these loudspeakers weighted (elements of vector g):

equation 3 can then be manipulated to represent the spatial cost of the squared error between the desired audio position and the audio position produced by the activated loudspeaker:

for FV, the spatial terms of the cost function are defined differently. The goal is to generate and audio object positions at the listener's left and right earsThe corresponding binaural response b. Conceptually, b is the 2 x 1 vector of filters (one for each ear), but it is more convenient to consider it as a 2 x 1 vector of complex values at a particular frequency. Continuing with the representation at a particular frequency, the desired binaural response may be derived from a set of HRTF indices per object position:

meanwhile, the 2 x 1 binaural response e produced by the loudspeakers at the listener's ears may be modeled as an mx1 vector g of 2×m acoustic transmission matrix H multiplied by complex speaker activation values:

e＝Hg (6)

the acoustic transmission matrix H may be based on a set of loudspeaker positionsModeling with respect to listener position. Finally, the spatial components of the cost function may be defined as the desired binaural response (equation 5) and loudspeaker productionRaw binaural response (et al)

Square error between equation 6):

conveniently, the spatial terms of the cost functions for CMAP and FV defined in both equations 4 and 7 may be rearranged as a matrix quadratic function as a function of speaker activation g:

where a represents an mxm square matrix, B represents a 1×m vector, and C represents a scalar. In this example, the rank of matrix A is 2, and thus when M>2, there are an infinite number of speaker activations g with a spatial error term equal to zero. Second term C of introduction cost function _proximity This uncertainty is removed and a specific solution is produced that has perceptually beneficial properties compared to other possible solutions. For both CMAP and FV, C _proximity Can be constructed such that the positionFar from the desired audio signal position->Is more penalized than the activation of a speaker that is located close to the desired location. This construction results in a sparse optimal set of speaker activations in which only speakers that are close to the location of the desired audio signal will be significantly activated and in fact result in a spatial reproduction of the audio signal, which may be more perceptually robust to listener movements around the set of speakers.

To this end, a second term C of the cost function _proximity May be defined as a distance weighted sum of absolute squares of speaker activations. This can be expressed in a matrix form in a simplified way as:

where D represents a diagonal matrix of distance penalties between the desired audio location and each speaker:

the distance penalty function may take many forms, but the following are useful parameterizations:

wherein,,representing the euclidean distance between the desired audio position and the speaker position, and alpha and beta represent the adjustable parameters. The parameter α indicates the global strength of the penalty; d, d ₀ Corresponding to the spatial extent of the distance penalty (at about d ₀ Loudspeakers at or further away will be penalized), and β accounts for the distance d ₀ Punishment of the abrupt nature of the initiation.

Combining the two terms of the cost function defined in equations 8 and 9a yields the overall cost function:

for this example, setting the derivative of the cost function with respect to g to zero and solving for g yields the best speaker activation solution:

in general, the optimal solution in equation 11 may produce speaker activation with a negative value. For CMAP construction of a flexible renderer, such negative activation may be undesirable, and thus in some examples, equation (11) may be minimized with all activations remaining positive.

Fig. 2C and 2D are diagrams illustrating an example set of speaker activations and object rendering positions. In these examples, speaker activation and object rendering positions correspond to speaker positions of 4, 64, 165, -87, and-4 degrees. Fig. 2C shows speaker activations 245a, 250a, 255a, 260a, and 265a, which include the optimal solution of equation 11 for these particular speaker locations. Fig. 2D depicts individual speaker locations as squares 267, 270, 272, 274, and 275, corresponding to speaker activations 245a, 250a, 255a, 260a, and 265a, respectively. Fig. 2D also shows ideal object positions (in other words, positions where audio objects are to be rendered) for a large number of possible object angles as points 276a, and corresponding actual rendering positions for these objects as points 278a, connected to the ideal object positions by dashed lines 279 a.

One class of embodiments relates to a method for rendering audio for playback by at least one (e.g., all or some) of a plurality of coordinated (orchestrated) intelligent audio devices. For example, a set of smart audio devices present in (in a system of) a user's home may be orchestrated to handle various simultaneous use cases, including flexible rendering (according to embodiments) of audio for playback by all or some of the smart audio devices (i.e., by speaker(s) of all or some of the smart audio devices). Many interactions with the system are considered, which require dynamic modification of the rendering. Such correction may, but need not, be focused on spatial fidelity.

Some embodiments are methods for rendering audio for playback by at least one (e.g., all or some) of a set of intelligent audio devices (or for playback by at least one (e.g., all or some) of another set of speakers). Rendering may include minimizing a cost function, wherein the cost function includes at least one dynamic speaker activation term. Examples of such dynamic speaker activation items may include (but are not limited to):

● The proximity of the speaker to one or more listeners;

● Proximity of the speaker to the attractive or repulsive force;

● Audibility of the speaker with respect to some locations (e.g., listener location or baby room);

● Capacity of the speaker (e.g., frequency response and distortion);

● Synchronization of speakers with respect to other speakers;

● Wake word performance; and

● Echo canceller performance.

The dynamic speaker activation item(s) may enable at least one of a variety of actions, including distorting the spatial presentation of audio away from a particular smart audio device so that a speaker may be better heard by the microphone of the particular smart audio device or so that a secondary audio stream may be better heard from the speaker(s) of the smart audio device.

Some embodiments implement rendering for coordinated (orchestrated) speaker(s) playback of multiple intelligent audio devices. Other embodiments implement rendering for playback by speaker(s) of another set of speakers.

Pairing a flexible rendering method (implemented according to some embodiments) with a set of wireless smart speakers (or other smart audio devices) can result in a very capable and easy-to-use spatial audio rendering system. When considering interactions with such systems, it is clearly desirable to dynamically modify the spatial rendering in order to optimize for other objectives that may occur during use of the system. To achieve this objective, one class of embodiments enhances existing flexible rendering algorithms (where speaker activation is a function of the previously disclosed spatial and proximity terms) with one or more additional dynamically configurable functions that depend on one or more properties of the audio signal being rendered, the speaker group, and/or other external inputs. According to some embodiments, the cost function of the existing flexible rendering given in equation 1 may add these one or more additional dependent terms, for example, according to the following equation:

In equation 12, the termRepresents an additional cost item, and->A set of one or more properties representing an audio signal being rendered (e.g., an object-based audio program), and +.>A set of one or more properties representing a loudspeaker that is rendering audio, and +.>Representing one or more additional external inputs. Each itemThe return cost as a function of the activation g associated with the combination of one or more properties of the audio signal, the loudspeaker and/or the external input is generally defined by the set +.>And (3) representing. It should be appreciated that in this example, the collectionAt least comprise from->Or->An element of any one of the above.

Examples of (a) include, but are not limited to:

● A desired perceived spatial position of the audio signal;

● The level of the audio signal (possibly time-varying); and/or

● The spectrum of the audio signal (which may vary over time).

Examples of (a) include, but are not limited to:

● The position of the loudspeaker in the listening space;

● The frequency response of the loudspeaker;

● Playback level limitations of the loudspeaker;

● Parameters of the dynamic processing algorithm inside the loudspeaker, such as limiter gain;

● Measurement or estimation of acoustic transmissions from each speaker to other speakers;

● Measurement of echo canceller performance on a loudspeaker; and/or

● The speakers are relatively synchronized with respect to each other.

Examples of (a) include, but are not limited to:

● The location of one or more listeners or speakers in the playback space;

● A measurement or estimate of acoustic transmission from each loudspeaker to the listening position;

● Measurement or estimation of acoustic transmissions from a speaker to a set of microphones;

● The location of some other landmark in the playback space; and/or

● A measurement or estimate of acoustic transmission from each speaker to some other landmark in the playback space;

using the new cost function defined in equation 12, the optimal activation group can be found by minimization and possibly post normalization with respect to g as previously specified in equations 2a and 2 b.

FIG. 2E is a flowchart outlining one example of a method that may be performed by a device or system, such as the device or system shown in FIG. 1A. As with the other methods described herein, the blocks of method 280 need not be performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described. The blocks of method 280 may be performed by one or more devices, which may be (or may include) a control system, such as control system 110 shown in fig. 1A.

In this embodiment, block 285 involves receiving, by the control system and via the interface system, audio data. In this example, the audio data includes one or more audio signals and associated spatial data. According to this embodiment, the spatial data indicates an expected perceived spatial position corresponding to the audio signal. In some examples, the desired perceived spatial location may be explicit, for example, as indicated by location metadata such as dolby panoramic location metadata. In other examples, the desired perceived spatial location may be implicit, e.g., the desired perceived spatial location may be a hypothetical location associated with a channel according to dolby 5.1, dolby 7.1, or other channel-based audio format. In some examples, block 285 relates to a rendering module of a control system that receives audio data via an interface system.

According to this example, block 290 involves rendering, by the control system, audio data for reproduction via a set of loudspeakers of the environment, resulting in a rendered audio signal. In this example, rendering each of the one or more audio signals included in the audio data involves determining relative activation of a set of loudspeakers in the environment by optimizing a cost function. According to this example, the cost is a function of a model of the perceived spatial location of the audio signal when played back on the set of loudspeakers in the environment. In this example, the cost is also a function of a measure of the proximity of the intended perceived spatial position of the audio signal to the position of each loudspeaker in the set of loudspeakers. In this embodiment, the cost is also a function of one or more additional dynamically configurable functions. In this example, the dynamically configurable functionality is based on one or more of the following: proximity of the loudspeakers to one or more listeners; proximity of the loudspeaker to the attractive force location, wherein attractive force is a factor in favor of relatively higher loudspeaker activation closer to the attractive force location; proximity of the loudspeaker to the repulsive force location, wherein repulsive force is a factor in favor of a relatively lower loudspeaker activation closer to the repulsive force location; the ability of each loudspeaker to be relative to other loudspeakers in the environment; the synchronization of the loudspeaker with respect to the other loudspeakers; wake word performance; or echo canceller performance.

In this example, block 295 involves providing the rendered audio signal to at least some of the group of loudspeakers of the environment via the interface system.

According to some examples, the model of perceived spatial locations may produce binaural responses corresponding to audio object locations at the listener's left and right ears. Alternatively or additionally, the model of perceived spatial location may place the perceived spatial location of the audio signal played from a set of loudspeakers at the centroid of the locations of the set of loudspeakers weighted by the associated activation gains of the loudspeakers.

In some examples, the one or more additional dynamically configurable functions may be based at least in part on a level of the one or more audio signals. In some examples, the one or more additional dynamically configurable functions may be based at least in part on a spectrum of the one or more audio signals.

Some examples of method 280 involve receiving loudspeaker layout information. In some examples, the one or more additional dynamically configurable functions may be based at least in part on the location of each loudspeaker in the environment.

Some examples of method 280 involve receiving loudspeaker specification information. In some examples, the one or more additional dynamically configurable functions may be based at least in part on the capabilities of each loudspeaker, which may include one or more of the following: frequency response, playback level limits, or parameters of one or more loudspeaker dynamic processing algorithms.

According to some examples, the one or more additional dynamically configurable functions may be based at least in part on measurements or estimates of acoustic transmissions from each loudspeaker to other loudspeakers. Alternatively or additionally, the one or more additional dynamically configurable functions may be based at least in part on listener or speaker positions of one or more persons in the environment. Alternatively or additionally, the one or more additional dynamically configurable functions may be based at least in part on measurements or estimates of acoustic transmissions from each loudspeaker to a listener or speaker location. The estimation of the acoustic transmission may be based, for example, at least in part, on walls, furniture, or other objects that may reside between each loudspeaker and the listener or speaker location.

Alternatively or additionally, the one or more additional dynamically configurable functions may be based at least in part on object locations of one or more non-loudspeaker objects or landmarks in the environment. In some such embodiments, the one or more additional dynamically configurable functions may be based at least in part on a measurement or estimate of acoustic transmission from each loudspeaker to the object location or landmark location.

Flexible rendering may be implemented by employing one or more appropriately defined additional cost terms to achieve many new and useful behaviors. All of the example acts listed below are to penalize certain loudspeakers under certain conditions that are deemed undesirable. The end result is that these loudspeakers are less activated in the spatial rendering of a set of audio signals. In many of these cases, one might consider simply turning down the undesired loudspeakers, regardless of any modification to the spatial rendering, but such a strategy might significantly reduce the overall balance of the audio content. For example, certain components of the mix may become completely inaudible. On the other hand, for the disclosed embodiments, integrating these penalties into the core optimization of the rendering allows the rendering adaptation and uses the remaining less penalized speakers to perform the best possible spatial rendering. This is a more elegant, adaptable, more efficient solution.

Example uses include, but are not limited to:

● Providing a more balanced spatial presentation around a listening area

It has been found that spatial audio is preferably presented across a loudspeaker at approximately the same distance from the intended listening area. The cost may be built such that loudspeakers that are significantly closer or farther than the average distance of the loudspeakers to the listening area are penalized, thus reducing the activation of the loudspeakers;

● Moving audio away or towards a listener or speaker

If a user of the system is attempting to speak with an intelligent voice assistant of or associated with the system, it may be beneficial to create a cost that penalizes loudspeakers closer to the speaker. In this way, the loudspeakers are less active, allowing the speaker's associated microphone to hear the speaker better;

to provide a more intimate experience for a single listener, minimizing playback levels for others in the listening space, speakers located far from the listener may be severely penalized so that only the speaker closest to the listener is most significantly activated;

● Moving audio away from or towards a landmark, zone or area

Certain locations near the listening space may be considered sensitive, such as a baby crib, office, reading area, learning area, etc. In this case, a penalty may be built for the cost of using speakers near the location, zone, or area;

Alternatively, for the same (or similar) situation above, the speaker system may have generated a measurement of the acoustic transmission from each speaker into the baby room, especially when one of the speakers (with attached or associated microphone) resides within the baby room. In this case, instead of using the physical proximity of the speaker to the baby room, a penalty may be built for using the cost of the speaker for high acoustic transmission to the room measured; and/or

● Optimal use of the capabilities of a loudspeaker

The capabilities of the different loudspeakers may vary significantly. For example, one popular smart speaker contains only a single 1.6 "full range driver with limited low frequency capability. On the other hand, another intelligent speaker contains a more capable 3 "woofer. These capabilities are typically reflected in the frequency response of the speaker and as such, a set of responses associated with the speaker may be utilized in a cost term. At a particular frequency, speakers that are weaker relative to other speakers (as measured from their frequency response) are penalized and therefore activated to a lesser extent. In some implementations, such frequency response values may be stored with the smart loudspeakers and then reported to a computing unit responsible for optimizing the flexible rendering;

Many speakers contain more than one driver, each of which is responsible for playing a different frequency range. For example, one popular smart speaker is a two-way design, including a woofer for lower frequencies and a tweeter for higher frequencies. Typically, such speakers contain crossover circuitry for dividing the full range playback audio signal into appropriate frequency ranges and transmitting to the corresponding drivers. Alternatively, such speakers may provide flexible renderer playback access for each individual driver, as well as information about the capabilities (e.g., frequency response) of each individual driver. By applying the cost term as described above, in some examples, the flexible renderer may automatically establish a frequency divider between two drivers based on its relative capabilities at different frequencies;

the example use of frequency response described above focuses on the inherent capabilities of speakers, but may not accurately reflect the capabilities of speakers placed in a listening environment. In some cases, the speaker frequency response as measured at the intended listening position may be obtained by some calibration procedure. Such measurements may be used instead of pre-calculated responses to better optimize the use of the speaker. For example, a speaker may be inherently very capable at a particular frequency, but may produce a very limited response at an intended listening position due to its placement (e.g., behind a wall or piece of furniture). Capturing the response and feeding the measurement into an appropriate cost term may prevent significant activation of such speakers;

The o frequency response is only one aspect of the playback capability of the loudspeaker. Many smaller loudspeakers start to distort and then reach their offset limits as the playback level increases, especially for lower frequencies. To reduce this distortion, many loudspeakers implement a dynamic process that limits the playback level below some limiting threshold that may vary with frequency. In the case where the speakers are close to or at these thresholds, while other speakers participating in the flexible rendering are not, it makes sense to reduce the signal level in limiting the speakers and transfer this energy to other less burdened speakers. According to some embodiments, this behavior may be automatically achieved by appropriately configuring the associated cost items. Such cost terms may relate to one or more of the following:

■ A global playback volume associated with a limiting threshold of the loudspeaker is monitored. For example, a loudspeaker whose volume level is close to its limit threshold may be subject to more penalty;

■ The dynamic signal level (and possibly also frequency) associated with the loudspeaker limit threshold (and possibly frequency) is monitored. For example, a loudspeaker whose monitored signal level is close to its limit threshold may be subject to more penalty;

■ Parameters of the dynamic processing of the loudspeaker, such as limiting gain, are monitored directly. In some such examples, the parameter indicates that a more restrictive loudspeaker may be subject to more penalty; and/or

■ The actual instantaneous voltage, current and power delivered by the amplifier to the microphone is monitored to determine whether the microphone is operating in a linear range. For example, a loudspeaker that operates less linearly may be subject to more penalties;

intelligent speakers with integrated microphones and interactive voice assistants typically employ some type of echo cancellation to reduce the level of audio signals played by the speaker picked up by the recording microphone. The greater this decrease, the greater the chance that the speaker will hear and understand the speaker in space. If the residual of the echo canceller is always high, this may indicate that the speaker is being driven into a non-linear region where prediction of the echo path becomes challenging. In this case, it may be meaningful to divert signal energy away from the speaker, and as such, it may be beneficial to consider a cost term of echo canceller performance. Such a cost term may assign high costs to speakers whose associated echo cancellers perform poorly;

In order to achieve predictable imaging when rendering spatial audio over multiple loudspeakers, playback over a set of loudspeakers is often required to be reasonably synchronized across time. For a wired loudspeaker this is given, but for a large number of wireless loudspeakers, synchronization can be challenging and the end result variable. In this case, it may be possible for each loudspeaker to report its relative degree of synchronization with the target, and this degree may then be fed into the synchronization cost term. In some such examples, loudspeakers with lower degrees of synchrony may be subject to more penalty and are therefore excluded from rendering. Additionally, certain types of audio signals may not need to be closely synchronized, e.g., components of an audio mix that are intended to be diffuse or non-directional. In some implementations, the components may be tagged with metadata as such, and the synchronization cost term may be revised such that the penalty is reduced.

Examples of embodiments are described next.

Each new cost function term will be similar to the proximity cost defined in equations 9a and 9bIt is also convenient to express as a weighted sum of the absolute squares of the loudspeaker activations:

Wherein W is _j Representing weightsDescribing the cost associated with activating speaker i for item j:

combining equations 13a and 13b with the matrix quadratic version of the CMAP and FV cost functions given in equation 10 yields a potentially advantageous implementation of the generic extended cost function (of some embodiments) given in equation 12:

C(g)＝g ^* Ag+Bg+C+g ^* Dg+∑ _j g ^* W _j g＝g ^* (A+D+∑ _j W _j )g+Bg+C (14)

with this definition of the new cost function term, the overall cost function is still matrix quadratic and the optimal set of activations g can be found by differentiation of equation 14 _opt To produce

Weight item w _ij Is considered as a given continuous penalty value for each of the loudspeakersIs useful. In an example embodiment, the penalty value is the distance from the object (to be rendered) to the loudspeaker under consideration. In another example embodiment, the penalty value indicates that a given loudspeaker cannot reproduce some frequencies. Based on the penalty value, the weight term w _ij The parameters can be as follows:

wherein alpha is _j Representing a pre-factor (which takes into account the global strength of the weight term), where τ _j Represents a penalty threshold (about or exceeding which the weight term becomes significant), and wherein f _j (x) Representing a monotonically increasing function. For example, have The weight term has the following form: />

Wherein alpha is _j 、β _j 、τ _j Is an adjustable parameter that indicates the global strength of the penalty, the abrupt nature of the penalty initiation, and the degree of penalty, respectively. Care should be taken in setting these adjustable values so that cost term C _j Relative to any other additional cost terms and C _spatial And C _proximity Is suitable for achieving the desired result. For example, empirically, if one wishes a particular penalty to obviously support other penalties, one will have its intensity α _j It may be appropriate to set about ten times the next maximum penalty strength.

If all loudspeakers are penalized, it is often convenient to subtract the minimum penalty from all weight terms in the post-processing so that at least one of the loudspeakers is not penalized:

w _ij →w′ _ij ＝w _ij -min _i (w _ij ) (18)

as described above, many possible use cases may be implemented using the new cost function terms described herein (and similar new cost function terms employed in accordance with other embodiments). Next, more specific details are described with the following three examples: moving audio toward a listener or speaker, moving audio away from a listener or speaker, and moving audio away from a landmark.

In a first example, what will be referred to herein as "attraction" is used to pull audio toward a location, which in some examples may be a listener or speaker's location, landmark location, furniture location, etc. This position may be referred to herein as an "attractive force position" or "attractor position". As used herein, "attractive force" is a relative greater advantage in favor of a closer position of attractive force High loudspeaker activation. According to this example, the weight w _ij In the form of equation 17, a continuous penalty value p _ij Fixed attractor position from ith speakerIs given by the distance of (2), and the threshold τ _j Given by the maximum of these distances for all speakers:

and (19 a)

To illustrate the use of "pulling" the audio towards the listener or speaker, in particular set α _j ＝20，β _j =3, and willSet as a vector corresponding to a listener/speaker position of 180 degrees. Alpha _j 、β _j And->These values of (2) are merely examples. In some embodiments, α _j May be in the range of 1 to 100 and beta _j May be in the range of 1 to 25.

Fig. 2F is a diagram of speaker activation in an example embodiment. In this example, FIG. 2F shows speaker activations 245b, 250b, 255b, 260b, and 265b, which include optimal solutions of the cost functions for the same speaker locations in FIGS. 1 and 2, plus the sum of w _ij The attractive force of the representation. FIG. 2G is a diagram of object rendering locations in an example embodiment. In this example, FIG. 2G shows corresponding ideal object positions 276b for a number of possible object angles and corresponding actual rendering positions 278b for those objects, connected to ideal object positions 276b by dashed lines 279 b. The actual rendering position 278b is oriented towards the fixed position Device for placing articlesThe effect of the attractor weights on the optimal solution of the cost function is illustrated.

In the second and third examples, a "repulsive force" is used to "push" the audio away from a position, which may be a listener position, a speaker position, or other positions, such as a landmark position, a furniture position, etc. In some examples, the repulsive force may be used to push audio out of an area or zone of the listening environment, such as an office area, a reading area, a bed or bedroom area (e.g., crib or bedroom), and so forth. According to some such examples, a particular location may be used as a representation of a zone or region. For example, the position representing the crib may be an estimated position of the head of the baby, an estimated sound source position corresponding to the baby, or the like. This position may be referred to herein as a "repulsive position" or "repulsive position". As used herein, "repulsive force" is a factor that favors relatively lower loudspeaker activation nearer the repulsive force location. According to this example, the repulsive position is relatively fixedDefinition p _ij And τ _j Similar to the attractive force in equation 19:

and (19 c)

To illustrate the use of "pushing" audio away from a listener or speaker, specifically set α _j ＝5，β _j =2, and willSet as a vector corresponding to a listener/speaker position of 180 degrees. Alpha _j βx and->These values of (2) are merely examples. As described above, in some examples, α _j May be in the range of 1 to 100 and beta _j May be in the range of 1 to 25. Fig. 2H is a diagram of speaker activation in an example embodiment. According to this example, FIG. 2H shows speaker activations 245c, 250c, 255c, 260c, and 265c, which include an optimal solution of the cost function for the same speaker location as in the previous figures, plus the sum of w _ij The repulsive force is shown. FIG. 2I is a diagram of an object rendering location in an example embodiment. In this example, FIG. 2I shows ideal object positions 276c for a large number of possible object angles and corresponding actual rendering positions 278c for those objects, which are connected to ideal object positions 276c by dashed lines 279 c. The actual rendering position 278c leaves the fixed position +.>The impact of the repulsive sub-weights on the optimal solution of the cost function is illustrated.

A third example use case is to "push" audio off an acoustically sensitive landmark, such as a door to a room of a sleeping infant. Similar to the last example, willSet as a vector corresponding to a gate position of 180 degrees (bottom center of the plot). In order to achieve a stronger repulsive force and tilt the sound field completely to the front of the main listening space, a is set _j ＝20，β _j =5. Fig. 2J is a diagram of speaker activation in an example embodiment. Again, in this example, fig. 2J shows speaker activations 245d, 250d, 255d, 260d, and 265d, which include optimal solutions for the same set of speaker locations, plus a stronger repulsive force. FIG. 2K is a diagram of object rendering locations in an example embodiment. And again, in this example, FIG. 2K shows ideal object positions 276d for a large number of possible object angles and corresponding actual rendering positions 278d for those objects, theThe corresponding actual rendering location is connected to the ideal object location 276d by a dashed line 279 d. The oblique orientation of the actual rendering location 278d illustrates the effect of the stronger repulsive sub-weights on the optimal solution of the cost function.

Returning now to fig. 2B, in this example, block 225 involves modifying a rendering process for the first audio signal based at least in part on at least one of the second audio signal, the second rendered audio signal, or a characteristic thereof to produce a modified first rendered audio signal. Various examples of a revised rendering process are disclosed herein. For example, the "characteristics" of the rendered signal may include the loudness or audibility estimated or measured at the intended listening position, whether in silence or in the presence of one or more additional rendered signals. Other examples of characteristics include parameters associated with the rendering of the signal, such as the expected spatial position of the constituent signals of the associated program stream, the position of the loudspeaker on which the signal is rendered, the relative activation of the loudspeaker according to the expected spatial position of the constituent signals, and any other parameters or states associated with the rendering algorithm used to generate the rendered signal. In some examples, block 225 may be performed by the first rendering module.

According to this example, block 230 involves modifying a rendering process for the second audio signal based at least in part on at least one of the first audio signal, the first rendered audio signal, or a characteristic thereof to produce a modified second rendered audio signal. In some examples, block 230 may be performed by a second rendering module.

In some implementations, modifying the rendering process for the first audio signal may involve warping the rendering of the first audio signal away from the rendering location of the second rendered audio signal and/or modifying the loudness of one or more of the first rendered audio signals in response to the loudness of one or more of the second audio signal or the second rendered audio signal. Alternatively or additionally, modifying the rendering process for the second audio signal may involve distorting the rendering of the second audio signal away from the rendering location of the first rendered audio signal and/or modifying the loudness of one or more of the second rendered audio signal in response to the loudness of one or more of the first audio signal or the first rendered audio signal. Some examples are provided below with reference to fig. 3 and below.

However, other types of rendering process modifications are within the scope of the present disclosure. For example, in some instances, modifying the rendering process of the first audio signal or the second audio signal may involve performing a spectral modification, an audibility-based modification, or a dynamic range modification. These modifications may or may not be related to loudness-based rendering modifications, depending on the particular example. For example, where the aforementioned primary spatial stream is rendered in an open-plan living area and the secondary stream comprising cooking cues is rendered in an adjacent kitchen, it may be desirable to ensure that the cooking cues remain audible in the kitchen. This may be done by estimating the loudness of the rendered cooking cue stream in the kitchen without the interfering first signal, then estimating the loudness of the first signal present in the kitchen, and finally dynamically modifying the loudness and dynamic range of the two streams across multiple frequencies, thereby ensuring audibility of the second signal in the kitchen.

In the example shown in fig. 2B, block 235 involves mixing at least the modified first rendered audio signal and the modified second rendered audio signal to produce a mixed audio signal. For example, block 235 may be performed by mixer 130b shown in fig. 2A.

According to this example, block 240 involves providing the mixed audio signal to at least some speakers of the environment. Some examples of the method 200 involve playback of the mixed audio signal by a speaker.

As shown in fig. 2B, some implementations may provide more than 2 rendering modules. Some such implementations may provide N rendering modules, where N is an integer greater than 2. Thus, some such implementations may include one or more additional rendering modules. In some such examples, each of the one or more additional rendering modules may be configured to receive additional audio program streams via the interface system. The additional audio program stream may comprise additional audio signals arranged to be reproduced by at least one speaker of the environment. Some such implementations may involve rendering the additional audio signal to be reproduced via at least one speaker of the environment to produce an additional rendered audio signal, and modifying a rendering process of the additional audio signal based at least in part on at least one of the first audio signal, the first rendered audio signal, the second rendered audio signal, or a characteristic thereof to produce a modified additional rendered audio signal. According to some such examples, the mixing module may be configured to mix the modified additional rendered audio signal with at least the modified first rendered audio signal and the modified second rendered audio signal to produce a mixed audio signal.

As described above with reference to fig. 1A and 2A, some embodiments may include a microphone system that includes one or more microphones in a listening environment. In some such examples, the first rendering module may be configured to modify a rendering process of the first audio signal based at least in part on the first microphone signal from the microphone system. The "first microphone signal" may be received from a single microphone or from 2 or more microphones, depending on the particular implementation. In some such implementations, the second rendering module may be configured to modify a rendering process of the second audio signal based at least in part on the first microphone signal.

As described above with reference to fig. 2A, in some examples, the location of one or more microphones may be known and may be provided to a control system. According to some such embodiments, the control system may be configured to estimate a first sound source position based on the first microphone signal and to modify the rendering process of at least one of the first audio signal or the second audio signal based at least in part on the first sound source position. For example, the first sound source location may be estimated according to a triangulation procedure based on DOA data from each of three or more microphones or groups of microphones having known locations. Alternatively or additionally, the first sound source position may be estimated from the amplitudes of signals received from two or more microphones. The microphone producing the highest amplitude signal may be assumed to be closest to the first sound source location. In some such examples, the first sound source location may be set to the location of the nearest microphone. In some such examples, the first sound source location may be associated with a location of a zone, where the zone is selected by processing signals from two or more microphones through a pre-trained classifier (e.g., a gaussian mixer model).

In some such embodiments, the control system may be configured to determine whether the first microphone signal corresponds to ambient noise. Some such implementations may involve modifying a rendering process of at least one of the first audio signal or the second audio signal based at least in part on whether the first microphone signal corresponds to ambient noise. For example, if the control system determines that the first microphone signal corresponds to ambient noise, modifying the rendering process of the first audio signal or the second audio signal may involve increasing the level of the rendered audio signal such that the perceived loudness of the signal in the presence of noise at the intended listening position is substantially equal to the perceived loudness of the signal in the absence of noise.

In some examples, the control system may be configured to determine whether the first microphone signal corresponds to human speech. Some such implementations may involve modifying a rendering process of at least one of the first audio signal or the second audio signal based at least in part on whether the first microphone signal corresponds to human speech. For example, if the control system determines that the first microphone signal corresponds to human speech (e.g., a wake word), the rendering process of modifying the first audio signal or the second audio signal may involve reducing the loudness of the rendered audio signal reproduced by speakers in the vicinity of the first sound source location as compared to the loudness of the rendered audio signal reproduced by speakers remote from the first sound source location. The rendering process of modifying the first audio signal or the second audio signal may alternatively or additionally involve modifying the rendering process to distort the intended positions of the constituent signals of the associated program streams away from the first sound source position and/or penalize the use of speakers in the vicinity of the first sound source position compared to speakers away from the first sound source position.

In some implementations, if the control system determines that the first microphone signal corresponds to human speech, the control system may be configured to reproduce the first microphone signal in one or more speakers near a location of an environment different from the first sound source location. In some such examples, the control system may be configured to determine whether the first microphone signal corresponds to a crying of the child. According to some such embodiments, the control system may be configured to reproduce the first microphone signal in one or more speakers near an environmental location corresponding to an estimated location of a caregiver (e.g., parent, relative, guardian, child care service provider, teacher, nurse, etc.). In some examples, the process of estimating the estimated position of the caretaker may be triggered by a voice command such as "< wake word >, not waking the baby". The control system will be able to estimate the location of the speaker (caregivers) from the location of the nearest intelligent audio device implementing the virtual assistant, by triangulation based on the DOA information provided by three or more local microphones, etc. According to some embodiments, the control system will have a priori knowledge of the baby room location (and/or listening devices therein) and then be able to perform the appropriate processing.

According to some such examples, the control system may be configured to determine whether the first microphone signal corresponds to a command. In some examples, if the control system determines that the first microphone signal corresponds to a command, the control system may be configured to determine a reply to the command and control at least one speaker in the vicinity of the first sound source location to reproduce the reply. In some such examples, the control system may be configured to revert to an unmodified rendering process of the first audio signal or the second audio signal after controlling the at least one speaker near the first sound source location to reproduce the reply.

In some implementations, the control system may be configured to execute the command. For example, the control system may be or may include a virtual assistant configured to control audio devices, televisions, home appliances, and the like, upon command.

By this definition of the minimum and more capable multi-stream rendering system shown in fig. 1A, 1B and 2A, dynamic management of simultaneous playback of multiple program streams can be achieved for many useful scenarios. Several examples will now be described with reference to fig. 3A and 3B.

First examine the previously discussed example involving playback of a spatial movie soundtrack in the living room and playback of a cooking tip in a connected kitchen at the same time. The spatial movie soundtrack is an example of the above-mentioned "first audio program stream", and the cooking tip audio is an example of the above-mentioned "second audio program stream". Fig. 3A and 3B show plan view examples of connected living spaces. In this example, living space 300 includes a living room at the upper left, a kitchen at the lower center, and a bedroom at the lower right. The boxes and circles 305a to 305h distributed across living space represent a set of 8 loudspeakers placed in a convenient location for space, but not in compliance with any standard prescribed layout (arbitrary placement). In fig. 3A, only the spatial movie soundtrack is played back, and all loudspeakers in living room 310 and kitchen 315 are used to create an optimized spatial reproduction around listener 320a sitting on sofa 325 facing television 330, taking into account the capacity and layout of the loudspeakers. This optimal reproduction of the movie soundtrack is visually represented by cloud line 335a which is located within the boundary of the active loudspeaker.

In fig. 3B, the cooking cues are simultaneously rendered and played back for the second listener 320B on a single loudspeaker 305g in the kitchen 315. The reproduction of this second program stream is visually represented by a cloud line 340 emanating from the loudspeaker 305 g. If these cooking cues are played back simultaneously without modifying the rendering of the movie soundtrack, as shown in fig. 3A, audio from the movie soundtrack coming from speakers in or near the kitchen 315 may interfere with the ability of the second listener to understand the cooking cues. Instead, in this example, the rendering of the spatial movie soundtrack is dynamically modified according to the rendering of the cooking cues. In particular, the rendering of the movie soundtrack is moved away from the speaker near the rendering location of the cooking cues (kitchen 315), with the smaller cloudy line 335B in fig. 3B being pushed away from the speaker near the kitchen visually representing the shift. In some implementations, if the cooking tip playback stops while the movie soundtrack is being played, the rendering of the movie soundtrack may be dynamically moved back to its original optimal configuration as seen in fig. 3A. Such dynamic shifting in the rendering of a spatial movie soundtrack may be achieved by many of the disclosed methods.

Many spatial audio mixes include a plurality of constituent audio signals designed to be played at specific locations in a listening space. For example, dolby 5.1 and 7.1 surround sound mixes are made up of 6 and 8 signals, respectively, meaning playback on speakers at specified canonical locations around the listener. An object-based audio format (e.g., dolby panoramic sound (Dolby Atmos)) is composed of constituent audio signals with associated metadata that describes possible time-varying 3D locations in a listening space where audio is to be rendered. Assuming that the renderer of the spatial movie soundtrack is able to render individual audio signals at any location with respect to any loudspeaker group, dynamic shifting of the rendering depicted in fig. 3A and 3B may be achieved by warping the intended locations of the audio signals within the spatial mix. For example, the 2D or 3D coordinates associated with the audio signal may be pushed away from the location of the speaker in the kitchen or alternatively pulled toward the upper left corner of the living room. The result of this warping is that loudspeakers in the vicinity of the kitchen are less used, since the warped position of the spatially mixed audio signal is now further away from this position. While this approach does achieve the goal of making the second audio stream more intelligible to the second listener, it does so at the cost of significantly changing the intended spatial balance of the movie soundtrack for the first listener.

A second method for implementing dynamic shifting to spatial rendering may be implemented by using a flexible rendering system. In some such embodiments, the flexible rendering system may be CMAP, FV, or a mixture of both, as described above. Some such flexible rendering systems attempt to reproduce a spatial mix whose all constituent signals are considered to be from their intended locations. In some examples, while doing so for each signal of the mix, priority is given to activating a loudspeaker near the desired location of that signal. In some implementations, additional items may be dynamically added to the optimization of rendering, which penalizes the use of certain loudspeakers based on other criteria. For example, what may be referred to as a "repulsive force" may be dynamically placed in a kitchen's location, highly penalizing the use of loudspeakers in the vicinity of that location and effectively pushing away the rendering of the spatial movie soundtrack. As used herein, the term "repulsive force" may refer to a factor corresponding to a relatively low speaker activation in a particular location or zone of the listening environment. In other words, the phrase "repulsive force" may refer to a factor that facilitates activation of a speaker relatively far from a particular location or region corresponding to the "repulsive force". However, according to some such embodiments, the renderer may still attempt to reproduce the desired spatial balance of the mix using the remaining, less penalized speakers. As such, this technique may be considered as a superior method of implementing dynamic shifting of rendering, compared to a method of simply warping the expected positions of constituent signals of a mix.

The described scenario of moving the rendering of the spatial movie soundtrack away from the cooking cues in the kitchen may be implemented using a minimal version of the multi-stream renderer depicted in fig. 1B. However, improvements to this scenario may be achieved by employing the more capable system depicted in fig. 2A. While shifting the rendering of the spatial movie soundtrack does improve the intelligibility of the cooking cues in the kitchen, the movie soundtrack may still be clearly audible in the kitchen. Depending on the instantaneous conditions of the two streams, the cooking cues may be masked by the movie soundtrack; for example, a loud moment in a movie soundtrack masks a soft moment in a cooking tip. To address this issue, dynamic corrections to the rendering of the cooking cues may be added in accordance with the rendering of the spatial movie soundtrack. For example, a method for dynamically changing an audio signal across frequency and time to maintain its perceived loudness in the presence of interfering signals may be performed. In this scenario, an estimate of the perceived loudness of the shifted movie soundtrack at the kitchen location may be generated and fed into such a process as an interference signal. The time and frequency variation levels of the cooking cues may then be dynamically modified to maintain their perceived loudness above the disturbance, thereby better maintaining intelligibility for the second listener. The desired estimate of the loudness of a movie soundtrack in the kitchen may be from a rendered speaker feed of the soundtrack, a signal from a microphone in or near the kitchen, or a combination thereof. This process of maintaining the perceived loudness of the cooking cues will typically increase the level of the cooking cues, and in some cases the overall loudness may become objectionable. To solve this problem, still another rendering correction may be employed. The interference space movie soundtrack may be dynamically tuned down in response to the loudness-modified cooking cues in the kitchen becoming too loud. Finally, some external noise sources may interfere with the audibility of both program streams simultaneously; for example, a blender may be used in the kitchen during cooking. The loudness estimates of the ambient noise sources in both the living room and the kitchen may be generated by microphones connected to the rendering system. For example, the estimate may be added to an estimate of the loudness of the soundtrack in the kitchen to affect the loudness modification of the cooking cues. At the same time, the rendering of the sound tracks in the living room may be additionally modified according to the ambient noise estimate to maintain the perceived loudness of the sound tracks in the living room in the presence of the ambient noise, thereby better maintaining audibility to listeners in the living room.

As can be seen, this example use case of the disclosed multi-stream renderer employs a number of interrelated modifications to the two program streams to optimize their simultaneous playback. In summary, these corrections to the flow can be listed as follows:

● Space film sound track

Removing the spatial rendering from the kitchen according to the cooking cues rendered in the kitchen

Dynamically reducing loudness according to loudness of cooking cues rendered in a kitchen

Dynamically increasing loudness based on estimation of loudness of interfering blender noise from kitchen in living room

● Cooking cues

Dynamically increasing loudness based on combined estimation of loudness of both movie soundtrack and stirrer noise in kitchen

A second example use case of the disclosed multi-stream renderer involves the playback of concurrent spatial program streams (e.g., music) and the response of an intelligent voice assistant to some of the user's queries. For existing smart speakers where playback is typically constrained to mono or stereo playback on a single device, interaction with the voice assistant is typically composed of the following phases:

1) Playing music

2) The user speaks the voice assistant wake word

3) The smart speaker recognizes the wake word and significantly turns down (dodges) the music

4) The user speaks a command to the intelligent assistant (i.e., "play next song")

5) The intelligent speaker recognizes the command, confirms the command by playing a certain voice response (i.e. "good, play next song") mixed over the evaded music via the speaker, and then executes the command

6) The intelligent loudspeaker adjusts the music back to the original volume

Fig. 4A and 4B illustrate examples of multi-stream renderers that provide simultaneous playback of spatial music mixes and voice assistant responses. Some embodiments provide improvements to the event chain described above when playing spatial audio on multiple orchestrated intelligent speakers. In particular, the spatial mixing may be removed from one or more speakers selected as appropriate for relaying the response from the voice assistant. Creating this space for the voice assistant response means that the spatial mix may be turned down less than in the existing case listed above, or may not be turned down at all. Fig. 4A and 4B depict this scenario. In this example, the modified event chain may occur as follows:

1) The spatial music program stream (cloud line 335c in fig. 4A) is being played for the user on numerous programmed intelligent speakers.

2) The user 320c speaks the voice assistant wake word.

3) The one or more intelligent speakers (e.g., speaker 305d and/or speaker 305 f) recognize the wake word and use the associated record from the microphone associated with the one or more intelligent speakers to determine the location of user 320c or which speaker(s) user 320c is closest to.

4) The rendering of the spatial musical mix is moved away from the location determined in the previous step (cloud line 335d in fig. 4B) when the speech assistant is expected to be rendered near the location in response to the program stream.

5) The user speaks a command to the intelligent assistant (e.g., to the intelligent speaker running the intelligent assistant/virtual assistant software).

6) The intelligent speaker recognizes the command, synthesizes the corresponding response program stream, and renders the response near the user's location (cloud line 440 in fig. 4B).

7) When the voice assistant response is complete, the rendering of the spatial music program stream will switch back to its original state (cloud line 335c in fig. 4A).

In addition to optimizing the simultaneous playback of the spatial music mix and the voice assistant response, the shifting of the spatial music mix may also improve the ability of the set of speakers in step 5 to understand the listener. This is because the music has been removed from the speakers in the vicinity of the listener, thereby improving the speech to other proportions of the associated microphone.

Similar to the scenes described for the previous scene with spatial movie mixing and cooking cues, the current scene may be further optimized beyond the optimization provided by shifting the rendering of the spatial mixing according to the speech assistant response. Shifting the spatial mix may not be sufficient by itself to make the speech assistant response fully understandable to the user. A simple solution is to also reduce the spatial mixing by a fixed amount, albeit less than the amount required for the current state. Alternatively, the loudness of the speech assistant's response program stream may be dynamically increased in accordance with the loudness of the spatial music mix program stream to maintain audibility of the response. As an extension, if this increase in response flow becomes too large, the loudness of the spatial music mix may also be dynamically cut.

Fig. 5A, 5B, and 5C illustrate a third example use case of the disclosed multi-stream renderer. This example involves managing the simultaneous playback of a spatial music mix program stream and a comfort noise program stream while attempting to ensure that the baby remains asleep in the adjacent room, but is able to hear whether the baby is crying or not. Fig. 5A depicts the starting point where a spatial music mix (represented by cloud line 335 e) plays in an optimal manner for many people at a party across all speakers in living room 310 and kitchen 315. In fig. 5B, the infant 510 is now attempting to sleep in the adjacent bedroom 505 drawn to the bottom right. To help ensure this, the spatial music mix is dynamically removed from the bedroom to minimize leakage in the bedroom, as depicted by cloud line 335f, while still maintaining a reasonable experience for people at the party. At the same time, a second program stream (represented by cloudy line 540) containing soothing white noise is played from the speakers 305h of the baby room to mask any remaining leakage from music in the adjacent room. In some examples, to ensure complete masking, the loudness of the white noise stream may be dynamically modified based on an estimate of the loudness of spatial music leaking into the baby's room. The estimate may be generated from a rendered speaker feed of spatial music, a signal from a microphone of the baby crib, or a combination thereof. Also, if the loudness of the spatial music mix becomes too great, the loudness of the spatial music mix may be dynamically attenuated based on the loudness modified noise. This is similar to the loudness processing between spatial movie mixes and cooking cues for the first scene. Finally, a microphone in the baby's room (e.g., a microphone associated with speaker 305h, which in some embodiments may be a smart speaker) may be configured to record audio from the baby (eliminate sound that may be picked up from spatial music and white noise), and if crying is detected (through machine learning, via pattern matching algorithms, etc.), then the combination of these processed microphone signals may act as a third program stream that may be played back simultaneously near listener 320d in living room 310, which listener 320d may be a parent or other caretaker. Fig. 5C depicts the reproduction of this additional stream with a cloud line 550. In this case, the spatial music mix may be additionally removed from the speaker near the parent playing the baby's cry, as shown by the modified shape of the cloud line 335g relative to the shape of the cloud line 335f of fig. 5B, and the program stream of the baby's cry may modify loudness according to the spatial music stream such that the baby's cry remains audible to the listener 320 d. The interconnect modifications considered in this example to optimize simultaneous playback of three program streams can be summarized as follows:

● Spatial music mixing in living room

Removing the spatial rendering from the baby room to reduce propagation into the room

Dynamically reducing loudness according to loudness of white noise rendered in baby crib

Rendering on a speaker near the parent according to the crying of the baby, removing the spatial rendering from the parent

● White noise

Dynamically increasing loudness according to an estimate of loudness of a music stream penetrating into a baby's room

● Recording crying of infants

The loudness is dynamically increased based on an estimate of the loudness of the music mix at the location of the parent or other caretaker.

Examples of how some of the mentioned embodiments may be implemented are described next.

In fig. 1B, each of rendering blocks 1..n may be implemented as the same instance of any single stream renderer, such as the CMAP, FV, or hybrid renderer mentioned previously. Constructing a multi-stream renderer in this manner has some convenient and useful properties.

First, if rendering is done in this hierarchical arrangement and each single-stream renderer instance is configured to operate in the frequency domain/transform domain (e.g., QMF), then the mixing of streams may also occur in the frequency domain/transform domain and the inverse transform only needs to be run once for the M channels. This is a significant efficiency improvement compared to running the inverse nxm transform and mixing in the time domain.

Fig. 6 shows a frequency domain/transform domain example of the multi-stream renderer shown in fig. 1B. In this example, a quadrature mirror analysis filter bank (QMF) is applied to each of the program streams 1 to N before each program stream is received by a corresponding one of the rendering modules 1 to N. According to this example, the rendering modules 1 to N operate in the frequency domain. After the mixer 630a mixes the outputs of the rendering modules 1 to N, the inverse synthesis filter bank 635a converts the mixing into the time domain and provides the mixed speaker feed signals in the time domain to the loudspeakers 1 to M. In this example, the quadrature mirror filter bank, rendering modules 1 through N, mixer 630a, and inverse filter bank 635a are components of control system 110 c.

Fig. 7 shows a frequency domain/transform domain example of the multi-stream renderer shown in fig. 2A. As in fig. 6, a quadrature mirror filter bank (QMF) is applied to each of the program streams 1 to N before each program stream is received by a corresponding one of the rendering modules 1 to N. According to this example, the rendering modules 1 to N operate in the frequency domain. In this embodiment, the time domain microphone signals from the microphone system 120b are also provided to the quadrature mirror filter bank, so that the rendering modules 1 to N receive microphone signals in the frequency domain. After the mixer 630b mixes the outputs of the rendering modules 1 to N, the inverse filter bank 635b transforms the mix into the time domain and provides the mixed speaker feed signals in the time domain to the loudspeakers 1 to M. In this example, the quadrature mirror filter bank, rendering modules 1 through N, mixer 630b, and inverse filter bank 635b are components of control system 110 d.

Another benefit of the level approach in the frequency domain is to calculate the perceived loudness of each audio stream and use this information to dynamically modify one or more of the other audio streams. To illustrate this embodiment, consider the previously mentioned example described above with reference to fig. 3A and 3B. In this case, there are two audio streams (n=2), a spatial movie soundtrack and a cooking tip. There may also be ambient noise picked up by one or more of the K microphones generated by the blender in the kitchen.

After each audio stream s is rendered separately and each microphone i is captured and transformed to the frequency domain, a source excitation signal E may be calculated that serves as a time-varying estimate of the perceived loudness of each audio stream s or microphone signal i _s Or E is _i . In this example, the source excitation signals are b bands of time t across for c microphones,Transform coefficient X via audio stream _s Or the transform coefficient X of the microphone signal _i Calculated from the rendered stream or captured microphone and using a frequency dependent time constant lambda _b Smoothing:

E _s (b，t，c)＝λ _b E _s (b，t-1，c)+(1-λ _b )|X _s (b，t，c)| ² (20a)

E _i (b，t，c)＝λ _b (b)E _i (b，t-1，c)+(1-λ _b )|X _i (b，t，c)| ² (20b)

in this example, the original source excitation is an estimate of the perceived loudness of each stream at a particular location. For spatial flow, this position is in the middle of cloud line 335B in fig. 3B, while for cooking tip flow, this position is in the middle of cloud line 340. The location of the blender noise picked up by the microphone may be based, for example, on the particular location(s) of the microphone(s) closest to the source of the blender noise.

The original source excitation must be transformed to the listening position of the audio stream(s) to be modified by it to estimate the perceived extent of the original source excitation as noise at the listening position of each target audio stream. For example, if audio stream 1 is a movie soundtrack and audio stream 2 is a cooking tip, thenWill be the transformed (noise) excitation. For each frequency band b, according to each loudspeaker c, by applying audibility scaling coefficients from the source audio stream s to the target audio stream x or by applying a _xi The transformation is calculated from the microphone i applied to the target audio stream x. A is that _xs And A _xi May be determined by using a distance ratio or an estimate of the actual audibility (which may vary over time).

In the equation 13a of the present invention,representing the original noise excitation calculated for the source audio stream without reference to the microphone input. In equation 13b, +.>Representing the raw noise excitation calculated by the reference microphone input. According to this example, the original noise excitation is then cross-streamed 1 to N, microphones 1 to K and output channels 1 to M +.>Or->Summing to obtain a total noise estimate for target stream x：/>

According to some alternative embodiments, the term in equation 14 is omittedThe total noise estimate may be obtained without reference to the microphone input.

In this example, the total raw noise estimate is smoothed to avoid perceptible artifacts caused by too fast correction of the target stream. According to this embodiment, the smoothing is based on using a fast start and slow release concept, similar to an audio compressor. Smoothed noise estimation for target stream xIn this example calculated as:

once there is a complete noise estimate for stream xThe previously calculated source excitation signal E can be reused _x (b, t, c) to determine a time-varying gain set G _x (b, t, c) to be applied to the target audio stream x, thereby ensuring that the target audio stream remains audible in noise. These gains may be calculated using any of a variety of techniques.

In one embodiment, a loudness function L { ·, } may be applied to the excitation to model various nonlinearities in the perception of human loudness and calculate specific loudness signals that describe the time-varying distribution of perceived loudness over frequency. Applying L { ·, · } to the excitation for the noise estimate and the rendered audio stream x gives an estimate of the specific loudness for each signal:

L _x (b，t，c)＝L{E _x (b，t，c)} (25b)

in equation 17a, L _xn Represents an estimate of specific loudness for noise, and in equation 17b, L _x Representing an estimate of the specific loudness for the rendered audio stream x. These specific loudness signals represent the perceived loudness of the signal when heard alone. However, if the two signals are mixed, masking may occur. For example, if the noise signal is much louder than the stream x signal, the noise signal will mask the stream x signal, relative to what is heard separatelyThe perceived loudness of the signal reduces the perceived loudness of the signal. This phenomenon can be modeled with a partial loudness function PL { ·, }, which takes two inputs. The first input is the stimulus of the signal of interest and the second input is the stimulus of the competing (noise) signal. The function returns a partial specific loudness signal PL that represents the perceived loudness of the signal of interest in the presence of the competing signal. In the presence of a noise signal, the partial specific loudness of the stream x signal can then be calculated directly from the excitation signal across frequency band b, time t and loudspeaker c:

in order to maintain audibility of the audio stream x signal in the presence of noise, a gain G may be calculated _x (b, t, c) to be applied to the audio stream x to increase the loudness until the signal is audible over noise, as shown in equations 8a and 8 b. Alternatively, if the noise is from another audio stream s, two sets of gains may be calculated. In one such example, a first set of gains G _x (b, t, c) to be applied to the audio stream x to increase its loudness and a second set of gains G _s (b, t) will be applied to competing audio stream s to reduce its loudness such that the combination of gains ensures audibility of audio stream x, as shown in equations 9a and 9 b. In both sets of equations,representing the partial specific loudness of the source signal in the presence of noise after the compensation gain has been applied.

So that

Again, so that

In some examples, the original gain may be further smoothed across frequency using a smoothing function S { · } before being applied to the audio stream to again avoid audible artifacts.And->The final compensation gain representing the target audio stream x and the competing audio stream s:

in one embodiment, these gains may be applied directly to all rendered output channels of the audio stream. In another embodiment, these gains may instead be applied to the objects of the audio stream prior to being rendered, for example, using the method described in U.S. patent application publication No. 2019/0037333A1, the disclosure of which is incorporated herein by reference. The methods involve calculating panning coefficients for each audio object associated with each of a plurality of predefined channel footprints based on spatial metadata of the audio object. The audio signal may be converted into a sub-mix related to the predefined channel footprint based on the calculated panning coefficients and the audio objects. Each sub-mix may indicate a sum of components of the plurality of audio objects associated with one of the predefined channel footprints. The sub-mix gain may be generated by applying audio processing to each sub-mix, and the object gain applied to each audio object may be controlled. The object gain may be a function of the panning coefficients for each audio object and the sub-mix gain associated with each predefined channel footprint. Applying gain to objects has some advantages, especially when combined with other processing of the stream.

Fig. 8 illustrates an embodiment of a multi-stream rendering system with an audio stream loudness estimator. According to this example, the multi-stream rendering system of fig. 8 is also configured to implement the loudness processing as described in equations 12a-21b, as well as the compensation gain application within each single-stream renderer. In this example, a quadrature mirror filter bank (QMF) is applied to each of program streams 1 and 2 before each program stream is received by the corresponding one of rendering modules 1 and 2. In an alternative example, a quadrature mirror filter bank (QMF) may be applied to each of the program streams 1 to N before each program stream is received by a corresponding one of the rendering modules 1 to N. According to this example, rendering modules 1 and 2 operate in the frequency domain. In this embodiment, loudness estimation module 805a calculates a loudness estimate for program stream 1, e.g., as described above with reference to equations 12 a-17 b. Similarly, in this example, loudness estimation module 805b calculates a loudness estimate for program stream 2.

In this embodiment, the time domain microphone signal from microphone system 120c is also provided to the quadrature mirror filter bank such that the loudness estimation module 805c receives the microphone signal in the frequency domain. In this embodiment, the loudness estimation module 805c calculates a loudness estimate for the microphone signal, e.g., as described above with reference to equations 12 b-17 a. In this example, the loudness processing module 810 is configured to implement the loudness processing as described in equations 18-21 b, and the compensation gain application for each single stream rendering module. In this embodiment, loudness processing module 810 is configured to alter the audio signal of program stream 1 and the audio signal of program stream 2 in order to maintain the perceived loudness of the audio signal in the presence of one or more interfering signals. In some examples, the control system may determine that the microphone signal corresponds to ambient noise over which the program stream should be raised. However, in some examples, the control system may determine that the microphone signal corresponds to a wake word, a command, a child's cry, or other such audio that may need to be heard by the smart audio device and/or one or more listeners. In some such embodiments, loudness processing module 810 may be configured to alter the microphone signal to maintain the perceived loudness of the microphone signal in the presence of the interfering audio signal of program stream 1 and/or the interfering audio signal of program stream 2. Here, the loudness processing module 810 is configured to provide appropriate gains to the rendering modules 1 and 2.

After the mixer 630c mixes the outputs of the rendering modules 1 to N, the inverse filter bank 635c converts the mixing into the time domain and provides the mixed speaker feed signals in the time domain to the loudspeakers 1 to M. In this example, the quadrature mirror filter bank, rendering modules 1 through N, mixer 630c, and inverse filter bank 635c are components of control system 110 e.

Fig. 9A illustrates an example of a multi-stream rendering system configured for crossfading of multiple rendered streams. In some such embodiments, crossfading of multiple rendered streams is used to provide a fluent experience when dynamically altering a rendering configuration. One example is the aforementioned use case of simultaneous playback of a spatial program stream (e.g., music), where the intelligent speech assistant responds to some interrogation by the listener, as described above with reference to fig. 4A and 4B. In this case, it is useful to instantiate and simultaneously crossfade between additional single-stream renderers using an alternative spatial rendering configuration, as shown in fig. 9A.

In this example, QMF is applied to program stream 1 before the program stream is received by rendering modules 1a and 1 b. Similarly, QMF is applied to program stream 2 before the program stream is received by rendering modules 2a and 2 b. In some examples, the output of rendering module 1a may correspond to the desired rendering of program stream 1 prior to wake word detection, while the output of rendering module 1b may correspond to the desired rendering of program stream 1 after wake word detection. Similarly, the output of rendering module 2a may correspond to the desired rendering of program stream 2 prior to wake word detection, while the output of rendering module 2b may correspond to the desired rendering of program stream 2 after wake word detection. In this embodiment, the outputs of rendering modules 1a and 1b are provided to crossfade module 910a, and the outputs of rendering modules 2a and 2b are provided to crossfade module 910b. For example, the crossfade time may be in the range of hundreds of milliseconds to several seconds.

After mixer 630d mixes the outputs of crossfading modules 910a and 910b, inverse filter bank 635d converts the mix into the time domain and provides the mixed speaker feed signals in the time domain to loudspeakers 1 to M. In this example, the quadrature mirror filter bank, rendering module, crossfade module, mixer 630d, and inverse filter bank 635d are components of control system 110 f.

In some embodiments, the rendering configuration used in each of the single stream renderers 1a, 1b, 2a, and 2b may be calculated in advance. This is particularly convenient and efficient for use cases like intelligent voice assistants, as the spatial configuration is generally known a priori and is not dependent on other dynamic aspects of the system. In other embodiments, it may not be possible or desirable to pre-calculate the rendering configuration, in which case the complete configuration of each single-stream renderer must be calculated dynamically at system runtime.

One of the practical considerations of implementing dynamic cost-flexible rendering (according to some embodiments) is complexity. In some cases, solving the unique cost function for each frequency band for each audio object in real time may not be feasible given that the object position (the position of each audio object to be rendered that may be indicated by metadata) may change multiple times per second. An alternative approach to reducing complexity at the cost of memory is to use a look-up table that samples the three-dimensional space of all possible object locations. The sampling need not be the same in all dimensions. Fig. 9B is a diagram indicating points of speaker activation in an example embodiment. In this example, 15 points are sampled for the x-dimension and y-dimension, and 5 points are sampled for the z-dimension. According to this example, each point represents M speaker activations, one for each of the M speakers in the audio environment. For example, for centroid amplitude panning (CMAP) rendering, speaker activation may be a single gain per speaker, or for Flexible Virtualization (FV) or hybrid CMAP/FV rendering, may be a complex-valued vector across multiple N frequencies. Other embodiments may include more samples or fewer samples. For example, in some embodiments, the spatial sampling of speaker activation may not be uniform. Some embodiments may involve more or fewer speaker activation samples in the x, y plane than shown in fig. 9B. Some such implementations may determine speaker activation samples in only one x, y plane. According to this example, each point represents M speaker activations for a CMAP or FV solution. In some implementations, a set of speaker activations as shown in fig. 9B may be stored in a data structure, which may be referred to herein as a "table" (or "cartesian table", as indicated in fig. 9B).

The desired rendering position does not necessarily correspond to a position for which a speaker activation has been calculated. In operation, some form of interpolation may be implemented in order to determine the actual activation of each speaker. In some such examples, tri-linear interpolation between speaker activations of 8 points closest to the desired rendering location may be used. Fig. 10 is a diagram of tri-linear interpolation between points indicating speaker activation, according to one example. According to this example, a solid circle 1003 at or near the vertex of the rectangular prism shown in fig. 10 corresponds to the position of the 8 points closest to the desired rendering position for which speaker activation has been calculated. In this example, the desired rendering location is a point within the rectangular prism presented in fig. 10. In this example, the process of continuous linear interpolation includes interpolating each point in the top plane to determine a first interpolation point 1005a and a second interpolation point 1005b, interpolating each point in the bottom plane to determine a third interpolation point 1010a and a fourth interpolation point 1010b, interpolating the first interpolation point 1005a and the second interpolation point 1005b to determine a fifth interpolation point 1015 in the top plane, interpolating the third interpolation point 1010a and the fourth interpolation point 1010b to determine a sixth interpolation point 1020 in the bottom plane, and interpolating the fifth interpolation point 1015 and the sixth interpolation point 1020 to determine a seventh interpolation point 1025 between the top plane and the bottom plane. While tri-linear interpolation is an effective interpolation method, those skilled in the art will appreciate that tri-linear interpolation is but one possible interpolation method that may be used to implement aspects of the present disclosure, and other examples may include other interpolation methods. For example, some embodiments may involve more or less interpolation in the x, y plane than shown in fig. 9B. Some such implementations may involve interpolation in only one x, y plane. In some implementations, the speaker activation for the desired rendering location will simply be set to the speaker activation for the location closest to the desired rendering location for which the speaker activation has been calculated.

As described above with reference to fig. 9B, in some examples, each data structure or table may correspond to a particular rendering configuration. In some examples, each data structure or table may correspond to one version (e.g., a simplified version or a full version) of a particular rendering configuration. The complexity of the rendering configuration and the time required to calculate the corresponding speaker activation is related to the number of points in the table (speaker activation), which in a three-dimensional example can be calculated as the product of 5 dimensions: the number of x, y and z points, the number M of loudspeakers in the audio environment and the number N of frequency bands involved. In some examples, this correlation between table size and complexity results from the need to minimize potentially unique cost functions to populate each point in the table. In some examples, complexity and time to calculate the speaker activation table may also be related to fidelity of rendering.

Fig. 11A and 11B illustrate examples of providing a rendering configuration computing service. The high quality rendering configuration calculations may be too complex for some audio devices (e.g., intelligent audio devices 1105a, 1105B, and 1105c shown in fig. 11A and 11B).

In the example shown in FIG. 11A, intelligent audio devices 1105a-1105c are configured to send a request 1110a to rendering configuration computing service 1115a. In this example, rendering configuration computing service 1115a is a cloud-based service, which in some instances may be implemented by one or more servers of a data center. According to this example, the intelligent audio devices 1105a-1105c are configured to send a request 1110a from the audio environment 1100a to the rendering configuration computing service 1115a via the internet. Here, rendering configuration calculation service 1115a provides rendering configuration 1120a to smart audio devices 1105a-1105c via the internet in response to request 1110a.

According to the example shown in fig. 11B, the intelligent audio devices 1105a-1105c are configured to send a request 1110B to the rendering configuration computing service 1115B. In this example, rendering configuration computing service 1115b is implemented by another more capable device of audio environment 1100 b. Here, the audio device 1105d is a relatively more capable intelligent audio device than the intelligent audio devices 1105a-1105 c. In some examples, the local device may be a device referred to herein as an orchestration device or "smart home hub. According to this example, the intelligent audio devices 1105a-1105c are configured to send a request 1110b to the rendering configuration computing service 1115b via a local network, such as a Wi-Fi network. In this implementation, the rendering configuration calculation service 1115b provides the rendering configuration 1120b to the intelligent audio devices 1105a-1105c via the local network in response to the request 1110b.

In the example shown in fig. 11A and 11B, network communication adds additional delay to the process of computing the rendering configuration. In a dynamic, lined audio system, validating rendering configuration changes with minimal delay is a desirable property. This is especially true when the rendering configuration changes in response to real-time events, such as rendering configurations that perform spatial avoidance in response to wake word utterances. For highly dynamic applications, delay can be said to be more important than quality, as any perceived lag in the system can negatively impact the user experience before reacting to an event. Thus, in many use cases, it may be highly desirable to obtain an efficient rendering configuration with low latency (ideally in the range of hundreds of milliseconds). Some disclosed examples include methods for implementing rendering configuration changes with low latency. Some disclosed examples implement a method for computing a relatively lower complexity version of a rendering configuration to meet latency requirements and then progressively transition to a higher quality version of the rendering configuration as it becomes available.

Another desirable property of dynamic flexible renderers is the ability to smoothly transition to a new rendering configuration regardless of the current state of the system. For example, if a third configuration change is requested while the system is in the middle of a transition between a lower complexity version of the rendering configuration and a higher quality version of the rendering configuration, some disclosed embodiments are configured to begin the transition to the new rendering configuration without waiting for the first rendering configuration transition to complete. The present disclosure includes methods for smoothly processing such transitions.

The disclosed examples are generally applicable to any renderer that applies a set of speaker activations (e.g., a set of M speaker activations) to a set of audio signals to generate an output (e.g., M outputs), whether the renderer operates in the time domain or the frequency domain. By ensuring that the rendering configuration transition between the current set of speaker activations and the new set of speaker activations is smooth and continuous and arbitrarily interruptible, the timing of the transition between rendering configurations can be effectively decoupled from the time taken to calculate the speaker activations of the new rendering configuration. Some such implementations enable progressive transition to a new, lower quality/complexity rendering configuration, and then to a corresponding higher quality/complexity rendering configuration, which may be calculated asynchronously and/or by devices other than the device performing the rendering. Some implementations that can support smooth, continuous, and optionally interruptible transitions also have the desirable property of allowing a new set of target render-activations to be dynamically updated at any time, regardless of any previous transitions that may be ongoing.

Rendering configuration transitions

Various methods for implementing smooth, dynamic, and arbitrarily interruptible rendering configuration transitions (which may also be referred to herein as renderer configuration transitions or simply "transitions") are disclosed herein. Fig. 12A and 12B illustrate examples of rendering configuration transitions. In the example shown in fig. 12A, the audio system initially reproduces audio rendered according to rendering configuration 1. For example, one or more control systems in an audio environment may render input audio data as loudspeaker feed signals according to rendering configuration 1. For example, rendering configuration 1 may correspond to a data structure ("table") as described above with reference to fig. 9B.

According to this example, a rendering transition indication is received at time a. In some examples, the rendering transition indication may be received by an orchestration device, such as a smart speaker or "smart home hub," configured to coordinate or orchestrate the rendering of audio devices in the audio environment. In this example, the rendering transition indication is an indication that the current rendering configuration should transition to rendering configuration 2. The rendering transition indication may, for example, correspond to (e.g., may be in response to) an event detected in the audio system, such as a wake word utterance, an incoming call, an indication that a second content stream (e.g., a content stream corresponding to the "cooking tip" example described above with reference to fig. 3B) is to be rendered by one or more audio devices of the audio environment, and so on. Rendering configuration 2 may, for example, correspond to a rendering configuration that performs spatial avoidance in response to a wake word utterance, a rendering configuration for "pushing" audio from an audio environment location in response to user input, and so forth. According to this example, during a transition time t, the audio system transitions from rendering configuration 1 to rendering configuration 2. After the transition time t, the audio system has transitioned completely to rendering configuration 2.

Fig. 12B illustrates an example of an interrupted renderer configuration transition. In the example shown in fig. 12B, the audio system initially reproduces audio rendered according to rendering configuration 1. According to this example, a first rendering transition indication is received at time C. In this example, the first rendering transition indication is an indication that the current rendering configuration should transition to rendering configuration 2. According to this example, the audio system would need to transition time t1 (from time C to time E) to transition completely from rendering configuration 1 to rendering configuration 2. However, before the transition time t1 elapses and before the audio system has completely transitioned to rendering configuration 2, in this example, a second rendering transition indication is received at time D. Here, the second rendering transition indication is an indication that the current rendering configuration should be transitioned to rendering configuration 3. According to this example, the transition from rendering configuration 1 to rendering configuration 2 is interrupted upon receipt of the second rendering transition indication. In this example, the audio system needs to transition time t2 (from time D to time F) to transition completely to rendering configuration 3.

Various mechanisms for managing transitions between rendering configurations are disclosed herein. In the example described above with reference to fig. 9A, crossfading of multiple rendered streams is used to provide a smooth transition when dynamically changing rendering configurations.

FIG. 13 presents blocks corresponding to an alternative embodiment for managing transitions between rendering configurations. As with other disclosed embodiments, the number, arrangement, and types of elements presented in fig. 13 are by way of example only. According to this example, the renderer 1315, QMF 1310, and inverse QMF 1335 are implanted via a control system 110g, which is an example of the control system 110 described above with reference to fig. 1A. Here, control system 110g is shown to receive a program stream, which in this example includes program stream audio data 1305a and program stream metadata 1305b. In this example, only a single renderer instance is implemented for a given program stream, which is a more efficient implementation than an implementation that maintains two or more renderer instances.

According to this embodiment, renderer 1315 maintains activation data structures 1312a and 1312B, which in this example are look-up tables A and B. Each of data structures 1312a and 1312b corresponds to a rendering configuration or a version of a rendering configuration (e.g., a simplified version or a full version). Table a corresponds to the current rendering configuration and table B corresponds to the target rendering configuration. For example, data structures 1312a and 1312B may correspond to a set of speaker activations, such as the set of speaker activations shown in fig. 9B and described above. In the example shown in fig. 13, the target rendering configuration corresponds to a rendering transition indication that has been received by the control system 110 g. In other words, the rendering transition indication is an indication that the control system 110g transitions from the current rendering configuration (corresponding to table a) to the new rendering configuration. In some examples, table B may correspond to a simplified version of the new rendering configuration. In other examples, table B may correspond to a full version of the new rendering configuration.

In this embodiment, when the renderer 1315 is rendering an audio object, the renderer 1315 calculates two sets of speaker activations for the position of the audio object: here, one set of speaker activations for the location of the audio object is based on table a, while another set of speaker activations for the location of the audio object is based on table B. According to this example, the tri-linear interpolation modules 1313a and 1313b are configured to determine the actual activation of each speaker. In some such examples, the tri-linear interpolation modules 1313a and 1313b are configured to determine the actual activation of each speaker from the tri-linear interpolation process between speaker activations of the nearest 8 points, as described above with reference to fig. 10. Other embodiments may use other types of interpolation. In still other embodiments, the actual activation of each speaker may be determined from speaker activation at more or less than the last 8 points.

In this example, the module 1314 of the control system 110g is configured to determine an amplitude normalized interpolation between the two sets of speaker activations based at least in part on the crossfade time 1311. According to this example, the module 1314 of the control system 110g is configured to determine a single speaker activation table based on interpolation of table a received from the tri-linear interpolation module 1313a and based on interpolation of table B received from the tri-linear interpolation module 1313B. In some examples, the rendering transition indication also indicates a crossfade time 1311, which corresponds to the transition time described above with reference to fig. 12A and 12B. However, in other examples, control system 110g will determine crossfade time 1311, for example, by accessing a stored crossfade time. In some examples, crossfade time 1311 may be configured according to user input to a device that includes control system 110 g.

In this embodiment, the module 1316 of the control system 110g is configured to calculate the last set of speaker activations for each audio object in the frequency domain from the amplitude normalized interpolation. In this example, inverse filter bank 1335 converts the last set of speaker activations to the time domain and provides speaker feed signals in the time domain to loudspeakers 1 to M. In alternative implementations, the renderer 1315 may operate in the time domain.

FIG. 14 presents blocks corresponding to another embodiment for managing transitions between rendering configurations. As with other disclosed embodiments, the number, arrangement, and types of elements presented in fig. 14 are by way of example only. In this example, fig. 14 includes the blocks of fig. 13 that may function as described above. However, in this example, control system 110g is also configured to implement table combination module 1405.

In some examples, for example, as described above with reference to fig. 12B, the transition from the first rendering configuration to the second rendering configuration may be interrupted. In some such examples, the second rendering transition indication may be received during a time interval in which the system is transitioning from the first rendering configuration to the second rendering configuration. The second rendering transition indication may, for example, indicate a transition to a third rendering configuration.

According to some examples, table combination module 1405 may be configured to handle interrupts that render configuration transitions. In some such examples, if the rendering configuration transition is interrupted by another rendering transition indication, table combination module 1405 may be configured to combine the data structures 1312a and 1312B described above (the previous two look-up tables a and B) to create a new current look-up table a' (data structure 1412 a). In some such embodiments, control system 110g is configured to replace data structure 1312B with a new target lookup table B' (data structure 1412B). The new target lookup table B' may for example correspond to a third rendering configuration.

In some examples, block 1414 of table combining module 1405 may be configured to implement the combining operation by applying the same amplitude normalized interpolation mechanism at each point of the a-and B-tables, then implement the interpolation process: in this embodiment, the interpolation process involves a separate tri-linear interpolation process performed on the contents of tables a and B by blocks 1413a and 1413B, respectively. The interpolation process may be based at least in part on the time at which the previous rendering configuration transition was interrupted (as indicated by the previous crossfade interruption time 1411 of fig. 14).

According to some embodiments, the new current lookup table a' (data structure 1412 a) may optionally have a reduced dimension as compared to one or more previous tables (a and/or B). The dimension selection of a 'allows a tradeoff between complexity and quality of one-time computation during transition to a new rendering configuration associated with a new target lookup table B'. After target table B is replaced with new target table B', in some examples, table combination module 1405 may be configured to continue real-time interpolation between tables until the rendering configuration transition is complete. In some implementations, table combination module 1405 (e.g., in combination with other disclosed features) may be configured to process multiple rendering configuration transition interrupts without interruption.

Progressive computation and application of rendering configurations is a way to meet the low latency objectives required for good user experience without sacrificing the quality of flexible rendering configurations (e.g., flexible rendering configurations that optimize playback of a heterogeneous set of arbitrarily placed audio devices). Given a set of constraints associated with a particular rendering configuration, some disclosed methods involve parallel computing a lower complexity solution (e.g., a lower complexity version of the rendering configuration) and a higher complexity solution (e.g., a higher complexity version of the rendering configuration). For example, the dimensions of the lower complexity version of the rendering configuration may be selected such that the lower complexity version of the rendering configuration may be calculated with minimal delay. In some instances, higher complexity solutions may be computed in parallel. According to some such examples, rendering configuration transitions may begin, for example, using the methods described above, once a lower complexity version of the rendering configuration is available. Later, when a higher complexity version of the rendering configuration becomes available, the transition to the lower complexity version of the rendering configuration may be interrupted by the transition to the higher complexity version of the rendering configuration (if the rendering configuration transition has not been completed), e.g., as described above.

Table 1 below provides some example dimensions of a high complexity and low complexity rendering configuration lookup table to illustrate the order of magnitude of the differences in its computation.

Configuration type	x	y	z	Frequency (N)	Minimizing cost function
						Low complexity	3	3	2	1	18
High complexity	9	9	5	20	8,100

TABLE 1

Additional examples of frequency domain implementations

FIG. 15 presents blocks corresponding to a frequency domain renderer according to one example. As with other disclosed embodiments, the number, arrangement, and types of elements presented in fig. 15 are by way of example only. According to this example, the frequency domain renderer 1515, QMF 1510, and inverse QMF 1535 are implemented via a control system 110h, which is an example of the control system 110 described above with reference to fig. 1A. Here, control system 110h is shown receiving a program stream, which in this example includes program stream audio data 1505a and program stream metadata 1505b. The reader may assume that blocks 1510, 1512, 1513, 1516, and 1535 of fig. 15 provide the functionality of blocks 1310, 1312, 1313, 1316, and 1335 of fig. 13, unless otherwise specified.

In this example, the frequency domain renderer 1515 is configured to apply a set of speaker activations to the program stream audio data 1505a to generate M outputs, one for each of the loudspeakers 1 to M. In addition to speaker activation, in this example, the frequency domain renderer 1515 is configured to apply varying delays to the M outputs, e.g., to time align the sound from each loudspeaker to the listening position. For example, where M speaker activations are represented by time-domain filter coefficients, these delays may be implemented as any combination of sample delays and group delays. In some examples where rendering is implemented in the frequency domain, the M speaker activations may be represented by N frequency domain filter coefficients, and the delays may be represented by a combination of transform block delays (implemented via transform block delay line module 1518 in the example of fig. 15) and residual linear phase terms (sub-block delays) applied by the frequency domain filter coefficients (implemented via sub-block delay module 1520 in the example of fig. 15). According to some examples, the sub-block delay may be implemented as a simple complex multiplier for each frequency band of the filter bank, the complex value for each frequency band being selected according to a linear phase term having a slope equal to the negative value of the sub-block delay. To get more accurate results, the sub-block delay may be implemented with a higher precision multi-tap filter that operates across blocks of the filter bank.

FIG. 16 presents blocks corresponding to another embodiment for managing transitions between rendering configurations. As with other disclosed embodiments, the number, arrangement, and types of elements presented in fig. 16 are by way of example only. According to this example, the frequency domain renderer 1615, QMF 1610 and inverse QMF 1635 are implemented via a control system 110h, which is an example of the control system 110 described above with reference to fig. 1A. Here, the control system 110h is shown as receiving a program stream, which in this example includes program stream audio data 1605a and program stream metadata 1605b.

To support rendering configuration transitions that may be continuously and arbitrarily interrupted without discontinuities, the aforementioned method of interpolating speaker activation may be used, as described above with reference to fig. 13. In the example shown in fig. 16, as in the example described above with reference to fig. 13, there are 2 active rendering configurations, each with its own data structure (data structures 1612a and 1612B, also referred to as look-up tables a and B in fig. 16). Thus, unless otherwise indicated, the reader may assume that blocks 1610, 1612a, 1612b, 1613a, 1613b, 1616, and 1635 of fig. 16 provide the functionality of blocks 1310, 1312a, 1312b, 1313a, 1313b, 1316, and 1335 of fig. 13. Furthermore, the reader may assume that block 1618 of fig. 16 provides the functionality of transform block delay line module 1518 of fig. 15 unless otherwise specified.

For example, the module 1614 of the control system 110g may be configured to determine an amplitude normalized interpolation between two sets of speaker activations based at least in part on the crossfade time 1611. According to this example, the module 1614 of the control system 110 is configured to: a single speaker activation table is determined based on the interpolation of table a received from tri-linear interpolation module 1613a and based on the interpolation of table B received from tri-linear interpolation module 1613B. In this embodiment, the module 1616 of the control system 110g is configured to calculate the last set of speaker activations for each audio object in the frequency domain according to the speaker activation table determined by the module 1614. Here, the module 1616 outputs speaker feeds, one for each of the loudspeakers 1 to M, to the transform block delay line module 1618.

According to this example, the transform block delay line module 1618 applies a set of delay lines, one for each speaker feed. As in the example described above with reference to fig. 15, the delay may be represented by a combination of a transform block delay (implemented via transform block delay line 1618 in the example of fig. 16) and a residual linear phase term (sub-block delay, which may also be referred to herein as a sub-block delay filter) applied according to the frequency domain filter coefficients. In this example, the sub-block delay is a residual phase term that allows a delay that is not an exact multiple of the frequency domain transform block size.

According to this example, each active rendering configuration also has its own corresponding delay and read offset. Here, read offset a is for a rendering configuration (or rendering configuration version) corresponding to table a, and read offset B is for a rendering configuration (or rendering configuration version) corresponding to table B. According to some examples, a "read offset a" corresponds to a set of M read offsets associated with rendering configuration a, one for each of the M channels. In such an example, "read offset B" corresponds to a set of M read offsets associated with rendering configuration B. In such embodiments, the comparison of delays and the selection using unit powers and or unit amplitudes and may be made on a per channel basis. As described above, according to some examples, after reading from each delay line, the sub-block delays associated with rendering configurations corresponding to tables a and B are implemented using additional filtering stages. In this example, the sub-block delay of the active rendering configuration corresponding to lookup table a is implemented by sub-block delay module 1620a, while the sub-block delay of the active rendering configuration corresponding to lookup table B is implemented by sub-block delay module 1620B.

In this example, the crossfading module 1625 crossfades the sets of delayed speaker feeds (the outputs of the sub-block delay module 1620a and the sub-block delay module 1620 b) for each configuration to produce a single set of M output speaker feeds. In some examples, the crossfade module 1625 may be configured to apply a crossfade window to each rendering configuration. According to some embodiments, the crossfade module 1625 may be configured to select a crossfade window based at least in part on the delay line read offsets a and B.

Many possible symmetric crossfade window pairs may be used. Thus, the crossfade module 1625 may be configured to select a crossfade window pair differently, depending on the particular implementation. In some implementations, if delay line read offsets a and B are not identical to the extent that they can be determined from transform block size samples, then the crossfade module 1625 may be configured to select a crossfade window to have a unit power sum. In practice, if the total delay of rendering configurations a and B are within the transform block size samples of each other, then the read offsets a and B will appear the same. For example, if the transform block includes 64 samples, at a sampling rate of 48 kHz, the corresponding time interval is approximately 1.333 milliseconds. The difference exceeds a threshold amount. According to some examples, such conditions may be expressed as follows:

w(i)2+(1-w(i))2＝1(30)

In equation 30, i represents a block index related to time, but in the frequency domain. One example of w (i) satisfying the criterion of equation 30 is:

fig. 17 shows an example of a cross-fade window pair with a unit power sum. In this example, the pair of windows w (i) and (1-w (i)) presented in FIG. 17 is based on equation 31.

However, as shown in fig. 16, if the read offsets a and B are equal for a given output channel, this means that the total delay (combined block and sub-block delay) associated with a given channel of rendering configurations a and B is similar (e.g., within the number of samples of the transform block size). In this case, the crossfade window should have a unit amplitude sum (also referred to herein as a "unit sum") rather than a unit power sum, since the combined signals are likely to be highly correlated. Some examples of w (i) meeting units and criteria are as follows:

w(i)＝i (32)

fig. 18A and 18B present examples of cross-fade window pairs having unit amplitude sums. Fig. 18A shows an example of a cross-fade window pair with a unit sum according to equation 32. Fig. 18B shows an example of a cross-fade window pair with a unit sum according to equation 33.

The previous example of a crossfade window is simple. However, the crossfading module 1625 of fig. 16 may require a more generalized window design approach to be able to handle rendering configuration transitions that may be continuously and arbitrarily interrupted without discontinuities. Additional reads from the block delay line may be required during each interrupt of the rendering configuration transition. In the embodiment as shown in fig. 16, if the transition from rendering configuration a to rendering configuration B is interrupted by receiving a second rendering transition indication indicating a transition to rendering configuration C, then 3 block delay line reads and three read offsets will be involved and the crossfade module 1625 should implement a 3-part crossfade window.

Fig. 19 presents blocks corresponding to another embodiment for managing transitions between first through L-th set of speaker activations. In this example, L is an integer greater than two. As with other disclosed embodiments, the number, arrangement, and types of elements presented in fig. 19 are by way of example only. According to this example, the frequency domain renderers 1915, QMF 1910 and inverse QMF 1935 are implemented via control system 110i, which is an example of control system 110 described above with reference to fig. 1A. Here, control system 110i is shown receiving a program stream, which in this example includes program stream audio data 1903a and program stream metadata 1903b.

In some examples, L groups of speaker activations may correspond to L rendering configurations. However, as described elsewhere herein, some implementations may involve multiple sets of speaker activations corresponding to multiple versions of a single rendering configuration. For example, a first set of speaker activations may be used to render a simplified version of the configuration, while a second set of speaker activations may be used to render a complete version of the configuration. In some such examples, a single rendering transition indication may result in a first transition to a simplified version of the rendering configuration and a second transition to a full version of the rendering configuration.

Thus, in the example described with reference to fig. 19 (and other examples disclosed herein), there may or may not be the same numerical relationship between the transition indication, the speaker activation set, and the rendering configuration. If there is a reduced-to-full transition in response to the received rendering transition indication, two sets of speaker activations may be determined for the rendering transition indication, and there may be two transitions corresponding to the rendering transition indication. However, if there is no reduced-to-full transition in response to the rendering transition indication, a set of speaker activations may be determined for the rendering transition indication and only one transition may correspond to the rendering transition indication.

For simplicity, the following discussion will assume that the L groups of speaker activations of FIG. 19 correspond to L rendering configurations. To support rendering configuration transitions that may be continuously and arbitrarily interrupted without discontinuities, the methods of interpolating speaker activations described above (e.g., as described above with reference to fig. 13, 14, and 16) may be used. Thus, unless otherwise noted, the reader may assume that blocks 1910, 1912a, 1912b, 1913a, 1913b, 1916, and 1935 of fig. 19 provide the functionality of blocks 1610, 1612a, 1612b, 1613a, 1613b, 1616, and 1635 of fig. 16. Further, the reader may assume that block 1918 of fig. 19 provides the functionality of block 1618 of fig. 16 unless otherwise indicated.

In the example shown in fig. 16, there are 2 active rendering configurations, each with a corresponding data structure (data structures 1612a and 1612B, also referred to as look-up tables a and B in fig. 16), and each with a corresponding sub-block delay module (sub-block delay modules 1620a and 1620B). In the example shown in FIG. 19, there are L active rendering configurations, each with a corresponding sub-block delay module (sub-block delay modules 1920 a-1920L).

However, in the example shown in fig. 19, there are still only two current rendering configuration data structures: data structure 1912a corresponds to the current rendering configuration (which may be a transition rendering configuration), while data structure 1912b corresponds to the target rendering configuration (in this example, rendering configuration L). This efficiency is achieved by the scalability of the embodiment described above with reference to fig. 14: in the example shown in fig. 19, the control system 110i is also configured to implement a table combination module 1905, which may function as described above with reference to the table combination module 1405 of fig. 14. Thus, in this example, only two activation look-up tables are implemented, regardless of the number of interrupts and the number of activation rendering configurations.

According to the example shown in fig. 19, the crossfade module 1925 is configured to implement an L-section crossfade window. The design of the L-section crossfade window set should take into account which of the L block delay line reads have matching read offsets and which have different read offsets (as may be determined given the transform block size). The crossfade window set design should also consider the values of the (L-1) rendering configurations that were part of the previous crossfade window set given the interrupt (i_0) to ensure smooth transitions.

In some embodiments, the following method may be used to design the set of L-section crossfade windows.

1. Given a symmetrical pair of decay and rise window functions w (i) and 1-w (i)), the first L-1 windows (windows corresponding to the current rendering configuration (s)) may be set as decay functions w (i), while the last window L (window corresponding to the target rendering configuration) may be set as rise functions 1-w (i).

L-1 decay windows can be defined by the last interrupt (i ₀ ) The previous window value at that time. As described above, i represents a block index related to time, but in the frequency domain. Thus, i ₀ May be samples or sample blocks in the frequency domain corresponding to a point in time.

A set of l windows (l _1...L (i) A) may be grouped into K groups by summing windows with matching read offsets. K may range from 1 to L. If none of the read offsets match, there will be L groups. If all read offsets match, there will be 1 group.

K window groups (k) _1…K (i) The sum of the powers of each of these is calculated, for example, as follows:

5. the normalization function may then be calculated as the sum of one divided by the powers of K window groups:

6. the square root of the normalization function can be used to scale the original set of L windows, which ensures that the sum of the powers of the window groups will be unity:

in some embodiments, the more generalized formulas of powers and constraints shown in equation 36 replace the formulas of equation 31. Using this approach means that the choice of w (i) will affect the shape of the simple power-preserving crossfade window pair used in the direct, non-interrupted transition.

One way to select w (i) is to combine equations 32 and 33:

by selecting the appropriate value for ≡ (e.g., +=0.75) and using equation 37 as the base window function w (i) in the L-section crossfade design algorithm described in the previous paragraph, a power-preserving crossfade window pair similar to equation 31 and a linear crossfade window pair similar to equation 33 can be obtained.

Fig. 20A shows an example of a cross-fade window pair with unit powers sum according to equations 31 and 37, where ∈=0.75. Other examples may have other oc values, such as 0.60, 0.65, 0.70, 0.80, 0.85, etc.

Fig. 20B shows an example of a cross-fade window pair with a unit sum according to equations 33 and 37, where ∈=0.75. Other examples may have other oc values.

Fig. 21, 22, 23, 24, and 25 are diagrams presenting examples of crossfade windows without, with some, or all of the matched read offsets. In these examples, the crossfade window is designed using equations 34-37, with the y-axis representing w (i) and the x-axis representing i. As described above, i represents a block index related to time, but in the frequency domain.

Where it isIn some examples, where the sum is 0.ltoreq.i<i ₁ During a corresponding time interval, a first rendering configuration transition from rendering configuration a to rendering configuration B occurs. The first rendering configuration transition may, for example, have been responsive to the first rendering transition indication. According to these examples, at AND i ₁ At or near the corresponding time, the first rendering configuration transition is interrupted because a second rendering transition indication is received indicating a transition to rendering configuration C.

In these examples, at AND i ₁ ≤i<i ₂ A second rendering configuration transition to rendering configuration C occurs during a time interval corresponding to and not requiring completion of the transition to rendering configuration B.

According to these examples, at AND i ₂ At or near the corresponding time, the second rendering configuration transition is interrupted because a third rendering transition indication is received indicating a transition to rendering configuration D. In these examples, at AND i ₂ ≤i<1 and a third rendering configuration transition to rendering configuration D occurs during a time interval in which the transition to rendering configuration C does not need to be completed. In these examples, the third rendering configuration transition is completed at a time corresponding to i=1.

In the example presented in FIG. 21, none of the read offsets of rendering configurations A-D match. Thus, all cross-fade windows for rendering configuration transitions are selected to have unit powers sums.

According to the example presented in FIG. 22, all read offsets of rendering configurations A-D match. Thus, all cross-fade windows for rendering configuration transitions are selected to have a unit sum.

In the example presented in fig. 23, only the read offsets of rendering configurations a and B match. Thus, in this example, the cross-fade window pair for the first rendering configuration transition (from rendering configuration a to rendering configuration B) is selected to have a unit sum, while the cross-fade window pair for the second and third rendering configuration transitions is selected to have a unit power sum.

According to the example presented in fig. 24, the read offsets of rendering configurations a and B match, and the read offsets of rendering configurations C and D match. However, the read offsets of rendering configurations B and C do not match. Thus, in this example, the cross-fade window pairs for the first rendering configuration transition (from rendering configuration a to rendering configuration B) and the third rendering configuration transition (from rendering configuration C to rendering configuration D) are selected to have a unit sum, while the cross-fade window pairs for the second rendering configuration transition are selected to have a unit power sum.

In the example presented in fig. 25, the read offsets of rendering configurations a and C match, and the read offsets of rendering configurations B and D match. However, the read offsets of rendering configurations a and B do not match, and the read offsets of rendering configurations C and D do not match. Thus, in this example, the cross-fade window pair for the first rendering configuration transition (from rendering configuration a to rendering configuration B) is selected to have a unit power sum. Then, the subsequent transition to rendering configuration C is selected to have a unit power sum between the window applied to rendering configuration B and the sum of the windows applied to rendering configurations A and C. Finally, for the transition to rendering configuration D, the crossfade window is designed to have a unity power sum between the summing window applied to configurations A and C and the summing window applied to configurations B and D.

Using a frequency domain renderer as shown in fig. 19 may allow for any number of discontinuities, but may not scale well in some instances. Thus, in some implementations, the number of active rendering configurations that require delay line reading and crossfading may be limited in order to keep the complexity limited. Fig. 26, 27, 28, 29 and 30 illustrate the same case as fig. 21 to 25, but limit the 3 activation rendering configurations. In these examples, by eliminating AND i ₂ The contribution of rendering configuration B at the corresponding time, the number of active rendering configurations is limited to three. In some alternative embodiments, by eliminating the sum i ₂ The number of active rendering configurations may be limited to three by the contribution of rendering configuration a at the corresponding time. In some examples, this situation may be generalized by limiting the number of active rendering configurations to three, except at the point of AND i ₂ Two other associated crossfade windows having maximum magnitudes at corresponding timesIn addition to the rendering configurations (B, C and D in this example), the latest rendering configuration is always included. Note that in these examples, the power sum remains uniform in all cases, as the contribution of the remaining rendering configurations is normalized.

Fig. 31 is a flowchart outlining an example of a method. As with other methods described herein, the blocks of method 3100 need not be performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described. In some examples, one or more blocks of such methods may be performed concurrently. The blocks of method 3100 may be performed by one or more devices, which may be (or may include) a control system, such as control system 110, control system 110a, or control system 110B shown in fig. 1A, 1B, and 2A and described above, or by one of the other disclosed control system examples.

In this embodiment, block 3105 involves receiving, by the control system and via the interface system, audio data. Here, the audio data includes one or more audio signals and associated spatial data. In this example, the spatial data indicates an expected perceived spatial location corresponding to the audio signal. According to some examples (e.g., for e.g., dolby Atmos ^TM Such as audio object implementations), the spatial data may be or may include location metadata. However, in some examples, the first set of speaker activations may correspond to a channel-based audio format. In some such examples, the desired perceived spatial location may correspond to a channel of a channel-based audio format (e.g., may correspond to a left channel, a right channel, a center channel, etc.).

In this example, block 3110 relates to rendering, by a control system, audio data for reproduction via a set of loudspeakers of an environment, resulting in a first rendered audio signal. In this embodiment, rendering the audio data for reproduction involves determining a first relative activation of a set of loudspeakers in the environment according to a first rendering configuration. In this example, the first rendering configuration corresponds to a first set of speaker activations. According to some examples, a first set of speaker activations may be used for each of a corresponding plurality of locations in three-dimensional space. In this embodiment, block 3115 involves providing, via the interface system, the first rendered audio signal to at least some of the set of loudspeakers of the environment.

In this example, block 3120 involves receiving, by the control system and via the interface system, a first rendering transition indication. In this embodiment, the first rendering transition indication indicates a transition from the first rendering configuration to the second rendering configuration.

In this embodiment, block 3125 involves determining, by the control system, a second set of speaker activations corresponding to the reduced version of the second rendering configuration. According to this example, block 3130 involves performing, by the control system, a first transition from a first set of speaker activations to a second set of speaker activations.

In this embodiment, block 3135 involves determining, by the control system, a third set of speaker activations corresponding to the full version of the second rendering configuration. In some examples, block 3135 may be performed concurrently with block 3125 and/or block 3130. In this example, block 3140 involves performing, by the control system, a second transition to a third set of speaker activations without completing the first transition.

According to some examples, the method 3100 may involve receiving, by the control system and via the interface system, a second rendering transition indication. In some such examples, the second rendering transition indication indicates a transition to a third rendering configuration. In some examples, method 3100 may involve determining, by the control system, a fourth set of speaker activations corresponding to the reduced version of the third rendering configuration. In some examples, the method 3100 may involve performing, by the control system, a third transition from a third set of speaker activations to a fourth set of speaker activations. In some examples, method 3100 may involve determining, by the control system, a fifth set of speaker activations corresponding to the full version of the third rendering configuration, and performing, by the control system, a fourth transition to the fifth set of speaker activations without completing the first transition, the second transition, or the third transition.

In some examples, method 3100 may involve sequentially receiving, by the control system and via the interface system, second through (N) th rendering transition indications. Some such methods may involve determining, by the control system, a first set of speaker activations and a second set of speaker activations for each of the second through (N) th rendering transition indications. The first set of speaker activations may correspond to a simplified version of the rendering configuration, and the second set of speaker activations may correspond to a complete version of the rendering configuration for each of the second through (N) th rendering transition indications. In some examples, the method 3100 may involve sequentially performing, by the control system, a third through (2N-1) th transition from a fourth set of speaker activations to a (2N) th set of speaker activations. In some examples, the method 3100 may involve performing, by the control system, a (2N) th transition to (2n+1) th group speaker activation without completing any of the first through (2N) th transitions. In some implementations, a single renderer instance may render the audio data for reproduction.

However, not all rendering transition indications will necessarily relate to a reduced to complete transition in response to a received rendering transition indication. As in the above example, if there is a reduced-to-full rendering transition in response to the received rendering transition indication, two sets of speaker activations may be determined for the rendering transition indication, and there may be two transitions corresponding to the rendering transition indication. However, if there is no reduced-to-full transition in response to the rendering transition indication, a set of speaker activations may be determined for the rendering transition indication and only one transition may correspond to the rendering transition indication.

In some examples, the method 3100 may involve receiving, by the control system and via the interface system, a second rendering transition indication. The second rendering transition indication may indicate a transition to a third rendering configuration. In some such examples, method 3100 may involve determining, by the control system, a fourth set of speaker activations corresponding to the third rendering configuration, and performing, by the control system, a third transition to the fourth set of speaker activations without completing the first transition or the second transition.

According to some examples, the method 3100 may involve receiving, by the control system and via the interface system, a third rendering transition indication. The third rendering transition indication may indicate a transition to a fourth rendering configuration. In some examples, method 3100 may involve determining, by the control system, a fifth set of speaker activations corresponding to the fourth rendering configuration, and performing, by the control system, a fourth transition to the fifth set of speaker activations without completing the first transition, the second transition, or the third transition.

In some examples, method 3100 may involve sequentially receiving, by the control system and via the interface system, second through (N) -th rendering transition indications, and determining, by the control system, fourth through (n+2) -th group speaker activations corresponding to the second through (N) -th rendering transition indications. According to some examples, the method 3100 may involve sequentially performing, by the control system, a third through (N) -th transition from a fourth set of speaker activations to an (n+1) -th set of speaker activations, and performing, by the control system, an (n+1) -th transition to an (n+2) -th set of speaker activations without completing any of the first through (N) -th transitions.

According to some embodiments, the first set of speaker activations, the second set of speaker activations, and the third set of speaker activations are frequency dependent speaker activations. In some such examples, applying frequency-dependent speaker activation may involve applying a model of perceived spatial location in the first frequency band that produces binaural responses corresponding to the audio object locations at the listener's left and right ears. Alternatively or additionally, applying frequency dependent speaker activation may involve applying a perceived spatial location model in at least the second frequency band that places the perceived spatial location of the audio signal played from a set of loudspeakers at the centroid of the locations of the set of loudspeakers weighted by the associated activation gains of the loudspeakers.

In some examples, at least one of the first set of speaker activations, the second set of speaker activations, or the third set of speaker activations may be a result of an optimization cost that is a function of a model of perceived spatial location of the audio signal being played when played back over a set of loudspeakers in the environment. In some examples, the cost may be a function of a measure of the proximity of the intended perceived spatial location of the audio signal to the location of each loudspeaker in the set of loudspeakers. Alternatively or additionally, the cost may be a function of a metric based on one or more additional dynamically configurable functions of one or more of: proximity of the loudspeakers to one or more listeners; proximity of the loudspeaker to an attractive force location, wherein attractive force is a factor in favor of relatively higher loudspeaker activation nearer the attractive force location; proximity of the loudspeaker to a repulsive force location, wherein repulsive force is a factor in facilitating relatively lower loudspeaker activation nearer the repulsive force location; the ability of each loudspeaker to be relative to other loudspeakers in the environment; the synchronization of the loudspeaker with respect to the other loudspeakers; wake word performance; and/or echo canceller performance.

According to some implementations, rendering audio data for rendering may involve determining a single set of interpolation activations according to a rendering configuration and applying the single set of interpolation activations to produce a single set of rendered audio signals. In some such examples, a single set of rendered audio signals may be fed into a set of loudspeaker delay lines. The set of loudspeaker delay lines may comprise one loudspeaker delay line for each of the plurality of loudspeakers.

In some examples, rendering the audio data for reproduction may be performed in the frequency domain. Thus, in some examples, rendering audio data for reproduction may involve determining and implementing loudspeaker delays in the frequency domain. According to some such examples, determining and implementing speaker delays in the frequency domain may involve determining and implementing a combination of transform block delays and sub-block delays applied by frequency domain filter coefficients. In some examples, the sub-block delay may be a residual phase term that allows a delay that is not an exact multiple of the frequency domain transform block size. Thus, in some examples, rendering audio data for rendering may involve implementing sub-block delay filtering. In some implementations, rendering audio data for reproduction may involve implementing a set of block delay lines with separate read offsets.

In some examples, rendering the audio data for rendering may involve determining and applying interpolation speaker activation and cross-fade windows for each rendering configuration. According to some such examples, rendering audio data for reproduction may involve implementing a set of block delay lines with separate delay line read offsets. In some such examples, the crossfade window selection may be based at least in part on the delay line read offset. In some examples, where the delay line read offsets differ by more than a threshold amount, the crossfade window may be designed to have a unit power sum. According to some examples, the crossfade window may be designed to have a unit sum if the delay line read offsets are the same or differ by no more than a threshold amount.

As described above, in some implementations, a single renderer instance may render audio data for reproduction.

Fig. 32 is a flow chart summarizing an example of another method. As with other methods described herein, the blocks of method 3200 need not be performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described. In some examples, one or more blocks of such methods may be performed concurrently. The blocks of method 3200 may be performed by one or more devices, which may be (or may include) a control system, such as control system 110, control system 110a, or control system 110B shown in fig. 1A, 1B, and 2A and described above, or by one of the other disclosed examples of control systems.

In this embodiment, block 3205 involves receiving, by the control system and via the interface system, audio data. In this example, the audio data includes one or more audio signals and associated spatial data. Here, the spatial data indicates an expected perceived spatial position corresponding to the audio signal.

In this example, block 3210 involves rendering, by the control system, the audio data for reproduction via a set of loudspeakers of the environment to produce a first rendered audio signal. According to this example, rendering the audio data for reproduction involves determining a first relative activation of a set of loudspeakers in the environment according to a first rendering configuration.

In this embodiment, the first rendering configuration corresponds to a first set of speaker activations. In some examples, a first set of speaker activations may be used for each of a corresponding plurality of locations in three-dimensional space. In some examples, the spatial data may be or may include location metadata. However, in some examples, the first set of speaker activations may correspond to a channel-based audio format. In some such examples, the desired perceived spatial location includes a channel of a channel-based audio format. In the embodiment shown in fig. 32, block 3215 involves providing a first rendered audio signal to at least some of the set of loudspeakers of the environment via the interface system.

According to this example, block 3220 involves sequentially receiving, by the control system and via the interface system, first through (L-1) th rendering transition indications. In this example, each of the first through (L-1) th rendering transition indications indicates a transition from a current rendering configuration to a new rendering configuration.

In this embodiment, block 3225 involves determining, by the control system, second through (L) th group speaker activations corresponding to the first through (L-1) th rendering transition indications. According to this example, block 3230 involves sequentially performing, by the control system, first through (L-2) th transitions from a first set of speaker activations to an (L-1) th set of speaker activations. In this embodiment, block 3225 relates to performing a (L-1) th transition to (L) th group speaker activation by the control system without completing any of the first through (L-2) th transitions.

In some implementations, a single renderer instance may render the audio data for reproduction. In some examples, rendering the audio data for reproduction may be performed in the frequency domain. According to some examples, rendering audio data for rendering may involve determining a single set of interpolation activations according to a rendering configuration and applying the single set of interpolation activations to produce a single set of rendered audio signals.

In some such examples, a single set of rendered audio signals may be fed into a set of loudspeaker delay lines. The set of loudspeaker delay lines may for example comprise one loudspeaker delay line for each of a plurality of loudspeakers.

An example embodiment is described with reference to fig. 33. As with the other figures provided herein, the types and numbers of elements shown in fig. 33 are provided by way of example only. Other embodiments may include more, fewer, and/or different types and numbers of elements. Fig. 33 depicts a plan view of a listening environment, which in this example is a living space. According to this example, the environment 3300 includes a living room 3310 at the upper left, a kitchen 3315 at the lower center, and a bedroom 3322 at the lower right. The boxes and circles distributed across living space represent a set of loudspeakers 3305a-3305h, at least some of which may be intelligent loudspeakers in some embodiments, placed in convenient locations to the space, but not following any standard prescribed layout (arbitrarily placed). In some examples, microphones 3305a-3305h may be coordinated to implement one or more disclosed embodiments. In this example, environment 3300 includes cameras 3311a-3311e distributed throughout the environment. In some implementations, one or more intelligent audio devices in environment 3300 may also include one or more cameras. The one or more intelligent audio devices may be single-use audio devices or virtual assistants. In some such examples, one or more cameras of the optional sensor system 130 may reside in or on the television 3330, in a mobile phone, or in a smart speaker (e.g., one or more of microphones 3305b, 3305d, 3305e, or 3305 h). Although cameras 3311a-3311e are not shown in each depiction of environments 3300 presented in this disclosure, in some implementations, each environment 3300 may still include one or more cameras.

Aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., disk) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general purpose processor, digital signal processor, or microprocessor programmed with software or firmware and/or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general purpose processor may be or include a computer system including an input device, memory, and a processing subsystem programmed (and/or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

Some embodiments may be implemented as a configurable (e.g., programmable) Digital Signal Processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform the required processing on the audio signal(s), including the execution of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed systems (or elements thereof) may be implemented as a general-purpose processor (e.g., a Personal Computer (PC) or other computer system or microprocessor, which may include an input device and memory) programmed and/or otherwise configured with software or firmware to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive system are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more microphones and/or one or more microphones). A general purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and/or keyboard), memory, and a display device.

Another aspect of the disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code for controlling one or more devices to perform one or more examples of the disclosed methods or steps thereof.

Various features and aspects will be understood from the example embodiments (EEEs) enumerated below:

eee1. An audio processing method comprising:

receiving, by the control system and via the interface system, audio data comprising one or more audio signals and associated spatial data indicative of an intended perceived spatial location corresponding to the audio signals;

rendering, by the control system, the audio data for reproduction via a set of loudspeakers of an environment to produce a first rendered audio signal, wherein rendering the audio data for reproduction involves determining a first relative activation of the set of loudspeakers in the environment according to a first rendering configuration, the first rendering configuration corresponding to the first set of loudspeaker activations;

providing the first rendered audio signal to at least some loudspeakers of the set of loudspeakers of the environment via the interface system;

Sequentially receiving, by the control system and via the interface system, first through (L-1) th rendering transition indications, each of the first through (L-1) th rendering transition indications indicating a transition from a current rendering configuration to a new rendering configuration;

determining, by the control system, second through (L) th set of speaker activations corresponding to the first through (L-1) th rendering transition indications;

sequentially performing, by the control system, first-to- (L-2) transitions from the first set of speaker activations to the (L-1) th set of speaker activations; and

the (L-1) th transition to the (L) th set of speaker activations is performed by the control system without completing any of the first through (L-2) th transitions.

EEE2. The method of claim EEE1 wherein a single renderer instance renders the audio data for reproduction.

EEE3. The method of claim EEE1 or claim EEE2 wherein rendering the audio data for reproduction comprises determining a single set of interpolation activations according to the rendering configuration and applying the single set of interpolation activations to produce a single set of rendered audio signals.

The method of EEE4 wherein the single set of rendered audio signals is fed into a set of loudspeaker delay lines, the set of loudspeaker delay lines including one loudspeaker delay line for each of a plurality of loudspeakers.

EEE5 the method of any one of claims EEE 1-EEE 4 wherein the rendering the audio data for reproduction is performed in the frequency domain.

The method of any one of EEEs 1-5, wherein the first set of speaker activations is for each of a corresponding plurality of locations in three-dimensional space.

EEE7. The method of any one of EEEs 1-6, wherein the spatial data comprises location metadata.

EEE8. the method of any one of claims EEE 1-EEE 5 wherein the first set of speaker activations corresponds to a channel-based audio format.

EEE9. the method of claim EEE8 wherein the desired perceived spatial location comprises a channel of the channel-based audio format.

EEE10. An apparatus configured to perform the method of any one of the EEEs 1 to 9.

EEE11. A system configured to perform the method of any one of the EEEs 1 to 9.

EEE12. One or more non-transitory media having stored thereon software comprising instructions for controlling one or more devices to perform the method of any of EEEs 1-9.

Although specific embodiments and applications have been described herein, it will be apparent to those of ordinary skill in the art that many more modifications than mentioned herein are possible without departing from the scope of the invention described and claimed herein. It is to be understood that while certain forms have been illustrated and described, the scope of the disclosure is not limited to the specific embodiments described and illustrated or the specific methods described.

Claims

1. An audio processing method, comprising:

rendering, by the control system, the audio data for reproduction via a set of loudspeakers of an environment to generate a first rendered audio signal, wherein rendering the audio data for reproduction involves determining a first relative activation of the set of loudspeakers in the environment according to a first rendering configuration, the first rendering configuration corresponding to a first set of loudspeaker activations;

Receiving, by the control system and via the interface system, a first rendering transition indication indicating a transition from the first rendering configuration to a second rendering configuration;

determining, by the control system, a second set of speaker activations corresponding to a reduced version of the second rendering configuration;

performing, by the control system, a first transition from the first set of speaker activations to the second set of speaker activations;

determining, by the control system, a third set of speaker activations corresponding to a full version of the second rendering configuration; and

a second transition to the third set of speaker activations is performed by the control system without completing the first transition.

2. The method of claim 1, wherein the first set of speaker activations, the second set of speaker activations, and the third set of speaker activations are frequency dependent speaker activations.

3. The method of claim 2, wherein the frequency-dependent speaker activation involves applying a model of perceived spatial location in at least a first frequency band, the model producing binaural responses corresponding to audio object locations at the listener's left and right ears.

4. A method as claimed in claim 3, wherein the frequency dependent loudspeaker activation involves applying a perceptual spatial location model in at least a second frequency band, the model placing the perceived spatial location of an audio signal played from a set of loudspeakers at the centroid of the locations of the set of loudspeakers weighted by their associated activation gains.

5. The method of any of claims 1-4, wherein at least one of the first set of speaker activations, the second set of speaker activations, or the third set of speaker activations is a result of optimizing a cost as a function of:

a model of perceived spatial location of audio signals played when played back over the set of loudspeakers in the environment;

a measure of the proximity of the intended perceived spatial location of the audio signal to the location of each loudspeaker in the set of loudspeakers; and

one or more additional dynamically configurable functions, wherein the one or more additional dynamically configurable functions are based on one or more of: proximity of the loudspeakers to one or more listeners; proximity of the loudspeaker to an attractive force location, wherein attractive force is a factor in favor of relatively higher loudspeaker activation closer to the attractive force location; proximity of the loudspeaker to a repulsive force location, wherein repulsive force is a factor facilitating relatively lower loudspeaker activation closer to said repulsive force location; the ability of each loudspeaker to be relative to other loudspeakers in the environment; synchronization of the loudspeakers with respect to other loudspeakers; wake word performance; or echo canceller performance.

6. The method of any one of claims 1 to 5, further comprising:

receiving, by the control system and via the interface system, a second rendering transition indication indicating a transition to a third rendering configuration;

determining, by the control system, a fourth set of speaker activations corresponding to the third rendering configuration; and

a third transition to the fourth set of speaker activations is performed by the control system without completing the first transition or the second transition.

7. The method of claim 6, further comprising:

receiving, by the control system and via the interface system, a third rendering transition indication indicating a transition to a fourth rendering configuration;

determining, by the control system, a fifth set of speaker activations corresponding to the fourth rendering configuration; and

a fourth transition to the fifth set of speaker activations is performed by the control system without completing the first transition, the second transition, or the third transition.

8. The method of any one of claims 1 to 5, further comprising:

sequentially receiving, by the control system and via the interface system, second through (N) th rendering transition indications;

Determining, by the control system, fourth through (n+2) -th group speaker activations corresponding to the second through (N) -th rendering transition indications;

sequentially performing, by the control system, third through (N) th transitions from the fourth set of speaker activations to an (n+1) th set of speaker activations; and

an (n+1) th transition to the (n+2) th set of speaker activations is performed by the control system without completing any of the first through (N) th transitions.

9. The method of any one of claims 1 to 5, further comprising:

determining, by the control system, a fourth set of speaker activations corresponding to a reduced version of the third rendering configuration;

performing, by the control system, a third transition from the third set of speaker activations to the fourth set of speaker activations;

determining, by the control system, a fifth set of speaker activations corresponding to a full version of the third rendering configuration; and

10. The method of any one of claims 1 to 5, further comprising:

determining, by the control system, a first set of speaker activations and a second set of speaker activations for each of the second through (N) th rendering transition indications, the first set of speaker activations corresponding to a simplified version of a rendering configuration and the second set of speaker activations corresponding to a full version of the rendering configuration for each of the second through (N) th rendering transition indications;

third to (2N-1) th transitions from fourth to (2N) th set of speaker activations are sequentially performed by the control system; and

the (2N) th transition to the (2n+1) th set of speaker activations is performed by the control system without any of the first through (2N) th transitions being completed.

11. The method of any of claims 1 to 10, wherein a single renderer instance renders the audio data for reproduction.

12. The method of any of claims 1 to 11, wherein rendering the audio data for reproduction comprises determining a single set of interpolation activations according to the rendering configuration and applying the single set of interpolation activations to produce a single set of rendered audio signals.

13. The method of claim 12, wherein the single set of rendered audio signals is fed into a set of loudspeaker delay lines, the set of loudspeaker delay lines including one loudspeaker delay line for each of a plurality of loudspeakers.

14. The method of any of claims 1 to 13, wherein rendering the audio data for reproduction further involves determining and implementing loudspeaker delays in the frequency domain.

15. The method of claim 14, wherein determining and implementing speaker delays in the frequency domain involves determining and implementing a combination of transform block delays and sub-block delays applied by frequency domain filter coefficients, the sub-block delays being residual phase terms that allow delays that are not exact multiples of a frequency domain transform block size.

16. The method of claim 14 or claim 15, wherein rendering the audio data for reproduction further involves implementing a set of transform block delay lines with separate read offsets.

17. The method of any of claims 14 to 16, wherein rendering the audio data for reproduction further involves implementing sub-block delay filtering.

18. The method of claim 17, wherein implementing the sub-block delay filtering involves implementing a multi-tap filter across the block of frequency domain transforms.

19. The method of any of claims 1-18, wherein the rendering the audio data for reproduction includes determining and applying an interpolated speaker activation and cross-fade window for each rendering configuration.

20. The method of claim 19, wherein rendering the audio data for reproduction further involves implementing a set of transform block delay lines with individual delay line read offsets, wherein a cross-fade window selection is based at least in part on the delay line read offsets, and wherein the cross-fade window is designed to have a unit power sum if the delay line read offsets are not identical.

21. The method of any of claims 1 to 20, wherein the rendering the audio data for reproduction is performed in the frequency domain.

22. The method of any of claims 1-21, wherein the first set of speaker activations are for each of a corresponding plurality of locations in three-dimensional space.

23. The method of any of claims 1 to 22, wherein the spatial data comprises location metadata.

24. The method of any of claims 1 to 21, wherein the first set of speaker activations corresponds to a channel-based audio format.

25. The method of claim 24, wherein the intended perceived spatial location comprises a channel of the channel-based audio format.

26. An apparatus configured to perform the method of any one of claims 1 to 25.

27. A system configured to perform the method of any one of claims 1 to 25.

28. One or more non-transitory media having software stored thereon, the software comprising instructions for controlling one or more devices to perform the method of any of claims 1-25.