WO2024252597A1 - Directivity control device for microphone array, directivity control method, and program - Google Patents

Abstract

One aspect of the present invention is a directivity control device that controls the directivity of an acoustic signal input from a microphone array having a plurality of microphones. This directivity control device comprises: a directivity characteristic determination processing unit that determines a desired directivity characteristic from a desired sound collection direction; a directivity characteristic correction processing unit that corrects the desired directivity characteristic by using spherical harmonic function expansion; a directivity control filter design processing unit that designs a directivity control filter corresponding to each of the microphones on the basis of the corrected desired directivity characteristic; and a convolution operation processing unit that performs convolution operations of the input acoustic signal and the directivity control filter, and generates an output acoustic signal of which the directivity is controlled to the desired directivity characteristic.

Description

Microphone array directivity control device, directivity control method, and program

One aspect of the present invention relates to a microphone array directivity control device, a directivity control method, and a program.

In recent years, the use of playback methods that use speakers and headphones to control and reproduce the direction and volume of sound propagation has become widespread. Research is also being conducted into sound reproduction technology for spaces such as public viewing venues and homes, and in particular the reproduction of the sound field of a desired space by using multiple speakers arranged to surround the listener. In this way, to reproduce the sound field of a certain space, it is necessary to obtain acoustic information about the space to be reproduced.

Generally, to obtain spatial information about an arbitrary sound field, it is necessary to record the sound field three-dimensionally, so there is a need to efficiently record the three-dimensional sound field using recording equipment consisting of microphones arranged three-dimensionally. In other words, there is a need for technology that can arbitrarily give a desired directional characteristic to a microphone array.

Sato, Koya, and Yoichi Haneda. "Directivity control of a finite cylindrical loudspeaker array based on circular harmonics and longitudinal multipole expression." Acoustical Science and Technology 40.2 (2019): 93-104.

Non-Patent Document 1 proposes a technology for arbitrarily setting desired directional characteristics for a microphone array. However, if a directional characteristic that is difficult to reproduce is set as the desired directional characteristic, a beamformer is designed in which the generated directional characteristic has a directional characteristic in a direction other than the target direction, called a side lobe. As a result, it is not possible to pick up sound in the target direction with high directivity.

The objective of this invention is to provide a technology that generates an output acoustic signal that is highly directional and controlled from an input acoustic signal input from a microphone array, so that the sound in the desired target direction is highly directional.

One aspect of the present invention is a directivity control device that controls the directivity of an input audio signal input from a microphone array having multiple microphones. The directivity control device has a directional characteristic determination processing unit that determines a desired directional characteristic from a desired sound pickup direction, a directional characteristic correction processing unit that corrects the desired directional characteristic using spherical harmonic function expansion, a directional control filter design processing unit that designs a directional control filter corresponding to each microphone based on the corrected desired directional characteristic, and a convolution calculation processing unit that performs a convolution calculation of the audio signal and the directional control filter to generate an output audio signal whose directivity is controlled to the desired directional characteristic.

One aspect of the present invention is a directivity control method for controlling the directivity of an input audio signal input from a microphone array having multiple microphones. The directivity control method includes the steps of determining a desired directional characteristic from a desired sound pickup direction, correcting the desired directional characteristic using spherical harmonic function expansion, designing a directional control filter corresponding to each microphone based on the corrected desired directional characteristic, and performing a convolution operation of the audio signal and the directional control filter to generate an output audio signal whose directivity is controlled to the desired directional characteristic.

One aspect of the present invention is a directivity control program that controls the directivity of an input sound signal input from a microphone array having multiple microphones. The directivity control program causes a computer having a processor and a storage device to execute at least some of the functions of the processing performed by each processing unit of the above-mentioned directivity control device.

According to one aspect of the present invention, a technology can be provided that generates an output audio signal in which sound from a desired target direction is controlled with high directivity from an input audio signal input from a microphone array.

FIG. 1 is a diagram showing an example of the configuration of a sound collection system including a directivity control device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a hardware configuration of the directivity control device shown in FIG. FIG. 3 is a block diagram showing an example of a software configuration of the directivity control device shown in FIG. FIG. 4 is a flowchart showing an example of a processing procedure and processing contents performed by the directivity control device shown in FIG. FIG. 5 is a diagram showing an example of an observation system for explaining a design method of a filter for directivity control using the least squares method. FIG. 6 is a diagram showing a polar coordinate system. FIG. 7 is a diagram showing spherical harmonics up to the third order.

Below, an embodiment of the present invention will be described with reference to the drawings.

[One embodiment]
(Configuration example)
(1) System FIG. 1 is a diagram showing the overall configuration of a sound collection system including a directivity control device according to an embodiment of the present invention.

The sound collection system according to one embodiment includes a microphone array MC, a directivity control device 1, and an input/output device 2 connected to the directivity control device 1.

The microphone array MC has a plurality of microphones MC ₁ to MC _M. For example, the microphones MC ₁ to MC _M are arranged at equal intervals on the same circumference. The microphones MC ₁ to MC _M are arranged so that their respective sound collection directions are oriented radially.

The microphones MC ₁ to MC _M of the microphone array MC convert sounds input from their respective sound collection directions into electrical signals, and output them to the directivity control device 1 as input acoustic signals IS ₁ to IS _M.

The input/output device 2 is, for example, a personal computer or a mobile terminal such as a smartphone used by a system administrator or user, and is used to input information on the desired directional characteristics to the directivity control device 1. The input/output device 2 may be, for example, a television remote control or an operation panel for a car navigation device, and the connection means between the input/output device 2 and the directivity control device 1 may be a wireless interface such as Bluetooth (registered trademark), WiFi (registered trademark), or a mobile communication network, other than a wired cable.

The directivity control device 1 is a device that generates an output audio signal OS in which sound in a desired target direction is controlled with high directivity by performing a convolution operation between an input audio signal IS input from each microphone _MC1 to _MCM of a microphone array MC and a directivity control filter corresponding to each microphone.

(2) Directivity control device 1
2 and 3 are block diagrams showing the hardware and software configurations, respectively, of the directivity control device 1. As shown in FIG.

The directional control device 1 includes a control unit 10 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 20 and a data storage unit 30, an input/output interface (hereinafter, the interface will be referred to as an I/F) 40 to which an input/output device 2 is connected, and an acoustic signal I/F 50 are connected to the control unit 10 via a bus 60.

The acoustic signal I/F 50 is used to receive input acoustic signals IS (IS ₁ to IS _M ) output by each microphone MC ₁ to MC _M of the microphone array MC, and to output an output acoustic signal OS in which sound in a desired target direction is controlled with high directivity.

The program storage unit 20 is configured, for example, by combining a non-volatile memory such as a HDD (Hard Disk Drive) or SSD (Solid State Drive) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a ROM (Read Only Memory), and stores middleware such as an OS (Operating System) as well as programs necessary to execute various control processes according to one embodiment.

The data storage unit 30 is, for example, a combination of a non-volatile memory such as an HDD or SSD, which can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium, and includes a directional characteristic storage unit 31 and a filter storage unit 32 as the main storage units required to implement one embodiment.

The directional characteristic memory unit 31 is used to store the desired directional characteristic determined by the control unit 10. The filter memory unit 32 is used to store the directional control filter calculated by the control unit 10.

The control unit 10 includes, as processing functions necessary to implement one embodiment, a directional characteristic determination processing unit 11, a directional characteristic correction processing unit 12, a directivity control filter design processing unit 13, and a convolution calculation processing unit 14. All of these processing units 11 to 14 are realized by causing the hardware processor of the control unit 10 to execute a program stored in the program storage unit 20.

The directional characteristic determination processing unit 11 receives the sound pickup direction signal DS, which indicates the desired sound pickup direction, input from the input/output device 2 via the input/output I/F 40, determines the directional characteristic of the microphone array MC, and stores the determined desired directional characteristic in the directional characteristic storage unit 31.

The directional characteristic correction processing unit 12 performs a process to correct the desired directional characteristic determined by the directional characteristic determination processing unit 11 using spherical harmonic function expansion by t-design.

The directivity control filter design processing unit 13 performs processing to design a directivity control filter based on the desired directivity corrected by the directivity correction processing unit 12, and stores the directivity control filter calculated by this processing in the filter storage unit 32.

The convolution calculation processing unit 14 performs a convolution calculation between the input acoustic signal IS (IS ₁ to IS _M ) input via the acoustic signal I/F 50 and the directivity control filter stored in the filter memory unit 32, and generates an output acoustic signal OS in which sound in the desired target direction is controlled with high directivity.

(Example of operation)
In the following, the basic technologies of the directivity control executed by the directivity control device 1 will be described, firstly, the directivity technology, the filter gain suppression by the penalty term, and the design of the directivity with suppressed side lobes.

(1) Directivity Technology In directional technology, control points are placed around a microphone array in which multiple microphones are lined up, and a directional control filter is designed based on the characteristics of transmission from the microphones to the control points. For example, multiple microphones are placed at equal intervals on the same circumference. The directional control filter is a filter that controls the direction in which each microphone picks up sound or the direction in which it does not pick up sound.

A typical method for designing a directional control filter is to use the least squares method. Figure 5 shows an example of an observation system to explain the design method for a directional characteristic filter using the least squares method.

Here, M (1≦m≦M) indicates the number of microphones, and Q (1≦q≦Q) indicates the number of control points. In addition, ω indicates the angular frequency (ω=2πf), and f indicates the frequency. W _m (ω) is a filter corresponding to the m-th microphone, and G _qm (ω) is a transfer function from the m-th microphone MC _m to the q-th control point CP _q . G(ω) is a transfer function matrix of Q rows and M columns that stores the transfer functions G _qm (ω) from each microphone MC ₁ to MC _M to each control point CP ₁ to CP _Q. The transfer function G _qm (ω) is given by the following equation.

Here, the superscript H denotes complex conjugate transpose. By solving the problem of minimizing the objective function J expressed in equation (3) for w(ω), the following directional control filter is obtained.

(2) Filter Gain Suppression by Penalty Term When designing a filter for controlling directivity, the filter is calculated in such a way that a filter gain that affects the output sound source from the filter is included.

Here, w _m (ω) represents the filter coefficient corresponding to the m-th microphone. The superscript * represents the complex conjugate. If the filter gain is large, the input signal also becomes larger proportionally, and the noise mixed in the picked-up signal is amplified, causing a decrease in performance. In response to this, there is a method of deriving a filter that controls directivity by using a penalty term, which will be described later, for the objective function that derives the filter. In this case, the sum of squares of the filter coefficients is used as the penalty term to suppress the filter gain.

Let's consider a directional control filter using the least squares method as an example, when a penalty term is used. If the penalty term is used in the objective function of equation (3), it becomes as follows.

Here, β(ω) is a regularization parameter that controls the relative weight between the loss term ||e|| ² and the penalty term ||W(ω)|| ^2. As with equation (4), the following directivity control filter W(ω) is obtained by solving the minimization problem for w(ω).

Here, I is a unit matrix with M rows and M columns, and M is the number of microphones.

(3) Design of directional characteristics with suppressed side lobes One of the issues in directional control using a microphone array or a speaker array is side lobes, which are directional characteristics in directions other than the desired direction. One of the factors that cause side lobes is setting a desired characteristic that is difficult to reproduce. A representative method of directional control is a method of minimizing the error between the desired directional characteristic and the reproduced directional characteristic. As the desired directional characteristic, the target direction is often set to 1 and other directions to 0, but since this is a directional characteristic that is difficult to reproduce and this method aims to minimize the overall directional characteristic, it is difficult to control the shape of the side lobe.

In the method based on Non-Patent Document 1, when performing directivity control using the least squares method, the desired directional characteristic is replaced with a window function to suppress side lobes, which are directional characteristics in directions other than the target direction. Because a window function is set as the desired directional characteristic, the beam width of the directional characteristic in the target direction becomes wider, but it has been confirmed that the side lobes can be suppressed.

(4) Spherical Harmonic Functions Consider the polar coordinate system defined in Fig. 6. In this case, the sound pressure S(r, ω) observed at an arbitrary point on the spherical surface is expressed by the following equation.

Here, Y _n,m (θ,φ) is a spherical harmonic function, and A _n,m (ω) is its expansion coefficient, which is defined by the following equation.

Here, P _n,m (·) is the associated Legendre function, and equation (12) is called the spherical harmonic expansion. Also, Y _n,m (θ,φ) ^* is the complex conjugate of the spherical harmonic function. When the maximum order is N, the order n and the order m are 0≦n≦N and -n≦m≦n, respectively. Also, the larger the order, the more complex the shape of the sound field can be expressed. Figure 7 shows spherical harmonic functions up to N=3.

Generally, the spherical harmonic expansion coefficients can be derived using control points spaced equally (θ, φ) using the following formula, which is an approximation of formula (10).

Here, _αq is a correction coefficient. As shown in the formula (11), when expressing the integral as a discrete sum, approximation is introduced, so accurate expansion coefficients cannot be obtained.

However, a method is known for deriving an optimal control point arrangement to match the continuous integral and the discrete sum. This method is called t-design and is described in the literature "Zotter, Franz, and Matthias Frank. Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforcement, and virtual reality. Springer Nature, 2019." In t-design, it is possible to accurately derive the spherical harmonic expansion coefficients A _n,m (ω) instead of approximations such as those in formula (11).

(5) Directivity Control in an embodiment of the present invention The following describes the sound collection directivity control executed by the directivity control device 1. Fig. 4 is a flowchart showing the processing procedure and processing contents of the directivity control of the microphone array MC executed by the directivity control device 1. Here, an example is described in which a filter is designed to realize desired directivity reproduction by solving the approximate gradient method with the sum of squares of the error between the desired directional characteristic corrected using the spherical harmonic function expansion of t-design and the reproduced directional characteristic as the target function, and an indicator function such that the filter coefficient falls within an arbitrary range.

(5-1) Determination of Desired Directivity A system administrator or a user inputs a desired sound collection direction to be reproduced in the microphone array MC to the input/output device 2. The input/output device 2 generates a sound collection direction signal DS representing the desired sound collection direction and sends it to the directivity control device 1.

The control unit 10 of the directivity control device 1 receives the sound collection direction signal DS from the input/output device 2 via the input/output I/F 40, and determines a desired directional characteristic from the sound collection direction signal DS under the control of the directional characteristic determination processing unit 11 in step S11. The desired directional characteristic corresponds to a signal observed at a control point CP _q arranged by t-design. The control unit 10 of the directivity control device 1 stores the desired directional characteristic determined by the directional characteristic determination processing unit 11 in the directional characteristic storage unit 31.

(5-2) Correction of Desired Directivity When the desired directional characteristic is determined by the directional characteristic determination processing unit 11, the control unit 10 of the directivity control device 1 then, under the control of the directional characteristic correction processing unit 12, in step S12, the directional characteristic correction processing unit 12 reads the desired directional characteristic from the directional characteristic storage unit 31 and corrects the desired directional characteristic. For this reason, the directional characteristic correction processing unit 12 calculates spherical harmonic expansion coefficients A _n,m for the input directional characteristic using the maximum order N _max according to the following formula. N _max is uniquely determined at the time of designing t-design.

(5-3) Design of Filter for Directivity Control Next, in step S13, the control unit 10 of the directivity control device 1 calculates a filter for directivity control by solving an optimization problem of the objective function expressed by equation (3) under the objective function set based on the desired directional characteristic corrected by the directional characteristic correction processing unit 12 and the range constraint of the microphone gain under the control of the directivity control filter design processing unit 13. Here, the optimization problem is defined as follows.

By finding the subdifferential with respect to w for equation (14), we obtain the following algorithm.

Here, γ is the update rate and prox represents the proximity operator. The update formula for the resulting algorithm is expressed as follows:

Here, k is the number of updates, _Tmax is the maximum number of updates, and the algorithm stops when the number of updates k exceeds the maximum number of updates _Tmax . By using the above algorithm, it is possible to calculate a filter for directivity control with a filter gain within a desired range.

The control unit 10 of the directivity control device 1 stores the directivity control filter calculated by the directivity control filter design processing unit 13 in the filter storage unit 32.

(5-4) Convolution of Input Acoustic Signal and Filter When the calculation and storage process of the filter for directivity control is completed, the control unit 10 of the directivity control device 1 thereafter executes the process of directivity control of the microphone array MC as follows.

That is, the control unit 10 of the directivity control device 1 receives the input acoustic signal IS from the microphone array MC via the acoustic signal I/F 50. At this time, if the input acoustic signal IS is an analog signal, it is converted into a digital signal by the acoustic signal I/F 50. Next, in step S14, under the control of the convolution calculation processing unit 14, the control unit 10 of the directivity control device 1 performs a convolution calculation between the input acoustic signal IS received by the acoustic signal I/F 50 and the directivity control filter stored in the filter storage unit 32. As a result, an acoustic signal is generated in which sound in the desired target direction is controlled with high directivity. The generated acoustic signal is converted into an analog signal by the acoustic signal I/F 50 and then output as the output acoustic signal OS.

(effect)
As described above, the directivity control device 1 according to one embodiment of the present invention is a device that generates an output audio signal OS in which sound in a desired target direction is controlled with high directivity by performing a convolution operation between the input audio signals _IS1 to _ISM input from each microphone _MC1 to _MCM of the microphone array MC and a directivity control filter corresponding to each microphone.

The directional characteristic determination processing unit 11 determines the desired directional characteristic from the input sound pickup direction.

In the directional characteristic correction processing unit 12, the desired directional characteristic is corrected using spherical harmonic expansion by t-design.

The directivity control filter design processing unit 13 designs a directivity control filter that achieves the desired directivity reproduction by solving the proximity gradient method with an indicator function that causes the filter coefficients to fall within a given range as the target function.

The convolution calculation processing unit 14 performs a convolution calculation between the input sound signal IS and the calculated directivity control filter to generate an output sound signal in which the sound in the desired target direction is controlled with high directivity.

[Other embodiments]
The present invention is not limited to the above embodiment. For example, in one embodiment, the directional characteristic correction process, the directional control filter design process, and the convolution calculation process executed by the directivity control device 1 are all realized by having a hardware processor (CPU) execute a program. However, some or all of these functions may be realized by using an integrated circuit configured for a specific application, such as an ASIC (Application Specific Integrated Circuit) or a DMC (Digital Signal Processor).

In short, this invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

MC ₁ to MC _M ... microphones CP ₁ to CP _Q ... control points 1 ... directivity control device 2 ... input device 10 ... control unit 11 ... directivity determination processing unit 12 ... directivity correction processing unit 13 ... directivity control filter design processing unit 14 ... convolution calculation processing unit 20 ... program storage unit 30 ... data storage unit 31 ... directivity storage unit 32 ... filter storage unit 40 ... input/output I/F
50...Acoustic signal I/F

Claims (6)

A directivity control device for controlling a directivity of an input sound signal input from a microphone array having a plurality of microphones,
a directional characteristic determination processing unit that determines a desired directional characteristic from a desired sound collection direction;
a directional characteristic correction processing unit that corrects the desired directional characteristic using a spherical harmonic function expansion;
a directivity control filter design processing unit that designs a directivity control filter corresponding to each microphone based on the corrected desired directional characteristic;
a convolution calculation processing unit that performs a convolution calculation of an acoustic signal and the directivity control filter to generate an acoustic signal whose directivity is controlled to the desired directional characteristic,
Directional control device. The directional characteristic correction processing unit derives expansion coefficients of the desired directional characteristic by a spherical harmonic function of the maximum order based on the control points designed by t-design, and corrects the desired directional characteristic by reconstructing the directional characteristic at an arbitrarily specified order based on the obtained expansion coefficients.
The directivity control device according to claim 1 . the directivity control filter design processing unit designs the filter by minimizing an error between the desired directivity characteristic and a reproduced directivity characteristic;
The directivity control device according to claim 1 . the directivity control filter design processing unit designs the filter so that the filter gain of each microphone falls within a predetermined range;
The directivity control device according to claim 1 . A directivity control method for controlling a directivity of an acoustic signal input from a microphone array having a plurality of microphones, comprising:
determining a desired directional characteristic from a desired sound collection direction;
correcting the desired directional characteristic using a spherical harmonic expansion;
designing a directivity control filter corresponding to each microphone based on the corrected desired directional characteristics;
performing a convolution operation of an acoustic signal and the directivity control filter to generate an output acoustic signal whose directivity is controlled to the desired directivity characteristic;
Directional control method. A computer having a processor and a storage device,
5. A method for controlling a directionality control device according to claim 1, comprising:
Directional control program.

Images