CN105682000B - A kind of audio-frequency processing method and system - Google Patents

Abstract

The present invention relates to a cloud audio processing method, a server and a system. For audio signals of different formats, according to the head rotation angle of the client, binaural transcoding is performed on the audio signals of different formats to generate binaural sounds of corresponding formats. channel audio signal; superimpose the two-channel signal of the corresponding format to obtain the audio binaural output virtual surround sound signal. The audio processing of the present invention is carried out on the cloud server, which is well adapted to the existing audio processing and storage based on the cloud architecture, thereby reducing the problems of low quality and heavy calculation of virtual surround sound generated by the mobile terminal. In addition, for the possible delay caused by the above-mentioned execution on the server, the present invention also includes smoothing the angle to eliminate the delay.

Description

An audio processing method and system

technical field

The present invention relates to the technical field of signal processing, in particular to an audio processing method, server and system.

Background technique

When using a virtual reality head-mounted display (HMD) to present content to users, virtual 3D audio technology is used to play audio content to users through stereo headphones. One way to improve the sense of presence is to track user head movements ( head tracking) to process the sound accordingly. For example, if the original sound is perceived by the user as coming from the front, when the user turns his head 90 degrees to the left, the sound should be processed so that the user perceives the sound as coming from the right 90 degrees. The virtual reality device here can be of many kinds, such as a display device with head tracking, or just a stereo headset with head tracking sensors.

There are also multiple ways to implement head tracking. It is more common to use a variety of motion sensors. Motion sensor kits typically include accelerometers, gyroscopes, and magnetic sensors. Each sensor has its own inherent strengths and weaknesses when it comes to motion tracking and absolute orientation. It is therefore common practice to use sensor "fusion" (sensor fusion) to combine the signals from the various sensors together to produce a more accurate motion detection result.

After getting the head rotation angle, you need to change the sound accordingly. One way is to convert the sound to the ambisonic domain and then transform the signal by using a rotation matrix. Ambisonic signals usually have more than two channels, and common media players only support two stereo channels, which makes it difficult to directly play ambisonic or other multi-channel audio signals.

In view of this, there is a need in the art for an effective and high-quality solution for generating and playing virtual surround sound.

Contents of the invention

In order to overcome the above-mentioned defects of the prior art, the object of the present invention is to provide a cloud audio processing method, server and system, which can effectively and high-quality generate virtual surround sound, and is mainly used for audio processing with virtual reality headsets Stereo headphones are played, and the generation of the virtual surround sound is carried out on the cloud server, which is well adapted to the existing network type based on the cloud architecture, and the generation and storage of the virtual surround sound are performed by the server, thus solving the current problem There is a problem that the client cannot play various 3603D audio, especially the audio suitable for virtual reality applications.

In order to achieve the above object, the present invention provides a cloud audio processing method. The audio processing method includes the following steps: acquiring the rotation angle of the user's head; Audio signals in different formats are binaurally transcoded to generate binaural audio signals in corresponding formats; the binaural signals in corresponding formats are superimposed to obtain audio binaural output virtual surround sound signals.

Preferably, the audio signals in different formats include binaural recording signals, ambisonic recording signals and audio object signals.

Preferably, performing binaural transcoding on the audio signals in different formats, and generating binaural transcoding audio signals in corresponding formats specifically includes:

For the binaural recording signal, interpolation is performed according to the rotation angle to generate a binaural recording binaural signal;

For the ambisonic recording signal, the ambisonic recording signal is adjusted according to the rotation angle, and the adjusted ambisonic recording signal is binaurally transcoded to generate an ambisonic recording binaural signal;

For the audio object signal, the audio object signal is adjusted according to the rotation angle, and the adjusted audio object signal is binaurally transcoded to generate an audio object binaural signal.

Preferably, if higher spatial precision is required, the audio object signal is rotated according to the rotation angle, the rotated audio object signal is encoded into a high-order B-format audio object signal, and the high-order B-format audio is generated after binaural transcoding Object binaural signal, superimposed with ambisonic recording binaural signal and binaural recording binaural signal;

If low complexity and low delay are required, the audio object signal is encoded into a first-order B-format audio object signal, superimposed with other first-order ambisonic recording signals, and then the superimposed mixed signal is binaurally transcoded according to the rotation angle to generate The mixed binaural signal of the audio object and the ambisonic recording signal is superimposed on the binaural signal of the binaural recording.

Preferably, the acquired rotation angle of the user's head rotation is specifically acquiring the rotation angle of the user's head rotation, and performing smoothing processing on the rotation angle.

The present invention also provides a cloud audio processing server. The server includes: an acquisition unit, which acquires the rotation angle of the user's head; an acquisition unit, which acquires audio signals in different formats; The unit is connected to the acquisition unit, and according to the rotation angle, binaural transcoding is performed on the audio signals in different formats to generate binaural audio signals in corresponding formats; the superposition unit is connected to the binaural transcoding unit , superimposing the binaural signals of the corresponding formats to obtain binaural audio output virtual surround sound signals.

Preferably, the audio signals in different formats include binaural recording signals, ambisonic recording signals and audio object signals.

Preferably, the binaural transcoding unit performs binaural transcoding on the audio signals of different formats, and generating a binaural transcoding audio signal of a corresponding format specifically includes:

For the binaural recording signal, interpolation is performed according to the rotation angle to generate a binaural recording binaural signal;

For the ambisonic recording signal, the ambisonic recording signal is adjusted according to the rotation angle, and the adjusted ambisonic recording signal is binaurally transcoded to generate an ambisonic recording binaural signal;

For the audio object signal, the audio object signal is adjusted according to the rotation angle, and the adjusted audio object signal is binaurally transcoded to generate an audio object binaural signal.

Preferably, if higher spatial precision is required, the binaural transcoding unit rotates the audio object signal according to the rotation angle, and encodes the rotated audio object signal into a high-order B-format audio object signal, which is generated after binaural transcoding The high-order B-format audio object binaural signal, the superposition unit superimposes the high-order B-format audio object binaural signal generated by the binaural transcoding unit, the ambisonic recording binaural signal, and the binaural recording binaural signal;

If low complexity and low delay are required, the binaural transcoding unit encodes the audio object signal into a first-order B-format audio object signal, superimposes it with other first-order ambisonic recording signals, and then double-binds the superimposed mixed signal according to the rotation angle. Ear transcoding generates a mixed binaural signal of an audio object and an ambisonic recording signal, and the superposition unit superimposes the mixed binaural signal and the binaural recording binaural signal generated by the binaural transcoding unit.

Preferably, the cloud server further includes a smoothing unit connected to the binaural transcoding unit and the acquisition unit respectively, the smoothing unit receives the rotation angle of the user's head rotation from the acquisition unit, and performs smoothing processing on the rotation angle .

The present invention also provides an audio playback system, the system includes a cloud audio processing server, and a client; the client includes a head tracking device, and the head tracking device captures the rotation angle of the head and uploads a To the cloud audio processing server, the cloud audio processor receives the rotation angle, generates an audio binaural output virtual surround sound signal, and transmits it to the client through the network.

According to the cloud audio processing method, server and system of the present invention, effectively and high-quality generation of virtual surround sound is mainly used for stereo earphone playback of audio in conjunction with virtual reality headsets, and the generation of the virtual surround sound is in the cloud It is carried out on the server, which is well adapted to the existing cloud-based network type, and the cloud server performs audio processing and storage, thus solving the problem that the existing client cannot play various 3603D audio, especially for virtual reality applications audio problem.

Using the cloud audio processing technology of the present invention can greatly enhance the sense of presence in multi-person voice communication, and users can turn their heads at will to pay attention to the sound in a certain direction, which is closer to the real multi-person conversation scene. Especially in scenarios where streaming media is used, the user's audio experience can be improved by adjusting spatial sound and audio orientation in real time. If the virtual reality video content is assisted, the user experience will be better improved.

Description of drawings

Fig. 1 is a functional block diagram of an embodiment of the cloud audio processing method of the present invention;

2a-c are schematic block diagrams of another embodiment of the cloud audio processing method of the present invention;

Fig. 3 is a schematic structural diagram of an embodiment of the audio processing server of the present invention;

Fig. 4 is a schematic structural diagram of another embodiment of the audio processing system of the present invention;

detailed description

Embodiment one: as shown in Figure 1, a kind of audio object processing comprises the following processing steps:

Obtain the rotation angle of the user's head through the head tracking device;

Encoding the audio object into a higher order (preferably 2nd or 3rd order) Ambisonic B-format signal according to said rotation angle;

Convert the Ambisonic B-format signal into a virtual loudspeaker array signal; take a first-order B-format signal [W ₁ X ₁ Y ₁ Z ₁ ] ^T as an example, convert it into a virtual loudspeaker array signal [L ₁ L ₂ ... L The process of _N ] ^T is to perform the following operations:

Wherein, N is the number of virtual speakers included in the virtual speaker topology. The G matrix used in the above formula is an ambisonic decoding matrix, which can be obtained by calculating the pseudo-inverse matrix.

The virtual speaker array signal of the audio object is binaurally transcoded based on the binaural room impulse response (BRIR) (usually 3D, ie contains height information), to obtain the binaural output virtual surround sound signal of the audio object. Specifically: transfer the virtual speaker signal to the two-way stereo BRIR matrix corresponding to the headphone signal, perform matrix multiplication on the two-way stereo matrix and the virtual speaker array signal, and obtain virtual surround sound.

The BRIR matrix is Then the virtual surround sound is There may be one or more audio signals.

The binaural room impulse response is preferably generated offline, which can be generated by real measurement or by special software. Therefore, it is not necessary to store a large amount of BRIR as in the prior art when online generation is used, reducing memory consumption.

When encoding an audio object into an Ambisonic B-format signal, the order in the horizontal direction is preferably greater than or equal to the order in the vertical direction. For example, when the encoding in the horizontal direction is preferably a 3-order Ambisonic B-format signal, the encoding in the vertical direction is preferably 2-order or 1 The second-order Ambisonic B-format signal is represented by H3V2 and H3V1 respectively. Since the height perception of human beings is lower than the resolution of the plane angle, the above method of appropriately reducing the order in a specific direction reduces the amount of calculation, but does not significantly reduce the user's perception of sound.

Processing the sound field signal, that is, the ambient sound, includes the following steps:

The ambient sound is converted into the binaural output virtual surround sound signal of the ambient sound, and then the audio object (the audio object at this time mainly refers to the sound content other than the ambient sound) and the respective binaural output of the ambient sound virtual Surround sound signals are mixed and binaurally output. Figure 1 shows a schematic block diagram of one embodiment of the method. Wherein, the binaural output virtual surround sound signal that described environmental sound (being the sound field signal in Fig. 1) is converted into environmental sound preferably comprises the following steps:

Obtain the 1st order Ambisonic B-format signal of the ambient sound;

According to the rotation angle, the ambisonic B-format signal of the ambient sound is rotated to obtain the rotated ambisonic B-format signal; specifically, a rotation matrix is generated according to the rotation angle, and then according to the rotation matrix, the The ambisonic B-format signal of the ambient sound (ie the signal to be adjusted) is rotated. The so-called rotation refers to multiplying the rotation matrix by the signal matrix to be adjusted. The rotation does not change the size of the audio signal matrix components, but only changes the direction of the components. The order of the rotation matrix is adapted to the audio signal matrix. For example, when the signal matrix to be adjusted is [W ₂ X ₂ Y ₂ ] ^T , the rotation matrix is When the signal matrix to be adjusted is [W ₂ X ₂ Y ₂ Z ₂ ] ^T , the rotation matrix is

The Ambisonic B-format signal after the rotation of the ambient sound is converted into a virtual speaker array signal; the virtual speaker array signal of the ambient sound is binaurally transcoded based on a head-related transform function (HRTF) (usually 2-dimensional, That is, the height information is not included), and the binaural output of the ambient sound is obtained to output the virtual surround sound signal. The name corresponding to HRTF in the time domain is HRIR (Head Related Impulse Response).

It should be pointed out that BRIR or HRIR can be used to filter audio objects or ambient sounds as needed. Since BRIR usually consists of a room model and a set of HRIR/HRTF describing the sound orientation, if the input signal already has room or environment information, using HRIR can meet the requirements.

The method for generating virtual surround sound is preferably based on the following assumptions when performing calculations: the virtual loudspeaker array has left-right symmetry, the user is on the central axis of the room, and the binaural room impulse response and head-related transfer function corresponding to the user also have Left-right symmetry. Based on this assumption, the high-order Ambisonic B-format symmetry optimization method can be used to significantly reduce the amount of computation and improve the computation efficiency.

The following describes how to encode an audio object into the ambisonic domain.

Encode an audio object to a first-order ambisonic signal:

s _i s _i is the i-th audio object, i=1..k, k is the number of audio objects. θ _i θ _i is the angle (azimuth angle) on the plane, and φ _i φ _i is the angle in the vertical direction. The W channel signal represents an omnidirectional sound wave, and the X channel signal, Y channel signal and Z channel signal respectively represent three mutually vertically oriented sound waves of X, Y, and Z along the space.

A first-order ambisonic B-format signal is expressed as

Likewise, encoding an audio object into a 2nd-order or 3rd-order Ambisonic B-format signal is preferably performed as defined in the following table:

The trigonometric function in the above table is an even function for the azimuth angle θ, then the corresponding component of the corresponding Ambisonic B-format signal is left-right symmetric, if the trigonometric function in the above table is an odd function for the azimuth angle θ, then the corresponding Ambisonic B- The corresponding components of the format signal are inverse left and right. Taking the first-order ambisonic B-format signal as an example, from the perspective of physical meaning and coordinates, w, x, and z are not divided into left and right, so if the listening position is left-right symmetrical, and assuming that the corresponding HRTF coefficients are also approximately left-right symmetrical, then w , the components of the binaural output corresponding to x, z are the same for the left and right channels of the output. And y is exactly the opposite for left and right. So the components of the binaural output corresponding to y are opposite for the left and right channels. For components with symmetry, a fast algorithm can be used, that is, symmetry optimization in the operation process, which can further reduce the amount of calculation.

In addition, since there may be a delay in the processing of the audio file by the server, the solution adopted is to obtain the rotation angle of the user's head rotation, and perform smoothing processing on the rotation angle. Therefore, small angle changes do not require new rotation direction processing, which effectively solves the delay problem of server processing.

Embodiment two:

Figures 2a-c describe an embodiment of enhancing the effect of immersive experience based on cloud-based multi-channel audio transmission. It should be noted that the present invention covers two application scenarios (1) audio real-time communication (conference scenario), as shown in Figure 2b; (2) audio downloading, as shown in Figure 2c;

For the two scenarios, there are three forms of input: a separate audio object, sound field input (wxy form), and binaural recording signals.

As shown in Figure 2b, for the audio download scenario:

The storage server stores binaural recording signals, ambisonic recording signals (sound field signals), and/or audio objects, and the binaural transcoding server obtains the above-mentioned signals from the storage server, and converts the audio objects into ambisonic signals on the binaural transcoding server side, For example, a first-order horizontal direction B-format signal, ie, wxy, is summed with other wxy signals (sound field signals). The binaural transcoding server rotates the wxy signal by using the rotation matrix according to the angle transmitted by the head tracking device of the client, converts the wxy signal into binaural, and then superimposes the binaural recording binaural signal to generate an audio download file. Compression is often required to reduce transmission bandwidth. Then the client downloads the compressed binaural audio. This approach will be more efficient, but the disadvantage is that if the audio object only uses the first-order B format, the resolution of spatial positioning will be reduced, but the preferred method based on cloud services If the binauralization process is placed on the client, the client If the wxy signal is downloaded from the server, the rotation operation does not need to go through the server.

If higher spatial precision is required, the binaural transcoding server first rotates the audio object according to the rotation angle, and encodes the rotated audio object signal into a high-order B format (such as three-order three), and other B-format signals in the binaural Channel domain superimposition: After binaural transcoding, a high-order B-format audio object binaural signal is generated, which is superimposed with the Ambisonic recording binaural signal and binaural recording binaural signal to generate an audio file.

Here we should note that head tracking is only a form and does not exclude other motion parameters, such as waving, etc. The present invention is equally applicable.

As shown in Figure 2c, for audio real-time communication (conference scenario):

The binaural transcoding server directly obtains the binaural recording microphone array, ambisonic microphone array, individual sound source or audio object, and performs the above-mentioned operations on the binaural recording microphone array, ambisonic microphone array, individual sound source or audio object on the binaural transcoding server side similar process.

Embodiment three:

As shown in Figure 3, a cloud audio processing server, the acquisition unit acquires the rotation angle of the user's head rotation transmitted by the head tracking device in the client; the acquisition unit collects binaural recording signals, ambisonic recording signals, audio Object: a binaural transcoding unit, which is respectively connected to the acquisition unit and the acquisition unit, and performs binaural transcoding on the audio signals of different formats according to the rotation angle, wherein for the binaural recording signal, according to the The above rotation angle is interpolated to generate a binaural recording binaural signal; and if higher spatial accuracy is required, the binaural transcoding unit rotates the audio object signal according to the rotation angle, and encodes the rotated audio object signal It is a high-order B-format audio object signal. After binaural transcoding, a high-order B-format audio object binaural signal is generated. The superposition unit generates a high-order B-format audio object binaural signal generated by the binaural transcoding unit. Ambisonic recording Superposition of binaural signals and binaural recording binaural signals;

If low complexity and low delay are required, the binaural transcoding unit encodes the audio object signal into a first-order B-format audio object signal, superimposes it with other first-order ambisonic recording signals, and then double-binds the superimposed mixed signal according to the rotation angle. Ear transcoding generates a mixed binaural signal of an audio object and an ambisonic recording signal, and the superposition unit superimposes the mixed binaural signal and the binaural recording binaural signal generated by the binaural transcoding unit to obtain an audio binaural The ear outputs a virtual surround sound signal.

In this embodiment, a cloud server is used to solve the problem of multi-channel audio transmission and playback supporting head tracking.

Embodiment four:

As shown in Figure 4, an audio processing system of the present invention mainly includes a client, a storage server, and a cloud audio processing server; the client includes a head tracking module, and the storage server stores multi-channel audio files and stores them in a specific way . The client head tracking module obtains the user's head movement such as the head rotation angle, and uploads the parameters to one or more cloud audio processing servers on the server side via the Internet, and performs corresponding processing on the multi-channel audio files: the cloud audio processing server from The storage server extracts audio signals in different formats, generates audio binaural output virtual surround sound signals according to the received rotation angle, and transmits the binaurally transcoded audio files to the client through the network.

The client downloads the above-mentioned processed audio file, and preferably plays it in a two-channel stereo format.

The preferred embodiment of the present invention has been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the specific details of the above embodiment, within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, These simple modifications all belong to the protection scope of the present invention.

In addition, it should be noted that the various specific technical features described in the above specific implementation manners may be combined in any suitable manner if there is no contradiction. In order to avoid unnecessary repetition, various possible combinations are not further described in the present invention.

In addition, various combinations of different embodiments of the present invention can also be combined arbitrarily, as long as they do not violate the idea of the present invention, they should also be regarded as the disclosed content of the present invention.

Claims (10)

1. A cloud audio processing method, characterized in that: the audio processing method comprises the following steps, Obtain the rotation angle of the user's head rotation; Acquiring audio signals in different formats, performing binaural transcoding on the audio signals in different formats according to the rotation angle, to generate binaural audio signals in corresponding formats; Superimposing the binaural signals of the corresponding formats to obtain audio binaural output virtual surround sound signals; The acquired rotation angle of the user's head rotation is specifically acquiring the rotation angle of the user's head rotation, and smoothing the rotation angle. 2. The cloud audio processing method according to claim 1, characterized in that: The audio signals in different formats include binaural recording signals, ambisonic recording signals and audio object signals. 3. The cloud audio processing method according to claim 2, characterized in that: Performing binaural transcoding on the audio signals of different formats, and generating binaural transcoding audio signals of corresponding formats specifically includes: For the binaural recording signal, interpolation is performed according to the rotation angle to generate a binaural recording binaural signal; For the ambisonic recording signal, the ambisonic recording signal is adjusted according to the rotation angle, and the adjusted ambisonic recording signal is binaurally transcoded to generate an ambisonic recording binaural signal; For the audio object signal, the audio object signal is adjusted according to the rotation angle, and the adjusted audio object signal is binaurally transcoded to generate an audio object binaural signal. 4. The cloud audio processing method according to claim 3, characterized in that: If higher spatial precision is required, the audio object signal is rotated according to the rotation angle, and the rotated audio object signal is encoded into a high-order B-format audio object signal, and the high-order B-format audio object binaural is generated after binaural transcoding channel signal, superimposed with ambisonic recording binaural signal and binaural recording binaural signal; If low complexity and low delay are required, the audio object signal is encoded into a first-order B-format audio object signal, superimposed with other first-order ambisonic recording signals, and then the superimposed mixed signal is binaurally transcoded according to the rotation angle to generate The mixed binaural signal of the audio object and the ambisonic recording signal is superimposed on the binaural signal of the binaural recording. 5. A cloud audio processing server, characterized in that the server comprises: The acquisition unit acquires the rotation angle of the user's head rotation; The acquisition unit collects audio signals in different formats; The binaural transcoding unit is respectively connected to the acquisition unit and the acquisition unit, and according to the rotation angle, performs binaural transcoding on the audio signals of different formats to generate a binaural audio signal of a corresponding format; A superposition unit, connected to the binaural transcoding unit, superimposes the binaural signals of the corresponding format to obtain audio binaural output virtual surround sound signals; The cloud audio processing server further includes a smoothing unit connected to the binaural transcoding unit and the acquisition unit respectively, the smoothing unit receives the rotation angle of the user's head rotation from the acquisition unit, and performs smoothing processing on the rotation angle. 6. The cloud audio processing server according to claim 5, characterized in that: The audio signals in different formats include binaural recording signals, ambisonic recording signals and audio object signals. 7. The cloud audio processing server according to claim 6, characterized in that: The binaural transcoding unit performs binaural transcoding on the audio signals of different formats, and generates binaural transcoding audio signals of corresponding formats specifically including: For the binaural recording signal, interpolation is performed according to the rotation angle to generate a binaural recording binaural signal; For the ambisonic recording signal, the ambisonic recording signal is adjusted according to the rotation angle, and the adjusted ambisonic recording signal is binaurally transcoded to generate an ambisonic recording binaural signal; For the audio object signal, the audio object signal is adjusted according to the rotation angle, and the adjusted audio object signal is binaurally transcoded to generate an audio object binaural signal. 8. The cloud audio processing server according to claim 7, characterized in that: If higher spatial precision is required, the binaural transcoding unit rotates the audio object signal according to the rotation angle, encodes the rotated audio object signal into a high-order B format audio object signal, and generates a high-order B format after binaural transcoding Format audio object binaural signal, the superimposition unit superimposes the high-order B-format audio object binaural signal generated by the binaural transcoding unit, the ambisonic recording binaural signal, and the binaural recording binaural signal; If low complexity and low delay are required, the binaural transcoding unit encodes the audio object signal into a first-order B-format audio object signal, superimposes it with other first-order ambisonic recording signals, and then double-binds the superimposed mixed signal according to the rotation angle. Ear transcoding generates a mixed binaural signal of an audio object and an ambisonic recording signal, and the superposition unit superimposes the mixed binaural signal and the binaural recording binaural signal generated by the binaural transcoding unit. 9. An audio playback system, characterized in that: the system includes the cloud audio processing server described in any one of claims 5-8, and a client; the client includes a head tracking device, and the head tracking device Grab the head rotation angle, upload it to the cloud audio processing server through the network, the cloud audio processor obtains audio signals in different formats, and generates audio binaural output virtual surround sound signals according to the rotation angle, and passes through the The above network is transmitted to the client. 10. The audio playback system according to claim 9, characterized in that: the system also includes a storage server that stores audio signals in different formats, and when a user requests to download an audio file to download, the cloud audio processing server starts from the A storage server retrieves the audio signal.