WO2022242480A1 - 三维音频信号编码方法、装置和编码器 - Google Patents

三维音频信号编码方法、装置和编码器 Download PDF

Info

Publication number
WO2022242480A1
WO2022242480A1 PCT/CN2022/091558 CN2022091558W WO2022242480A1 WO 2022242480 A1 WO2022242480 A1 WO 2022242480A1 CN 2022091558 W CN2022091558 W CN 2022091558W WO 2022242480 A1 WO2022242480 A1 WO 2022242480A1
Authority
WO
WIPO (PCT)
Prior art keywords
coefficients
representative
virtual
virtual speakers
current frame
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/091558
Other languages
English (en)
French (fr)
Inventor
高原
刘帅
王宾
王喆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to BR112023023662A priority Critical patent/BR112023023662A2/pt
Priority to JP2023571383A priority patent/JP7656090B2/ja
Priority to EP22803804.8A priority patent/EP4322158A4/en
Priority to CA3220588A priority patent/CA3220588A1/en
Priority to KR1020237040819A priority patent/KR20240001226A/ko
Publication of WO2022242480A1 publication Critical patent/WO2022242480A1/zh
Priority to US18/511,191 priority patent/US20240087580A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/008—Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/002—Dynamic bit allocation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
    • G10L19/16—Vocoder architecture
    • G10L19/167—Audio streaming, i.e. formatting and decoding of an encoded audio signal representation into a data stream for transmission or storage purposes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/0204—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using subband decomposition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/032—Quantisation or dequantisation of spectral components
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2420/00—Techniques used stereophonic systems covered by H04S but not provided for in its groups
    • H04S2420/11—Application of ambisonics in stereophonic audio systems

Definitions

  • the present application relates to the field of multimedia, in particular to a three-dimensional audio signal encoding method, device and encoder.
  • three-dimensional audio technology has been widely used in wireless communication (such as 4G/5G, etc.) voice, virtual reality/augmented reality, and media audio.
  • Three-dimensional audio technology is an audio technology that acquires, processes, transmits, renders and replays sound and three-dimensional sound field information in the real world. "Extraordinary listening experience.
  • a collection device such as a microphone collects a large amount of data to record 3D sound field information, and transmits 3D audio signals to a playback device (such as a speaker, earphone, etc.), so that the playback device can play 3D audio.
  • a playback device such as a speaker, earphone, etc.
  • the three-dimensional audio signal can be compressed, and the compressed data can be stored or transmitted.
  • encoders can compress 3D audio signals using pre-configured multiple virtual speakers.
  • the computational complexity for the encoder to compress and encode the 3D audio signal is relatively high. Therefore, how to reduce the computational complexity of compressing and encoding 3D audio signals is an urgent problem to be solved.
  • the present application provides a three-dimensional audio signal encoding method, device and encoder, thereby reducing the computational complexity of compressing and encoding the three-dimensional audio signal.
  • the present application provides a method for encoding a three-dimensional audio signal, which can be executed by an encoder, and specifically includes the following steps: the encoder obtains the fourth number of coefficients of the current frame of the three-dimensional audio signal, and the fourth number of coefficients After the frequency-domain eigenvalues of the coefficients, according to the frequency-domain eigenvalues of the fourth number of coefficients, select the third number of representative coefficients from the fourth number of coefficients, and then select the candidate virtual speaker set from the third number of representative coefficients Select the second number of representative virtual speakers of the current frame, and encode the current frame according to the second number of representative virtual speakers of the current frame to obtain a code stream.
  • the fourth number of coefficients includes a third number of representative coefficients, and the third number is smaller than the fourth number, indicating that the third number of representative coefficients is part of the fourth number of coefficients.
  • the current frame of the three-dimensional audio signal is a higher order ambisonics (HOA) signal; the frequency domain characteristic value of the coefficient is determined according to the coefficient of the HOA signal.
  • HOA ambisonics
  • the encoder selects some coefficients from all the coefficients of the current frame as representative coefficients, and uses a smaller number of representative coefficients to replace all the coefficients of the current frame to select representative virtual speakers from the set of candidate virtual speakers, thus effectively reducing the The computational complexity of searching for a virtual speaker is reduced, thereby reducing the computational complexity of compressing and encoding a three-dimensional audio signal and reducing the computational burden of an encoder.
  • the encoder encodes the current frame according to the representative virtual speakers of the second number of current frames, and obtaining the code stream includes: the encoder generates a virtual speaker signal according to the representative virtual speakers of the second number of current frames and the current frame; The signal is encoded to obtain a code stream.
  • the encoder Since the frequency-domain eigenvalues of the coefficients of the current frame characterize the sound field characteristics of the three-dimensional audio signal, the encoder selects the representative coefficients of the representative sound field components of the current frame according to the frequency-domain eigenvalues of the coefficients of the current frame, and uses the representative coefficients from the candidate virtual
  • the representative virtual speaker of the current frame selected in the speaker set can fully represent the sound field characteristics of the 3D audio signal, thereby further improving the ability of the encoder to generate a virtual speaker signal when compressing and encoding the 3D audio signal to be encoded using the representative virtual speaker of the current frame. Accuracy, in order to improve the compression rate of the three-dimensional audio signal compression encoding, reduce the bandwidth occupied by the encoder to transmit the code stream.
  • selecting a third number of representative coefficients from the fourth number of coefficients according to the frequency domain characteristic values of the fourth number of coefficients includes: the encoder according to the frequency domain characteristics of the fourth number of coefficients value, selecting representative coefficients from at least one subband included in the spectrum range indicated by the fourth number of coefficients to obtain a third number of representative coefficients.
  • selecting representative coefficients from at least one subband included in the spectrum range indicated by the fourth number of coefficients to obtain the third number of representative coefficients includes: the encoder according to at least one subband The frequency-domain eigenvalues of the coefficients in each sub-band in the band, Z representative coefficients are respectively selected from each sub-band to obtain a third number of representative coefficients, and Z is a positive integer. Since the encoder selects representative coefficients according to the frequency-domain eigenvalues of the coefficients within the spectrum range indicated by all the coefficients of the current frame, thus ensuring that each sub-band has representative coefficients selected, which improves the performance of the encoder in the current frame indicated by all the coefficients. The equalization of representative coefficients is selected in the spectrum range.
  • representative coefficients are selected from at least one subband included in the spectral range indicated by the fourth number of coefficients to obtain the first Three numbers of representative coefficients include: the encoder determines the weight of each sub-band according to the frequency-domain feature value of the first candidate coefficient in each sub-band in at least two sub-bands; adjusts the weights in each sub-band according to the respective weights of each sub-band The frequency-domain eigenvalues of the second candidate coefficients are obtained by obtaining the adjusted frequency-domain eigenvalues of the second candidate coefficients in each subband, and the first candidate coefficients and the second candidate coefficients are partial coefficients in the subbands; according to at least two subbands The adjusted frequency-domain feature values of the second candidate coefficients in the sub-band, and the frequency-domain feature values of coefficients other than the second candidate coefficients in at least two subbands, determine a third number of
  • the encoder adjusts the probability of selecting the coefficients in the subband according to the weight of the subband, which further improves the accuracy that the representative coefficients selected by the encoder represent the coefficients of all subbands in terms of sound field distribution and audio characteristics.
  • the encoder can divide the spectrum range equally to obtain at least two subbands, and the number of coefficients contained in the at least two subbands is different; or, the encoder can also divide the spectrum range equally to obtain at least two subbands, then at least two subbands Each subband in the band contains the same number of coefficients.
  • selecting the second number of representative virtual speakers of the current frame from the candidate virtual speaker set according to the third number of representative coefficients includes: the encoder according to the third number of representative coefficients of the current frame, the candidate The set of virtual speakers and the number of voting rounds determine the first number of virtual speakers and the first number of voting values, and select the second number of representative virtual speakers of the current frame from the first number of virtual speakers according to the first number of voting values, The second number is smaller than the first number, indicating that the representative virtual speakers of the second number of current frames are part of the virtual speakers in the candidate virtual speaker set. Understandably, the virtual speaker corresponds to the voting value one by one.
  • the first number of virtual speakers includes a first virtual speaker
  • the first number of voting values includes voting values of the first virtual speaker
  • the first virtual speaker corresponds to the voting value of the first virtual speaker.
  • the voting value of the first virtual speaker is used to represent the priority of the first virtual speaker.
  • the set of candidate virtual speakers includes a fifth number of virtual speakers, the fifth number of virtual speakers includes a first number of virtual speakers, the first number is less than or equal to the fifth number, the number of voting rounds is an integer greater than or equal to 1, and the voting round number is less than or equal to the fifth number.
  • the second quantity is preset, or, the second quantity is determined according to the current frame.
  • the encoder uses the result of correlation calculation between the three-dimensional audio signal to be encoded and the virtual speaker as the selection indicator of the virtual speaker. Moreover, if the encoder transmits a virtual speaker for each coefficient, the goal of high-efficiency data compression cannot be achieved, and a heavy computational burden will be imposed on the encoder. In the method for selecting a virtual speaker provided in the embodiment of the present application, the encoder uses a small number of representative coefficients to replace all the coefficients of the current frame to vote for each virtual speaker in the candidate virtual speaker set, and selects the representative virtual speaker of the current frame according to the voting value .
  • the encoder uses the representative virtual speaker of the current frame to compress and encode the 3D audio signal to be encoded, which not only effectively improves the compression rate of the 3D audio signal, but also reduces the computational complexity of the encoder searching for the virtual speaker. Therefore, the computational complexity of compressing and encoding the three-dimensional audio signal is reduced and the computational burden of the encoder is reduced.
  • the second number is used to represent the number of representative virtual speakers of the current frame selected by the encoder.
  • the larger the second number the larger the number of representative virtual speakers in the current frame, the more sound field information of the three-dimensional audio signal; the smaller the second number, the smaller the number of representative virtual speakers in the current frame, and the more sound field information of the three-dimensional audio signal. few. Therefore, the number of representative virtual speakers of the current frame selected by the encoder can be controlled by setting the second number.
  • the second number may be preset, and for another example, the second number may be determined according to the current frame.
  • the value of the second quantity may be 1, 2, 4 or 8.
  • selecting the second number of representative virtual speakers of the current frame from the first number of virtual speakers includes: the encoder according to the first number of voting values, and The final voting value of the sixth number of previous frames, obtain the final voting value of the seventh number of current frames corresponding to the seventh number of virtual speakers and the current frame, according to the final voting value of the seventh number of current frames, from the seventh number of virtual speakers
  • the representative virtual speakers of the second number of current frames are selected from the speakers, and the second number is less than the seventh number, indicating that the representative virtual speakers of the second number of current frames are part of the virtual speakers of the seventh number of virtual speakers.
  • the seventh number of virtual speakers includes the first number of virtual speakers
  • the seventh number of virtual speakers includes the sixth number of virtual speakers
  • the virtual speakers included in the sixth number of virtual speakers are the previous frames of the three-dimensional audio signal A virtual speaker representative of the previous frame used for encoding.
  • the sixth number of virtual speakers included in the representative virtual speaker set of the previous frame is in one-to-one correspondence with the sixth number of final voting values of the previous frame.
  • the virtual speaker may not be able to form a one-to-one correspondence with the real sound source, and because in the actual complex scene, there may be A limited number of virtual speaker sets cannot represent all sound sources in the sound field.
  • the virtual speakers searched between frames may jump frequently, and this jump will obviously affect the auditory experience of the listener. , leading to obvious discontinuity and noise in the three-dimensional audio signal after decoding and reconstruction.
  • the method for selecting a virtual speaker provided by the embodiment of this application inherits the representative virtual speaker of the previous frame, that is, for the virtual speaker with the same number, adjusts the initial voting value of the current frame with the final voting value of the previous frame, so that the encoder is more inclined to Select the representative virtual speaker of the previous frame, thereby reducing the frequent jump of the virtual speaker between frames, enhancing the continuity of the signal orientation between frames, and improving the stability of the sound image of the three-dimensional audio signal after reconstruction. Ensure the sound quality of the reconstructed 3D audio signal.
  • the method further includes: the encoder obtains a first correlation degree between the current frame and the previous frame representing a virtual speaker set, and if the first correlation degree does not meet the multiplexing condition, obtain a three-dimensional audio signal The fourth number of coefficients of the current frame of and the frequency-domain feature values of the fourth number of coefficients.
  • the set of representative virtual speakers of the previous frame includes a sixth number of virtual speakers, the virtual speakers included in the sixth number of virtual speakers are representative virtual speakers of the previous frame used for encoding the previous frame of the three-dimensional audio signal, the first The degree of correlation is used to determine whether to reuse the set of representative virtual speakers of the previous frame when encoding the current frame.
  • the encoder can first determine whether the current frame can be encoded by multiplexing the representative virtual speaker set of the previous frame. Executing the process of searching for the virtual speaker effectively reduces the computational complexity of the encoder searching for the virtual speaker, thereby reducing the computational complexity of compressing and encoding the three-dimensional audio signal and reducing the computational burden of the encoder. In addition, it can also reduce the frequent jumps of virtual speakers between frames, enhance the continuity of orientation between frames, improve the stability of the sound image of the reconstructed 3D audio signal, and ensure the accuracy of the reconstructed 3D audio signal. sound quality.
  • the encoder then selects representative coefficients, uses the representative coefficients of the current frame to vote for each virtual speaker in the candidate virtual speaker set, and selects according to the voting value
  • the representative virtual speaker of the current frame is used to reduce the computational complexity of compressing and encoding the 3D audio signal and reduce the computational burden of the encoder.
  • the method further includes: the encoder may also collect the current frame of the 3D audio signal, so as to compress and encode the current frame of the 3D audio signal to obtain a code stream, and transmit the code stream to the decoding end.
  • the encoder may also collect the current frame of the 3D audio signal, so as to compress and encode the current frame of the 3D audio signal to obtain a code stream, and transmit the code stream to the decoding end.
  • the present application provides a three-dimensional audio signal coding device, and the device includes various modules for executing the three-dimensional audio signal coding method in the first aspect or any possible design of the first aspect.
  • a three-dimensional audio signal encoding device includes a coefficient selection module, a virtual speaker selection module and an encoding module.
  • the coefficient selection module is used to obtain the fourth number of coefficients of the current frame of the three-dimensional audio signal, and the frequency domain characteristic value of the fourth number of coefficients; the coefficient selection module is also used to obtain the fourth number of coefficients according to the Frequency domain feature value, select a third number of representative coefficients from the fourth number of coefficients, the third number is less than the fourth number; the virtual speaker selection module is used to select from the candidate virtual speaker set according to the third number of representative coefficients Selecting a second number of representative virtual speakers of the current frame; the encoding module is configured to encode the current frame according to the second number of representative virtual speakers of the current frame to obtain a code stream.
  • These modules can perform the corresponding functions in the method example of the first aspect above. For details, refer to the detailed description in the method example, and details are not repeated here.
  • the present application provides an encoder, which includes at least one processor and a memory, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the first Operation steps of the three-dimensional audio signal encoding method in one aspect or any possible implementation manner of the first aspect.
  • the present application provides a system, the system includes the encoder as described in the third aspect, and a decoder, the encoder is used to perform the three-dimensional audio in the first aspect or any possible implementation manner of the first aspect In the operation steps of the signal encoding method, the decoder is used to decode the code stream generated by the encoder.
  • the present application provides a computer-readable storage medium, including: computer software instructions; when the computer software instructions are run in the encoder, the encoder is made to perform any possible implementation of the first aspect or the first aspect Operational steps of the method described in the method.
  • the present application provides a computer program product.
  • the encoder is made to perform the operation steps of the method described in the first aspect or any possible implementation manner of the first aspect. .
  • FIG. 1 is a schematic structural diagram of an audio codec system provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of a scene of an audio codec system provided by an embodiment of the present application
  • FIG. 3 is a schematic structural diagram of an encoder provided in an embodiment of the present application.
  • FIG. 4 is a schematic flowchart of a method for encoding and decoding a three-dimensional audio signal provided in an embodiment of the present application
  • FIG. 5 is a schematic flowchart of a method for selecting a virtual speaker provided by an embodiment of the present application
  • FIG. 6 is a schematic flowchart of a method for encoding a three-dimensional audio signal provided in an embodiment of the present application
  • FIG. 7 is a schematic flowchart of a method for selecting representative coefficients of a three-dimensional audio signal provided by an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of a method for selecting a virtual speaker provided by an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of another method for selecting a virtual speaker provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of another method for selecting a virtual speaker provided by the embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of a three-dimensional audio signal encoding device provided by the present application.
  • FIG. 12 is a schematic structural diagram of an encoder provided by the present application.
  • Sound is a continuous wave produced by the vibration of an object. Objects that vibrate to emit sound waves are called sound sources. When sound waves propagate through a medium (such as air, solid or liquid), the auditory organs of humans or animals can perceive sound.
  • a medium such as air, solid or liquid
  • Characteristics of sound waves include pitch, intensity, and timbre.
  • Pitch indicates how high or low a sound is.
  • Pitch intensity indicates the volume of a sound.
  • Pitch intensity can also be called loudness or volume.
  • the unit of sound intensity is decibel (decibel, dB). Timbre is also called fret.
  • the frequency of sound waves determines the pitch of the sound. The higher the frequency, the higher the pitch.
  • the number of times an object vibrates within one second is called frequency, and the unit of frequency is hertz (Hz).
  • the frequency of sound that can be recognized by the human ear is between 20Hz and 20000Hz.
  • the amplitude of the sound wave determines the intensity of the sound. The greater the amplitude, the greater the sound intensity. The closer the distance to the sound source, the greater the sound intensity.
  • the waveform of the sound wave determines the timbre.
  • the waveforms of sound waves include square waves, sawtooth waves, sine waves, and pulse waves.
  • sounds can be divided into regular sounds and irregular sounds.
  • Random sound refers to the sound produced by the sound source vibrating randomly. Random sounds are, for example, noises that affect people's work, study, and rest.
  • a regular sound refers to a sound produced by a sound source vibrating regularly. Regular sounds include speech and musical tones.
  • regular sound is an analog signal that changes continuously in the time-frequency domain. This analog signal may be referred to as an audio signal.
  • An audio signal is an information carrier that carries speech, music and sound effects.
  • the human auditory system Since the human auditory system has the ability to distinguish the location and distribution of sound sources in space, when the listener hears the sound in the space, he can not only feel the pitch, intensity and timbre of the sound, but also feel the direction of the sound. .
  • Three-dimensional audio technology refers to the assumption that the space outside the human ear is a system, and the signal received at the eardrum is a three-dimensional audio signal that is output by filtering the sound from the sound source through a system outside the ear.
  • a system other than the human ear can be defined as a system impulse response h(n)
  • any sound source can be defined as x(n)
  • the signal received at the eardrum is the convolution result of x(n) and h(n) .
  • the three-dimensional audio signal described in the embodiment of the present application may refer to a higher order ambisonics (higher order ambisonics, HOA) signal.
  • Three-dimensional audio can also be called three-dimensional audio, spatial audio, three-dimensional sound field reconstruction, virtual 3D audio, or binaural audio.
  • the sound pressure p satisfies formula (1), is the Laplacian operator.
  • the space system outside the human ear is a sphere, and the listener is at the center of the sphere, the sound from outside the sphere has a projection on the sphere, and the sound outside the sphere is filtered out.
  • the sound source is distributed on the sphere, use the sphere
  • the sound field generated by the above sound source is used to fit the sound field generated by the original sound source, that is, the three-dimensional audio technology is a method of fitting the sound field.
  • the formula (1) equation is solved in the spherical coordinate system, and in the passive spherical region, the solution of the formula (1) is the following formula (2).
  • r represents the radius of the ball
  • ⁇ represents the horizontal angle
  • k represents the wave number
  • s represents the amplitude of the ideal plane wave
  • m represents the order number of the three-dimensional audio signal (or the order number of the HOA signal).
  • represents ⁇ The spherical harmonics of the direction, Spherical harmonics representing the direction of the sound source.
  • the three-dimensional audio signal coefficients satisfy formula (3).
  • formula (3) can be transformed into formula (4).
  • N is an integer greater than or equal to 1.
  • the value of N is an integer ranging from 2 to 6.
  • the coefficients of the 3D audio signal described in the embodiments of the present application may refer to HOA coefficients or ambient stereo (ambisonic) coefficients.
  • the three-dimensional audio signal is an information carrier carrying the spatial position information of the sound source in the sound field, and describes the sound field of the listener in the space.
  • Formula (4) shows that the sound field can be expanded on the spherical surface according to the spherical harmonic function, that is, the sound field can be decomposed into the superposition of multiple plane waves. Therefore, the sound field described by the three-dimensional audio signal can be expressed by the superposition of multiple plane waves, and the sound field can be reconstructed through the coefficients of the three-dimensional audio signal.
  • the HOA signal includes a large amount of data for describing the spatial information of the sound field. If the acquisition device (such as a microphone) transmits the three-dimensional audio signal to a playback device (such as a speaker), a large bandwidth needs to be consumed.
  • the encoder can use spatial squeezed surround audio coding (spatial squeezed surround audio coding, S3AC) or directional audio coding (directional audio coding, DirAC) to compress and code the 3D audio signal to obtain a code stream, and transmit the code stream to the playback device.
  • the playback device decodes the code stream, reconstructs the three-dimensional audio signal, and plays the reconstructed three-dimensional audio signal. Therefore, the amount of data transmitted to the playback device and the bandwidth occupation of the three-dimensional audio signal are reduced.
  • the computational complexity of compressing and encoding the three-dimensional audio signal by the encoder is relatively high, which occupies too much computing resources of the encoder. Therefore, how to reduce the computational complexity of compressing and encoding 3D audio signals is an urgent problem to be solved.
  • the embodiment of the present application provides an audio coding and decoding technology, especially a three-dimensional audio coding and decoding technology for three-dimensional audio signals, and specifically provides a coding and decoding technology that uses fewer channels to represent three-dimensional audio signals, so as to improve the traditional audio codec system.
  • Audio coding (or commonly referred to as coding) includes two parts of audio coding and audio decoding. Audio encoding is performed on the source side and typically involves processing (eg, compressing) raw audio to reduce the amount of data needed to represent the raw audio for more efficient storage and/or transmission. Audio decoding is performed at the destination and usually involves inverse processing relative to the encoder to reconstruct the original audio. The encoding part and the decoding part are also collectively referred to as codec.
  • FIG. 1 is a schematic structural diagram of an audio codec system provided by an embodiment of the present application.
  • the audio codec system 100 includes a source device 110 and a destination device 120 .
  • the source device 110 is configured to compress and encode the 3D audio signal to obtain a code stream, and transmit the code stream to the destination device 120 .
  • the destination device 120 decodes the code stream, reconstructs the 3D audio signal, and plays the reconstructed 3D audio signal.
  • the source device 110 includes an audio acquirer 111 , a preprocessor 112 , an encoder 113 and a communication interface 114 .
  • the audio acquirer 111 is used to acquire original audio.
  • Audio acquirer 111 may be any type of audio capture device for capturing real world sounds, and/or any type of audio generation device.
  • the audio acquirer 111 is, for example, a computer audio processor for generating computer audio.
  • the audio fetcher 111 can also be any type of memory or storage that stores audio. Audio includes real world sounds, virtual scene (eg: VR or augmented reality (augmented reality, AR)) sounds and/or any combination thereof.
  • the preprocessor 112 is configured to receive the original audio collected by the audio acquirer 111, and perform preprocessing on the original audio to obtain a three-dimensional audio signal.
  • the preprocessing performed by the preprocessor 112 includes channel conversion, audio format conversion, or denoising.
  • the encoder 113 is configured to receive the 3D audio signal generated by the preprocessor 112, and compress and encode the 3D audio signal to obtain a code stream.
  • the encoder 113 may include a spatial encoder 1131 and a core encoder 1132 .
  • the spatial encoder 1131 is configured to select (or search for) a virtual speaker from the candidate virtual speaker set according to the 3D audio signal, and generate a virtual speaker signal according to the 3D audio signal and the virtual speaker.
  • the virtual speaker signal may also be referred to as a playback signal.
  • the core encoder 1132 is used to encode the virtual speaker signal to obtain a code stream.
  • the communication interface 114 is used to receive the code stream generated by the encoder 113, and send the code stream to the destination device 120 through the communication channel 130, so that the destination device 120 reconstructs a 3D audio signal according to the code stream.
  • the destination device 120 includes a player 121 , a post-processor 122 , a decoder 123 and a communication interface 124 .
  • the communication interface 124 is configured to receive the code stream sent by the communication interface 114 and transmit the code stream to the decoder 123 . So that the decoder 123 reconstructs the 3D audio signal according to the code stream.
  • the communication interface 114 and the communication interface 124 can be used to pass through a direct communication link between the source device 110 and the destination device 120, such as a direct wired or wireless connection, etc., or through any type of network, such as a wired network, a wireless network, or any other Combination, any type of private network and public network or any combination thereof, send or receive raw audio related data.
  • Both the communication interface 114 and the communication interface 124 can be configured as a one-way communication interface as indicated by an arrow pointing from the source device 110 to the corresponding communication channel 130 of the destination device 120 in Figure 1, or a two-way communication interface, and can be used to send and receive messages etc., to establish the connection, confirm and exchange any other information related to the communication link and/or data transmission, such as encoded code stream transmission, etc.
  • the decoder 123 is used to decode the code stream and reconstruct the 3D audio signal.
  • the decoder 123 includes a core decoder 1231 and a spatial decoder 1232 .
  • the core decoder 1231 is used to decode the code stream to obtain the virtual speaker signal.
  • the spatial decoder 1232 is configured to reconstruct a 3D audio signal according to the candidate virtual speaker set and the virtual speaker signal to obtain a reconstructed 3D audio signal.
  • the post-processor 122 is configured to receive the reconstructed 3D audio signal generated by the decoder 123, and perform post-processing on the reconstructed 3D audio signal.
  • the post-processing performed by the post-processor 122 includes audio rendering, loudness normalization, user interaction, audio format conversion or denoising, and the like.
  • the player 121 is configured to play the reconstructed sound according to the reconstructed 3D audio signal.
  • the audio acquirer 111 and the encoder 113 may be integrated on one physical device, or may be set on different physical devices, which is not limited.
  • the source device 110 shown in FIG. 1 includes an audio acquirer 111 and an encoder 113, which means that the audio acquirer 111 and the encoder 113 are integrated on one physical device, and the source device 110 may also be called an acquisition device.
  • the source device 110 is, for example, a media gateway of a wireless access network, a media gateway of a core network, a transcoding device, a media resource server, an AR device, a VR device, a microphone, or other audio collection devices. If the source device 110 does not include the audio acquirer 111, it means that the audio acquirer 111 and the encoder 113 are two different physical devices, and the source device 110 can obtain the original audio from other devices (such as: collecting audio devices or storing audio devices).
  • the player 121 and the decoder 123 may be integrated on one physical device, or may be set on different physical devices, which is not limited.
  • the destination device 120 shown in FIG. 1 includes a player 121 and a decoder 123, indicating that the player 121 and the decoder 123 are integrated on one physical device, and the destination device 120 can also be called a playback device, and the destination device 120 Has functions to decode and play reconstructed audio.
  • the destination device 120 is, for example, a speaker, an earphone or other devices for playing audio. If the destination device 120 does not include the player 121, it means that the player 121 and the decoder 123 are two different physical devices.
  • the destination device 120 After the destination device 120 decodes the code stream and reconstructs the 3D audio signal, it transmits the reconstructed 3D audio signal to other playback devices. (such as speakers or earphones), the reconstructed three-dimensional audio signal is played back by other playback devices.
  • other playback devices such as speakers or earphones
  • FIG. 1 shows that the source device 110 and the destination device 120 may be integrated on one physical device, or may be set on different physical devices, which is not limited.
  • the source device 110 may be a microphone in a recording studio, and the destination device 120 may be a speaker.
  • the source device 110 can collect the original audio of various musical instruments, transmit the original audio to the codec device, and the codec device performs codec processing on the original audio to obtain a reconstructed 3D audio signal, and the destination device 120 plays back the reconstructed 3D audio signal.
  • the source device 110 may be a microphone in the terminal device, and the destination device 120 may be an earphone.
  • the source device 110 may collect external sounds or audio synthesized by the terminal device.
  • the source device 110 and the destination device 120 are integrated in a virtual reality (virtual reality, VR) device, an augmented reality (Augmented Reality, AR) device, a mixed reality (Mixed Reality, MR) devices or Extended Reality (XR) devices, VR/AR/MR/XR devices have the functions of collecting original audio, playing back audio, and encoding and decoding.
  • the source device 110 can collect the sound made by the user and the sound made by the virtual objects in the virtual environment where the user is located.
  • the source device 110 or its corresponding function and the destination device 120 or its corresponding function may be implemented using the same hardware and/or software or by separate hardware and/or software or any combination thereof. According to the description, the existence and division of different units or functions in the source device 110 and/or the destination device 120 shown in FIG. 1 may vary according to actual devices and applications, which is obvious to a skilled person.
  • the audio codec system may also include other devices.
  • the audio codec system may also include device-side devices or cloud-side devices. After the source device 110 collects the original audio, it preprocesses the original audio to obtain a three-dimensional audio signal; and transmits the three-dimensional audio to the end-side device or the cloud-side device, and the end-side device or the cloud-side device realizes the encoding of the three-dimensional audio signal function to decode.
  • the encoder 300 includes a virtual speaker configuration unit 310 , a virtual speaker set generation unit 320 , an encoding analysis unit 330 , a virtual speaker selection unit 340 , a virtual speaker signal generation unit 350 and an encoding unit 360 .
  • the virtual speaker configuration unit 310 is configured to generate virtual speaker configuration parameters according to the encoder configuration information, so as to obtain multiple virtual speakers.
  • the encoder configuration information includes but is not limited to: the order of the 3D audio signal (or generally referred to as the HOA order), encoding bit rate, user-defined information, and so on.
  • the virtual speaker configuration parameters include but are not limited to: the number of virtual speakers, the order of the virtual speakers, the position coordinates of the virtual speakers, and so on.
  • the number of virtual speakers is, for example, 2048, 1669, 1343, 1024, 530, 512, 256, 128, or 64.
  • the order of the virtual loudspeaker can be any one of 2nd order to 6th order.
  • the position coordinates of the virtual loudspeaker include horizontal angle and pitch angle.
  • the virtual speaker configuration parameters output by the virtual speaker configuration unit 310 are used as the input of the virtual speaker set generation unit 320 .
  • the virtual speaker set generating unit 320 is configured to generate a candidate virtual speaker set according to virtual speaker configuration parameters, and the candidate virtual speaker set includes a plurality of virtual speakers. Specifically, the virtual speaker set generation unit 320 determines a plurality of virtual speakers included in the candidate virtual speaker set according to the number of virtual speakers, and determines the coefficients of the virtual speakers according to the position information (such as: coordinates) of the virtual speakers and the order of the virtual speakers .
  • the method for determining the coordinates of the virtual speakers includes, but is not limited to: generating multiple virtual speakers according to the equidistant rule, or generating a plurality of virtual speakers with non-uniform distribution according to the principle of auditory perception; and then, generating the virtual speakers according to the number of virtual speakers coordinate.
  • the coefficients of the virtual speaker can also be generated according to the above-mentioned generation principle of the three-dimensional audio signal. Put ⁇ s in formula (3) and are respectively set as the position coordinates of the virtual speakers, Indicates the coefficients of the virtual speaker of order N.
  • the coefficients of the virtual speakers may also be referred to as ambisonics coefficients.
  • the encoding analysis unit 330 is used for encoding and analyzing the 3D audio signal, for example, analyzing the sound field distribution characteristics of the 3D audio signal, that is, the number of sound sources, the directionality of the sound source, and the dispersion of the sound source of the 3D audio signal.
  • the coefficients of multiple virtual speakers included in the candidate virtual speaker set output by the virtual speaker set generation unit 320 are used as the input of the virtual speaker selection unit 340 .
  • the sound field distribution characteristics of the three-dimensional audio signal output by the encoding analysis unit 330 are used as the input of the virtual speaker selection unit 340 .
  • the virtual speaker selection unit 340 is configured to determine a representative virtual speaker matching the 3D audio signal according to the 3D audio signal to be encoded, the sound field distribution characteristics of the 3D audio signal, and the coefficients of multiple virtual speakers.
  • the encoder 300 in this embodiment of the present application may not include the encoding analysis unit 330, that is, the encoder 300 may not analyze the input signal, and the virtual speaker selection unit 340 uses a default configuration to determine the representative virtual speaker.
  • the virtual speaker selection unit 340 determines a representative virtual speaker matching the 3D audio signal only according to the 3D audio signal and the coefficients of the plurality of virtual speakers.
  • the encoder 300 may use the 3D audio signal obtained from the acquisition device or the 3D audio signal synthesized by using artificial audio objects as the input of the encoder 300 .
  • the 3D audio signal input by the encoder 300 may be a time domain 3D audio signal or a frequency domain 3D audio signal, which is not limited.
  • the position information representing the virtual speaker and the coefficient representing the virtual speaker output by the virtual speaker selection unit 340 serve as inputs to the virtual speaker signal generation unit 350 and the encoding unit 360 .
  • the virtual speaker signal generating unit 350 is configured to generate a virtual speaker signal according to the three-dimensional audio signal and attribute information representing the virtual speaker.
  • the attribute information representing the virtual speaker includes at least one of position information representing the virtual speaker, coefficients representing the virtual speaker, and coefficients of a three-dimensional audio signal. If the attribute information is the position information representing the virtual speaker, determine the coefficient representing the virtual speaker according to the position information representing the virtual speaker; if the attribute information includes the coefficient of the three-dimensional audio signal, obtain the coefficient representing the virtual speaker according to the coefficient of the three-dimensional audio signal.
  • the virtual speaker signal generation unit 350 calculates the virtual speaker signal according to the coefficients of the 3D audio signal and the coefficients representing the virtual speaker.
  • matrix A represents the coefficients of virtual speakers
  • matrix X represents the HOA coefficients of the HOA signal.
  • Matrix X is the inverse of matrix A.
  • w represents the virtual speaker signal.
  • the virtual loudspeaker signal satisfies formula (5).
  • a -1 represents the inverse matrix of matrix A.
  • the size of the matrix A is (M ⁇ C)
  • C represents the number of virtual speakers
  • M represents the number of channels of the N-order HOA signal
  • a represents the coefficient of the virtual speaker
  • the size of the matrix X is (M ⁇ L)
  • L represents the number of coefficients of the HOA signal
  • x represents the coefficient of the HOA signal.
  • the coefficients representing virtual speakers may refer to HOA coefficients representing virtual speakers or ambisonics coefficients representing virtual speakers.
  • the virtual speaker signal output by the virtual speaker signal generating unit 350 serves as an input of the encoding unit 360 .
  • the encoding unit 360 is configured to perform core encoding processing on the virtual speaker signal to obtain a code stream.
  • Core encoding processing includes but not limited to: transformation, quantization, psychoacoustic model, noise shaping, bandwidth extension, downmixing, arithmetic coding, code stream generation, etc.
  • the spatial encoder 1131 may include a virtual speaker configuration unit 310, a virtual speaker set generation unit 320, a coding analysis unit 330, a virtual speaker selection unit 340, and a virtual speaker signal generation unit 350, that is, the virtual speaker configuration unit 310, the virtual The speaker set generation unit 320 , the encoding analysis unit 330 , the virtual speaker selection unit 340 and the virtual speaker signal generation unit 350 realize the function of the spatial encoder 1131 .
  • the core encoder 1132 may include an encoding unit 360 , that is, the encoding unit 360 implements the functions of the core encoder 1132 .
  • the encoder shown in Figure 3 can generate one virtual speaker signal or multiple virtual speaker signals. Multiple virtual speaker signals can be obtained by multiple executions of the encoder shown in FIG. 3 , or can be obtained by one execution of the encoder shown in FIG. 3 .
  • FIG. 4 is a schematic flowchart of a method for encoding and decoding a three-dimensional audio signal provided by an embodiment of the present application.
  • the process of encoding and decoding a 3D audio signal performed by the source device 110 and the destination device 120 in FIG. 1 is taken as an example for illustration.
  • the method includes the following steps.
  • the source device 110 acquires a current frame of a three-dimensional audio signal.
  • the source device 110 can collect original audio through the audio acquirer 111 .
  • the source device 110 may also receive the original audio collected by other devices; or obtain the original audio from the storage in the source device 110 or other storages.
  • the original audio may include at least one of real-world sounds collected in real time, audio stored by the device, and audio synthesized from multiple audios. This embodiment does not limit the way of acquiring the original audio and the type of the original audio.
  • the source device 110 After acquiring the original audio, the source device 110 generates a three-dimensional audio signal according to the three-dimensional audio technology and the original audio, so as to provide the listener with an "immersive" sound effect when playing back the original audio.
  • a specific method of generating a three-dimensional audio signal reference may be made to the description of the preprocessor 112 in the foregoing embodiment and the description of the prior art.
  • the audio signal is a continuous analog signal.
  • the audio signal can be sampled first to generate a frame sequence digital signal.
  • a frame can consist of multiple samples.
  • a frame may also refer to sample points obtained by sampling.
  • a frame may also include subframes obtained by dividing the frame.
  • a frame may also refer to subframes obtained by dividing a frame. For example, a frame with a length of L sampling points is divided into N subframes, and each subframe corresponds to L/N sampling points.
  • Audio coding and decoding generally refers to processing a sequence of audio frames containing multiple sample points.
  • An audio frame may include a current frame or a previous frame.
  • the current frame or previous frame described in various embodiments of the present application may refer to a frame or a subframe.
  • the current frame refers to a frame that undergoes codec processing at the current moment.
  • the previous frame refers to a frame that has undergone codec processing at a time before the current time.
  • the previous frame may be a frame at a time before the current time or at multiple times before.
  • the current frame of the 3D audio signal refers to a frame of 3D audio signal that undergoes codec processing at the current moment.
  • the previous frame refers to a frame of 3D audio signal that has undergone codec processing at a time before the current time.
  • the current frame of the 3D audio signal may refer to the current frame of the 3D audio signal to be encoded.
  • the current frame of the 3D audio signal may be referred to as the current frame for short.
  • the previous frame of the 3D audio signal may be simply referred to as the previous frame.
  • the source device 110 determines a candidate virtual speaker set.
  • the source device 110 has a set of candidate virtual speakers pre-configured in its memory.
  • Source device 110 may read the set of candidate virtual speakers from memory.
  • the set of candidate virtual speakers includes a plurality of virtual speakers.
  • the virtual speakers represent speakers that virtually exist in the spatial sound field.
  • the virtual speaker is used to calculate a virtual speaker signal according to the 3D audio signal, so that the destination device 120 plays back the reconstructed 3D audio signal.
  • virtual speaker configuration parameters are pre-configured in the memory of the source device 110 .
  • Source device 110 generates a set of candidate virtual speakers based on virtual speaker configuration parameters.
  • the source device 110 generates a set of candidate virtual speakers in real time according to its own computing resource (eg, processor) capability and characteristics of the current frame (eg, channel and data volume).
  • the source device 110 selects a representative virtual speaker of the current frame from the candidate virtual speaker set according to the current frame of the three-dimensional audio signal.
  • the source device 110 votes for the virtual speaker according to the coefficient of the current frame and the coefficient of the virtual speaker, and selects the representative virtual speaker of the current frame from the set of candidate virtual speakers according to the voting value of the virtual speaker.
  • a limited number of representative virtual speakers of the current frame are searched from the set of candidate virtual speakers as the best matching virtual speakers of the current frame to be encoded, so as to achieve the purpose of data compression on the 3D audio signal to be encoded.
  • FIG. 5 is a schematic flowchart of a method for selecting a virtual speaker provided by an embodiment of the present application.
  • the method flow described in FIG. 5 is an illustration of the specific operation process included in S430 in FIG. 4 .
  • the process of selecting a virtual speaker performed by the encoder 113 in the source device 110 shown in FIG. 1 is taken as an example for illustration.
  • the function of the virtual speaker selection unit 340 As shown in Figure 5, the method includes the following steps.
  • the encoder 113 acquires representative coefficients of the current frame.
  • the representative coefficient may refer to a frequency domain representative coefficient or a time domain representative coefficient.
  • the representative coefficients in the frequency domain may also be referred to as representative frequency points in the frequency domain or representative coefficients in the frequency spectrum.
  • the time-domain representative coefficients may also be referred to as time-domain representative sampling points.
  • the encoder 113 selects the representative virtual speaker of the current frame from the candidate virtual speaker set according to the voting value of the representative coefficient of the current frame for the virtual speakers in the candidate virtual speaker set. Execute S440 to S460.
  • the encoder 113 votes for the virtual speakers in the candidate virtual speaker set according to the representative coefficient of the current frame and the coefficient of the virtual speaker, and selects (searches) the representative virtual speaker of the current frame from the candidate virtual speaker set according to the final voting value of the current frame of the virtual speaker. speaker.
  • searches searches the representative virtual speaker of the current frame from the candidate virtual speaker set according to the final voting value of the current frame of the virtual speaker. speaker.
  • the encoder first traverses the virtual speakers contained in the candidate virtual speaker set, and uses the representative virtual speaker of the current frame selected from the candidate virtual speaker set to compress the current frame.
  • the results of virtual speakers selected in consecutive frames are quite different, the sound image of the reconstructed 3D audio signal will be unstable, and the sound quality of the reconstructed 3D audio signal will be reduced.
  • the encoder 113 can update the initial voting value of the current frame of the virtual speaker contained in the candidate virtual speaker set according to the final voting value of the previous frame representing the virtual speaker in the previous frame, and obtain the virtual speaker’s
  • the final voting value of the current frame is to select the representative virtual speaker of the current frame from the set of candidate virtual speakers according to the final voting value of the current frame of the virtual speaker. Therefore, by referring to the representative virtual speaker of the previous frame to select the representative virtual speaker of the current frame, the encoder tends to select the same virtual speaker as the representative virtual speaker of the previous frame when selecting the representative virtual speaker of the current frame for the current frame, The continuity of orientation between consecutive frames is increased, which overcomes the problem that the results of virtual speakers selected in consecutive frames are quite different. Therefore, the embodiment of the present application may also include S530.
  • the encoder 113 adjusts the initial voting value of the current frame of the virtual speaker in the candidate virtual speaker set according to the final voting value of the previous frame representing the virtual speaker in the previous frame, and obtains the final voting value of the current frame of the virtual speaker.
  • the encoder 113 votes for the virtual speakers in the candidate virtual speaker set according to the representative coefficient of the current frame and the coefficient of the virtual speaker, and after obtaining the initial voting value of the current frame of the virtual speaker, according to the previous frame representing the virtual speaker in the previous frame, the final The voting value adjusts the initial voting value of the current frame of the virtual speaker in the candidate virtual speaker set to obtain the final voting value of the current frame of the virtual speaker.
  • the representative virtual speaker of the previous frame is the virtual speaker used by the encoder 113 when encoding the previous frame.
  • the encoder 113 if the current frame is the first frame in the original audio, the encoder 113 performs S510 to S520. If the current frame is any frame above the second frame in the original audio, the encoder 113 can first judge whether to reuse the representative virtual speaker of the previous frame to encode the current frame or judge whether to search for a virtual speaker to ensure The continuity of the orientation and reduce the coding complexity.
  • the embodiment of the present application may also include S540.
  • the encoder 113 judges whether to perform virtual speaker search according to the representative virtual speaker of the previous frame and the current frame.
  • the encoder 113 may execute S510 first, that is, the encoder 113 obtains the representative coefficient of the current frame, and the encoder 113 judges whether to perform virtual speaker search according to the representative coefficient of the current frame and the coefficient representing the virtual speaker of the previous frame, if The encoder 113 determines to perform virtual speaker search, and then executes S520 to S530.
  • the encoder 113 determines to multiplex the representative virtual speaker of the previous frame to encode the current frame.
  • the encoder 113 multiplexes the representative virtual speaker of the previous frame and the current frame to generate a virtual speaker signal, encodes the virtual speaker signal to obtain a code stream, and sends the code stream to the destination device 120, that is, executes S450 and S460.
  • the source device 110 generates a virtual speaker signal according to the current frame of the 3D audio signal and the representative virtual speaker of the current frame.
  • the source device 110 generates a virtual speaker signal according to the coefficients of the current frame and the coefficients representing the virtual speaker of the current frame.
  • a virtual speaker signal For a specific method of generating a virtual speaker signal, reference may be made to the prior art and the description of the virtual speaker signal generating unit 350 in the foregoing embodiments.
  • the source device 110 encodes the virtual speaker signal to obtain a code stream.
  • the source device 110 may perform coding operations such as transformation or quantization on the virtual speaker signal to generate a code stream, so as to achieve the purpose of data compression on the 3D audio signal to be coded.
  • coding operations such as transformation or quantization on the virtual speaker signal to generate a code stream, so as to achieve the purpose of data compression on the 3D audio signal to be coded.
  • the source device 110 sends the code stream to the destination device 120.
  • the source device 110 may send the code stream of the original audio to the destination device 120 after all encoding of the original audio is completed.
  • the source device 110 may also encode the 3D audio signal in real time in units of frames, and send a code stream of one frame after encoding one frame.
  • code streams For a specific method of sending code streams, reference may be made to the prior art and the descriptions of the communication interface 114 and the communication interface 124 in the foregoing embodiments.
  • the destination device 120 decodes the code stream sent by the source device 110, reconstructs a 3D audio signal, and obtains a reconstructed 3D audio signal.
  • the destination device 120 After receiving the code stream, the destination device 120 decodes the code stream to obtain a virtual speaker signal, and then reconstructs a three-dimensional audio signal according to the candidate virtual speaker set and the virtual speaker signal to obtain a reconstructed three-dimensional audio signal.
  • the destination device 120 plays back the reconstructed 3D audio signal.
  • the destination device 120 transmits the reconstructed 3D audio signal to other playback devices, and the reconstructed 3D audio signal is played by other playback devices, so that the listener is placed in an "immersive" experience in places such as theaters, concert halls, or virtual scenes. The sound effect is more realistic.
  • each coefficient of the 3D audio signal needs to be correlated with the coefficient of each virtual speaker, and the encoding
  • the controller poses a heavy computational burden.
  • An embodiment of the present application provides a method for selecting coefficients of a three-dimensional audio signal.
  • the encoder uses the representative coefficient of the three-dimensional audio signal to perform a correlation operation with the coefficient of each virtual speaker to select a representative virtual speaker, thereby reducing the computational complexity of the encoder searching for a virtual speaker. .
  • FIG. 6 is a schematic flowchart of a method for encoding a three-dimensional audio signal provided by an embodiment of the present application.
  • the process of selecting coefficients of a three-dimensional audio signal performed by the encoder 113 in the source device 110 in FIG. 1 is taken as an example for illustration. Specifically realize the function of the virtual speaker selection unit 340 .
  • the method flow described in FIG. 6 is an illustration of the specific operation process included in S510 in FIG. 5 . As shown in Fig. 6, the method includes the following steps.
  • the encoder 113 acquires a fourth number of coefficients of the current frame of the 3D audio signal, and frequency-domain feature values of the fourth number of coefficients.
  • the encoder 113 may sample the current frame of the HOA signal to obtain L ⁇ (N+1) 2 sampling points, that is, obtain the fourth number of coefficients.
  • N represents the order of the HOA signal. For example, assuming that the duration of the current frame of the HOA signal is 20 milliseconds, the encoder 113 samples the current frame at a frequency of 48 KHz to obtain 960 ⁇ (N+1) 2 sampling points in the time domain. Sampling points may also be referred to as time-domain coefficients.
  • the frequency domain coefficients of the current frame of the 3D audio signal may be obtained by performing time-frequency conversion according to the time domain coefficients of the current frame of the 3D audio signal.
  • the method for transforming the time domain into the frequency domain is not limited.
  • the method of transforming the time domain into the frequency domain is, for example, Modified Discrete Cosine Transform (MDCT), and then 960 ⁇ (N+1) 2 frequency domain coefficients in the frequency domain can be obtained.
  • Frequency domain coefficients may also be referred to as spectral coefficients or frequency bins.
  • the encoder 113 selects a third number of representative coefficients from the fourth number of coefficients according to the frequency-domain feature values of the fourth number of coefficients.
  • the encoder 113 divides the spectrum range indicated by the fourth number of coefficients into at least one subband. Wherein, the encoder 113 divides the spectrum range indicated by the fourth number of coefficients into a subband. It can be understood that the spectrum range of this subband is equal to the spectrum range indicated by the fourth number of coefficients, which is equivalent to the coder 113. The spectrum range indicated by the fourth number of coefficients is divided.
  • the encoder 113 divides the spectrum range indicated by the fourth number of coefficients into at least two subbands, in one case, the encoder 113 divides the spectrum range indicated by the fourth number of coefficients equally into at least two subbands, at least two Each of the subbands contains the same number of coefficients.
  • the encoder 113 performs unequal division on the spectrum range indicated by the fourth number of coefficients, and the number of coefficients contained in at least two subbands obtained by division is different, or each subband in the at least two subbands obtained by division
  • the number of coefficients included varies.
  • the encoder 113 may perform unequal division on the spectrum range indicated by the fourth number of coefficients according to the low frequency range, the middle frequency range and the high frequency range in the spectrum range indicated by the fourth number of coefficients, so that the low frequency range, the middle frequency range and the Each spectral range in the high frequency range includes at least one subband.
  • Each of the at least one subband in the low frequency range contains the same number of coefficients.
  • Each of the at least one subband in the intermediate frequency range contains the same number of coefficients.
  • Each subband of at least one subband in the high frequency range contains the same number of coefficients.
  • the subbands in the three spectral ranges of the low frequency range, the middle frequency range and the high frequency range may contain different numbers of coefficients.
  • the encoder 113 selects representative coefficients from at least one subband included in the spectrum range indicated by the fourth number of coefficients according to the frequency-domain feature values of the fourth number of coefficients to obtain a third number of representative coefficients.
  • the third number is smaller than the fourth number, and the fourth number of coefficients includes the third number of representative coefficients.
  • the method flow described in FIG. 7 is an illustration of the specific operation process included in S620 in FIG. 7 . As shown in Fig. 7, the method includes the following steps.
  • the encoder 113 selects Z representative coefficients from each subband according to frequency-domain feature values of coefficients in each subband of at least one subband, so as to obtain a third number of representative coefficients.
  • Z is a positive integer.
  • the encoder 113 selects Z representative coefficients from each subband according to the descending order of the frequency domain eigenvalues of the coefficients in each subband of at least one subband, and Z representative coefficients selected from each subband
  • the representative coefficients are composed to obtain a third number of representative coefficients.
  • the encoder 113 sorts the frequency-domain eigenvalues of the b(i) coefficients in the i-th subband from large to small, according to the order of the frequency-domain eigenvalues of the b(i) coefficients in the i-th sub-band
  • K(i) representative coefficients are selected starting from the coefficient of the largest frequency-domain eigenvalue in the i-th subband.
  • the value of K(i) can be preset, or can be generated according to predetermined rules, for example, starting from the coefficient of the largest frequency-domain eigenvalue in the i-th subband, the encoder 113 selects the frequency domain of the coefficient The 50% coefficients with the largest eigenvalues are used as representative coefficients.
  • the encoder 113 may first determine the weight of each subband in the at least two subbands, using The respective weights of each subband adjust the frequency-domain eigenvalues of the coefficients in each subband, and then select a third number of representative coefficients from at least two subbands.
  • S620 may further include the following steps.
  • the encoder 113 determines the respective weight of each subband according to the frequency domain feature value of the first candidate coefficient in each subband of the at least two subbands.
  • the first candidate coefficients may refer to partial coefficients within a subband.
  • the embodiment of the present application does not limit the number of first candidate coefficients, and the number of first candidate coefficients may be one or at least two.
  • the encoder 113 may select the first candidate coefficient according to the method described in S6201. Understandably, the encoder 113 selects Z representative coefficients from each subband according to the descending order of the frequency-domain feature values of the coefficients in each subband of at least two subbands, and takes the Z representative coefficients as each The first candidate coefficients of subbands. For example, at least two subbands include the first subband, and Z representative coefficients are selected from the first subband as first candidate coefficients of the first subband.
  • the encoder 113 determines the weight of the subband according to the frequency domain feature value of the first candidate coefficient in the subband and the frequency domain feature values of all the coefficients in the subband.
  • the encoder 113 calculates the weight w(i) of the i-th subband according to the frequency-domain feature values of the candidate coefficients of the i-th subband and the frequency-domain feature values of all coefficients of the i-th subband.
  • the weight w(i) of the i-th subband satisfies formula (6).
  • p represents the frequency domain eigenvalue of the coefficient of the current frame
  • K(i) represents the number of coefficients of the i-th subband
  • ai[j] represents the coefficient number of the j-th coefficient of the i-th sub-band
  • sfb[i] Indicates the starting coefficient number of the i-th sub-band
  • b(i) indicates the number of coefficients contained in the i-th sub-band
  • the encoder 113 respectively adjusts the frequency-domain feature values of the second candidate coefficients in each sub-band according to the respective weights of each sub-band, to obtain the adjusted frequency-domain feature values of the second candidate coefficients in each sub-band.
  • the second candidate coefficients may refer to partial coefficients within a subband.
  • the embodiment of the present application does not limit the number of second candidate coefficients, and the number of second candidate coefficients may be one or at least two.
  • the encoder 113 may select the second candidate coefficient according to the method described in S6201. Understandably, the encoder 113 selects Z representative coefficients from each subband according to the descending order of the frequency-domain feature values of the coefficients in each subband of at least two subbands, and uses the Z representative coefficients as each The second candidate coefficients for subbands. In this case, the numbers of the first candidate coefficients and the second candidate coefficients may be the same or different. For the first candidate coefficient and the second candidate coefficient in one subband, the first candidate coefficient and the second candidate coefficient may be the same coefficient or different coefficients.
  • the encoder 113 may adjust the frequency-domain feature values of some coefficients of each subband.
  • the second candidate coefficients may also refer to all coefficients in the subband.
  • the numbers of the first candidate coefficients and the second candidate coefficients are different. Understandably, the encoder 113 adjusts the frequency-domain feature values of all coefficients in each subband.
  • the encoder 113 adjusts the frequency-domain feature values of the K(i) coefficients of the i-th subband according to the weight w(i) of the i-th sub-band, and the adjusted K(i) coefficients of the i-th subband
  • the frequency-domain eigenvalues satisfy formula (7).
  • P(a i [j]) represents the frequency-domain eigenvalue corresponding to the j-th coefficient of the i-th subband
  • P′(a i [j]) represents the adjusted frequency domain corresponding to the j-th coefficient of the i-th sub-band Eigenvalue
  • K(i) represents the number of coefficients of the i-th subband
  • ai[j] represents the coefficient number of the j-th coefficient of the i-th sub-band
  • w(i) represents the weight of the i-th sub-band
  • the encoder 113 determines a third number of represent coefficients.
  • the encoder 113 sorts the frequency-domain eigenvalues of all coefficients in at least two subbands from large to small, according to the order of frequency-domain eigenvalues of all coefficients in at least two subbands from large to small, from at least two Starting with the coefficient of the largest frequency-domain eigenvalue in subbands, select the third number of representative coefficients.
  • the frequency-domain eigenvalues of all the coefficients in at least two sub-bands include the adjusted frequency-domain eigenvalues of the second candidate coefficients, and at least two sub-bands except Frequency-domain eigenvalues of coefficients other than the second candidate coefficient.
  • the encoder 113 determines a third number of representative coefficients according to the adjusted frequency-domain eigenvalues of the second candidate coefficients in at least two subbands and the frequency-domain eigenvalues of coefficients other than the second candidate coefficients in at least two subbands .
  • the frequency-domain feature values of all the coefficients in at least two sub-bands are adjusted frequency-domain feature values of the second candidate coefficients.
  • the encoder 113 determines a third number of representative coefficients according to the adjusted frequency-domain feature values of the second candidate coefficients in at least two subbands.
  • the third number may be preset or generated according to predetermined rules. For example, the encoder 113 selects 20% coefficients with the largest frequency-domain characteristic values of all coefficients in at least two subbands as representative frequency points.
  • the encoder 113 selects a second number of representative virtual speakers of the current frame from the candidate virtual speaker set according to the third number of representative coefficients.
  • the encoder 113 uses the third number of representative coefficients of the current frame of the 3D audio signal to perform a correlation operation with the coefficients of each virtual speaker in the candidate virtual speaker set, and selects the second number of representative virtual speakers of the current frame.
  • the encoder selects some coefficients from all the coefficients of the current frame as representative coefficients, and uses a smaller number of representative coefficients to replace all the coefficients of the current frame to select a representative virtual speaker from the candidate virtual speaker set, thus effectively reducing the number of virtual speakers that the encoder searches for.
  • the computational complexity of the loudspeaker is reduced, thereby reducing the computational complexity of compressing and encoding the three-dimensional audio signal and reducing the computational burden of the encoder.
  • a frame of N-order HOA signal has 960 (N+1) 2 coefficients, and this embodiment can select the first 10% of the coefficients to participate in the virtual speaker search.
  • the encoding complexity is compared with that of the full coefficients participating in the virtual speaker search. Coding complexity is reduced by 90%.
  • the encoder 113 encodes the current frame according to the second number of representative virtual speakers of the current frame to obtain a code stream.
  • the encoder 113 generates a virtual speaker signal according to the second number of representative virtual speakers of the current frame and the current frame, and encodes the virtual speaker signal to obtain a code stream.
  • a specific method of generating a code stream reference may be made to the prior art and the descriptions of the encoding unit 360 and S450 in the foregoing embodiments.
  • the encoder 113 After the encoder 113 generates the code stream, it sends the code stream to the destination device 120, so that the destination device 120 decodes the code stream sent by the source device 110, reconstructs the 3D audio signal, and obtains the reconstructed 3D audio signal.
  • the encoder Since the frequency-domain eigenvalues of the coefficients of the current frame characterize the sound field characteristics of the three-dimensional audio signal, the encoder selects the representative coefficients of the representative sound field components of the current frame according to the frequency-domain eigenvalues of the coefficients of the current frame, and uses the representative coefficients from the candidate virtual
  • the representative virtual speaker of the current frame selected in the speaker set can fully represent the sound field characteristics of the 3D audio signal, thereby further improving the ability of the encoder to generate a virtual speaker signal when compressing and encoding the 3D audio signal to be encoded using the representative virtual speaker of the current frame. Accuracy, in order to improve the compression rate of the three-dimensional audio signal compression encoding, reduce the bandwidth occupied by the encoder to transmit the code stream.
  • the encoder 113 may select the second number of representative virtual speakers of the current frame according to the voting values of the third number of representative coefficients of the current frame to the virtual speakers in the candidate virtual speaker set.
  • the method flow described in FIG. 8 is an illustration of the specific operation process included in S630 in FIG. 7 . As shown in Figure 8, the method includes the following steps.
  • the encoder 113 determines the first number of virtual speakers and the first number of voting values according to the third number of representative coefficients of the current frame, the set of candidate virtual speakers, and the number of voting rounds.
  • Voting rounds are used to limit the number of times a virtual speaker can be voted on.
  • the number of voting rounds is an integer greater than or equal to 1, and the number of voting rounds is less than or equal to the number of virtual speakers contained in the candidate virtual speaker set, and the number of voting rounds is less than or equal to the number of virtual speaker signals transmitted by the encoder.
  • the set of candidate virtual speakers includes a fifth number of virtual speakers, the fifth number of virtual speakers includes a first number of virtual speakers, the first number is less than or equal to the fifth number, and the number of voting rounds is an integer greater than or equal to 1, and The number of voting rounds is less than or equal to the fifth number.
  • the virtual speaker signal also refers to a transmission channel representing the virtual speaker in the current frame corresponding to the current frame. Usually the number of virtual speaker signals is less than or equal to the number of virtual speakers.
  • the number of voting rounds may be preconfigured, or determined according to the computing capability of the encoder, for example, the number of voting rounds is determined according to the encoding rate and/or encoding application scenarios of the encoder .
  • the number of voting rounds is determined according to the number of directional sound sources in the current frame. For example, when the number of directional sound sources in the sound field is 2, set the number of voting rounds to 2.
  • the embodiment of the present application provides three possible implementation manners for determining the first number of virtual speakers and the first number of voting values, and the three manners are described in detail below.
  • the number of voting rounds is equal to 1.
  • the encoder 113 samples a plurality of representative coefficients, it obtains the voting values of each representative coefficient of the current frame to all virtual speakers in the candidate virtual speaker set, and the accumulation is the same Voting values of the numbered virtual speakers, the first number of virtual speakers and the first number of voting values are obtained.
  • the set of candidate virtual speakers includes the first number of virtual speakers.
  • the first number is equal to the number of virtual speakers included in the set of candidate virtual speakers. Assuming that the set of candidate virtual speakers includes a fifth number of virtual speakers, the first number is equal to the fifth number.
  • the first number of voting values includes voting values of all virtual speakers in the set of candidate virtual speakers.
  • the encoder 113 may use the first number of voting values as the final voting values of the first number of virtual speakers in the current frame, and execute S6302, that is, the encoder 113 selects the first number of virtual speakers from the first number of voting values according to the first number of voting values. Two numbers of virtual speakers representing the current frame.
  • the first number of virtual speakers includes a first virtual speaker
  • the first number of voting values includes voting values of the first virtual speaker
  • the first virtual speaker corresponds to the voting value of the first virtual speaker.
  • the voting value of the first virtual speaker is used to represent the priority of the first virtual speaker.
  • the priority can also be described as a tendency instead, that is, the voting value of the first virtual speaker is used to represent the tendency of using the first virtual speaker when encoding the current frame. It can be understood that the greater the voting value of the first virtual speaker, the higher the priority or the higher the tendency of the first virtual speaker.
  • the encoder 113 prefers to select the first virtual speaker to encode the current frame.
  • the difference from the above-mentioned first possible implementation is that after the encoder 113 obtains the voting values of each representative coefficient of the current frame for all virtual speakers in the candidate virtual speaker set, from each A representative coefficient selects part of the voting values from the voting values of all virtual speakers in the candidate virtual speaker set, accumulates the voting values of the virtual speakers with the same number in the virtual speakers corresponding to the partial voting values, and obtains the first number of virtual speakers and the first number voting value.
  • the set of candidate virtual speakers includes the first number of virtual speakers. The first number is less than or equal to the number of virtual speakers included in the set of candidate virtual speakers.
  • the first number of voting values includes voting values of some virtual speakers included in the candidate virtual speaker set, or the first number of voting values includes voting values of all virtual speakers included in the candidate virtual speaker set.
  • the difference from the above-mentioned second possible implementation is that the number of voting rounds is an integer greater than or equal to 2, and for each representative coefficient of the current frame, the encoder 113 performs All the virtual speakers in the set will vote for at least 2 rounds, and the virtual speaker with the largest voting value will be selected in each round. After performing at least 2 rounds of voting on all virtual speakers for each representative coefficient of the current frame, the voting values of virtual speakers with the same number are accumulated to obtain the first number of virtual speakers and the first number of voting values.
  • the encoder 113 selects a second number of representative virtual speakers of the current frame from the first number of virtual speakers according to the first number of voting values.
  • the encoder 113 selects representative virtual speakers of the second number of current frames from the first number of virtual speakers according to the first number of voting values, and the voting values of the second number of representative virtual speakers of the current frame are greater than a preset threshold .
  • the encoder 113 may also select a second number of representative virtual speakers of the current frame from the first number of virtual speakers according to the first number of voting values. For example, according to the descending order of the first number of voting values, determine the second number of voting values from the first number of voting values, and correspond the first number of virtual speakers to the second number of voting values The virtual speaker of is used as the representative virtual speaker of the second number of current frames.
  • the encoder 113 may use virtual speakers with different numbers as The current frame's representative virtual speaker.
  • the second quantity is smaller than the first quantity.
  • the first number of virtual speakers includes a second number of virtual speakers representative of the current frame.
  • the second number can be preset, or the second number can be determined according to the number of sound sources in the sound field of the current frame, for example, the second number can be directly equal to the number of sound sources in the sound field of the current frame, or according to
  • the encoder uses a small number of representative coefficients to replace all the coefficients of the current frame to vote for each virtual speaker in the candidate virtual speaker set, the representative virtual speaker of the current frame is selected according to the voting value. Furthermore, the encoder uses the representative virtual speaker of the current frame to compress and encode the 3D audio signal to be encoded, which not only effectively improves the compression rate of the 3D audio signal, but also reduces the computational complexity of the encoder searching for the virtual speaker. Therefore, the computational complexity of compressing and encoding the three-dimensional audio signal is reduced and the computational burden of the encoder is reduced.
  • the encoder 113 adjusts the candidate virtual speaker according to the final voting value of the previous frame representing the virtual speaker in the previous frame
  • the initial voting value of the current frame of the virtual speaker in the set, and the final voting value of the current frame of the virtual speaker is obtained.
  • FIG. 9 it is a schematic flowchart of another method for selecting a virtual speaker provided by the embodiment of the present application. Wherein, the method flow described in FIG. 9 is an illustration of the specific operation process included in S6302 in FIG. 8 .
  • the encoder 113 obtains the seventh number of final voting values of the current frame corresponding to the seventh number of virtual speakers and the current frame according to the first number of initial voting values of the current frame and the sixth number of final voting values of the previous frame.
  • the encoder 113 may determine the first number of virtual speakers and the first number of voting values according to the current frame of the three-dimensional audio signal, the set of candidate virtual speakers, and the number of voting rounds according to the method described in S6301 above, and further, the first number of virtual speakers The voting value is used as the initial voting value of the current frame of the first number of virtual speakers.
  • the virtual speaker and the initial voting value of the current frame there is a one-to-one correspondence between the virtual speaker and the initial voting value of the current frame, that is, one virtual speaker corresponds to one initial voting value of the current frame.
  • the first number of virtual speakers includes the first virtual speaker
  • the first number of current frame initial voting values includes the first virtual speaker's current frame initial voting value
  • the first virtual speaker and the first virtual speaker's current frame initial voting value correspond.
  • the current frame initial voting value of the first virtual speaker is used to represent the priority of using the first virtual speaker when encoding the current frame.
  • the sixth number of virtual speakers included in the representative virtual speaker set of the previous frame is in one-to-one correspondence with the sixth number of final voting values of the previous frame.
  • the sixth number of virtual speakers may be a representative virtual speaker of a previous frame used by the encoder 113 to encode the previous frame of the 3D audio signal.
  • the encoder 113 updates the first number of initial voting values of the current frame according to the final voting values of the sixth number of previous frames, that is, the encoder 113 calculates the first number of virtual speakers and the sixth number of virtual speakers.
  • the sum of the initial voting value of the current frame of the virtual speaker and the final voting value of the previous frame is obtained, and the final voting value of the seventh number of the current frame corresponding to the seventh number of virtual speakers and the current frame is obtained.
  • the seventh number of virtual speakers includes the first number of virtual speakers, and the seventh number of virtual speakers includes the sixth number of virtual speakers.
  • the encoder 113 selects a representative virtual speaker of the second number of current frames from the seventh number of virtual speakers according to the final voting value of the seventh number of current frames.
  • the encoder 113 selects a representative virtual speaker of the second number of current frames from the seventh number of virtual speakers according to the final voting value of the seventh number of current frames, and the current frame of the second number of current frames representing the virtual speaker finally The voting value is greater than the preset threshold.
  • the encoder 113 may also select a representative virtual speaker of the second number of current frames from the seventh number of virtual speakers according to the final voting value of the seventh number of current frames. For example, according to the descending order of the final voting values of the seventh current frame, determine the second final voting value of the current frame from the seventh final voting value of the current frame, and set the seventh virtual speaker The virtual speaker associated with the final voting value of the second number of current frames is used as the representative virtual speaker of the second number of current frames.
  • the encoder 113 may combine the virtual speakers with different numbers. Acts as the representative virtual speaker for the current frame.
  • the second quantity is smaller than the seventh quantity.
  • the seventh number of virtual speakers includes the second number of virtual speakers representative of the current frame.
  • the second number may be preset, or the second number may be determined according to the number of sound sources in the sound field of the current frame.
  • the encoder 113 may encode the second number of representatives of the current frame The virtual speaker is used as the representative virtual speaker of the second number of previous frames, and the next frame of the current frame is encoded by using the representative virtual speaker of the second number of previous frames.
  • the virtual speaker may not be able to form a one-to-one correspondence with the real sound source, and because in the actual complex scene, there may be The virtual speaker cannot represent the independent sound source in the sound field.
  • the virtual speaker searched between frames may jump frequently. This frequent jump will obviously affect the listener's auditory experience, resulting in decoding reconstruction Visible discontinuities and noise appear in the rear 3D audio signal.
  • the method for selecting a virtual speaker inherits the representative virtual speaker of the previous frame, that is, for the virtual speaker with the same number, adjusts the initial voting value of the current frame with the final voting value of the previous frame, so that the encoder is more inclined to Select the representative virtual speaker of the previous frame, thereby reducing the frequent jump of the virtual speaker between frames, enhancing the continuity of the orientation between frames, improving the stability of the sound image of the reconstructed three-dimensional audio signal, and ensuring Sound quality of the reconstructed 3D audio signal.
  • adjust the parameters to ensure that the final voting value of the previous frame will not be inherited for too long, so as to prevent the algorithm from being unable to adapt to scenes where the sound field changes such as sound source movement.
  • the embodiment of the present application provides a method for selecting a virtual speaker.
  • the encoder can first judge whether the representative virtual speaker set of the previous frame can be reused to encode the current frame. If the encoder reuses the representative virtual speaker set of the previous frame The set of speakers encodes the current frame, thereby avoiding the encoder from performing a virtual speaker search process, effectively reducing the computational complexity of the encoder to search for virtual speakers, thus reducing the computational complexity of compressing and encoding the three-dimensional audio signal and easing reduce the computational burden of the encoder.
  • FIG. 10 is a schematic flowchart of a method for selecting a virtual speaker provided by an embodiment of the present application.
  • the encoder 113 obtains a first degree of correlation between the current frame of the 3D audio signal and the representative virtual speaker set of the previous frame.
  • the sixth number of virtual speakers contained in the representative virtual speaker set of the previous frame is the representative virtual speaker of the previous frame used for encoding the previous frame of the 3D audio signal.
  • the first correlation degree is used to represent the priority of multiplexing the representative virtual speaker set of the previous frame when encoding the current frame.
  • the priority can also be described as a tendency instead, that is, the first degree of correlation is used to determine whether to reuse the representative virtual speaker set of the previous frame when encoding the current frame. Understandably, the greater the first correlation degree of the representative virtual speaker set of the previous frame, the higher the tendency of the representative virtual speaker set of the previous frame, and the encoder 113 is more inclined to select the representative virtual speaker of the previous frame for the current frames are encoded.
  • the encoder 113 judges whether the first correlation degree satisfies the multiplexing condition.
  • the encoder 113 is more inclined to search for the virtual speaker, and encodes the current frame according to the representative virtual speaker of the current frame.
  • S610 is executed, and the encoder 113 obtains the fourth frame of the current frame of the three-dimensional audio signal. number of coefficients, and the frequency-domain eigenvalues of the fourth number of coefficients.
  • the encoder 113 selects the third number of representative coefficients from the fourth number of coefficients according to the frequency-domain eigenvalues of the fourth number of coefficients, the largest representative coefficient among the third number of representative coefficients As the coefficient of the current frame for obtaining the first correlation degree, the encoder 113 obtains the first correlation degree between the largest representative coefficient among the third representative coefficients of the current frame and the representative virtual loudspeaker set of the previous frame, if the first correlation degree does not meet the multiplexing condition, execute S630, that is, the encoder 113 selects the second number of representative virtual speakers of the current frame from the candidate virtual speaker set according to the third number of representative coefficients.
  • the encoder 113 prefers to select the representative virtual speaker of the previous frame to encode the current frame, and the encoder 113 executes S670 and S680.
  • the encoder 113 generates a virtual speaker signal according to the representative virtual speaker set of the previous frame and the current frame.
  • the encoder 113 encodes the virtual speaker signal to obtain a code stream.
  • the method for selecting a virtual speaker uses the correlation between the representative coefficient of the current frame and the representative virtual speaker of the previous frame to judge whether to perform a virtual speaker search, and ensures that the selection of the correlation of the representative virtual speaker of the current frame is accurate. In the case of high degree, the complexity of the coding end is effectively reduced.
  • the encoder includes hardware structures and/or software modules corresponding to each function.
  • the present application can be implemented in the form of hardware or a combination of hardware and computer software with reference to the units and method steps of the examples described in the embodiments disclosed in the present application. Whether a certain function is executed by hardware or computer software drives the hardware depends on the specific application scenario and design constraints of the technical solution.
  • the 3D audio signal encoding method according to this embodiment is described in detail above with reference to FIG. 1 to FIG. 10 , and the 3D audio signal encoding device and encoder provided according to this embodiment will be described below in conjunction with FIG. 11 and FIG. 12 .
  • FIG. 11 is a schematic structural diagram of a possible three-dimensional audio signal encoding device provided by this embodiment.
  • These three-dimensional audio signal encoding devices can be used to implement the function of encoding three-dimensional audio signals in the above method embodiments, and thus can also achieve the beneficial effects of the above method embodiments.
  • the three-dimensional audio signal encoding device may be the encoder 113 shown in Figure 1, or the encoder 300 shown in Figure 3, or a module (such as a chip) applied to a terminal device or a server .
  • the three-dimensional audio signal encoding device 1100 includes a communication module 1110 , a coefficient selection module 1120 , a virtual speaker selection module 1130 , an encoding module 1140 and a storage module 1150 .
  • the three-dimensional audio signal coding apparatus 1100 is used to implement the functions of the encoder 113 in the method embodiments shown in FIGS. 6 to 10 above.
  • the communication module 1110 is used for acquiring the current frame of the 3D audio signal.
  • the communication module 1110 may also receive the current frame of the 3D audio signal acquired by other devices; or acquire the current frame of the 3D audio signal from the storage module 1150 .
  • the current frame of the 3D audio signal is the HOA signal; the frequency-domain eigenvalues of the coefficients are determined according to the two-dimensional vector, and the two-dimensional vector includes the HOA coefficients of the HOA signal.
  • the coefficient selection module 1120 is configured to obtain the fourth number of coefficients of the current frame of the 3D audio signal, and the frequency domain feature values of the fourth number of coefficients.
  • the coefficient selection module 1120 is further configured to select a third number of representative coefficients from the fourth number of coefficients according to frequency-domain feature values of the fourth number of coefficients, and the third number is smaller than the fourth number.
  • the coefficient selection module 1120 is used to realize related functions of S610 and S620.
  • the coefficient selection module 1120 is specifically configured to select representative coefficients from at least one subband included in the spectrum range indicated by the fourth number of coefficients according to the frequency-domain feature values of the fourth number of coefficients, to obtain a third number of representative coefficients.
  • the number of coefficients included in at least two sub-bands is different; or, the number of coefficients included in each sub-band of the at least two sub-bands is the same.
  • the coefficient selection module 1120 is specifically configured to select Z representative coefficients from each sub-band according to the frequency-domain characteristic value of the coefficients in each sub-band to obtain a third number of representative coefficients, where Z is a positive integer.
  • the coefficient selection module 1120 is specifically configured to determine the weight of each subband according to the frequency-domain feature value of the first candidate coefficient in each subband of the at least two subbands; The respective weights of each subband adjust the frequency domain eigenvalues of the second candidate coefficients in each subband to obtain the adjusted frequency domain eigenvalues of the second candidate coefficients in each subband, the first candidate coefficients and the second candidate coefficients is part of the coefficients in the sub-band; according to the adjusted frequency-domain eigenvalues of the second candidate coefficients in at least two sub-bands, and the frequency-domain eigenvalues of coefficients other than the second candidate coefficients in at least two sub-bands, determine the first Three quantities represent coefficients.
  • the virtual speaker selection module 1130 is configured to select a second number of representative virtual speakers of the current frame from the candidate virtual speaker set according to the third number of representative coefficients.
  • the virtual speaker selection module 1130 is used to realize related functions of S630.
  • the virtual speaker selection module 1130 is specifically configured to determine the first number of virtual speakers and the first number of voting values according to the third number of representative coefficients of the current frame, the set of candidate virtual speakers and the number of voting rounds, the virtual speaker and the voting value One-to-one correspondence, the first number of virtual speakers includes the first virtual speaker, the first number of voting values includes the voting value of the first virtual speaker, the first virtual speaker corresponds to the voting value of the first virtual speaker, and the first virtual speaker The voting value is used to characterize the priority of using the first virtual speaker when encoding the current frame, the candidate virtual speaker set includes the fifth number of virtual speakers, the fifth number of virtual speakers includes the first number of virtual speakers, and the number of voting rounds is An integer greater than or equal to 1, and the number of voting rounds is less than or equal to the fifth number; according to the first number of voting values, select the second number of virtual speakers representing the current frame from the first number of virtual speakers, and the second number is less than first quantity.
  • the virtual speaker selection module 1130 is further configured to obtain the seventh number of current frames corresponding to the seventh number of virtual speakers and the current frame according to the first number of voting values and the sixth number of final voting values of previous frames
  • the seventh number of virtual speakers includes the first number of virtual speakers
  • the seventh number of virtual speakers includes the sixth number of virtual speakers
  • the virtual speakers included in the sixth number of virtual speakers are for the three-dimensional audio signal.
  • the representative virtual speaker of the previous frame used for encoding the previous frame; according to the final voting value of the seventh number of current frames, select the representative virtual speaker of the second number of current frames from the seventh number of virtual speakers, and the second number is less than seventh number.
  • the virtual speaker selection module 1130 is further configured to acquire a first degree of correlation between the current frame and the representative virtual speaker set of the previous frame, where the representative virtual speaker set of the previous frame includes a sixth number of virtual speakers, and the sixth number
  • the virtual speaker included in the virtual speaker is the representative virtual speaker of the previous frame used for encoding the previous frame of the 3D audio signal
  • the first correlation is used to determine whether to reuse the representative virtual speaker of the previous frame when encoding the current frame.
  • a set of loudspeakers if the first degree of correlation does not meet the multiplexing condition, obtain the fourth number of coefficients of the current frame of the three-dimensional audio signal, and the frequency domain feature values of the fourth number of coefficients.
  • the encoding module 1140 is configured to encode the current frame according to the second number of representative virtual speakers of the current frame to obtain a code stream.
  • the coding module 1140 is used to realize related functions of S640.
  • the encoding module 1140 is specifically configured to generate a virtual speaker signal according to the second number of representative virtual speakers of the current frame and the current frame; and encode the virtual speaker signal to obtain a code stream.
  • the storage module 1150 is used to store the coefficients related to the three-dimensional audio signal, the candidate virtual speaker set, the representative virtual speaker set of the previous frame, and the selected coefficients and virtual speakers, etc., so that the encoding module 1140 encodes the current frame to obtain a code stream , and transmit the code stream to the decoder.
  • the three-dimensional audio signal encoding device 1100 in the embodiment of the present application may be implemented by an application-specific integrated circuit (application-specific integrated circuit, ASIC), or a programmable logic device (programmable logic device, PLD), and the above-mentioned PLD may be Complex programmable logical device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL) or any combination thereof.
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • CPLD Complex programmable logical device
  • FPGA field-programmable gate array
  • GAL generic array logic
  • FIG. 12 is a schematic structural diagram of an encoder 1200 provided in this embodiment. As shown in FIG. 12 , the encoder 1200 includes a processor 1210 , a bus 1220 , a memory 1230 and a communication interface 1240 .
  • the processor 1210 may be a central processing unit (central processing unit, CPU), and the processor 1210 may also be other general-purpose processors, digital signal processors (digital signal processing, DSP), ASIC , FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general purpose processor may be a microprocessor or any conventional processor or the like.
  • the processor can also be a graphics processing unit (graphics processing unit, GPU), a neural network processing unit (neural network processing unit, NPU), a microprocessor, or one or more integrated circuits used to control the execution of the program of the present application.
  • graphics processing unit graphics processing unit, GPU
  • neural network processing unit neural network processing unit, NPU
  • microprocessor or one or more integrated circuits used to control the execution of the program of the present application.
  • the communication interface 1240 is used to realize the communication between the encoder 1200 and external devices or devices.
  • the communication interface 1240 is used to receive 3D audio signals.
  • Bus 1220 may include a path for transferring information between the components described above (eg, processor 1210 and memory 1230).
  • the bus 1220 may also include a power bus, a control bus, a status signal bus, and the like. However, for clarity of illustration, the various buses are labeled as bus 1220 in the figure.
  • encoder 1200 may include multiple processors.
  • the processor may be a multi-CPU processor.
  • a processor herein may refer to one or more devices, circuits, and/or computing units for processing data (eg, computer program instructions).
  • the processor 1210 may call the coefficients related to the three-dimensional audio signal stored in the memory 1230, the set of candidate virtual speakers, the set of representative virtual speakers of the previous frame, selected coefficients and virtual speakers, and the like.
  • the encoder 1200 includes only one processor 1210 and one memory 1230 as an example.
  • the processor 1210 and the memory 1230 are respectively used to indicate a type of device or device.
  • the quantity of each type of device or equipment can be determined according to business needs.
  • the memory 1230 may correspond to the storage medium used for storing the coefficients related to the three-dimensional audio signal, the candidate virtual speaker set, the representative virtual speaker set of the previous frame, and the selected coefficients and virtual speakers in the above method embodiment, for example, a disk , such as a mechanical hard drive or solid state drive.
  • the above-mentioned encoder 1200 may be a general-purpose device or a special-purpose device.
  • the encoder 1200 may be a server based on X86 or ARM, or other dedicated servers, such as a policy control and charging (policy control and charging, PCC) server, and the like.
  • policy control and charging policy control and charging, PCC
  • the embodiment of the present application does not limit the type of the encoder 1200 .
  • the encoder 1200 may correspond to the three-dimensional audio signal encoding device 1100 in this embodiment, and may correspond to a corresponding subject performing any method in FIG. 6 to FIG. 10 , and the three-dimensional audio signal
  • the above-mentioned and other operations and/or functions of each module in the encoding device 1100 are for realizing the corresponding flow of each method in FIG. 6 to FIG. 10 , and for the sake of brevity, details are not repeated here.
  • the method steps in this embodiment may be implemented by means of hardware, and may also be implemented by means of a processor executing software instructions.
  • Software instructions can be composed of corresponding software modules, and software modules can be stored in random access memory (random access memory, RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or known in the art any other form of storage medium.
  • An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium.
  • the storage medium may also be a component of the processor.
  • the processor and storage medium can be located in the ASIC.
  • the ASIC can be located in a network device or a terminal device.
  • the processor and the storage medium may also exist in the network device or the terminal device as discrete components.
  • all or part of them may be implemented by software, hardware, firmware or any combination thereof.
  • software When implemented using software, it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product comprises one or more computer programs or instructions. When the computer program or instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are executed in whole or in part.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, network equipment, user equipment, or other programmable devices.
  • the computer program or instructions can be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions can be downloaded from a website, computer, A server or data center transmits to another website site, computer, server or data center by wired or wireless means.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrating one or more available media. Described usable medium can be magnetic medium, for example, floppy disk, hard disk, magnetic tape; It can also be optical medium, for example, digital video disc (digital video disc, DVD); It can also be semiconductor medium, for example, solid state drive (solid state drive) , SSD).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Acoustics & Sound (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Mathematical Physics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Stereophonic System (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

本申请公开了一种三维音频信号编码方法、装置和编码器，涉及多媒体领域。该方法包括：编码器获取到三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值后，根据第四数量个系数的频域特征值，从第四数量个系数中选取第三数量个代表系数，并根据第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器，进而，根据第二数量个当前帧的代表虚拟扬声器对当前帧进行编码，得到码流。由于编码器利用较少数量的代表系数代替全部系数从候选虚拟扬声器集合中选取代表虚拟扬声器，因此有效地降低了编码器搜索虚拟扬声器的计算复杂度，以及对三维音频信号进行压缩编码的计算复杂度，减轻了编码器的计算负担。

Description

三维音频信号编码方法、装置和编码器
本申请要求于2021年05月17日提交国家知识产权局、申请号为202110535832.3、申请名称为“三维音频信号编码方法、装置和编码器”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及多媒体领域，尤其涉及一种三维音频信号编码方法、装置和编码器。
背景技术
随着高性能计算机和信号处理技术的飞速发展，收听者对语音、音频体验提出了越来越高的要求，浸入式音频能够满足人们在这方面的需求。例如，三维音频技术在无线通信(例如4G/5G等等)语音、虚拟现实/增强现实和媒体音频等方面得到了广泛应用。三维音频技术是对真实世界中的声音和三维声场信息进行获取、处理、传输和渲染回放的音频技术，使声音具有强烈的空间感、包围感及沉浸感，给收听者以“身临其境”的非凡听觉体验。
通常，采集设备(如：麦克风)采集大量的数据记录三维声场信息，向回放设备(例如扬声器，耳机等)传输三维音频信号，以便于回放设备播放三维音频。由于三维声场信息的数据量较大，导致需要大量的存储空间存储数据，以及传输三维音频信号的带宽需求较高。为了解决上述问题，可以对三维音频信号进行压缩，存储或传输压缩数据。目前，编码器可以采用预先配置的多个虚拟扬声器对三维音频信号进行压缩。但是，编码器对三维音频信号进行压缩编码的计算复杂度较高。因此，如何降低对三维音频信号进行压缩编码的计算复杂度是一个亟待解决的问题。
发明内容
本申请提供了三维音频信号编码方法、装置和编码器，由此可以降低对三维音频信号进行压缩编码的计算复杂度。
第一方面，本申请提供了一种三维音频信号编码方法，该方法可以由编码器执行，具体包括如下步骤：编码器获取到三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值后，根据第四数量个系数的频域特征值，从第四数量个系数中选取第三数量个代表系数，进而，根据第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器，以及，根据第二数量个当前帧的代表虚拟扬声器对当前帧进行编码，得到码流。其中，所述第四数量个系数包括第三数量个代表系数，第三数量小于第四数量，表示第三数量个代表系数是第四数量个系数中的部分系数。
其中，三维音频信号的当前帧为高阶立体混响(higher order ambisonics，HOA)信号；系数的频域特征值是依据HOA信号的系数确定的。
如此，由于编码器从当前帧的全部系数中选取部分系数作为代表系数，利用较少数量的代表系数代替当前帧的全部系数从候选虚拟扬声器集合中选取代表虚拟扬声器， 因此有效地降低了编码器搜索虚拟扬声器的计算复杂度，从而降低了对三维音频信号进行压缩编码的计算复杂度以及减轻了编码器的计算负担。
另外，编码器根据第二数量个当前帧的代表虚拟扬声器对当前帧进行编码，得到码流包括：编码器根据第二数量个当前帧的代表虚拟扬声器和当前帧生成虚拟扬声器信号；对虚拟扬声器信号进行编码得到码流。
由于当前帧的系数的频域特征值表征了三维音频信号的声场特性，编码器依据当前帧的系数的频域特征值选取当前帧的具有代表性声场成分的代表系数，利用代表系数从候选虚拟扬声器集合中选取的当前帧的代表虚拟扬声器能够充分地表征三维音频信号的声场特性，从而进一步地提高了编码器利用当前帧的代表虚拟扬声器对待编码的三维音频信号进行压缩编码时生成虚拟扬声器信号的准确性，以便于提升对三维音频信号进行压缩编码的压缩率，降低编码器传输码流所占用的带宽。
在一种可能的实现方式中，根据第四数量个系数的频域特征值，从第四数量个系数中选取第三数量个代表系数，包括：编码器根据第四数量个系数的频域特征值，从第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到第三数量个代表系数。
例如，根据第四数量个系数的频域特征值，从第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到第三数量个代表系数包括：编码器根据至少一个子带中每个子带中系数的频域特征值，从每个子带中分别选取Z个代表系数，以得到第三数量个代表系数，Z为正整数。由于编码器在当前帧的全部系数指示的频谱范围内，根据系数的频域特征值选取代表系数，从而确保每个子带均有代表系数被选中，提高了编码器在当前帧的全部系数指示的频谱范围内选取代表系数的均衡性。
又如，当至少一个子带包括至少两个子带时，根据第四数量个系数的频域特征值，从第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到第三数量个代表系数包括：编码器根据至少两个子带中每个子带内的第一候选系数的频域特征值确定每个子带的权重；根据每个子带各自的权重分别调整每个子带内的第二候选系数的频域特征值，得到每个子带内的第二候选系数的调整后频域特征值，第一候选系数和第二候选系数为子带内的部分系数；根据至少两个子带内的第二候选系数的调整后频域特征值，以及至少两个子带内除第二候选系数之外的系数的频域特征值，确定第三数量个代表系数。如此，编码器根据子带的权重调整该子带内的系数被选中的机率，进一步地提高了编码器选取的代表系数从声场分布和音频特性上代表了全子带系数的准确性。
其中，编码器可以不等分划分频谱范围得到至少两个子带，则至少两个子带包含的系数的数量不同；或者，编码器也可以等分划分频谱范围得到至少两个子带，则至少两个子带中每个子带包含的系数的数量相同。
在另一种可能的实现方式中，根据第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器包括：编码器根据当前帧的第三数量个代表系数、候选虚拟扬声器集合和投票轮数确定第一数量个虚拟扬声器和第一数量个投票值，根据第一数量个投票值，从第一数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器，第二数量小于第一数量，表示第二数量个当前帧的代表虚拟扬声器是 候选虚拟扬声器集合中的部分虚拟扬声器。可理解的，虚拟扬声器与投票值一一对应。例如，第一数量个虚拟扬声器包括第一虚拟扬声器，第一数量个投票值包括第一虚拟扬声器的投票值，第一虚拟扬声器与第一虚拟扬声器的投票值对应。第一虚拟扬声器的投票值用于表征第一虚拟扬声器的优先级。候选虚拟扬声器集合包括第五数量个虚拟扬声器，第五数量个虚拟扬声器包括第一数量个虚拟扬声器，第一数量小于或等于第五数量，投票轮数为大于或等于1的整数，且投票轮数小于或等于第五数量。第二数量是预设的，或者，第二数量是根据当前帧确定的。
目前，在虚拟扬声器搜索过程中，编码器依据待编码的三维音频信号和虚拟扬声器之间的相关计算的结果作为虚拟扬声器的选择衡量指标。而且，若编码器对每一个系数传输一个虚拟扬声器，则无法达到高效数据压缩的目的，会对编码器造成沉重的计算负担。本申请实施例提供的选择虚拟扬声器的方法，编码器利用较少数量的代表系数代替当前帧的全部系数对候选虚拟扬声器集合中每个虚拟扬声器进行投票，依据投票值选取当前帧的代表虚拟扬声器。进而，编码器利用当前帧的代表虚拟扬声器对待编码的三维音频信号进行压缩编码，不仅有效地提升了对三维音频信号进行压缩编码的压缩率，而且降低了编码器搜索虚拟扬声器的计算复杂度，从而降低了对三维音频信号进行压缩编码的计算复杂度以及减轻了编码器的计算负担。
第二数量用于表征编码器选取的当前帧的代表虚拟扬声器的数量。第二数量越大表示当前帧的代表虚拟扬声器的数量越大，三维音频信号的声场信息越多；第二数量越小表示当前帧的代表虚拟扬声器的数量越小，三维音频信号的声场信息越少。因此，可通过设置第二数量控制编码器选取的当前帧的代表虚拟扬声器的数量。例如，第二数量可以是预设的，又如，第二数量可以是根据当前帧确定的。示例地，第二数量的取值可以是1、2、4或8。
在另一种可能的实现方式中，根据第一数量个投票值，从第一数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器包括：编码器根据第一数量个投票值，以及第六数量个在先帧最终投票值，获取第七数量个虚拟扬声器与当前帧对应的第七数量个当前帧最终投票值，根据第七数量个当前帧最终投票值，从第七数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器，第二数量小于第七数量，表示第二数量个当前帧的代表虚拟扬声器是第七数量个虚拟扬声器中的部分虚拟扬声器。其中，第七数量个虚拟扬声器包括第一数量个虚拟扬声器，且第七数量个虚拟扬声器包括第六数量个虚拟扬声器，第六数量个虚拟扬声器包含的虚拟扬声器为对三维音频信号的在先帧进行编码所使用的在先帧的代表虚拟扬声器。在先帧的代表虚拟扬声器集合包含的第六数量个虚拟扬声器与所述第六数量个在先帧最终投票值一一对应。
在虚拟扬声器搜索过程中，由于真实声源的位置与虚拟扬声器的位置不一定重合，会导致虚拟扬声器不一定能够与真实声源形成一一对应关系，且由于在实际的复杂场景下，可能出现有限数量的虚拟扬声器集合无法表征声场中所有声源的情况，此时，帧与帧之间搜索到的虚拟扬声器可能会发生频繁跳变，这种跳变会明显地影响听音者的听觉感受，导致解码重建后三维音频信号中出现明显的不连续和噪声现象。本申请的实施例提供的选择虚拟扬声器的方法通过继承在先帧的代表虚拟扬声器，即对于相同编号的虚拟扬声器，用在先帧最终投票值调整当前帧初始投票值，使得编码器更倾 向于选择在先帧的代表虚拟扬声器，从而降低帧与帧之间的虚拟扬声器的频繁跳变，增强了帧之间的信号方位的连续性，提高了重建后三维音频信号的声像的稳定性，确保重建后三维音频信号的音质。
在另一种可能的实现方式中，该方法还包括：编码器获取当前帧与在先帧的代表虚拟扬声器集合的第一相关度，若第一相关度不满足复用条件，获取三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值。在先帧的代表虚拟扬声器集合包括第六数量个虚拟扬声器，第六数量个虚拟扬声器包含的虚拟扬声器为对三维音频信号的在先帧进行编码所使用的在先帧的代表虚拟扬声器，第一相关度用于确定对当前帧进行编码时是否复用在先帧的代表虚拟扬声器集合。
如此，编码器可以先判断是否可以复用在先帧的代表虚拟扬声器集合对当前帧进行编码，如果编码器复用在先帧的代表虚拟扬声器集合对当前帧进行编码，从而，避免编码器再执行搜索虚拟扬声器的过程，有效地降低了编码器搜索虚拟扬声器的计算复杂度，因此降低了对三维音频信号进行压缩编码的计算复杂度以及减轻了编码器的计算负担。另外，还可以降低帧与帧之间的虚拟扬声器的频繁跳变，增强了帧之间的方位的连续性，提高了重建后三维音频信号的声像的稳定性，确保重建后三维音频信号的音质。如果编码器不能复用在先帧的代表虚拟扬声器集合对当前帧进行编码，编码器再选取代表系数，利用当前帧的代表系数对候选虚拟扬声器集合中每个虚拟扬声器进行投票，依据投票值选取当前帧的代表虚拟扬声器，来达到降低对三维音频信号进行压缩编码的计算复杂度以及减轻编码器的计算负担的目的。
可选地，方法还包括：编码器还可以采集三维音频信号的当前帧，以便于对三维音频信号的当前帧进行压缩编码得到码流，将码流传输至解码端。
第二方面，本申请提供了一种三维音频信号编码装置，所述装置包括用于执行第一方面或第一方面任一种可能设计中的三维音频信号编码方法的各个模块。例如，三维音频信号编码装置包括系数选择模块、虚拟扬声器选择模块和编码模块。所述系数选择模块，用于获取三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值；所述系数选择模块，还用于根据第四数量个系数的频域特征值，从第四数量个系数中选取第三数量个代表系数，第三数量小于第四数量；所述虚拟扬声器选择模块，用于根据第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器；所述编码模块，用于根据第二数量个当前帧的代表虚拟扬声器对当前帧进行编码，得到码流。这些模块可以执行上述第一方面方法示例中的相应功能，具体参见方法示例中的详细描述，此处不做赘述。
第三方面，本申请提供一种编码器，该编码器包括至少一个处理器和存储器，其中，所述存储器用于存储一组计算机指令；当处理器执行所述一组计算机指令时，执行第一方面或第一方面任一种可能实现方式中的三维音频信号编码方法的操作步骤。
第四方面，本申请提供一种系统，系统包括如第三方面所述的编码器，以及解码器，所述编码器用于执行第一方面或第一方面任一种可能实现方式中的三维音频信号编码方法的操作步骤，所述解码器用于解码所述编码器生成的码流。
第五方面，本申请提供一种计算机可读存储介质，包括：计算机软件指令；当计算机软件指令在编码器中运行时，使得编码器执行如第一方面或第一方面任意一种可 能的实现方式中所述方法的操作步骤。
第六方面，本申请提供一种计算机程序产品，当计算机程序产品在编码器上运行时，使得编码器执行如第一方面或第一方面任意一种可能的实现方式中所述方法的操作步骤。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
图1为本申请实施例提供的一种音频编解码系统的结构示意图；
图2为本申请实施例提供的一种音频编解码系统的场景示意图；
图3为本申请实施例提供的一种编码器的结构示意图；
图4为本申请实施例提供的一种三维音频信号编解码方法的流程示意图；
图5为本申请实施例提供的一种选择虚拟扬声器方法的流程示意图；
图6为本申请实施例提供的一种三维音频信号编码方法的流程示意图；
图7为本申请实施例提供的一种选择三维音频信号的代表系数方法的流程示意图；
图8为本申请实施例提供的一种选择虚拟扬声器方法的流程示意图；
图9为本申请实施例提供的另一种选择虚拟扬声器方法的流程示意图；
图10为本申请实施例提供的另一种选择虚拟扬声器方法的流程示意图；
图11为本申请提供的一种三维音频信号编码装置的结构示意图；
图12为本申请提供的一种编码器的结构示意图。
具体实施方式
为了下述各实施例的描述清楚简洁，首先给出相关技术的简要介绍。
声音(sound)是由物体振动产生的一种连续的波。产生振动而发出声波的物体称为声源。声波通过介质(如：空气、固体或液体)传播的过程中，人或动物的听觉器官能感知到声音。
声波的特征包括音调、音强和音色。音调表示声音的高低。音强表示声音的大小。音强也可以称为响度或音量。音强的单位是分贝(decibel，dB)。音色又称为音品。
声波的频率决定了音调的高低。频率越高音调越高。物体在一秒钟之内振动的次数称为频率，频率单位是赫兹(hertz，Hz)。人耳能识别的声音的频率在20Hz～20000Hz之间。
声波的幅度决定了音强的强弱。幅度越大音强越大。距离声源越近，音强越大。
声波的波形决定了音色。声波的波形包括方波、锯齿波、正弦波和脉冲波等。
根据声波的特征，声音可以分为规则声音和无规则声音。无规则声音是指声源无规则地振动发出的声音。无规则声音例如是影响人们工作、学习和休息等的噪声。规则声音是指声源规则地振动发出的声音。规则声音包括语音和乐音。声音用电表示时，规则声音是一种在时频域上连续变化的模拟信号。该模拟信号可以称为音频信号。音频信号是一种携带语音、音乐和音效的信息载体。
由于人的听觉系统具有辨别空间中声源的位置分布的能力，则听音者听到空间中的声音时，除了能感受到声音的音调、音强和音色外，还能感受到声音的方位。
随着人们对听觉体验的关注和品质要求与日俱增，为了增强声音的纵深感、临场 感和空间感，则三维音频技术应运而生。从而听音者不仅感受到来自前、后、左和右的声源发出的声音，而且感受到自己所处空间被这些声源产生的空间声场(简称“声场”(sound field))所包围的感觉，以及声音向四周扩散的感觉，营造出一种使听音者置身于影院或音乐厅等场所的“身临其境”的音响效果。
三维音频技术是指将人耳以外的空间假设为一个系统，耳膜处接收到的信号为声源发出的声音经过耳朵以外系统滤波输出的三维音频信号。例如，人耳以外的系统可以定义为系统冲击响应h(n)，任意一个声源可以定义为x(n)，耳膜处接收到的信号为x(n)和h(n)的卷积结果。本申请实施例所述的三维音频信号可以是指高阶立体混响(higher order ambisonics，HOA)信号。三维音频也可以称为三维音效、空间音频、三维声场重建、虚拟3D音频或双耳音频等。
众所周知，声波在理想介质中传播，波数为k＝w/c，角频率为w＝2πf，其中，f为声波频率，c为声速。声压p满足公式(1)，
Figure PCTCN2022091558-appb-000001
为拉普拉斯算子。
Figure PCTCN2022091558-appb-000002
假设人耳以外的空间系统是一个球形，听音者处于球的中心，从球外传来的声音在球面上有一个投影，过滤掉球面以外的声音，假设声源分布在这个球面上，用球面上的声源产生的声场来拟合原始声源产生的声场，即三维音频技术就是一个拟合声场的方法。具体地，在球坐标系下求解公式(1)等式方程，在无源球形区域内，该公式(1)方程解为如下公式(2)。
Figure PCTCN2022091558-appb-000003
其中，r表示球半径，θ表示水平角，
Figure PCTCN2022091558-appb-000004
表示俯仰角，k表示波数，s表示理想平面波的幅度，m表示三维音频信号的阶数序号(或称为HOA信号的阶数序号)。
Figure PCTCN2022091558-appb-000005
表示球贝塞尔函数，球贝塞尔函数又称为径向基函数，其中，第一个j表示虚数单位，
Figure PCTCN2022091558-appb-000006
不随角度变化。
Figure PCTCN2022091558-appb-000007
表示θ,
Figure PCTCN2022091558-appb-000008
方向的球谐函数，
Figure PCTCN2022091558-appb-000009
表示声源方向的球谐函数。三维音频信号系数满足公式(3)。
Figure PCTCN2022091558-appb-000010
将公式(3)代入公式(2)，公式(2)可以变形为公式(4)。
Figure PCTCN2022091558-appb-000011
其中，
Figure PCTCN2022091558-appb-000012
表示N阶的三维音频信号系数，用于近似描述声场。声场是指介质中有声波存在的区域。N为大于或等于1的整数。比如，N的取值范围为2至6的整数。本申请的实施例所述的三维音频信号的系数可以是指HOA系数或环境立体声(ambisonic)系数。
三维音频信号是一种携带声场中声源的空间位置信息的信息载体，描述了空间中听音者的声场。公式(4)表明声场可以在球面上按球谐函数展开，即声场可以分解为多个平面波的叠加。因此，可以将三维音频信号描述的声场使用多个平面波的叠加来表达，并通过三维音频信号系数重建声场。
相对5.1声道的音频信号或7.1声道的音频信号，由于N阶的HOA信号有(N+1) 2个声道，则HOA信号包括用于描述声场的空间信息的数据量较多。若采集设备(比如：麦克风)将该三维音频信号传输到回放设备(比如：扬声器)，需要消耗较大的带宽。目前，编码器可以利用空间压缩环绕音频编码(spatial squeezed surround audio  coding，S3AC)或定向音频编码(directional audio coding，DirAC)对三维音频信号进行压缩编码得到码流，向回放设备传输码流。回放设备对码流进行解码，并重建三维音频信号，播放重建后三维音频信号。从而降低向回放设备传输三维音频信号的数据量，以及带宽的占用。但是，编码器对三维音频信号进行压缩编码的计算复杂度较高，占用编码器过多的计算资源。因此，如何降低对三维音频信号进行压缩编码的计算复杂度是一个亟待解决的问题。
本申请实施例提供一种音频编解码技术，尤其是提供一种面向三维音频信号的三维音频编解码技术，具体提供一种采用较少的声道表示三维音频信号的编解码技术，以改进传统的音频编解码系统。音频编码(或通常称为编码)包括音频编码和音频解码两部分。音频编码在源侧执行，通常包括处理(例如，压缩)原始音频以减少表示该原始音频所需的数据量，从而更高效地存储和/或传输。音频解码在目的侧执行，通常包括相对于编码器作逆处理，以重建原始音频。编码部分和解码部分也合称为编解码。下面将结合附图对本申请实施例的实施方式进行详细描述。
图1为本申请实施例提供的一种音频编解码系统的结构示意图。音频编解码系统100包括源设备110和目的设备120。源设备110用于对三维音频信号进行压缩编码得到码流，向目的设备120传输码流。目的设备120对码流进行解码，并重建三维音频信号，播放重建后三维音频信号。
具体地，源设备110包括音频获取器111、预处理器112、编码器113和通信接口114。
音频获取器111用于获取原始音频。音频获取器111可以是任意类型的用于捕获现实世界声音的音频采集设备，和/或任意类型的音频生成设备。音频获取器111例如是用于生成计算机音频的计算机音频处理器。音频获取器111也可以为存储音频的任意类型的内存或存储器。音频包括现实世界声音、虚拟场景(如：VR或增强现实(augmented reality，AR))声音和/或其任意组合。
预处理器112用于接收音频获取器111采集的原始音频，并对原始音频进行预处理，得到三维音频信号。例如，预处理器112执行的预处理包括声道转换、音频格式转换或去噪声等。
编码器113用于接收预处理器112生成的三维音频信号，对三维音频信号进行压缩编码得到码流。示例地，编码器113可以包括空间编码器1131和核心编码器1132。空间编码器1131用于根据三维音频信号从候选虚拟扬声器集合选取(或称为搜索)虚拟扬声器，根据三维音频信号和虚拟扬声器生成虚拟扬声器信号。虚拟扬声器信号也可以称为回放信号。核心编码器1132用于对虚拟扬声器信号进行编码，得到码流。
通信接口114用于接收编码器113生成的码流，通过通信信道130向目的设备120发送码流，以便于目的设备120根据码流重建三维音频信号。
目的设备120包括播放器121、后处理器122、解码器123和通信接口124。
通信接口124用于接收通信接口114发送的码流，并将码流传输给解码器123。以便于解码器123根据码流重建三维音频信号。
通信接口114和通信接口124可用于通过源设备110与目的设备120之间的直连通信链路，例如直接有线或无线连接等，或者通过任意类型的网络，例如有线网络、 无线网络或其任意组合、任意类型的私网和公网或其任意类型的组合，发送或接收原始音频的相关数据。
通信接口114和通信接口124均可配置为如图1中从源设备110指向目的设备120的对应通信信道130的箭头所指示的单向通信接口，或双向通信接口，并且可用于发送和接收消息等，以建立连接，确认并交换与通信链路和/或例如编码后的码流传输等数据传输相关的任何其它信息，等等。
解码器123用于对码流进行解码，并重建三维音频信号。示例地，解码器123包括核心解码器1231和空间解码器1232。核心解码器1231用于对码流进行解码，得到虚拟扬声器信号。空间解码器1232用于根据候选虚拟扬声器集合和虚拟扬声器信号重建三维音频信号，得到重建后三维音频信号。
后处理器122用于接收解码器123生成的重建后三维音频信号，对重建后三维音频信号进行后处理。例如，后处理器122执行的后处理包括音频渲染、响度归一化、用户交互、音频格式转换或去噪声等。
播放器121用于根据重建后三维音频信号播放重建的声音。
需要说明的是，音频获取器111和编码器113可以集成在一个物理设备上，也可以设置在不同的物理设备上，不予限定。示例地，如图1所示的源设备110包括音频获取器111和编码器113，表示音频获取器111和编码器113集成在一个物理设备上，则源设备110也可称为采集设备。源设备110例如是无线接入网的媒体网关、核心网的媒体网关、转码设备、媒体资源服务器、AR设备、VR设备、麦克风或者其他采集音频设备。若源设备110不包括音频获取器111，表示音频获取器111和编码器113是两个不同的物理设备，源设备110可以从其他设备(如：采集音频设备或存储音频设备)获取原始音频。
另外，播放器121和解码器123可以集成在一个物理设备上，也可以设置在不同的物理设备上，不予限定。示例地，如图1所示的目的设备120包括播放器121和解码器123，表示播放器121和解码器123集成在一个物理设备上，则目的设备120也可称为回放设备，目的设备120具有解码和播放重建音频的功能。目的设备120例如是扬声器、耳机或其他播放音频的设备。若目的设备120不包括播放器121，表示播放器121和解码器123是两个不同的物理设备，目的设备120对码流解码重建三维音频信号后，将重建后三维音频信号传输给其他播放设备(如：扬声器或耳机)，由其他播放设备回放重建后三维音频信号。
此外，图1示出了源设备110和目的设备120可以集成在一个物理设备上，也可以设置在不同的物理设备上，不予限定。
示例地，如图2中的(a)所示，源设备110可以是录音棚中的麦克风，目的设备120可以是扬声器。源设备110可以采集各种乐器的原始音频，将原始音频传输至编解码设备，编解码设备对原始音频进行编解码处理，得到重建后三维音频信号，由目的设备120回放重建后三维音频信号。又示例地，源设备110可以是终端设备中的麦克风，目的设备120可以是耳机。源设备110可以采集外界的声音或终端设备合成的音频。
又示例地，如图2中的(b)所示，源设备110和目的设备120集成在虚拟现实(virtual reality，VR)设备、增强现实(Augmented Reality，AR)设备、混合现实(Mixed Reality， MR)设备或扩展现实(Extended Reality，XR)设备中，则VR/AR/MR/XR设备具备采集原始音频、回放音频和编解码的功能。源设备110可以采集用户发出的声音和用户所处的虚拟环境中虚拟物体发出的声音。
在这些实施例中，源设备110或其对应功能和目的设备120或其对应功能可以使用相同硬件和/或软件或通过单独的硬件和/或软件或其任意组合来实现。根据描述，图1所示的源设备110和/或目的设备120中的不同单元或功能的存在和划分可能根据实际设备和应用而有所不同，这对技术人员来说是显而易见的。
上述音频编解码系统的结构只是示意性说明，在一些可能的实现方式中，音频编解码系统还可以包括其他设备，例如，音频编解码系统还可以包括端侧设备或云侧设备。源设备110采集到原始音频后，对原始音频进行预处理，得到三维音频信号；并将三维音频传输至端侧设备或云侧设备，由端侧设备或云侧设备实现对三维音频信号进行编解码的功能。
本申请实施例提供的音频信号编解码方法主要应用于编码端。结合图3对编码器的结构进行详细说明。如图3所示，编码器300包括虚拟扬声器配置单元310、虚拟扬声器集合生成单元320、编码分析单元330、虚拟扬声器选择单元340、虚拟扬声器信号生成单元350和编码单元360。
虚拟扬声器配置单元310用于根据编码器配置信息生成虚拟扬声器配置参数，以便得到多个虚拟扬声器。编码器配置信息包括但不限于：三维音频信号的阶数(或通常称为HOA阶数)，编码比特率，用户自定义信息，等。虚拟扬声器配置参数包括但不限于：虚拟扬声器的数量，虚拟扬声器的阶数，虚拟扬声器的位置坐标，等。虚拟扬声器的数量例如是2048、1669、1343、1024、530、512、256、128或64等。虚拟扬声器的阶数可以是2阶至6阶中任一个。虚拟扬声器的位置坐标包括水平角和俯仰角。
虚拟扬声器配置单元310输出的虚拟扬声器配置参数作为虚拟扬声器集合生成单元320的输入。
虚拟扬声器集合生成单元320用于根据虚拟扬声器配置参数生成候选虚拟扬声器集合，候选虚拟扬声器集合包括多个虚拟扬声器。具体地，虚拟扬声器集合生成单元320根据虚拟扬声器的数量确定了候选虚拟扬声器集合包括的多个虚拟扬声器，以及根据虚拟扬声器的位置信息(如：坐标)和虚拟扬声器的阶数确定虚拟扬声器的系数。示例地，虚拟扬声器的坐标确定方法包括但不限于：按等距规则产生多个虚拟扬声器，或者根据听觉感知原理生成非均匀分布的多个虚拟扬声器；然后，根据虚拟扬声器的数量生成虚拟扬声器的坐标。
根据上述三维音频信号的生成原理也可以生成虚拟扬声器的系数。将公式(3)中的θ s和
Figure PCTCN2022091558-appb-000013
分别设置为虚拟扬声器的位置坐标，
Figure PCTCN2022091558-appb-000014
表示N阶的虚拟扬声器的系数。虚拟扬声器的系数也可以称作ambisonics系数。
编码分析单元330用于对三维音频信号进行编码分析，例如分析三维音频信号的声场分布特征，即三维音频信号的声源数量、声源的方向性和声源的弥散度等特征。
虚拟扬声器集合生成单元320输出的候选虚拟扬声器集合包括的多个虚拟扬声器的系数作为虚拟扬声器选择单元340的输入。
编码分析单元330输出的三维音频信号的声场分布特征作为虚拟扬声器选择单元340的输入。
虚拟扬声器选择单元340用于根据待编码的三维音频信号、三维音频信号的声场分布特征和多个虚拟扬声器的系数确定与三维音频信号匹配的代表虚拟扬声器。
不限定的是，本申请实施例的编码器300还可以不包括编码分析单元330，即编码器300可以不对输入信号进行分析，虚拟扬声器选择单元340采用一种默认配置确定代表虚拟扬声器。例如，虚拟扬声器选择单元340仅根据三维音频信号和多个虚拟扬声器的系数确定与三维音频信号匹配的代表虚拟扬声器。
其中，编码器300可以将从采集设备获取的三维音频信号或采用人工音频对象合成的三维音频信号作为编码器300的输入。另外，编码器300输入的三维音频信号可以是时域三维音频信号也可以是频域三维音频信号，不予限定。
虚拟扬声器选择单元340输出的代表虚拟扬声器的位置信息和代表虚拟扬声器的系数作为虚拟扬声器信号生成单元350和编码单元360的输入。
虚拟扬声器信号生成单元350用于根据三维音频信号和代表虚拟扬声器的属性信息生成虚拟扬声器信号。代表虚拟扬声器的属性信息包括代表虚拟扬声器的位置信息、代表虚拟扬声器的系数和三维音频信号的系数中至少一个。若属性信息为代表虚拟扬声器的位置信息，根据代表虚拟扬声器的位置信息确定代表虚拟扬声器的系数；若属性信息包括三维音频信号的系数，根据三维音频信号的系数获取代表虚拟扬声器的系数。具体地，虚拟扬声器信号生成单元350根据三维音频信号的系数和代表虚拟扬声器的系数计算虚拟扬声器信号。
示例地，假设矩阵A表示虚拟扬声器的系数，矩阵X表示HOA信号的HOA系数。矩阵X为矩阵A的逆矩阵。采用最小二乘方法求得理论的最优解w，w表示虚拟扬声器信号。虚拟扬声器信号满足公式(5)。
w＝A -1X    公式(5)
其中，A -1表示矩阵A的逆矩阵。矩阵A的大小为(M×C)，C表示代表虚拟扬声器的数量，M表示N阶的HOA信号的声道的数量，a表示代表虚拟扬声器的系数，矩阵X的大小为(M×L)，L表示HOA信号的系数的数量，x表示HOA信号的系数。代表虚拟扬声器的系数可以是指代表虚拟扬声器的HOA系数或代表虚拟扬声器的ambisonics系数。例如，
Figure PCTCN2022091558-appb-000015
虚拟扬声器信号生成单元350输出的虚拟扬声器信号作为编码单元360的输入。
编码单元360用于对虚拟扬声器信号进行核心编码处理，得到码流。核心编码处理包括但不限于：变换、量化、心理声学模型、噪声整形、带宽扩展、下混、算数编码、码流产生等。
值得注意的是，空间编码器1131可以包括虚拟扬声器配置单元310、虚拟扬声器 集合生成单元320、编码分析单元330、虚拟扬声器选择单元340和虚拟扬声器信号生成单元350，即虚拟扬声器配置单元310、虚拟扬声器集合生成单元320、编码分析单元330、虚拟扬声器选择单元340和虚拟扬声器信号生成单元350实现了空间编码器1131的功能。核心编码器1132可以包括编码单元360，即编码单元360实现了核心编码器1132的功能。
图3所示的编码器可以生成一个虚拟扬声器信号，也可以生成多个虚拟扬声器信号。多个虚拟扬声器信号可以由图3所示的编码器多次执行得到，也可以由图3所示的编码器一次执行得到。
接下来，结合附图对三维音频信号的编解码过程进行说明。图4为本申请实施例提供的一种三维音频信号编解码方法的流程示意图。在这里由图1中源设备110和目的设备120执行三维音频信号编解码过程为例进行说明。如图4所示，该方法包括以下步骤。
S410、源设备110获取三维音频信号的当前帧。
如上述实施例所述，若源设备110携带音频获取器111，源设备110可以通过音频获取器111采集原始音频。可选地，源设备110也可以接收其他设备采集的原始音频；或者从源设备110中的存储器或其他存储器获取原始音频。原始音频可以包括实时采集的现实世界声音、设备存储的音频和由多个音频合成的音频中至少一种。本实施例对原始音频的获取方式以及原始音频的类型不予限定。
源设备110获取到原始音频后，根据三维音频技术和原始音频生成三维音频信号，以便于回放原始音频时，为听音者提供“身临其境”的音响效果。生成三维音频信号的具体方法可以参考上述实施例中预处理器112的阐述和现有技术的阐述。
另外，音频信号是一个连续的模拟信号。在音频信号处理过程中，可以先对音频信号进行采样，生成帧序列的数字信号。帧可以包括多个采样点。帧也可以指采样得到的采样点。帧也可以包括对帧划分得到的子帧。帧也可以指对帧划分得到的子帧。例如一帧长度为L个采样点，划分为N个子帧，那么每个子帧对应L/N个采样点。音频编解码通常是指处理包含多个采样点的音频帧序列。
音频帧可以包括当前帧或在先帧。本申请的各个实施例所述的当前帧或在先帧可以是指帧或是子帧。当前帧是指在当前时刻进行编解码处理的帧。在先帧是指在当前时刻之前时刻已进行编解码处理的帧。在先帧可以是当前时刻的前一时刻或者前多个时刻的帧。本申请的实施例中，三维音频信号的当前帧是指在当前时刻进行编解码处理的一帧三维音频信号。在先帧是指在当前时刻之前时刻已进行编解码处理的一帧三维音频信号。三维音频信号的当前帧可以是指三维音频信号的待编码当前帧。三维音频信号的当前帧可以简称为当前帧。三维音频信号的在先帧可以简称为在先帧。
S420、源设备110确定候选虚拟扬声器集合。
在一种情形下，源设备110的存储器中预先配置有候选虚拟扬声器集合。源设备110可以从存储器中读取候选虚拟扬声器集合。候选虚拟扬声器集合包括多个虚拟扬声器。虚拟扬声器表示空间声场中虚拟存在的扬声器。虚拟扬声器用于根据三维音频信号计算虚拟扬声器信号，以便于目的设备120回放重建后三维音频信号。
在另一种情形下，源设备110的存储器中预先配置有虚拟扬声器配置参数。源设 备110根据虚拟扬声器配置参数生成候选虚拟扬声器集合。可选地，源设备110根据自身的计算资源(如：处理器)能力和当前帧的特征(如：信道和数据量)实时生成候选虚拟扬声器集合。
生成候选虚拟扬声器集合的具体方法可以参考现有技术，以及上述实施例中虚拟扬声器配置单元310和虚拟扬声器集合生成单元320的阐述。
S430、源设备110根据三维音频信号的当前帧，从候选虚拟扬声器集合中选取当前帧的代表虚拟扬声器。
源设备110根据当前帧的系数与虚拟扬声器的系数对虚拟扬声器进行投票，根据虚拟扬声器的投票值从候选虚拟扬声器集合中选择当前帧的代表虚拟扬声器。从候选虚拟扬声器集合中搜索有限数量的当前帧的代表虚拟扬声器，作为待编码的当前帧的最佳匹配虚拟扬声器，从而实现对待编码的三维音频信号进行数据压缩的目的。
图5为本申请实施例提供的一种选择虚拟扬声器方法的流程示意图。图5所述的方法流程是对图4中S430所包括的具体操作过程的阐述。在这里由图1所示的源设备110中编码器113执行选择虚拟扬声器过程为例进行说明。具体地实现虚拟扬声器选择单元340的功能。如图5所示，该方法包括以下步骤。
S510、编码器113获取当前帧的代表系数。
代表系数可以是指频域代表系数或时域代表系数。频域代表系数也可以称为频域代表频点或频谱代表系数。时域代表系数也可以称为时域代表采样点。获取当前帧的代表系数的具体方法可以参考下述图6和图7所述的S610和S620的阐述。
S520、编码器113根据当前帧的代表系数对候选虚拟扬声器集合中虚拟扬声器的投票值，从候选虚拟扬声器集合中选取当前帧的代表虚拟扬声器。执行S440至S460。
编码器113根据当前帧的代表系数与虚拟扬声器的系数对候选虚拟扬声器集合中的虚拟扬声器进行投票，根据虚拟扬声器的当前帧最终投票值从候选虚拟扬声器集合中选择(搜索)当前帧的代表虚拟扬声器。选取当前帧的代表虚拟扬声器的具体方法可以参考下述图8和图9所述的S630的阐述。
需要说明的是，编码器先遍历候选虚拟扬声器集合包含的虚拟扬声器，利用从候选虚拟扬声器集合中选取的当前帧的代表虚拟扬声器对当前帧进行压缩。但是，若连续帧选取的虚拟扬声器的结果差异较大，会导致重建后三维音频信号的声像不稳定，降低重建后三维音频信号的音质。在本申请的实施例中，编码器113可以依据在先帧的代表虚拟扬声器的在先帧最终投票值对候选虚拟扬声器集合包含的虚拟扬声器的当前帧初始投票值进行更新处理，得到虚拟扬声器的当前帧最终投票值，则根据虚拟扬声器的当前帧最终投票值从候选虚拟扬声器集合中选取当前帧的代表虚拟扬声器。从而，通过参考在先帧的代表虚拟扬声器来选取当前帧的代表虚拟扬声器，使编码器对当前帧选择当前帧的代表虚拟扬声器时倾向于选择与在先帧的代表虚拟扬声器相同的虚拟扬声器，增加连续帧之间的方位的连续性，克服了连续帧选取的虚拟扬声器的结果差异较大的问题。因此，本申请的实施例还可以包括S530。
S530、编码器113根据在先帧的代表虚拟扬声器的在先帧最终投票值调整候选虚拟扬声器集合中虚拟扬声器的当前帧初始投票值，获得虚拟扬声器的当前帧最终投票值。
编码器113根据当前帧的代表系数与虚拟扬声器的系数对候选虚拟扬声器集合中的虚拟扬声器进行投票，得到虚拟扬声器的当前帧初始投票值后，根据在先帧的代表虚拟扬声器的在先帧最终投票值调整候选虚拟扬声器集合中虚拟扬声器的当前帧初始投票值，获得虚拟扬声器的当前帧最终投票值。在先帧的代表虚拟扬声器为编码器113对在先帧进行编码时使用的虚拟扬声器。调整候选虚拟扬声器集合中虚拟扬声器的当前帧初始投票值的具体方法可以参考下述图9所述的S6302a至S6302b的阐述。
在一些实施例中，若当前帧是原始音频中第一帧，编码器113执行S510至S520。若当前帧是原始音频中第二帧以上的任意一帧，编码器113可以先判断是否复用在先帧的代表虚拟扬声器对当前帧进行编码或判断是否进行搜索虚拟扬声器，确保连续帧之间的方位的连续性，并降低编码复杂度。本申请的实施例还可以包括S540。
S540、编码器113根据在先帧的代表虚拟扬声器和当前帧判断是否进行虚拟扬声器搜索。
若编码器113确定进行虚拟扬声器搜索，执行S510至S530。可选地，编码器113可以先执行S510，即编码器113获取当前帧的代表系数，编码器113根据当前帧的代表系数和在先帧的代表虚拟扬声器的系数判断是否进行虚拟扬声器搜索，若编码器113确定进行虚拟扬声器搜索，再执行S520至S530。
若编码器113确定不进行虚拟扬声器搜索，执行S550。
S550、编码器113确定复用在先帧的代表虚拟扬声器对当前帧进行编码。
编码器113复用在先帧的代表虚拟扬声器和当前帧生成虚拟扬声器信号，对虚拟扬声器信号进行编码得到码流，向目的设备120发送码流，即执行S450和S460。
判断是否进行虚拟扬声器搜索的具体方法可以参考下述图10所述的S650至S660的阐述。
S440、源设备110根据三维音频信号的当前帧和当前帧的代表虚拟扬声器生成虚拟扬声器信号。
源设备110根据当前帧的系数和当前帧的代表虚拟扬声器的系数生成虚拟扬声器信号。生成虚拟扬声器信号的具体方法可以参考现有技术，以及上述实施例中虚拟扬声器信号生成单元350的阐述。
S450、源设备110对虚拟扬声器信号进行编码得到码流。
源设备110可以对虚拟扬声器信号进行变换或量化等编码操作，生成码流，从而实现对待编码的三维音频信号进行数据压缩的目的。生成码流的具体方法可以参考现有技术，以及上述实施例中编码单元360的阐述。
S460、源设备110向目的设备120发送码流。
源设备110可以对原始音频全部编码完成后，向目的设备120发送原始音频的码流。或者，源设备110也可以以帧为单位，实时对三维音频信号进行编码处理，对一帧编码完成后发送一帧的码流。发送码流的具体方法可以参考现有技术，以及上述实施例中通信接口114和通信接口124的阐述。
S470、目的设备120对源设备110发送的码流进行解码，重建三维音频信号，得到重建后三维音频信号。
目的设备120接收到码流后，对码流进行解码得到虚拟扬声器信号，再根据候选 虚拟扬声器集合和虚拟扬声器信号重建三维音频信号，得到重建后三维音频信号。目的设备120回放重建后三维音频信号。或者，目的设备120将重建后三维音频信号传输给其他播放设备，由其他播放设备播放重建后三维音频信号，使得听音者置身于影院、音乐厅或虚拟场景等场所的“身临其境”的音响效果更加逼真。
目前，在虚拟扬声器搜索过程中，为了衡量候选虚拟扬声器集合中每个虚拟扬声器与三维音频信号之间的关系，三维音频信号的每个系数要与每个虚拟扬声器的系数进行相关运算，对编码器造成了沉重的计算负担。本申请实施例提供一种三维音频信号的系数选择方法，编码器利用三维音频信号的代表系数与每个虚拟扬声器的系数进行相关运算选取代表虚拟扬声器，从而降低编码器搜索虚拟扬声器的计算复杂度。
接下来，结合附图对三维音频信号的系数选择方法进行详细说明。图6为本申请实施例提供的一种三维音频信号编码方法的流程示意图。在这里由图1中源设备110中编码器113执行三维音频信号的系数选择过程为例进行说明。具体地实现虚拟扬声器选择单元340的功能。其中，图6所述的方法流程是对图5中S510所包括的具体操作过程的阐述。如图6所示，该方法包括以下步骤。
S610、编码器113获取三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值。
假设三维音频信号是HOA信号，编码器113可以对HOA信号的当前帧进行采样，得到L·(N+1) 2个采样点，即得到第四数量个系数。N表示HOA信号的阶数。示例地，假设HOA信号的当前帧的时长为20毫秒，编码器113根据48KHz频率对当前帧进行采样，得到时域上的960·(N+1) 2个采样点。采样点也可以称为时域系数。
三维音频信号的当前帧的频域系数可以是根据三维音频信号的当前帧的时域系数进行时频转换得到。时域转变为频域的方法不予限定。时域转变为频域的方法例如是修正的离散余弦变换(Modified Discrete Cosine Transform，MDCT)，则可以得到频域上960·(N+1) 2个频域系数。频域系数也可以称为频谱系数或频点。
采样点的频域特征值满足p(j)＝norm(x(j))，其中，j＝1,2…L，L表示采样时刻的数量，x表示三维音频信号的当前帧的频域系数，例如MDCT系数，norm为求取二范数运算；x(j)表示第j个采样时刻的(N+1) 2个采样点的频域系数。
采样点的频域特征值还可以是HOA信号中任意声道系数。通常情况下，选择第0阶对应的声道系数。因此HOA信号的频域特征值满足p(j)＝x 0(j)，其中x 0(j)表示第0阶第j个频点的频域系数。
采样点的频域特征值还可以是HOA信号中多个声道系数的均值。因此HOA信号的频域特征值满足p(j)＝mean(x(j))，其中，mean表示求取平均运算。
S620、编码器113根据第四数量个系数的频域特征值，从第四数量个系数中选取第三数量个代表系数。
编码器113将第四数量个系数指示的频谱范围划分为至少一个子带。其中，编码器113将第四数量个系数指示的频谱范围划分为一个子带，可以理解的，该一个子带的频谱范围等于第四数量个系数指示的频谱范围，相当于编码器113未对第四数量个系数指示的频谱范围进行划分。
如果编码器113将第四数量个系数指示的频谱范围划分为至少两个子带，在一种 情形下，编码器113将第四数量个系数指示的频谱范围等分为至少两个子带，至少两个子带中每个子带包含相同数量的系数。
在另一情形下，编码器113对第四数量个系数指示的频谱范围进行不等分，划分得到的至少两个子带包含的系数的数量不同，或者划分得到的至少两个子带中每个子带包含的系数的数量均不同。例如，编码器113可以根据第四数量个系数指示的频谱范围中的低频范围、中频范围和高频范围，对第四数量个系数指示的频谱范围进行不等分，使得低频范围、中频范围和高频范围中每个频谱范围包括至少一个子带。低频范围内的至少一个子带中每个子带包含相同数量的系数。中频范围内的至少一个子带中每个子带包含相同数量的系数。高频范围内的至少一个子带中每个子带包含相同数量的系数。低频范围、中频范围和高频范围中三个频谱范围内的子带可以包含不同数量的系数。
示例地，编码器113根据心理声学模型将第四数量个系数指示的频谱范围划分为T个子带，例如T＝44。第i个子带的起始系数序号记为sfb[i]，i＝1,2…T，表示i的取值范围为1至T。第i个子带包含的系数个数记为b(i)。假设低频范围包括10个子带，b(1)＝4，表示第1个子带包含4个系数，b(10)＝4，表示第10个子带包含4个系数。中频范围包括20个子带，b(11)＝8，表示第11个子带包括8个系数；b(30)＝8，表示第30个子带包括8个系数。高频范围包括14个子带，b(31)＝16，表示第31个子带包括16个系数；b(44)＝16，表示第44个子带包括16个系数。
进一步地，编码器113根据第四数量个系数的频域特征值，从第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，得到第三数量个代表系数。第三数量小于第四数量，第四数量个系数包含第三数量个代表系数。
在一种可能的实现方式中，图7所述的方法流程是对图7中S620所包括的具体操作过程的阐述。如图7所示，该方法包括以下步骤。
S6201、编码器113根据至少一个子带的每个子带中系数的频域特征值，从每个子带中选取Z个代表系数，以得到第三数量个代表系数。Z为正整数。
例如，编码器113根据至少一个子带的每个子带中系数的频域特征值的从大到小的顺序，分别从每个子带中选取Z个代表系数，从每个子带中选取的Z个代表系数组成得到第三数量个代表系数。
示例地，编码器113对第i个子带中b(i)个系数的频域特征值进行从大到小的排序，根据第i个子带中b(i)个系数的频域特征值的从大到小的顺序，从第i个子带中最大频域特征值的系数开始，选取K(i)个代表系数。第i个子带内K(i)个代表系数对应的系数序号记为a i[j]，j＝0,…K(i)-1，表示j的取值范围为0至K(i)-1。其中，K(i)的取值可以是预先设定的，也可以是按照预定的规则产生的，例如从第i个子带中最大频域特征值的系数开始，编码器113选取系数的频域特征值最大的50％个系数作为代表系数。
在另一种可能的实现方式中，至少一个子带包括至少两个子带时，对于至少两个子带中的每个子带，编码器113可以先确定至少两个子带中每个子带的权重，利用每个子带各自的权重分别对每个子带内系数的频域特征值进行调整，再从至少两个子带中选取第三数量个代表系数。如图7所示，S620还可以包含以下步骤。
S6202、编码器113根据至少两个子带中每个子带内的第一候选系数的频域特征值 确定每个子带各自的权重。
第一候选系数可以是指子带内的部分系数。本申请实施例对第一候选系数的数量不予限定，第一候选系数的数量可以是一个或至少两个。在一些实施例中，编码器113可以根据S6201所述的方法选取第一候选系数。可理解的，编码器113根据至少两个子带的每个子带中系数的频域特征值的从大到小的顺序，从每个子带中选取Z个代表系数，分别将Z个代表系数作为每个子带的第一候选系数。例如，至少两个子带包括第一子带，将从第一子带中选取Z个代表系数作为第一子带的第一候选系数。
编码器113根据子带内的第一候选系数的频域特征值和该子带内所有系数的频域特征值确定该子带的权重。
示例地，编码器113根据第i个子带的候选系数的频域特征值和第i个子带的所有系数的频域特征值，计算第i个子带的权重w(i)。第i个子带的权重w(i)满足公式(6)。
Figure PCTCN2022091558-appb-000016
其中，p表示当前帧的系数的频域特征值，K(i)表示第i个子带的系数的数量，ai[j]表示第i个子带的第j个系数的系数序号，sfb[i]表示第i个子带的起始系数序号，b(i)表示第i个子带包含的系数数量，j＝0,…K(i)-1，i＝1,2…T。
S6203、编码器113根据每个子带各自的权重分别调整每个子带内的第二候选系数的频域特征值，得到每个子带内的第二候选系数的调整后频域特征值。
第二候选系数可以是指子带内的部分系数。本申请实施例对第二候选系数的数量不予限定，第二候选系数的数量可以是一个或至少两个。在一些实施例中，编码器113可以根据S6201所述的方法选取第二候选系数。可理解的，编码器113根据至少两个子带的每个子带中系数的频域特征值的从大到小的顺序，从每个子带中选取Z个代表系数，将Z个代表系数分别作为每个子带的第二候选系数。在该情况下，第一候选系数和第二候选系数的数量可以相同也可以不同。对于一个子带内的第一候选系数和第二候选系数，第一候选系数和第二候选系数可以是相同的系数也可以不同的系数。编码器113可以对每个子带的部分系数的频域特征值进行了调整。
第二候选系数也可以是指子带内的全部系数。在该情况下，第一候选系数和第二候选系数的数量不同。可理解的，编码器113对每个子带的全部系数的频域特征值进行了调整。
示例地，编码器113根据第i个子带的权重w(i)，调整第i个子带的K(i)个系数的频域特征值，第i个子带的K(i)个系数的调整后频域特征值满足公式(7)。
P′(a i[j])＝P(a i[j])·W(i)     公式(7)
其中，j＝1,2…K(i)。P(a i[j])表示第i个子带的第j个系数对应的频域特征值，P′(a i[j])表示第i个子带的第j个系数对应的调整后频域特征值，K(i)表示第i个子带的系数的数量，ai[j]表示第i个子带的第j个系数的系数序号，w(i)表示第i个子带的权重，j＝0,…K(i)-1，i＝1,2…T。
S6204、编码器113根据至少两个子带内的第二候选系数的调整后频域特征值，以及至少两个子带内除第二候选系数之外的系数的频域特征值，确定第三数量个代表系数。
编码器113对至少两个子带内的所有系数的频域特征值进行从大到小的排序，根 据至少两个子带内的所有系数的频域特征值的从大到小的顺序，从至少两个子带中最大频域特征值的系数开始，选取第三数量个代表系数。
可理解的，如果第二候选系数是子带内的部分系数，至少两个子带内的所有系数的频域特征值包括第二候选系数的调整后频域特征值，以及至少两个子带内除第二候选系数之外的系数的频域特征值。编码器113根据至少两个子带内的第二候选系数的调整后频域特征值，以及至少两个子带内除第二候选系数之外的系数的频域特征值，确定第三数量个代表系数。
如果第二候选系数是子带内的全部系数，至少两个子带内的所有系数的频域特征值为第二候选系数的调整后频域特征值。编码器113根据至少两个子带内的第二候选系数的调整后频域特征值，确定第三数量个代表系数。
第三数量可以是预先设定的，也可以是按照预定的规则生成的，例如编码器113选取至少两个子带内的所有系数的频域特征值最大的20％个系数作为代表频点。
S630、编码器113根据第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器。
编码器113利用三维音频信号的当前帧的第三数量个代表系数与候选虚拟扬声器集合中每个虚拟扬声器的系数进行相关运算，选取第二数量个当前帧的代表虚拟扬声器。
由于编码器从当前帧的全部系数中选取部分系数作为代表系数，利用较少数量的代表系数代替当前帧的全部系数从候选虚拟扬声器集合中选取代表虚拟扬声器，因此有效地降低了编码器搜索虚拟扬声器的计算复杂度，从而降低了对三维音频信号进行压缩编码的计算复杂度以及减轻了编码器的计算负担。例如一帧N阶的HOA信号有960·(N+1) 2个系数，本实施例可以选取前10％的系数参与虚拟扬声器搜索，此时编码复杂度相较于全系数参与虚拟扬声器搜索的编码复杂度降低了90％。
S640、编码器113根据第二数量个当前帧的代表虚拟扬声器对当前帧进行编码，得到码流。
编码器113根据第二数量个当前帧的代表虚拟扬声器和当前帧生成虚拟扬声器信号，以及对虚拟扬声器信号进行编码得到码流。生成码流的具体方法可以参考现有技术，以及上述实施例中编码单元360和S450的阐述。
编码器113在生成码流后，向目的设备120发送码流，以便于目的设备120对源设备110发送的码流进行解码，重建三维音频信号，得到重建后三维音频信号。
由于当前帧的系数的频域特征值表征了三维音频信号的声场特性，编码器依据当前帧的系数的频域特征值选取当前帧的具有代表性声场成分的代表系数，利用代表系数从候选虚拟扬声器集合中选取的当前帧的代表虚拟扬声器能够充分地表征三维音频信号的声场特性，从而进一步地提高了编码器利用当前帧的代表虚拟扬声器对待编码的三维音频信号进行压缩编码时生成虚拟扬声器信号的准确性，以便于提升对三维音频信号进行压缩编码的压缩率，降低编码器传输码流所占用的带宽。
本申请的实施例中，编码器113可以根据当前帧的第三数量个代表系数对候选虚拟扬声器集合中虚拟扬声器的投票值，选择第二数量个当前帧的代表虚拟扬声器。图8所述的方法流程是对图7中S630所包括的具体操作过程的阐述。如图8所示，该方 法包括以下步骤。
S6301、编码器113根据当前帧的第三数量个代表系数、候选虚拟扬声器集合和投票轮数确定第一数量个虚拟扬声器和第一数量个投票值。
投票轮数用于限定对虚拟扬声器进行投票的次数。投票轮数为大于或等于1的整数，且投票轮数小于或等于候选虚拟扬声器集合包含的虚拟扬声器的数量，以及投票轮数小于或等于编码器传输的虚拟扬声器信号的数量。例如，候选虚拟扬声器集合包括第五数量个虚拟扬声器，第五数量个虚拟扬声器包括第一数量个虚拟扬声器，第一数量小于或等于第五数量，投票轮数为大于或等于1的整数，且投票轮数小于或等于所述第五数量。虚拟扬声器信号也是指当前帧对应的当前帧的代表虚拟扬声器的传输通道。通常情况下虚拟扬声器信号的数量小于或等于虚拟扬声器的数量。
在一种可能的实现方式中，投票轮数可以是预先配置的，也可以是根据编码器的计算能力确定的，比如，投票轮数是根据编码器的编码速率和/或编码应用场景确定的。
在另一种可能的实现方式中，投票轮数是根据当前帧中指向性声源的数量确定的。例如，当声场中指向性声源数量为2时，设置投票轮数为2。
本申请实施例提供了确定第一数量个虚拟扬声器和第一数量个投票值的三种可能实现方式，下面对三种方式分别进行详述。
在第一种可能的实现方式中，投票轮数等于1，编码器113采样到多个代表系数后，获取当前帧的每个代表系数对候选虚拟扬声器集合中所有虚拟扬声器的投票值，累加相同编号的虚拟扬声器的投票值，得到第一数量个虚拟扬声器和第一数量个投票值。可理解的，候选虚拟扬声器集合包括第一数量个虚拟扬声器。第一数量等于候选虚拟扬声器集合包括的虚拟扬声器的数量。假设候选虚拟扬声器集合包括第五数量个虚拟扬声器，则第一数量等于第五数量。第一数量个投票值包括候选虚拟扬声器集合中所有虚拟扬声器的投票值。编码器113可以将第一数量个投票值作为第一数量个虚拟扬声器的当前帧最终投票值，执行S6302，即编码器113根据第一数量个投票值，从第一数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器。
其中，虚拟扬声器与投票值一一对应，即一个虚拟扬声器对应一个投票值。例如，第一数量个虚拟扬声器包括第一虚拟扬声器，第一数量个投票值包括第一虚拟扬声器的投票值，第一虚拟扬声器与第一虚拟扬声器的投票值对应。第一虚拟扬声器的投票值用于表征第一虚拟扬声器的优先级。优先级也可以替换描述为倾向性，即第一虚拟扬声器的投票值用于表征对当前帧进行编码时使用第一虚拟扬声器的倾向性。可理解的，第一虚拟扬声器的投票值越大，表示第一虚拟扬声器的优先级越高或倾向性越高，相对于候选虚拟扬声器集合中比第一虚拟扬声器的投票值小的虚拟扬声器，编码器113更倾向选择第一虚拟扬声器对当前帧进行编码。
在第二种可能的实现方式中，与上述第一种可能的实现方式的区别在于，编码器113获取当前帧的每个代表系数对候选虚拟扬声器集合中所有虚拟扬声器的投票值后，从每个代表系数对候选虚拟扬声器集合中所有虚拟扬声器的投票值中选取部分投票值，累加部分投票值对应的虚拟扬声器中相同编号的虚拟扬声器的投票值，得到第一数量个虚拟扬声器和第一数量个投票值。可理解的，候选虚拟扬声器集合包括第一数量个虚拟扬声器。第一数量小于或等于候选虚拟扬声器集合包括的虚拟扬声器的数量。第 一数量个投票值包括候选虚拟扬声器集合包括的部分虚拟扬声器的投票值，或者，第一数量个投票值包括候选虚拟扬声器集合包括的全部虚拟扬声器的投票值。
在第三种可能的实现方式中，与上述第二种可能的实现方式的区别在于，投票轮数为大于或等于2的整数，对于当前帧的每个代表系数，编码器113对候选虚拟扬声器集合中所有虚拟扬声器进行至少2轮次投票，每轮选择最大投票值的虚拟扬声器。对当前帧的每个代表系数对所有虚拟扬声器进行至少2轮次投票后，累加相同编号的虚拟扬声器的投票值，得到第一数量个虚拟扬声器和第一数量个投票值。
S6302、编码器113根据第一数量个投票值，从第一数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器。
编码器113根据第一数量个投票值，从第一数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器，而且，第二数量个当前帧的代表虚拟扬声器的投票值大于预设阈值。
编码器113也可以根据第一数量个投票值，从第一数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器。例如，按照第一数量个投票值的从大到小的顺序，从第一数量个投票值中确定第二数量个投票值，并将第一数量个虚拟扬声器中与第二数量个投票值对应的虚拟扬声器作为第二数量个当前帧的代表虚拟扬声器。
可选地，若第一数量个虚拟扬声器中不同编号的虚拟扬声器的投票值相同，且该不同编号的虚拟扬声器的投票值大于预设阈值，则编码器113可以将不同编号的虚拟扬声器均作为当前帧的代表虚拟扬声器。
需要说明的是，第二数量小于第一数量。第一数量个虚拟扬声器包括第二数量个当前帧的代表虚拟扬声器。第二数量可以是预设的，或者，第二数量可以是根据当前帧的声场中声源的数量确定的，例如，第二数量可以直接等于当前帧的声场中声源的数量，或者是按照预设算法对当前帧的声场中声源的数量进行处理，将处理得到的数量作为第二数量；其中，预设算法可以根据需要进行设计，例如，预设算法可以是：第二数量＝当前帧的声场中声源的数量+1，或第二数量＝当前帧的声场中声源的数量-1等等。
由于编码器利用较少数量的代表系数代替当前帧的全部系数对候选虚拟扬声器集合中每个虚拟扬声器进行投票，依据投票值选取当前帧的代表虚拟扬声器。进而，编码器利用当前帧的代表虚拟扬声器对待编码的三维音频信号进行压缩编码，不仅有效地提升了对三维音频信号进行压缩编码的压缩率，而且降低了编码器搜索虚拟扬声器的计算复杂度，从而降低了对三维音频信号进行压缩编码的计算复杂度以及减轻了编码器的计算负担。
为了增加连续帧之间的方位的连续性，克服了连续帧选取的虚拟扬声器的结果差异较大的问题，编码器113根据在先帧的代表虚拟扬声器的在先帧最终投票值调整候选虚拟扬声器集合中虚拟扬声器的当前帧初始投票值，获得虚拟扬声器的当前帧最终投票值。如图9所示，为本申请实施例提供的另一种选择虚拟扬声器方法的流程示意图。其中，图9所述的方法流程是对图8中S6302所包括的具体操作过程的阐述。
S6302a、编码器113根据第一数量个当前帧初始投票值，以及第六数量个在先帧最终投票值，获取第七数量个虚拟扬声器与当前帧对应的第七数量个当前帧最终投票 值。
编码器113可以依据上述S6301所述的方法，根据三维音频信号的当前帧、候选虚拟扬声器集合和投票轮数确定第一数量个虚拟扬声器和第一数量个投票值，进而，将第一数量个投票值作为第一数量个虚拟扬声器的当前帧初始投票值。
虚拟扬声器与当前帧初始投票值一一对应，即一个虚拟扬声器对应一个当前帧初始投票值。例如，第一数量个虚拟扬声器包括第一虚拟扬声器，第一数量个当前帧初始投票值包括第一虚拟扬声器的当前帧初始投票值，第一虚拟扬声器与第一虚拟扬声器的当前帧初始投票值对应。第一虚拟扬声器的当前帧初始投票值用于表征对当前帧进行编码时使用第一虚拟扬声器的优先级。
在先帧的代表虚拟扬声器集合包含的第六数量个虚拟扬声器与第六数量个在先帧最终投票值一一对应。第六数量个虚拟扬声器可以是编码器113对三维音频信号的在先帧进行编码所使用的在先帧的代表虚拟扬声器。
具体地，编码器113根据第六数量个在先帧最终投票值，更新第一数量个当前帧初始投票值，即编码器113计算第一数量个虚拟扬声器与第六数量个虚拟扬声器中相同编号的虚拟扬声器的当前帧初始投票值和在先帧最终投票值之和，获取第七数量个虚拟扬声器与当前帧对应的第七数量个当前帧最终投票值。第七数量个虚拟扬声器包括第一数量个虚拟扬声器，且第七数量个虚拟扬声器包括第六数量个虚拟扬声器。
S6302b、编码器113根据第七数量个当前帧最终投票值，从第七数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器。
编码器113根据第七数量个当前帧最终投票值，从第七数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器，而且，第二数量个当前帧的代表虚拟扬声器的当前帧最终投票值大于预设阈值。
编码器113也可以根据第七数量个当前帧最终投票值，从第七数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器。例如，按照第七数量个当前帧最终投票值的从大到小的顺序，从第七数量个当前帧最终投票值中确定第二数量个当前帧最终投票值，并将第七数量个虚拟扬声器中与第二数量个当前帧最终投票值关联的虚拟扬声器作为第二数量个当前帧的代表虚拟扬声器。
可选地，若第七数量个虚拟扬声器中不同编号的虚拟扬声器的投票值相同，且该不同编号的虚拟扬声器的投票值大于预设阈值，则编码器113可以将该不同编号的虚拟扬声器均作为当前帧的代表虚拟扬声器。
需要说明的是，第二数量小于第七数量。第七数量个虚拟扬声器包括第二数量个当前帧的代表虚拟扬声器。第二数量可以是预设的，或者，第二数量可以是根据当前帧的声场中声源的数量确定的。
另外，编码器113对当前帧的下一帧进行编码前，如果编码器113确定复用在先帧的代表虚拟扬声器对下一帧进行编码，编码器113可以将第二数量个当前帧的代表虚拟扬声器作为第二数量个在先帧的代表虚拟扬声器，利用第二数量个在先帧的代表虚拟扬声器对当前帧的下一帧进行编码。
在虚拟扬声器搜索过程中，由于真实声源的位置与虚拟扬声器的位置不一定重合，会导致虚拟扬声器不一定能够与真实声源形成一一对应关系，且由于在实际的复杂场 景下，可能出现虚拟扬声器无法表征声场中独立声源情况，此时，帧与帧之间搜索到的虚拟扬声器可能会发生频繁跳变，这种频繁跳变会明显地影响听音者的听觉感受，导致解码重建后三维音频信号中出现明显的不连续和噪声。本申请的实施例提供的选择虚拟扬声器的方法通过继承在先帧的代表虚拟扬声器，即对于相同编号的虚拟扬声器，用在先帧最终投票值调整当前帧初始投票值，使得编码器更倾向于选择在先帧的代表虚拟扬声器，从而降低帧与帧之间的虚拟扬声器的频繁跳变，增强了帧之间的方位的连续性，提高了重建后三维音频信号的声像的稳定性，确保重建后三维音频信号的音质。另外调整参数确保在先帧最终投票值不会继承太过久远，避免算法无法适应声源移动等声场变化的场景。
此外，本申请实施例提供还一种选择虚拟扬声器的方法，编码器可以先判断是否可以复用在先帧的代表虚拟扬声器集合对当前帧进行编码，如果编码器复用在先帧的代表虚拟扬声器集合对当前帧进行编码，从而，避免编码器再执行虚拟扬声器搜索过程，有效地降低了编码器搜索虚拟扬声器的计算复杂度，因此降低了对三维音频信号进行压缩编码的计算复杂度以及减轻了编码器的计算负担。如果编码器不能复用在先帧的代表虚拟扬声器集合对当前帧进行编码，编码器再选取代表系数，利用当前帧的代表系数对候选虚拟扬声器集合中每个虚拟扬声器进行投票，依据投票值选取当前帧的代表虚拟扬声器，来达到降低对三维音频信号进行压缩编码的计算复杂度以及减轻了编码器的计算负担的目的。图10为本申请实施例提供的一种选择虚拟扬声器方法的流程示意图。在编码器113获取三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值，即S610之前，如图10所示，该方法包括以下步骤。
S650、编码器113获取三维音频信号的当前帧与在先帧的代表虚拟扬声器集合的第一相关度。
在先帧的代表虚拟扬声器集合包含的第六数量个虚拟扬声器，第六数量个虚拟扬声器包含的虚拟扬声器为对三维音频信号的在先帧进行编码所使用的在先帧的代表虚拟扬声器。第一相关度用于表征对当前帧进行编码时复用在先帧的代表虚拟扬声器集合的优先级。优先级也可以替换描述为倾向性，即第一相关度用于确定对当前帧进行编码时是否复用在先帧的代表虚拟扬声器集合。可理解的，在先帧的代表虚拟扬声器集合的第一相关度越大，表示在先帧的代表虚拟扬声器集合的倾向性越高，编码器113更倾向选择在先帧的代表虚拟扬声器对当前帧进行编码。
S660、编码器113判断第一相关度是否满足复用条件。
若第一相关度不满足复用条件，表示编码器113更倾向进行虚拟扬声器搜索，根据当前帧的代表虚拟扬声器对当前进行编码，执行S610，编码器113获取三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值。
可选地，编码器113也可以在根据第四数量个系数的频域特征值，从第四数量个系数中选取第三数量个代表系数之后，将第三数量个代表系数中最大的代表系数作为获取第一相关度的当前帧的系数，则编码器113获取当前帧的第三数量个代表系数中最大的代表系数与在先帧的代表虚拟扬声器集合的第一相关度，若第一相关度不满足复用条件，执行S630，即编码器113根据第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器。
若第一相关度满足复用条件，表示编码器113更倾向选择在先帧的代表虚拟扬声器对当前帧进行编码，编码器113执行S670和S680。
S670、编码器113根据在先帧的代表虚拟扬声器集合和当前帧生成虚拟扬声器信号。
S680、编码器113对虚拟扬声器信号进行编码，得到码流。
本申请实施例提供的选择虚拟扬声器的方法，利用当前帧的代表系数与在先帧的代表虚拟扬声器的相关度判断是否进行虚拟扬声器搜索，在确保当前帧的代表虚拟扬声器的相关度的选择准确度情况下，有效地降低了编码端的复杂度。
可以理解的是，为了实现上述实施例中的功能，编码器包括了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本申请中所公开的实施例描述的各示例的单元及方法步骤，本申请能够以硬件或硬件和计算机软件相结合的形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用场景和设计约束条件。
上文中结合图1至图10，详细描述了根据本实施例所提供的三维音频信号编码方法，下面将结合图11和图12，描述根据本实施例所提供的三维音频信号编码装置和编码器。
图11为本实施例提供的可能的三维音频信号编码装置的结构示意图。这些三维音频信号编码装置可以用于实现上述方法实施例中编码三维音频信号的功能，因此也能实现上述方法实施例所具备的有益效果。在本实施例中，该三维音频信号编码装置可以是如图1所示的编码器113，或者如图3所示的编码器300，还可以是应用于终端设备或服务器的模块(如芯片)。
如图11所示，三维音频信号编码装置1100包括通信模块1110、系数选择模块1120、虚拟扬声器选择模块1130、编码模块1140和存储模块1150。三维音频信号编码装置1100用于实现上述图6至图10中所示的方法实施例中编码器113的功能。
通信模块1110用于获取三维音频信号的当前帧。可选地，通信模块1110也可以接收其他设备获取的三维音频信号的当前帧；或者从存储模块1150获取三维音频信号的当前帧。三维音频信号的当前帧为HOA信号；系数的频域特征值是依据二维向量确定的，二维向量包括HOA信号的HOA系数。
系数选择模块1120用于获取三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值。
系数选择模块1120还用于根据第四数量个系数的频域特征值，从第四数量个系数中选取第三数量个代表系数，第三数量小于第四数量。
当三维音频信号编码装置1100用于实现图6至图10所示的方法实施例中编码器113的功能时，系数选择模块1120用于实现S610和S620的相关功能。
具体地，系数选择模块1120具体用于根据第四数量个系数的频域特征值，从第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，得到第三数量个代表系数。其中，至少两个子带包含的系数的数量不同；或者，至少两个子带中每个子带包含的系数的数量相同。
例如，系数选择模块1120具体用于根据每个子带中系数的频域特征值，从每个子 带中选取Z个代表系数，得到第三数量个代表系数，Z为正整数。
又如，当至少一个子带包括至少两个子带时，系数选择模块1120具体用于根据至少两个子带中每个子带内的第一候选系数的频域特征值确定每个子带的权重；根据每个子带各自的权重分别调整每个子带内的第二候选系数的频域特征值，得到每个子带内的第二候选系数的调整后频域特征值，第一候选系数和第二候选系数为子带内的部分系数；根据至少两个子带内的第二候选系数的调整后频域特征值，以及至少两个子带内除第二候选系数之外的系数的频域特征值，确定第三数量个代表系数。
虚拟扬声器选择模块1130用于根据第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器。
当三维音频信号编码装置1100用于实现图6至图10所示的方法实施例中编码器113的功能时，虚拟扬声器选择模块1130用于实现S630的相关功能。
示例地，虚拟扬声器选择模块1130具体用于根据当前帧的第三数量个代表系数、候选虚拟扬声器集合和投票轮数确定第一数量个虚拟扬声器和第一数量个投票值，虚拟扬声器与投票值一一对应，第一数量个虚拟扬声器包括第一虚拟扬声器，第一数量个投票值包括第一虚拟扬声器的投票值，第一虚拟扬声器与第一虚拟扬声器的投票值对应，第一虚拟扬声器的投票值用于表征对当前帧进行编码时使用第一虚拟扬声器的优先级，候选虚拟扬声器集合包括第五数量个虚拟扬声器，第五数量个虚拟扬声器包括第一数量个虚拟扬声器，投票轮数为大于或等于1的整数，且投票轮数小于或等于第五数量；根据第一数量个投票值，从第一数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器，第二数量小于第一数量。
可选地，虚拟扬声器选择模块1130还用于根据第一数量个投票值，以及第六数量个在先帧最终投票值，获取第七数量个虚拟扬声器与当前帧对应的第七数量个当前帧最终投票值，第七数量个虚拟扬声器包括第一数量个虚拟扬声器，且第七数量个虚拟扬声器包括第六数量个虚拟扬声器，第六数量个虚拟扬声器包含的虚拟扬声器为对三维音频信号的在先帧进行编码所使用的在先帧的代表虚拟扬声器；根据第七数量个当前帧最终投票值，从第七数量个虚拟扬声器中选取第二数量个当前帧的代表虚拟扬声器，第二数量小于第七数量。
可选地，虚拟扬声器选择模块1130还用于获取当前帧与在先帧的代表虚拟扬声器集合的第一相关度，在先帧的代表虚拟扬声器集合包括第六数量个虚拟扬声器，第六数量个虚拟扬声器包含的虚拟扬声器为对三维音频信号的在先帧进行编码所使用的在先帧的代表虚拟扬声器，第一相关度用于确定对当前帧进行编码时是否复用在先帧的代表虚拟扬声器集合；若第一相关度不满足复用条件，获取三维音频信号的当前帧的第四数量个系数，以及第四数量个系数的频域特征值。
编码模块1140用于根据第二数量个当前帧的代表虚拟扬声器对当前帧进行编码，得到码流。
当三维音频信号编码装置1100用于实现图6至图10所示的方法实施例中编码器113的功能时，编码模块1140用于实现S640的相关功能。
示例地，编码模块1140具体用于根据第二数量个当前帧的代表虚拟扬声器和当前帧生成虚拟扬声器信号；对虚拟扬声器信号进行编码得到码流。
存储模块1150用于存储与三维音频信号相关的系数，候选虚拟扬声器集合，在先帧的代表虚拟扬声器集合，以及选取的系数和虚拟扬声器等，以便于编码模块1140对当前帧进行编码得到码流，并将码流传输至解码器。
应理解的是，本申请实施例的三维音频信号编码装置1100可以通过专用集成电路(application-specific integrated circuit，ASIC)实现，或可编程逻辑器件(programmable logic device，PLD)实现，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)，现场可编程门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic，GAL)或其任意组合。也可以通过软件实现图6至图10所示的三维音频信号编码方法时，三维音频信号编码装置1100及其各个模块也可以为软件模块。
有关上述通信模块1110、系数选择模块1120、虚拟扬声器选择模块1130、编码模块1140和存储模块1150更详细的描述可以参考图6至图10所示的方法实施例中相关描述直接得到，这里不加赘述。
图12为本实施例提供的一种编码器1200的结构示意图。如图12所示，编码器1200包括处理器1210、总线1220、存储器1230和通信接口1240。
应理解，在本实施例中，处理器1210可以是中央处理器(central processing unit，CPU)，该处理器1210还可以是其他通用处理器、数字信号处理器(digital signal processing，DSP)、ASIC、FPGA或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者是任何常规的处理器等。
处理器还可以是图形处理器(graphics processing unit，GPU)、神经网络处理器(neural network processing unit，NPU)、微处理器或一个或多个用于控制本申请方案程序执行的集成电路。
通信接口1240用于实现编码器1200与外部设备或器件的通信。在本实施例中，通信接口1240用于接收三维音频信号。
总线1220可以包括一通路，用于在上述组件(如处理器1210和存储器1230)之间传送信息。总线1220除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都标为总线1220。
作为一个示例，编码器1200可以包括多个处理器。处理器可以是一个多核(multi-CPU)处理器。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(例如计算机程序指令)的计算单元。处理器1210可以调用存储器1230存储的与三维音频信号相关的系数，候选虚拟扬声器集合，在先帧的代表虚拟扬声器集合，以及选取的系数和虚拟扬声器等。
值得说明的是，图12中仅以编码器1200包括1个处理器1210和1个存储器1230为例，此处，处理器1210和存储器1230分别用于指示一类器件或设备，具体实施例中，可以根据业务需求确定每种类型的器件或设备的数量。
存储器1230可以对应上述方法实施例中用于存储与三维音频信号相关的系数，候选虚拟扬声器集合，在先帧的代表虚拟扬声器集合，以及选取的系数和虚拟扬声器等信息的存储介质，例如，磁盘，如机械硬盘或固态硬盘。
上述编码器1200可以是一个通用设备或者是一个专用设备。例如，编码器1200 可以是基于X86、ARM的服务器，也可以为其他的专用服务器，如策略控制和计费(policy control and charging，PCC)服务器等。本申请实施例不限定编码器1200的类型。
应理解，根据本实施例的编码器1200可对应于本实施例中的三维音频信号编码装置1100，并可以对应于执行根据图6至图10中任一方法中的相应主体，并且三维音频信号编码装置1100中的各个模块的上述和其它操作和/或功能分别为了实现图6至图10中的各个方法的相应流程，为了简洁，在此不再赘述。
本实施例中的方法步骤可以通过硬件的方式来实现，也可以由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于网络设备或终端设备中。当然，处理器和存储介质也可以作为分立组件存在于网络设备或终端设备中。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机程序或指令。在计算机上加载和执行所述计算机程序或指令时，全部或部分地执行本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、网络设备、用户设备或者其它可编程装置。所述计算机程序或指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机程序或指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，数字视频光盘(digital video disc，DVD)；还可以是半导体介质，例如，固态硬盘(solid state drive，SSD)。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (21)

  1. 一种三维音频信号编码方法，其特征在于，包括：
    获取三维音频信号的当前帧的第四数量个系数，以及所述第四数量个系数的频域特征值；
    根据所述第四数量个系数的频域特征值，从所述第四数量个系数中选取第三数量个代表系数，所述第三数量小于所述第四数量；
    根据所述第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器；
    根据所述第二数量个当前帧的代表虚拟扬声器对所述当前帧进行编码，得到码流。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述第四数量个系数的频域特征值，从所述第四数量个系数中选取第三数量个代表系数包括：
    根据所述第四数量个系数的频域特征值，从所述第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到所述第三数量个代表系数。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述第四数量个系数的频域特征值，从所述第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到所述第三数量个代表系数包括：
    根据所述至少一个子带中每个子带中系数的频域特征值，从所述每个子带中分别选取Z个代表系数，以得到所述第三数量个代表系数，Z为正整数。
  4. 根据权利要求2所述的方法，其特征在于，当所述至少一个子带包括至少两个子带时，所述根据所述第四数量个系数的频域特征值，从所述第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到所述第三数量个代表系数包括：
    根据所述至少两个子带中每个子带内的第一候选系数的频域特征值确定所述每个子带各自的权重；
    根据所述每个子带各自的权重分别调整所述每个子带内的第二候选系数的频域特征值，得到所述每个子带内的第二候选系数的调整后频域特征值，所述第一候选系数和所述第二候选系数为所述子带内的部分系数；
    根据所述至少两个子带内的第二候选系数的调整后频域特征值，以及所述至少两个子带内除第二候选系数之外的系数的频域特征值，确定所述第三数量个代表系数。
  5. 根据权利要求1-4中任一项所述的方法，其特征在于，所述根据所述第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器包括：
    根据所述当前帧的第三数量个代表系数、所述候选虚拟扬声器集合和投票轮数确定第一数量个虚拟扬声器和第一数量个投票值，所述虚拟扬声器与所述投票值一一对应，所述第一数量个虚拟扬声器包括第一虚拟扬声器，所述第一虚拟扬声器的投票值用于表征所述第一虚拟扬声器的优先级，所述候选虚拟扬声器集合包括第五数量个虚拟扬声器，所述第五数量个虚拟扬声器包括所述第一数量个虚拟扬声器，所述第一数量小于或等于所述第五数量，所述投票轮数为大于或等于1的整数，且所述投票轮数小于或等于所述第五数量；
    根据所述第一数量个投票值，从所述第一数量个虚拟扬声器中选取所述第二数量个当前帧的代表虚拟扬声器，所述第二数量小于所述第一数量。
  6. 根据权利要求5所述的方法，其特征在于，所述根据所述第一数量个投票值，从所述第一数量个虚拟扬声器中选取所述第二数量个当前帧的代表虚拟扬声器包括：
    根据所述第一数量个投票值，以及第六数量个在先帧最终投票值，获取第七数量个虚拟扬声器与所述当前帧对应的第七数量个当前帧最终投票值，所述第七数量个虚拟扬声器包括所述第一数量个虚拟扬声器，且所述第七数量个虚拟扬声器包括所述第六数量个虚拟扬声器，在先帧的代表虚拟扬声器集合包含的第六数量个虚拟扬声器与所述第六数量个在先帧最终投票值一一对应，所述第六数量个虚拟扬声器是用于对所述三维音频信号的在先帧进行编码时使用的虚拟扬声器；
    根据所述第七数量个当前帧最终投票值，从所述第七数量个虚拟扬声器中选取所述第二数量个当前帧的代表虚拟扬声器，所述第二数量小于所述第七数量。
  7. 根据权利要求1-6中任一项所述的方法，其特征在于，所述方法还包括：
    获取所述当前帧与在先帧的代表虚拟扬声器集合的第一相关度，所述在先帧的代表虚拟扬声器集合包括第六数量个虚拟扬声器，所述第六数量个虚拟扬声器包含的虚拟扬声器为对所述三维音频信号的在先帧进行编码所使用的在先帧的代表虚拟扬声器，所述第一相关度用于确定对所述当前帧进行编码时是否复用所述在先帧的代表虚拟扬声器集合；
    若所述第一相关度不满足复用条件，获取所述三维音频信号的当前帧的第四数量个系数，以及所述第四数量个系数的频域特征值。
  8. 根据权利要求1-7中任一项所述的方法，其特征在于，所述三维音频信号的当前帧为高阶立体混响HOA信号；所述系数的频域特征值是依据HOA信号的系数确定的。
  9. 一种三维音频信号编码装置，其特征在于，包括：
    系数选择模块，用于获取三维音频信号的当前帧的第四数量个系数，以及所述第四数量个系数的频域特征值；
    所述系数选择模块，还用于根据所述第四数量个系数的频域特征值，从所述第四数量个系数中选取第三数量个代表系数，所述第三数量小于所述第四数量；
    虚拟扬声器选择模块，用于根据所述第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器；
    编码模块，用于根据所述第二数量个当前帧的代表虚拟扬声器对所述当前帧进行编码，得到码流。
  10. 根据权利要求9所述的装置，其特征在于，所述系数选择模块根据所述第四数量个系数的频域特征值，从所述第四数量个系数中选取第三数量个代表系数时，具体用于：
    根据所述第四数量个系数的频域特征值，从所述第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到所述第三数量个代表系数。
  11. 根据权利要求10所述的装置，其特征在于，所述系数选择模块根据所述第四数量个系数的频域特征值，从所述第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到所述第三数量个代表系数时，具体用于：
    根据所述至少一个子带中每个子带中系数的频域特征值，从所述每个子带中分别 选取Z个代表系数，以得到所述第三数量个代表系数，Z为正整数。
  12. 根据权利要求10所述的装置，其特征在于，当所述至少一个子带包括至少两个子带时，所述系数选择模块根据所述第四数量个系数的频域特征值，从所述第四数量个系数指示的频谱范围包含的至少一个子带选取代表系数，以得到所述第三数量个代表系数时，具体用于：
    根据所述至少两个子带中每个子带内的第一候选系数的频域特征值确定所述每个子带各自的权重；
    根据所述每个子带各自的权重分别调整所述每个子带内的第二候选系数的频域特征值，得到所述每个子带内的第二候选系数的调整后频域特征值，所述第一候选系数和所述第二候选系数为所述子带内的部分系数；
    根据所述至少两个子带内的第二候选系数的调整后频域特征值，以及所述至少两个子带内除第二候选系数之外的系数的频域特征值，确定所述第三数量个代表系数。
  13. 根据权利要求9-12中任一项所述的装置，其特征在于，所述虚拟扬声器选择模块根据所述第三数量个代表系数从候选虚拟扬声器集合中选取第二数量个当前帧的代表虚拟扬声器时，具体用于：
    根据所述当前帧的第三数量个代表系数、所述候选虚拟扬声器集合和投票轮数确定第一数量个虚拟扬声器和第一数量个投票值，所述虚拟扬声器与所述投票值一一对应，所述第一数量个虚拟扬声器包括第一虚拟扬声器，所述第一虚拟扬声器的投票值用于表征所述第一虚拟扬声器的优先级，所述候选虚拟扬声器集合包括第五数量个虚拟扬声器，所述第五数量个虚拟扬声器包括所述第一数量个虚拟扬声器，所述第一数量小于或等于所述第五数量，所述投票轮数为大于或等于1的整数，且所述投票轮数小于或等于所述第五数量；
    根据所述第一数量个投票值，从所述第一数量个虚拟扬声器中选取所述第二数量个当前帧的代表虚拟扬声器，所述第二数量小于所述第一数量。
  14. 根据权利要求13所述的装置，其特征在于，所述虚拟扬声器选择模块根据所述第一数量个投票值，从所述第一数量个虚拟扬声器中选取所述第二数量个当前帧的代表虚拟扬声器时，具体用于：
    根据所述第一数量个投票值，以及第六数量个在先帧最终投票值，获取第七数量个虚拟扬声器与所述当前帧对应的第七数量个当前帧最终投票值，所述第七数量个虚拟扬声器包括所述第一数量个虚拟扬声器，且所述第七数量个虚拟扬声器包括所述第六数量个虚拟扬声器，在先帧的代表虚拟扬声器集合包含的第六数量个虚拟扬声器与所述第六数量个在先帧最终投票值一一对应，所述第六数量个虚拟扬声器是用于对所述三维音频信号的在先帧进行编码时使用的虚拟扬声器；
    根据所述第七数量个当前帧最终投票值，从所述第七数量个虚拟扬声器中选取所述第二数量个当前帧的代表虚拟扬声器，所述第二数量小于所述第七数量。
  15. 根据权利要求9-14中任一项所述的装置，其特征在于，所述虚拟扬声器选择模块，还用于：
    获取所述当前帧与在先帧的代表虚拟扬声器集合的第一相关度，所述在先帧的代表虚拟扬声器集合包括第六数量个虚拟扬声器，所述第六数量个虚拟扬声器包含的虚 拟扬声器为对所述三维音频信号的在先帧进行编码所使用的在先帧的代表虚拟扬声器，所述第一相关度用于确定对所述当前帧进行编码时是否复用所述在先帧的代表虚拟扬声器集合；
    若所述第一相关度不满足复用条件，获取所述三维音频信号的当前帧的第四数量个系数，以及所述第四数量个系数的频域特征值。
  16. 根据权利要求9-15中任一项所述的装置，其特征在于，所述三维音频信号的当前帧为高阶立体混响HOA信号；所述系数的频域特征值是依据HOA信号的系数确定的。
  17. 一种编码器，其特征在于，所述编码器包括至少一个处理器和存储器，其中，所述存储器用于存储计算机程序，使得所述计算机程序被所述至少一个处理器执行时实现如权利要求1-8中任一项所述的三维音频信号编码方法。
  18. 一种系统，其特征在于，所述系统包括如权利要求17所述的编码器，以及解码器，所述编码器用于执行上述权利要求1-8中任一项所述的方法的操作步骤，所述解码器用于解码所述编码器生成的码流。
  19. 一种计算机程序，其特征在于，所述计算机程序被执行时实现如权利要求1-8中任一项所述的三维音频信号编码方法。
  20. 一种计算机可读存储介质，其特征在于，包括计算机软件指令；当计算机软件指令在编码器中运行时，使得所述编码器执行如权利要求1-8中任一项所述的三维音频信号编码方法。
  21. 一种计算机可读存储介质，其特征在于，包括如权利要求1-8中任一项所述的三维音频信号编码方法所获得的码流。
PCT/CN2022/091558 2021-05-17 2022-05-07 三维音频信号编码方法、装置和编码器 Ceased WO2022242480A1 (zh)

Priority Applications (6)

Application Number Priority Date Filing Date Title
BR112023023662A BR112023023662A2 (pt) 2021-05-17 2022-05-07 Método e aparelho de codificação de sinal de áudio tridimensional e codificador
JP2023571383A JP7656090B2 (ja) 2021-05-17 2022-05-07 三次元オーディオ信号コーディング方法及び装置、並びにエンコーダ
EP22803804.8A EP4322158A4 (en) 2021-05-17 2022-05-07 METHOD AND DEVICE FOR ENCODING THREE-DIMENSIONAL AUDIO SIGNALS AND ENCODER
CA3220588A CA3220588A1 (en) 2021-05-17 2022-05-07 Three-dimensional audio signal coding method and apparatus, and encoder
KR1020237040819A KR20240001226A (ko) 2021-05-17 2022-05-07 3차원 오디오 신호 코딩 방법, 장치, 및 인코더
US18/511,191 US20240087580A1 (en) 2021-05-17 2023-11-16 Three-dimensional audio signal coding method and apparatus, and encoder

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110535832.3 2021-05-17
CN202110535832.3A CN115376527B (zh) 2021-05-17 2021-05-17 三维音频信号编码方法、装置和编码器

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/511,191 Continuation US20240087580A1 (en) 2021-05-17 2023-11-16 Three-dimensional audio signal coding method and apparatus, and encoder

Publications (1)

Publication Number Publication Date
WO2022242480A1 true WO2022242480A1 (zh) 2022-11-24

Family

ID=84059746

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/091558 Ceased WO2022242480A1 (zh) 2021-05-17 2022-05-07 三维音频信号编码方法、装置和编码器

Country Status (9)

Country Link
US (1) US20240087580A1 (zh)
EP (1) EP4322158A4 (zh)
JP (1) JP7656090B2 (zh)
KR (1) KR20240001226A (zh)
CN (1) CN115376527B (zh)
BR (1) BR112023023662A2 (zh)
CA (1) CA3220588A1 (zh)
TW (1) TWI834163B (zh)
WO (1) WO2022242480A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118136027A (zh) * 2022-12-02 2024-06-04 华为技术有限公司 场景音频编码方法及电子设备
CN116222639B (zh) * 2022-12-08 2025-05-13 苏州伟创电气科技股份有限公司 基于编码器的校正处理方法、装置、设备及介质
CN119049483A (zh) * 2023-05-27 2024-11-29 华为技术有限公司 场景音频解码方法及电子设备

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102547549A (zh) * 2010-12-21 2012-07-04 汤姆森特许公司 编码解码2或3维声场环绕声表示的连续帧的方法和装置
EP2934025A1 (en) * 2014-04-15 2015-10-21 Thomson Licensing Method and device for applying dynamic range compression to a higher order ambisonics signal
IN201627036613A (zh) * 2016-10-26 2016-11-18 Qualcomm Inc
CN106463130A (zh) * 2014-07-02 2017-02-22 杜比国际公司 用于对hoa信号表示的子带内的主导方向信号的方向进行编码/解码的方法和装置
WO2018073258A1 (en) * 2016-10-19 2018-04-26 Holosbase Gmbh Decoding and encoding apparatus and corresponding methods
CN110767242A (zh) * 2013-05-29 2020-02-07 高通股份有限公司 声场的经分解表示的压缩
CN111670583A (zh) * 2018-02-01 2020-09-15 高通股份有限公司 可扩展的统一的音频渲染器
CN114582356A (zh) * 2020-11-30 2022-06-03 华为技术有限公司 一种音频编解码方法和装置

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB0419346D0 (en) * 2004-09-01 2004-09-29 Smyth Stephen M F Method and apparatus for improved headphone virtualisation
CN107103910B (zh) * 2011-10-21 2020-09-18 三星电子株式会社 帧错误隐藏方法和设备以及音频解码方法和设备
US9473870B2 (en) * 2012-07-16 2016-10-18 Qualcomm Incorporated Loudspeaker position compensation with 3D-audio hierarchical coding
JP6412931B2 (ja) * 2013-10-07 2018-10-24 ドルビー ラボラトリーズ ライセンシング コーポレイション 空間的オーディオ・システムおよび方法
ES2976769T3 (es) * 2013-12-02 2024-08-08 Top Quality Telephony Llc Medio de almacenamiento legible por ordenador y producto de software informático
EP4539046A1 (en) * 2014-03-21 2025-04-16 Dolby International AB Method for compressing a higher order ambisonics (hoa) signal, method for decompressing a compressed hoa signal, apparatus for compressing a hoa signal, and apparatus for decompressing a compressed hoa signal
KR20250051142A (ko) * 2014-06-27 2025-04-16 돌비 인터네셔널 에이비 Hoa 데이터 프레임 표현의 데이터 프레임들 중 특정 데이터 프레임들의 채널 신호들과 연관된 비차분 이득 값들을 포함하는 코딩된 hoa 데이터 프레임 표현
EP2963949A1 (en) * 2014-07-02 2016-01-06 Thomson Licensing Method and apparatus for decoding a compressed HOA representation, and method and apparatus for encoding a compressed HOA representation
US9747910B2 (en) * 2014-09-26 2017-08-29 Qualcomm Incorporated Switching between predictive and non-predictive quantization techniques in a higher order ambisonics (HOA) framework
WO2016142002A1 (en) * 2015-03-09 2016-09-15 Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. Audio encoder, audio decoder, method for encoding an audio signal and method for decoding an encoded audio signal
US9881628B2 (en) * 2016-01-05 2018-01-30 Qualcomm Incorporated Mixed domain coding of audio
US10972851B2 (en) * 2017-10-05 2021-04-06 Qualcomm Incorporated Spatial relation coding of higher order ambisonic coefficients
US12462815B2 (en) * 2018-07-03 2025-11-04 Qualcomm Incorporated Synchronizing enhanced audio transports with backward compatible audio transports

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102547549A (zh) * 2010-12-21 2012-07-04 汤姆森特许公司 编码解码2或3维声场环绕声表示的连续帧的方法和装置
CN110767242A (zh) * 2013-05-29 2020-02-07 高通股份有限公司 声场的经分解表示的压缩
EP2934025A1 (en) * 2014-04-15 2015-10-21 Thomson Licensing Method and device for applying dynamic range compression to a higher order ambisonics signal
CN106463130A (zh) * 2014-07-02 2017-02-22 杜比国际公司 用于对hoa信号表示的子带内的主导方向信号的方向进行编码/解码的方法和装置
WO2018073258A1 (en) * 2016-10-19 2018-04-26 Holosbase Gmbh Decoding and encoding apparatus and corresponding methods
IN201627036613A (zh) * 2016-10-26 2016-11-18 Qualcomm Inc
CN111670583A (zh) * 2018-02-01 2020-09-15 高通股份有限公司 可扩展的统一的音频渲染器
CN114582356A (zh) * 2020-11-30 2022-06-03 华为技术有限公司 一种音频编解码方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4322158A4

Also Published As

Publication number Publication date
JP2024520944A (ja) 2024-05-27
EP4322158A4 (en) 2024-08-07
CN115376527B (zh) 2026-04-07
BR112023023662A2 (pt) 2024-01-30
US20240087580A1 (en) 2024-03-14
CN115376527A (zh) 2022-11-22
JP7656090B2 (ja) 2025-04-02
TWI834163B (zh) 2024-03-01
TW202247148A (zh) 2022-12-01
CA3220588A1 (en) 2022-11-24
KR20240001226A (ko) 2024-01-03
EP4322158A1 (en) 2024-02-14

Similar Documents

Publication Publication Date Title
TWI834163B (zh) 三維音頻訊號編碼方法、裝置和編碼器
TWI844036B (zh) 三維音訊訊號編碼方法、裝置、編碼器、系統、電腦程式和電腦可讀儲存介質
WO2022253187A1 (zh) 一种三维音频信号的处理方法和装置
WO2022257824A1 (zh) 一种三维音频信号的处理方法和装置
CN115376529B (zh) 三维音频信号编码方法、装置和编码器
WO2022242479A1 (zh) 三维音频信号编码方法、装置和编码器
WO2022242481A1 (zh) 三维音频信号编码方法、装置和编码器
WO2024146408A1 (zh) 场景音频解码方法及电子设备
WO2024212638A1 (zh) 场景音频解码方法及电子设备
WO2025097787A1 (zh) 一种音频编码方法和装置及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22803804

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2022803804

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 202337077205

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2023571383

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 3220588

Country of ref document: CA

ENP Entry into the national phase

Ref document number: 2022803804

Country of ref document: EP

Effective date: 20231108

REG Reference to national code

Ref country code: BR

Ref legal event code: B01A

Ref document number: 112023023662

Country of ref document: BR

ENP Entry into the national phase

Ref document number: 20237040819

Country of ref document: KR

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 112023023662

Country of ref document: BR

Kind code of ref document: A2

Effective date: 20231110

WWG Wipo information: grant in national office

Ref document number: 202337077205

Country of ref document: IN