WO2024251169A1 - Procédé de reconnaissance de la parole, dispositif et support de stockage - Google Patents

Procédé de reconnaissance de la parole, dispositif et support de stockage Download PDF

Info

Publication number
WO2024251169A1
WO2024251169A1 PCT/CN2024/097609 CN2024097609W WO2024251169A1 WO 2024251169 A1 WO2024251169 A1 WO 2024251169A1 CN 2024097609 W CN2024097609 W CN 2024097609W WO 2024251169 A1 WO2024251169 A1 WO 2024251169A1
Authority
WO
WIPO (PCT)
Prior art keywords
speaker
vector representation
decoder
word
speech recognition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/097609
Other languages
English (en)
Chinese (zh)
Inventor
史莫晗
杜志浩
陈谦
俞帆
张仕良
李泱泽
张结
戴礼荣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Publication of WO2024251169A1 publication Critical patent/WO2024251169A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/16—Speech classification or search using artificial neural networks
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/18—Speech classification or search using natural language modelling
    • G10L15/183—Speech classification or search using natural language modelling using context dependencies, e.g. language models
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/20—Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems

Definitions

  • the present invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method, device and storage medium.
  • An end-to-end speaker-dependent speech recognition model can be used to recognize the speech of multiple speakers.
  • the existing end-to-end speaker-dependent speech recognition model cannot obtain speech recognition results with high accuracy.
  • the embodiments of the present invention provide a speech recognition method, a device and a storage medium, which are used to improve the accuracy of speech recognition results of multiple speakers.
  • an embodiment of the present invention provides a speech recognition method, wherein a speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer, and the method includes:
  • the first vector representation, the first n-1 words and a weighted sum of the correlation coefficients to the portrait feature vectors of the multiple speakers are input into the speech recognition decoder to obtain the nth word.
  • an embodiment of the present invention provides a speech recognition device, wherein the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer, and the device includes:
  • An acquisition module is configured to acquire speech signals of multiple speakers and portrait feature vectors of the multiple speakers, wherein the speech signals include the speech of the multiple speakers;
  • a first encoding module configured to obtain a first vector representation corresponding to the speech signal through the speech recognition encoder, and to obtain a second vector representation corresponding to the speech signal through the speaker encoder, wherein the first vector representation is configured to perform speech recognition, and the second vector representation is configured to perform speaker recognition;
  • a second encoding module is configured to encode the first n-1 words output by the speech recognition decoder through the text encoder to obtain a third vector representation corresponding to the n-1th word;
  • a first decoding module is configured to input the first vector representation, the second vector representation and the third vector representation into the speaker decoder to obtain a speaker vector representation corresponding to the nth word; and determine a speaker corresponding to the nth word according to correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of the plurality of speakers;
  • the second decoding module is configured to input the first vector representation, the first n-1 words and the weighted sum of the correlation coefficients to the portrait feature vectors of the multiple speakers into the speech recognition decoder to obtain the nth word.
  • an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the speech recognition method as described in the first aspect.
  • an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon.
  • the executable code When executed by a processor of an electronic device, the processor can at least implement the speech recognition method described in the first aspect.
  • an embodiment of the present invention provides a speech recognition method, the method comprising:
  • a request triggered by a terminal device by calling a speech recognition service wherein the request includes speech signals of multiple speakers and portrait feature vectors of the multiple speakers, and the speech signals include the speech of the multiple speakers;
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer;
  • the speech recognition output information includes text sequences corresponding to each of the multiple speakers.
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder and a text encoder including an attention layer.
  • the speech signal can be first encoded by the speech recognition encoder in the speech recognition model to obtain a first vector representation set for speech recognition, and the speech signal can be encoded by the speaker encoder to obtain a second vector representation set for speaker recognition. Then, the first n-1 words output by the speech recognition decoder are encoded by the text encoder to obtain a third vector representation corresponding to the n-1th word.
  • the first vector representation, the second vector representation and the third vector representation are input into the speaker decoder to obtain the speaker vector representation corresponding to the nth word, and the speaker corresponding to the nth word is determined according to the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of multiple speakers.
  • the weighted sum of the first vector representation, the first n-1 words and the correlation coefficient to the portrait feature vectors of multiple speakers is input into the speech recognition decoder to obtain the nth word.
  • a text encoder including an attention layer is added to the speech recognition model.
  • the first n-1 words output by the speech recognition decoder are encoded by the text encoder, so that the context information of the entire sentence can be better aggregated to obtain a third vector representation containing context information corresponding to the n-1th word. That is, a vector representation that is more accurately set for speaker recognition can be obtained, thereby improving the accuracy of the speaker recognition results corresponding to each word.
  • FIG1 is a flow chart of a speech recognition method provided by an embodiment of the present invention.
  • FIG2 is a schematic diagram of the structure of a speech recognition model provided by an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of the structure of a speech recognition encoder and a speech recognition decoder provided in an embodiment of the present invention
  • FIG4 is a flow chart of determining a speaker vector representation corresponding to an n-th word through a speaker decoder
  • FIG5 is a flow chart of another speech recognition method provided by an embodiment of the present invention.
  • FIG6 is a flow chart of obtaining the nth word through a speech recognition decoder
  • FIG7 is a schematic diagram of an application of a speech recognition method provided by an embodiment of the present invention.
  • FIG8 is a flow chart of another speech recognition method provided by an embodiment of the present invention.
  • FIG9 is a schematic diagram of a speech recognition process in a cloud service mode provided by an embodiment of the present invention.
  • FIG10 is a schematic diagram of the structure of a speech recognition device provided by an embodiment of the present invention.
  • FIG. 11 is a schematic diagram of the structure of an electronic device provided in this embodiment.
  • the user information including but not limited to user device information, user personal information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
  • SA-ASR Speaker-Attributed Automatic Speech Recognition
  • Serialized Output Training When transcribing speech (i.e., converting speech into text), special delimiters are used to connect the texts spoken by different speakers, and the transcribed texts are arranged in the order in which the different speakers start speaking to generate a text sequence, and the text sequence is output.
  • a speech segment includes two speakers, speaker A said "How is the weather today", and speaker B said "The weather is good today”. Then when performing serialized output training on the speech segment, the output will be: How is the weather today ⁇ sc>The weather is good today, and each word or sentence is associated with the corresponding speaker label.
  • Speaker Profile refers to a set of feature vectors, which contains the feature vectors of the profiles of multiple speakers.
  • the feature vectors of the profiles of each speaker can be extracted from a speech signal of each speaker.
  • End-to-End Different from modular processing of the target task, the target task is processed by a complete model, that is, the input is the original data and the output is the final processing result.
  • the overall objective function can be directly optimized during learning and training.
  • the existing end-to-end speaker-dependent speech recognition model mainly includes speech recognition encoder, speaker encoder
  • the model lacks a context information perception module.
  • the context information corresponding to each word is not fully considered, so that the predicted speaker corresponding to each word is inaccurate, thereby affecting the final speech recognition result.
  • an embodiment of the present invention provides a new speech recognition scheme.
  • a text encoder including an attention layer is added to the speech recognition model.
  • the text encoder can encode the first n-1 words that have been output by the speech recognition decoder to integrate the context information of the sentence, so that the speaker vector representation corresponding to the nth word is more accurate, so that the speaker encoder can more accurately predict the speaker corresponding to each word.
  • FIG. 1 is a flow chart of a speech recognition method provided by an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:
  • the speech recognition solution provided by the embodiment of the present invention can use a pre-trained speech recognition model to process the speech signals of multiple speakers to be recognized, so as to obtain each word corresponding to the speech signal and the speaker corresponding to each word. Since the training process of the speech recognition model is similar to the process of using the speech recognition model to recognize the speech signals of multiple speakers to be recognized, the process of using the speech recognition model is described here.
  • the structure of the speech recognition model can be referred to as shown in FIG2.
  • the speech recognition model mainly includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer.
  • the speech recognition encoder is mainly configured to convert speech signals of multiple speakers into a first vector representation configured to perform speech recognition.
  • the speaker encoder is mainly configured to convert speech signals of multiple speakers into a second vector representation configured to perform speaker recognition.
  • the speech recognition decoder is mainly configured to complete speech recognition. Speech-to-text recognition is used to identify all the words spoken by each speaker in the speech signal, and finally output the speech recognition results corresponding to each speaker.
  • the speaker decoder is mainly configured to determine the correspondence between each word and the speaker.
  • the text encoder is mainly configured to integrate the context information of the text, and determine the speaker vector representation corresponding to each word according to the context information corresponding to each word, so that the speaker vector representation corresponding to each word is more accurate.
  • the specific structure of the speech recognition encoder and the speech recognition decoder can refer to that shown in FIG3.
  • the speech recognition encoder can include multiple cascaded encoders, and each encoder can include two sublayers: an attention layer and a feedforward neural network layer.
  • the speech recognition decoder can include multiple cascaded decoders, and each decoder includes an attention layer and a feedforward neural network layer.
  • the structure of the speech encoder is similar to that of the speech recognition encoder, and reference can be made to the specific structure shown in FIG3.
  • the speaker encoder can include multiple cascaded encoders, and each encoder includes an attention layer and a feedforward neural network layer.
  • the structure of the speech decoder is similar to that of the speech recognition decoder, and the speaker decoder includes multiple cascaded decoders, and each decoder includes an attention layer and a feedforward neural network layer.
  • the text encoder includes at least one cascaded encoder, and each encoder includes an attention layer and a feedforward neural network layer.
  • the number of encoders included in the speech recognition encoder and the speaker encoder can be set according to actual needs, and is not limited here.
  • the number of decoders included in the speech recognition decoder and the speaker decoder can be set according to actual needs.
  • the attention layer in the decoder here can include a self-attention layer and a source-target attention layer.
  • the speech recognition model mainly includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder and a text encoder including an attention layer. Then, when the speech recognition model is used to perform speech recognition on the speech signals of multiple speakers, the specific execution process of each encoder and decoder in the speech recognition model is as follows: First, the speech signals of multiple speakers and the portrait feature vectors of multiple speakers are obtained. Among them, the speech signal contains the speech of multiple speakers.
  • the speaker's portrait feature vector refers to the speaker's feature vector, and the portrait feature vector corresponding to the speaker can be extracted from a segment of the speaker's speech signal.
  • the portrait features of the multiple speakers may be extracted in advance to obtain the portrait feature vectors of each speaker.
  • the speech signals of each speaker may be encoded by an encoder having the same structure as the speaker encoder to obtain the portrait feature vectors of each speaker.
  • the speech signal may be processed to extract effective speech information and obtain an acoustic feature sequence corresponding to the speech signal.
  • the specific implementation process may include: segmenting the acquired speech signal containing multiple speakers, cutting the speech signal into frames, then extracting acoustic features from each frame of the speech signal, and generating an acoustic feature sequence corresponding to the speech signal based on the acoustic features corresponding to each frame of the speech signal.
  • the acoustic feature sequence corresponding to the speech signal is input into the speech recognition encoder to obtain the first vector representation corresponding to the speech signal through the speech recognition encoder.
  • the first vector representation is set to perform speech recognition, that is, the first vector representation can characterize the speech features of the speech signal.
  • the corresponding acoustic feature sequence is input into the speaker encoder to obtain the second vector representation corresponding to the speech signal through the speaker encoder.
  • the second vector representation is set to perform speaker recognition, that is, the second vector representation can be set to represent the timbre and other features of the speaker of the speech signal.
  • the speech recognition encoder can convert the acoustic feature sequence corresponding to the speech signal into a vector representation that can be used for speech recognition, so that the speech recognition process can be performed directly based on the first vector representation to obtain each word corresponding to the speech signal.
  • the speaker encoder can convert the acoustic feature sequence corresponding to the speech signal into a vector representation for speaker recognition, so that the speaker recognition process corresponding to each word can be performed directly based on the second vector representation.
  • prediction is performed word by word. For example, after the speech recognition decoder determines the first word corresponding to the speech signal, the first word regresses to the input end of the speech recognition decoder and is set to determine the second word corresponding to the speech signal. The words corresponding to the speech signal are determined in sequence until the end symbol is encountered. Since the specific implementation process is basically the same when predicting each word in the speech signal, the prediction process of the nth word is used as an example here, assuming that the speech recognition decoder has output the first n-1 words. It should be noted that the first recognized word is predicted based on the set starting character or empty character as the input of the speech recognition decoder.
  • the text encoder When determining the nth word, we must first obtain the third vector representation containing context information corresponding to the n-1th word, that is, determine the nth word in combination with the information of the first n-1 words, so that the determination of the nth word is more accurate. Since the text encoder is mainly configured to integrate the context information corresponding to each word to obtain a vector representation containing context information, after obtaining the first n-1 words output by the speech recognition decoder, the text encoder encodes the first n-1 words output by the speech recognition decoder to obtain the third vector representation corresponding to the n-1th word.
  • the text encoder determines the third vector representation corresponding to the n-1th word, it first obtains the first n-1 words that the speech recognition encoder finally outputs after various processing, and encodes the first n-1 words that have been determined, so as to obtain the encoding result corresponding to the n-1th word from the encoding result as the third vector representation corresponding to the n-1th word.
  • the third vector representation corresponding to the 4th word is to be obtained, the first 3 words that the speech recognition encoder has output are first obtained, and the first 3 words are encoded to obtain the third vector representation corresponding to the 4th word.
  • the specific implementation process of encoding the first n-1 words that have been output by the speech recognition decoder through the text encoder to obtain the third vector representation corresponding to the n-1th word may include: obtaining the weighted vector representations of the attention coefficients corresponding to the first n-1 words obtained after the self-attention layer in the speech recognition decoder performs attention calculations on the first n-1 words respectively; and inputting the weighted vector representations of the attention coefficients corresponding to the first n-1 words into the text encoder to obtain the third vector representation corresponding to the n-1th word output by the text encoder.
  • what is input into the text editor is the weighted encoded vectors corresponding to the first n-1 words output by the self-attention layer in the speech recognition decoder.
  • the speech recognition decoder may include at least one decoder, each of which may include Including an attention layer, and having several different attention layers, such as a self-attention layer and a source-target attention layer.
  • the self-attention layer in the above-mentioned speech recognition decoder can be the first layer of the speech recognition decoder, that is, the self-attention layer contained in the first decoder. At this time, it is necessary to collect the results of the self-attention layer in the first layer of the speech recognition encoder for the first n-1 words, respectively, to form a sequence, and send it to the text encoder.
  • the text encoder will output the third vector representation after the fusion context information corresponding to each word of the first n-1 words.
  • the third vector representation corresponding to the n-1th word is taken out from the third vector representation corresponding to the first n-1 words.
  • the reason why the text encoder can fuse text context information is because the text encoder includes an attention layer, such as an attention layer that adopts a self-attention mechanism or other attention mechanisms. In this article, the self-attention layer is taken as an example.
  • the self-attention layer in the first layer performs attention calculation on the first n-1 words respectively, which can be expressed as the following formula: in, Represents the weighted vector representation of the attention coefficient corresponding to the n-1th word output by the first layer of the speech recognition decoder, represents the embedding vector representation corresponding to the n-1th word in the first layer of the input speech recognition decoder, represents the multi-head attention computation performed by the self-attention layer in the first layer of the speech recognition decoder, The embedding vector representations corresponding to the first n-1 words in the first layer of the input speech recognition decoder are represented.
  • the embedding vector representation is set to represent the vector representation after the embedding coding process, and the embedding vector representation can be obtained by embedding coding or embedding coding and position coding the first n-1 words respectively to obtain the embedding vector representations corresponding to the first n-1 words.
  • the formula can be used. Get the embedding vector representation corresponding to the first n-1 words.
  • PosEnc represents position encoding
  • Embed represents embedding encoding
  • y [1:n-1] represents the first n-1 words output by the speech recognition decoder.
  • the text editor After obtaining the weighted vector representations of the attention coefficients corresponding to the first n-1 characters, the text editor processes the weighted vector representations of the attention coefficients corresponding to the first n-1 characters to obtain a third vector representation corresponding to the n-1th character.
  • the text editor may process the weighted vector representations of the attention coefficients corresponding to the first n-1 characters according to the formula Calculate the third vector representation corresponding to the n-1th word. represents the third vector representation corresponding to the n-1th word, Context-Enc represents the context text encoding, Represents the weighted vector of attention coefficients corresponding to the first n-1 words.
  • the first vector representation, the second vector representation and the third vector representation are input into the speaker decoder to obtain the speaker vector representation corresponding to the nth word.
  • the speaker vector representation can be used to characterize the characteristics of the speaker.
  • the speech recognition features, speaker recognition features, and context information features of the speech signal are combined, so that the speaker vector representation corresponding to the nth word determined is of higher quality, thereby improving the accuracy of predicting the speaker corresponding to each word. This makes the transcription results corresponding to the speech signal more accurate.
  • the speaker corresponding to the nth word is then determined based on the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of multiple speakers.
  • the correlation coefficient is set to characterize the correlation between the speaker vector representation corresponding to each word and the portrait feature vectors of multiple speakers, that is, the correlation coefficient is set to represent the correlation between the nth word and each speaker contained in the speech signal.
  • the speaker corresponding to the nth word when determining the speaker corresponding to the nth word, first obtain the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of multiple speakers, and then determine the speaker corresponding to the nth word based on the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of multiple speakers.
  • the speaker vector corresponding to the nth word is represented by qn
  • the portrait feature vector of the first speaker is d1
  • the portrait feature vector of the second speaker is d2
  • the portrait feature vector of the third speaker is d3
  • the correlation coefficient ⁇ n,1 between the speaker vector representation qn corresponding to the nth word and d1 is obtained
  • the correlation coefficient ⁇ n,2 between the speaker vector representation qn corresponding to the nth word and d2 is obtained
  • the correlation coefficient ⁇ n,3 between the speaker vector representation qn corresponding to the nth word and d3 is obtained, and then the speaker corresponding to the nth word is determined based on these three correlation coefficients.
  • the speaker corresponding to the maximum correlation coefficient can be selected from multiple correlation coefficients to be determined as the speaker corresponding to the nth word.
  • the weighted sum of the multiple correlation coefficients for the portrait feature vectors of multiple speakers is calculated, where the weighted sum can be regarded as the weighted speaker vector representation corresponding to the nth word.
  • the formula Calculate the weighted sum of multiple correlation coefficients for multiple speaker portrait feature vectors in represents the weighted sum
  • K represents the total number of speakers contained in the speech signal
  • ⁇ n ,k represents the correlation coefficient between the speaker vector corresponding to the nth word and the portrait feature vector of the kth speaker
  • dk represents the portrait feature vector of the Kth speaker.
  • the weighted sum of the first vector representation, the first n-1 words, and the correlation coefficients for the portrait feature vectors of multiple speakers is input into the speech recognition decoder to obtain the nth word.
  • the weighted sum is the weighted speaker vector representation corresponding to the nth word.
  • the weighted speaker vector representation corresponding to the nth word is introduced when determining the nth word.
  • the nth word obtained in this way includes not only the specific content of the word but also the speaker corresponding to the word. For example, the specific content of the output nth word is "I", and the corresponding speaker is "Xiao Li".
  • each word in the speech signal and the speaker corresponding to each word can be determined in turn, and finally the recognition result in SOT format corresponding to each speaker can be determined.
  • two speakers are identified in a speech signal, and the speaker corresponding to the recognized text "How is the weather today” is speaker a, and the speaker corresponding to the recognized text "The weather is good today” is speaker b, and the output content is: How is the weather today ⁇ sc>The weather is good today.
  • a text encoder including an attention layer is added to the speech recognition model.
  • the text encoder encodes the first n-1 words output by the speech recognition decoder, so that the context information of the entire sentence can be better aggregated to obtain a third vector representation containing context information corresponding to the n-1th word. That is, based on the first vector representation, the second vector representation and the third vector representation, a speaker vector representation that is more accurately set for speaker recognition can be obtained, thereby improving the accuracy of speaker recognition results corresponding to each word.
  • a speaker decoder includes multiple cascaded decoders.
  • the processing of the encoder corresponding to each layer is exemplarily described in conjunction with FIG. 4 .
  • FIG4 is a flow chart of determining a speaker vector representation corresponding to an n-th word through a speaker decoder; as shown in FIG4 , the speaker decoder includes a plurality of cascaded decoders, each of which includes an attention layer and a feedforward neural network layer, and the method includes the following steps:
  • the speaker decoder in the embodiment of the present invention includes multiple decoders, each of which is set at a different level. Each decoder performs decoding processing layer by layer according to the level, and the output of the decoder in the previous layer is used as the input of the decoder in the next layer.
  • each decoder includes an attention layer and a feedforward neural network layer, and the decoding processing is performed by two sublayers of the attention layer and the feedforward neural network layer respectively. First, the vector representation input to the decoder is transmitted to the attention layer for processing, and the processing result is input to the feedforward neural network layer.
  • the feedforward neural network layer processes the processing result output by the attention layer, and the processed result is input to the attention layer in the decoder of the next layer, and the attention layer in the decoder of the next layer processes it, that is, each decoder in the speaker decoder is processed in sequence according to this method.
  • the processing process in each decoder may include: first, the first vector representation, the second vector representation and the third vector representation are input into the speaker decoder, so as to obtain the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word through the attention layer in the first layer decoder. Then, the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word is input into the feedforward neural network layer in the first layer decoder, so as to obtain the speaker vector representation corresponding to the n-1th word input into the second layer decoder through the feedforward neural network layer in the first layer decoder. In other words, the output result of the feedforward neural network layer in the first layer decoder (the speaker vector representation corresponding to the n-1th word) is used as the input of the attention layer in the second layer decoder.
  • the attention layer in the first layer decoder can be calculated according to the formula Calculate the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word.
  • the feedforward neural network layer in the first decoder can be calculated according to the formula Calculate the speaker vector representation corresponding to the n-1th word.
  • the speaker vector representation corresponding to the n-1th word input to the second layer decoder represents the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word output by the attention layer of the first decoder, Represents the first layer of a feed-forward neural network in the speaker encoder.
  • the attention layers in the second layer decoder and the decoders above the second layer include a self-attention layer and a source-target attention layer. Then, after obtaining the speaker vector representation corresponding to the n-1th word input to the second layer decoder through the feedforward neural network layer in the first layer decoder, it is output to the self-attention layer in the second layer decoder, and the self-attention layer in the second layer decoder performs self-attention calculation on the speaker vector representation corresponding to the n-1th word and the speaker vector representation corresponding to the first n-1 words, and obtains the vector representation corresponding to the n-1th word corresponding to the self-attention layer in the second layer decoder after the self-attention coefficient weighted, and inputs the vector representation corresponding to the n-1th word after the self-attention coefficient weighted to the source-target attention layer in the second layer decoder.
  • the source-target attention layer in the second layer decoder performs attention calculation on the vector representation weighted by the self-attention coefficient corresponding to the n-1th word and the second vector representation to obtain the vector representation weighted by the attention coefficient corresponding to the n-1th word corresponding to the source-target attention layer in the second layer decoder, and inputs the vector representation weighted by the attention coefficient corresponding to the n-1th word to the feedforward neural network layer in the second layer decoder.
  • the feedforward neural network layer in the second decoder processes the vector representation weighted by the attention coefficient corresponding to the n-1th word to obtain the speaker vector representation corresponding to the n-1th word input to the third encoder.
  • the decoders above the second layer perform calculations and processing respectively according to the above method until the speaker vector representation corresponding to the n-1th word output by the last layer decoder in the speaker decoder is determined, and the speaker vector representation corresponding to the n-1th word is determined based on the speaker vector representation corresponding to the n-1th word output by the last layer decoder and the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word output by the attention layer in the first layer decoder.
  • the detailed processing process is referred to the above description and will not be repeated here.
  • the self-attention layers in the second layer decoder and the second layer and above decoders can be calculated according to the formula Calculate the weighted vector representation of the self-attention coefficient corresponding to the n-1th word of the source-target attention layer input to the decoder of this layer.
  • l represents the number of layers corresponding to the encoder, and l>1, represents the n-1th output of the self-attention layer in the lth encoder
  • the weighted vector representation of the self-attention coefficient corresponding to the word represents the speaker vector representation corresponding to the n-1th word output by the feedforward neural network layer in the l-1th layer encoder, represents the multi-head attention of the self-attention layer in the l-th encoder layer, Represents the speaker vector representation corresponding to the first n-1 words output by the feedforward neural network layer in the l-1th encoder layer.
  • the source-target attention layer in the second layer decoder and the decoders above the second layer can be calculated according to the formula Calculate the weighted vector representation of the attention coefficient corresponding to the n-1th word of the feedforward neural network layer input to the decoder of this layer.
  • l represents the number of layers corresponding to the encoder, and l>1, represents the weighted vector representation of the attention coefficient corresponding to the n-1th word output by the source-target attention layer in the l-th encoder (i.e., the weighted vector representation of the attention coefficient corresponding to the n-1th word input to the feedforward neural network layer in the l-th encoder), It represents the weighted vector representation of the self-attention coefficient corresponding to the n-1th word output by the self-attention layer in the lth layer encoder, represents the multi-head attention of the source-target attention layer in the l-th encoder layer, and H spk represents the second vector representation.
  • the feedforward neural network layers in the second layer decoder and the decoders above the second layer can be calculated according to the formula Calculate the speaker vector representation corresponding to the n-1th word input to the next layer of encoder.
  • l represents the number of layers corresponding to the encoder, and l>1
  • l represents the speaker vector representation corresponding to the n-1th word output by the feedforward neural network in the l-th layer encoder (i.e., the speaker vector representation corresponding to the n-1th word input to the l+1-th layer encoder)
  • the output of the feedforward neural network layer in each layer decoder is determined in sequence until the speaker vector representation corresponding to the n-1th word output by the last layer decoder in the speaker decoder is determined. Finally, the speaker decoder skips the speaker vector representation corresponding to the n-1th word output by the last layer decoder and the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word output by the attention layer in the first layer decoder, and predicts the speaker vector representation corresponding to the nth word based on the skip-connection result.
  • the speaker vector representation corresponding to the n-1th word output by the last layer decoder in the speaker decoder is The speaker vector weighted by the attention coefficient corresponding to the n-1th word output by the attention layer in the first layer decoder is expressed as According to the formula Determine the speaker vector representation q n corresponding to the nth word, where W q represents the weight set in the speaker encoder.
  • the speaker decoder includes a multi-layer decoder
  • the first layer decoder processes the first vector representation, the second vector representation and the third vector representation containing context information to obtain the first layer decoder.
  • the attention layer in the decoder outputs the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word, that is, the first layer decoder combines the context information when processing, so that the speaker representation weighted by the attention coefficient corresponding to the n-1th word is more accurate.
  • the feedforward neural network layer in the first layer decoder processes the speaker vector weighted by the attention coefficient corresponding to the n-1th word obtained after processing in combination with the context information, so that the speaker vector representation corresponding to the n-1th word input to the second layer decoder has a higher quality and can better reflect the speaker characteristics corresponding to the n-1th word. Then, the speaker vector representation corresponding to each word with higher quality can be obtained through the processing of multiple layers of decoders in turn, thereby making the speaker results predicted by each word more accurate.
  • the speaker corresponding to each character can be further determined based on the speaker vector representation corresponding to each character.
  • the speaker corresponding to the nth character is determined based on the correlation coefficient between the speaker vector representation corresponding to the nth character and the portrait feature vectors of multiple speakers.
  • the correlation coefficient between the speaker vector representation corresponding to the nth character and the portrait feature vectors of multiple speakers is first determined.
  • the specific process of determining the correlation coefficient between the speaker vector representation corresponding to the nth character and the portrait feature vectors of multiple speakers is exemplified in conjunction with Figure 5.
  • FIG5 is a flow chart of another speech recognition method provided by an embodiment of the present invention. As shown in FIG5 , in order to improve the accuracy of the speech recognition result of the method, based on the above embodiment, the method may further include the following steps:
  • a correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined by jointly scoring using a set similarity algorithm and a set scoring model including an attention module.
  • the correlation coefficient can be set to represent the posterior probability of the nth word corresponding to each speaker.
  • the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined according to the set similarity algorithm. That is to say, according to the set similarity algorithm, the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of each speaker can be determined respectively.
  • the set similarity algorithm can be a cosine similarity algorithm, for example, according to the formula The first similarity between the speaker vector corresponding to the nth word and the portrait feature vectors of each speaker can be determined.
  • the speaker vector representation corresponding to the nth word is respectively calculated with the portrait feature vector of each speaker to obtain the probability that the nth word corresponds to each speaker.
  • the determination of the first similarity only considers the speaker vector representation corresponding to the nth word, and does not consider the speaker vector representation corresponding to the first n words. Therefore, the determination of the first similarity is independent of other speakers.
  • the speaker vector representations of other speakers are fully combined. Specifically, by adding a scoring model containing an attention module to the speech recognition model, the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker is obtained through the scoring model containing the attention module.
  • the specific implementation process of determining the second similarity may be: obtaining the speaker vector representation corresponding to each of the first n words, and splicing the speaker vector representation corresponding to each of the first n words with the portrait feature vector of any speaker. Inputting the multiple spliced vector representations into the set scoring model including the attention module to obtain the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker.
  • the formula Determine the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of each speaker represents the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of the kth speaker
  • tanh is the hyperbolic tangent function
  • CD-scorer represents the set scoring model including the attention module.
  • q [1:n] represents the speaker vector representation corresponding to the first n words
  • d k represents the portrait feature vector of the kth speaker.
  • the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vectors of each speaker can be generated according to this formula.
  • the implementation method for determining the second similarity can be: obtain the speaker vector representation corresponding to each of the first n words, and concatenate the speaker vector representation corresponding to each of the first n words with the kth speaker feature vector to obtain a first vector sequence with a length of n. And input the first vector sequence into the scoring model (for example, using the structure of the transfomer model). In the structure), a second vector sequence is obtained.
  • the second vector sequence is a sequence of values between [-1, 1] obtained after tanh transformation, and the second similarity between the speaker vector corresponding to the last determined nth word and the portrait feature vector of the kth speaker is taken out. Tanh is an example of a mapping function, which is not limited to this.
  • the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined according to the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker and the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker.
  • the first similarity and the second similarity are calculated for each speaker, the first similarity and the second similarity between the nth word and each speaker are added to obtain a first sum value, and the first sum value is divided by the sum of the first similarity and the second similarity between the nth word and each speaker to obtain the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of each speaker.
  • the formula Determine the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of each speaker.
  • ⁇ n,k represents the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of the kth speaker
  • exp represents an exponential function with e as the base.
  • K is the total number of speakers included in the speech signal
  • j is the jth speaker.
  • k in the above can be any one of the K speakers, then the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of each speaker can be obtained according to the above formula.
  • the speaker corresponding to the nth word can be determined based on the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of the multiple speakers.
  • a first similarity between the speaker vector representation corresponding to the nth word and a portrait feature vector of any speaker is determined according to a set similarity algorithm, and a second similarity between the speaker vector representation corresponding to each of the first n words and a portrait feature vector of any speaker is obtained through a set scoring model including an attention module, and then a correlation coefficient between the speaker vector representation corresponding to the nth word and a portrait feature vector of any speaker is determined according to the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker and the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker, that is, the correlation coefficient determined by combining the set similarity algorithm and the set scoring model including the attention module is more accurate, so that the speech recognition model has better performance, thereby further improving the accuracy of the speech recognition result.
  • the above embodiment introduces the method of determining the speaker vector representation corresponding to the nth word and the speaker vector representation corresponding to the plurality of speakers.
  • the specific implementation method of the correlation coefficient between the portrait feature vectors is to calculate the weighted sum of the correlation coefficient to the portrait feature vectors of the multiple speakers after determining the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of the multiple speakers. Specifically, the product values of the multiple correlation coefficients and the portrait feature vectors of the respective speakers are obtained, and then the multiple product values are summed to obtain the weighted speaker feature vector corresponding to the nth word.
  • a speech recognition decoder includes multiple cascaded decoders.
  • the processing process of the encoder corresponding to each layer is exemplarily illustrated in conjunction with Figure 6.
  • FIG6 is a flow chart of obtaining the nth word through a speech recognition decoder; as shown in FIG6 , the speech recognition decoder includes a plurality of cascaded decoders, each of which includes an attention layer and a feedforward neural network layer, and the method includes the following steps:
  • the weighted vector representation of the attention coefficient corresponding to the n-1th word and the weighted sum are input into the feedforward neural network layer in the first layer decoder to obtain the vector representation corresponding to the n-1th word input into the second layer decoder.
  • the speech recognition decoder in the embodiment of the present invention includes a multi-layer decoder, and each layer of the decoder performs decoding processing layer by layer according to the level, and the output of the upper layer of the decoder is used as the input of the next layer of the decoder.
  • each decoder includes an attention layer and a feedforward neural network layer, and the two sublayers process the vector representation input to the encoder respectively.
  • the function of the speech decoder is to predict each word one by one in an autoregressive manner.
  • the nth word when performing the nth iteration, it is set to predict the nth word corresponding to the input speech signal, and the nth word is predicted according to the weighted sum of the weighted vector representation of the attention coefficient corresponding to the first n-1 words of the input, the n-1 word, and the portrait feature vectors of multiple speakers by the correlation coefficient.
  • the predicted nth word is obtained according to the probability of the nth word corresponding to each word in the dictionary V. Assuming that there are 4950 words in the dictionary, determine the probability on of the nth word corresponding to each word in the dictionary V, that is, on is 4950 probabilities, and select the word with the largest probability among the 4950 probabilities obtained as the prediction result of the nth word.
  • embedding encoding is performed on the first n-1 words respectively to obtain the embedding vector representation corresponding to each of the first n-1 words.
  • embedding processing can be performed on the first n-1 words respectively to obtain the processed vector representation
  • position encoding processing can be performed on the processed vector representation to obtain the embedding vector representation corresponding to each of the first n-1 words.
  • PosEnc represents the position encoding
  • Embed represents the embedding function
  • y [1:n-1] represents the first n-1 words that the speech recognition decoder has output.
  • the embedded vector representations and the first vector representation corresponding to the first n-1 characters are input into the attention layer in the first decoder.
  • the attention layer in the first decoder processes the embedded vector representations and the first vector representation corresponding to the first n-1 characters to obtain the vector representation weighted by the attention coefficient corresponding to the n-1th character corresponding to the attention layer in the first decoder, and inputs it into the feedforward neural network layer in the first decoder.
  • the attention layer in each decoder in the speech recognition encoder may include a self-attention layer and a source-target attention layer. Then, after the speaker encoder receives the embedding vector representation and the first vector representation corresponding to the first n-1 words, the embedding vector representation corresponding to the first n-1 words is input into the self-attention layer in the first layer decoder to obtain the vector representation weighted by the self-attention coefficient corresponding to the n-1th word output by the self-attention layer in the first layer decoder, and the vector representation is input into the source-target attention layer in the first layer decoder.
  • the source-target attention layer in the first layer decoder calculates the vector representation weighted by the self-attention coefficient corresponding to the n-1th word and the first vector representation to obtain the vector representation weighted by the attention coefficient corresponding to the n-1th word output by the source-target attention layer. Then, the weighted sum of the vector representation weighted by the attention coefficient corresponding to the n-1th word and the correlation coefficient to the portrait feature vectors of multiple speakers is input into the feedforward neural network layer in the first layer decoder to obtain the vector representation corresponding to the n-1th word input into the second layer decoder. According to the above method, the processing operations of the second layer and the decoders above the second layer are completed respectively until the vector representation corresponding to the n-1th word output by the last layer decoder in the speech recognition decoder is obtained.
  • the self-attention layer in the first layer and above decoders can be calculated according to the formula Calculate the weighted vector representation of the self-attention coefficient corresponding to the n-1th word of the source-target attention layer input to the decoder of this layer.
  • l represents the number of layers corresponding to the encoder
  • l represents the weighted vector representation of the self-attention coefficient corresponding to the n-1th word output by the self-attention layer in the l-th encoder
  • the weighted vector representation of the self-attention coefficient corresponding to the n-1th word input to the source-target attention layer in the l-th encoder represents the vector representation of the n-1th word output by the feedforward neural network layer in the l-1th encoder layer
  • the source-target attention layer in the first and higher decoders can be calculated according to the formula Calculate the weighted vector representation of the attention coefficient corresponding to the n-1th word of the feedforward neural network layer input to the decoder of this layer.
  • l represents the layer corresponding to the encoder Number, and represents the weighted vector representation of the attention coefficient corresponding to the n-1th word output by the source-target attention layer in the l-th encoder (i.e., the weighted vector representation of the attention coefficient corresponding to the n-1th word input to the feedforward neural network layer in the l-th encoder), It represents the weighted vector representation of the self-attention coefficient corresponding to the n-1th word output by the self-attention layer in the lth layer encoder, denotes the multi-head attention of the source-target attention layer in the l-th encoder layer, and Hasr denotes the first vector representation.
  • the feedforward neural network layer in the first layer decoder can be calculated according to the formula Calculate the vector representation corresponding to the n-1th word input to the second layer encoder.
  • l represents the number of layers corresponding to the encoder
  • W spk is the weight corresponding to the speaker encoder
  • the formula Calculate the vector representation corresponding to the n-1th word input to the next layer of encoder (that is, the output of the feedforward neural network layer is used as the input of the next layer of encoder).
  • l represents the number of layers corresponding to the encoder, and l>1.
  • the nth word is determined based on the vector representation corresponding to the n-1th word output by the last layer of the decoder in the speech recognition decoder. Specifically, the vector representation corresponding to the n-1th word output by the last layer of the decoder is processed using the softmax operation to obtain the posterior probability o n corresponding to the nth word, and then the nth word is predicted based on the posterior probability o n corresponding to the nth word.
  • the vector representation corresponding to the n-1th word output by the last layer of the decoder is According to the formula Determine the posterior probability o n corresponding to the nth word, where softmax is the activation function, and W o and b o are the weights corresponding to the speech recognition encoder.
  • the first n-1 words that have been output are embedded and encoded.
  • the embedded vector representation of the n-1th word is taken out as the query in the self-attention layer, and the embedded vector representations corresponding to the n-1 words are used as key and value for self-attention processing to obtain the result of the n-1th word paying attention to the first n-1 words, that is, the encoding vector weighted by the self-attention coefficient corresponding to the n-1th word, and then added to the embedded vector representation of the n-1th word before weighting.
  • the addition result is the weighted vector representation of the self-attention coefficient corresponding to the n-1th word, and it is used as one of the inputs of the source-target attention layer.
  • the other two inputs are two first vector representations, and finally the weighted vector representation of the attention coefficient corresponding to the n-1th word output by the source-target attention layer in the first layer is obtained.
  • the weighted vector representation of the attention coefficient corresponding to the n-1th word is input into the feedforward neural network layer in the first layer encoder to obtain the first layer encoder.
  • the output of the encoder is the input of the second layer encoder.
  • the output is calculated with This item is not added when the feedforward neural network layer in other layers of encoder is processed.
  • the feedforward neural network layer in the first layer of encoder introduces The purpose is to introduce the weighted speech vector representation of the nth word corresponding to each speaker in the process of speech recognition, so that the recognized nth word does not only include the specific content corresponding to the word but also includes the speaker corresponding to the word.
  • the speech recognition encoder can directly output the recognition results in SOT format, such as a text sequence of one speaker ⁇ sc> another speaker's text sequence.
  • the speech decoder predicts each word one by one in an autoregressive manner.
  • each layer of decoders in the speech encoder decodes the weighted sum of the portrait feature vectors of multiple speakers of the first n-1 words of the input, the weighted vector representation of the attention coefficient corresponding to the n-1th word, and the correlation coefficient layer by layer, and the output of the previous layer of decoders is used as the input of the next layer of decoders.
  • the nth word is determined according to the vector representation corresponding to the n-1th word output by the last layer of decoders in the speech recognition decoder.
  • the nth word obtained in this way does not contain the content corresponding to the word and the speaker corresponding to the word, so that the text content in the SOT format can be directly generated based on the predicted words in the future.
  • the context information is comprehensively considered when determining each word, so that the predicted words can be more accurate.
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, a text encoder including an attention layer, and a joint scorer.
  • the joint scorer includes a speaker-independent scorer and a speaker-dependent scorer.
  • the speaker-independent scorer can determine the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker according to the set similarity algorithm.
  • the speaker-dependent scorer includes a set scoring model including an attention module, so as to obtain the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker through the set scoring model including the attention module.
  • each word corresponding to the speech signal is predicted one by one in an iterative manner.
  • the specific recognition prediction process of each word is basically the same, and the recognition prediction process of the nth word is used as an example here.
  • the speech signals of multiple speakers and the portrait feature vectors of multiple speakers are obtained, wherein the speech signals contain the speech of multiple speakers.
  • the speech signals of multiple speakers are input into the speech recognition encoder and the speaker encoder, and the first vector representation corresponding to the speech signal is obtained through the speech recognition encoder, and the second vector representation corresponding to the speech signal is obtained through the speaker encoder.
  • the first vector representation is set to perform speech recognition
  • the second vector representation is set to perform speaker recognition.
  • the first n-1 words that have been output by the speech recognition decoder are obtained, and the first n-1 words that have been output by the speech recognition decoder are encoded by the text encoder to obtain the third vector representation corresponding to the n-1th word.
  • the first vector representation, the second vector representation, and the third vector representation are input into the speaker decoder to obtain the speaker vector representation corresponding to the nth word.
  • the speaker vector representation corresponding to the nth word is spliced with the portrait feature vector of any speaker, and the spliced vector representation is input into the speaker-independent scorer.
  • the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined according to the set similarity algorithm.
  • the speaker vector representation corresponding to each of the first n words is spliced with the portrait feature vector of any speaker, and the multiple spliced vector representations are input into the scoring model including the attention module set in the speaker-dependent scorer to obtain the second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker.
  • the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined. Then, the weighted sum of the correlation coefficient to the portrait feature vectors of multiple speakers is calculated.
  • the weighted sum of the first vector representation, the first n-1 words, and the correlation coefficients for the portrait feature vectors of multiple speakers is input into the speech recognition decoder to obtain the posterior probability corresponding to the nth word. Then, the nth word is predicted based on the posterior probability corresponding to the nth word. Finally, a text sequence in SOT format is generated based on each word output by the speech recognizer.
  • a two-stage decoding process is performed.
  • the first stage decoding is to predict each word corresponding to the speech signal one by one according to the method in each of the above-mentioned embodiments.
  • the second stage decoding process is performed.
  • the second stage decoding is to obtain the prediction results of all N words after the first stage decoding process, and use the prediction results of these N words as the input of the text encoder, and then calculate the correlation coefficients between the speaker vector representation corresponding to an nth word and the portrait feature vectors of multiple speakers one by one, that is, to obtain the probability distribution of the speaker corresponding to each word in the N words, and according to the probability distribution of the speaker corresponding to each word in the N words, to re-obtain the speaker corresponding to each word, so that the result of the N words finally obtained can be made more accurate.
  • Its specific implementation process can be shown in Figure 8.
  • FIG8 is a flow chart of another speech recognition method provided by an embodiment of the present invention. As shown in FIG8 , in order to improve the accuracy of the speech recognition result of the method so that the speech recognition method can better combine the global context information, based on the above embodiment, the method may further include the following steps:
  • the complete text sequence output by the first stage speech recognition decoder is first determined. Then the complete text sequence is input into the text encoder, so that the text encoder encodes the text sequence to obtain the fourth vector representation corresponding to each word in the text sequence. Then, the first vector representation, the second vector representation and the fourth vector representation are input into the speaker decoder to obtain the speaker vector representation corresponding to each word. According to the correlation coefficient between the speaker vector representation corresponding to each word and the portrait feature vectors of multiple speakers, the speaker corresponding to each word is re-determined.
  • the first stage decoding is to predict the nth word based on the first n-1 words, and only the context information corresponding to the word can be obtained, but no context information.
  • the second stage decoding is to predict again based on all N words after all N words have been predicted once, so that the context information and context information can be seen for each word, which can make the prediction results of the speaker corresponding to each word obtained by recalculating more accurate.
  • the speech recognition method provided by the embodiment of the present invention can be executed in the cloud, and several computing nodes (cloud servers) can be deployed in the cloud, and each computing node has computing, storage and other processing resources.
  • cloud multiple computing nodes can be organized to provide a certain service.
  • a computing node can also provide one or more services.
  • the cloud can provide the service by providing a service interface to the outside, and the user calls the service interface to use the corresponding service.
  • the cloud can provide a service interface for speech recognition service.
  • the user calls the service interface through the terminal device to trigger a speech recognition service request to the cloud.
  • the request includes speech signals of multiple speakers and portrait feature vectors of multiple speakers.
  • the speech signal contains the speech of multiple speakers.
  • the cloud determines the computing node that responds to the request and uses the processing resources in the computing node to perform the following steps:
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder, and a text encoder including an attention layer;
  • the speech recognition output information is sent to the terminal device, wherein the speech recognition output information includes text sequences corresponding to multiple speakers.
  • the user can call the speech recognition service through the terminal device E1 illustrated in Figure 9 to perform speech recognition on the target speech signal to obtain a text sequence prediction corresponding to the target speech signal.
  • the service interface for the user to call the service includes a software development kit (Software Development Kit, referred to as SDK), an application programming interface (Application Programming Interface, referred to as API), and the like.
  • Figure 9 illustrates the situation of the API interface.
  • the speech recognition service is provided by the service cluster E2 and the service cluster E2 includes at least one computing node.
  • the service cluster E2 executes the steps in the aforementioned embodiment to obtain a text sequence corresponding to the target speech signal, including multiple speakers respectively, and feeds it back to the terminal device E1.
  • FIG 10 is a structural schematic diagram of a speech recognition device provided in an embodiment of the present invention.
  • the speech recognition model includes a speech recognition encoder, a speaker encoder, a speech recognition decoder, a speaker decoder and a text encoder including an attention layer.
  • the device includes: an acquisition module 11, a first encoding module 12, a second encoding module 13, a first decoding module 14, and a second decoding module 15.
  • the acquisition module 11 is configured to acquire speech signals of multiple speakers and portrait feature vectors of the multiple speakers, wherein the speech signals include the speech of the multiple speakers.
  • the first encoding module 12 is configured to obtain a first vector representation corresponding to the speech signal through the speech recognition encoder, and to obtain a second vector representation corresponding to the speech signal through the speaker encoder, wherein the first vector representation is configured to perform speech recognition, and the second vector representation is configured to perform speaker recognition.
  • the second encoding module 13 is configured to encode the first n-1 words output by the speech recognition decoder through the text encoder to obtain a third vector representation corresponding to the n-1th word.
  • the first decoding module 14 is configured to input the first vector representation, the second vector representation and the third vector representation into the speaker decoder to obtain the speaker vector representation corresponding to the nth word; and determine the speaker corresponding to the nth word according to the correlation coefficients between the speaker vector representation corresponding to the nth word and the portrait feature vectors of the multiple speakers.
  • the second decoding module 15 is configured to input the first vector representation, the first n-1 words and the weighted sum of the correlation coefficients to the portrait feature vectors of the multiple speakers into the speech recognition decoder to obtain the nth word.
  • the first decoding module 14 is further configured to: determine, for the portrait feature vector of any speaker among the multiple speakers, a first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker according to a set similarity algorithm; obtain the speaker vector representation corresponding to each of the first n words; and respectively compare the speaker vector representation corresponding to each of the first n words with the portrait feature vector of any speaker.
  • the invention relates to a method for obtaining a first speaker vector corresponding to each of the first n words and a portrait feature vector of any speaker, wherein the first speaker vector corresponding to the first n words is concatenated; the plurality of concatenated vector representations are input into a set scoring model including an attention module to obtain a second similarity between the speaker vector representation corresponding to each of the first n words and the portrait feature vector of any speaker; and the correlation coefficient between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker is determined according to the first similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker and the second similarity between the speaker vector representation corresponding to the nth word and the portrait feature vector of any speaker.
  • the second encoding module 13 is also configured to: obtain the weighted vector representations of the attention coefficients corresponding to the first n-1 words respectively obtained after the self-attention layer in the speech recognition decoder performs attention calculations on the first n-1 words respectively; input the weighted vector representations of the attention coefficients corresponding to the first n-1 words respectively into the text encoder to obtain a third vector representation corresponding to the n-1th word output by the text encoder.
  • the text encoder includes at least one cascaded encoder, each of which includes an attention layer and a feedforward neural network layer.
  • the speech recognition decoder includes a plurality of cascaded decoders, each of which includes an attention layer and a feedforward neural network layer; based on this, the second decoding module 15 is also configured to: perform embedding coding on the first n-1 characters respectively to obtain the embedding vector representations corresponding to the first n-1 characters respectively; input the embedding vector representations corresponding to the first n-1 characters respectively and the first vector representation into the first layer decoder of the speech recognition decoder to obtain the weighted vector representation of the attention coefficient corresponding to the n-1th character output by the attention layer in the first layer decoder; input the weighted vector representation of the attention coefficient corresponding to the n-1th character and the weighted sum into the feedforward neural network layer in the first layer decoder to obtain the vector representation corresponding to the n-1th character input to the second layer decoder; determine the nth character according to the vector representation corresponding to the n-1th character output by the last layer decoder in the speech recognition
  • the speaker decoder includes a plurality of cascaded decoders, each of which includes an attention layer and a feedforward neural network layer; based on this, the first decoding module 14 is further configured to: input the first vector representation, the second vector representation and the third vector representation into a first layer decoder in the speaker decoder to obtain a speaker vector representation weighted by the attention coefficient corresponding to the n-1th word output by the attention layer in the first layer decoder; input the speaker vector representation weighted by the attention coefficient corresponding to the n-1th word into the feedforward neural network layer in the first layer decoder to obtain a speaker vector representation corresponding to the n-1th word input to the second layer decoder; determine the speaker vector representation corresponding to the n-1th word output by the last layer decoder in the speaker decoder; and determine the speaker vector representation corresponding to the n-1th word according to the speaker vector representation corresponding to the n-1th word output by the last layer decoder and the speaker vector representation weighted by
  • the second decoding module 15 is further configured to: determine the complete text sequence output by the speech recognition decoder; encode the text sequence through the text encoder to obtain the text sequence; A fourth vector representation corresponding to each word in the column; inputting the first vector representation, the second vector representation and the fourth vector representation into the speaker decoder to obtain the speaker vector representation corresponding to each word; and re-determining the speaker corresponding to each word according to the correlation coefficients between the speaker vector representation corresponding to each word and the portrait feature vectors of the multiple speakers.
  • the device shown in FIG. 10 can execute the steps in the speech recognition method in the aforementioned embodiment.
  • the device shown in FIG. 10 can execute the steps in the speech recognition method in the aforementioned embodiment.
  • the embodiment of the present invention also provides an electronic device, as shown in FIG11 , which may include: a processor 21, a memory 22, and a communication interface 23.
  • the memory 22 stores executable codes, and when the executable codes are executed by the processor 21, the processor 21 implements the speech recognition method in the above embodiment.
  • an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored.
  • the processor can at least implement the speech recognition method provided in the aforementioned embodiment.
  • each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software.
  • the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
  • the embodiments of the present disclosure provide a speech recognition method, device and storage medium.
  • the specific implementation scheme is to add a text encoder including an attention layer to the speech recognition model.
  • the text encoder encodes the first n-1 words that have been output by the speech recognition decoder, so as to better aggregate the context information of the entire sentence to obtain a third vector representation containing context information corresponding to the n-1th word, that is, in this way, a vector representation that is more accurately set for speaker recognition can be obtained, thereby improving the accuracy of the speaker recognition results corresponding to each word.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

La présente divulgation concerne un procédé de reconnaissance de la parole, un dispositif et un support de stockage. Le procédé consiste à : acquérir une première représentation vectorielle correspondant à un signal vocal au moyen d'un codeur de reconnaissance de la parole et acquérir une deuxième représentation vectorielle correspondant au signal vocal au moyen d'un codeur de locuteur ; au moyen d'un codeur de texte, coder des premiers n-1 mots qui ont été délivrés par un décodeur de reconnaissance de la parole, pour obtenir une troisième représentation vectorielle correspondant à un (n-1)-ième mot ; entrer la première représentation vectorielle, la deuxième représentation vectorielle et la troisième représentation vectorielle dans un décodeur de locuteur pour obtenir une représentation vectorielle de locuteur correspondant à un n-ième mot ; selon des coefficients de corrélation entre la représentation vectorielle de locuteur correspondant au n-ième mot et des vecteurs de caractéristiques de portrait d'une pluralité de locuteurs, déterminer un locuteur correspondant au n-ième mot ; et entrer la première représentation vectorielle, les n-1 premiers mots et la somme pondérée des coefficients de corrélation sur les vecteurs de caractéristiques de portrait de la pluralité de locuteurs dans le décodeur de reconnaissance de la parole pour obtenir le n-ième mot.
PCT/CN2024/097609 2023-06-08 2024-06-05 Procédé de reconnaissance de la parole, dispositif et support de stockage Ceased WO2024251169A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310680032.X 2023-06-08
CN202310680032.XA CN116825095A (zh) 2023-06-08 2023-06-08 语音识别方法、设备和存储介质

Publications (1)

Publication Number Publication Date
WO2024251169A1 true WO2024251169A1 (fr) 2024-12-12

Family

ID=88123382

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/097609 Ceased WO2024251169A1 (fr) 2023-06-08 2024-06-05 Procédé de reconnaissance de la parole, dispositif et support de stockage

Country Status (2)

Country Link
CN (1) CN116825095A (fr)
WO (1) WO2024251169A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116825095A (zh) * 2023-06-08 2023-09-29 阿里巴巴（中国）有限公司 语音识别方法、设备和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109215662A (zh) * 2018-09-18 2019-01-15 平安科技（深圳）有限公司 端对端语音识别方法、电子装置及计算机可读存储介质
US20200043467A1 (en) * 2018-07-31 2020-02-06 Tencent Technology (Shenzhen) Company Limited Monaural multi-talker speech recognition with attention mechanism and gated convolutional networks
CN111009237A (zh) * 2019-12-12 2020-04-14 北京达佳互联信息技术有限公司 语音识别方法、装置、电子设备及存储介质
CN111899727A (zh) * 2020-07-15 2020-11-06 苏州思必驰信息科技有限公司 用于多说话人的语音识别模型的训练方法及系统
CN113889089A (zh) * 2021-09-29 2022-01-04 北京百度网讯科技有限公司 语音识别模型的获取方法、装置、电子设备以及存储介质
CN116825095A (zh) * 2023-06-08 2023-09-29 阿里巴巴（中国）有限公司 语音识别方法、设备和存储介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115713939B (zh) * 2023-01-06 2023-04-21 阿里巴巴达摩院(杭州)科技有限公司 语音识别方法、装置及电子设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200043467A1 (en) * 2018-07-31 2020-02-06 Tencent Technology (Shenzhen) Company Limited Monaural multi-talker speech recognition with attention mechanism and gated convolutional networks
CN109215662A (zh) * 2018-09-18 2019-01-15 平安科技（深圳）有限公司 端对端语音识别方法、电子装置及计算机可读存储介质
CN111009237A (zh) * 2019-12-12 2020-04-14 北京达佳互联信息技术有限公司 语音识别方法、装置、电子设备及存储介质
CN111899727A (zh) * 2020-07-15 2020-11-06 苏州思必驰信息科技有限公司 用于多说话人的语音识别模型的训练方法及系统
CN113889089A (zh) * 2021-09-29 2022-01-04 北京百度网讯科技有限公司 语音识别模型的获取方法、装置、电子设备以及存储介质
CN116825095A (zh) * 2023-06-08 2023-09-29 阿里巴巴（中国）有限公司 语音识别方法、设备和存储介质

Also Published As

Publication number Publication date
CN116825095A (zh) 2023-09-29

Similar Documents

Publication Publication Date Title
CN109785824B (zh) 一种语音翻译模型的训练方法及装置
CN106683677B (zh) 语音识别方法及装置
CN109979432B (zh) 一种方言翻译方法及装置
CN114097026A (zh) 语音识别的上下文偏置
US11721324B2 (en) Providing high quality speech recognition
EP4273855B1 (fr) Procédé et appareil de reconnaissance de la parole et support de stockage
KR20230086737A (ko) 단순화된 스트리밍 및 비스트리밍 스피치 인식을 위한 캐스케이드 인코더들
CN111081230A (zh) 语音识别方法和设备
CN112489651B (zh) 语音识别方法和电子设备、存储装置
JP7329393B2 (ja) 音声信号処理装置、音声信号処理方法、音声信号処理プログラム、学習装置、学習方法及び学習プログラム
CN112951211B (zh) 一种语音唤醒方法及装置
US11380315B2 (en) Characterizing accuracy of ensemble models for automatic speech recognition by determining a predetermined number of multiple ASR engines based on their historical performance
WO2024088262A1 (fr) Système et procédé de traitement de données pour modèle de reconnaissance vocale, et procédé de reconnaissance vocale
CN119547136A (zh) 用于罕见词语音辨识的场境感知神经置信度估计
CN117935787B (zh) 一种数据筛选标注方法、装置、电子设备和存储介质
EP3939034B1 (fr) Caractérisation de la précision de modèles d'ensemble pour la reconnaissance vocale automatique
CN115273862B (zh) 语音处理的方法、装置、电子设备和介质
CN113470617A (zh) 语音识别方法以及电子设备、存储装置
Thukroo et al. Spoken language identification system for Kashmiri and related languages using mel-spectrograms and deep learning approach
CN114495914B (zh) 语音识别方法、语音识别模型的训练方法及相关装置
KR20240068723A (ko) Rnn-T로 구현된 자동 음성 인식 시스템에서 음향과 텍스트 표현의 융합
WO2025183894A1 (fr) Décodage spéculatif par rap pour réduire la latence globale d'applications vocales
WO2024251169A1 (fr) Procédé de reconnaissance de la parole, dispositif et support de stockage
CN117711376A (zh) 语种识别方法、系统、设备及存储介质
CN113793599B (zh) 语音识别模型的训练方法和语音识别方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24818701

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE