JPH1097286A - Word / phrase classification processing method, collocation extraction method, word / phrase classification processing device, speech recognition device, machine translation device, collocation extraction device, and word / phrase storage medium - Google Patents

Abstract

(57) [Summary] [Problem] Automatically classify words and collocations. Kind Code: A1 A token is assigned to a word class string having a probability of occurrence in text data that is equal to or greater than a predetermined value, and a set in which words and tokens included in the word / token string of text data are mixed is defined as a word / token of text data. The sequence is divided so that the generation probability of the sequence is maximized, and the token is replaced with a collocation existing in the text data.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a word / phrase classification processing method, a collocation extraction method, a word / phrase classification processing device, a speech recognition device, a machine translation device, a collocation extraction device, and a word / phrase storage medium. This is suitable for automatically extracting collocations from text data and automatically classifying words and collocations.

[0002]

2. Description of the Related Art Conventional word classification processing apparatuses include, for example,
"Brown, P., Della Pietra,
V. , DeSouza, P .; , Lai, J. et al. , Merc
er, R .; (1992) "Class-Based n
-Gram Models of Natural La
nguage ".Computational Lin
guistics, Vol. 18, No4, pp. 46
7-479 ", a single word used in text data is statistically processed to automatically classify the single word.
Speech recognition and machine translation were performed using the classification result of the single word.

[0003]

However, the conventional word classification processing device cannot automatically classify words and collocations together, and cannot determine the correspondence and similarity between words and collocations or collocations and collocations. However, since speech recognition and machine translation cannot be performed using this method, speech recognition and machine translation cannot be performed accurately.

Accordingly, a first object of the present invention is to provide a word / phrase classification processing method and a word / phrase classification processing device that can automatically classify words and collocations collectively.

A second object of the present invention is to provide a collocation extracting apparatus capable of extracting a collocation from a large amount of text data at high speed. A third object of the present invention is to provide a speech recognition device capable of performing accurate speech recognition by using the correspondence and similarity between words and collocations or between collocations and collocations.

A fourth object of the present invention is to provide a machine translation apparatus capable of performing accurate machine translation by using the correspondence and similarity between words and collocations or between collocations and collocations.

[0007]

According to the present invention, in order to achieve the first object described above, words and collocations are classified together and words and collocations are mixed. Class is generated.

[0008] This makes it possible not only to classify words and words collectively, but also to classify words and collocations or collocations and collocations together. And similarity can be easily determined.

According to another aspect of the present invention, a word class obtained by classifying words is mapped to a one-dimensional sequence of words in text data to generate a one-dimensional sequence of word classes, and a primary class of the word class in text data is generated. In the original sequence, a word class sequence in which the degree of adhesion between adjacent word classes is all greater than or equal to a predetermined value is extracted, a token is added to the word class sequence, words and tokens are classified together, and Is replaced by a collocation belonging to the word class sequence.

Thus, a token is assigned to the word class string, the word class string is regarded as one word, and the word included in the text data and the word class string to which the token is assigned are treated equally, and Classification processing can be performed without distinction from collocations. In addition, by mapping a word class obtained by classifying words to a one-dimensional sequence of words in text data to generate a one-dimensional sequence of word classes, and extracting collocations based on the degree of adhesion between adjacent word classes, Extraction of collocations from data can be performed at high speed.

According to the present invention, in order to achieve the second object, a word class in which words are classified is mapped to a one-dimensional string of words in text data to generate a one-dimensional string of word classes. Then, in the one-dimensional sequence of the word classes of the text data, a word class sequence in which the degree of adhesion between adjacent word classes is all equal to or greater than a predetermined value is extracted, and the individual word classes constituting the word class sequence are adjacent to the text data. Each word that exists is extracted separately to extract collocations.

[0012] Thus, collocations can be extracted based on the word class sequence, and the number of word classes in which those words are classified is smaller than the number of different words existing in the text data. In the one-dimensional sequence of word classes, it is better to extract a word class sequence in which the degree of adhesion between adjacent word classes is equal to or more than a predetermined value. Compared with the case of extracting the above word strings, the amount of calculation and the memory capacity can be reduced, the collocation extraction processing can be performed at high speed, and memory resources can be saved. Note that the word class string may include a word string that does not exist in the one-dimensional string of the words in the text data. The individual words to be taken are separately taken out as collocations.

According to the present invention, in order to achieve the third object, words and collocations included in predetermined text data are classified and stored in a class in which words and collocations are mixed. By referring to the word / syllable dictionary, the pronunciation voice is recognized.

Thus, speech recognition can be performed using the correspondence and similarity between words and collocations or between collocations and collocations, and accurate processing can be performed. According to the present invention, in order to achieve the fourth object described above, words and collocations included in predetermined text data are classified and stored in a class in which words and collocations are mixed and stored. -Based on the collocation dictionary, the example original sentence stored in the example sentence collection is made to correspond to the input original sentence.

Thus, even when an original sentence in which words of the example original sentence stored in the example sentence collection are replaced with collocations is input, machine translation can be performed by applying the example original sentence to the input original sentence. In addition, accurate machine translation using the correspondence and similarity between words and collocations or between collocations and collocations becomes possible.

[0016]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a word / phrase classification processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. In this embodiment, words and collocations included in predetermined text data are classified into classes in which words and collocations are mixed.

FIG. 1 is a diagram showing words and words according to an embodiment of the present invention.
It is a block diagram which shows the functional structure of a collocation classification processing apparatus. In FIG. 1, a word classifying unit 1 extracts words different from each other from a one-dimensional string of words in text data, and divides a set of the extracted words to generate a word class.

FIG. 2 explains the processing of the word classifying means 1 and is a one-dimensional sequence of words (w ₁ w ₂ w ₃ w ₄ ... W _T ) composed of T words contained in the text data. From
V vocabulary words {v ₁ , v ₂ , v ₃ , v ₄ ,... Arranged in the order of appearance frequency in text data
, _{V V} }, and the words {v ₁ , v ₂ , v ₃ , v ₄ ,..., As a vocabulary of the text data.
v _V } is assigned an initialization class. Here, the number T of words is, for example, 50 million,
The number V of vocabularies is, for example, 7,000.

In the example of FIG. 2, the frequency of appearance in the text data is high, for example, "the", "a", "in",
“Of” corresponds to words v ₁ , v ₂ , v ₃ , and v ₄ as vocabularies, respectively. The words {v ₁ , v ₂ , v ₃ , v ₄ ,..., _{V V} } as V vocabularies to which the initialization class is assigned are converted into C word classes {C ₁ , C ₂ , C ₃ ,
C ₄ ,..., C _C }. Here, the number C of the word classes is, for example, 500.

In FIG. 2, for example, "spea"
k "," say "," tell "," talk "...
There are classified as word class _{C 1, "he", "} she",
"She" ··· is classified in the word class C _5, "ca
r "," track "," wagon "··· are classified in the word class _{C 32," Toyota ","} Nissa
n "," GM "··· is an example that has been classified as word class C _300.

The classification of the words composed of the V words {v ₁ , v ₂ , v ₃ , v ₄ ,..., _{V V} } as V vocabulary is, for example, such that two words existing in text data are When two word classes that belong to each other are merged, the one that minimizes the decrease in the generation probability of the original text data is integrated into the same word class. Here, the generation probability of the original text data by the class-by model can be expressed by using the average mutual information AMI, and the average mutual information AMI can be expressed by the following equation.

[0022]

(Equation 1)

Here, Pr (C_i) Is text data
One-dimensional sequence of words (w₁w_Twow_Threew_Four... w_T)
Replaced by the word class to which the word belongs, the text
Class C in a one-dimensional sequence of word classes_iOut of
Current probability, Pr (C_j) Is the primary word of the text data
Original column (w₁w_Twow_Threew_Four... w_T) To which the word belongs
If you replace it with a word class,
Class C in a one-dimensional sequence of word classes_jProbability of occurrence of Pr
(C_i, C_j) Is a one-dimensional sequence of words in the text data
(W₁w_Twow_Threew_Four... w_T) To which the word belongs
When replaced with a word class, the text data
Word class C in one-dimensional sequence of word classes_iNext to
Word class C _jIs the probability that appears.

FIG. 3 is a block diagram showing an example of a functional configuration of the word classifying means 1 of FIG. 3, the initialization class setting unit 10 extracts different words from each other from the one-dimensional string of words in the text data _{{w 1 w 2 w 3 w} 4 ··· w T}, words having a predetermined frequency ｛V ₁ , v
_{2, v 3, v 4,} ···, v V each unique word class {C _{1 of}, C 2, C 3,} C 4, assigns ..., and C _V}.

The temporary merging unit 11 generates a set of word classes {C
₁, C_Two, C_Three, C_Four, ..., C _MTwo words from｝
Class II C_i, C_j｝ Is taken out and temporarily merged. average
The mutual information calculation unit 12 calculates the temporary merging of the text data.
Word class ｛C₁, C_Two, C_Three, C_Four, ..., C
_M-1The average mutual information AMI for｝ is given by equation (1).
Calculated. In this case, a set of M word classes
｛C₁, C_Two, C _Three, C_Four, ..., C_MTwo from｝
Word class $ C_i, C_jHow to remove｝
Is M (M−1) / 2, so M (M−
1) It is necessary to calculate the average mutual information AMI twice.
is there.

Based on the M (M-1) / 2 pieces of average mutual information AMI calculated by the provisional merge, the merging unit 13 generates two word classes {C _i, which maximize the average mutual information AMI. , C _j } to a set of word classes {C ₁ , C ₂ ,
C ₃ , C ₄ ,..., C _M } Thus, the words belonging to any of the merged word classes {C _i , C _j } are classified into the same word class.

The word class sequence generating means 2 shown in FIG.
Word string (w₁w_Twow_Threew _Four... w_T)
Each word to be formed is represented by a word class {C} to which the word belongs.₁,
C_Two, C_Three, C_Four, ..., C_VBy replacing with｝
Then, a word class sequence of the text data is generated.

FIG. 4 shows a word class of text data.
It is a figure showing an example of a dimension sequence. In FIG. 4, word classification
By means 1, C word classes {C₁, C_Two, C_Three, C
_Four, ..., C_C｝ Has been generated, and
For example, word class C₁The vocabulary v₁, V₃₇,
.. Belong to the word class C_TwoThe vocabulary
Lee v_Three, V_Fifteen, ... belong to the word class C_Three
The vocabulary v_Two, V_Four... belong to
Word class C_FourThe vocabulary v ₇, V₉,
.. Belong to the word class C_FiveThe vocabulary
Lee v₆, V ₈, V₂₆, V_V, ... belong to
Word Class C₆The vocabulary v ₆, V_{twenty three}...
Belongs to the word class C₇The vocabulary v
_Five, V _Ten, ... belong.

In the one-dimensional string (w ₁ w ₂ w ₃ w ₄ ... W _T ) of the words of the text data, for example, the word as the vocabulary indicated by the word w ₁ is represented by v ₁₅ and the word w ₂ is represented by w ₁₅ The word as vocabulary is v ₂ , the word w
_The vocabulary indicated by ₃ is the word v ₂₃ and the word w ₄
Words as vocabulary indicated is v _4, word w ₅ is word v ₅ as vocabulary shown, the words in the vocabulary word w ₆ are indicated v _15, word w ₇ words v ₅ as vocabulary indicated, word The word as the vocabulary indicated by w ₈ is v ₂₆ , the word as the vocabulary indicated by word w ₉ is v ₃₇ , the word as the vocabulary indicated by word w ₁₀ is v ₂ ,..., the vocabulary indicated by word w _T word is assumed to be v _8.

In this case, the vocabulary v_FifteenIs the word kura
S_TwoThe word w₁Is the word class C_TwoTo
Mapped, vocabulary v_TwoIs the word class C_Three
The word w_TwoIs the word class C_ThreeMapi
Vocabulary v _{twenty three}Is the word class C₆Belongs to
The word w_ThreeIs the word class C₆Mapped to
Vocabulary v_FourIs the word class C_ThreeBelongs to
So the word w_FourIs the word class C_ThreeIs mapped to
Cabrio_FiveIs the word class C₇Belongs to
Word w_FiveIs the word class C₇Mapped to the vocabulary
Rally v_FifteenIs the word class C_TwoThe word w
₆Is the word class C_TwoMapped to vocabulary
v_FiveIs the word class C₇The word w₇Is simply
Word Class C₇Is mapped to the vocabulary v₂₆Is
Word class C_FiveThe word w₈Is the word kura
S_FiveIs mapped to the vocabulary v₃₇Is the word
Las C₁The word w₉Is the word class C₁
Is mapped to the vocabulary v_TwoIs the word class C
_ThreeThe word w_TenIs the word class C_ThreeTo
Pinged, ..., vocabulary v₈Is a word class
C_FiveThe word w_TIs the word class C_FiveNima
Be pinged.

That is, the one-dimensional sequence (w ₁ w ₂ w ₃ w ₄ ... W _T ) of the words of the text data is represented by C word classes {C ₁ , C ₂ , C ₃ , C ₄ ,. ., As a result of mapping by C _C }, a one-dimensional sequence of word classes of text data (C ₂ C ₃ C ₆ C ₃ C ₇ C ₂ C ₇ C ₅ C ₁₎
C ₃ ... C ₅ ) are generated in a one-to-one correspondence.

The word class string extracting means 3 shown in FIG. 1 converts a word class string in which the degree of adhesion between word classes in the one-dimensional string of the word class of the text data is all equal to or more than a predetermined value into one-dimensional word class of the text data. Extract from a column. Here, the degree of adhesion between the word classes is an index indicating the strength of connection between the word classes constituting the word class sequence. As an expression of the degree of adhesion, for example, the mutual information M
I, correlation coefficient, cosine measure, liklihood
ratio.

In the following description, a case where a word class string is extracted from a one-dimensional string of a word class of text data by using the mutual information MI as the degree of adhesion between word classes will be described as an example.

FIG. 5 is a diagram showing an example of a word class string extracted by the word class string extracting means 3. In FIG. 5, a one-dimensional string (w ₁ w ₂ w ₃
.. w _T ), the one-dimensional sequence (C ₂ C ₃ C ₆ C ₃ C ₇ C ₂ C ₇ ...) of the word class of the text data is obtained as a result of mapping to w ₄ w ₅ w ₆ w _7. C ₅ ) is one-on-one
It is assumed that it is generated in correspondence. One-dimensional sequence (C ₂ C ₃ C ₆ C ₃ C ₇ C ₂ C) of the word class of this text data
From ₇ ··· C _5), two adjacent word classes (C _i, C _j) sequentially taken out, two adjacent word classes (C _i, mutual information MI about C _j)
(C _i , C _j ) is calculated by the following equation (2).

MI (C _i , C _j ) = log {Pr (C _i , C _j ) / (Pr (C _i ) Pr (C _j ))} (2) and two adjacent word classes (C _i, C _j) mutual information MI (C _i, C _j) for the case of more than a predetermined threshold value TH, 2 single word class thereof adjacent (C
_i , C _j ) are connected by a class chain.

For example, in FIG. 5, the mutual information MI (C ₂ ) for two adjacent word classes (C ₂ , C ₃ )
₂ , C ₃ ), two adjacent word classes (C ₃ , C ₆ )
Mutual information MI (C ₃ , C ₆ ) for
Mutual information M about two word classes (C ₆ , C ₃ )
I (C ₆ , C ₃ ), two adjacent word classes (C ₃ ,
C ₇₎ mutual information MI about (C _3, C _7), mutual information MI about two adjacent word classes _{(C 7, C 2) (} C 7, C 2), 2 single word class adjacent (C _2, C ₇₎ mutual information MI (C ₂ for,
C ₇ ),... Are sequentially calculated by equation (2).

Then, the mutual information MI (C_Two, C_Three),
Mutual information MI (C_Three, C₇), Mutual information MI
(C₇, C_Two), ... are greater than or equal to the threshold value TH
Information MI (C_Three, C₆), Mutual information MI (C₆,
C_Three), Mutual information MI (C_Two, C₇), ...
If the value is less than TH, two adjacent word classes
(C_Two, C _Three), (C_Three, C₇), (C₇, C_Two),
・ ・ By connecting each with a class chain, the word
Class column C_Two-C_Three, C_Three-C₇-C_TwoExtract ...
I do.

FIG. 6 is a block diagram showing an example of a functional configuration of the word class string extracting means 3 of FIG. In FIG. 6, the word class extracting unit 30 sequentially extracts two adjacent word classes (C _i , C _j ) from a one-dimensional sequence of the word classes of the text data.

The mutual information calculator 31 calculates the two word classes (C _i ,
Mutual information MI (C _i of C _j), is calculated by the C _j) (2) expression.

The class chain connecting unit 32 connects two word classes (C _i , C _j ) whose mutual information MI (C _i , C _j ) is equal to or larger than a predetermined threshold value by a class chain. The token assigning means 4 in FIG. 1 assigns a token to the word class strings connected by the class chain by the word class string extracting means 3.

FIG. 7 shows the state of the token provided by the token providing means 4.
FIG. 6 is a diagram illustrating an example of a token obtained. In FIG.
A word class sequence connected by a lath chain is, for example, C₁
-C _Three, C₁-C₇, ..., C_Two-C_Three, C_Two−
C₁₁, ..., C₃₀₀-C₃₂, ..., C₁-C_Three-C
₈₀, C₁-C_Four-C_Five, C_Three-C₇-C_Two, ..., C
₁-C₉-C₁₁-C₃₂, ... In this case, the word
Class column C₁-C_ThreeFor the token t₁And simply
Word class sequence C₁-C₇For the token t_Two, And
..., word class sequence C_Two-C_ThreeFor the token t_Three
And the word class sequence C_Two-C₁₁For the token t
_Four, ..., word class sequence C₃₀₀-C₃₂Against
T token_Five,..., Word class sequence C₁
-C_Three-C₈₀For the token t₆Grant the word club
Row C₁-C_Four-C_FiveFor the token t₇, And
Word class sequence C_Three-C₇-C_TwoFor the token t₈To
…, Word class sequence C₁-C₉-C₁₁-C₃₂
For the token t₉Is given.

The word / token string generation means 5 in FIG.
One-dimensional sequence of words in the text data (w ₁w_Twow_Threew_Fourw_Five
w₆w₇... w_T), Word class string extracting means 4
Talks word strings belonging to word class strings extracted by
By replacing with words, words and words in the text data
Generate a one-dimensional sequence of kuns.

FIG. 8 is a diagram showing an example of a one-dimensional sequence of words and tokens of text data. In FIG. 8, a one-dimensional string (w ₁ w ₂ w ₃ w ₄ w ₅ w
₆ w ₇ ... W _T ), the result is a one-dimensional sequence (C ₂ C ₃ ) of the word class of the text data.
C ₆ C ₃ C ₇ C ₂ C ₇ ... C ₅ ) are generated in a one-to-one correspondence, and word class strings C ₂ -C ₃ and C ₃ -C _7- connected by class chains. It is assumed that tokens t ₃ , t ₈ ,... Are given to C ₂ ,.

In this case, the words connected by the class chain
Class column C_Two-C_ThreeWord string of text data belonging to
(W₁w_Two) To the token t_ThreeReplace with
Word class sequence C_Three-C₇-C_TwoBelongs to
Word string of text data (w_Fourw _Fivew₆) To the token t₈
Can be replaced by words / toes in text data.
One-dimensional sequence of kung (t_Threew_Threet₈w₇... w_T)Generate a
I do.

FIG. 9 shows words and tokens of text data.
FIG. 5 is a diagram showing an example of a one-dimensional sequence in English. Figure
9 (b) one-dimensional string of words in the text data (w₁w_Two
w_Threew_Fourw_Fivew₆w ₇w₈w₉w_Tenw₁₁w₁₂w₁₃w₁₄w
_Fifteen) As “Hewent to th” in FIG.
e apartment by bus and sh
e sent to New York by pla
ne ”corresponds to the one-dimensional sequence of this word.
(W₁w_Twow_Threew_Fourw_Fivew₆w₇w₈w₉w_Tenw₁₁w₁₂
w₁₃w ₁₄w_Fifteen) One-dimensional one-to-one correspondence of word classes
The column is (C) in FIG._FiveC₉₀C _ThreeC_{twenty one}C₁₈C₁₀₁C₃₂C
_TwoC_FiveC₉₀C_ThreeC₆₃C₂₈C₁₀₁C₃₂What is given in)
And

The one-dimensional sequence (C_FiveC₉₀C_Three
C_{twenty one}C₁₈C₁₀₁C₃₂C_TwoC_FiveC₉₀C _ThreeC₆₃C₂₈C₁₀₁C
₃₂), Two adjacent word classes (C_i,
C_j) Mutual information MI (C_i, C_j) Calculate and mutual
Information amount MI (C₆₃, C₂₈) Is greater than or equal to a predetermined threshold TH,
Mutual information MI (C_Five, C₉₀), MI (C₉₀, C_Three),
MI (C_Three, C_{twenty one}), MI (C_{twenty one}, C₁₈), MI
(C₁₈, C₁₀₁), MI (C₁₀₁, C₃₂), MI
(C₃₂, C_Two), MI (C_Two, C_Five), MI (C_Five, C
₉₀), MI (C₉₀, C_Three), MI (C_Three, C₆₃), MI
(C₂₈, C₁₀₁) And MI (C₁₀₁ , C₃₂) Is the prescribed threshold
If the value is less than TH, two adjacent word classes
(C₆₃, C₂₈), As shown in FIG.
Are tied together.

Two words connected by this class chain
Class (C₆₃, C₂₈) Is the token t ₁Is replaced by
As shown in FIG. 9E, a one-dimensional sequence of words and tokens
(W₁w_Twow_Threew_Fourw_Fivew₆w₇w₈w₉w_Tenw₁₁t₁
w₁₄w_Fifteen) Is generated.

The word / token classification means 6 in FIG. 1 performs a set of N words {w ₁ , w ₂ , w ₃ , w _{4 ,.} ｝ Or L
Set of tokens {t ₁ , t ₂ , t ₃ , t ₄ ,.
, T _L }, so that D words and tokens {T ₁ ,
T ₂ , T ₃ , T ₄ ,..., T _D } are generated.

In the word / token classification means 6, the word class sequence to which the token has been assigned is regarded as one word, and the words {w ₁ , w ₂ ,
w ₃ , w ₄ ,..., w _N } and tokens {t ₁ , t ₂ ,
Since t ₃ , t ₄ ,..., t _L } can be treated equivalently, the words {w ₁ , w ₂ , w ₃ , w ₄ ,.
w _N } and tokens {t ₁ , t ₂ , t ₃ , t ₄ ,.
FIG. 10 that can perform classification processing without distinction from t _L ｝
FIG. 3 is a block diagram showing a functional configuration of the word / token classification means 6 of FIG.

In FIG. 10, an initialization class setting section 40
Extracts different words and different tokens from the word / token sequence of the text data, and generates N words {w ₁ , w ₂ , w ₃ , w ₄ ,.
.., w _N } and L tokens {t ₁ , t ₂ , t ₃ , t
_4, allocates ..., each unique word token class {T ₁ and _{t L}, T 2, T} 3, T 4, ···, the T _Y}.

The temporary merging unit 41 generates two word / token classes {T _i , T _j } from the set of word / token classes {T ₁ , T ₂ , T ₃ , T ₄ ,..., T _M }. Take out and temporarily merge.

The average mutual information calculation section 42 calculates the word / token class {T ₁ , T
The average mutual information AMI for ₂ , T ₃ , T ₄ ,..., T _M-1 } is calculated by equation (1). In this case, M
Set of word classes and token classes {T ₁ , T ₂ ,
From T ₃ , T ₄ ,..., T _M }, two words / token classes {T _i , T _j } are extracted.
Since only M (M-1) / 2 exist, M (M-1) /
It is necessary to calculate the average mutual information AMI twice.

Based on the M (M−1) / 2 average mutual information AMI calculated by the temporary merge, the main merge unit 43 generates two words / maximum average mutual information AMI.
A token class {T _i , T _j } is a set of word classes and token classes {T ₁ , T ₂ , T ₃ , T ₄ ,.
Take out from T _Mす る and perform full merging. This allows
The words and tokens belonging to any of the merged word / token classes {T _i , T _j } are classified into the same word class / token class.

The collocation unit 7 in FIG. 1 converts the tokens in the word / token class into word / token sequence generation units 5
Is reversely replaced with the word string replaced by, to generate a collocation.
FIG. 11 is a diagram for explaining the relationship between class chains and collocations.

In FIG. 11, for example, word class C
₃₀₀And word class C₃₂Are connected by a class chain,
Class string C connected by the class chain₃₀₀-C₃₂
To the token t_FiveIs given. Also the word
"Toyota", "Nissan", "GM" ...
A words like word class C₃₀₀Belong to the word "c"
ar "," track "," wagon "...
B words are in word class C ₃₂It belongs to

In this case, as a candidate for the collocation,
As shown in (b), “Toyotacar”, “Toyotacar”
yota track ”,“ Toyota wago ”
n "," Nissant car "," Nissant t "
rack "," Nissanwagon "," GM c
ar "," GM track "," GM wago "
n ”,..., the number of permutations of A words belonging to word class C ₃₀₀ and B words belonging to word class C ₃₂
Only × B of collocation candidates are generated. Since the collocation candidates include collocations that do not exist in the text data, the text data is scanned to extract only the collocations present in the text data from these collocation candidates. For example, text data includes “Nissa
“n track” and “Toyota wagon” exist, but “Toyota car”, “Toyota wagon”
a track "," Nissan car "," N
issan wagon ”,“ GM car ”,“ GM
When “track” and “GM wagon” do not exist, as shown in FIG. 11C, “Nissant t”
Only “track” and “Toyota wagon” are extracted from the text data as collocations.

FIG. 12 shows C word classes {C ₁ ,
_{C 2, C 3, C 4} , ···, C C}, D number word token classes _{{T 1, T 2, T} 3, T 4, ···, T D}
And D word / syllable classes {R ₁ , R ₂ , R ₃ ,
R _4, ···, is a diagram showing an example of R _D}.

In FIG. 12A, C word classes {C ₁ , C ₂ , C ₃ , C ₄ ,..., C _C } are generated by the word classifying means 1 of FIG. "He",
Words such as “she”, “it”,.
Belongs to _{5, "York", "London} " words such as ... belong to the word class _{C 28, "car", "} tra
ck "," wagon "words such as ... belong to the word class _{C 32," new ","} old " words such as ... belong to the word class _{C 63," Toyota ","} Nis
words such as “san”, “GM”,.
It belongs to ₃₀₀ . The text data includes “New York” and “Nissantrac”.
It is assumed that a number of collocations of "k" and "Toyota wagon" exist.

The C word classes {C ₁ , C ₂ ,
_{C 3, C 4, ···,} C C} one-dimensional string of words in the text data _{(w 1 w 2 w 3 w} 4 ··· w T) in a one-dimensional word classes mapped one-to-one correspondence Figure 1
Determines that the degree of adhesion between the word class C ₆₃ to which “new” belongs and the word class C ₂₈ to which “York” belongs is large, and classifies the word class C ₆₃ and the word class C ₂₈ into classes. Connect with a chain. In addition, the word class string extracting means 3 determines that the word class C _{300 to} which “Toyota” and “Nissan” belong and “track” and “wag”
It determines that the tackiness of the word class C ₃₂ which on "belongs is large, connecting the word class C ₃₀₀ and the word class C ₃₂ in the class chain.

The token assigning means 4 generates the word class string C ₆₃
The token t ₁ is given to -C _28, word class sequence C ₃₀₀ -
The token t ₅ given to the C _32. The word / token string generation means 5 generates a one-dimensional string (w ₁ w ₂
w ₃ w ₄ ··· w _T) to present "New York"
The replaced by a token t _1, exist in a one-dimensional string of words of text data _{(w 1 w 2 w 3 w} 4 ··· w T) "N
issan track ”and“ Toyota wag ”
to generate a one-dimensional string of words or token by replacing the on "in the token t _5.

The word / token classification means 6
"He", "she",
“It”, “London”, “car”, “trac”
k ”,“ wagon ”... and“ t ”₁”,
"T_FiveClassification processing for tokens such as "
12 (b) D words / token class @T₁,
T _Two, T_Three, T_Four, ..., T_DGenerate｝.

The word / token class {T ₁ , T ₂ ,
In T ₃ , T ₄ ,..., T _D }, for example, “h
e "," she "," it " word or token, such as ... belong to the word token class _{T 5," t 1 ",} " L
ondon "word or token, such as ... belong to the word token class _{T 28," car ","} track ",
Words or tokens such as “wagon”, “t ₅ ”... Belong to the word / token class T ₃₂ , and “new”, “o”
ld "word or token, such as ... belong to the word token class _{T 63," Toyota ","} Nissa
n "," GM "word or token, such as ... are belong to the word token class T _300. In this way, the words,
Token class {T ₁ , T ₂ , T ₃ , T ₄ ,..., T
_{In D}単 語, words and tokens are classified in a mixed manner without distinction between words and tokens.

The collocation unit 7 converts the word / word of FIG.
Token class {T ₁ , T ₂ , T ₃ , T ₄ ,..., T
Tokens such as “t ₁ ” and “t ₅ ” existing in _D
By reversely permuting words and collocation classes {R ₁ , R ₂ , R ₃ , R ₄ ,..., R _D } in FIG. Generate. For example, the token t ₁ that belongs to the word token class T ₂₈ is the word token string generating means 5,
"New" existing in a one-dimensional string of words of text data
This token t ₁
Is replaced with “New York” to generate a word / collocation class R ₂₈ and a word / token class T ₃₂
The token t ₅ that belong to, by the word token string generating means 5, exist in a one-dimensional string of words in the text data "Nissan track" and "Toyota
wagon ", the token t
₅ for “Nissan track” and “Toyota
By inverse replaced by wagon in ", to generate the word-phrase class R _32.

FIG. 13 is a block diagram showing a system configuration for realizing the word / phrase classification processing apparatus of FIG. FIG.
3, the memory interfaces 42, 46, the CPU 43, the ROM 44, the work RAM 45, the RAM 47, the driver 71, and the communication interface 72 of the word / phrase classification processing unit 41 are connected to each other via a bus 48, and the text data 40 When input to the processing unit 41, the CPU 43 processes the text data 40 according to a program stored in the ROM 44, and performs a classification process of words and collocations of the text data 40. The classification processing result of the words and collocations in the text data 40 is
It is stored in the collocation dictionary 49. The text data 40
Interface processing results of classification of words and words and collocations
2 can also be transmitted and received via the communication network 73.

A program for classifying words and collocations is stored in the hard disk 74 and the IC memory card 7.
5. Magnetic tape 76, floppy disk 77 or C
The program may be loaded into the RAM 47 from a storage medium such as a D-ROM or a DVD-ROM, such as an optical disk 78, and then executed by the CPU 43.

Further, a program for classifying words and collocations can be extracted from the communication network 73 via the communication interface 72. As a communication network 73 connected to the communication interface 72,
For example, LAN (Local Area Network)
k), WAN (Wide Area Network)
k), the Internet, an analog telephone network, a digital telephone network (ISDN: Integral Service D)
It is possible to use a wireless communication network such as digital network (PTE), PHS (Personal Handy System) and satellite communication.

FIG. 14 shows the word / phrase classification processing apparatus of FIG.
6 is a flowchart showing the operation of the first embodiment. In FIG.
First, as shown in step S1, word clustering
Perform processing. In this word clustering process,
One-dimensional sequence of words (w₁w_Twow _Threew_Four... w_TAs)
V words that are different from each other
｛V ₁, V_Two, V_Three, V_Four, ..., v_VExtract｝,
Set of V words ｛v₁, V _Two, V_Three, V_Four, ...,
v_VLet｝ be C word classes ｛C₁, C_Two, C_Three, C_Four,
..., C_CThe first clustering process of dividing into｝
Do.

Here, V words {v ₁ , v ₂ , v ₃ ,
v ₄ ,..., v _V }, respectively, the word class {C ₁ , C
₂ , C ₃ , C ₄ ,..., C _V }
Word classes {C ₁ , C ₂ , C ₃ , C ₄ ,..., C
_V }, the number of V word classes {C ₁ , C ₂ , C ₃ , C ₄ ,..., C _V } is reduced by one, and C word classes {C _1, C _2,
When generating C ₃ , C ₄ ,..., C _C }, V is 70
When the number is 00 and becomes large, the number of calculations of the average mutual information AMI of the equation (1) for performing the merge processing becomes enormous, which is not realistic. Therefore, window processing is performed to reduce the number of word classes to be merged.

FIG. 15 is a diagram for explaining window processing.
is there. In FIG. 15A, V data of text data
The word $ v₁, V_Two, V_Three, V _Four, ..., v_V｝ Each
V word classes assigned to them₁, C_Two, C
_Three, C_Four, ..., C_V｝ Of the text data
C + 1 words assigned to words with high appearance frequency
Class II C₁, C_Two, C_Three, C_Four, ..., C _C, C
_{C + 1}、, and C + 1 word classes ｛C₁,
C_Two, C_Three, C_Four, ..., C_C, C_{C + 1}}about
Perform merge processing.

Here, as shown in FIG.
Word class ｛C₁, C_Two, C_Three, C_Four, ...,
C_M｝ Is C + 1 word classes in the window
｛C₁, C_Two, C_Three, C_Four, ..., C_C, C_{C + 1}Puni
When the merge process is performed, M word classes
｛C₁, C_Two, C_Three, C_Four, ..., C_MOne 数
Reduced M-1 word classes ｛C₁, C_Two, C_Three,
C_Four, ..., C_M-1｝ And in the window
C + 1 word classes ｛C₁, C_Two, C_Three, C_Four,
・ ・ 、 C _C, C_{C + 1}The number of｝ is also reduced by one to C word classes.
｛C₁, C_Two, C_Three, C_Four, ..., C_CIt becomes｝.

[0071] In this case, as shown in FIG. 15 (c), the window outside the word class _{{C C + 1, ···,} C M-1} of the largest word classes the frequency of occurrence of a text data Put _{CC + 1} in the window so that the number of word classes in the window is kept constant.

Then, there are no word classes outside the window, and the C word classes {C ₁ , C ₂ ,
When C ₃ , C ₄ ,..., C _C } are generated, the word clustering process ends.

In the above embodiment, the number of word classes in the window is set to C + 1. However, the number of word classes may be less than V other than C + 1, or may be changed in the middle.

FIG. 16 is a flowchart showing the word clustering process in step S1. 16, first, as shown in step S10, based on the text data as the T dimensional row of words _{(w 1 w 2 w 3 w} 4 ··· w T), all except the overlapping The frequency of appearance of the _V words {v ₁ , v ₂ , v ₃ , v ₄ ,..., _{V V} } is checked, and these V words {v ₁ , v ₂ , v ₃ ,
v ₄ ,..., v _V } are arranged in descending order of the frequency of occurrence, and these V words {v ₁ , v ₂ , v ₃ , v ₄ ,.
, _{V V} } are represented by V word classes {C ₁ ,
C ₂ , C ₃ , C ₄ ,..., C _V }.

Next, as shown in step S11, V word classes {C ₁ , C ₂ , C ₃ , C ₄ ,.
Of the words of C _V }, less than V words of the C + 1 word class from words of the word class having a high frequency of occurrence are defined as words of the word class in one window.

Next, as shown in step S12, among the words of the word class in one window, provisional pairs of all combinations are created, and the average mutual information AMI when the provisional pairs are provisionally merged is calculated. It is calculated by equation (1).

Next, as shown in step S13, the average mutual information AMI for the temporary pairs of all the combinations is determined.
Among them, the temporary pair having the maximum average mutual information amount AMI is main-merged to reduce the word class by one, and the word of the word class in one window after the main merge is updated.

Next, as shown in step S14, it is determined whether there are no more word classes outside the window and there are C word classes in the window. If this condition is not satisfied, the flow advances to step S15. The words of the class having the maximum appearance frequency outside the current window are put in the window, and the process returns to step S12 to repeat the above processing to reduce the number of word classes.

On the other hand, the condition of step S14 holds,
If there are no word classes outside the window and the number of word classes becomes C, the process proceeds to step S16, and the C word classes {C ₁ , C ₂ , C ₃ , C ₄ ,.
.., C _C } are stored in the memory.

Next, as shown in step S2 of FIG. 14, a class chain extraction process is performed. In this class chain extraction processing, in a one-dimensional sequence of word classes of text data generated based on the first clustering processing in step S1, two adjacent word classes having a mutual information amount equal to or more than a predetermined threshold value are included. Are connected by a chain to extract a set of word class strings connected by the chain.

FIG. 17 is a flowchart showing a first embodiment of the class chain extracting process in step S2. In FIG. 17, first, as shown in step S20, two adjacent word classes (C _i , C _j ) are extracted from the one-dimensional sequence of the word classes of the text data.

Next, as shown in step S21, the two word classes (C _i ,
Mutual information MI (C _i for C _j), is calculated by the C _j) (2) expression.

Next, as shown in step S22, it is determined whether or not the mutual information amount MI (C _i , C _j ) calculated in step S21 is equal to or greater than a predetermined threshold value TH, and the mutual information amount MI (C _i , C _j ) is determined. If C _i , C _j ) is equal to or greater than the predetermined threshold value TH, the process proceeds to step S23, where the two word classes (C _i , C _j ) extracted in step S20 are connected in a class chain and stored in a memory. , Mutual information MI
When (C _i , C _j ) is smaller than a predetermined threshold TH,
Step S23 is skipped.

[0084] Next, as shown in step S24, the word class connected by class chain stored in the memory, to determine whether the class chain ending with the word classes C _i is present, the words If the class chain ending with the class C _i is present, step S
Proceed to 25, connecting the word class C _j in class chain that ends with the word class C _i.

On the other hand, in step S24, the word
S_iIf there is no class chain ending with
In this case, step S25 is skipped. Next, step S
As shown in FIG. 26, the primary
From the source column, two adjacent word classes (C_i, C
_jJudge whether all of them have been taken out and
Two word classes (C_i, C_jPlace where all of them were taken out
End the class chain extraction process and
Two word classes (C _i, C_j) Has not been taken out
If not, return to step S20 and repeat the above processing.
You.

FIG. 18 is a flowchart showing a second embodiment of the class chain extracting process in step S2. In FIG. 18, first, as shown in step S201,
Two adjacent word classes (C _i , C _j ) are sequentially extracted from the one-dimensional sequence of the word classes of the text data. Then, the extracted two word classes (C _i ,
For C _j ), the mutual information MI (C _i , C _j ) is calculated by equation (2) to extract all the class chains of length 2 from the one-dimensional sequence of the word classes of the text data.

Next, as shown in step S202, all class chains of length 2 are replaced with objects. Here, the object represents the same token as the above-mentioned token, but the token given to the class chain of length 2 is particularly called an object.

Next, as shown in step S203, the one-dimensional sequence of the text data class is processed in step S2.
02, the class chain of length 2 to which the object is assigned is replaced with the object, and a class of the text data and a one-dimensional sequence of the object are generated.

Next, as shown in step S204, one object existing in the one-dimensional sequence of the text data class and the object is regarded as one class,
Mutual information MI about two classes (C _i , C _j )
(C _i , C _j ) is calculated by equation (2). That is,
When the mutual information MI (C _i , C _j ) in the one-dimensional sequence of the class of the text data and the object is calculated between one adjacent class and one class, one adjacent class is calculated. And between one object (a class chain of length 2)
And one object (class chain of length 2) and one object (class chain of length 2) adjacent to each other.

Next, as shown in step S205, the mutual information MI (C _i ,
C _j ) is determined to be greater than or equal to a predetermined threshold TH, and the mutual information MI (C _i , C _j ) is determined to be equal to or smaller than the predetermined threshold T
If not less than H, the process proceeds to step S26, where two classes adjacent to each other extracted in step S204, or one class and one object adjacent to each other,
Alternatively, when two objects adjacent to each other are connected by a class chain and the mutual information amount MI (C _i , C _j ) is smaller than a predetermined threshold value TH, step S206 is skipped.

FIG. 19 is a diagram showing a class of text data and a class chain extracted in a one-dimensional sequence of objects. In FIG. 19, when a class chain is extracted between one class and one class adjacent to each other, a class chain (object) having a length of 2
Is generated, and a class chain is extracted between one class and one object adjacent to each other, a class chain of length 3 is generated, and a class chain between one adjacent object and one object is generated. When a class chain is extracted, a class chain having a length of 4 is generated.

Next, as shown in step S207 of FIG. 18, it is determined whether or not the class chain extraction processing has been performed a predetermined number of times. If the predetermined number of times has not been performed, the flow returns to step S202 to perform the above processing. repeat.

As described above, the class chain of length 2 is replaced with an object, and the mutual information MI (C _i ,
By repeatedly calculating C _j ), a class chain having an arbitrary length can be extracted.

Next, as shown in step S3 of FIG. 14, a token replacement process is performed. In this token replacement processing, a unique token is made to correspond to the word class string extracted in the class chain extraction processing in step S2, and a word string belonging to this word class string is searched for from a one-dimensional string of words in the text data. By replacing the word string of the data with the corresponding token, a one-dimensional string of words and tokens for the text data is generated.

FIG. 20 is a flowchart showing the token replacement processing in step S3. In FIG. 20, first, as shown in step S30, the extracted class chains are sorted according to a predetermined rule except for duplication, tokens are made to correspond to the respective class chains, and names are given to the class chains. Here, the sorting of the class chains is performed, for example, in ASCII code order.

Next, as shown in step S31, one class chain corresponding to the token is extracted. Next, as shown in step S32, it is determined whether or not there is a word string belonging to the word class string connected by the class chain in the one-dimensional string of words of the text data, and the word connected by the class chain is determined. If there is a word string belonging to the class string, the process proceeds to step S33, where the corresponding word string in the text data is replaced with one token, and the word string belonging to the word class string connected by the class chain is the primary word of the text data. The above processing is repeated until there is no longer existing in the original sequence.

On the other hand, if there is no word string belonging to the word class string connected by the class chain, step S34
It is determined whether the collocation / token replacement process has been completed for all the class chains corresponding to the tokens in step S30. If the collocation / token replacement process has not been completed for all the class chains, the process proceeds to step S30. Returning to S31, one new class chain is taken out, and the above processing is repeated.

Next, as shown in step S4 of FIG. 14, a word / token clustering process is performed. In the word / token clustering process, different words and different tokens are extracted from a one-dimensional sequence of words and tokens of text data, and a set in which words and tokens coexist is defined as a word / token class {T ₁ , T ₂ , T ₃ , T ₄ ,..., T _D }.

FIG. 21 shows the word / token in step S4.
It is a flowchart which shows a clustering process. FIG.
In step S1, as shown in step S40,
One-dimensional words and tokens of the text data obtained in step 3
Using the sequence as input data, the first word class
Perform clustering in the same way as the stalling process.
And more words / token class @T₁, T_Two, T_Three, T
_Four, ..., T_DGenerate｝. This second cluster
The tokening process does not distinguish between words and tokens,
Treated as one word. Also, each generated
Word / token class @T ₁, T_Two, T_Three, T_Four,
・ ・ 、 T_D含 ん contains words and tokens as its elements
In.

Next, a data output process is performed as shown in step S5 of FIG. In this data output process,
Of the word strings existing in the one-dimensional string of words of the text data, those corresponding to the tokens are extracted as collocations, and the word / token class {T ₁ , T ₂ , T ₃ , T ₄ ,.
By replacing the tokens in T _D } with collocations, a set in which words and collocations are mixed is converted into a word / composition class {R ₁ ,
A third clustering process for dividing into R ₂ , R ₃ , R ₄ ,..., R _D } is performed.

FIG. 22 is a flowchart showing the data output processing in step S5. In FIG. 22, first,
As shown in step S50, one token t _K is extracted from one word / token class T _i .

Next, as shown in step S51, a one-dimensional sequence of words in the text data is scanned, and step S51 is executed.
In 52, the token t extracted in step S50
_It is determined whether there is a word string belonging to the word class string connected by the class chain corresponding to _K. And
When the word string belonging to the word class string connected by the class chain corresponding to the token t _K exists in the one-dimensional string of the word of the text data, the process proceeds to step S53, and the process of regarding this word string as a collocation is repeated. replacing the token t _K in these collocation obtained by scanning a one-dimensional string of words in the text data.

On the other hand, if the word string belonging to the word class string connected by the class chain corresponding to the token t _K does not exist in the one-dimensional string of the word of the text data, step S
Proceeding to 54, it is determined whether or not processing has been completed for all tokens. If processing has not been completed for all tokens, processing has proceeded to step S50, and the above processing is repeated.

For example, the token replacement processing in step S3
, A one-dimensional string (w₁w_Two
w_Threew_Four... w_T), The word string (w₁w_Two),
(W₁₃w₁₄), ... are tokens t₁Replaced by the word
Column (w_Fourw_Fivew₆), (W₁₇w ₁₈), ... are tokens
t_TwoIs replaced by the token t₁Corresponding to
As a word,₁-W_Two, W₁₃-W₁₄...
Extracted from the data_TwoCollocations corresponding to
As ｛w_Four-W_Five-W₆, W₁₇-W₁₈,···}But
Extracted from text data.

One word / token class T _i is a set of words W _i and a set of tokens J _i = {t _i1 , t _i2,.
t _in }, and the token class T _i is {W _i {J _i }
And one token t _im in the set of tokens J _i is represented by a set of collocations V _im = ｛v _im ⁽¹⁾ , v _im ⁽²⁾ ,
... If the inverse token replacement is made to {}, one word / syllable class R _i becomes

[0106]

(Equation 2)

Is given by As described above, according to the word / phrase classification processing device according to one embodiment of the present invention,
Words and collocations can be classified without distinction.

Next, a speech recognition apparatus according to an embodiment of the present invention will be described. FIG. 23 is a block diagram showing the configuration of a speech recognition device that performs speech recognition using the word / phrase classification processing result obtained by the word / phrase classification processing device of FIG.

In FIG. 23, predetermined text data 4
0, the word and the collocation classification processing unit 4
1, the words and collocations are classified into classes in which words and collocations coexist, and the classified words and collocations are stored in the word / composition dictionary 49.

On the other hand, a pronunciation sound composed of a plurality of words and collocations is converted into an analog sound signal by a microphone 50, converted into a digital sound signal by an A / D converter 51, and input to a feature extraction unit 52. You. Feature extraction unit 52
Performs, for example, LPC analysis on a digital audio signal and extracts characteristic parameters such as cepstrum coefficients and logarithmic power. The feature parameters extracted by the feature extraction unit 52 are output to the speech recognition unit 54, refer to a language model 55 such as a phoneme hidden Markov model, and classify words and collocations stored in the word and collocation dictionary 49. While referring to the result, speech recognition is performed for each word and collocation.

FIG. 24 is a diagram showing an example in which speech recognition is performed using the result of the word / phrase classification processing. In FIG. 24, a pronunciation voice uttered “Today is fine weather” is input to the microphone 50, and a speech model is applied to this pronunciation voice, for example, the recognition result “Today is fine weather” and “Today is fine weather” Is a static electricity ". The recognition results of these speech models are processed by a language model to obtain a word / syllable dictionary 4
9 is performed, and if the collocation word “sunny weather” is registered in the word / syllable dictionary 49, a high probability is given to the recognition result “today is clear weather”, and Is given a low probability.

As described above, according to the speech recognition apparatus according to one embodiment of the present invention, by performing speech recognition with reference to the word / syllable dictionary 49, more accurate recognition processing becomes possible.

Next, a machine translation apparatus according to one embodiment of the present invention will be described. FIG. 25 is a block diagram showing a configuration of a machine translation apparatus that performs machine translation using the word / phrase classification processing result obtained by the word / phrase classification processing apparatus of FIG.

In FIG. 25, predetermined text data 4
0, the word and the collocation classification processing unit 4
1, the words and collocations are classified into classes in which words and collocations coexist, and the classified words and collocations are stored in the word / composition dictionary 49. The example original sentence and the example translated sentence corresponding to the example original sentence are stored in the example sentence collection 60 in association with each other.

When the original sentence is input to the example search section 61, the class to which the word of the input original sentence belongs is searched with reference to the word / syllable dictionary 49, and is composed of the words or the collocations belonging to the same class as the class. The example sentence to be used is searched from the example sentence collection 60. The example original sentence retrieved from the example sentence collection 60 and its example translation are input to the example application unit 62,
A translation for the input original text is generated by replacing the translation in the example translation text with a translation for the word of the input original text.

FIG. 26 is a diagram showing an example in which speech recognition is performed using the result of the word / phrase classification processing. In FIG. 26, “Toyota” and “Kohlberg Kr”
avis Robert & Co. "Belong to the same class," gained "and" lost "belong to the same class," 2 "and" 1 "belong to the same class," 30 1/4 "and" 80 1 / "2" belongs to the same class.

As an original, “Toyota gain”
d 2 to 30 ４. Is input from the example sentence collection 60 as "Kohlberg K" as an example original sentence.
ravis Robert & Co. lost 1
to 80 1/2. Is searched, and the example translation sentence “Kohlberg Kra” for the example original sentence is searched.
vis Robert & Co. The company has dropped $ 1 to close $ 1/20. Is also searched.

Next, the original word “Kohlberg” in the original example text is used.
Kravis Robert & Co. "Toyota" for the original word "Toyota" of the input source text belonging to the same class as "" and the translation "Kohlbe" for the example translation
rg Kravis Robert & Co. Is replaced with the translation of the input source text "gained" of the input source text "gained" that belongs to the same class as the source text "lost" of the example source text, and the translation "down" of the example source translation. By replacing “1” with “2” and replacing the numerical value “80 1/2” of the example translation with “30 1/4”, the translated sentence “Toyota
The dollar rose to a closing price of $ 301/2. Is output.

As described above, according to the machine translation apparatus of one embodiment of the present invention, by performing machine translation with reference to the word / syllable dictionary 49, more accurate translation processing becomes possible.

As described above, one embodiment of the present invention has been described. However, the present invention is not limited to the above-described embodiment, and various other modifications are possible within the technical idea of the present invention. . For example, in the above-described embodiment, a case has been described in which the word / phrase classification processing device is applied to a speech recognition device and a machine translation device. However, the word / phrase classification processing device may be used for a character recognition device. Further, in the above-described embodiment, the case of classifying a mixture of words and collocations has been described. However, only collocations may be extracted and the extracted collocations may be classified.

[0121]

As described above, according to the word / phrase classification processing device of the present invention, words and collocations included in text data are classified together to generate a class in which words and collocations are mixed. By doing so, it is possible not only to classify words and words collectively, but also to classify words and collocations or collocations and collocations, and to easily correlate words and collocations or collocations and collocations and to measure similarities. Can be determined.

Further, according to one aspect of the present invention, a token is assigned to the word class string of the text data, the word class string is regarded as one word, and the word included in the text data and the word class assigned with the token are added. Since the strings are treated equally and classified, and then the corresponding word class strings are replaced with the word strings existing in the text data, the classification process can be performed without distinction between words and collocations. Extraction of collocations from text data can be performed at high speed.

Further, according to the collocation extraction device of the present invention, each word constituting the word string of the text data is replaced with the word class to which the word belongs, and the word whose probability of occurrence in the text data is not less than a predetermined value is obtained. By extracting the collocations existing in the text data after extracting the class sequence, the collocations can be extracted at high speed.

Further, according to the speech recognition apparatus of the present invention, speech recognition can be performed using the correspondence and similarity between words and collocations or collocations and collocations, and accurate processing becomes possible.

According to the machine translation apparatus of the present invention, even when an original sentence in which words of the example original sentence stored in the example sentence collection are replaced with collocations is input, the example original sentence is applied to the input original sentence. Machine translation, and accurate machine translation using the correspondence and similarity between words and collocations or collocations and collocations becomes possible.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a functional configuration of a word / phrase classification processing device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a word clustering process of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 3 is a block diagram illustrating a functional configuration of a word classification unit in FIG. 1;

FIG. 4 is a diagram illustrating a word class sequence generation process of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 5 is a diagram illustrating a class chain extraction process of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 6 is a block diagram illustrating a functional configuration of a word class string extracting unit in FIG. 1;

FIG. 7 is a diagram showing a relationship between a class chain and a token by the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 8 is a diagram illustrating token replacement processing of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 9 is a diagram showing an example of an English sentence in a token replacement process by the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 10 is a block diagram showing a functional configuration of the word / token classification means of FIG. 1;

FIG. 11 is a diagram showing the relationship between tokens and collocations by the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 12 is a diagram showing a result of a word / phrase classification process performed by the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 13 is a block diagram illustrating a system configuration of a word / phrase classification processing device according to an embodiment of the present invention.

FIG. 14 is a flowchart showing a word / phrase classification process of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 15 is a diagram illustrating window processing of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 16 is a flowchart showing a word clustering process of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 17 is a flowchart showing a first embodiment of a class chain extraction process of the word / phrase classification processing device according to the present invention.

FIG. 18 is a flowchart showing a second embodiment of the class chain extraction processing of the word / phrase classification processing device according to the present invention.

FIG. 19 is a diagram illustrating a second embodiment of the class chain extraction processing of the word / phrase classification processing device according to the present invention.

FIG. 20 is a flowchart showing token replacement processing of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 21 is a flowchart showing word / token clustering processing of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 22 is a flowchart showing data output processing of the word / phrase classification processing device according to one embodiment of the present invention.

FIG. 23 is a block diagram showing a functional configuration of a speech recognition device according to one embodiment of the present invention.

FIG. 24 is a diagram illustrating a voice recognition method according to an embodiment of the present invention.

FIG. 25 is a block diagram illustrating a functional configuration of a machine translation apparatus according to an embodiment of the present invention.

FIG. 26 is a diagram illustrating a machine translation method according to an embodiment of the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 word classification means 2 word class string generation means 3 word class string extraction means 4 token assignment means 5 word / token string generation means 6 word / token classification means 7 word replacement means 40 text data 41 word / word word classification processing units 42, 46 Memory interface 43 CPU 44 ROM 45 Work RAM 47 RAM 48 Bus 49 Word / phrase dictionary 50 Microphone 51 A / D converter 52 Feature extraction unit 53 Buffer memory 54 Voice recognition unit 55 Language model 60 Example sentence collection 61 Example search unit 62 Example application Department

Claims (17)

[Claims] 1. A method for extracting V words different from each other from text data as a one-dimensional string of a plurality of words,
Generating a first clustering obtained by dividing a set of words into C word classes; and in the one-dimensional sequence of word classes of the text data generated based on the first clustering, adjacent words A step of extracting a set of word class strings in which the degree of adhesion between the classes is all equal to or greater than a predetermined value, and causing a unique token to correspond to the word class string, and searching a word string belonging to the word class string from the text data Generating a one-dimensional sequence of words and tokens for the text data by replacing the word sequence of the text data with corresponding tokens; anda one-dimensional sequence of words and tokens for the text data. A set in which different words and different tokens are extracted, and the words and the tokens are mixed. Generating a second clustering divided into words / token classes; extracting word strings corresponding to the tokens among word strings existing in the text data as collocations; and extracting tokens in the word / token classes. By replacing the word with the collocation, a set in which the word and the collocation are mixed is defined as a word / word.
Generating a third clustering divided into collocation classes. 2. The word / phrase classification method according to claim 1, wherein the first clustering is generated based on an average mutual information amount of the word classes. 3. The method according to claim 1, wherein the second clustering is generated based on an average mutual information of the word and token classes. Generating a word class in which words included in the text data are classified; and generating a one-dimensional string of the word class by mapping the word class to a one-dimensional string of the words in the text data. In a one-dimensional sequence of word classes of the text data,
Extracting, from the one-dimensional sequence of the word classes of the text data, a word class sequence in which the degree of adhesion between adjacent word classes is all equal to or greater than a predetermined value; and combining the words included in the text data with the word class sequence. And extracting individual words adjacent to the text data from the individual word classes constituting the word class sequence to extract collocations; and extracting the word class sequence into the words Replacing with a collocation belonging to a class sequence. 5. A step of generating a word class in which words included in text data are classified; and a step of mapping the word class to a one-dimensional string of words in the text data to generate a one-dimensional string of word classes. In a one-dimensional sequence of word classes of the text data,
Extracting, from a one-dimensional sequence of the word classes of the text data, a word class sequence in which the degree of adhesion between adjacent word classes is all equal to or greater than a predetermined value; Separately extracting individual words existing adjacent to the data to extract a collocation word. 6. A word classifying means for extracting words different from each other from a word string of text data, generating a word class by dividing the set of extracted words, and forming a one-dimensional string of words of the text data A word class sequence generating means for generating a one-dimensional sequence of the word class of the text data by replacing each word to be performed with the word class to which the word belongs; ,
A word class sequence extracting unit that extracts a word class sequence in which the degree of adhesion between adjacent word classes is all equal to or greater than a predetermined value from a one-dimensional sequence of the word classes of the text data; A token assigning means for assigning a token to a word class string, and replacing the word string belonging to the word class string extracted by the word class string extracting means with the token in the one-dimensional string of words of the text data. A word / token sequence generating means for generating a one-dimensional sequence of words / tokens of the text data; and dividing a set in which words and tokens contained in the one-dimensional sequence of words / tokens of the text data are mixed to form a word・
Word / token classifying means for generating a token class; and collocation replacing means for generating a collocation by inversely replacing the token in the word / token class with the word string replaced by the word / token string generating means. A word / phrase classification processing device, comprising: 7. An initialization class setting unit that extracts words different from each other from a one-dimensional string of words of the text data, and assigns a unique word class to each word having a predetermined appearance frequency. A temporary merging unit that extracts two word classes from a set of word classes and temporarily merges them; an average mutual information calculating unit that calculates an average mutual information amount of the temporarily merged word classes of the text data; 7. The word / phrase classification processing device according to claim 6, further comprising: a main merge unit that performs a main merge of two word classes having the largest average mutual information amount among a set of classes. 8. A word class extracting section, comprising: a word class extracting section for sequentially extracting two adjacent word classes from a one-dimensional string of the word class of the text data; and a word class extracting section. A mutual information calculating unit that calculates a mutual information amount of the two extracted word classes; and a class chain connecting unit that connects two word classes having the mutual information amount equal to or more than a predetermined threshold value with a class chain. 7. The word / phrase classification processing device according to claim 6, wherein: 9. The word / token classification means extracts a different word and a different token from a one-dimensional string of words / tokens of the text data, and extracts each of the words and tokens having a predetermined appearance frequency. An initialization class setting unit for allocating a unique word / token class; a temporary merging unit for taking out two words / token classes from a set of word / token classes; and temporarily merging the words / token classes; An average mutual information calculation unit for calculating an average mutual information amount for a class; and a main merge unit for performing a main merge of two word / token classes having the maximum average mutual information amount among the set of the word / token classes. 7. The word / phrase classification processing device according to claim 6, comprising: 10. A collocation extracting means for extracting a collocation from text data, and a word / composition classification means for classifying together words and collocations included in the text data to generate a class in which words and collocations are mixed. A word / phrase classification processing device comprising: 11. The method according to claim 1, wherein the class is generated based on an average mutual information amount of the class.
0, a word / syllable classification processing device. 12. A word classifying means for classifying a word included in text data to generate a word class, wherein each word constituting a one-dimensional sequence of words of the text data is represented by the word class to which the word belongs. A word class string generating means for generating a one-dimensional string of the word class of the text data by substituting;
Word class string extracting means for extracting, from a one-dimensional string of the word classes of the text data, a word class string in which the degree of adhesion between adjacent word classes is all equal to or greater than a predetermined value; individual word classes constituting the word class string A collocation extracting means for separately extracting each word existing adjacent to the text data and extracting a collocation from the text data. 13. The collocation extraction apparatus according to claim 12, wherein the word class is generated based on an average mutual information amount of the word class. 14. A word / concatenation dictionary storing words and collocations included in predetermined text data in a class in which words and concatenations are mixed, and the word / concatenation dictionary and a predetermined hidden Markov dictionary. And a voice recognition unit for recognizing the pronunciation voice by referring to the model. 15. A word / syllable dictionary that stores words and collocations included in predetermined text data in a class in which words and collocations are mixed, an example original sentence, and an example translation sentence for the example original sentence. An example sentence collection in which an example original sentence composed of words or collocations belonging to the same class as the class to which the input original word belongs is searched from the example sentence collection; and A machine translation apparatus comprising: an example application unit configured to generate a translation of the input original text by replacing a translation in an example translation of the original text with a translation of a word of the input original text. 16. A word / syllable storage medium that stores words and collocations included in predetermined text data in a class in which words and collocations are mixed, wherein the class is a class of the class. A word / syllable storage medium characterized by being generated based on an average mutual information amount. 17. A function of extracting different words from a one-dimensional string of words in text data, dividing the set of extracted words to generate a word class, and A function of generating a one-dimensional column of the word class of the text data by replacing each constituent word with the word class to which the word belongs; A function of extracting a word class string in which the degree of adhesion between word classes is all equal to or greater than a predetermined value; a function of assigning a token to the word class string; and a one-dimensional string of words of the text data, A function of generating a one-dimensional string of words / tokens of the text data by replacing the word string to which the text data belongs with the token; - and the words and token that is included in the one-dimensional string of tokens by dividing the set of mixed word -
A computer-readable program storing a program for causing a computer to execute a function of generating a token class and a function of generating a collocation by reversely replacing a token in the word / token class with a word string existing in the text data Storage media.