JPH07129589A - Morpheme analyzer - Google Patents

Abstract

PURPOSE:To obtain partial morpheme linking likelihood through the use of a statistic morpheme linking frequency and to effectively execute morpheme analytic processing based on the value. CONSTITUTION:A morpheme analytic processing module 1 repeatedly calls a morpheme dictionary consulting module 2 and a morpheme linking likelihood calculation module 4 to an inputted natural language sentence so as to successively generate the sequence of proper morphemes. The morpheme dictionary consulting module 2 reads a morpheme to be a candidate from a morpheme dictionary 3. The morpheme linking likelihood calculation module 4 calculates morpheme linking likelihood through the use of morpheme linking frequency data 5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a morphological analyzer, which is used in the field of information processing, and particularly, to processing natural language such as kana-kanji conversion, machine translation, information retrieval, etc. by using the morpheme connection frequency. The present invention relates to a morphological analysis device for performing.

[0002]

2. Description of the Related Art In a conventional morpheme analysis device, a morpheme candidate obtained from a morpheme dictionary is subjected to a concatenation check by a concatenation table indicating whether or not a morpheme can be concatenated, or by a grammar. The number of morpheme candidates was narrowed down. Furthermore, regarding the control of the processing flow and the determination of the output result, heuristics such as the longest matching method that prioritizes the appearance frequency of a single morpheme or the morpheme having a longer appearance form have been used.

[0003]

However, in the conventional method, since the analysis is performed using the concatenation table or the grammar, the information obtained therefrom is only whether or not the morphemes can be concatenated. The number of morpheme candidates cannot be sufficiently narrowed down just by passing the check, which causes an explosion in the number of candidates. This leaves a lot of ambiguity, which causes a decrease in processing speed or a decrease in reliability of the processing result.

Since only the appearance frequency of morphemes is used, information about the connection with the preceding and following morphemes cannot be obtained, and an erroneous result is likely to occur.

The priority of the branches that occur in the course of processing,
Or one must resort to general heuristics that ignore the individuality of word concatenation, such as the longest match method to give the likelihood of the final output candidate. This method is difficult to tune to improve system performance. As described above, the conventional method has many problems.

Therefore, the main object of the present invention is to obtain a partial morpheme connection likelihood using the morpheme connection frequency,
It is an object of the present invention to provide a morphological analysis device that can effectively perform morphological analysis processing based on the value.

[0007]

The invention according to claim 1 is
A morphological analyzer that divides an input natural language sentence for each morpheme and outputs the result, and selects an appropriate morpheme using a morpheme dictionary and concatenation frequency data.

In the invention of claim 2, the morpheme dictionary of claim 1 is used to subtract other morpheme information of the morpheme from the appearance form of the morpheme.

In the invention according to claim 3, in the concatenation frequency data of claim 1, attention is paid to a part or all of the morpheme information, and the frequency of the morphemes concatenated with each other is calculated in advance.

In the invention according to claim 4, the partial morpheme connection likelihood is calculated using the connection frequency for a plurality of morpheme candidates obtained by the morpheme dictionary of claim 2.

The invention according to claim 5 has determined in claim 4 a problem which is always present in the process, such as determining the priority order when the process branches and preventing the explosion of the number of morpheme string candidates. Utilizing partial morpheme concatenated likelihood to improve the efficiency and optimization of processing.

In the invention according to claim 6, in order to select the maximum likelihood one from the finally obtained morpheme sequence candidates, the partial morpheme connection likelihoods obtained in claim 4 are accumulated over the entire sentence. The value is the likelihood of the output result.

[0013]

The morpheme analyzer according to the present invention obtains not only the possibility of concatenation but also the concatenation likelihood by using the morpheme concatenation frequency, and it is possible to narrow down the morpheme candidates without using heuristics. On the other hand, it is possible to give a likelihood as a measure of reliability.

[0014]

1 is a schematic block diagram of an embodiment in which the present invention is applied to a Japanese sentence. In FIG. 1, an input sentence is given to a morpheme analysis processing module 1. This morpheme analysis processing module 1 reads a morpheme from a morpheme dictionary 3 by a morpheme dictionary drawing module 2 and a morpheme connection frequency calculation module 4 by a morpheme connection likelihood calculation module 4. The process is executed while reading the data 5.

FIG. 2 is a diagram showing an example of the morpheme dictionary shown in FIG. In FIG. 2, the morpheme dictionary is a dictionary for subtracting other morpheme information (standard form, part of speech, inflectional form, inflectional form, etc.) of the morpheme from the appearance form of the morpheme.

FIG. 3 is a diagram showing an example of morpheme connection frequency data. The morpheme concatenation frequency data is the appearance frequency of a set of n contiguous morphemes (referred to as n-gram frequency, where the concatenation number n is an integer of 1 or more). When n = 1, it is equivalent to the appearance frequency of each morpheme.

In this embodiment, as the morpheme connection frequency,
When n = 1 (hereinafter referred to as the word monogram frequency) and n = 2 (hereinafter referred to as the word bigram frequency) with respect to the connection frequency in the case of connecting all morpheme information, the part-of-speech, inflectional form, and utilization A total of four types of concatenation, considering n = 1 (hereinafter referred to as part-of-speech monogram frequency) and n = 2 (hereinafter referred to as part-of-speech bigram frequency) with respect to the concatenation frequency when three types of items are connected. Frequency is assumed.

The processing procedure of the morphological analysis processing apparatus using the connection frequency will be described in detail below.

(0) First, a sentence head morpheme is set in a dummy of a morpheme string candidate that appears in the next (1), an input sentence is also set in an unprocessed partial natural language sentence, and a cumulative morpheme connection likelihood Yp = The process of (1) starts from the state of being initialized to 1. Here, the dummy sentence head morpheme is a virtual morpheme that does not appear in the output result, but is used only for the concatenation likelihood calculation between the sentence head and the first morpheme.

(1) A morpheme candidate that matches the leading partial character string of the unprocessed partial natural language sentence following the morpheme string candidate is subtracted from the morpheme dictionary. If no morpheme can be looked up in the dictionary, the morpheme sequence candidate is considered to have failed, and the morpheme sequence candidate is removed.

(2) The last morpheme c1 of the morpheme string candidate
, And the bigram frequency between c1 and each of the plurality of morpheme candidates generally obtained in (1), the morpheme concatenated likelihood shown in the following equations (1) to (3). Find Yc.

Morphological concatenated likelihood Yc = (C · Wc + Pc) / (C + 1) ... (1) Word concatenated likelihood Wc = Wb (c1, c2) / Wm (c1) ... (2) Part-of-speech concatenated likelihood Pc = Pb (C1, c2) / Pm (c1) ... (3) where C: weighting coefficient of word concatenation likelihood to part-of-speech concatenation likelihood Wm (c1): word monogram frequency of morpheme c1 Wb (c1, c2): morpheme c1 , C2 word bigram frequency Pm (c1): part-of-speech monogram frequency of morpheme c1 Pb (c1, c2): part-of-speech bigram frequency between morphemes c1 and c2 (3) If Yc is 0, it is regarded as a failure because it is not connected Remove morpheme string candidates. When Yc> 0, the morpheme is added to the morpheme string candidate as concatenable and the morpheme is removed from the unprocessed partial natural language sentence. Further, the morpheme connected likelihood Yc is accumulated in the accumulated morpheme connected likelihood Yp.

(4) The morpheme string candidates are prioritized according to the cumulative morpheme concatenated likelihood Yp, and when the number of candidates is too large, the ones with the lowest priorities are deleted to obtain an appropriate number.

(5) For each of the plurality of morpheme sequence candidates generally remaining in (4), (1) to (4) are applied to the unprocessed partial natural language sentence until the following end condition is satisfied. Repeat.

<End condition 1> For all morpheme string candidates, there is no unprocessed partial natural language sentence, and the process is completely completed.

<Termination condition 2> There is no unprocessed partial natural language sentence for a preset number of morpheme string candidates, and the process is partially completed.

<End condition 3> When all the morpheme string candidates fail, in other words, the next morpheme cannot be looked up in the dictionary, or Yc becomes 0, and there is no morpheme string candidate necessary for proceeding with the analysis. If

(6) When the termination condition is satisfied, the value of the finally obtained cumulative morpheme concatenated likelihood Yp is set as the likelihood.
The unprocessed partial natural language sentence is output as a morpheme string candidate with likelihood likelihood morpheme string candidate.

FIG. 4 is a flow chart for explaining the specific operation of the embodiment of the present invention. In FIG. 4, the basic processing flow of morphological analysis using the concatenation frequency is shown by taking the following (input sentence 1) as an example.

“This is the secretariat” (input sentence 1) First, it is assumed that there is one morpheme registered for each of “here” and “wa”. At this time, the morpheme string candidate, the unprocessed partial natural language sentence (hereinafter, referred to as an unprocessed part) and the cumulative morpheme connected likelihood are as follows, for example.

Cumulative morpheme connected likelihood Yp morpheme sequence candidate unprocessed part (1) 0.60 (here) “It is the secretariat” (1) By morpheme dictionary lookup for the unprocessed part, the following multiple morpheme candidates are obtained. Suppose

Things (0.00) Office work (0.50) Secretariat (0.80) However, the numerical value in the parentheses is the morpheme connection likelihood Yc with the immediately preceding morpheme "wa", and 0.00 means not connected. To do.

Then, the unprocessed part of the morpheme string candidate is
Each is divided into multiple combinations as follows. In addition, if the cumulative morpheme connection likelihood becomes 0.00, it is regarded as a failure and is excluded from the subsequent processing.

Cumulative morpheme connected likelihood Yp Unprocessed part of morpheme sequence candidate (2a) 0.60 × 0.00 = 0.00 (this is the case) “It is a bureau” → Failed (2b) 0.60 × 0.50 = 0.30 (this is clerical) “Station (2c) 0.60 × 0.80 = 0.48 (this is the secretariat) “Yes” where (2a) fails with a cumulative morpheme concatenated likelihood of 0.00.

Further, it is assumed that the following plural morpheme candidates are obtained by the morpheme dictionary lookup for the unprocessed portions of (2b) and (2c). For (2b), it is (0.50) for (2c) and (0.20) for (0.80) However, the value in parentheses is the morpheme concatenated likelihood Yc with the previous morpheme. Then, the morpheme string candidate and the unprocessed part are as follows.

Cumulative morpheme connected likelihood Yp Unprocessed part of morpheme sequence candidate (3b) 0.30 × 0.50 = 0.15 (this is the secretariat) “is” (3c) 0.48 × 0.20 = 0.09 (this is the secretariat) “su” (3d) 0.48 × 0.80 = 0.33 (This is the secretariat)… → End Repeat the same process until the end condition is satisfied.

As can be seen from this example, generally, in the morphological analysis process, the number of result candidates tends to increase as the process progresses. However, in the method using the concatenation frequency, the cumulative morpheme concatenated likelihood is obtained at each time point of the processing, so that a candidate with a low possibility can be deleted if necessary.

[0038]

As described above, according to the present invention, by using the morpheme concatenation frequency information for the morpheme analysis processing, it is possible to improve the accuracy of the processing result.
Generally, there is a trade-off relationship between the improvement of the processing speed and the improvement of the reliability of the processing result. Against this, by adjusting the number of morpheme string candidates in the middle of the processing, it is possible to adjust as necessary. It is possible to achieve an appropriate balance.

[Brief description of drawings]

FIG. 1 is a schematic block diagram of an embodiment of the present invention.

FIG. 2 is a diagram showing a specific example of a morpheme dictionary.

FIG. 3 is a diagram showing a specific example of morpheme connection frequency data.

FIG. 4 is a flowchart showing a basic processing flow of morphological analysis using a concatenation frequency.

[Explanation of symbols]

 1 morpheme analysis processing module 2 morpheme dictionary lookup module 3 morpheme dictionary 4 morpheme connection likelihood calculation module 5 morpheme connection frequency data

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Eiichiro Sumida, 2-chome, Kodai, Seika-cho, Soraku-gun, Kyoto Prefecture ATTR Voice Translation Research Laboratory (72) Inventor, Kura Furuse Soraku Prefecture 2-2 Kodai, Seika-cho, Gunma Incorporated, E.T.R. Co., Ltd. Voice Translation Research Laboratory (72) Inventor, Shinichi Murakami 2-2, Kodai, Seika-cho, Soraku-gun, Kyoto E.T. R Speech Translation and Communication Research Laboratory

Claims (6)

[Claims] 1. A morphological analyzer that divides an input natural language sentence for each morpheme and outputs the result, wherein an appropriate morpheme is selected using a morpheme dictionary and concatenation frequency data. A morphological analyzer. 2. The morpheme analysis apparatus according to claim 1, wherein the morpheme dictionary is used to subtract other morpheme information of the morpheme from the appearance form of the morpheme. 3. The morpheme analysis apparatus according to claim 1, wherein the connection frequency data is obtained by paying attention to a part or all of the morpheme information and calculating in advance the frequency of morphemes connected to each other. 4. The morpheme analysis apparatus according to claim 1, wherein a partial morpheme connection likelihood is calculated using the connection frequency for a plurality of morpheme candidates obtained by the morpheme dictionary. 5. The partial morpheme concatenation likelihood is used for a task that is always present in a processing process, such as determining a priority order when processing branches and preventing explosion of the number of morpheme string candidates. The morphological analysis device according to claim 4, wherein the efficiency and the optimization of the processing are achieved. 6. A value obtained by accumulating the partial morpheme concatenated likelihood over the entire sentence is used as the likelihood of the output result in order to select the maximum likelihood morpheme sequence candidate finally obtained. The morphological analyzer according to claim 4.