JPH02122366A - natural language processing device - Google Patents

Abstract

(57) [Abstract] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

[Detailed description of the invention] Technical field The present invention relates to a natural language processing device, and more particularly to a natural language processing device.

The present invention relates to natural language processing devices such as a natural language morphological analysis device, a kana-kanji conversion device 4, a kanji-kana conversion device, etc. in machine translation 1 proofreading support, etc.

BACKGROUND ART Conventionally, as a longest clause matching method, there is a method that outputs a clause candidate whose reading length is the longest among two clauses.

In addition, the Kana-Kanji conversion method (Japanese Unexamined Patent Publication No. 59-221733) that uses reading length, connection probability, and frequency calculates the evaluation value of a word using an evaluation formula using three parameters, and the integral value of the evaluation value is It outputs the maximum candidate word string.

In this way, in the conventional method, in order to select the correct candidate, the extracted candidates are evaluated, and output candidates are determined based on the evaluation results.

But because natural language has ambiguities.

The optimal candidate cannot be accurately determined by a single evaluation. In order to perform natural language processing more appropriately, it is necessary to select plausible candidates using multiple evaluations (evidence).

In addition, a natural language processing device has been proposed that selects more accurate candidates to output by providing multiple pieces of evidence regarding the likelihood of a candidate and combining them to calculate accuracy and likelihood. There is.

This method provides evidence of the likelihood of a candidate in the form of basic probabilities and performs 1-synthesis, and expresses ambiguous evidence using an empirical numerical value called likelihood, making it easy to combine evidence obtained by different types of evaluations. This is one of the advantages of this method. However, conversely, when designing a natural language processing device using this method using the assets of the conventional method, there are the following problems. In the conventional method, the candidate with the highest candidate evaluation value is output. Therefore, it is not necessary that the evaluation value accurately represents the likelihood of the candidate, and it is sufficient that the evaluation value ranking for the extracted candidate matches the likelihood ranking of the candidate.

Therefore, when calculating the basic probability from the evaluation value instead of giving the likelihood of a candidate directly in the form of a pseudo-likelihood distribution or basic probability, the basic probability may not be given appropriately.

Purpose The present invention was made in view of the above-mentioned circumstances.
In order to achieve the above object, the present invention provides a natural language processing device with higher performance by converting the evaluation result indicated by the evaluation value into an appropriate basic probability and using it as evidence. a word dictionary that holds information; a candidate extractor that extracts candidates consisting of words or combinations of words based on input character strings; an evaluation unit that provides multiple pieces of evidence regarding the likelihood of the candidates; In a natural language processing device that has a synthesis calculation unit that synthesizes multiple pieces of evidence for likelihood, and that uses the synthesized evidence to determine candidates to output, the evaluation value is determined for one evaluation of likelihood. It is characterized by providing two types of evidence: one based on the evaluation and the other based on the ranking of the evaluation. Hereinafter, the present invention will be specifically explained based on examples.

FIG. 1 is a block diagram for explaining one embodiment of the natural language processing device of the present invention. A character string to be analyzed is inputted by an input device 1 and sent to a candidate extraction section 2. In FIG. The character string to be analyzed is a kana character string indicating the pronunciation in the case of a kana-kanji conversion device, and a written character string containing a mixture of kanji and kana in the case of a kanji-kana conversion device or a morphological analysis device for proofreading support. In addition, in the case of machine translation, the input string is the language to be analyzed.

The candidate extraction unit 2 searches the word dictionary 3 using the input character string and extracts candidates consisting of words that match the input character string.

The evaluation unit 4 evaluates the extracted candidates using connection information between words, etc., and provides multiple pieces of evidence regarding the likelihood of the candidates.

The present invention is characterized by a method of converting the evaluation value into a basic probability and providing it as evidence when the evaluation result is expressed as an evaluation value and the evaluation value does not directly represent the basic probability that indicates the likelihood. be. The details will be described later.

The synthesis calculation section 5 synthesizes the evidence given by the evaluation section, which is represented by the basic probability, using the combination rule in Dempster Schaefer's probability theory.

The output candidate determination unit 6 calculates the likelihood and accuracy of the candidates from the combined evidence and determines the candidates to be output. Information on candidates to be output is sent to an output device, and the candidates are displayed. When the present invention is applied to machine translation, etc., information on output candidates is sent to the next stage of processing.

Next, a method of providing evaluation values as evidence will be explained using a kana-kanji conversion device using the present invention as an example.

In this embodiment, candidates are extracted in units of clauses, and for evaluation, independent parts that are connected to the extracted candidates are further extracted. Then, an evaluation value calculated from the reading length of the free-standing part of the extracted candidate, ease of connection, and frequency,
Evaluation values similarly calculated for independent parts connected to the candidate are converted into basic probabilities and used as evidence of the likelihood of the candidate.

This evaluation value is given so that the more likely candidate has a larger value, but the likelihood increases exponentially with respect to the evaluation value. Furthermore, even if the evaluation values are the same, it is more likely that the evaluation value has a higher rank.

In order to use such evaluation values as evidence, as shown in FIG. The calculated basic probabilities are respectively provided to the synthesis calculation unit 5 as evidence.

FIG. 3 shows a flowchart for converting evaluation values into evidence expressed in basic probabilities.

First, from the evaluation value of each candidate, convert it to a pseudo-likelihood distribution by referring to a conversion table that describes the likelihood (possibility) corresponding to the evaluation value. Since the likelihood increases exponentially with respect to the value, by creating a conversion table in this way, appropriate evidence can be obtained.

The pseudo-likelihood distribution is further converted into basic probabilities using the difference in its values.

Conversion from evaluation rankings to basic probabilities is also performed after converting evaluation rankings to pseudo-likelihood distributions in the same manner. In this embodiment, the evaluation ranking conversion table reflects the likelihood of the evaluation method. Therefore, although the evaluation values are calculated in the same way, different evidence is given depending on whether the evaluation is for the independent part of the candidate or for the independent part connected to the candidate, even if the evaluation value is the same and the evaluation rank is the same. It turns out.

Note that in this example, the conversion processing speed is improved by converting the evaluation value into a pseudo-likelihood distribution using a conversion table.
There is also a method of calculating using a conversion formula.

Furthermore, a method of directly converting the evaluation value and rank into a basic probability without converting the evaluation value into a pseudo-likelihood distribution is also within the scope of the present invention.

The present invention can also be implemented in a case where the evaluation value calculation method or candidate extraction unit is different from this embodiment.

In addition, in the above example, the reading length of the words in the free-standing part, the word appearance frequency, and the ease of connection between words are calculated using different evaluation formulas, and the D-8 calculation is performed using the values as evidence. However, the D-5 calculation may be performed using the values of each item such as length of word pronunciation, word frequency, and ease of connection between words as evidence, and any set of each item may be calculated by After calculating with the evaluation formula,
The D-3 calculation may be performed using that value as evidence.

Of course, as for how to use the evidence, calculations are performed using two types of evidence: evidence based on the evaluation value and evidence based on the evaluation rank.

Effects As is clear from the above explanation, according to the present invention, the evaluation value obtained from one evaluation is given both the value and the ranking as evidence, so when combining it with other evidence, the evaluation value is This appropriately represents the likelihood of a value, and has the effect of enabling high-performance natural language processing.

Furthermore, since two pieces of evidence are given for each evaluation, the number of times the pieces of evidence are combined increases, which tends to increase the accuracy. Therefore, the distinction between suitable candidates and unsuitable candidates, which is indicated by likelihood and accuracy, is more clearly shown than in the past.

Furthermore, according to claim 2, even if the same evaluation factors are used, the likelihood and probability can be calculated more strictly and appropriate candidates can be presented.

[Brief explanation of the drawing]

Fig. 1 is a block diagram showing an embodiment of the natural language processing device of the present invention, Fig. 2 is a block diagram of the evaluation section of the kana-kanji conversion device of the embodiment of the present invention, and Fig. 3 shows the evaluation value. It is a flowchart for converting into evidence expressed by basic probability. DESCRIPTION OF SYMBOLS 1... Input device, 2... Candidate extraction part, 3... Word dictionary, 4... Evaluation part, 5... Synthesis calculation part, 6...
Output candidate determining unit, 7...output device. Diagram Diagram Likelihood, Accuracy

Claims (1)

[Claims] 1. A word dictionary that holds word information, a candidate extraction unit that extracts candidates consisting of words or word combinations based on input character strings, and a plurality of A natural language processing device that has an evaluation unit that provides evidence and a composition calculation unit that combines multiple pieces of evidence regarding the likelihood of a candidate, and uses the combined evidence to determine a candidate to output. For one evaluation,
Evidence based on evaluation value and evidence based on evaluation ranking.
A natural language processing device characterized by providing one. 2. The above evaluation value includes at least two factors: the length of the character string of the written (or pronunciation) word, the frequency of appearance of the word, and the ease of connection between words. 2. The natural language processing device according to claim 1, wherein the natural language processing device uses independent evaluation values.