JPH0895976A - Natural language analyzer - Google Patents

Abstract

PURPOSE: To utilize the repeated appearance of a character string and the appearance conditions of a character string following the repeatedly appearing character string for extracting a unregistered word. CONSTITUTION: An input part 1 transmits an inputted sentence to an unregistered word candidate extracting part 2 and a language analyzing part 5. The unregistered word candidate extracting part 2 extracts unregistered word candidates out of the document sent from the input part 1 and records them in an unregistered word candidate appearance information recording part 3 together with character strings to be connected before and after. An unregistered word candidate evaluating part 4 evaluates the respective unregistered word candidates by analyzing the information recorded in the unregistered word candidate appearance information recording part 3. While referring to the evaluated result of the unregistered word candidate appearance information evaluating part 4, the language analyzing part 5 performs language analysis for analyzing the snetence sent from the input part 1 and transmitting the result to an output part 6. The output part 6 outputs the analyzed result. Thus, natural language analysis is performed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a natural language analyzer widely used in the field of language processing.

[0002]

2. Description of the Related Art In recent years, the field of natural language processing has been rapidly developing with the practical use of machine translation and natural language interfaces. Along with this, the natural language sentences to be analyzed have become diverse.

However, natural language is constantly changing, and the words used therein are also constantly changing. Therefore, it is essential for the natural language analysis device to have the ability to properly handle words that have not been incorporated when the system was created, so-called unregistered words.

On the other hand, conventionally, an unregistered word is detected by a method such that the character of the part whose analysis has failed is an unregistered word or the character type block including the failed part is an unregistered word. Was assumed to be a noun, or was assumed to be the most frequently occurring part-of-speech that succeeded in connection verification. These methods can also be considered as heuristics that take advantage of the fact that many unregistered words are nouns composed of Kanji or Katakana, and have proved to be reasonably effective.

[0005]

However, the accuracy of detection by the above method is not always sufficient, and humans cannot recognize words whose character boundaries do not match word divisions or words of relatively low frequency of occurrence. It was difficult to detect even words that could be easily grasped. For example, "...
The salmon has gone sardines. ……. It cannot be used anymore if you eat sea bream. When the sentence "..." is found, a human being can understand the ra-gyu five-verb verb "ne-ni-ru-ru" without using advanced content such as meaning. However, it was difficult to detect this by the conventional technique.

An object of the present invention is to provide a natural language analysis apparatus capable of appropriately processing unregistered words in which such character types are mixed or unregistered words of a part of speech having a relatively low frequency of occurrence. is there.

[0007]

A natural language analysis apparatus according to the present invention includes an input section for receiving an input sentence, an output section for outputting an analysis result of the language analysis section, and an input sentence for receiving the input sentence from the input section. An unregistered word candidate extraction unit that extracts a character string that may be an unregistered word from a sentence, an unregistered word candidate extracted by the unregistered word candidate extraction unit, and a character string adjacent to the unregistered word candidate An unregistered word appearance information recording unit, an unregistered word candidate evaluation unit that evaluates an unregistered word candidate based on the information recorded in the unregistered word appearance information recording unit, and the unregistered word candidate. A language analysis unit for analyzing the input sentence received from the input unit with reference to the evaluation result of the evaluation unit.

The natural language analysis method of the present invention extracts an unregistered word candidate from an input sentence, records the unregistered word candidate and a character string adjacent to the unregistered word candidate, and records the unregistered word candidate. It is characterized in that the unregistered word candidate appearance information is acquired from the content, each unregistered word candidate is evaluated, the input sentence is analyzed using the unregistered word candidate evaluation, and the analysis result is output.

[0009]

Embodiments of the present invention will now be described with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of the invention of claim 1.

This natural language analysis apparatus has an input unit 1 for receiving an input sentence, a language analysis unit 5 for analyzing the input sentence received from the input unit 1, and an output unit for outputting an analysis result of the language analysis unit 5. 6, an unregistered word candidate extraction unit 2 that receives an input sentence from the input unit 1 and extracts a character string that may be an unregistered word from the sentence, and an unregistered word candidate extraction unit 2 extracts the unregistered word candidate extraction unit 2. An unregistered word appearance information recording unit 3 that records a registered word candidate and a character string adjacent to the unregistered word candidate,
An unregistered word candidate evaluation unit 4 that evaluates an unregistered word candidate based on the information recorded in the unregistered word candidate appearance information recording unit 3 is provided.

The text input to the input unit 1 is the data line 1
2 to the unregistered word candidate extraction unit 2 and the data line 15 to the language analysis unit 6. Unregistered word candidate extraction unit 2
Extracts an unregistered word candidate from the input sentence, and connects the unregistered word candidate and a character string adjacent to the unregistered word candidate to the data line 2
It records in the unregistered word candidate appearance information recording unit 3 through 3.
The unregistered word candidate evaluation unit 4 acquires unregistered word candidate appearance information from the unregistered word candidate appearance information recording unit 3 through the data line 34, and evaluates each unregistered word candidate. The language analysis unit 5
The input sentence sent from the input unit 1 is analyzed using the unregistered word candidate evaluation of the unregistered word candidate evaluation unit obtained through the data line 45, and the analysis result is sent to the output unit 6 through the data line 56. The output unit 6 outputs the analysis result.

Next, the operation principle will be described using a specific example.

First, it is assumed that the sentence shown in FIG. 4 (a) is input.

The unregistered word candidate extraction unit 2 extracts unregistered word candidates as shown in FIG. 4B from the sentence of FIG. 4A, and the unregistered word candidate appearance information recording unit 3 together with the adjacent character string. To record. FIG. ₅ shows a part of the recorded contents of the unregistered word candidate appearance information recording unit 3 at this time.

The unregistered word candidate evaluation unit 4 evaluates each unregistered word candidate based on the recorded contents of the unregistered word candidate appearance information recording unit 3. FIG. ₆ shows an example of the evaluation method, and FIG. ₇ shows an example of the evaluation result.

The language analysis unit 5 analyzes the sentence sent from the input unit 1 with reference to the evaluation of the unregistered word candidate evaluation unit 4. FIG. ₈ shows an example of the analysis result.

In the unregistered word candidate appearance information recording unit 3 of this embodiment, the portions up to the sentence boundary of the preceding and succeeding character strings are recorded. However, depending on the evaluation method of the unregistered word candidate evaluation unit 4, only the preceding or succeeding one may be sufficient. It is not the essence of the present invention that the length of the recorded character string is up to the sentence boundary, and a shorter or longer character string may be recorded.

Next, the invention of claim 2 will be described.

FIG. 2 is a block diagram showing an embodiment of the invention of claim 2.

This natural language analysis apparatus has an input unit 1 for receiving an input sentence, a language analysis unit 5 for analyzing the input sentence received from the input unit 1, and an output unit for outputting an analysis result of the language analysis unit 5. 6, an unregistered word candidate extraction unit 2 that receives an input sentence from the input unit 1 and extracts a character string that may be an unregistered word from the sentence, and an unregistered word candidate extraction unit 2 extracts the unregistered word candidate extraction unit 2. An unregistered word appearance information recording unit 3 that records a registered word candidate and a character string adjacent to the unregistered word candidate,
A character meaning dictionary 7 storing characters and meaning information corresponding to the characters, based on information recorded in the unregistered word candidate appearance information recording unit 3 and information stored in the character meaning dictionary 7. An unregistered word candidate evaluation unit 4 that evaluates unregistered word candidates.

The text input to the input unit 1 is the data line 1
2 to the unregistered word candidate extraction unit 2 and the data line 15 to the language analysis unit 6. Unregistered word candidate extraction unit 2
Extracts an unregistered word candidate from the input sentence, and connects the unregistered word candidate and a character string adjacent to the unregistered word candidate to the data line 2
It records in the unregistered word candidate appearance information recording unit 3 through 3.
The unregistered word candidate evaluation unit 4 receives the unregistered word candidate appearance information from the unregistered word candidate appearance information recording unit 3 through the data line 34 and the meaning of the constituent character string of each unregistered word candidate from the character meaning dictionary 7 through the data line 47. Obtain information and evaluate each unregistered word candidate. The language analysis unit 5 analyzes the input sentence sent from the input unit 1 by using the unregistered word candidate evaluation of the unregistered word candidate evaluation unit obtained through the data line 45, and outputs the analysis result through the data line 56. Send to Part 6. The output unit 6 outputs the analysis result.

A specific example will be described below.

First, it is assumed that the sentence shown in FIG. 9A is input.

The unregistered word candidate extraction unit 2 extracts unregistered word candidates as shown in FIG. 9B from the sentence of FIG. 9A, and the unregistered word candidate appearance information recording unit 3 together with the adjacent character string. To record. FIG. 10 shows a part of the recorded contents of the unregistered word candidate appearance information recording unit 3 at this time.

The unregistered word candidate evaluation unit 4 evaluates each unregistered word candidate based on the recorded contents of the unregistered word candidate appearance information recording unit 3 and the semantic information obtained from the character meaning dictionary 7.
FIG. 11A shows an example of the semantic information obtained from the character dictionary 7, an example of the evaluation method using the semantic information is shown in FIG. 11B, and an example of the evaluation result is shown in FIG.

The language analysis unit 5 analyzes the sentence sent from the input unit 1 with reference to the evaluation of the unregistered word candidate evaluation unit 4. FIG. 13 shows an example of the analysis result.

FIG. 3 is a block diagram showing an embodiment of the invention of claim 3.

This natural language analyzing apparatus refers to the input section 1 for receiving an input sentence, an updatable dictionary 8 storing information for language analysis, and the input sentence received from the input section 1 with reference to the dictionary 8. A language analysis unit 5 that analyzes and updates the contents of the dictionary 8 based on the analysis result; and the language analysis unit 5
An unregistered word candidate extraction unit 2 that receives an input sentence from the output unit 6 that outputs the analysis result of 1 and the input unit 1 and extracts a character string that may be an unregistered word from the sentence; The unregistered word appearance information recording unit 3 which records the unregistered word candidate extracted by the candidate extraction unit 2 and the character string adjacent to the unregistered word candidate, and the unregistered word candidate appearance information recording unit 3 are recorded. An unregistered word candidate evaluation unit 4 that evaluates unregistered word candidates based on the information and updates the contents of the dictionary 8 based on the evaluation result.

The text input to the input unit 1 is the data line 1
2 to the unregistered word candidate extraction unit 2 and the data line 15 to the language analysis unit 6. Unregistered word candidate extraction unit 2
Extracts an unregistered word candidate from the input sentence, and connects the unregistered word candidate and a character string adjacent to the unregistered word candidate to the data line 2
It records in the unregistered word candidate appearance information recording unit 3 through 3.
The unregistered word candidate evaluation unit 4 acquires the unregistered word candidate appearance information from the unregistered word candidate appearance information recording unit 3 through the data line 34, evaluates each unregistered word candidate, and based on the evaluation result, the data line 48. To update the contents of the dictionary 8. The language analysis unit 5 analyzes the input sent from the input unit 1 using the unregistered word candidate evaluation of the unregistered word candidate evaluation unit obtained through the data line 45 and the dictionary information obtained through the data line 58. , The contents of the dictionary 8 are updated through the data line 58 based on the analysis result, and the analysis result is sent to the output unit 6 through the data line 56. The output unit 6 outputs the analysis result.

A specific example will be described below.

First, it is assumed that the sentence shown in FIG. 14A is input.

The unregistered word candidate extraction unit 2 extracts unregistered word candidates as shown in FIG. 14B from the sentence of FIG. 14A, and the unregistered word candidate appearance information recording unit 3 together with the adjacent character string.
To record. Unregistered word candidate appearance information recording unit 3 at this time
FIG. 15 shows a part of the recorded contents of FIG.

The unregistered word candidate evaluation unit 4 evaluates each unregistered word candidate based on the recorded contents of the unregistered word candidate appearance information recording unit 3, and updates the contents of the dictionary 8 based on the evaluation result. To do. FIG. 16 shows an example of the evaluation result, and FIG. 17 shows an example of updating the contents of the dictionary 8 based on the evaluation result.

The language analysis unit 5 analyzes the sentence sent from the input unit 1 while referring to the evaluation of the unregistered word candidate evaluation unit 4, and updates the contents of the dictionary 8 based on the analysis result.

FIG. 18 shows an example of updating the contents of the dictionary 8 based on the analysis result.

As another specific example, the unregistered word candidate extraction unit 2
A case will be described in which extraction is performed using the extraction template.

It is assumed that the text shown in FIG. 19 is input. Further, it is assumed that the unregistered word candidate extraction unit 2 uses the extraction template shown in FIG. However, [] in the extraction template means a character string of an arbitrary length. The unregistered word candidate extraction unit 2 extracts unregistered word candidates by applying the extraction template to the input sentence. As a result, the unregistered word candidate appearance information recording unit 3 records the unregistered word candidates shown in FIG.

As another specific example, the unregistered word candidate evaluation unit 4
A case will be described where is evaluated using the appearance frequency.

The unregistered word candidate evaluation unit 4 searches the information recorded in the unregistered word candidate appearance information recording unit 3, and counts each surface word and the number of appearances of the surface word. FIG. 22 shows an example of the counting result. The unregistered word candidate evaluation unit 4 evaluates each surface layer based on the counting result. FIG. 23 shows an example of the evaluation result.

As another specific example, the unregistered word candidate evaluation unit 4
A case will be described in which the evaluation is performed using a sequence of candidate sounds.

FIG. 24 shows an example of an evaluation method used by the unregistered word candidate evaluation unit 4.

The unregistered word candidate evaluation unit 4 evaluates the unregistered word candidates recorded in the unregistered word candidate appearance information recording unit 3 by applying the evaluation method. An example of the evaluation result is shown in FIG.

As another example, FIG. 26 shows an example having a plurality of evaluation sections.

An unregistered word candidate evaluation unit α and an unregistered word candidate evaluation unit β that evaluate unregistered word candidates by two different methods.
There is a comprehensive evaluation section that combines the evaluations of both evaluation sections. At this time, the entire part surrounded by the dotted line can be regarded as one unregistered word candidate evaluation part, and can be said to be the same invention as claim 1.

Although an example having two evaluation sections is shown here, the case of having three or more evaluation sections can be considered in the same manner.

In the present embodiment, the explanation, numerical values, symbols,
, Etc. are used, but these expressions are not the essence of the present invention, and may be appropriately replaced with equivalent explanations, numerical values, symbols, and the like.

[0048]

As described above, according to the present invention, unregistered word candidates are evaluated based on the number of appearances of the corresponding candidate and the adjacent character string information. It becomes possible to extract an unregistered word, an unregistered word that is a part of speech with a low occurrence frequency, and the like with high accuracy.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of the invention of claim 1;

FIG. 2 is a block diagram showing an embodiment of the invention of claim 2;

FIG. 3 is a block diagram showing an embodiment of the invention of claim 3;

FIG. 4 is a block diagram showing an example of input sentence and extraction of unregistered word candidates.

FIG. 5 is an explanatory diagram showing an example of contents of an unregistered word candidate appearance information recording unit.

FIG. 6 is an explanatory diagram showing an example of an evaluation method of an unregistered word candidate evaluation unit.

FIG. 7 is an explanatory diagram showing an example of an unregistered word candidate candidate evaluation result.

FIG. 8 is an explanatory diagram showing an example of an analysis result of an input sentence.

FIG. 9 is an explanatory diagram showing an example of an input sentence and extraction of unregistered word candidates.

FIG. 10 is an explanatory diagram showing an example of the contents of an unregistered word candidate appearance information recording unit.

FIG. 11 is an explanatory diagram showing an example of an evaluation method using the contents of the character meaning dictionary and the contents of the character meaning dictionary.

FIG. 12 is an explanatory diagram showing an example of an evaluation result using the contents of a character meaning dictionary.

FIG. 13 is an explanatory diagram showing an example of an analysis result of an input sentence.

FIG. 14 is an explanatory diagram showing an example of input sentence and extraction of unregistered word candidates.

FIG. 15 is an explanatory diagram showing an example of contents of an unregistered word candidate appearance information recording unit.

FIG. 16 is an explanatory diagram showing an example of an unregistered word candidate evaluation result.

FIG. 17 is an explanatory diagram showing an example of updating the contents of a dictionary using an unregistered word candidate evaluation result.

FIG. 18 is an explanatory diagram showing an example of dictionary content update based on an analysis result of a language analysis unit.

FIG. 19 is an explanatory diagram showing an example of an input sentence.

FIG. 20 is an explanatory diagram showing an example of an extraction template.

FIG. 21 is an explanatory diagram showing an example of unregistered word candidates extracted by the extraction template.

FIG. 22 is an explanatory diagram showing an example of frequency information.

FIG. 23 is an explanatory diagram showing an example of an evaluation result using frequency information.

FIG. 24 is an explanatory diagram showing an example of an evaluation method using a sequence of sounds.

FIG. 25 is an explanatory diagram showing an example of an evaluation result using a sequence of sounds.

FIG. 26 is a block diagram showing an example having a plurality of evaluation units.

[Description of marks _Nos.]

 1 input unit 2 unregistered word candidate extraction unit 3 unregistered word candidate appearance information recording unit 4 unregistered word candidate evaluation unit 5 language analysis unit 6 output unit 7 character semantic dictionary 8 dictionary

Claims (4)

[Claims] 1. An input unit that receives an input sentence, an output unit that outputs an analysis result of the language analysis unit, a character string that receives the input sentence from the input unit, and may be an unregistered word from the sentence. An unregistered word candidate extraction unit that extracts the unregistered word candidate, and an unregistered word appearance information recording unit that records the unregistered word candidate extracted by the unregistered word candidate extraction unit and a character string adjacent to the unregistered word candidate, With reference to the unregistered word candidate evaluation unit that evaluates unregistered word candidates based on the information recorded in the unregistered word appearance information recording unit, the evaluation result of the unregistered word candidate evaluation unit, and from the input unit A natural language analysis device, comprising: a language analysis unit that analyzes a received input sentence. 2. A character semantic dictionary that stores characters and semantic information corresponding to the characters, further comprising: referring to the character semantic dictionary when the unregistered word candidate evaluation unit evaluates unregistered word candidates. The natural language analysis device according to claim 1, which is performed. 3. An updatable dictionary storing information for language analysis is further provided, wherein the unregistered word candidate evaluation unit updates the contents of the dictionary based on an evaluation result, and the language analysis unit based on the analysis result. 2. The natural language analysis device according to claim 1, wherein the content of the dictionary is updated by referring to the information of the dictionary when the language analysis unit analyzes the input sentence. 4. An unregistered word candidate is extracted from an input sentence, the unregistered word candidate and a character string adjacent to the unregistered word candidate are recorded, and an unregistered word candidate appears from the recorded content. A natural language analysis method characterized by acquiring information, evaluating each unregistered word candidate, analyzing the input sentence using the unregistered word candidate evaluation, and outputting the analysis result.