JPH0749872A - Automatic key word extraction system - Google Patents

Abstract

PURPOSE:To narrow down key-word extracted words by recognizing a phrase which is important when subjects are represented, term by term, and extracting a noun and an 's'-series inflection verb stem included therein as key words. CONSTITUTION:A word dividing means l divides a sentence into word units first by using a prepared dictionary 2 for word division. Then a part-d-speech recognizing means 3 recognizes the parts of speech of the respective words by utilizing a dictionary 4 for part-of-speech recognition. On the basis of the part-of-speech information recognized by the phrase recognizing means 3, nouns which succeed are recognized as one combined noun from the data divided into the word units and a particle and an auxiliary verb are recognized as one phrase by putting other words which are present right before together. Lastly, a key word extracting means 6 recognizes phrases which are used together with terms, term by term, and important when, specially, the subject is represented, in order by utilizing the obtained phrase information and a term dictionary 7, and the stems of the noun and 's'-series inflection verb in the phrases as stems that are suitable as key words.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an automatic keyword extraction method, and more particularly to an information retrieval system using a computer,
The present invention relates to an automatic keyword extraction method for extracting effective keywords from a document text of a Japanese natural language sentence when searching the document text.

[0002]

2. Description of the Related Art In a conventional automatic keyword extraction method, a Japanese natural language sentence in a document text is first word-divided based on a word-division dictionary, and then the stems of a noun and a verb are used as keyword candidates. Based on the information in the unnecessary word dictionary prepared in advance, and were extracted as candidates, the unnecessary words were removed from the nouns and syllables stems extracted as candidates. It should be noted that the unnecessary word dictionary is used to register and accumulate words that are considered inappropriate as keywords by the user based on experience.

[0003]

According to the above-described conventional automatic keyword extraction method, all nouns and sa variation stems are selected as candidates from the Japanese natural language sentences in the document text regardless of the subject of the sentence. Since it will be extracted, there is the inconvenience of preparing an unnecessary word dictionary and accumulating unnecessary words.
Nevertheless, there was a drawback that it contained many words that were not suitable as keywords.

The object of the present invention is to eliminate the above-mentioned drawbacks,
It is to provide an automatic keyword extraction method that can extract only words related to the subject of a sentence as candidates.

[0005]

According to an automatic keyword extraction method of the present invention, an information retrieval system using a computer divides a document text of a Japanese natural language sentence into word units by referring to a word division dictionary. In the keyword automatic extraction method for extracting effective keywords when searching the document text, a part-of-speech recognition dictionary storing part-of-speech information of each word and the part-of-speech recognition dictionary for data divided into words Referring to the part-of-speech recognition means for adding the part-of-speech information, and referring to the part-of-speech information added by the part-of-speech recognition means, a plurality of consecutive noun words are collectively referred to as 1
A phrase recognition means for combining particles and auxiliary verbs with words immediately preceding them into words and combining them into phrases, and a phrase dictionary that is used together with a phrase to store the relational information between the particles forming the sentence and the phrases. , And a keyword extracting means for extracting a keyword by referring to the phrase dictionary and focusing on a phrase including a phrase from the data collected in the phrase by the phrase recognition means.

[0006]

Embodiments of the present invention will now be described with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention.

As shown in FIG. 1, the automatic keyword extraction method of the present embodiment is necessary for word division means 1 for dividing the document text of the target Japanese natural language sentence into word units, and for performing word division. Dictionary for storing word-division information 2, part-of-speech recognition means 3 for recognizing and adding part-of-speech information to data divided into words, and part-of-speech recognition for storing information necessary for part-of-speech recognition Dictionary 4, phrase recognition means 5 for collecting word-by-word data to which part-of-speech information is added for each phrase, and keyword extraction means 6 for extracting keywords from the data collected in the phrase by paying attention to the phrase.
And a vocabulary dictionary 7 in which information about vocabulary necessary for keyword extraction is stored.

FIG. 2 shows an example sentence of a Japanese natural language sentence for explaining the operation of this embodiment. The operation of this embodiment will be described below with reference to the example sentence of FIG.

First, the word dividing means 1 performs the word dividing process by the same method as the conventional method. That is, the Japanese natural language sentence in the specified document text is used as input data,
It is divided into word units by using the word division dictionary 2 prepared in advance. The result of division by the word dividing means 1 is shown in FIG. In FIG. 3, “Δ” represents the division of each word.

Next, the part-of-speech recognition means 3 uses the part-of-speech recognition dictionary 4 to recognize the part-of-speech of each word. Here, the part-of-speech is a verb used in normal Japanese grammar, a sa verb, an adjective, an adjective, a noun, an adverb, a conjunction, a conjunction, a verb,
These are the 11 parts of speech of auxiliary verbs and particles. It should be noted that although the sa verb is a part of the verb, it is treated separately from other verbs because it is a verbized noun and includes words that are keywords. In addition, the “defective” used in the following description is a collection of four parts of speech: verb, sa verb, adjective, and adjective verb.

The part-of-speech recognition dictionary 4 basically includes not only the notation of each word and two pieces of information on the part-of-speech, but also about words that change their endings such as five parts-parts of verbs, sa verbs, adjectives, adjectives and auxiliary verbs. Two types of information are stored: a basic form, which is an expression, and a stem, which is a part where the notation does not change. Information of the part-of-speech recognition dictionary 4 necessary for the example sentence of FIG. 2 is shown in FIG.

Explaining with an example, the word "identification" (reference numeral 31) in FIG. 3 is "identification" in FIG.
It is recognized from the information (reference numeral 41) that the part of speech is a "noun". Similarly, for the word “compare” (reference numeral 32) in FIG. 3, “compare” (reference numeral 4) in FIG.
It is recognized from the information in 2) that the part-of-speech is "sa verb," its basic form is "compare," and the stem is "comparison."

FIG. 5 shows the result of processing the word segmentation means 1 and the part-of-speech recognition means 3 on the example sentence of FIG. In FIG. 5, “Δ” indicates the division of each word, and the column indicated by the reference number A is the storage location of the recognized part-of-speech information.

Next, the phrase recognition unit 5 recognizes a phrase from the data divided into words based on the part-of-speech information recognized by the part-of-speech recognition unit 3 and the following two rules. The rules used by the phrase recognition means 5 are as follows. (1) When nouns are consecutive, the consecutive nouns are combined into one noun. (2) Particles and auxiliary verbs are grouped together with other words that exist immediately before into one phrase.

Explaining with an example, the words "identification" (reference numeral 51) and "result" (reference numeral 52) in FIG.
Since the noun is continuous, it is recognized as one noun “identification result” according to the above-mentioned rule (1). Further, since the word “to” (reference numeral 53) in FIG. 5 is a postpositional particle, the noun “identification result” immediately preceding the noun existing according to the above rule (2)
Recognized as one clause "identification result △ and" collectively.
Similarly, since the word “te” (reference numeral 55) in FIG. 5 is also a particle, it is collectively referred to as the immediately preceding sa verb “comparison” (reference numeral 54) according to the above-mentioned rule (2). Recognize it as one phrase, "Compare and △".

FIG. 6 shows the result obtained by processing the word segmentation means 1, the part-of-speech recognition means 3, and the clause recognition means 5 on the example sentence of FIG.
Shown in. In FIG. 6, “Δ” indicates the division of each word,
"★" represents a delimiter of each clause, and the column indicated by reference number A is the storage location of the recognized part-of-speech information.

Finally, the keyword extracting means 6 will be described. The keyword extraction unit 6 uses the phrase information obtained by the phrase recognition unit 5 and the vocabulary dictionary 7, and the vocabulary necessary for constructing a sentence by using the verb with each vocabulary and those phrases. Among the bunsetsus, the bunsetsus that are particularly important when expressing the subject are sequentially recognized, and the stems of the nouns and sa verbs that are important when expressing the subject are extracted as appropriate keywords. It should be noted that the denotation dictionary 7 used for this processing contains information about particles, which is included in a clause necessary for constructing a sentence together with a denotation, and in particular, each subject is a subject. The information of the particle included in the clause which is judged to be important when constructing is stored.
Information of the vocabulary dictionary 7 regarding the example sentence of FIG. 2 is shown in FIG.

Explaining with an example, the word "comparing" in the phrase "comparing Δ te" (reference numeral 65) containing the adjective (sa verb) in FIG. 6 is "comparing" in FIG. (Reference number 4
Since the basic form is "comparing" from the information of 2), the information of "compare" (reference numeral 71) in FIG. 7 is referred to by using the basic form "compare". In other words, for "comparing", the particles are "ga" and "wo".
A phrase containing either "to" is necessary to compose a sentence, and the particles "ga" and "wo" in that phrase are included in the phrase that is important when expressing the subject. is there. Therefore, from “comparison △ te” toward the beginning of the sentence, we search for a phrase in which the particle contains either “ga”, “wo” or “to”, and the phrase “comparison” composes the sentence. The phrase necessary for this is "identification result △ and" in FIG.
2), it is determined that the phrase is “topic identification result Δ” (reference number 61). “Compare” in FIG. 7 (reference numeral 7
From the information in 1), it can be seen that the phrase containing the particle "to" is not important when expressing the subject. Therefore, in the phrase "identification result Δto" (reference numeral 62) containing the particle "to" in FIG. It is determined that the noun "identification result" in is not extracted as a keyword. On the contrary, since the phrase containing the particle "o" is important when representing the subject, the noun "topic identification result" in the phrase "topic identification result △ wo" (reference number 64) containing the particle "o" is It is determined to be extracted as a keyword.

Similarly, the word “go” in the phrase “go Δta” (reference numeral 68) including the adjective (verb) in FIG. 6 is also judged. First, “Go” in FIG. 4 (reference numeral 43)
Since the basic form "do" can be obtained from the information of "do", the basic form "do" is used to "do" in FIG.
Refer to the information of (reference number 72). In other words, "do"
For, a clause whose particle is either "ga" or "wo" is necessary to compose a sentence, and the particle is "ga" or "wo".
Information that is important when a clause that is one of Therefore, the phrase necessary for composing a sentence is searched for the phrase in which the particle is “ga” “wo” from the direction of “go △” toward the beginning of the sentence. It is judged that it is a clause of “evaluation Δ” in 6 (reference numeral 67). At this time, the “topic identification result Δ in FIG.
”” (Reference numeral 64) is also a phrase that includes the particle “”, but since it was previously determined to be a phrase necessary to compose a sentence by the “comparison” phrase, it is a duplicate phrase. Do not judge that the phrase is necessary for "do". As described above, FIG.
From the information of “do” (reference number 72) of, the phrase including the particle “o” is an important phrase when expressing the subject,
It is determined that the noun “evaluation” in “evaluation Δ” in FIG. 6 (reference numeral 67), which is a clause necessary to compose a sentence, is extracted as a keyword.

Note that the phrase "by topic identification result Δ" (reference numeral 61), which is a clause including the particle "by" in FIG.
“Topic identification model Δno” (reference number 66), which is a clause including “by hand Δ” (reference number 63) and a particle “no”
In this case, “compare” (reference numeral 72) and “perform” in FIG. 7 are applied to any of the terms “compare” and “perform”.
Based on the information (reference numeral 72), it is not determined that the clause is necessary when composing a sentence. Therefore, these three clauses, that is, “according to the topic identification result Δ” in FIG. 6 (reference numeral 6
No keywords are extracted from 1), “by human Δ” (reference number 63), and “of topic identification model Δ” (reference number 66).

FIG. 8 shows the results obtained by performing the respective processes of the word dividing means 1, the part-of-speech recognition means 3, the clause recognition means 5, and the keyword extraction means 6 on the example sentence of FIG. 2 together with the extraction results by the conventional method. Show. In FIG. 8, “Δ” indicates the division of each word, “★” indicates the division of each clause, the column of reference number A shows the recognized part-of-speech information, and the column of reference number B shows the keyword according to this embodiment. The determination by the conventional method is shown in the column of reference numeral C. Figure 8
Referring to, between the method of the present invention and the conventional method,
"Identification topic model" (reference number 81), "identification result"
(Reference number 82), “manpower” (reference number 83), and “topic identification model” (reference number 84), differences are recognized, and it can be seen that a large number of unnecessary keyword candidates are not extracted by the method of the present invention.

[0023]

As described above, according to the automatic keyword extraction method of the present invention, when a keyword is extracted from a document text of a Japanese natural language sentence by an information retrieval system using a computer, the keywords are grouped into phrases and then Pay attention to the phrase, which phrase is needed in the natural language sentence, and the phrase that is important when the phrase is representative of the subject, are judged from the notation method without performing the semantic analysis of the sentence, and By recognizing the clauses that are important when expressing the subject and extracting the nouns and sa verbs included in them as keywords, there is an effect that the words extracted as keywords can be narrowed down.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention.

FIG. 2 is an explanatory diagram showing an example sentence of a document text for explaining the present embodiment.

FIG. 3 is an explanatory diagram of a result of performing word division processing on the example sentence of FIG.

FIG. 4 is an explanatory diagram showing information in a part-of-speech recognition dictionary for the example sentence of FIG.

5 is an explanatory diagram showing a part-of-speech recognition processing result for the example sentence of FIG. 2. FIG.

FIG. 6 is an explanatory diagram showing a phrase recognition processing result for the example sentence of FIG. 2;

FIG. 7 is an explanatory diagram showing information of a vocabulary dictionary for the example sentence of FIG. 2;

FIG. 8 is an explanatory diagram comparing a keyword extraction result for the example sentence of FIG. 2 with a related art.

[Explanation of symbols]

 1 word dividing means 2 word dividing dictionary 3 part-of-speech recognition means 4 part-of-speech recognition dictionary 5 phrase recognition means 6 keyword extraction means 7 vocabulary dictionary

Claims (1)

[Claims] 1. An information retrieval system using a computer divides a document text of a Japanese natural language sentence into word units by referring to a word segmentation dictionary, and then identifies effective keywords when retrieving the document text. In the keyword automatic extraction method for extracting, a part-of-speech recognition dictionary in which part-of-speech information of each word is stored, and a part-of-speech recognition unit that adds part-of-speech information to data divided into words by referring to the part-of-speech recognition dictionary. A phrase recognition unit that refers to the part-of-speech information added by the part-of-speech recognition unit to group a plurality of consecutive noun words into one word, and then combines the particle and auxiliary verb with the immediately preceding word to group them into phrase units; A phrase is stored from the data collected in the phrase by the phrase recognition means and the phrase dictionary that stores the relational information between the particle and the phrase that are used together with the phrase to compose the sentence. An automatic keyword extraction method, comprising: keyword extraction means for extracting a keyword by referring to the vocabulary including the phrase.