JPH01112331A - Automatic evaluation device for significance of key word - Google Patents

Abstract

PURPOSE:To perform retrieval by a significant key word by evaluating the significance of an extracted key word, by providing a word significance evaluation part which gives a proper evaluation point at every authorization item on an authorized word and decides the significance of a word in sequence of the total number of the evaluation points. CONSTITUTION:In a thesaurus and significant word dictionary collation part 11, the word collated by performing the collation of the word in an extraction noun table 5 with a thesaurus 12 and a significant word dictionary 13 is registered in a key word candidate table 14 as a key word candidate. The authorized results of a collateral word authorizing part 15, a high/middle-order word authorizing part 16, an appearance position authorizing part 17, and an appearance frequency authorizing part 18 are registered in a word feature authorizing table 19, and a word significance evaluation part 20 gives the evaluation point proper to every authorization item on the word authorized based on the result of the table 19, then, the significance of the word is decided in sequence of the total number of the evaluation points. In such a way, it is possible to reduce the number of the key words attached on input press news, etc., to less than 1/2, and to perform the retrieval by the key word of a news data base with high accuracy and high efficiency.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to an automatic keyword importance evaluation device, and more specifically, to a device for automatically extracting keywords from individual articles for searching newspaper article databases, etc. The present invention also relates to a device that evaluates the statistical, syntactic, and semantic importance of these keywords in articles, and ranks the keywords in order of integrated importance.

[Conventional technology]

Conventionally, the free-term method and the controlled keyword method are known as methods for automatically extracting key words from newspaper articles and the like.

In the free-term method, first, divide the target newspaper article, etc. into characters at the transition points of kanji, hiragana, etc., or
Keywords are extracted by focusing on delimiters such as ",", ".", etc., and words are further divided into parts of speech using a separation dictionary. Next, prefixes and suffixes are removed from the separated words by comparing them with a dictionary in which prefixes and suffixes are registered, and compound words are further divided using a word dictionary in which minimum unit words are registered. For example, "information search"
Divide into "information" and "search".

Next, there is a unit word dictionary in which numerical unit words are registered, as well as common words consisting of hiragana and kanji strings called unnecessary words or stop words such as ``yesterday'', ``trend'', and ``now''. Create unnecessary word dictionaries in which words that are not keywords are registered, check these dictionaries with the separated words, remove numerical unit words and stop words, and also remove numbers. Use nouns as keywords among the remaining words.

The controlled keyword method is a method in which words that are used as keywords in the free-term method processing are checked against a dictionary in which words that are keywords are registered, and keywords are selected.

[Problem that the invention seeks to solve]

Both the free term method and controlled keyword method of the above-mentioned prior art are only for keyword extraction.
It did not even evaluate and output the statistical, syntactic, and semantic importance of keywords in articles. As a result, the number of keywords assigned to newspaper articles by keyword experts called indexers is usually 5 to 6.
However, according to the conventional technology, more than 20 keywords are assigned, and for this reason, when keyword searching a newspaper article database, etc., a large number of unnecessary articles match the keyword search. However, these methods have drawbacks such as low accuracy and inefficiency, and the need to reserve a large amount of storage space for unnecessary keywords in the database.

An object of the present invention is to perform a search using important keywords by evaluating the importance of the extracted keywords when extracting keywords from individual newspaper articles, etc., in order to make keyword searches highly accurate and efficient. An object of the present invention is to provide an automatic keyword importance evaluation device that makes it possible to evaluate the importance of keywords.

[Means and operations for solving the problem] The automatic keyword importance evaluation device of the present invention includes an input processing unit 1, a noun extraction unit, an affix/numerical deletion unit, an unnecessary word deletion unit, a thesaurus/important word dictionary collation unit, Parallel word recognition section, Upper middle word recognition section, Occurrence position recognition section, Appearance frequency recognition section, Single word importance evaluation section, Prefix dictionary, Suffix dictionary, -Fune fishing words such as "yesterday" and "trend" An unnecessary word dictionary that registers words that cannot be used as keywords,
9. Words that can be keywords. In addition, synonyms and hypernyms are used to record the interrelationships between these words.

9. A thesaurus dictionary that shows word relationships such as hyponyms and related words, as well as proper names, place names, etc. that are particularly important words that you want to use as keywords. It consists of a dictionary of important words that have been recorded.

The input processing unit reads articles from a newspaper article database recorded on a magnetic storage device, etc., and the noun extraction unit extracts "ha", "ga", and "wo" from the read articles.
Extract the kanji katakana sequence immediately before the particle such as as a noun,
Register them in the extracted noun table. The affix and numeral deletion section deletes prefixes, suffixes, and numerals from each word by comparing each word in the extracted noun table with a prefix dictionary and a suffix dictionary. The number word in the word is also deleted and the extracted noun table is updated. The unnecessary word deletion unit compares the words in the extracted noun table with an unnecessary word dictionary, deletes the checked unnecessary words, and updates the extracted noun table.

The thesaurus/key word dictionary collation unit collates the words in the updated extracted noun table with the thesaurus and key word dictionary and registers the collated words as keyword candidates in the keyword candidate table.

In the parallel word recognition department, the words in the keyword candidate table are "A, BJ, rA and B", rA, B in the original article.
Words that are expressed in parallel, such as J, are recognized as parallel words, and the upper-middle word recognition department recognizes words in the keyword candidate table as upper-middle words that have hyponyms in the thesaurus. The position verification section verifies whether the words in the keyword candidate table appear in the original article from the beginning of the sentence to the predetermined character, and the appearance frequency verification section verifies the words in the keyword candidate table. Count the total number of times it appears in the original article.

The certification results of each of these certification departments are registered in the word feature certification table, and the word importance evaluation section uses the results of the word feature certification table to assign each certification item to the words certified by each certification department above. A unique evaluation score is given, and then the evaluation points are totaled for each word and the importance of the words is determined in order of the total sum.

〔Example〕

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a basic configuration diagram of an embodiment of the keyword importance automatic evaluation device of the present invention. Reference numeral 1 denotes an input device such as a keyboard or computer phototypesetting. Reference numeral 2 denotes a database read by the input device 1 and recorded in the form of character codes in a magnetic storage device, etc., which is herein referred to as a newspaper article database. 3 is an input processing unit that reads from the newspaper article database 2;

4 is "ha", "ga", " from the loaded newspaper article.
This is a noun extraction unit that extracts the kanji-katakana sequence located immediately before a particle such as "wo" as a noun.

5 is the noun extracted by the noun extraction unit 4 is 9. This is the extracted noun table that is recorded.

6 compares each word in the extracted noun table 5 with the prefix dictionary 7 and suffix dictionary 8, deletes the prefix, suffix, and particle in each word, and This is an affix and numeral deletion unit that also deletes the numerals inside and updates the extracted noun table 5. 7 and 8 are a prefix dictionary (including classifiers) and a suffix dictionary (also including classifiers), respectively.

Reference numeral 9 denotes an unnecessary word deletion unit that compares each word in the updated extracted noun table 5 with the unnecessary word dictionary 10, deletes the checked unnecessary words, and updates the extracted noun table 5. . Reference numeral 10 is an unnecessary word dictionary in which words such as ``yesterday'' and ``trend'' that are commonplace and cannot be used as keywords are registered.

Reference numeral 11 denotes a thesaurus/key word matching unit that matches each word in the updated extracted noun table 5 with the thesaurus dictionary 12 and key word dictionary 13. 12 is a thesaurus dictionary which registers words that can be used as keywords and further shows word relationships such as synonyms, hypernyms, hyponyms, and related words as mutual relationships between these words.

13 is an important word dictionary in which proper names, place names, etc. that are considered to be particularly important words are registered as keywords. 14 is a keyword candidate table in which words matched by the thesaurus/key word dictionary matching unit 11 are registered as keyword candidate words.

Reference numeral 15 denotes a parallel word recognition unit that determines whether or not words in the keyword candidate table 14 are expressed concurrently in the original newspaper article. Reference numeral 16 denotes an upper-middle word recognition unit that certifies words in the keyword candidate table 14 that have lower-order words in the thesaurus dictionary 12 as upper-middle words.

Reference numeral 17 denotes an appearance position determination unit that determines whether the word in the keyword candidate table 14 appears at a position in the original newspaper article from the beginning of the sentence to a predetermined character.

Reference numeral 18 denotes an appearance frequency recognition unit that counts how many times a word in the keyword candidate table 14 appears in the original newspaper article. 19 is each certification department 15-18
This is a feature certification table in which the certification results are registered.

20 gives a unique evaluation point for each certification item to each word certified in each of the above-mentioned certification sections 15 to 18 based on the various characteristic certification table 19, and then assigns an evaluation point to each word. This is a word importance evaluation unit that determines the importance of words in order of the total sum. 21 is word importance evaluation unit 2
A printing device 22 outputs a result of 0, and a result file 22 registers the results of the word importance evaluation section 20.

First, a newspaper article that is a target for keyword extraction is read from an input device 1 such as a keyboard or computer typesetting, and is recorded in a magnetic storage device or the like to form a newspaper article database 2. From this newspaper article database 2, the input processing section 3 inputs newspaper articles to be extracted for keywords. Noun extraction part 4
``ha'', ``ga'', ``from among the newspaper articles to be processed.
Kanji-katakana sequences located immediately before particles such as "wo" are extracted as nouns and are registered in the extracted noun table 5. FIG. 2(A) shows part of the contents of extracted nouns registered in the extracted noun table 5.

Next, the affix/number deletion unit 6 deletes the prefix # (including the particle) 7 from the word registered in the extracted noun table 5.
, prefixes and suffixes in individual words by checking with a suffix dictionary (including suffixes) 8.

The extracted noun table 5 is updated by deleting the particle and also deleting the numeral in each word. FIG. 2(b) shows a part of the extracted noun table 5 from which this affix/number has been deleted. Next, the unnecessary word deletion unit 9 compares each word in the updated extracted noun table 5 with the unnecessary word dictionary 10, and the words "ts-shu", "yesterday", [Trend J
The extracted noun table 5 is updated by deleting unnecessary words using the common word ``nato'' as a keyword. In Figure 2 (c),
A part of the extracted noun table 5 from which unnecessary words have been deleted is shown.

Next, the thesaurus/key word dictionary matching unit 11 matches each word in the updated extracted noun table 5 with the thesaurus dictionary 12 and key word dictionary 13, and selects matched words. It is registered in the keyword candidate table 14 as a keyword candidate.

As shown in FIG. 2(d), 9. is added to the keyword candidate table 14 in this way. Shows some of the words recorded.

Next, the parallel word recognition unit 15 uses the keyword candidate table 14
Regarding the words in the middle, they are found in the newspaper articles in the newspaper article database 2, such as "A and B", "A and B", rA, BJ.
For words in the keyword candidate table 14, the upper middle word recognition unit 16 determines whether or not they are expressed in parallel, such as A and B, and registers the results in the feature recognition table 19. A word with a lower term in the thesaurus is recognized as an upper middle term, and the result is registered in a characteristic recognition table 19. Next, the appearance position recognition unit 17 determines whether the appearance position of the word in the keyword candidate table 14 in the original newspaper article is N from the beginning of the sentence to a predetermined character position, and the result is are registered in the characteristics recognition table 19. In addition, experiments have confirmed that the optimum value is around the 80th to 90th character from the beginning of a sentence, and that there is little meaning in smaller or larger characters.

Next, the appearance frequency recognition unit 18 uses the keyword candidate table 1
The total number of times the words in 4 appear in the original newspaper article is counted and the results are registered in the feature recognition table 19.

FIG. 3 shows an example of the contents of the various feature recognition table 19.

Whether each word in the keyword candidate table 14 has been certified by each of the certification units 15 to 18, if yes, [0];
If there is no item, it is shown with no mark.

Next, the word importance evaluation section 20 gives a unique evaluation point for each certification item to each word certified by each of the above-mentioned certification sections 15 to 18 based on the various feature certification table 19, and then, The evaluation scores for each word are totaled, the importance of the two words is determined based on the total sum, and the result is output to the printing device 21 and registered in a result file 22 such as a magnetic storage device. FIG. 4 shows an example of the results of evaluating the importance of words, in which words are arranged in the order of their evaluated importance.

The accuracy of the overall ranking of keyword importance has been confirmed through experiments.
00 articles as experimental samples.

Up to 95% of the necessary keywords in these 200 articles are
It is included in the top 10 words of importance in each article. Therefore, for example, by using the top 10 keywords in the output results of this device, keywords can be added to the input newspaper article, whereas in the conventional technology, 20 or more keywords are attached to each newspaper article. The number of keywords is 1
/2 or less, and as a result, keyword searches in newspaper article databases can be performed with high accuracy and efficiency, and the storage capacity for keywords in the database can also be reduced to 1/2 or less.

〔Effect of the invention〕

As explained above, in addition to the conventional technology, the keyword importance automatic evaluation device of the present invention includes a parallel word recognition section, an upper middle recognition section, an appearance position recognition section, an appearance frequency recognition section, and a word importance evaluation section. The parallel word certification section certifies whether keyword candidate words are expressed in parallel, and the upper middle word certification section certifies whether keyword candidate words are upper middle words in the thesaurus. The appearance position recognition section certifies whether the keyword candidate word appears in the original newspaper article from the beginning of the sentence to a predetermined character position.The appearance frequency recognition section certifies the keyword candidate word. The word importance evaluation department counts the total number of times a word appears in the original newspaper article, and then assigns a unique rating to each word recognized in each of the abovementioned departments. An evaluation point is given, and then the evaluation points are totaled for each word, and the importance of the words is determined with high precision in the order of the total sum.

For this reason, conventional technology only extracts keywords from individual newspaper articles, etc., but more than 20 keywords are attached, including many inappropriate keywords. When keywords were used in a keyword search, many inappropriate articles were extracted, resulting in low search accuracy and inefficiency.In contrast, this device not only extracts keywords, but also extracts a large number of inappropriate articles. keywords in the original article, statistically, syntactically,
By being able to output in order of overall semantic importance, for example, by using the top 10 output results of this device as keywords, it is possible to reduce the number of keywords attached to input newspaper articles, etc. by half or less. As a result, keyword searches in the article database are made highly accurate and efficient.
Also, the storage capacity for keywords in the database is 1
There is an advantage that it can be made less than /2.

[Brief explanation of the drawing]

FIG. 1 is a basic configuration diagram of an embodiment of the automatic keyword importance evaluation device of the present invention, FIG. FIG. 3 is a diagram showing an example of the contents of the feature recognition table shown in FIG. 1, and FIG. 4 is a diagram showing an example of the results of evaluating the importance of words in the keyword candidate table. 1... Input device, 2... Newspaper article database,
3... Input processing unit, 4... Noun extraction unit, 5...
Extracted noun table, 6... Affix/number deletion unit, 7... Prefix dictionary. 8... Suffix dictionary, 9... Unnecessary word deletion section, 10.
...Unnecessary word dictionary, 11...Thesaurus/important word dictionary collation unit. 12... Thesaurus dictionary, 13... Important word dictionary, 14... Keyword candidate table, 15... Parallel word recognition section, 16... Upper middle word recognition section, 17... Occurrence position recognition Department. 18... Appearance frequency recognition section, 19... Feature recognition table, 20... Word importance evaluation section, 21... Printing device. 22...Result file. 2nd / 滲p -7°f Shitsuichi Kai P (c) (d) -'7
"l Shino, Shap

Claims (1)

[Claims] (1) In an automatic keyword importance evaluation device that automatically extracts keywords from individual newspaper articles, etc., and automatically evaluates the statistical, syntactic, and semantic importance of those keywords in the article, Database, extraction noun table, keyword candidate table, word feature recognition table, prefix and suffix dictionary, unnecessary word dictionary with common words that cannot be keywords, synonyms, hypernyms, hyponyms, related words, etc. A thesaurus dictionary showing relationships, especially an important word dictionary that registers important words to be used as keywords, an input processing unit that reads articles from the article database, and extracts nouns from the read articles and stores them in the extracted noun table. The registered noun extraction unit compares each word in the extracted noun table with the prefix and suffix dictionary, deletes prefixes, suffixes, particles, number words, etc., and updates the extracted noun table. an affix/numerical deletion unit that performs a check on the words in the extracted noun table with the unnecessary word dictionary, deletes the checked unnecessary words, and updates the extracted noun table; a thesaurus/key word dictionary matching unit that matches the words in the updated extracted noun table with the thesaurus dictionary and the key word dictionary and registers the matched words as keyword candidates in the keyword candidate table; and the keywords. a recognition unit that refers to the original article in the article database to identify parallel words, upper middle words, appearance positions, appearance frequencies, etc. for each word in the candidate table, and registers the results in the word feature recognition table; a word importance evaluation unit that assigns unique evaluation points for each certification item to the words certified by the certification unit based on the results of the various characteristic certification tables, and determines the importance of the words in order of the total sum; An automatic keyword importance evaluation device characterized by: