JPH10320403A - Method and device for generating retrieval expression, and record medium - Google Patents

Abstract

(57) [Summary] [Problem] By focusing on the importance of keywords and preferentially selecting keywords having a low appearance frequency in document data as target keywords, it is possible to maintain search accuracy at or above a certain value A search expression creating device. A keyword extracting unit and a document set dividing unit are provided.
1, a keyword document frequency dictionary 12 and a search formula creation unit 33 are provided. When there are a plurality of keywords as search keyword candidates, the number of documents (document frequency) including each keyword is determined by a keyword document. The keyword is read out from the frequency dictionary 12 and compared, and a keyword with a low document frequency is preferentially selected as a search keyword. The search expression creation unit 33 converts the search keyword into a logical operator “and”,
And "or" to create a search expression.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document database for retrieving a specific document from electronic documents such as academic papers and technical documents accumulated in a large amount, and a method for creating a document stored in advance in an electronic document. The present invention relates to a document search technology applied to various support systems used for supporting the development of ideas, and in particular, a searcher can efficiently search for a document of interest using keywords extracted from an electronic document. The present invention relates to a technique for creating a search formula to be performed by trial and error.

[0002]

2. Description of the Related Art A keyword is extracted from a document database in which electronic documents to be searched are stored, and a required search formula is created by trial and error in cooperation with a searcher by a combination of a logical product and a logical sum of the keywords. There is known a search formula creation device.

FIG. 3 is a functional block diagram of this type of conventional search expression creating apparatus. The search formula creation device 30 includes functional blocks of a keyword extraction unit 31, a document set division unit 32, and a search formula creation unit 33, which are formed by a computer device reading and executing a predetermined program. Note that it is assumed that each document is provided with necessary / unnecessary designation information indicating whether the searcher is a necessary document of interest or an unnecessary document of no interest.

The keyword extracting unit 31 performs a process of extracting a plurality of keywords for each document from the plurality of documents by a known morphological analysis process. Also, discrimination information indicating whether a keyword appears in each document and identification information indicating whether the document is a necessary document or an unnecessary document are output as a document set together with a document identifier such as a document name and a document number. Code 31
B illustrates the contents of the document set output from the keyword extraction unit 31.

[0005] The document set dividing section 32 divides the document set in stages based on the discrimination information, and determines a plurality of search keywords that are the basis for creating a search formula used for document search. In this case, by dividing the document set by the discrimination information of one (small) keyword as much as possible, it is possible to extract the intention of the searcher who distinguishes the necessary documents from the unnecessary documents. The plurality of search keywords determined by the document set division unit 32 are combined by the logical operator “and” or “or” in the search expression creation unit 33, and are output to the subsequent processing as a search expression.

[0006] The document set dividing process in the document set dividing section 32 is performed based on, for example, "ID3" which is a known decision tree (a logical expression represented by a tree structure) learning algorithm. Hereinafter, this decision tree learning algorithm “ID
The outline of the document set division process by “3” will be described with reference to FIG. First, the document set sent from the keyword extraction unit 31 is set as an initial document set Set ₀ (step S10).
1). Next, the "undivided" flag of the initial document set Set ₀ is turned on (step S102), and this is set as Set _i (step S103). Next, this document set Set
Necessary documents in _i , each keyword t included in unnecessary documents
_{For j} (1 ≦ j ≦ N), a mutual information amount I (t _j ) representing a relative relationship between the information amount of the individual document and the information amount of the entire document is calculated (step S104). Mutual information I (t _j )
Is, specifically, the information amount H about the undivided document set.
Is subtracted from the information amount H (t _j ) of the document set that includes the keyword t _j and the document set that does not include the keyword t _j . The information amounts H and H (t _j ) are expressed by the following equations (1) and (2), respectively.

[0007]

(Equation 1)

The parameters in the equations (1) and (2) are as follows. p _i: Set _i need the number of documents in, n _i: Set _i unnecessary number of documents _{in, s i: p i + n} i, p i (t j): necessary number of documents that contain the keyword t _j in Set _{i , n i (t j):} Set i unnecessary number of documents that contain the keyword t _{j in, s i (t j):} p i (t j) + n i (t j), p i not (t j): Set _i need the number of documents that do not contain the keyword t _{j in, n i not (t j)} : Set i unnecessary number of documents that do not contain the keyword t _{j in, s i not (t j)} : p i not (t j) + N _i not (t _j ), h (a, b, c):-{a / c · log ₂ (a / c) + b / c · log ₂ (b / c)}

Next, a keyword t which can maximize the value of the mutual information I (t _k ) from a plurality of keywords t _j
k is selected and set as a search keyword (step S
105). This mutual information I (t _k ) has a positive finite value (>
In the case of 0) (step S106), the search keyword t _k
Set _i 'a document set consisting of number of documents containing the flag of the search keyword set of documents consisting of number of documents that do not contain t _k Set _i "divided as, for each document set divided" undivided " Turn on (step S107)
To S110). i ′, i ″ are already document sets Set _i ′, S
If et _i ″ does not exist, any value may be used.On the other hand, if the mutual information amount I (t _k ) is a zero value (= 0), the document set is not divided (step S106).

Thereafter, the flag of “undivided” of the set Set _i is turned off (step S111). If there is a document set with the “undivided” flag on, the process returns to step S103 (step S112, Yes), and the process is repeated until there is no document set with the “undivided” flag on. Then, the process ends when the “undivided” flag is turned off for all the document sets (step S11).
2, No).

The details of the decision tree learning algorithm "ID3" can be referred to "Knowledge Acquisition and Learning Series 1: Introduction to Knowledge Acquisition" (Michalski, RS, et al., Kyoritsu Shuppan). In addition, the process using the algorithm “ID3” may be replaced with, for example, a known algorithm “C4.5” or another algorithm that divides a document set using only the presence or absence of a keyword in a document. It is possible. See “C4.5 Programs for Machine Learning” for details of “C4.5”.
(Quinlan, JR, published by Morgan Kaufmann Publishers).

FIG. 5 is an explanatory diagram showing a process in which the above-mentioned search formula creating apparatus 30 divides one document set into a plurality of document sets and creates a search formula by trial and error. Hereinafter, with reference to FIG. 5, a description will be given of a procedure for creating a conventional search expression. First, from the initial document set Set ₀ output from the keyword extraction unit 31, a keyword that maximizes the mutual information is determined based on the above-described decision tree learning algorithm “ID3”, and is set as a search keyword. Here, it is assumed that the search keyword kwd3 has been determined. Then, by this search Kwd3, divides the initial document set Set _0, the search keyword Kwd3 in a set Set ₂ required documents and unnecessary documents that do not contain the set Set ₁ the search keyword Kwd3 necessary documents including.

The document set Set ₁ cannot be further divided, while the document set Set ₂ can be further divided. Therefore, the search keyword kwd2 that maximizes the mutual information in the document set Set ₂ is determined, and the document set Sed is determined by the search keyword kwd2.
The t _2, divides search in Kwd2 a set Set ₃ of unnecessary documents that do not contain a search set of necessary and unnecessary documents containing the keyword kwd2 Set _4. Document Set Set ₄
Can be further divided, so this document set Set ₄
Is determined as the search keyword, and the keyword kwd1 with the maximum mutual information amount is determined.
A set Set _{5 of} necessary documents including d1 and a set Set _{6 of} documents not including the search keyword kwd1 are divided.
Since the document sets Set ₅ and Set ₆ cannot be further divided, the division processing ends.

The plurality of search keywords kwd1 to kwd3 determined in the above-mentioned division processing are sequentially stored in storage means (not shown), and are passed to the retrieval formula creation unit 33 when the division processing is completed. The search expression creation unit 33 combines the search keywords, which are the result of the document set division unit 32, with the logical operators "and" and "or" to combine the search keywords que.
Create ry. Reference numeral 33B illustrates a search formula output from the search formula creation unit 33. At t ₆ , the division process ends because no further division is possible.

The plurality of search keywords kwd1 to kwd3 determined in the above-mentioned division processing are sequentially stored in storage means (not shown), and are passed to the retrieval formula creation unit 33 when the division processing is completed. The search expression creation unit 33 combines the search keywords, which are the result of the document set division unit 32, with the logical operators "and" and "or" to combine the search keywords que.
Create ry. Reference numeral 33B illustrates a search formula output from the search formula creation unit 33.

[0016]

In the above-mentioned conventional retrieval formula creation device 30, the retrieval keyword used for dividing the document set is determined based on the mutual information amount and the necessary / unnecessary document discrimination information. Does not consider whether the keyword is truly important to the searcher. For this reason, the created search formula may include an unimportant search keyword, and when this search formula is used in actual document search processing, sufficient search accuracy may not be obtained.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an improved method of using a computer to create a search formula capable of maintaining a search accuracy in a document search at a certain value or more by reflecting the importance of a keyword. Is to do. Another object of the present invention is to provide a search formula creation device suitable for implementing the above method and a recording medium for implementing the search formula creation method with a general-purpose computer device.

[0018]

According to the present invention, there is provided a method for creating a retrieval formula according to the present invention, comprising the steps of: morphologically analyzing a specified document group in a document database to extract a plurality of keywords; And the mutual information amount obtained by subtracting the information amount of the document group including and not including the keyword from the total information amount of the specified document group is maximized, and Determining a keyword that minimizes the number of documents including the keyword as a search keyword; and combining the determined search keyword with a logical expression to create a search expression used for searching the document database. And

According to another search formula creation method of the present invention, a process of morphologically analyzing a specified document group in a document database to extract a plurality of keywords, and the number of documents in which the extracted individual keywords appear are determined for each keyword. The detection step, the importance of the keyword calculated based on the monotonically decreasing function with respect to the number of documents, and the information amount of the document group including and not including the keyword are subtracted from the total information amount of the designated document group. And a process of determining the keyword having the largest total value as a search keyword, and combining the determined search keywords with a logical formula to create a search formula used for document search. It is characterized by including.

According to another aspect of the present invention, there is provided a search formula generating apparatus for generating a search formula for searching for a specific document from a document database. A plurality of keywords are extracted by the analysis, and discrimination information indicating whether or not each extracted keyword is included in the document, and designation information indicating whether the document is a necessary document or an unnecessary document, together with identification information of each designated document. A keyword extracting unit that generates a set of documents, a mutual information amount obtained by subtracting the information amount of the document group including and not including the individual keywords from the total information amount of the specified document group, and the keyword appear. A single keyword is determined as a search keyword based on the number of documents, and one document is determined using the determined search keyword. A document set dividing unit that divides the document set into a plurality of document sets, and a search formula creating unit that creates the search formula by combining search keywords used in dividing the document set with a logical expression. Features.

A dictionary may be further provided in which the number of documents in which the extracted keywords appear is counted in advance and held for each keyword. In this case, the document set division unit is configured to search out the number of relevant documents held in the dictionary and determine the search keyword when dividing the document set.

The document set dividing unit calculates the importance of the keyword by using a keyword in which the mutual information amount is maximum and the number of documents is minimum, or a monotone decreasing function with respect to the number of documents. The keyword having the maximum value obtained by adding the calculated importance and the mutual information at a fixed ratio is determined as the search keyword.

According to another aspect of the present invention, there is provided a recording medium for extracting a plurality of keywords from a group of designated documents in a document database, and determining whether each keyword is included in the document. Keyword extraction processing for generating a document set in which information and discrimination information indicating whether the document is a necessary document or an unnecessary document are collected together with identification information of each designated document; A single keyword is determined as a search keyword based on the mutual information amount obtained by subtracting the information amount from the total information amount of all document groups and the number of documents in which the keyword appears, and one keyword is determined using the determined search keyword. A document set division process for dividing one document set into a plurality of document sets, and a search expression used for dividing the document set used in a logical expression. A search expression creation processing for creating the search expression combined with a program for causing a computer to execute the device comprising recorded in a form readable is the computer device.

[0024]

Embodiments of the present invention will be described below in detail. FIG. 1 is a functional block diagram showing an embodiment of a search formula creation device to which the present invention is applied. Note that FIG.
The same reference numerals are given to the same functions as those of the conventional search-expression creating apparatus 30 described above, and redundant description is omitted.

The retrieval formula creation device 10 of the present embodiment includes a keyword extraction unit 31, an improved document set division unit 11, and a keyword document frequency dictionary 1 formed by a computer device reading and executing a predetermined program.
2. It is provided with the functions of the search formula creation unit 33. The above-mentioned program is usually stored in a storage means incorporated in the computer device, and is read out and used at any time by a main control unit (CPU) of the computer device. It may be stored in a medium, for example, a CD-ROM, and installed in the storage means at the time of use.

The document set division unit 11 divides the initial document set output from the keyword extraction unit 31 using the above-described decision tree learning algorithm “ID3”. In general, a keyword that appears in a document with a low frequency can be an important keyword for distinguishing a plurality of documents. Therefore, in the present embodiment, the number of documents in which the keyword appears (document frequency)
Is stored in the keyword document frequency dictionary 12, the document frequency of each keyword is read out from the keyword document frequency dictionary 12, and the importance of the keyword is determined using a monotonically decreasing function for the document frequency, for example, the reciprocal of the document frequency. calculate. The importance may be stored in advance in the keyword document frequency dictionary 12 for each keyword together with the document frequency.

The importance thus obtained and the mutual information amount are added at a fixed ratio, and a keyword having a large sum is preferentially determined as a search keyword.
Then, the document set is divided according to the determined search keywords, and the search keywords determined in the division process are output to the search creating unit 33.

FIG. 2 is an explanatory diagram showing a process in which one document set 31A is divided into a plurality of document sets in a stepwise manner and a search formula is created in the search formula creating apparatus 10. Hereinafter, a search formula creation procedure according to the present embodiment will be described with reference to FIG.

First, a keyword having the maximum mutual information is determined as a search keyword from the initial document set Set ₀ output from the keyword extraction unit 31. here,
It is assumed that the search keyword kwd3 has been determined. The initial document set Se is determined by the determined search keyword kwd3.
t ₀ is divided into a set Set _{1 of} necessary documents including the search keyword kwd3 and a set Set _{2 of} documents not including the search keyword kwd3.

The document set Set ₁ cannot be further divided. On the other hand, the document set Set ₂ can be further divided. In this set ₂ , the mutual information amount for each keyword is calculated. Identify the keyword with the largest amount of information. In the case of this example, the document set Set ₂
, Two keywords kwd2 and kwd4 are specified. When a plurality of keywords are candidates for a search keyword as described above, the document frequency of the corresponding keyword stored in the keyword document frequency dictionary 12 is read, and the respective values are compared. In this example, the keyword kwd4
Is determined to be a search keyword because the document frequency is relatively low. Then, the document set Set ₂ is set to a document set Set not including the search keyword kwd4.
₃ and a set Set _{4 of} documents including the search keyword kwd4
And split into

Since the document set Set ₄ can be further divided, this document set Set ₄ is used as a search keyword kwd.
Set _{5 of} required documents including “1” and search keyword k
The divided into a set Set ₆ of the document that does not contain wd1. Since the document sets Set ₅ and Set ₆ cannot be further divided, the division processing ends.

The plurality of search keywords kwd1, kwd3, and kwd4 determined in the above-described division processing are sequentially stored in storage means (not shown), and are output to the search expression creating unit 33 when the division processing is completed. Search expression creation unit 33
Then, the search keywords received from the document set division unit 11 are combined by the logical operators “and” and “or” to create a search expression query. Reference numeral 33A illustrates a search formula output from the search formula creation unit 33.

As described above, according to the search expression creation apparatus 10 of the present embodiment, when a plurality of search keyword candidates having the maximum mutual information I are specified, the search result is obtained from the keyword document frequency dictionary 12. A keyword having a large value obtained by reading out the document frequency of the keyword and performing comparison, preferentially selecting a keyword with a low document frequency, determining the keyword as a search keyword, and adding the importance and mutual information at a fixed ratio. Is determined as a search keyword, a quick search expression creation process can be performed, and an important keyword is included in the created search expression.

[0034]

As is apparent from the above description, according to the present invention, there is an effect that a retrieval formula is created in which the importance of each keyword is considered. Further, by using the retrieval formula obtained thereby, it is possible to maintain the retrieval accuracy of the document data at a certain value or more, and there is also an effect that the efficiency of the retrieval processing is greatly improved.

[Brief description of the drawings]

FIG. 1 is a functional configuration diagram showing an embodiment of a search formula creation device of the present invention.

FIG. 2 is a schematic diagram of information obtained in a process of the search formula creation device of the present invention.

FIG. 3 is a functional configuration diagram of a conventional search expression creation device.

FIG. 4 is a diagram showing a procedure in a document set division process of the conventional search expression creating apparatus.

FIG. 5 is a schematic diagram of information obtained in a processing process of a conventional search expression creating device.

[Explanation of symbols]

 10, 30 retrieval formula creation device 31 keyword extraction unit 11, 32 document set division unit 12 keyword document frequency dictionary 33 search formula creation unit

Claims (7)

[Claims] 1. A process of morphologically analyzing a specified document group in a document database to extract a plurality of keywords, a process of detecting, for each keyword, the number of documents in which the extracted individual keywords appear, and including the keyword. Determining, as a search keyword, a keyword that maximizes the mutual information obtained by subtracting the information amount of the document group and the information amount of the document group not included from the total information amount of the specified document group and minimizes the number of documents including the keyword And a step of combining the determined search keywords with a logical expression to create a search expression used for searching the document database. 2. A step of morphologically analyzing a specified document group in a document database to extract a plurality of keywords, a step of detecting, for each keyword, the number of documents in which the extracted individual keywords appear, At a fixed ratio, the importance of the keyword calculated based on the monotone decreasing function and the mutual information obtained by subtracting the information amount of the document group including and not including the keyword from the total information amount of the specified document group Computer, comprising: a step of determining a keyword having the maximum total value as a search keyword; and a step of combining the determined search keywords with a logical expression to create a search expression used for document search. A method for creating a search formula using a device. 3. An apparatus for creating a retrieval formula for searching for a specific document from a document database, comprising extracting a plurality of keywords from a specified document group in the document database by morphological analysis and extracting the extracted individual keywords. A keyword extracting unit that generates a document set in which discrimination information indicating whether the document is included in the document and designation information indicating whether the document is a required document or an unnecessary document together with identification information of each designated document; A single keyword is used as a search keyword based on the mutual information obtained by subtracting the information amount of the document group including and not including the keyword from the total information amount of the specified document group and the number of documents in which the keyword appears. A document set dividing unit that decides and divides one document set into a plurality of document sets using the determined search keyword; Search expression creating apparatus characterized by having a search expression creation unit for creating the search expression combined with a logical expression a search keyword used when the division of the document set. 4. The apparatus further comprises a dictionary in which the number of documents in which the extracted keywords appear is counted in advance and held for each keyword, and wherein the document set division unit holds the document set when dividing the document set. 4. The search formula creation device according to claim 3, wherein the search keyword is determined by finding out the number of documents. 5. The document set division unit is configured to determine, as the search keyword, a keyword in which the mutual information amount is maximum and the number of documents is minimum. Search formula creation device. 6. The document set division unit calculates the importance of the keyword using a monotonically decreasing function with respect to the number of documents, and sums the calculated importance and the mutual information at a fixed ratio. The search expression creating apparatus according to claim 4, wherein a keyword having a maximum value is determined as the search keyword. 7. A morphological analysis process for extracting a plurality of keywords from a specified document group in a document database, discrimination information indicating whether each keyword is included in the document, and whether the document is a required document or an unnecessary document Keyword extraction processing that generates a document set in which the discrimination information indicating the information is collected together with the identification information of each designated document, and the information amount of the document group including and not including the individual keywords is subtracted from the total information amount of the designated document group A single keyword is determined as a search keyword based on the mutual information amount obtained and the number of documents in which the keyword appears, and a document set is divided into a plurality of document sets using the determined search keyword. A set dividing process and a search formula creating the search formula by combining a search keyword used in dividing the document set with a logical formula And a computer-readable storage medium storing a program for causing a computer device to execute the program.