JPH11203309A - Method for preparing retrieval expression and device therefor - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide a search formula creation device capable of creating a search formula capable of maintaining a search accuracy at a certain value or more in consideration of a compound word composed of a plurality of words. A morphological analysis unit and a keyword extraction unit.
2. The retrieval formula creation device 10 is provided with the functions of the compound word processing unit 11, the document set division unit 33, and the retrieval formula creation unit 34. When the noun phrases are continuous in a plurality of words extracted by the keyword extracting unit 32, the corresponding words are combined into a compound word in the compound word processing unit 11, and the combined words are combined into a document set output by the keyword extracting unit 32 and It is used for dividing a document set in the document set dividing unit 33. If a compound word is more effective than a word at the time of document set division, this compound word is determined as a search keyword, and is reflected in the search formula created by the search formula creation unit 34.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document database for retrieving specific information from a large amount of stored electronic documents, and a pre-stored example of an electronic document for supporting document creation and idea development. Related to the document search technology applied to various support systems to be used, in particular, using a keyword extracted from an electronic document, a search formula for a searcher to efficiently search for a document of interest is tried. It relates to a method of making mistakes.

[0002]

2. Description of the Related Art There is known a retrieval formula creation apparatus which extracts words from a document database in which electronic documents to be searched are stored, and combines the words by trial and error to create a required search formula. FIG. 3 is a functional configuration diagram of this type of conventional search expression creating apparatus. The search formula creation device 30 is formed by a computer device reading and executing a predetermined program, and is formed by a morphological analysis unit 31, a keyword extraction unit 32, a document set division unit 33, and a search formula creation unit 3.
4 functional blocks. Note that each document is provided with necessary / unnecessary designation information indicating whether the document is a necessary document that the searcher is interested in or an unnecessary document that is not interested.

[0003] The morphological analysis unit 31 performs morphological analysis for each document from a plurality of input documents. In the figure, reference numeral 31
B shows an output example of the morphological analysis unit 31.
In the following description, it is assumed that five documents having the following contents are input.

Document No. 1 (necessary document) Telephone service such as Telehodai (note) (note: a trademark of Nippon Telegraph and Telephone Corporation) Document No. 2 (unnecessary document) Free admission to museums by subway ticket Document No. 3 (necessary document) Various telephone discount services are popular. Document number 4 (unnecessary document) Securities companies sell bonds by telephone Document number 5 (necessary document) Telehodai's subscribers increase

[0005] A keyword extraction unit 32 extracts words that can be used as keywords based on a morpheme group for each document. Further, it outputs determination information indicating presence / absence of a word in each document and designation information for determining whether the document is a necessary document or an unnecessary document as a document set together with a document identifier such as a document name and a document number. Reference numeral 32B is
This is an example of a document set output from the keyword extraction unit 32. “1” to “5” are document identifiers, and “necessary”
/ "Unnecessary" is designation information, and "o" / "x" is discrimination information.

The document set division unit 33 divides the document set in stages based on the discrimination information “O” / “X”, and generates a search keyword as a basis for creating a search formula used for document search. It is determined from the extracted word group. In this case, by dividing the document set by the discrimination information of as few keywords as possible, it is possible to extract the intention of the searcher who distinguishes the necessary documents from the unnecessary documents. The search keywords determined by the document set division unit 33 are combined by the logical operator “and” or “or” in the search expression creation unit 34, and are output to the subsequent processing as a search expression.

The document set dividing process in the document set dividing section 33 is performed, for example, by using a well-known MDL (Minimum Descri
ption Length: minimum description length).
This MDL principle is based on the idea that “more necessary documents and unnecessary documents are combined with as few keywords as possible (search expression).
Can be more accurately expressed the intention of a human (searcher) by discriminating with the "." However, since strict realization of the MDL principle requires a large amount of processing, Is generally realized approximately in order to reduce the processing amount. As a method of approximately realizing the MDL principle, for example, a decision tree (a logical expression represented by a tree structure) learning algorithm “I
D3 "is known. About this "ID3",
"Knowledge Acquisition and Learning Series 1: Introduction to Knowledge Acquisition" (Mic
halski, R .; S. Other editions, Kyoritsu Shuppan) are described in detail.

Hereinafter, this decision tree learning algorithm “ID
The outline of the document set division process by “3” will be described with reference to FIG. First, the document set sent from the keyword extraction unit 32 is set as an initial document set Set ₀ (step S20).
1). Next, the “undivided” flag of the initial document set Set ₀ is turned on (step S202). This is Set _i
(Step S203). Next, this document set Se
Necessary documents in t _i , each keyword t _j included in unnecessary documents
For (1 ≦ j ≦ N), a mutual information amount I (t _j ) representing the relative relationship between the information amount of the individual document and the information amount of the entire document is calculated (step S204). Mutual information I (t _j )
Is, specifically, the information amount H about the undivided document set.
Is subtracted from the information amount H (t _j ) of the document set including the keyword t _j and the document set not including the keyword t _j from the following expression (1). I (t _j ) = H−H (t _j ) (1)

However, the parameters in equation (1) are as follows. p _i: Set _i need the number of documents in, n _i: Set _i unnecessary number of documents _{in, s i: p i + n} i, p i (t j): necessary number of documents that contain the keyword t _j in Set _{i , n i (t j):} Set i unnecessary number of documents that contain the keyword t _{j in, s i (t j):} p i (t j) + n i (t j), p i not (t j): Set _i need the number of documents that do not contain the keyword t _{j in, n i not (t j)} : Set i unnecessary number of documents that do not contain the keyword t _{j in, s i not (t j)} : p i not (t j) + N _i not (t _j ), h (a, b, c):-{a / c · log ₂ (a / c) + b / c · log ₂ (b / c)}

The information amounts H and H (t _j ) are expressed by the following equations (2) and (3), respectively.

[0011]

(Equation 1)

Next, a keyword t that can maximize the value of the mutual information I (t _k ) from a plurality of keywords t _j
k is selected and set as a search keyword (step S
205). This mutual information I (t _k ) has a positive finite value (>
In the case of 0) (step S206), the search keyword t _k
Set _i 'a document set consisting of number of documents containing the flag of the search keyword set of documents consisting of number of documents that do not contain t _k Set _i "divided as, for each document set divided" undivided " Turn on (step S207)
To S210). i ′, i ″ are already document sets Set _i ′,
If Set _i ″ does not exist, any value may be used.On the other hand, if the mutual information I (t _k ) is a zero value (= 0), the document set is not divided (step S206).

Thereafter, the flag of “undivided” of the set Set _i is turned off (step S211). If there is a document set with the “undivided” flag on, the process returns to step S103 (step S212: Yes), and the process is repeated until there is no document set with the “undivided” flag on. Then, the process ends when the “undivided” flag is turned off for all the document sets (step S212:
No).

The above-mentioned algorithm "ID3" can be replaced with, for example, a known algorithm such as "C4.5". The algorithm of “C4.5” is described in “C4.5 Programs for Machine Learning”.
(Quinlan, JR, published by Morgan Kaufmann Publishers).

FIG. 5 is a diagram showing a process in which one document set is divided into a plurality of document sets and the search formula is created by trial and error in the search formula creating apparatus 30. Hereinafter, an outline of a conventional search formula creation method will be described with reference to FIG. First, from the initial document set Set ₀ generated by the keyword extraction unit 32 (see reference numeral 32B), a keyword having the maximum mutual information is determined based on the decision tree learning algorithm “ID3”, and this is set as a search keyword. . Here, it is assumed that the search keyword “Telehodai” has been determined.

The document set division unit 33 uses the search keyword “Telehodai” to generate an initial document set Set0.
Is a set of required documents Set ₁ including the search keyword.
And a set Set ₂ of necessary documents and unnecessary documents that are not included. The document set Set ₁ cannot be further divided. On the other hand, the document set Set ₂ can be further divided. Therefore, a search keyword “discount” that maximizes the mutual information in the document set Set ₂ is determined,
A document set Set ₂ is set by the search keyword, and a set Set _{3 of} unnecessary documents not including the search keyword “discount”.
And a set Set ₄ of necessary and unnecessary documents to be included.

Since the document set Set ₄ can be further divided, a keyword “telephone” having the maximum mutual information in the document set Set ₄ is determined as a search keyword, and a set Set of necessary documents including the search keyword is set.
₅ and a set Set _{6 of} documents not included. Since the document sets Set ₅ and Set ₆ cannot be further divided, the division processing ends.

Each of the search keywords "tele-hodai", "discount", and "telephone" determined in the above-described division processing.
Are sequentially stored in a storage unit (not shown), and are passed to the search formula creation unit 34 when the division processing is completed. In the search formula creation unit 34, each search keyword that is the result of the document set division unit 33 is defined by the logical operators "and", "or", "no".
The search expression "query" is created by combining with "t". FIG.
Is an example of a search formula output from the search formula creating unit.

[0019]

By the way, in the above-mentioned conventional retrieval formula creation device 30, the word group extracted based on the morphological analysis processing is used as the attribute of the document. Therefore, for example, when a specific meaning is recalled for a phrase including a plurality of words due to the morphological analysis (hereinafter, referred to as a phrase).
Even if such a phrase is called a compound word), it may be divided into individual constituent words, or one word may be erroneously divided into a plurality of words, which is useful as a search keyword. There was a problem that the property was impaired. This will be described with an example in which three documents having the following contents are input. Document number 1 (necessary document) Facility installation fee is required to set up the telephone ... Document number 2 (unnecessary document) Payment of the payment of the competition facility installation was prompted by telephone ... Document number 3 (necessary) Document) Telephone installation does not require facility contribution

In this example, a set of documents having contents as shown in FIG. 6 is obtained from the keyword extracting unit 32. In this case, a compound word such as “facility contribution” exists in the documents of document numbers 1 and 3, but the words obtained by the morphological analysis are “facility” and “payment”. An appropriate search formula cannot be obtained just by combining them. Further, in this example, since the extracted words are included in all the documents, it is difficult to determine a search keyword and quickly create a search formula for distinguishing necessary documents and unnecessary documents.

SUMMARY OF THE INVENTION An object of the present invention is to improve the search accuracy in a document search or the like at a fixed value or more, and to quickly determine a search keyword and create a search expression without dividing a compound word. It is an object of the present invention to provide a search method creating method. Another object of the present invention is to provide a search formula creation device suitable for implementing the above search formula creation method.

[0022]

According to a first aspect of the present invention, there is provided a method for creating a search formula using a computer, wherein a specified document group input for creating the search formula includes a morpheme. Analyzing and extracting words in document units, specifying a group of words as unique compound words when words corresponding to noun phrases are continuous, and extracting the extracted individual words and the specified compound For a word, a mutual information amount represented by a difference between the information amount of the document group including and not including the word or the compound word and the total information amount of the specified document group is calculated, and the mutual information amount is set to the maximum. Deciding an individual word or compound word to be used as a search keyword, and creating a search formula having the determined search keyword as an element.

According to another aspect of the present invention, there is provided a search formula generating apparatus for generating a search formula used for document search of a document group including necessary documents of interest to a searcher and unnecessary documents of no interest. Word extracting means for extracting a word for each document from a specified document group to which specified information for identifying the required document or the unnecessary document has been added, and determining the part of speech of each extracted word, Compound word processing means for specifying a group of words as a unique compound word in the case where words corresponding to a phrase are continuous; discriminating information indicating whether each word and the compound word are included in the document; A document set generating means for generating a document set in which the assigned designation information is collected together with document identification information; and a document set including these words or compound words among the extracted words and the specified compound words. Individual words or compound words that maximize the mutual information represented by the difference between the information amount of the document group and the document group not included and the total information amount of the specified document group are determined as search keywords, and the determined search is performed. Document set dividing means for dividing one document set into a plurality of document sets by using a keyword, wherein the search keywords determined each time the division of the document set is repeated are combined with a logical expression to create the search expression It is characterized by doing.

[0024] In the retrieval formula creating apparatus, the compound word processing means combines the extracted words in a predetermined order based on a preset compound word creating criterion, and converts the obtained word group into a unique compound word. It may be configured to be specified as

The document set dividing means is configured to successively determine, as the search keyword, a word or a compound word that maximizes the mutual information based on, for example, a predetermined minimum description length principle.

[0026]

Embodiments of the present invention will be described below in detail. FIG. 1 is a functional block diagram showing a search formula creation device suitable for implementing the above search formula creation method. Components having the same functions as those of the conventional search formula creation device 30 described with reference to FIG. And a duplicate description is omitted. For convenience of explanation, it is assumed that a document group input to the present apparatus is given in advance with necessary / unnecessary designation information indicating whether the document is necessary or unnecessary for a user or the like.

The retrieval formula creation device 10 of the present embodiment includes a morphological analysis unit 31, a keyword extraction unit 32, a compound word processing unit 11, and a document set division formed by reading and executing a predetermined program by a computer device. Part 33,
It is provided with each function of the search formula creation unit 34. The above program is usually stored in an internal storage device or an external storage device of the computer device, and is read and executed as needed. However, a recording medium separable from the computer device, for example, a CD-ROM or FD Etc., or stored in a program server or the like connected to the computer device and the local network, and installed in the internal storage device or the external storage device at the time of use, and provided for execution at any time. Good.

The compound word processing unit 11 determines the part of speech of the word group extracted by the morphological analysis unit 31 and the keyword extraction unit 32, and specifies a unique compound word. The specification of the compound word is performed, for example, by combining a plurality of corresponding words when a word group corresponding to a noun phrase is continuously extracted as a result of the part-of-speech determination.
Alternatively, a predetermined compound word creation criterion is set by a system parameter or the like, and corresponding words are successively combined in a predetermined order based on the compound word creation criterion. In this case, various forms can be considered as a compound word creation criterion. For example, a combination procedure of words to be used as a compound word is set in advance, and words are combined in accordance with this procedure. . Alternatively, words corresponding to a predetermined number of noun phrases may be combined regardless of whether the noun phrases are continuous. The compound words specified in this way are used in document collection.

When this compound word is effective in distinguishing a necessary document from an unnecessary document in the document set dividing unit 33,
That is, when the mutual information amount described above increases, the compound word is determined as a search keyword, and is reflected in the search formula created by the search formula creating unit 34. If multiple words or compound words are suggested search terms,
For example, based on frequency information of individual words held in a document in advance, a word or compound word having a smaller number of documents, that is, a word having a lower appearance frequency is appropriately determined as a search keyword.

Next, a description will be given of a method for creating a retrieval formula using the retrieval formula producing device 10 having the above configuration with reference to FIG. Here, it is assumed that three documents having the following contents are input for convenience. Document number 1 (necessary document) Facility installation fee is required to set up the telephone ... Document number 2 (unnecessary document) Payment of the payment of the competition facility installation was prompted by telephone ... Document number 3 (necessary) Document) Telephone installation does not require facility contribution

When the above document is input (step S10)
1) The search formula creation device 10 performs morphological analysis on the input document by the morphological analysis unit 31 and extracts a morpheme group for each document (step S102). Reference numeral 31A in FIG. 1 shows the result of the morphological analysis. The keyword extraction unit 32 extracts words that can be used as keywords based on the morpheme group of each document (step S103). Also,
If the noun phrases are consecutive (step S104: Yes),
The corresponding words are combined into a compound word (step S10
5). In this example, a combination of words extracted consecutively, such as “facility” and “contribution”, is specified as a compound word “facility contribution”. The keyword extraction unit 32 also stores a word in each document, discrimination information indicating whether or not the compound word generated in step S105 appears, and identification information indicating whether the document is a required document or an unnecessary document. The documents are collected together with document identifiers such as numbers to create a document set (step S106). Reference numeral 32A in FIG. 1 exemplifies the contents of a document set output from the keyword extracting unit 32. In this document set, the compound term "facility contribution"
It can be seen that is an effective keyword for distinguishing a necessary document from an unnecessary document.

Thereafter, the document set division unit 33 divides the document set created in step S106 stepwise based on the above-mentioned mutual information (step S1).
07), the search formula creation unit 34 creates a search formula based on the search keywords sequentially determined in the process of the division process (step S108). Reference numeral 34A in FIG. 1 exemplifies a search formula output from the search formula creation unit 34. This example shows that a search formula is created using only the compound word “facility contribution” determined as a search keyword. If there is an input of a new document for which a search expression is to be created, the process returns to step S101 (step S10).
9: Yes), the above series of processing is repeated. If there is no other input document, the process ends (step S109: N
o).

As described above, the retrieval formula creating apparatus 10 of the present embodiment specifies a unique compound word by combining words corresponding to noun phrases that are successively extracted, and generates a unique compound word by creating a document set and a search keyword. , It is possible to avoid a problem caused by morphological analysis in which a specific word is divided into a plurality of words, and the usefulness of a keyword as a search word is guaranteed.

In addition, an appropriate search formula that could not be obtained by simply combining divided words can be easily obtained by using a compound word as a search formula, and required documents and unnecessary documents can be obtained by using fewer search keywords. And can be distinguished. From this, it is obtained based on the above-mentioned assumption of the MDL principle that "a search formula that distinguishes necessary documents from unnecessary documents with fewer search words is a search formula that accurately expresses the intention of a human (searcher)." The retrieval formula is more accurate than the conventional method. Furthermore, by applying the created search formula to the search processing, a result with high search accuracy can be obtained.

[0035]

As is apparent from the above description, according to the present invention, it is possible to create a search formula with fewer search keywords without dividing the entire document database. Further, by using the retrieval formula obtained by the present invention, it is possible to maintain the retrieval accuracy of the document at a certain value or more, and there is also an effect that the efficiency of the retrieval processing is greatly improved.

[Brief description of the drawings]

FIG. 1 is a functional block diagram illustrating an embodiment of a search formula creation device according to an embodiment of the present invention.

FIG. 2 is a processing procedure diagram in the search expression creating apparatus of the embodiment.

FIG. 3 is a functional block diagram of a conventional search expression creation device.

FIG. 4 is an explanatory diagram of a processing procedure in a conventional search expression creation device.

FIG. 5 is a schematic diagram of information obtained in a conventional dividing process.

FIG. 6 shows a document set creation result corresponding to a conventional input document group.

[Explanation of symbols]

 10, 30 retrieval formula creation device 11 compound word processing unit 31 morphological analysis unit 32 keyword extraction unit 33 document set division unit 34 search formula creation unit

Claims (6)

[Claims] 1. A morphological analysis is performed on a designated document group input for creating a retrieval formula to extract words in document units, and a word group in the case where words corresponding to a noun phrase are continuous is uniquely compounded. Specifying the word as a word, and for the extracted individual words and the specified compound word, the information amount of the document group including and not including the word or compound word and the total information amount of the specified document group Calculating the mutual information represented by the difference between the two, and determining individual words or compound words that maximize the mutual information as search keywords, and creating a search formula using the determined search keywords as elements. A method for creating a retrieval formula using a computer device. 2. The designated document group is a document group to which designated information for identifying a necessary document of interest or an unnecessary document not of interest to a searcher is added, and the designated document assigned to each document is provided. 2. The method according to claim 1, wherein information is reflected in the mutual information amount. 3. The method according to claim 1, wherein the compound word is obtained by combining word groups extracted based on a predetermined compound word creation criterion in a predetermined order. 4. An apparatus for creating a retrieval formula for use in document retrieval of a group of documents including a required document of interest and an unnecessary document of no interest to a searcher, wherein a specification for identifying the required document or the unnecessary document is provided. A word extraction means for extracting words for each document from a specified document group to which information is added; determining a part of speech of each extracted word; and, when words corresponding to a noun phrase are continuous, uniquely identify the word group. A compound word processing means for specifying a compound word, a document set in which individual words, discrimination information indicating whether or not the compound word is included in the document, and the designation information given to the document together with document identification information A set of document generating means for generating, from among the extracted words and the specified compound words, the information amount of a document group including and not including the words or compound words and the specified document group Individual words or compound words that maximize the mutual information represented by the difference from the total information amount are determined as search keywords, and one document set is divided into a plurality of document sets using the determined search keywords. A search formula creating apparatus, comprising: a document set dividing unit, wherein the search formula is created by combining search keywords determined every time the document set is divided by a logical formula. 5. An apparatus for creating a retrieval formula for use in document retrieval of a group of documents including a required document of interest and an unnecessary document of no interest to a searcher, wherein a specification for identifying the required document or the unnecessary document is provided. Word extraction means for extracting words for each document from a specified document group to which information has been added; and combining the extracted words in a predetermined order based on a preset compound word creation criterion. Compound word processing means for specifying a unique compound word, identification information indicating whether each word and the compound word are included in the document, and the designation information given to the document are collected together with document identification information. A document set generation unit for generating a document set; and, among the extracted words and the identified compound words, information amounts of documents including and not including the words or the compound words, Individual words or compound words that maximize the mutual information represented by the difference from the total information amount of the specified document group are determined as search keywords, and one document set is divided into multiple documents using the determined search keywords. A search formula creating apparatus, comprising: a document set dividing unit that divides a document into sets, wherein the search keywords determined each time the document set is divided repeatedly are combined with a logical formula to create the search formula. 6. The document set dividing means is configured to sequentially determine a word or compound word that maximizes the mutual information amount as the search keyword based on a predetermined minimum description length principle. The retrieval formula creation device according to claim 3, wherein