JPH10320421A - Document search method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to retrieve a document at a high speed. SOLUTION: This document retrieving method consists of a word dividing means which divides a text into words, an index file operating means which saves and retrieves results obtained by an inputted registered document into words, a retrieval request analyzing means which analyzes an inputted retrieval request, and a document retrieving means which specifies a document complying with the retrieval request by using the above-mentioned index file operating means. In this case, the word dividing means divides the text into the words according to how easily a word border of an array of two characters or characters having longer length than that is formed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a document search method capable of searching a document at high speed in a document management system, an image management system, a database management system, and the like.

[0002]

2. Description of the Related Art Generally, in this type of document search method, an index file for search is generated by dividing text into words.
In Japanese Unexamined Patent Publication No. H07-85033 and Japanese Unexamined Patent Publication No. Hei 7-85033, morphological analysis of text is performed as a means for generating an index file for searching by dividing the text into words (hereinafter, this method is referred to as "morphological analysis method"). Is called).

[0003]

The conventional morphological analysis method has the following three problems.

(1) In order to perform morphological analysis, it is necessary to prepare a large-scale word dictionary in advance. Therefore, creating a word dictionary requires labor. Further, when a new word not found in the dictionary is generated, the new word needs to be added in order, so that there is a problem that the maintenance cost becomes enormous.

[0005] (2) Morphological analysis involves errors. If there is an error in the morphological analysis, a document that should be retrieved is not originally retrieved or a document that does not need to be retrieved is retrieved, so that a retrieval result desired by the user cannot be obtained. For example, if "Machida-shi, Tokyo" can be analyzed as "Tokyo / Tokyo / Machida / City", a document containing this can be
It is possible to search with the search term "Tokyo", but if this is incorrectly analyzed as "Tokyo / Kyoto / Machida / City", it cannot be searched with the search term "Tokyo" .

(3) If all the words resulting from the morphological analysis are recorded in the index, the index file becomes large. Therefore, it can be said that words that are not clearly used during the search should be excluded.

[0007]

According to the first aspect of the present invention,
Word division means for dividing text into words, index file operation means for storing and searching the results of word division of an input registered document, search request analysis means for analyzing input search requests, and analysis of search requests In a document search method including a document search unit that specifies a document that matches a search request using the index file operation unit according to a result, the word division unit may detect a word boundary between two or more characters. The text is divided into words based on the likelihood of becoming. Therefore, it is possible to completely automatically create a word boundary of a character sequence having a length of two or more characters, so that it is not necessary to prepare a huge word dictionary, Since a plurality of cutting methods can be realized, it is easy to obtain a search result desired by the user. Furthermore, the size of the index file can be avoided by taking into account the nature of the words when dividing by the word dividing means and not cutting out those that are not used for search as words.

According to a second aspect of the present invention, there is provided the word dividing means of the second aspect.
It is characterized in that the likelihood of becoming a word boundary in a character sequence of characters or longer is taken as information of a combination of character types. Therefore, the search can be performed with a very small amount of data.

The invention according to claim 3 is characterized in that, during the analysis of the search request by the search request analysis means, the division processing is applied to the word in the search request using the word division means. I do. Therefore, search omission can be effectively prevented.

According to a fourth aspect of the present invention, there is provided the word dividing means of the second aspect.
It is characterized in that the likelihood of becoming a word boundary of a character sequence of characters or longer is a word boundary evaluation value by a numerical value. Therefore, the degree of search omission can be adjusted by changing the division threshold according to the target data or the user's intention.

According to a fifth aspect of the present invention, in the result obtained by dividing the text by the word dividing means, if the word boundary evaluation value between adjacent words is equal to or less than a certain threshold, those adjacent words are separated. It is characterized in that a combination is generated as a division result. Therefore, it is possible to effectively prevent erroneous searches and missed searches.

According to a sixth aspect of the present invention, in the result obtained by dividing the text by the word dividing means, if the word boundary evaluation value between consecutive words is equal to or less than a certain threshold, those consecutive words are separated. It is characterized in that a combination is generated as a division result. Therefore, the combination of all combinations of index words including the word boundary evaluation value equal to or less than the combination threshold can also be used as the index word, so that the possibility of erroneous search can be greatly reduced.

According to a seventh aspect of the present invention, the word dividing means comprises:
It is characterized in that unnecessary elements are not extracted using an unnecessary character table of characters not used as an index. Therefore, the characters not used as the index are excluded from the target of the division processing by using the unnecessary character table, so that the index file can be prevented from being enlarged.

According to an eighth aspect of the present invention, the word dividing means comprises:
It is characterized in that unnecessary elements are not extracted using a table of non-prefixed characters. Therefore, by using a table of non-prefixed characters for non-prefixed characters, it is possible to determine that such a character is not a word boundary when it comes at the beginning of a word boundary. Can be

According to a ninth aspect of the present invention, the word dividing means comprises:
It is characterized in that words having a length equal to or longer than a preset upper limit value are not extracted. Therefore, there is basically no limit on the length of a word cut out by the word division means.
Some words have very long words, and registering all of them may cause an increase in the size of the index file. However, since an upper limit value is set for the word length, it is possible to avoid an increase in the size of the index file. .

According to a tenth aspect of the present invention, the word dividing means divides a word having a length equal to or greater than a predetermined upper limit into a plurality of character strings having a length equal to the upper limit included in the word. It is characterized by extracting. Therefore, a word having a length exceeding the upper limit value is extracted by being divided into a character string having a length equal to the upper limit value, so that it is possible to avoid a case where a word having a length equal to or longer than the upper limit value cannot be searched. .

The invention according to claim 11 is characterized in that the word dividing means can set an upper limit for each character type.
Therefore, in Japanese, there is a large difference in the distribution of word lengths depending on the character types that make up the word, but since the upper limit of the word length is set in consideration of the character type, search omission can be prevented and indexing can be prevented. The file can be prevented from becoming large.

According to a twelfth aspect of the present invention, the word dividing means sets the plurality of predetermined non-divided character strings on the basis of the tendency to form a word boundary of a sequence of two or more characters. It is characterized in that it is extracted as it is without performing the process of dividing it into words. Therefore, if word division is performed uniformly, words may not be correctly divided.However, a non-divided character string is prepared in advance, and a character string that should not be divided by excluding it from the division target is unnecessary. The division accuracy can be improved without division.

According to a thirteenth aspect of the present invention, when a certain non-divided character string includes another non-divided character string with respect to a plurality of predetermined non-divided character strings,
Only the included non-divided character strings are extracted as words, and the included non-divided character strings are not extracted as words. Therefore, even when the non-divided character string includes an inclusive relation, the character string can be correctly divided and the division accuracy is improved.

According to a fourteenth aspect of the present invention, when a certain non-divided character string includes another non-divided character string with respect to a plurality of predetermined non-divided character strings,
It is characterized in that both included non-divided character strings and included non-divided character strings are extracted as words. Therefore, even when a non-divided character string includes an inclusive relation, the character string can be correctly divided without causing an erroneous search, and the division accuracy is improved.

[0021] According to a fifteenth aspect of the present invention, the word dividing means sets a plurality of unnecessary character strings based on a word boundary based on the tendency to be a word boundary of an arrangement of two or more characters. It is characterized by not performing the process of dividing into words and not extracting as words. Therefore, unnecessary character strings that are rarely used as search words such as conjunctions are excluded from division and are not extracted as words. The size can be reduced.

[0022]

FIG. 1 shows a first embodiment of the present invention.
It will be described based on. First, FIG. 1 shows an outline of the process of the document search method, and the word dividing means has a function of dividing a text into words. The index file operation means has a function of storing and retrieving a result obtained by dividing the input registered document into words on a secondary storage device as a file.
The search request analysis means has a function of analyzing the input search request. The search processing unit has a function of specifying a document that matches the search request by using the index file operation unit according to the analysis result of the search request.

At the time of document registration, a registered document (accepted by a registered document input unit, not shown) is cut into a group of words by a word dividing unit, and is recorded in an index file by an index file operating unit. As a format of the index file, an inverted file in which a document including the word is recorded for each word may be used.

At the time of document search, a search request (accepted by a search request input unit (not shown))
Is analyzed by the search request analysis means, and based on the result, the search processing means specifies a document including the search word through the index file operation means, and generates a search result. The search request has a format in which the search words are combined by the logical operators and and or, but may also accept other proximity operators and the like. However, in order to be able to process the proximity operator, it is necessary not only to simply record the document containing the word for each word, but also to record the appearance position of each document in the index file.

The word dividing means divides the text into words on the basis of the tendency of the arrangement of two or more characters to form word boundaries. Information on the ease of word boundaries is
It is described in a format as shown in Table 1.

[0026]

[Table 1]

In other words, this expresses the likelihood of becoming a word boundary for a two-character set, and "○" indicates that the two-character set is likely to be a word boundary, and "x" indicates that it is unlikely to be. . The word dividing means divides the text using the two-character set marked with “○” in Table 1 as a word break. For example, “Machida-shi, Tokyo” is divided into “Tokyo / Tokyo / Machida / City”. Actually, in order to reduce the amount of data to be prepared, a table in which only one of two character sets of “O” or “X” is recorded is used. Although the above description is directed to a two-character set, it goes without saying that the present invention can be similarly applied to a three-character set or more.

A second embodiment of the present invention will be described with reference to Table 2. In the above-described first embodiment, it is necessary to represent the likelihood of becoming a word boundary in binary form of “○” / “×” and hold it as a table. In Japanese, the number of different characters is huge, about 7,000, so the number of entries in this table is extremely large, 7,000 x 7,000 = 49,000,000. Even if a table that records only one of two character sets of “O” or “X” is used, the number is still large.

[0029]

[Table 2]

In the present embodiment, character types such as kanji, hiragana, katakana, and alphanumeric characters are used to reduce the size of the table as shown in Table 1 described above. Since the change in the character type (for example, the point at which the character changes from kanji to hiragana) is often a word break, a table such as Table 2 that describes whether or not the character break is a word break for each combination of character types is prepared. Here, the meanings of “” ”and“ × ”are the same as in the case of Table 1 described above, but“ ▽ ”refers to a table describing the likelihood of a two-character set becoming a word boundary. Means that. For example, if "Tokyo in the 21st century" is "2
It can be seen from Table 2 that “1” is not a word boundary and that “East” is a word boundary from Table 2. For other portions, the word boundaries are determined using the table shown in Table 1 (in this case, the requirements shown in Table 2 are deleted). In this way, the amount of data to be recorded in the table for the two-character set can be reduced.

Next, a third embodiment of the present invention will be described with reference to FIG. The difference between FIG. 2 and FIG. 1 is that the search request analysis means analyzes the search request in cooperation with the word division means (in FIG. 2, the search request analysis means and the word division means are connected). ing). The reason why the search request analyzing means and the word dividing means are linked in this way is that, in the method shown in FIG. 1, if a compound word is included in the search request, it is finely divided at the time of document registration. This is because the compound word in the search request cannot be searched, and search omission occurs. Therefore, in the method shown in FIG. 2, the character string in the search request is not used as it is as a search word, but is passed to word division means and divided into words. When the division result is two or more words, an analysis result in a form in which these words are combined by an and operator is generated and passed to the search processing means.

For example, if the search request is "Governor of Tokyo", the search request is passed to the search processing means as is "Governor of Tokyo" in the method shown in FIG. 1, but the embodiment of the present invention shown in FIG. According to the above, "Governor of Tokyo" is divided by the word dividing means and passed to the search processing means. That is, if it is divided into "Tokyo / Tokyo / Governor" by the word dividing means,
It will be processed in the same way as the search request "Tokyo and Tokyo and Governor". As a result, search omission can be prevented.

Alternatively, if the index file includes a word appearance position, the division result by the word division means is 2
When the number of words becomes more than one, a method of generating an analysis result in a form in which these words are combined by a proximity operator and passing the result to a search processing unit is also conceivable. For example, "Governor of Tokyo" becomes "Tokyo dis
t (2) Governor dist (1) Governor ". Here, “dist (n)” represents a proximity operator that makes a search result a document in which the preceding and following words appear at a distance of n characters.

That is, when the search request is “Governor of Tokyo”, the method using “and” searches for a document containing “Governor of Okinawa visited Tokyo”. Then, it is possible to avoid detecting such a document.

Next, a fourth embodiment of the present invention is shown in Table 3.
It will be described based on. In each of the above embodiments,
The probability of becoming a word boundary was represented by binary values of “O” / “X”. However, in practice, the likelihood of becoming a word boundary cannot be determined in a binary manner, so in the present embodiment, as shown in the table of Table 3, a numerical value (this is called a word boundary evaluation value) is used. Express.

[0036]

[Table 3]

Thus, the word division means prepares a division threshold, and sets a point having a word boundary evaluation value larger than the value as a division point. For example, assuming that the division threshold is 0.20, if the search request is “Machida-shi, Tokyo”, “Tokyo / Tokyo / Machida / City” is obtained as in the first embodiment. On the other hand, if the division threshold is set to a small value of 0.10, the image is divided into “East / Kyoto / Tokyo / Machida / City”.

The advantage of introducing such a word boundary evaluation value is that the division threshold can be changed in accordance with the target data or the user's intention, and the degree of search omission can be adjusted. By reducing the division threshold, it is possible to reduce a search omission that fails to search for a document that must be searched, even if erroneous searches for documents that should not be searched are increased. Conversely, by increasing the division threshold, erroneous searches can be reduced even if search omissions increase. Since the extent to which search omission / erroneous search can be tolerated differs for each user, it is advantageous to be able to adjust the division threshold by setting the division threshold, whereby a user-friendly search system can be constructed. Of course, expressing the likelihood of becoming a word boundary with an evaluation value can be applied to the expression of the likelihood between character types shown in Table 1.

The word boundary evaluation values shown in Table 3 can be easily obtained from a corpus obtained by morphological analysis. The following is an example of a morphologically analyzed corpus, in which "|" indicates a word boundary.

Tokyo | capital | governor | is | Okinawa | prefecture | governor | and |

Using this corpus, the total number of appearances and the number of occurrences across words are counted for all two-character groups. In the example above, the two character set "Kyoto" is 2
Appears twice and appears once across words. Therefore, the word boundary evaluation value for "Kyoto" is 1/2
= 0.5. On the other hand, since “Tokyo” appears once and does not appear across words, the word boundary evaluation value is 0 /
1 = 0.0. The morphologically analyzed corpus is provided by EDR (Electronic Dictionary Research Institute), RWCP (New Generation Information Processing Organization), or the like, and is easily available.

Next, a fifth embodiment of the present invention will be described. In the above-described fourth embodiment, since the word division is performed based on a simple index called a word boundary evaluation value, the division result may be incorrect. When the above-mentioned word boundary evaluation value table (Table 3) is used, "Governor of Kyoto Prefecture"
Since the word boundary evaluation value (0.15) for "Tokyo" is smaller than the word boundary evaluation value (0.30) for "Kyoto", the sentence "Kyoto / Future / Governor" can be used regardless of the division threshold. Can not be divided into In fact, the division threshold is 0.40
If so, it will be "Kyoto / Governor". If the search word is "Kyoto", and if the division threshold is 0.30 or less, "Kyo and Miyako"
An erroneous search occurred because the search was performed as
In the above case, the search is performed as “Kyoto”, but the division is “Governor of Kyoto Prefecture” or “Kyoto / Governor”. I do.

In order to solve such a problem, in the present embodiment, when registering a document, the word division processing is executed as follows. Among the divided index words, adjacent search words satisfying a specific condition are recombined, and the combined one is also used as an index word. Here, the specific condition is that a word boundary evaluation value of a two-character group that straddles an adjacent search word is equal to or less than a newly introduced combination threshold. For example, assuming that the division threshold is 0.10 and the newly introduced combination threshold is 0.40, "Kyoto Governor" is divided into "Kyoto / Tokyo / Governor / Governor" by the division threshold, and "Kyoto", "Tokyo" Is generated by recombination. Therefore, “Kyoto”, “Tokyo”, “Fu”, “Governor”, “Kyoto”, and “Tokyo” are finally extracted as index words from “Governor of Kyoto Prefecture”.

On the other hand, at the time of retrieval, the following processing is performed.
At the time of retrieval, among the index words divided based on the division threshold, those used as elements of recombination are not extracted, and only the combined one is used as a search word. If the search request is "Kyoto", "Kyoto" and "Miyako" are combined as "Kyoto" and are not used as search words, but only "Kyoto" is used. Therefore, it is possible to prevent erroneous searches and search omissions that occur in the case of the above-described fourth embodiment.

Next, a sixth embodiment of the present invention will be described. In the above fifth embodiment, "Kyoto Prefecture"
Is searched as "Kyoto and prefecture", so "Kyoto" and "Prefecture" appear at different positions, and a document that does not include "Kyoto prefecture" itself is erroneously searched. Sometimes. As described in the third embodiment, the possibility of an erroneous search is lower than that of the processing performed by "Tokyo and Tokyo and Governor", but there is still the possibility of an erroneous search.

Therefore, in the present embodiment, the combination of all combinations of index words including a word boundary evaluation value equal to or less than the combination threshold value is also used as an index word. According to this method,
In addition to “Kyo”, “Tokyo”, “Fu”, “Governor”, “Kyoto” and “Tokyo” extracted from “Governor of Kyoto” in the fifth embodiment, the compound word “Kyoto” is also extracted. It is extracted as an index word. As a result, the possibility of an erroneous search can be significantly reduced.

The seventh embodiment of the present invention will be described with reference to Table 4. In the first embodiment, as described above, the likelihood of becoming a word boundary must be represented as a binary value of “○” / “x” and held as a table. In Japanese, the number of different characters is huge, about 7,000, so the number of entries in this table is 7,000 x 7,000 = 49,00
It will be a huge number of 0,000. Even if a table in which only one of two character sets of “O” or “X” is recorded is used, the number is still large. In the present embodiment, in order to reduce the size of the table as shown in Table 1 above, an unnecessary character table relating to characters not used as an index as shown in Table 4 is used.

[0048]

[Table 4]

For example, characters such as "," (reading), "." (Punctuation), "," (comma), "("")" (parentheses) do not form words and need to be recorded in the index. Nor. Therefore, such unnecessary characters are recorded in another unnecessary character table as exemplified in Table 4, and are excluded from the target of the division processing by the word division means. Since it is not necessary to record characters included in the unnecessary character table in the divided table, the size of the divided table can be reduced.

For example, "Central processing unit (CPU)
Is attached. Is given to the word division means as a division target. At this time, the “central processing unit”, “CPU”, and “
The division processing is performed on the four parts of “attach” using the method of the first embodiment described above. That is, it is not necessary to determine whether or not a two-character set including unnecessary characters such as “place (”, “(C”, “U)” is a word boundary by referencing a division table. Since the processing is simplified, the processing can be speeded up accordingly, and unnecessary characters in the text are not extracted as words, so that the index file can be downsized.

The unnecessary characters included in the unnecessary character table are, of course, not limited to the exemplified five characters.

An eighth embodiment of the present invention will be described with reference to Table 5. In the seventh embodiment described above, by recording characters that do not form words in the unnecessary character table, the size of the divided table, the speed of processing, and the size of the index file are reduced. However, in Japanese, there are characters that constitute a word but do not become the head (prefix) of the word, such as a prompt sound such as "tsu", a repellent sound such as "n", and a relentless sound such as "ya". Therefore, in the present embodiment, a non-prefixed character table for characters that do not have a prefix as shown in Table 5 is used.

[0053]

[Table 5]

If a character that does not become a prefix is recorded in another non-prefixed character table, if such a character comes to the head of a word boundary, it can be determined that the character is not a word boundary without referring to the division table. Therefore, a two-character set in which such a character is at the beginning of a word boundary can be excluded from the divided table, and the divided table can be downsized. In the non-prefixed character table shown in Table 5, ten characters such as "tsu" are prepared as non-prefixed characters, but it is needless to say that the characters are not limited to the exemplified characters.

As a result, for example, when “track” is a target to be divided by the word dividing means, the method of the first embodiment uses the division table for three two-character sets “tra”, “ra”, and “c”. It is necessary to determine whether or not a word boundary is referred to, but according to the method of the present embodiment,
It can be determined from the fact that the character "tsu" is a non-prefix character because the character "tsu" is a break. Therefore, it is only necessary to refer to the division table for the two “tracks” and “tracks”. Since the dividing process is thus simplified, the process can be speeded up.

A ninth embodiment of the present invention will be described. In the first embodiment described above, there is no restriction on the length of the words cut out by the word dividing means. Therefore, when the method as in the fifth embodiment is used, the compound name (for example, “dichlorobenzenebiphenyl”) is used.
A long word such as is extracted without being divided into its constituent words (“dichloro”, “benzene”, “biphenyl”). However, registering all such words undesirably increases the number of different words registered in the index file explosively and increases the size of the index file.

Therefore, in this embodiment, such a situation is prevented from occurring by setting an upper limit value for the length of a word to be divided by the word dividing means. That is, the word division means does not extract words having a length equal to or longer than the preset upper limit value. For example, if the upper limit is set to 10 (characters), the word “dichlorobenzenebiphenyl” (14
Characters) will not be registered in the index file. As a result,
The index file can be prevented from becoming large.

Next, a tenth embodiment of the present invention will be described. According to the ninth embodiment, it is possible to avoid an increase in the size of an index file by setting an upper limit value for the length of a word to be extracted. However, if the value exceeds the upper limit, it is simply not registered, so that there may be a case where a word longer than the upper limit cannot be searched. Therefore, in the present embodiment, a word having a length exceeding the upper limit value is extracted by being divided into a plurality of character strings having a length equal to the upper limit value.

Here, as a method of dividing a long word,
For example, there is a method of decomposing a word into a character string equal to the upper limit of the minimum number covering the entire word. Assuming that the upper limit is 10 (characters), words with 20 characters or less are divided into two strings: the first 10 characters and the last 10 characters.
Words of 30 characters or less are divided into three parts: a character string of 10 characters from the beginning, a character string of 10 characters from the 11th character from the beginning, and a character string of 10 characters from the end. In the case of a longer word, it may be similarly divided.

When this method is used, for example, the word "dichlorobenzene biphenyl" in the above example is divided into two character strings of "dichlorobenzene biff" and "benzene biphenyl", and the extracted result is obtained. Therefore, according to the present embodiment, it is possible to suppress an increase in the number depending on the setting of the upper limit value, and to search for a word having a length exceeding the upper limit value.

An eleventh embodiment of the present invention will be described.
In Japanese, there is a great difference in the distribution of word lengths depending on the character types that make up the word. For example, there are compound words that are relatively short for Kanji, but quite long for Katakana. Therefore, when an upper limit is set for the length of a word to be extracted as in the ninth or tenth embodiment, a problem occurs unless the character type is considered. That is, if the upper limit is set in accordance with the kanji, the number of Katakana words that are determined to be equal to or higher than the upper limit increases, which may cause a search omission or the like. On the other hand, if the upper limit value is set in accordance with Katakana, almost no kanji word is determined to be equal to or more than the upper limit value, which is insufficient for suppressing an increase in the size of the index file.

Therefore, in the present embodiment, the upper limit value can be freely set for each character type. For example, the upper limit is 6 (characters) for kanji, and 10 (characters) for katakana. According to this method, an increase in the size of the index file can be avoided without causing search omission.

A twelfth embodiment of the present invention will be described with reference to Table 6. In the method of the above-described first embodiment, basically, division is performed based on a simple principle, so that words may not be correctly divided in some cases. For example, in order to divide “in” into “in” and “in”, it is necessary to record “in” in the division table as being divided. In this case, however, “smell” (smell) Will also be split.

Therefore, in the present embodiment, the word character string to be excluded from the object of division is recorded as a non-divided character string in another table as shown in Table 6, and when the word division means performs the division processing, If there is a part that matches this non-divided character string in the target, that part is excluded from the division target.

[0065]

[Table 6]

As a result, it is possible to avoid the inconvenience of unnecessarily dividing a character string that should not be divided, such as “odor”, and to improve the accuracy of division by the word dividing means. If the table of non-divided character strings shown in Table 6 is used, for example, if “Kusaya no smell and taste” is to be divided, “Kusaya” and “smell” are recorded in the non-divided character string table. It is extracted as a word without referring to the table. Then, only the remaining "no" and "to taste" portions are subjected to the same division processing as in the case described above.

A thirteenth embodiment of the present invention will be described.
In the method of the twelfth embodiment, for example, when “smell” is recorded in the table of the non-divided character string,
The word "Hey" (nephew) cannot be extracted from the sentence "He has a nephew" (he has a nephew). In order to avoid such inconvenience, "Oi" may be recorded in a table of non-divided character strings. However, in this case "smell"
Contains "Oi", so in order to correctly process the sentence "I have him", only the included ("Oi") is extracted as a word and the included ("Oi") Smell ") need not be extracted as a word. In the present embodiment, by adopting such a method, even if a character string included in the non-divided character string table includes an inclusive relation, it can be correctly divided. That is,
The division accuracy is improved.

A fourteenth embodiment of the present invention will be described.
Even in the case of the method according to the thirteenth embodiment, it may not be sufficient. For example, when “Kusaya to smell and taste” is a processing target, only the included character string “smell” is extracted, and not “smell”. Therefore, if the search word is "smell", the search must be performed by separating one word such as "ni and sui". As a result, not only sentences in which “Ni” and “Oi” successively become “Odor”, but also documents in which “Ni” and “Oi” appear separately are searched.

In order to solve this problem, in the present embodiment, not only the included characters but also the included characters are extracted from the non-divided character string table. In other words, from "Kusaya and smell and taste", not only "smell" but also "smell"
Is also extracted, and when the search word is "odor", the search is performed as "odor". As a result, the erroneous search described above does not occur, and the search accuracy is improved.

The fifteenth embodiment of the present invention will be described with reference to Table 7. In document search, words such as conjunctions are rarely used as search words. However, the method of the first embodiment described above is wasteful because all the divided words are extracted. On the other hand, if such unnecessary words can be excluded and registered in the index file during document registration, the index file can be downsized. Thus, in the present embodiment, word character strings to be excluded such as “but”, “hence”, and “again” are recorded as unnecessary character strings in a table as shown in Table 7, and are excluded from the target of the division processing. .

[0071]

[Table 7]

Therefore, for example, if "but cake is delicious" is to be divided, "but" is recorded in the unnecessary character string table, and is excluded from the division processing. Then, only the remaining “, the cake is delicious” portion is subjected to the dividing process. As a result, words to be finally extracted are reduced, and the size of the index file can be reduced.

[0073]

According to the first aspect of the present invention, a word dividing means for dividing a text into words, an index file operating means for storing and retrieving a result obtained by dividing the inputted registered document into words, In a document search method comprising: a search request analysis unit for analyzing a search request; and a document search unit for specifying a document that matches the search request by using the index file operation unit in accordance with the analysis result of the search request. Text is divided into words based on the likelihood of a word line of a character sequence of characters or longer, so it is likely to be a word boundary of a character sequence of two or more characters. Because it can be created completely automatically, there is no need to prepare a huge word dictionary, and since multiple ways of cutting words can be realized, It has the effect of say it becomes easy to obtain a search result.

According to the second aspect of the present invention, the probability that a word boundary of a sequence of characters having two or more characters in the word dividing means is determined as information of a combination of character types, so that an extremely small data amount You can search with.

According to the third aspect of the present invention, during the analysis of the search request by the search request analysis means, the division processing is applied to the word in the search request using the word division means. Search omission can be effectively prevented.

According to the fourth aspect of the present invention, the likelihood of becoming a word boundary of a sequence of characters having a length of two or more characters of the word dividing means is set as a word boundary evaluation value by a numerical value.
By changing the division threshold according to the target data or the user's intention, the degree of search omission can be adjusted.

According to the invention described in claim 5, in the result obtained by dividing the text by the word dividing means, if the word boundary evaluation value between adjacent words is equal to or smaller than a certain threshold value, the adjacent word Are generated as a result of division, it is possible to effectively prevent erroneous searches and missed searches.

According to the invention described in claim 6, in the result obtained by dividing the text by the word dividing means, if the word boundary evaluation value between consecutive words is equal to or less than a certain threshold value, the continuous word Is generated as a segmentation result, so that the combination of all index terms including the word boundary evaluation value equal to or less than the combination threshold value can also be used as the index term, which may lead to erroneous search. Can be greatly reduced.

According to the seventh aspect of the present invention, the word division means does not use the unnecessary character table of characters not used as an index to extract unnecessary elements, so that the unnecessary character table is not used for characters not used as an index. Since the file is excluded from the target of the dividing process, it is possible to avoid an increase in the size of the index file and to speed up the dividing process.

According to the eighth aspect of the present invention, the word dividing means does not use a table of non-prefixed characters to extract unnecessary elements. Therefore, when the non-prefixed character comes to the head of a word boundary, It can be determined that it is not a boundary, and it is possible to avoid an increase in the size of the index file and to speed up the division processing.

According to the ninth aspect of the present invention, there is basically no limitation on the length of the words cut out by the word dividing means, and some words are very long. However, since the word division means does not extract words having a length equal to or larger than a preset upper limit, it is possible to avoid an increase in the size of the index file, to speed up the processing, and to perform the processing necessary for word division. The size of the table can be reduced.

According to the tenth aspect, the word dividing means divides a word having a length equal to or larger than a predetermined upper limit into a plurality of character strings having a length equal to the upper limit included in the word. In addition to the effect of the ninth aspect of the present invention, it is possible to avoid a situation in which a word having a length equal to or longer than the upper limit cannot be searched.

According to the eleventh aspect of the present invention, in Japanese, there is a great difference in the distribution of word lengths depending on the character types constituting words, but the word dividing means can set an upper limit for each character type. Since the upper limit of the word length is set in consideration of the character type, search omission can be prevented, and the index file can be prevented from becoming large.

According to the twelfth aspect of the present invention, if word division is performed uniformly, a word may not be correctly divided. However, the word dividing means may use two characters or less for a plurality of predetermined non-divided character strings. Because the process of dividing words into words based on the likelihood of becoming a word boundary of a sequence of characters longer than that is performed without extracting the words as they are,
Character strings that should not be divided by being excluded from the division target are not unnecessarily divided, and the division accuracy can be improved.

According to the thirteenth aspect of the present invention, the word dividing means sets a plurality of predetermined non-divided character strings
When a certain non-divided character string includes another non-divided character string, only the included non-divided character string is extracted as a word,
Since the included non-divided time sequence is not extracted as a word, in addition to the effect of the twelfth aspect, even when the non-divided character string includes an inclusive relation,
Division can be performed correctly, and division accuracy can be improved.

According to the fourteenth aspect of the present invention, the word dividing means sets a plurality of predetermined non-divided character strings
When a certain non-divided character string includes another non-divided character string, both the included non-divided character string and the included non-divided character string are extracted as words.
In addition to the effects of the invention described in 2, even when a non-divided character string includes an inclusive relation, the character string can be correctly divided without causing an erroneous search, and the division accuracy can be improved.

According to the fifteenth aspect of the present invention, for an unnecessary character string which is hardly used as a search word such as a conjunction, the word dividing means performs two-character processing for a plurality of predetermined unnecessary character strings. Alternatively, the process of dividing words into words based on the likelihood of becoming a word boundary of a sequence of characters having a longer length is not performed, and is not extracted as a word. Therefore, words to be finally extracted can be reduced, and the index file can be reduced in size.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating an outline of processing of a document search method according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing an outline of processing of a document search method according to a third embodiment of the present invention.

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification code FIG06F 15/413 310B

Claims (15)

[Claims] 1. A word dividing means for dividing a text into words, an index file operating means for storing and retrieving a result of word division of an input registered document, and a search request analyzing means for analyzing an input search request. A document search unit that specifies a document that matches the search request by using the index file operation unit according to the analysis result of the search request; A document search method, wherein a text is divided into words based on the likelihood of becoming a word boundary in a line. 2. The document search method according to claim 1, wherein the likelihood of becoming a word boundary of a sequence of characters having a length of two or more characters in the word dividing means is information on a combination of character types. 3. The method according to claim 1, wherein, during the analysis of the search request by the search request analysis means, the division processing is applied to the word in the search request by using the word division means. Document search method. 4. The document search method according to claim 1, wherein the likelihood of forming a word boundary between two or more characters in the word division means is a word boundary evaluation value by a numerical value. . 5. In the result obtained by dividing the text by the word dividing means, if the word boundary evaluation value between adjacent words is equal to or less than a certain threshold, the result of combining the adjacent words is used as the result of the division. 5. The document search method according to claim 4, wherein the document is generated as. 6. In a result obtained by dividing a text by the word dividing means, if a word boundary evaluation value between consecutive words is equal to or less than a certain threshold, a combination of those consecutive words is used as a result of the division. 5. The document search method according to claim 4, wherein the document is generated as. 7. The document search method according to claim 1, wherein the word division unit does not extract unnecessary elements using an unnecessary character table of characters not used as an index. 8. The document search method according to claim 1, wherein the word division means does not extract unnecessary elements using a table of non-prefixed characters. 9. The document search method according to claim 1, wherein the word dividing means does not extract words having a length equal to or longer than a preset upper limit value. 10. The word dividing means extracts a word having a length equal to or greater than a predetermined upper limit value by dividing the word into a plurality of character strings having a length equal to the upper limit value included in the word. 2. The document search method according to claim 1, wherein 11. The document search method according to claim 9, wherein the word dividing means can set an upper limit for each character type. 12. The word dividing means performs a process of dividing a plurality of predetermined non-divided character strings into words based on the tendency of the arrangement of characters having a length of two or more characters to be a word boundary. 2. The document search method according to claim 1, wherein the word is extracted as it is without performing it. 13. When a certain non-divided character string includes another non-divided character string with respect to a plurality of preset non-divided character strings, the word dividing means only includes the included non-divided character string. 13. The document search method according to claim 12, wherein the character string is extracted as a word, and the included non-divided character string is not extracted as a word. 14. A word dividing means, for a plurality of preset non-divided character strings, when a certain non-divided character string includes another non-divided character string, 13. The document search method according to claim 12, wherein both the included non-divided character strings are extracted as words. 15. The word dividing means performs a process of dividing a plurality of unnecessary character strings set in advance into words based on the likelihood of becoming a word boundary of an arrangement of two or more characters. 2. The document search method according to claim 1, wherein the document is not extracted as a word.