JPH10254894A - Similar document search device, similar document search method, and similar document search storage medium - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To improve the efficiency of automatic classification of document data by realizing highly accurate similarity calculation processing. A word is extracted from an arbitrary document as a comparison source and an arbitrary comparison destination document stored in an external storage device, and each extracted word is extracted from the comparison source document and the comparison destination document. Are stored individually in the word information storage buffers 4n and 4p together with the appearance position information in the document, and it is determined whether or not there is a specific word common to both the comparison source and the comparison destination documents. With respect to both the source and destination documents, a word corresponding to the word within a certain distance from the specific word is searched for, and the weight coefficient determining unit 4c searches for a word within a certain distance from the searched specific word. , A weight coefficient corresponding to the distance from the specific word in both documents is calculated for the word corresponding to the word in
b, based on the calculation result of the weight coefficient determining unit 4c,
The similarity is calculated according to the appearance position of the word of the comparison destination document with respect to the comparison source document.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a similar document search device, a similar document search method, and a similar document search storage medium.

[0002]

2. Description of the Related Art In recent years, a large amount of electronic document data has been distributed, and a device for searching a document database for a document similar to a designated document has been put to practical use for the purpose of automatic classification and the like. Is coming. In this apparatus, when searching for a similar document of a designated document (hereinafter referred to as a comparison source document), the similarity between the comparison source document and each document in the database (these are referred to as comparison destination documents) is determined. Calculate and output the one with the larger value as the search result.

[0003] As a method of calculating the similarity, a comparison source document,
A vector space method is generally used in which the similarity increases as the comparison destination document includes more common words.

In such a vector space method, first, words are extracted from both a comparison source document and a comparison destination document, and vector data is created from them by using the weight of each word as an element.

As the weight for each word, the frequency of occurrence of each word in the document is used. When the word exists, a positive value is set, and when the word does not exist, 0 is set.

In the vector space method, vector data of both words extracted from both the source document and the destination document are calculated, and then the inner product is calculated, and the value of the inner product is used as the similarity.

[0007]

As described above, according to the similarity calculation method using the existing vector space method, it is necessary to weight each word when calculating the similarity of a document. When both word A and word B exist in both the source document and the destination document, and word A and word B are adjacent or close to each other in the text of the source document, the destination document Even if the word A and the word B are adjacent or close to each other in the text of the above, the word A and the word B are similar to the case where the word A and the word B exist independently. Weighting has been done.

[0008] In such a plurality of documents, a set of words that appear adjacent or close to each other is a major feature that determines the content of the document. Is not performed, and only the uniformized appearance frequency is used as a weighting element. As a result, it becomes difficult to obtain sufficient accuracy in the similarity calculation processing, and There is a problem that it is difficult to reduce the burden on the user when performing automatic classification of document data.

The present invention has been made in view of the above circumstances. It is an object of the present invention to include, in a process of calculating a similarity, a set of words existing near each other in a comparison source document in a comparison destination document. A similar document search device that has a function of setting the weight of those words to be large, realizes highly accurate similarity calculation processing, and can improve the efficiency of automatic classification of a large number of document data, A similar document search method and a similar document search storage medium are provided.

[0010]

According to a first aspect of the present invention, there is provided a similar document search apparatus which extracts words from an arbitrary document as a comparison source and an arbitrary comparison destination document stored in a document storage unit. Extraction means, and word position information for individually storing each word extracted from the comparison source document and the comparison destination document by the extraction means together with the appearance position information in the comparison source document and the comparison destination document A storage unit, a determination unit that determines whether a specific word common to both the comparison source and the comparison destination documents exists with reference to the word position information storage unit, Searching means for searching for a word corresponding to the specific word within a certain distance from the specific word with respect to both the comparison source and the comparison destination documents by referring to the word position information storage means for the specific word,
A weight for calculating a weight coefficient corresponding to a distance from a specific word in both the comparison source and the comparison destination documents for a word corresponding to the specific word within a certain distance from the specific word searched by the search means. Coefficient calculating means, and similarity calculating means for calculating, based on the calculation result of the weighting coefficient calculating means, a similarity corresponding to the word appearance position of the word of the comparison destination document with respect to the comparison source document. It is assumed that.

According to the present invention, the extracting means extracts words from an arbitrary document to be compared and an arbitrary document to be compared stored in the document storage means, and the extraction source document and the compared document are extracted. Are stored individually in the word position information storage unit together with the appearance position information in the comparison source document and the comparison destination document, and the determination unit compares the words with reference to the word position information storage unit. It is determined whether or not a specific word common to both the source document and the comparison destination document exists, and the search unit refers to the word position information storage unit for the determined specific word, and compares the comparison source and the comparison destination. In both documents, a word corresponding to the word within a certain distance from the specific word is searched for, and the weight coefficient calculating means corresponds to the word corresponding to the word within a certain distance from the searched specific word. Source for words, A weight coefficient corresponding to the distance from a specific word in both documents of the comparison target is calculated, and the similarity calculation means calculates the weight coefficient according to the weight coefficient calculation means based on the calculation result of the comparison source document with respect to the comparison source document. Is calculated based on the appearance position of the word.

Therefore, when a set of words existing within a certain distance from each other in the comparison source document is included in the comparison destination document, the calculation of the weighting coefficients of those words allows for a high degree of accuracy taking into account the similarity of the context. Similar document search can be performed.

According to a second aspect of the present invention, there is provided a similar document search apparatus according to the first aspect, wherein the weighting factor calculating means specifies both the comparison source and the comparison destination documents. The weighting coefficient is set to a larger value as the distance of the word corresponding to the word from the word is smaller.

According to the present invention, the smaller the distance from a specific word is, the larger the weighting factor is added, and a high-precision similar document search becomes possible.

A similar document search device according to a third aspect of the present invention is the similar document search device according to the first aspect of the present invention, wherein the maximum weighting factor obtained by the weighting factor calculating means is used for a space vector method. Term data calculation means for calculating one term data of an inner product between word vectors according to the above, and inner product data normalizing means for normalizing the inner product data using the maximum weighting coefficient. ,
Each time the maximum weight coefficient obtained by the weight coefficient calculating means is multiplied by the number of appearances of the specific word in the processing of the inner product data normalizing means, the weight coefficient calculating means performs weight coefficient calculating processing. It is assumed that.

According to the present invention, when calculating the similarity as described above, the same processing as in the case where only the number of appearances as in the conventional example is used as the weighting factor can be performed.
In addition, the inner product operation and the normalization operation can be accurately performed.

According to a fourth aspect of the present invention, in the similar document search method, a word is extracted from an arbitrary document to be compared and an arbitrary document to be compared stored in the document storage unit, and the document of the comparison source is extracted. And each word extracted from the comparison destination document is separately stored in the word location information storage means together with the appearance position information in the comparison source document and the comparison destination document,
By referring to the word position information storage means, it is determined whether or not a specific word common to both the comparison source document and the comparison destination document exists, and the word position information storage means is determined for the determined specific word. By referring to both the source and destination documents, a word corresponding to the word within a certain distance from the specific word is searched for, and a certain distance from the searched specific word is calculated by the weight coefficient calculating means. A weight coefficient corresponding to the distance from a specific word in both the comparison source document and the comparison destination document is calculated for a word corresponding to the word in the calculation result, and the similarity calculation means calculates the weight result of the weight coefficient calculation means. Based on
It is characterized in that a similarity is calculated in accordance with the appearance position of the word of the comparison destination document with respect to the comparison source document.

According to this invention, using the configuration of the similar document search device according to the first aspect of the present invention, the comparison destination document includes a set of words existing within a certain distance from each other in the comparison source document. In this case, it is possible to establish a highly accurate similar document search method that takes into account the similarity of the context by calculating the weighting coefficients of those words.

A similar-document storage medium according to a fifth aspect of the present invention comprises the steps of: extracting a word from an arbitrary document as a comparison source and an arbitrary comparison-destination document stored in the document storage means; Storing each word extracted from the comparison source document and the comparison destination document together with the appearance position information in the comparison source document and the comparison destination document in the word position information storage unit; A procedure of referring to the position information storage means to determine whether or not a specific word common to both the document of the comparison source and the document of the comparison destination exists; and a step of determining the word position information storage means for the determined specific word. A procedure for searching for a word corresponding to the word within a certain distance from a specific word for both the reference source and the comparison target documents, and a method for searching for a word within a certain distance from the specific word searched. To the word equivalent to the word And calculating the weighting factor according to the distance from the specific word in the document of both the comparison source and the comparison destination, and comparing the comparison source document based on the calculation result by the procedure of calculating the weighting factor. A program for calculating a degree of similarity according to the appearance position of a word in the preceding document.

According to the present invention, this storage medium is read and used by an existing computer device or the like, so that the existing computer device or the like can be used in the same manner as the similar document search device according to the first aspect of the present invention. When the comparison target document includes a set of words existing within a certain distance from each other in the comparison source document, a highly accurate similar document search that takes into account the similarity of the context by calculating the weighting coefficients of those words It can be used as a device for performing

[0021]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a hardware configuration diagram of the present embodiment. As shown in FIG.
It comprises a display device 1, an input device 2, an external storage device 3, a control device 4, a memory 5, and a communication device 6, and each device is connected via a bus.

The display device 1 is composed of, for example, a color liquid crystal display and its controller, and displays search results and the like.

The input device 2 includes a keyboard, a mouse, and the like, and executes input of various data and commands to the control device 4.

The external storage device 3 comprises a hard disk and a controller, and stores document data to be searched and indexes of words included in each document.

FIG. 2 shows the storage format of the document data and word information data stored in the external storage device 3 of this embodiment.

Each document data is composed of a header part and a text data part representing titles, creation date / time data, creator data and the like of each document, and these are stored in the order of ID numbers.

The word information data is stored in association with each document, and the result of extracting a noun by performing a morphological analysis on the text data of the document data is stored. FIG. 4 shows an example of word information data created from one document in association with the original text.

As shown in FIG. 4, the number of appearances of each word in the text and each appearance position are stored in the word information data.

The control device 4 is composed of a CPU, and is connected to each of the above hardware devices via a bus, and controls each device and performs processing such as data transfer between the devices.

The memory 5 is composed of a dynamic RAM, and as shown in FIG. 3, a program section for storing programs for the control device 5 to execute various controls and processes, and for storing data necessary for the processes. It consists of a buffer section.

The program section of the memory 5 includes a similarity calculating section 4b, a weight coefficient determining section 4c, and a term for processing a subroutine called by the main processing section 4a in addition to a main processing section 4a for controlling the entire processing. It has a data adder 4d, an inner product data normalizer 4e, a document list display 4f, a document selector 4g, and a document content display 4h.

The buffer unit includes a comparison source document data storage buffer 4m, a comparison source word information storage buffer 4n, a comparison destination word information storage buffer 4p, and a similarity storage buffer 4.
q.

Here, the comparison source document data storage buffer 4
m has a structure capable of storing the document data shown in FIG. The comparison source word information storage buffer 4n and the comparison destination word information storage buffer 4p have a structure capable of storing word information data in the format shown in FIG.

The similarity storage buffer 4q has a structure in which the similarity of each document can be stored for each document title.
The buffer unit has an area 4s for each work variable.

This area 4s stores a document ID counting variable idDoc, a counting variable iSrc, iDst, kSrc, kDst, a weight coefficient variable coefSrc, and the like, which are used in the processing described later.

The communication device 6 transmits and receives data to and from the outside via a communication line, and includes, for example, a LAN line and a LAN controller.

Next, the flow of processing of the apparatus having the above-described components according to this embodiment will be described with reference to FIG.

Here, in the external storage device 3 in advance,
As shown in FIG. 2, it is assumed that a total of N pieces of document data and word information data in the above-described format from ID (identification) number 0 to N-11 are stored.

The overall control of the apparatus of this embodiment is performed by the main processing section 4a stored in the program section of the memory 5.

First, the document data input by the user from the input device 2 or the document data received by the communication device 6 is
It is stored in the comparison source document data storage buffer 4m (step 5a). The comparison source document data is composed of a header portion representing each attribute of the title, creation date and time, and the document of the creator, and a text data portion.

Next, the comparison source document data storage buffer 4m
The main processing unit 4a performs a morphological analysis on the text data part of the comparison source document data stored in the text data part based on a program stored in advance, and extracts nouns included in the text data part (step 5b). Original word information data is created and stored in the comparison source word information storage buffer 4n (step 5c).

The structure of the comparison source word information data is as shown in FIG.
The information has the same structure as that of the word information data shown in (a), and stores information on the number of appearances of each word and the appearance position in the text.

Next, the main processing section 4a stores 0 in the document ID counting variable idDoc (step 5).
d). Subsequently, the word information data in the external storage device 3 corresponding to the document ID counting variable idDoc is stored in the comparison target word information storage buffer 4p (step 5e).

Subsequently, the similarity calculation unit 4b is activated, and the contents of the comparison source document data stored in the comparison source word information storage buffer 4n and the comparison source document data stored in the comparison destination word information storage buffer 4p Is calculated with reference to (step 5f).

The details of the processing performed by the similarity calculating section 4b in step 5e will be described below with reference to FIG.

Here, the headword of the i-th word stored in the comparison source document data is wordSrc [i], the number of occurrences is nApSrc [i], and the j-th occurrence position is posSrc [i] [j ], The headword of the i-th word stored in the word information data of the comparison destination is word
Dst [i], the number of appearances thereof will be represented by nApDst [i], and the jth occurrence position thereof will be represented by posDst [i] [j].

The weight coefficient for each word vector of the comparison source is stored in the real variable coffSrc [iSrc], the total number of words stored in the comparison source document data is nSrC, and the word information data of the comparison destination is stored. ND the total number of words
It is represented by st.

Further, count variables iSrc, iDs
t, kSrc, and kDst are prepared in an area 4s for a working variable in the memory 5, and the following processing is executed.

First, 0 is stored in the similarity storage variable S and initialized (step 6a).

Hereinafter, words commonly included in the comparison source document and the comparison destination document are found in the processing from step 6b to step 6d.

First, 0 is substituted for the variable iSrc (step 6b). Next, the real variable cofeSrc [iSr
c] is substituted with 1.0 (step 6c). Next, 0 is substituted for the variable iDst (step 6d). Next, 1.0 is substituted for the real variable cofeDst [iDst] (step 6e).

Next, the similarity calculation unit 4b uses wordSrc
The character string content is compared to determine whether [iSrc] is equal to wordSt [iDst] (step 6f). If they are equal, control is transferred to step 6g, and if they are not equal, control is transferred to step 6i.

In step 6g, the weighting factor determination unit 4c is activated, and determines a weighting factor variable coefSrc [iSrc] related to one element of the word vector to be compared. The details of the weight coefficient determination process will be described later with reference to FIG.

Next, the term data adder 4d is activated, and adds one term data of the inner product of the word vector to the similarity degree storage variable S with reference to the value of coefSrc [iSrc] determined in step 6g ( Step 6h).

More specifically, the addition processing in the term data addition section 4d is performed according to the following equation (1).

[0057]

(Equation 1)

Here, nApSrc [iSrc] is the number of appearances of the comparison source word, and nApDst [iDst] is the number of appearances of the comparison target word, which are used as weights in general inner product calculation. is there.

The product of these is added to the weight coefficient variable c obtained earlier.
oefSrc [iSrc].

After the addition process is completed, the process proceeds to step 6i.
Move on to In step 6i, 1 is added to the value of the variable iDst. Here, if the value of the variable iDst is smaller than the total number nDSt (step 6j), the process returns to step 6f and repeats the processing. If the value of the variable iDst is not smaller than the total number nDSt, 1 is added to the value of the variable iSrc (step 6k).

Next, if the value of the variable iSrc is smaller than the total number nSrc, the process returns to step 6e and repeats the processing. If the value of the variable iSrc is equal to or greater than the total number nSrc in the processing in step 6k, the inner product data normalizing unit 4e is activated, and the value of the similarity storage variable S is calculated as the product of the sizes of the word vectors of the comparison source and the comparison destination. (Step 6m) to perform a normalization process (Step 6m).

The formula used in the process of the inner product data normalizing section 4e is shown in the following equation (2).

[0063]

(Equation 2)

The magnitude of the word vector of the comparison source is determined by the variable iS of the product of the number of appearances nApSrc [iSrc] of the word of the comparison source and the weight coefficient variable coefSrc [iSrc].
The square root of the sum of rc and the size of the word vector of the comparison destination are represented by the number of appearances nApDst [iDst] of the word of the comparison destination
Is obtained by the square root of the sum of the square of the variable iDst.

After finishing the processing in the inner product data normalizing section 4e described above, the similarity calculation processing shown in step 5f of FIG. 5 is finished (step 6n).

Next, the weight coefficient determination processing (step 7a) of the weight coefficient determination section 4c will be described with reference to FIG.

First, in the processing from step 7b to step 7p described below, the word wordSrc of the comparison source is used.
Word wo that exists near [iSrc] and is compared
It is checked whether there is a word existing near rdDst [iDst], and if it exists, the value of the weight coefficient variable coefSrc [iSrc] of the word to be compared is reset so as to be minimum. .

That is, first, the variable k representing the word to be compared
Substitute 0 for Src (step 7b). Thereafter, if the variable iSrc and the variable kSrc are the same (step 7
c) Since the same word is represented, the control moves to step 7s. When the variable iSrc and the variable kSrc are different,
Word wordSrc [iSrc] and word wordSrc
Compare whether [kSrc] is equal or not (step 7)
d).

If both are equal, control is passed to step 7e.
If not, the process proceeds to step 7s.

In step 7e, the words wordSrc [iSrc] appearing in the text and the words wordSrc
With reference to all the appearance positions of [kSrc], the absolute value (minimum value) minSrc of the difference between the appearance positions is obtained (step 7e).

The specific processing here will be described later with reference to FIG.

Thereafter, the minimum value obtained in step 7e is
For example, if it is equal to or less than 80 (80 characters) (step 7f), the control is shifted to step 7g, and if not, step 7s
Transfer control to. In the following step 7g, 0 is substituted for kDst representing the word of the comparison source (step 7g).

Thereafter, if iDst and kDst are the same (step 7h), they represent the same word, and therefore control transfers to step 7q. If iDSt and kDSt are different (step 7h), the word wordDst [iDst] and the word w
It is compared whether ordDst [kDst] is equal (step 7i).

If they are equal, the control moves to step 7j, and if they are not equal, the control moves to step 7q.

In step 7j, the words wordDst [iDst] and the words wordst [k
Dst], the absolute value (minimum value) minDst of the difference between the appearance positions is determined (step 7j).

The specific processing here will be described later with reference to FIG.
This will be described with reference to FIG.

Thereafter, the minimum value mi obtained in step 7j
If nDst is 80 or less (step 7k), step 7
m, otherwise move to step 7q.

In step 7m, the minSr obtained earlier
Referring to the values of c and minDst, the value of the real number variable coef of the weight coefficient is calculated so that the smaller the values of minSrc and minDst, the larger the value (step 7).
m).

Here, specifically, the value of the real number variable coef is obtained by the following equation (3).

[0080]

(Equation 3)

It should be noted that the value of the real number variable coef cannot be less than 1; therefore, minSrc / 80, mi
The value of nDst / 80 is set to 1 at the maximum.

Next, the value of the real number variable coef obtained here is compared with the previously obtained variable coefSrc [iSr
c] is compared (step 7n).

If the value of the real number variable coef is larger than the already obtained variable coefSrc [iSrc],
The value of coef obtained in step 7m is adopted as the new maximum value of coefSrc [iSrc] (step 7p). Subsequently, in step 7q, the value of the variable kDst is incremented (step 7q), and it is compared whether the value is smaller than the total number of words nDst of the comparison target document (step 7r). If smaller, the processing from step 7h is repeated. Otherwise, control is transferred to step 7s.

In step 7s, the value of the variable kSrc is incremented, and the value is set to the total number of words nSr in the comparison target document.
It is compared whether it is smaller than c (step 7t), and if it is smaller, the processing from step 7c is repeated. Otherwise, the weight coefficient determination processing ends (step 7u).

Next, in the process of step 7e, the word wordSrc [iSr
c] and the absolute value minSr of the difference between the appearance positions with reference to all the appearance positions of the word wordSrc [kSrc].
The details of the process for obtaining c will be described with reference to FIG.

In this process, first, j0 is set to 0 (step 8a), and in the loop from step 8b to step 8j, the value of j0 is changed from 0 to nApSrc [iSrc]. In the loop from to 8h, the value of j1 is changed from 0 to nApSrc
[KSrc], and for all combinations of j0 and j1, the appearance position p of the word represented by kSrc
osSrc [kSrc] [jo], the distance of the occurrence position posSrc [iSrc] [j1] of the word represented by iSrc, that is, the absolute value dSrc of the difference represented by the following equation (4)
Is calculated (step 8c).

[0087]

(Equation 4)

Further, in the case of the initial state (j0 = 0 and j1
= 0) (step 8d), or when the value of dSrc obtained in step 8c is smaller than the minimum value minSrc already obtained in the previous processing (step 8e), a new minSrc value is obtained in step 8c. DS
The value of rc is adopted (step 8f).

When these processes are completed, the process ends (step 8k).

Next, in step 7j, the absolute value minDst of the difference between the appearance positions is referred to by referring to all the appearance positions of the words wordDst [iDst] and the word wordDst [kDst] appearing in the text to be compared. The details of the process to be determined will be described with reference to FIG.

In this processing, first, j0 = 0 (step 9a), the value of j0 is changed from 0 to nApDst [iDst] in a loop from step 9b to step 9j, and step 9c inside this loop is executed. In the loop up to step 9h, the value of j1 is changed from 0 to nApDst
[KDst], and for all combinations of j0 and j1, the appearance position p of the word represented by kDst
The distance between the occurrence position posDst [iDst] [j1] of the word represented by osDst [kDst] [j0] and iDst, that is, the absolute value of the difference represented by the following equation 5 is calculated (step 9c).

[0092]

(Equation 5)

Further, by the processing of steps 9d to 9f, the case of the initial state (j0 = 0 and j1
= 0) (step 9d), or when the value of dDst obtained in step 9c is smaller than the minimum value minDst of the distance already obtained in the previous processing (when the comparison in step 9e is successful), a new minDst The value of dDst obtained in step 9c is adopted as the value of (step 9f).

When these processes are completed, the process ends (step 9k).

Next, after calculating the similarity in step 5f as described above, as shown in FIG. 5, the obtained similarity is stored in the similarity storage buffer 4q in a position corresponding to idDoc (step 5f). 5g).

Next, the value of idDoc is incremented (step 5h). Further, the value of idDoc is compared with the total number N of documents stored in the external storage device 3, and i
If dDoc is smaller than the total number N, the processing from step 5e is repeated, otherwise, control is transferred to step 5j (step 5i).

In step 5j, the document list display section 4
f is activated. The document list display unit 4f refers to the contents of the similarity storage buffer 4q and the title data in each document data, and starts from the document with the highest similarity (from the one with the lowest similarity to the one with the highest similarity). Then, the list of the document titles is displayed on the screen of the display device 1 together with the corresponding similarity (similarity 1, similarity 2,...) Information (step 5j).

FIG. 10 shows the state of the screen of the display device 1 in the document title-list display state in this case.

Subsequently, in step 5k, the document selection section 4g is activated. The document selection unit 4g allows the user to select one or a plurality of document titles displayed on the screen using the input device 2.

Next, the document content display section 4h is activated, and the content of the document corresponding to the document title selected in step 5k is displayed (step 5m). Thereby, the user can display and confirm the contents of the document having the desired similarity on the screen according to his / her selection. Thus, the process ends (step 5n).

The above is the flow of processing in the present embodiment.

The present invention is not limited to the above embodiment.

For example, step 6 of the term data adder 4d
In the addition processing of f, the calculation formula shown in the embodiment may not be used as long as the value of the weight coefficient variable is reflected in the result.

Step 7m of the weight coefficient determining unit 4c
The calculation formula used in minSrc or minSrc
The present invention need not be the one shown in the present embodiment as long as the weight coefficient increases as the value of Dst decreases.

Further, steps 7f and 7h shown in FIG.
In the above, the value of 80 was used in comparing the distances, but this value can also be set to another appropriate value.

In addition, various modifications can be made without departing from the spirit of the present invention.

[0107]

According to the present invention described above, the following effects can be obtained.

According to the first aspect of the present invention, in the case where a set of words existing within a certain distance from each other in a comparison source document is included in a comparison destination document, the weight coefficient of those words is calculated to determine the context. A similar document search device capable of performing a similar document search with high accuracy in consideration of similarity can be provided.

According to the second aspect of the present invention, it is possible to provide a similar document search apparatus capable of performing a high-accuracy similar document search by adding a larger weighting factor as the distance between specific words becomes smaller. it can.

According to the third aspect of the present invention, when calculating the similarity, it is possible to perform the same processing as in the case where only the number of appearances as in the conventional example is used as the weighting coefficient, and it is simple and accurate. A similar document search device capable of performing inner product operation and normalization operation can be provided.

According to the fourth aspect of the present invention, by using the configuration of the similar document search apparatus according to the first aspect of the present invention, a set of words existing within a certain distance from each other in the comparison source document is compared with the comparison destination document. , It is possible to provide a similar document search method capable of establishing a highly accurate similar document search method that takes into account the similarity of context by calculating weighting coefficients of those words.

According to the fifth aspect of the present invention, when an existing computer device or the like includes a set of words existing within a certain distance from each other in a comparison source document, the comparison destination document includes:
It is possible to provide a similar document search storage medium that can be used as a device for performing a highly accurate similar document search that takes into account the similarity of context by calculating the weighting coefficients of those words.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a hardware configuration of a device according to an embodiment.

FIG. 2 is an explanatory diagram showing a storage format of document data and word information data of the present embodiment.

FIG. 3 is a diagram showing a configuration of a memory of the present embodiment device.

FIG. 4 is a diagram showing an example of word information data of the present embodiment device in association with an original text.

FIG. 5 is a flowchart showing a flow of overall processing of the apparatus according to the embodiment.

FIG. 6 is a flowchart illustrating details of a process performed by a similarity calculation unit of the embodiment.

FIG. 7 is a flowchart showing a process of a weight coefficient determination unit of the present embodiment.

FIG. 8 is a flowchart illustrating a process of calculating a distance between words appearing in a comparison source text according to the present embodiment.

FIG. 9 is a flowchart illustrating a process of calculating a distance between words appearing in a comparison target text according to the present embodiment.

FIG. 10 is a diagram showing a screen of the display device in the case of title-view display of the present embodiment device.

[Explanation of symbols]

 Reference Signs List 1 display device 2 input device 3 external storage device, 4 control device 4a main processing unit 4b similarity calculation unit 4c weight coefficient determination unit 4d term data addition unit 4e inner product data normalization unit 4f document list display unit 4g document selection unit 4h document Content display section 4m Comparison source document data storage buffer 4n Comparison source word information storage buffer 4p Comparison destination word information storage buffer 4q Similarity storage buffer 4s Area for various work variables 5 Memory 6 Communication device

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Yukio Nakamoto 1381-1, Shinmachi, Ome-shi, Tokyo Toshiba Computer Engineering Co., Ltd. (72) Takuya Nishina 1381-1, Shinmachi, Ome-shi, Tokyo Toshiba Computer (72) Inventor Naohide Kubota 1381 Shinmachi, Ome-shi, Tokyo Toshiba Computer Engineering Co., Ltd.

Claims (5)

[Claims] An extracting means for extracting a word from an arbitrary document to be compared and an arbitrary document to be compared stored in a document storing means; and extracting means for extracting a word from the document to be compared and the document to be compared by the extracting means. Word position information storage means for individually storing each extracted word together with the appearance position information in the comparison source document and the comparison destination document; and the comparison source and comparison with reference to the word position information storage means Determining means for determining whether or not a specific word common to both of the preceding documents exists; and comparing the specific word determined by the determining means with reference to the word position information storage means for comparison and comparison. Searching means for searching for a word corresponding to the word within a certain distance from a specific word in both of the preceding documents; and searching for the word within a certain distance from the specific word searched by the searching means. Equivalent Weighting factor calculating means for calculating a weighting factor corresponding to the distance from a specific word in both the comparison source and the comparison target documents for the word to be compared, and the comparison source based on the calculation result of the weighting factor calculation means A similarity calculating unit configured to calculate a similarity according to an appearance position of a word of a document to be compared with the document of the similar document. 2. The weighting factor calculation means according to claim 2, wherein the weighting factor is set to a larger value as the distance of a word corresponding to the word from a specific word is shorter for both the source document and the destination document. The similar document search device according to claim 1, wherein 3. The similar document search device according to claim 1, wherein one term data of an inner product between word vectors is calculated by a space vector method using a maximum weighting factor obtained by the weighting factor calculation means. Data calculating means, further comprising inner product data normalizing means for normalizing inner product data using the maximum weighting coefficient, term processing means, in the processing of the inner product data normalizing means by the weighting coefficient calculating means A similar document search device, wherein the weight coefficient calculating means performs a weight coefficient calculating process each time the obtained maximum weight coefficient is multiplied by the number of appearances of the specific word. 4. Extracting words from an arbitrary document serving as a comparison source and an arbitrary comparison destination document stored in a document storage unit, and extracting each word extracted from the comparison source document and the comparison destination document. Are stored individually in the word position information storage means together with the appearance position information in the comparison source document and the comparison destination document, and are referred to both the comparison source and comparison destination documents by referring to the word position information storage means. It is determined whether or not there is a specific word that is common to the specific word, and the determined specific word is referred to the word position information storage means, and the specific word is fixed for both the source and destination documents. The word corresponding to the word within the distance of the search is searched for, and the weighting factor calculating means compares the word corresponding to the word within a certain distance from the searched specific word with both the comparison source and the comparison destination. From a specific word in the document Calculating a weight coefficient corresponding to the distance, and calculating, by the similarity calculating means, a similarity corresponding to an appearance position of a word of the document of the comparison destination with respect to the document of the comparison source based on the calculation result of the weight coefficient calculating means. A similar document search method, characterized in that: 5. A procedure for extracting words from an arbitrary document serving as a comparison source and an arbitrary comparison destination document stored in a document storage unit, and a procedure for extracting words from the comparison source document and the comparison destination document. Storing the words individually in the word position information storage unit together with the appearance position information in the comparison source document and the comparison destination document; and referring to the word position information storage unit, A procedure for determining whether or not a specific word common to both documents exists; and referring to the word position information storage means for the determined specific word, for both the source and destination documents. Searching for a word corresponding to the word within a certain distance from the specific word, and comparing a word corresponding to the word within a certain distance from the searched specific word with a comparison source and a comparison destination. Specific in both documents Calculating a weighting factor according to the distance from the word, and calculating the similarity according to the word appearance position of the word of the comparison target document with respect to the comparison source document based on the calculation result obtained by the procedure of calculating the weighting factor. A storage medium for similar document search, characterized by storing a calculation procedure and a program consisting of: