JPH1031677A - Document search device - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide a document retrieval apparatus capable of retrieving document data semantically similar to a retrieval request described in a predetermined language from a plurality of document data each described in a different language. SOLUTION: A stored document database 2 storing a plurality of document data, each described in a different language, and a document feature vector indicating a semantic feature corresponding to each of the plurality of document data, A search request input unit 3 for inputting the search request, a search request feature vector generation unit 4 for generating a search request feature vector indicating a semantic feature of the input search request, and a search unit 5.
The search unit 5 extracts, based on the inner product value of the generated search request feature vector and each document feature vector, document data semantically similar to the input search request from the plurality of document data in the stored document database 2. Search and output.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document retrieval apparatus for retrieving document data described in a natural language, and more particularly, to a document data satisfying a retrieval request from a mixture of a plurality of document data described in different languages. The present invention relates to a document search device that is effective for obtaining information of a document.

[0002]

2. Description of the Related Art As a method of supplementing a rigid retrieval method that presupposes notation matching such as keyword retrieval when retrieving document data, a method of retrieving document data using a feature vector of a word has already been proposed. Have been.

[0003] In the proposed document retrieval system,
A feature space is defined by several feature words (usually several tens to several hundreds), a feature vector is created by quantifying the relationship between each word and the feature word, and a feature vector of the word included in the document data is created. The sum is used as the feature vector of the document data. Then, a feature vector is similarly calculated for a search request from the operator, and the degree of approximation between mutually normalized (equalized vector length) feature vectors in both the document data and the search request is calculated. The inner product value) is calculated, and it is determined that the larger the degree of approximation is, the closer the document data is to the search request.

In the case of this document search system, it is necessary to create feature vectors for a considerable number of words in order to achieve practically high search accuracy, which is a great deal for the builder of a document search apparatus. Japanese Patent Application Laid-Open No. 6-195388 discloses a technique for automatically assigning a feature vector of a word to reduce the burden.

[0005]

[Problems to be Solved by the Invention] With the development of network technology, there is almost no need to be aware of the existence of borders when obtaining necessary information, and there is a demand to obtain and use a vast amount of information overflowing overseas. Although growing, there is still the barrier of language differences. In a conventional document search method using a feature vector, if a word dictionary to which a feature vector is assigned relates to a certain language A, only document data described in the language A is to be searched. However, there is a problem that the range of information to be obtained is limited, and the practicability is not excellent.

Therefore, an object of the present invention is to provide a document retrieval apparatus capable of retrieving document data semantically similar to a retrieval request described in a predetermined language from a plurality of document data each described in a different language. That is.

Another object of the present invention is to search for document data that is semantically similar to a search request described in a predetermined language from a plurality of document data each described in a different language. It is an object of the present invention to provide a document search device that can easily add document data to be used.

Still another object of the present invention is to provide a method for retrieving document data semantically similar to a retrieval request described in a predetermined language from a plurality of document data each described in a different language. It is an object of the present invention to provide a document search apparatus capable of translating and presenting semantically approximated document data obtained by using a search request description language.

[0009]

According to a first aspect of the present invention, there is provided a document search apparatus comprising: a plurality of document data written in different languages; and a document showing a semantic characteristic corresponding to each of the plurality of document data. A document data storage unit storing the characteristic data, a request input unit for inputting a search request described in a predetermined language, and request characteristic data indicating semantic characteristics of the search request input by the request input unit A request feature data detection unit for detecting a search request; and a search request input by the request input unit based on the request feature data detected by the request feature data detection unit from a plurality of document data in the document data storage unit. And a retrieval unit that retrieves and outputs semantically approximated document data that approximates.

According to the first aspect of the present invention, even if the description language of the input search request does not match the description language of the stored document data to be searched, the document search apparatus searches for document data that is semantically similar. Can output. Therefore, the operator can obtain the requested document data regardless of the description language of each of the plurality of document data in the document data storage unit. The range of sources is set wider to improve practicability.

According to a second aspect of the present invention, there is provided the document search apparatus, wherein the apparatus according to the first aspect further comprises a plurality of words for each of different languages, and a word feature vector indicating a semantic feature corresponding to each word. And a word dictionary unit storing data and the document feature data and the requested feature data, respectively, and normalize the sum of the word feature vector data in the word dictionary unit corresponding to each word constituting the document data and the search request. And the semantically approximated document data is document data having a large inner product value between the corresponding unit vector data and the unit vector data corresponding to the search request among the plurality of document data in the document data storage unit. It is configured as follows.

According to the second aspect of the present invention, the document data which is semantically similar to the search request input from the plurality of document data in the stored document data is obtained by the inner product value calculation processing of the vector data. Therefore, the document data requested by the operator can be searched (specified) easily and promptly.

According to a third aspect of the present invention, the word dictionary section of the second aspect of the present invention is arranged such that all words having the same meaning in different languages are associated with the same word feature vector data. Be composed.

According to the third aspect of the present invention, the word dictionary section is configured such that all synonymous words between different languages are associated with one word feature vector data. The memory capacity consumed by the dictionary unit is suppressed, and the memory of the search device is effectively used.

Further, word feature vector data can be uniquely obtained for words in any language as long as they are synonyms, so that document data that is semantically similar to the search request can be searched accurately regardless of the description language. Can be output.

According to a fourth aspect of the present invention, there is provided the document search apparatus according to any one of the first to third aspects, further comprising:
A document data input unit for inputting new document data, a document characteristic data generation unit for generating document characteristic data of the document data input by the document data input unit, and a document input by the document data input unit The data is stored in the document data storage unit in association with the document feature data generated by the document feature data generation unit.

According to the fourth aspect of the present invention, the document characteristic data required for the search is generated regardless of the description language of the new document data input, and the document data to be searched is converted to the document data. It can be additionally stored in the document data storage together with the document characteristic data. Therefore, it is possible to easily construct a document data storage unit for document data to be searched.

According to a fifth aspect of the present invention, there is provided a document search apparatus according to any one of the first to fourth aspects, further comprising:
A translation dictionary unit that holds data necessary for translation between different languages; and a search unit that outputs a search when the description language of the semantically approximated document data does not match a predetermined language that describes the search request. A translation processing unit for translating the semantically approximate document data into a predetermined language with reference to the contents.

According to the document search device of the present invention, even if the description language of the search request input by the operator does not match the description language of the document data to be searched, The information possessed can be translated and presented in the description language of the search request, that is, the language notation requested by the operator. Therefore, even when the information of the requested document data is described in a language that cannot be read, the operator can acquire the requested information without a language barrier.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments 1 to 1 of the present invention will be described.
3 will be described with reference to the drawings.

FIG. 1 is a block diagram of a document search apparatus applied to the first to third embodiments of the present invention. In the figure, a document search device is inputted from a word dictionary database 1, a stored document database 2, a search request input unit 3 composed of, for example, a keyboard, and the like, for inputting a search request described in a desired natural language. A search request feature vector generation unit 4, a search unit 5, a display unit 6 for displaying data such as processing results, a key input unit, etc., for generating a feature vector of the searched search request. It includes a new document input unit 7 for inputting (unregistered) document data, a language determination unit 8 for determining the description language of given data, a document feature vector generation unit 9, a translation dictionary database 10, and a translation processing unit 11. .

FIG. 2 is a diagram showing a configuration example of the word dictionary database 1 of FIG. In FIG. 2, the word dictionary database 1 includes word dictionaries DA, DB, DC, DD,... For a plurality of different natural languages A, B, C, D,. Includes a plurality of word data Wi (i = 1, 2, 3,...), And each word data Wi is associated with a word feature vector Vi indicating its semantic feature.

FIG. 3 is a diagram showing a configuration example of the stored document database 2 of FIG. In FIG. 3, the stored document database 2 includes a plurality of document data SDi registered in the device and described in different natural languages, and a document feature vector VDi corresponding to each document data SDi. Note that the document feature vector VDi of the document data SDi can be obtained by the document feature vector generation unit 9, the details of which will be described later.

The translation dictionary database 10 is a database that is referred to by the translation processing unit 11 when translating the document data SDi described in a certain natural language into another natural language. This defines the conditioning in the translation into.

(Embodiment 1) FIG. 4 is a partial configuration diagram of the document search apparatus of FIG. 1 necessary for the document data search processing operation of Embodiment 1 of the present invention, and FIG. 5 is an embodiment of the present invention. 9 is a flowchart of a document data search processing operation according to the first embodiment.

FIG. 6 is a diagram for explaining a procedure for obtaining a search request feature vector from a search request in the document data search processing operation according to the first embodiment of the present invention. FIG. 7 is a diagram for explaining a procedure for obtaining document data similar to a search request in the document data search processing operation according to the first embodiment of the present invention.

Next, as a first embodiment of the present invention, a processing operation of searching and outputting document data SDi that is semantically similar to a search request from the stored document database 2 in the document search apparatus of FIG. 1 will be described with reference to FIG. This will be described according to the flowchart of FIG.

First, the search request input in the search request input unit 3 is sent to the search request feature vector generation unit 4 (S301). Next, the search request feature vector generation unit 4
Performs a morphological analysis of a search request input with reference to the word dictionary database 1, extracts word feature vectors of each word included in the search request, and generates a unit vector obtained by normalizing the sum of these word feature vectors. The search request feature vector is sent to the search unit 5 (S302).

FIG. 6 shows the detailed procedure from the search request by the search request feature vector generation unit 4 to the obtaining of the search request feature vector. Figure 6 shows the “future of personal computer communication”
Is shown, the search request SR may be a word such as “PC”, a sentence composed of a plurality of words, or a plurality of sentences. Document. In FIG. 6, when a search request SR “future of personal computer communication” is input, the search request feature vector generation unit 4 converts this into a morpheme (word).
It is analyzed and decomposed into “PC”, “communication (no)”, and “future”, and a word feature vector Vi corresponding to each morpheme (word data Wi) is extracted from the word dictionary database 1 corresponding to the description language of the search request. In this case, V1, V2, and V3 have the same vector length (for example, length 1).
In this example, the size of each word feature vector Vi is all the same, but the vector length may be changed for each word and weighted depending on the situation. For example, since the search request SR is related to a specialized field, the weight of the technical terms in the search request SR can be changed to improve the search efficiency.

Finally, the sum of V1, V2, and V3 is normalized to obtain a search request feature vector VS.

Here, as shown in FIG. 2, the word dictionary database 1 can accept a search request SR in an arbitrary natural language within the range by supporting a plurality of natural languages. That is, in FIG. 2, by associating the word data Wi group of synonyms between natural languages A, B, C, D,... With one word feature vector Vi, the word data Wi of any natural language can be obtained. Since the word feature vector Vi is obtained in exactly the same way, the search request feature vector V indicating the semantic feature of the search request SR is obtained.
S can be easily obtained. For example, the example in FIG. 2 shows that the same word feature vector V1 can be extracted whether the search request SR is “house” in language A or “house” in language B.

Returning to FIG. 5, as shown in FIG. 7, the retrieval unit 5 transmits a document request vector VDi indicating the semantic characteristics of each document to each document data SDi held in the stored document database 2 and a retrieval request. The inner product VS · VDi of the feature vector VS is calculated, and the inner product value is defined as the semantic similarity Ai between the search request SR and the document data SDi. Therefore, the document data SDi in which the semantic similarity Ai increases,
That is, the document data SDi that is considered to be semantically close to the search request SR are sent to the display unit 6 in order (S3).
03). At this time, only the one that is semantically closest may be sent, the top n items may be sent, or one having a semantic approximation Ai that satisfies a certain threshold may be sent. Finally, the display unit 6 displays the sent document data SDi.
Is displayed (S304), and the series of processing ends.

According to this document data search method, the degree of semantic similarity between the search request and each piece of document data is calculated in a manner that does not depend on the description language at all. The search range can be easily expanded for document data as a search target.

(Embodiment 2) FIG. 8 is a partial configuration diagram of the document search apparatus of FIG. 1 necessary for the document data registration processing operation of Embodiment 2 of the present invention, and FIG. 9 is an embodiment of the present invention. 13 is a flowchart of a document data registration processing operation according to the second embodiment.

FIG. 10 is an explanatory diagram of data created by the language determination section in the document data registration processing operation according to the second embodiment of the present invention. FIG. 11 is an explanatory diagram of an operation of adding new document data to a document feature vector and storing the new document data in a stored document database in the document data registration processing operation according to the second embodiment of the present invention.

Next, as a second embodiment of the present invention, FIG.
The processing operation for registering new document data in the stored document database 2 in the document search device will be described with reference to the flowchart of FIG.

First, new document data is input from the new document input unit 7 and sent to the language determination unit 8 (S401). As shown in FIG. 10, the language determination unit 8 determines the description language of the input document data by using the character features and syntax features of the new document data, as shown in FIG. Is sent to the document feature vector generation unit 9 (S402).

The method of language determination by the language determination unit 8 may be a method of simply determining only characters, or a method of determining languages A, B,
For C, D,..., An analysis process using the word dictionary database 1 may be performed on the input new document data, and a determination may be made based on the degree of the analysis possibility. A method may be used in which some characteristic words appearing most frequently in the document data are searched for and determined in the input document data.

Returning to FIG. 9, the document feature vector generation unit 9
Among the word data Wi held in the word dictionary database 1 as shown in FIG.
While referring to word data Wi corresponding to the description language of
The input document data SD is subjected to morphological analysis to obtain a word feature vector Vi of each word Wi included in the input document data SD.
Is extracted, and a unit vector obtained by normalizing the sum of the word feature vectors Vi is associated with the document data SD as a feature vector VD of the input document data SD (S40).
3), stored in the stored document database 2 (S404),
A series of processing ends.

According to the document data registration method described above,
There is no need to manually classify document data having different description languages, and the data can be registered and stored as stored document data in the stored document database 2 centrally. Further, since the document feature vector can be automatically calculated for the document data, it is possible to continuously perform the input of the document data and the execution of the above-described search processing. It can be easily expanded to data.

(Embodiment 3) FIG. 12 is a partial block diagram of the document search apparatus of FIG. 1 required for the operation of translating the document data obtained by the search according to Embodiment 3 of the present invention.
FIG. 3 is a flowchart of a translation processing operation of document data obtained by the search according to the third embodiment of the present invention.

Next, a third embodiment of the present invention will be described with reference to FIG.
A processing operation of translating and presenting document data obtained by searching in the document search device of the present embodiment will be described with reference to the flowchart of FIG.

First, steps S501 to S5 in FIG.
The procedures up to 03 are the same as those in FIG.
Since the procedure from step S301 to step S303 is followed, detailed description is omitted. Note that step S30 in FIG.
In FIG. 3, the document data obtained by the search is sent directly to the display unit 6, but in FIG.

In the language determination section 8, the second embodiment is used.
In exactly the same manner as described in step S402 of FIG. 9, the description language of the document data is determined using the character features and syntax features of the document data sent from the search unit 5 and The determination result is sent to the translation processing unit 11 together with the document data (S504). At this time, the translation processing unit 1
The format of the data sent to 1 is the same as the format of the new document data shown in FIG.

When it is known that the search request SR has been morphologically analyzed and described in the language A, the translation processing unit 11 first checks the description language of the document data, and converts the document data into a language other than the language A. If it is described, it is determined that the translation processing of the language A is necessary (S505), and the translation processing is performed with reference to the translation dictionary database 10 (S50).
6) The translation result is sent to the display unit 6 and displayed (S50).
7), a series of processing ends.

On the other hand, if the document data is described in the description language A of the search request SR, it is determined that the translation process is unnecessary (S
505), the document data sent from the language determination unit 8 is sent to the display unit 6 as it is and displayed (S508), and a series of processing ends.

According to the above-described document data translation method, even if the document data containing necessary information is described in any language, a single language that the operator can understand, that is, the description language of the search request SR The necessary information can be obtained by the expression of.

[Brief description of the drawings]

FIG. 1 is a block diagram of a document search device applied to Embodiments 1 to 3 of the present invention.

FIG. 2 is a diagram showing a configuration example of a word dictionary database 1 of FIG.

FIG. 3 is a diagram showing a configuration example of a stored document database 2 of FIG. 1;

FIG. 4 is a partial configuration diagram of the document search device of FIG. 1 necessary for a document data search processing operation according to the first embodiment of the present invention;

FIG. 5 is a flowchart of a document data search processing operation according to the first embodiment of the present invention.

FIG. 6 is a diagram illustrating a procedure for obtaining a search request feature vector from a search request in the document data search processing operation according to the first embodiment of the present invention.

FIG. 7 is a diagram illustrating a procedure for obtaining document data approximate to a search request in the document data search processing operation according to the first embodiment of the present invention.

FIG. 8 is a partial configuration diagram of the document search device of FIG. 1 required for a document data registration processing operation according to the second embodiment of the present invention.

FIG. 9 is a flowchart of a document data registration processing operation according to the second embodiment of the present invention.

FIG. 10 is an explanatory diagram of data created by a language determination unit in a document data registration processing operation according to the second embodiment of the present invention.

FIG. 11 is an explanatory diagram of an operation of adding new document data to a new document data and storing it in a stored document database in a document data registration operation according to the second embodiment of the present invention;

FIG. 12 is a partial configuration diagram of the document search device in FIG. 1 required for a translation operation of document data obtained by searching according to the third embodiment of the present invention.

FIG. 13 is a flowchart of a translation processing operation of document data obtained by searching according to the third embodiment of the present invention.

[Explanation of symbols]

REFERENCE SIGNS LIST 1 word dictionary database 2 stored document database 3 search request input unit 4 search request feature vector generation unit 5 search unit 7 new document input unit 8 language determination unit 9 document feature vector generation unit 10 translation dictionary database 11 translation processing unit Vi word feature vector Wi word data SDi document data VDi document feature vector SR search request VS search request feature vector Ai Semantic similarity between search request SR and document data SDi where i = 1, 2, 3,... The same reference numerals indicate the same or corresponding parts.

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁶ Identification code Agency reference number FI Technical display location G06F 15/403 330C

Claims (5)

[Claims] A document data storage unit that stores a plurality of document data, each of which is described in a different language, and document feature data corresponding to each of the plurality of document data and indicating a semantic feature thereof; A request input unit for inputting a search request described in a language of the following; a request characteristic data detection unit for detecting required characteristic data indicating a semantic characteristic of the search request input by the request input unit; A semantic approximation document that semantically approximates the detection request input by the request input unit from the plurality of document data in the document data storage unit based on the required characteristic data detected by the characteristic data detection unit. A document search device comprising: a search unit for searching and outputting data. 2. The apparatus according to claim 1, further comprising: a word dictionary unit storing a plurality of words for each of different languages and word feature vector data indicating a semantic feature corresponding to each word; Each of the feature data is unit vector data obtained by normalizing a sum of the word feature vector data in the word dictionary unit corresponding to each word constituting each of the document data and the search request, and the semantic approximation. The document data is document data having a large inner product value between the corresponding unit vector data and the unit vector data corresponding to the search request among the plurality of document data in the document data storage unit. Item 2. The document search device according to Item 1. 3. The document search device according to claim 2, wherein in the word dictionary section, all of the words having the same meaning in different languages are associated with the same word feature vector data. 4. A document data input unit for inputting new document data, and a document characteristic data generating unit for generating the document characteristic data of the document data input by the document data input unit, The document data input by the document data input unit is stored in the document data storage unit in association with the document feature data generated by the document feature data generation unit. 4. The document search device according to any one of claims 1 to 3. 5. A translation dictionary unit for holding data necessary for translation between different languages, and a description language of the semantically approximate document data matches a predetermined language describing the search request when the search unit outputs a search. 5. The document search device according to claim 1, further comprising: a translation processing unit that translates the semantically approximated document data into the predetermined language by referring to the contents of the translation dictionary unit when not. .