JPH04246772A - Image retrieving system by sketch - Google Patents

Abstract

PURPOSE:To facilitate the designation of a shape, and also, to execute the retrieval by an instruction of a partial shape by retrieving the shape to be retrieved by only instructing it by a sketch on a displayed similar image. CONSTITUTION:In the beginning, on a display part 2, an index is superposed on an image and displayed. A user sketches a shape to be retrieved on a display image with a light pen 3. By a correlation calculating means 53 and a function approximating means 53, the index which is displayed is approximated to the sketch. An index comparing means 55 compares the index changed by this approximation and the index given to an image read out of an accumulating part 1, and an image in which a distance of both of them is near is displayed on the display part 2 through an image rearranging means 56.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for searching an image database using contour sketches.

[Conventional technology]

[0002] Conventionally, known image database search methods include a search method in which an intended shape is entered as a keyword or word, and a search method in which similar images are presented.

0003

[Problems to be Solved by the Invention] In the prior art, when expressing a shape in words, there are various ways to express even one shape. It was quite difficult to create a dictionary of expressions. Furthermore, it is quite difficult to express complex shapes accurately in words. Furthermore, even when searching by keyword, it required considerable skill to express the intended shape using limited keywords. Furthermore, in the case of a search by presenting similar images, a considerable amount of image processing technology is required to extract information about the shape from the input image, and it is also necessary to perform a search by partial shape based on the extracted shape. In some cases, there was a problem in that it was not possible to specify which part's shape to focus on in the search. There is also a search method that presents sketches as similar images, but conventionally, it is possible to search by overall shape;
Search by partial shape was not possible.

[0004] The present invention has been made to solve this problem, and even when it is difficult to express a three-dimensional shape in words, it is possible to search by sketching a partial shape of the shape to be searched. The purpose is to make it possible.

[0005]

[Means for Solving the Problems] In order to achieve the above object, the present invention is provided with an image storage section in which indexes are assigned to image structures using shape representation by functions, and an index is assigned to images. means for superimposing and displaying; means for inputting a sketch of a contour shape; and means for correlating the input sketch with an index of the displayed image and approximating a function of the index to the sketch. , means for retrieving from the image storage section an image to which an index close to the index value changed by the approximation is provided.

[0006]

[Effect] First, the index is superimposed on the similar image and displayed, so the user can see which part of the image structure is described as what kind of index, and the user can change the index to transform the displayed index. Sketch the shape you want to search on the image as a new contour for the index. Then, the coordinate data of the outline of the sketch is detected, the index for which the shape was sketched is detected from the position of the coordinate data, and the function corresponding to the index corresponding to the sketched part is approximated to the sketch. By retrieving an image having an index close to the function parameter obtained by this approximation from the image storage section, it is possible to search for an image structure close to the sketched shape.

[0007]

[Embodiment] An embodiment of the present invention will be described below. In this embodiment, an example of searching an image database to which a three-dimensional shape index is given using a hyperquadratic function will be described.

The hyperquadratic function is expressed by the following equation 1, where ε1 is a parameter representing the shape viewed from the side, and ε2 is a parameter representing the shape viewed from above. Also, a1, a2,
a3 is a scale parameter representing the size in each of the x, y, and z axis directions. Note that ω indicates an angle in the latitude direction, and η indicates an angle in the longitude direction.

[Math 1]

Here, parameters ε1, ε2, a1, a
2. By changing a3, various shapes such as a cylinder, a cube, and a sphere can be expressed. In the example,
As indices, the parameters ε1, ε2 of this function
, a1, a2, a3, and the transformation matrix M for transforming the coordinate system of the hyperquadratic function to the coordinate system of the projection plane that shows the image structure are also given as indexes, and as the search target image. This will be explained using an example using an image of a coffee cup.

FIG. 1 is a block diagram of an embodiment of a device configuration for realizing an image search method according to the present invention. Reference numeral 1 denotes a storage unit that stores images and indexes, and 2 an image display unit that has an overlay screen that displays images and indexes. 3 is a light pen for drawing a sketch of the outline shape in the display unit 2; 4 is a control device for controlling the light pen 3; the coordinate data input by the light pen 3 is sent from the light pen control device 4; It is input to the control section 5. The control unit 5 is composed of, for example, a microprocessor, and generally performs data processing according to a control program written in a memory (not shown). The control unit 5 functionally includes a coordinate position detection means 50, an index reading means 51, a two-dimensional projection means 52, a correlation calculation means 53,
It consists of a function approximation means 54, an index comparison means 55, and an image rearrangement means 56.

FIG. 2 is a flowchart for explaining the operation of FIG. 1, and one procedure of image retrieval according to the present invention will be explained below in accordance with this flowchart.

An example of the screen configuration of the display section 2 is shown in FIG. Figure 3
, 10, 11, 12, 13, and 14 are windows for displaying searched candidate images, and 15 is a window for displaying the image selected first among the candidate images and its index. It is a window. Reference numeral 16 indicates a button for instructing a search, and 17 indicates a button for instructing to draw a sketch.

Index Display (2) Initially, the window 15 of the display section 2 displays the basic shape of a coffee cup index as shown in FIG. The main body 40 and the handle 41 are described using different hyperquadratic functions. Further, the left half of the hyperquadratic function describing the handle 41 is embedded in the main body 40.

Display of image structures and index ■Using the light pen 3, the user presses the search button 16 on the display section 2.
instruct. The light pen control device 4 detects the designated coordinate point and transfers it to the coordinate position detection means 50. The coordinate position detecting means 50 detects from the input coordinates that the search button 16 has been designated, and issues instructions to the index reading means 51 and the index comparing means 53. The index reading means 51 reads out the index of the image stored in the storage section 1. At this time, the simultaneously read images are sent to image rearranging means 56. The index comparing means 53 compares the parameter value of the index currently displayed on the window 15 and the value of the index of the image read out from the storage section 1. In other words, the difference between both parameter values is calculated as a distance. The calculated distance of each image is sent to the image rearranging means 54, and the images read from the storage unit 1 are rearranged in order of decreasing distance. Then, the image with the smallest distance is displayed in window 15, and the second and subsequent images are displayed in windows 10-14.

If the user wants to detect a cup with a slightly different shape, he first points the button 17 on the display unit 2 using the light ben 3. The designated coordinate values are sent from the light pen control device 4 to the coordinate position detection means 50, and selection of the button 17 is detected from the position of the coordinate point. Then, the coordinate position detection means 50 sends a command to the index display means 51, and the index reading means 51 reads the index of the image currently displayed on the window 15 from the storage section 1, and uses the transformation matrix M to read out the index of the image currently displayed on the window 15. By displaying it on the overlay screen of the window 15, the image structure and the index are superimposed. 42 in FIG. 5 is the image of the retrieved coffee cup, and 43 shows the shape when the index assigned to that image is displayed in an overlay. In reality, these 42 and 43 are superimposed, 44, the portion where the index is superimposed is displayed with the color of the image slightly changed.

Sketch Input ■ Next, on the window 15, use the light pen 3 to sketch the outline of the part whose shape you want to change, as shown in FIG. In Figure 6, the outline of the top of the cup body is not circular, but is transformed into a square-shaped cup (Figure 6
60) is shown.

The sketched data is sent to the coordinate position detecting means 50, and the positional relationship between each index displayed on the window 15 and the sketched coordinate data string is checked.

Two-Dimensional Projection■ First, an instruction is sent to the two-dimensional projection means 52 to project the contour shape of the top surface of the three-dimensional index (corresponding to the cross-sectional shape of the xy plane of the hyperquadratic function) and the contour shape of the side surface (corresponding to the cross-sectional shape of the (equivalent to the cross-sectional shape on a plane including the z-axis of the following function) are separately perspectively projected onto the overlay plane, and the contour shape is extracted. In other words,
Each cross-sectional shape is perspectively projected onto the overlay plane. 71 and 72 in FIG. 7 are outlines of the top and side parts of the main body;
3 and 74 are examples in which the contours of the upper and side surfaces of the handle are taken out. Coordinate data in the coordinate system in which the image is displayed (coordinate data on the overlay) is obtained from the extracted contour data, and this coordinate data is used in the correlation calculation means 5.
Sent to 3.

Correlation calculation (2) The correlation calculation means 53 calculates the correlation between the coordinate data of the extracted contour shape and the coordinate data of the sketch. 7 in Figure 7
5 is the correlation with the contour of the top surface of the index of the cup body;
76 is the correlation with the lateral contour of the Kakitan general index, 7
7 is an example in which the correlation between the coordinate values on the screen of the window 15 (on the perspective screen) is calculated, ie, the correlation with the top contour of the cup handle, and 78 with the side contour of the index of the cup handle. . Then, the contour shape part with the highest correlation is judged to be the shape meant by the sketch (
75 in Figure 7). In the case of the sketch in FIG. 6, the correlation with the upper contour 71 of the main body portion of the cup is greatest, and the sketch is determined to be the shape of the upper surface of the cup.

Coordinate transformation of coordinate data (2) and function approximation (2) The sketched coordinate data string is sent to the function approximation means 54. First, since the sketch shape has been detected by the correlation calculation means 53 to be the shape of the top surface of the cup, the shape of the input sketch is the shape of the cup viewed from directly above, which is converted into the shape of the cup to be searched by the transformation matrix M. This corresponds to the shape of the result of conversion. Therefore, the function approximation means 54 uses matrix 1/
Inverse transformation is performed using M to obtain coordinate data on the xy plane. 80 in FIG. 8 is the input sketch, and 81 is the sketch data on the xy plane (the plane seen from directly above) that has been inversely transformed by the matrix 1/M. Then, a two-dimensional hyperquadratic function, that is, the following equation 2, is approximated to the coordinate data of the transformed sketch.

[Math 2]

The result of approximation of the above equation 2 is shown at 82 in FIG. 83 shows a sketch, and the thick line 84 is the shape of the approximated function.

Parameter comparison ■ and search ■ Since the parameters (a3, a4, ε) of the function obtained by approximation are the shape of the upper surface of the main body, the parameters (a1, a2) of the hyperquadratic function that are the index , ε1), and the parameters other than these three are read out from the storage unit 1 and the parameter values obtained from the sketch based on the index values of the displayed image that have not been changed by the sketch. The distance between the image and the index value is calculated by the index comparison means 55. Then, based on the calculated distance, the image rearranging means 5
6, the images are rearranged in descending order of distance, and the images are displayed on the display unit 2. In other words, a coffee cup whose mouth portion is nearly square as shown in FIG. 9 is retrieved.

It should be noted that the present invention is not limited to the above embodiments. For example, although an example of searching for a three-dimensional shape using a hyperquadratic function has been shown, the search need not be a hyperquadratic function or three-dimensional. It is easy to apply this embodiment to a two-dimensional shape, and any function that can express a three-dimensional shape (or two-dimensional shape) can be applied to the present invention, and is not specified.

In addition, in the embodiment, the shape is specified by sketching one part, but it is also possible to draw sketches of multiple parts at the same time.Furthermore, when specifying the entire shape, it is possible to draw the sketch of the entire outline. You can search by entering a sketch.

In addition, in the embodiment, only the side and top surfaces of the index are compared with the sketch to determine the position of the sketch, but it is also possible to further compare the right side, left side, bottom, etc. Not allowed.

[0026]

[Effects of the Invention] As described above, according to the present invention, a search can be performed simply by specifying the shape to be searched for by sketching it on an image, and there is no need to describe the shape in words or keywords. This has the effect of making instructions easier. Further, by specifying a partial shape by sketching based on an image that has already been searched, an excellent effect can be achieved in searching by partial shape. Further, in the present invention, by displaying the image and the index at the same time, it is possible to easily determine which part of the image structure is assigned as the index.

Furthermore, the method of deforming an index using a sketch according to the present invention can also be applied to shape operations in three-dimensional modeling.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing the configuration of an apparatus according to an embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of FIG. 1;

FIG. 3 is an example of a screen configuration of the display unit 2. FIG.

FIG. 4 shows the default shape of the coffee cup index.

FIG. 5 is an example of displaying an image and an index at the same time.

FIG. 6 is an example of a sketch.

FIG. 7 is an example of the shape of each top surface and side surface of the index.

FIG. 8 is an explanatory diagram showing an example of approximating a function to a sketch.

FIG. 9 shows searched coffee cups.

[Explanation of symbols]

1 Image/index storage section 2 Image/index display area 3. Light pen 4 Light pen control device 5 Control section 50 Coordinate position detection means 51 Index reading means 52 Two-dimensional projection means 53 Correlation calculation means 54 Function approximation means 55 Index comparison means 56 Image rearrangement means

Claims (1)

[Claims] Claim 1: In an image database comprising an image storage section in which indexes are assigned to image structures using function-based shape representation, means for superimposing and displaying the index on the image, and inputting a sketch of a contour shape. means for correlating the input sketch with an index of the displayed image to approximate a function of the index to the sketch; and assigning an index close to a value of the index changed by the approximation. 1. An image retrieval method using a sketch, comprising means for retrieving the retrieved image from the image storage section.