JPH10207618A - User interface device and instruction input method - Google Patents

Abstract

(57) [Summary] [Problem] To provide a user interface device which performs input by image processing and which is easier to use and which reduces a user's operational burden. A user interface device for performing input by image processing of an input image, comprising: means for switching between a pointing mode and other modes based on an image processing result of the input image. And A user interface device for performing input by image processing of an input image, wherein at least a mode for switching a mode of moving a cursor, a mode of selection, and a mode of double clicking is switched based on an image processing result of the input image. It is characterized by having.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a user interface device and an input method for inputting by image processing.

[0002]

2. Description of the Related Art A mouse is overwhelmingly used as an input device of a computer. However, operations that can be performed with the mouse include moving a cursor and selecting a menu, and are merely useful as a two-dimensional pointing device. The mouse can handle two-dimensional information, and it is difficult to select a deep object such as an object in a three-dimensional space. Also, when creating an animation, it is difficult to make a natural movement with an input device such as a mouse to make the movement of the character.

In order to compensate for the difficulty of pointing in a three-dimensional space, a device for inputting information in six axes by pushing or rotating a ball in a desired direction, or a hand such as a so-called data glove, data suit, or cyber glove. Devices that can be used in such devices have been developed. However, these devices are not as widely used as initially expected due to poor operability and the like.

On the other hand, recently, a direct instruction type input device has been developed in which a user can input information intended by the user by hand or gesture without handling a special device.

For example, light is emitted, light reflected by a user's hand is received, and this is imaged to perform feature extraction and shape recognition processing, thereby executing control according to the shape of the hand, and moving the hand. Some of them move the cursor by an amount corresponding to the amount or change the viewpoint in the three-dimensional model.

[0006] Alternatively, there is a method in which the same processing as described above is performed by capturing a video of a user's hand motion and analyzing the video image.

[0007] With such a device, the user can easily input by gesture without wearing a special device.

[0008]

However, in this type of apparatus, each mode such as a cursor movement mode, a selection mode, and a double click mode is fixedly used, and in order to change the mode, the mode is changed. For this reason, an explicit operation is required, and there is a problem that the user is burdened with the operation.

The present invention has been made in consideration of the above circumstances, and provides a user interface device for performing input by image processing, which is easier to use and has a less user-friendly operation, and an instruction input method. The purpose is to do.

[0010]

SUMMARY OF THE INVENTION The present invention relates to a user interface device for performing input by image processing of an input image, wherein a pointing mode and other modes are set based on the image processing result of the input image. And means for switching by switching.

The present invention also relates to a user interface device for performing input by image processing of an input image, wherein at least a mode for moving a cursor, a mode for selection, and a mode for double-clicking are set in an image of the input image. A switching means is provided based on the processing result.

Preferably, the apparatus further comprises means for designating a recognition method (recognition engine) which restricts image processing contents for each selectable object in the selection mode, and performs image processing of the input image for the selected object. In doing so, image processing may be performed according to the recognition method specified for the object.

Preferably, for each selectable object in the selection mode, means for designating a recognition method (recognition engine) which restricts image processing contents, the object being located in the vicinity of the displayed object indicated by the cursor And means for presenting information indicating the recognition method designated for.

Preferably, the apparatus may further comprise means for presenting an image processing result of the input image in a predetermined shape on a cursor.

Further, the present invention provides a first method for inputting a reflection image.
A user interface device comprising: a first device and a second device for performing input by image processing of an input image, wherein the second device restricts image processing of an input image with respect to the first device. Means for specifying a recognition method (recognition engine), wherein the first device performs means for performing predetermined image processing based on the specification of the recognition method, and converts the input image and the image processing result to the second Means for returning to the apparatus.

Preferably, when the first device does not have an image processing means (recognition engine) suitable for a necessary recognition method, the second device stores information necessary for image processing suitable for the recognition method in the second device. The information processing apparatus may further include a unit that requests transfer of a (recognition engine), and the second device may further include a unit that transfers the requested information to the first device.

Preferably, each of the first device and the second device is activated by another device when information necessary for image processing suitable for a predetermined recognition method in the own device is activated first. Means for notifying the information that has become inactive, and, when receiving a notification from another apparatus to inactivate information necessary for image processing suitable for the predetermined recognition method, deactivating the information. Means may be further provided.

The present invention is also a method for inputting an instruction by a user interface device for inputting by image processing of an input image, wherein the input image of a target object is image-processed, and pointing is performed based on the result of the image processing. The mode is switched between a mode to be performed and another mode.

Further, the present invention provides a first method for inputting a reflection image.
An instruction input method by a user interface device including an apparatus and a second apparatus for performing input by image processing of an input image, wherein an image processing content of the input image is transmitted from the second apparatus to the first apparatus. The first device performs a predetermined image processing based on the specification of the recognition method, and converts the input image and the image processing result into the second image. It is returned to the device.

According to the present invention, a cursor movement mode,
The user does not need to perform an explicit operation for changing each mode such as the selection mode and the double-click mode.

Further, since the point designated by the user is read by the recognition processing and reflected on the movement of the cursor on the screen, the calibration by the user's operation is unnecessary.

Further, if an image processing means (recognition engine) suitable for a necessary recognition method is used, an improvement in input accuracy and an improvement in user operability can be expected.

If the input image is displayed translucently on the cursor after being made translucent, the operation status can be fed back to the user.

As described above, according to the present invention, it is possible to provide an easy-to-use user interface device in which the operational burden on the user is reduced.

Further, according to the present invention, since a certain degree of recognition processing is performed on the first device side (device side), the load can be distributed and the speed of the overall recognition processing can be improved.

Further, it is possible to enhance the function of a device having an image input function.

The invention relating to each apparatus described above is also valid as an explanation of a method.

Further, the above-mentioned invention is also realized as a machine-readable medium storing a program for causing a computer to execute the corresponding procedure or means.

[0029]

Embodiments of the present invention will be described below with reference to the drawings.

First, a first embodiment will be described.

FIG. 1 is a diagram showing a configuration example of a user interface device according to an embodiment of the present invention.

This user interface device is, for example,
It is suitable to be applied to a computer having a graphic user interface. That is, icons such as a cursor, a slider bar, a scroll bar, a pull-down menu, a box, a link, and an application are displayed on the display screen, and a user inputs an instruction such as moving a cursor, selecting an icon, or starting an application using an input device. In this system, an input device receives an input by performing image processing on an object such as a user's hand without requiring a dedicated device such as a mouse.

In the present embodiment, the shape of the light reflected by the target object such as the user's hand is captured as an image (or the light reflected from the target object of the background light is captured as an image). , Movement, distance information, and the like, and performs predetermined control (for example, control on an input / output device or activation of application software) according to the shape or the like. The user performs an intended input by hand motion or the like. It provides functions that can be used. In addition, since each mode such as cursor movement, icon selection, and application startup is switched according to the image processing result, the user does not need to perform an explicit operation for switching each mode.

This user interface device comprises an image input unit 10, an image storage unit 11, a shape interpretation unit 12, an interpretation rule storage unit 13, a presentation unit 14, and a cursor switching unit 15.

FIG. 2 shows an example of the configuration of the image input unit 10.

The image input unit 10 includes, for example, a light emitting unit 101 for irradiating a target object with light such as near-infrared light by a light emitting element such as an LED, and a light receiving element in which reflected light from the target object is arranged in a two-dimensional array. Light extraction unit 102 that receives light at
And a timing control unit 103 for controlling operation timings of the light emitting unit 101 and the reflected light extracting unit 102. By taking the difference between the amount of light received by the reflected light extraction unit 102 when the light emitting unit 101 is emitting light and the amount of light received by the reflected light extraction unit 102 when the light emitting unit 101 is not emitting light , Correcting the background, and emitting light 1
Only the component of light reflected from the target object of light from 01 is extracted. Note that the image input unit 10 has no light emitting unit,
It may have only a light receiving unit such as a D camera.

FIG. 3 shows the relationship between the display device 20, the housing 8 of the image input unit 10, and the target object 22. For example, when the user's hand 22 is brought in front of the image input unit 10, a reflected light image from the hand is obtained. At this time, each pixel value of the reflected light image is affected by the properties of the object (specular reflection, scattering, absorption, etc. of the light), the direction of the object surface, the distance of the object, and the like. When an object scatters light uniformly, the amount of reflected light has a close relationship with the distance to the object. Since a hand or the like has such a property, the reflected light screen when the hand is extended reflects the distance of the hand, the inclination of the hand (partially different distance), and the like. Therefore,
By extracting such information, input and generation of various information becomes possible.

The image storage unit 11 receives a predetermined time (for example, every 1/30 second, every 1/60 second, 1/1
(For example, every 00 seconds), and sequentially stores two-dimensional images of the image detection target object.

The shape interpretation unit 12 sequentially takes in the two-dimensional images stored in the image storage unit 11 as an N × N (for example, 64 × 64) dot matrix. Each pixel has a gradation (for example, 8 bits = 256 gradations).

The shape interpreting unit 12 extracts a predetermined feature amount from the dot matrix, and interprets the shape based on the interpretation rules stored in the interpretation rule storage unit 13. Then, an instruction corresponding to the matching interpretation rule is output as an interpretation result. If there is no matching interpretation rule, the method of extracting a predetermined feature amount from the dot matrix is changed as necessary (for example, when performing threshold processing of the dot matrix, the threshold value is changed). Alternatively, the matching process may be performed again. If there is no finally applicable interpretation rule, there is no input.

The interpretation rule storage unit 13 stores interpretation rules for shape interpretation. For example, a feature amount of a target object such as a user's hand in a dot matrix, for example, a predetermined value such as a shape, an area, an uppermost point, and a center of gravity, and an instruction corresponding thereto are stored as interpretation rules. Instructions include icon selection, application activation, cursor movement, and the like. In the case of cursor movement, the movement amount of the cursor according to the movement direction and distance of the hand is also instructed.
For example, the state where the thumb and the index finger are opened and set up corresponds to the cursor movement (in this case, for example, the movement distance / direction of the tip of the index finger corresponds to the movement distance / direction of the cursor), and the thumb and the index finger are closed and set up The state where the cursor is positioned corresponds to the selection of the icon where the cursor is located, the thumb and forefinger are raised, and the state where the palm is inverted in the case of cursor movement corresponds to the activation of the application corresponding to the icon where the cursor is located Such rules are conceivable.

Representative examples of the extraction of the characteristic amount from the dot matrix in the shape interpretation by the shape interpretation unit 12 include extraction of distance information and extraction of an area. If the object has a uniform and uniform scattering surface, the reflected light image can be regarded as a distance image. Therefore, the three-dimensional shape of the object viewed from the light receiving unit can be extracted. If the object is a hand, the inclination of the palm can be detected. The tilt of the palm appears as a partial difference in distance. If the pixel value changes when the hand is moved, it can be regarded that the distance has moved. Also, since there is almost no reflected light from a distant object such as the background, the shape of the object can be easily cut out by processing to cut out an area above a certain threshold from the reflected light image. For example, if the object is a hand, it is extremely easy to cut out a silhouette image thereof. Even in the case of using a distance image, it is often the case that an area is extracted once using a threshold, and then the distance information within the area is used.

There are various methods of matching the feature quantity extracted from the dot matrix with the interpretation rule. For example, vectorization for extracting a vector from an image, extraction of a deformed state of a shape based on a shape model, spectrum analysis based on a distance value on a scanning line, and the like.

If there is no suitable shape, the matching process may be performed again, for example, by changing the threshold value. If there is no final conforming shape, it is assumed that there is no input.

When the shape interpreting unit 12 recognizes that the instruction is to activate an application or a function of an OS, the software is activated.

The presenting unit 14 is a display device, and the shape interpreting unit 1
2. The presentation reflecting the interpretation result by No. 2 is performed. For example, a cursor is moved, and a message is presented as necessary.

The cursor switching unit 15 controls cursor switching based on the result of interpretation by the shape interpretation unit 12.

FIGS. 4 to 6 show examples of operation procedures of the user interface device of the present embodiment.

First, the control state C of the cursor is initialized (C
← 0), the selection state S is initialized (S ← 0), the cursor information I is initialized (I ← 0), and the recognition engine flag R is initialized (R ← 0) (step S1).

Next, the reflection image is read and the image storage unit 11 reads the reflection image.
(Step S2).

Next, a dot matrix is read into the shape interpreting section 12 (step S3).

Next, the mode indicated by the gesture is determined from the feature quantity extracted from the dot matrix and the interpretation rule by the shape interpretation unit 12 (step S4).

Thereafter, the processing is divided depending on the result of the determination and the parameter value.

Gesture of cursor control and C = 0
And S = 0 and R = 0 (step S
5) This is a process for entering the cursor control. C ← 1 is set (step S11), and the process returns to step S2.

The cursor control gesture is, for example, as shown in FIG.
Is recognized.

Gesture of cursor control and C = 1
And S = 0 and R = 0 (step S
6) Processing during cursor movement. In this case, first
The coordinates (x, y) of the neighboring point are calculated from the dot matrix (step S12), and the cursor is moved to the calculated coordinates (x, y) (step S13). Then, the calculated coordinates (x, y) are held. It is assumed that Cp = (x, y) (step S14). If there is an object in Cp (step S15), the state of the object is set (I ← object state) (step S16). If there is no object (step S15), I = O is set (step S17). The process returns to step S2.

Cursor control gesture and C = 1
If S = 1 (step S7), the process returns to the cursor control. S ← 0, R ← 0, I ← 0 (step S18), and the process returns to step S2.

Gesture of selection and C = 1 and
If S = 0 and R = 0 (step S8), the process is for selecting an object. In this case, first, S ← 1
(Step S19), an object closest to Cp is searched (Step S20), and the searched object is selected (Step S21). Here, if the selected object has a designated recognition engine (step S
22), R ← 1 (step S23), and step S2
Return to When there is no designated recognition engine, if the object is a link object (step S24), the link destination is skipped (step S25), C ← 0, S ← 0, I ← 0 (step S26), and the process returns to step S2. If it is not a link object (step S24), the process returns to step S2.

The selection gesture is, for example, when the hand shape shown in FIG. 11 is recognized.

Here, the recognition engine extracts a predetermined characteristic amount from the dot matrix, as will be described in detail later. For example, the recognition engine calculates the vertical movement amount of the highest point of the target object shape in the dot matrix. There are various things, such as the highest point vertical engine to extract.

If the gesture of selection and R = 1 (step S9), this is a process of moving the selected object, and recognition is performed along the recognition object (step S27), and the process returns to step S2.

Double-click gesture and C =
If it is 1 (step S10), it is a double click process. In this case, the object closest to Cp is opened (step S28), and C ← 0, S ← 0, I ← 0, R ← 0.
(Step S29), and returns to Step S2.

The double-click gesture is, for example, when the hand shape shown in FIG. 14 is recognized.

In cases other than the above, other recognition processing is performed (step S30), and the process returns to step S2.

Hereinafter, the present embodiment will be described using a specific example.

First, when the shape of the hand with the thumb and forefinger extended as shown in FIG. 7 is recognized, the cursor movement is instructed.

For example, in the state of FIG. 8A, the shape of the hand in FIG.
When the user moves the hand shape in the state shown in FIG. 7, the cursor 201 moves on the display screen in response to the movement. In this case, a certain point in the dot matrix of the shape of the user's hand, for example, the highest point where the vertical position in the image is the highest (for example, corresponding to the tip of the index finger), or the most light receiving portion in the image The moving amount / direction at the nearest point (for example, the point with the highest gradation) close to is extracted.

As shown in FIG. 8B, the input image 202 may be displayed translucently on the cursor 201, and the recognition status may be fed back to the user.

When the cursor is on various objects such as a slider bar and a link node, the shape of the cursor is changed so as to exhibit its function.

For example, as shown in FIG. 9A, when the slider bar is located at the position where the cursor has moved, the shape of the cursor is deformed as indicated by 203 in the figure. FIG. 9 (b)
, The input image may be displayed on the cursor as the translucent 205.

Here, the arrow indicated by 204 in FIG. 9A indicates that the operation of the slider bar to be operated is restricted in the vertical direction. That is, a vertical engine is specified for the slider bar. In this case, no matter how the user moves his hand, only the movement in the vertical direction is recognized.

When a link node is located at a position where the cursor has moved as shown in FIG. 10A, the shape of the cursor is changed as shown at 206 in the figure. Note that FIG.
As shown in (b), the input image 207 may be displayed translucently on the cursor.

Next, after the cursor is moved to a desired position with the hand shape shown in FIG. 7, the index finger is extended as shown in FIG. 11, and when the shape of the hand with the thumb closed is recognized, it is indicated by the cursor. It is assumed that selection of an object to be performed is instructed. This is equivalent to a single mouse click.

For example, in the state as shown in FIG.
With a hand shape of one, a slider bar is selected. The arrow indicated by 208 in the figure indicates that the operation of the slider bar to be operated is restricted in the vertical direction. As shown in FIG. 12B, the input image 209 may be displayed translucently on the cursor.

Further, for example, in the state of FIG.
In this case, the link node “company profile” is selected, and the display content changes as shown in FIG. In addition,
As shown in FIG. 13B, the input image 210 may be displayed translucently on the cursor.

Next, after the cursor is moved to a desired position with the hand shape shown in FIG. 7, it is recognized that the wrist is rotated 180 degrees without changing the hand shape itself as shown in FIG. First, a double click is instructed for the object indicated by the cursor.

For example, in FIG. 15A, the cursor on the "Internet" icon is moved in the shape of the hand in FIG. 7, and then the hand is rotated as shown in FIG. Then, a double click of "Internet" is accepted. FIG. 15B shows a state immediately before an icon is selected and the icon is opened. As shown in FIG. 16, the input image 212 may be displayed translucently on the cursor.

For example, first, the cursor is moved to “file” as shown in FIG. At this time, as shown in FIG.
3 may be displayed translucently.

Next, when “File” is selected, FIG.
As shown in (a), a pull-down menu is displayed. Here, the arrow indicated by 214 in the figure indicates that the operation of the pull-down menu to be operated is restricted in the vertical direction.

Next, as shown in FIG. 18 (a), the cursor is positioned on "save" and the hand is shaped as shown in FIG. 14, so that the double-click of "save" is accepted. Here, as shown in FIG. 18B, the input image 213 may be displayed translucently on the cursor.

When "Save" is double-clicked, the shape of the cursor is changed, for example, as shown at 216 in FIG. This indicates that the document is being saved by selecting the save. Again, as shown in FIG.
The input image 217 may be displayed translucently on the cursor.

Further, for example, first, as shown in FIG. 20A, the cursor is moved to "file", "file" is selected, further moved to print, and "print" is selected. Then, a menu corresponding to “print” is displayed. At this time, as shown in FIG. 20B, the input image 219 may be displayed translucently on the cursor.

Next, the recognition engine will be described.

FIG. 21 shows a flowchart of processing relating to the recognition engine.

In this embodiment, a recognition engine is specified for each object as needed.

The recognition engine extracts a predetermined feature amount from the dot matrix. That is, if there is a recognition engine specified for the selected object (step S31),
The feature is extracted from the dot matrix according to the specified recognition engine. If no recognition engine is specified (step S31), normal recognition is executed (step S32).

The recognition engine includes, for example, a nearest point vertical engine 121 for extracting the vertical movement amount of the nearest point of the target object shape in the dot matrix, and a nearest point horizontal engine for extracting the horizontal movement amount of the nearest point. Direction engine 12
2. The nearest point diagonal engine 123 for extracting the amount of diagonal movement of the nearest point, the centroid vertical engine 124 for extracting the amount of vertical displacement of the center of gravity, and the centroid for extracting the amount of horizontal movement of the center of gravity. Point horizontal engine 125, center-of-gravity point diagonal engine 126 for extracting the amount of movement of the center of gravity in a diagonal direction, center-of-gravity point vertical engine 127 for extracting the amount of movement of the center of gravity in the vertical direction, and amount of horizontal movement of the center of gravity Center-of-gravity point horizontal direction engine 128 for extracting the center-of-gravity point diagonal direction engine 12 for extracting the amount of movement of the center-of-gravity point in the diagonal direction
9. Edge extraction engine 130 for extracting the edge of the target object shape in the dot matrix, area calculation engine 131 for calculating the area of the target object shape in the dot matrix, rotation of the nearest point of the target object shape in the dot matrix around the x-axis The nearest point x-axis rotation angle engine 132 for extracting the angle, the nearest point y-axis rotation angle engine 133 for extracting the y-axis rotation angle of the nearest point, and the nearest point z-axis for extracting the nearest point z-axis rotation angle Rotation angle engine 134,
A centroid x-axis rotation angle engine 135 for extracting the centroid point x-axis rotation angle, a centroid point y-axis rotation angle engine 136 for extracting the centroid point y-axis rotation angle, a z-axis rotation angle for the centroid point The rotation angle engine 13 about the center of gravity point z axis
7. A centroid x-axis rotation angle engine 138 for extracting the x-axis rotation angle of the centroid point, a centroid point y-axis rotation angle engine 139 for extracting the y-axis rotation angle of the centroid point, and a z point for the centroid point
A variety of things can be considered, such as a center-of-gravity point z-axis rotation angle engine 140 for extracting the rotation angle around the axis, and a recognition engine 141 based on a predetermined engine weighting combination.

FIG. 22 shows an example of a description of a recognition engine for each object.

As described above, according to the present embodiment, an explicit operation by the user for changing each mode such as the cursor movement mode, the selection mode, and the double-click mode becomes unnecessary.

Further, since the point designated by the user is read by the recognition process and reflected on the movement of the cursor on the screen, the calibration by the user's operation is unnecessary.

Further, if a recognition engine is used, improvement in input accuracy and improvement in operability of a user can be expected.

Further, if the input image is made translucent and displayed over the cursor, the operation status can be fed back to the user.

As described above, according to the present embodiment, it is possible to provide a more user-friendly user interface device in which the operational burden on the user is reduced.

Next, a second embodiment will be described.

The second embodiment is basically the same as the first embodiment. However, in the present embodiment, a part of the recognition processing in the image input unit side housing (hereinafter referred to as the device side) is performed. And a dot matrix of the input image and a predetermined recognition result are passed from the image input unit 10 side to the main body side. It is desirable that the recognition process performed on the device side be a light load process.

FIG. 23 shows a configuration example of the interface device according to the present embodiment.

In FIG. 23, the main body control section 32,
The presentation unit 14 and the cursor switching unit 15 are arranged, and the image input unit 10, the image storage unit 11, the recognition engine control unit 30, the active list 31, and some predetermined recognition engines 121, 122, 142, and 143 are provided on the device side. , 144
Is arranged.

The main body control unit 32 corresponds to, for example, the shape interpretation unit 12 and the interpretation rule storage unit 13 (including the recognition engine) in FIG. 1, but is not limited thereto, and has another configuration. , Even if it performs other recognition processing,
It does not matter if it uses a recognition engine.

FIG. 24 shows an example of the description of the active list storage unit when the vertical slider bar is selected. In this case, it is indicated that the cursor engine 142 and the closest point vertical direction engine 121 are designated.

In such a configuration, a list of recognition engines to be activated or a list of recognition engines to be deactivated are transmitted from the main body to the device. On the device side, this list is stored in the active list storage unit 31, a predetermined feature amount is extracted from the input image as a recognition result by the recognition engine control unit 30 according to the specified recognition engine, and the input image and the recognition result are stored in the main unit. return.

According to this embodiment, since a certain degree of recognition processing is performed on the device side, the load can be distributed, and the speed of the overall recognition processing can be improved.

Further, the function of a device having an image input function can be enhanced.

FIG. 27 shows still another configuration example of the interface device according to the present embodiment.

In FIG. 27, the configuration in the main unit is almost the same as that in FIG. 25. However, in this configuration example, when the recognition engine exists in a plurality of places such as the main unit side and the device side, it becomes active first. The side having the recognition engine communicates with the other side having the same recognition engine so as to deactivate the recognition engine.

FIG. 28 shows an example of the operation procedure of the active notification in this case, and FIG. 29 shows an example of the operation procedure of the active notification reception.

First, an image is input on the device side (step S39), and the recognition is executed on the main body side and / or the device side (step S40). Then, the side that has performed the recognition sends the recognition result, the image matrix, and the active list (or the inactive list) to the notification partner (step S41).

Next, the side receiving the notification receives the recognition result, the image matrix, and the active list (or the inactive list) (step S42), and recognizes the ON recognition engine (or the inactive list in the received active list). Is rewritten to OFF in the active list held (step S).
43). Then, other processing is executed as needed (step S44).

[0108] Each of the above functions can be realized as software. Further, the present invention can be embodied as a machine-readable medium storing a program for causing a computer to execute the above-described procedures or means.

The present invention is not limited to the above-described embodiments, but can be implemented with various modifications within the technical scope.

[0110]

According to the present invention, each mode, such as a cursor movement mode, a selection mode, and a double-click mode, is switched based on the image processing result, so that an explicit operation by the user for changing the mode is performed. Becomes unnecessary.

Further, since the point designated by the user is read by the recognition process and reflected on the movement of the cursor on the screen, the calibration by the user's operation is unnecessary.

As described above, according to the present invention, it is possible to provide an easier-to-use user interface device in which the operational burden on the user is reduced.

Further, according to the present invention, since a certain degree of recognition processing is performed on the first device side (device side), the load can be distributed and the speed of the overall recognition processing can be improved.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration example of an interface device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a configuration example of an image input unit;

FIG. 3 is a diagram for explaining a relationship between a display device, a housing of an image input unit, and a target object;

FIG. 4 is an exemplary flowchart illustrating an example of an operation procedure of the user interface device of the embodiment.

FIG. 5 is an exemplary flowchart illustrating an example of an operation procedure of the user interface device of the embodiment.

FIG. 6 is an exemplary flowchart illustrating an example of an operation procedure of the user interface device of the embodiment.

FIG. 7 is a diagram showing an example of an input image showing a gesture of cursor control;

FIG. 8 is a diagram showing a screen display example.

FIG. 9 is a diagram showing a screen display example.

FIG. 10 is a diagram showing a screen display example.

FIG. 11 is a diagram showing an example of an input image showing a gesture of selection

FIG. 12 is a diagram showing a screen display example.

FIG. 13 shows a screen display example.

FIG. 14 is a diagram showing an example of an input image showing a double-click gesture.

FIG. 15 is a diagram showing a screen display example.

FIG. 16 shows a screen display example.

FIG. 17 is a diagram showing a screen display example.

FIG. 18 is a diagram showing a screen display example.

FIG. 19 is a diagram showing a screen display example.

FIG. 20 is a diagram showing a screen display example.

FIG. 21 is a diagram for explaining processing according to a specified recognition engine;

FIG. 22 is a diagram illustrating an example of a description of a recognition engine for each object.

FIG. 23 is a diagram illustrating a configuration example of an interface device according to a second embodiment of the present invention;

FIG. 24 is a diagram showing an example of a description in an active list storage unit when a vertical slider bar is selected.

FIG. 25 is a diagram showing another configuration example of the interface device according to the second embodiment of the present invention.

FIG. 26 is an exemplary flowchart illustrating an example of the operation procedure of the user interface device of the embodiment.

FIG. 27 is a diagram showing still another configuration example of the interface device according to the second embodiment of the present invention;

FIG. 28 is an exemplary flowchart illustrating an example of an operation procedure of the user interface device of the embodiment.

FIG. 29 is an exemplary flowchart illustrating an example of the operation procedure of the user interface device of the embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Image input part 11 ... Image storage part 12 ... Shape interpretation part 13 ... Interpretation rule storage part 14 ... Presentation part 15 ... Cursor switching part 101 ... Light emitting part 102 ... Reflected light extraction part 103 ... Timing control part 20 ... Display device 8 ... Case 30 ... Recognition engine control unit 31, 34 ... Active list storage unit 32 ... Main body control unit 33 ... Recognition engine storage unit 121 ... Nearest point vertical direction engine 122 ... Nearest point horizontal direction engine 142 ... Cursor engine 143 ... Selection engine 144: Double-click engine 151-15N: Recognition engine

Continued on the front page (72) Inventor Shunichi Numazaki 1 Tokoba Toshiba-cho, Komukai-ku, Kawasaki-shi, Kanagawa Inside the R & D center of Toshiba Corporation

Claims (10)

[Claims] 1. A user interface device for performing input by image processing of an input image, comprising: means for switching between a pointing mode and other modes based on an image processing result of the input image. Characteristic user interface device. 2. A user interface device for performing input by image processing of an input image, wherein at least a mode of moving a cursor, a mode of selection, and a mode of double clicking are set based on an image processing result of the input image. A user interface device, comprising: means for switching by switching. 3. An image processing apparatus according to claim 1, further comprising means for designating a recognition method in which image processing content is restricted for each selectable object in the selection mode. 3. The user interface device according to claim 1, wherein image processing is performed in accordance with a recognition method specified for. 4. A means for designating a recognition method which restricts image processing contents for each selectable object in a selection mode, and a recognition designated for an object in the vicinity of a displayed object indicated by a cursor. 3. The user interface device according to claim 1, further comprising: means for presenting information indicating a method. 5. The user interface device according to claim 1, further comprising means for presenting an image processing result of the input image in a predetermined shape on a cursor. 6. A user interface device comprising: a first device for inputting a reflection image; and a second device for inputting by image processing of the input image, wherein the second device is connected to the first device. In response to the input image, there is provided means for designating a recognition method which restricts image processing content of the input image, wherein the first device performs predetermined image processing based on the designation of the recognition method, Means for returning an image processing result to the second device. 7. The first device requests the second device to transfer information necessary for image processing suitable for the recognition method, when the first device does not hold image processing means suitable for the required recognition method. 7. The user interface device according to claim 6, further comprising: a unit; and wherein the second device further includes a unit configured to transfer the requested information to the first device. 8. The information processing apparatus according to claim 1, wherein the first device and the second device are activated by another device when information necessary for image processing suitable for a predetermined recognition method in the device is activated first. Means for notifying the information that has become inactive, and, when receiving a notification from another device to deactivate information necessary for image processing suitable for the predetermined recognition method, deactivating the information. 7. The user interface device according to claim 6, further comprising: 9. A method for inputting an instruction by a user interface device for performing input by image processing of an input image, comprising: a mode in which an input image of a target object is image-processed; An instruction input method characterized by switching between other modes. 10. A method for inputting an instruction by a user interface device including a first device for inputting a reflection image and a second device for inputting the input image by image processing, wherein the second device receives the first image from the second device. The first device performs a predetermined image processing based on the specification of the recognition method, and performs the predetermined image processing on the input device. And returning a processing result to the second device.