JPH09237151A - Graphical user interface - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a graphical user interface with small quantity of operation restriction. SOLUTION: A user puts a red cap 1 on a finger and moves the finger. The motion is photographed by a video camera 1. An information processor 4 detects the position of the picture of the red cap in the photographed picture a mouse cursor 5 on a display screen is moved in accordance with the detected position. When the mouse cursor 5 is positioned at an icon 6 on the display screen, the user utters voice, 'This one' so that voice is inputted in the information processor 4 with a microphone 3 and a control instruction and the selection processing of the icon corresponding to 'This one' are executed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a pointing device and a graphical user interface for inputting information using the pointing device.

[0002]

2. Description of the Related Art Conventionally, a keyboard is often used to input information to an information processing device. Recently, a GUI that uses a pointing device such as a mouse or a touch panel to input a specific command (processing instruction) by specifying the position of the display screen of the display device.
(Graphical user interface) is also proposed. A mouse is used as such a GUI type pointing device, but the mouse has one to three click buttons, which causes confusion for the operator.

Therefore, a pointing motion using a human finger, that is, a motion of pointing an object with a finger is photographed by a video camera, a direction pointed by the finger is identified from the photographed result, and the direction is extended to a three-dimensional space. Then, there is a proposal in which the position on the screen is specified by obtaining the position virtually touched on the screen. The references include the following.

"Man-Machine Interface Evolution" Richard A Bolt (MIT) Translated by Yuji Tsuruoka (Personal Media) "Media Lab" Stuart Brand Co-translated by Kenji Muro / Kumi Aso (Fukutake Shoten) "Virtual Tech Lab" (Industrial) Study group)

[0005]

However, since the position pointing device for detecting the position pointed by the finger in this way takes into consideration the pointing of a specific position on a large screen, the position pointing device is like a display screen of a personal computer. It is not suitable for specifying the position on a small screen.
In the conventional method, it is necessary to identify the position of the finger and the direction in which the finger points in the three-dimensional image, but these are extremely difficult technical problems. Also, with a mouse etc., after specifying the position by moving the mouse,
Instructions such as starting an application program are given by operating a click button on the mouse, but such instructions must rely on the keyboard.

A pointing device such as a mouse requires an operation space such as a desk, and a physically handicapped person who cannot operate a click button of a mouse cannot operate a personal computer.

Therefore, in view of the above points, the present invention has an object to provide a new graphical user interface with a limited number of operations.

[0008]

In order to achieve such an object, the invention according to claim 1 displays an object by a first notation on a display screen of a display device and designates the position of the display screen. In the graphical user interface for displaying the second notation and inputting an instruction about the object of the first notation designated by the position designation by moving the second notation, the position of the moving object is referred to as the object. Is equipped with a position detection means for contactless detection and a display control means for moving and displaying the second notation in correspondence with the detected position and movement direction.

According to a second aspect of the present invention, in the graphical user interface according to the first aspect, input means for inputting a voice instruction for the object of the first notation, and the instruction content from the input voice. It is characterized by further comprising voice recognition means for recognizing and information processing means for executing the recognized instruction content.

According to a third aspect of the present invention, in the graphical user interface according to the second aspect, the object of the first notation that matches the position of the second notation of the first aspect is stored in the information processing means. It is characterized in that it is a processing target.

According to a fourth aspect of the present invention, in the graphical user interface according to the third aspect, the second instruction is given as the instruction for the object according to the second aspect.
When a voice indicating that a processing target of the information processing means is indicated by the position indicated by and the type of the processing are input to the input means by voice, the information processing means executes the type of processing indicated by the voice. It is characterized by doing.

According to a fifth aspect of the present invention, in the graphical user interface according to the fourth aspect, the first notation according to the first aspect is a graphic, and the object schematically displayed by the graphic is a program. Is characterized by.

According to a sixth aspect of the present invention, in the graphical user interface according to the fifth aspect, the type of the processing according to the fourth aspect is activation of a program.

According to a seventh aspect of the present invention, in the graphical user interface according to the fourth aspect, the first notation according to the first aspect is a graphic, and an object schematically displayed by the graphic is a character and / or an image. The data is composed of

According to an eighth aspect of the present invention, in the graphical user interface according to the seventh aspect, the type of the processing according to the fourth aspect is display of characters and / or images.

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 shows the basic configuration of the present invention. The user attaches a member having optical characteristics, in this example, a red rubber cap 1 to the finger. Video camera 2
Takes a picture of this cap 1. The video signal captured by the video camera 2 is input to the information processing device 4. The voice generated by the user is converted into a voice signal by the microphone 3 and input to the information processing device 4.

In the information processing device 4, the input video signal is subjected to image processing to detect the position of the cap 1, and the cap 1
Position and mouse cursor on the display screen (indicated by arrows in the figure)
Correspond the positions of 5. Further, the input voice signal is voice-recognized, and an information processing command instructed by voice is executed.
In this embodiment, the movement of the cap 1 corresponds to the conventional movement operation of the mouse body, and the input of the information processing command by voice corresponds to the operation of the click button of the mouse.

A system configuration for realizing such a function is shown in FIG. In FIG. 2, a well-known personal computer with a video input function can be used as the information processing device 4. Therefore, the points relating to the present invention and different from the related art will be mainly described.

The CPU 10 executes the operation control of the components described below based on the system program stored in the system memory 11 and the hard disk storage device (HD) 16, and also processes the signals input from the video camera 2 and the microphone 3. Then, the processing relating to the present invention for converting into a control signal for internal processing is executed.

The system memory 11 relates to a system program executed by the CPU 10, a font table for converting a character code into an image (font) for display, and the present invention. The figure for displaying the position of the cap 1 on the display screen of the CRT 18 , A second notation of the present invention, referred to as a mouse cursor). The voice input interface 12 converts an analog voice signal output from the microphone 3 into a digital voice signal, internally stores a predetermined amount (corresponding to the length of a word), and delivers it to the CPU 10. The video capture board 13 converts an analog color video signal output from the video camera 2 into a digital color video signal, accumulates a predetermined amount (for one screen), and delivers it to the CPU 10.

The video memory 14 stores image information for one screen to be displayed on the CRT 18. Video memory 14
Image information is read by the CRT controller 17 and transferred to the CRT 18. The image information is visually displayed on the CRT 18.

Floppy disk drive (FD) 15
Accepts a floppy disk and reads and writes information on the floppy disk.

The hard disk storage device (HD) 16 stores various types of information processing, for example, application programs for performing document editing processing, character and / or image data created by these programs, for example, document text. Stored programs,
The data described above is schematically displayed on the display screen in the form of a graphic called an icon (first notation of the present invention). In the present embodiment, conventionally, since the mouse cursor is positioned on the icon on the display screen or the character string indicating the information processing command, the command to be input by the operation of the click button is input by voice. A table described as a voice pattern corresponding to a processing command (referred to as a control command conversion table) 1
6A is stored in the HD 16. Here, the control command is a command that can be recognized by the CPU 10, for example, a macro command, and conventionally, a command that can be executed on the CRT 18 is displayed as a character string, and execution is instructed by designating the position of the mouse and operating a click button. This command is included in the control command. In addition, a control command conventionally includes a command which is executed by operating a click button by positioning a mouse cursor on a graphic called an icon.

What is graphically displayed by an icon is an object to be processed by an application program such as an image such as a document or a photograph in addition to an application program, and is collectively called an object. Part of the contents of the control conversion table 16A is shown in FIG. It should be noted that the voice data to be a voice pattern is a voice parameter extracted from a voice signal as is well known in the related art, but is written in characters for the sake of description and a control command is written in a character string representing its function.

In FIG. 4, "this" is a voice parameter corresponding to a control command for designating (selecting) the object of the icon designated by the mouse cursor. “Put” is a voice parameter corresponding to a control command for instructing a stop position after the movement of the designated icon. “Open” is a voice parameter corresponding to the object of the specified icon, for example, a control command for displaying document data on the display screen. "Execution" is an object of a designated icon, for example, a voice parameter corresponding to a control command for activating a document editing program. “Close” is a voice parameter corresponding to a control command for closing (ending display) the currently displayed window. In addition, well-known system control commands, such as copy, paste, erase, print, etc., system control commands to be input in relation to mouse cursor position designation and voice parameters corresponding to the system control commands are described in the control command conversion table. There is.

The pointing operation executed in the above system configuration and the system operation executed in association with the pointing operation will be described with reference to the flowchart of FIG. The processing procedure of FIG. 3 is repeatedly executed by the CPU 10 in a very short cycle.

The user puts the cap 1 on the finger and causes the video camera 2 to photograph the cap 1. One screen of video data captured by the video camera 2 is captured by the CPU 10 via the video capture board 13 and temporarily stored in the work area in the system memory (steps S1 and S2).

C when the video data for one screen is captured
The PU 10 calculates the position of the cap 1 on the screen from the video data. The position of the cap 1 at time t is represented by the two-dimensional coordinate position of the pixel on the screen by (Px, Py) (t),
When the RGB color information is acquired from the video camera 2, the color information of the R (red) pixel is set to [f (x, y,
t); x-1 ,. . N, y-1 ,. . . , N],

[0029]

## EQU1 ## (Px, Py) (t) = Arg {max f
(X, y, t)} x = 1 ,. . . , N, y-1 ,. . . , N 1 (step S20).

Based on the position of the cap on the screen, the CPU 10 moves the corresponding position of the mouse cursor of the CRT 18 (Q
i, Qj) (t) is acquired using a conversion table (a table in which each pixel position on the photographing screen and the pixel position on the CRT screen corresponding to this pixel position are described).

The image of the mouse cursor figure to be displayed at the position on the screen of the CRT 18 thus obtained is written in the storage area of the video memory 14 corresponding to the display position. As a result, the mouse cursor is displayed on the display screen of the CRT 18 (steps S21 → S22).

Since the above-described processing procedure is repeatedly executed in a cycle of the order of the operating clock of the CPU 10, when the cap 1 moves, the position of the mouse cursor on the display screen of the CRT 18 also appears to move correspondingly. .

The user looks at the display screen of the CRT 18 and moves the cap 1 so as to move the mouse cursor to a desired icon. When the mouse cursor is positioned on the desired icon, the user says "this". This voice is converted into an electric signal by the microphone 3 and taken into the work area in the system memory 11 by the CPU 10 via the voice input interface 12 (step S).
1). When the required amount of voice data is captured, the CPU1
0 performs voice recognition. More specifically, the characteristic parameter is extracted from the captured voice data.

Next, a control command having a characteristic parameter similar to the extracted characteristic parameter is searched and extracted from the control command conversion table in the HD 16 (steps S3 → S1).
0). This control command is delivered to the control command processing program of the CPU 10. In the case of this example, since the icon has been selected, the same icon selection processing as in the related art, such as changing the color of the icon, is performed.

Next, when the user says "execute", the CP
U10 recognizes the control command instructed by voice recognition,
The control command is executed (steps S1 to S3 → S10).
→ S11). As a result, the application program of the selected icon is activated and becomes ready for use.

As described above, in the present embodiment, since the cap 1 put on the user's finger can be moved and the control command can be input by voice, it is not necessary to use a contact type pointing device such as a mouse or a keyboard. Information can be input to the information processing device by contact. Further, since it does not require an operation space such as a desk, it is suitable for an instruction operation in a place where the operation environment is restricted. In addition, the pointing device of the present invention can be operated even by a person who has trouble in operating the mouse, such as a physically handicapped person. In addition to this embodiment, the following example can be carried out.

1) In this embodiment, the red cap 1 is photographed as an object having optical characteristics, and the position on the photographing screen is detected. Even if the red cap 1 is attached to the tip of a long rod instead of the finger, the same effect can be obtained.

2) In this embodiment, as the optical feature, a specific color is used, but in addition, a light emitting object having a brightness equal to or higher than a threshold value, for example, a light pen can be used.

Further, a specific gesture of a finger, for example, a small gesture of rock-paper-scissors may be used as an optical feature.
In this case, conventionally known shape recognition is performed on the photographed data to detect the image position such as the position of the center of gravity of the slightly shaped portion.

3) When the control command input by voice is executed, if the CPU 10 confirms that the position designation by the cap 1 is being performed immediately before and then executes the input control command, This can be distinguished from a voice-only control command and can prevent malfunction.

4) A series of control operations can be instructed to the object of the icon designated by the cap 1. For example, when the user wants to display the document text on the screen and print it, the user specifies the icon of the document text with the cap 1 and then voice-inputs "Open this and print". The CPU 10 recognizes the input voice and uses the control command conversion table to extract the control command (command code) corresponding to the control command (characteristic parameter of voice). The plurality of fetched control commands are stored in the system memory 11, and the control commands are sequentially executed.

5) In the present embodiment, the example in which the microphone 3 is installed on the video camera 2 has been shown, but the user may carry the microphone if a microphone for wirelessly transmitting an audio signal is used.

6) In the present embodiment, the CRT 18 is provided corresponding to the position of the object having the optical characteristics in the photographing screen of the video camera 2.
Although the mouse cursor on the screen is moved, the position of the object having optical characteristics may be detected using the automatic tracking device and the detected position itself may be associated with the position of the mouse cursor on the screen of the CRT 18.

7) As the object for detecting the position of the moving object, an object having optical characteristics may be used, and also a transmitter that emits ultrasonic waves or radio waves may be used.

8) In addition to the control command described in this embodiment, a control command that has been conventionally executed by operating a click button of a mouse, for example, a well-known control command such as movement of an icon on a display screen is voiced. It goes without saying that you can enter.

[0046]

As described above, according to the invention of claim 1, the second notation (mouse cursor) on the display screen can be moved only by moving the object in the three-dimensional space. It is not restricted by the operation space like a conventional mouse. Further, it is possible to avoid difficult technical problems such as identification of the position of the finger in the three-dimensional space and the pointing direction of the finger as in the conventional method.

According to the inventions of claims 2, 3, and 4,
It is possible to input a large number of control commands from the mouse by voice inputting the control commands which were performed by the click button of the mouse. Further, with the mouse, it is necessary to learn complicated operations such as one-time operation of the click button, two-time operation, and moving the mouse while operating the click button. However, the invention of claim 2 requires such operation. Not.

According to the fifth and sixth aspects of the invention, movement of the object and
A desired program can be started by voice.

In the seventh and eighth aspects of the invention, various data of characters and images created by the program can be displayed.

[Brief description of drawings]

FIG. 1 is a perspective view showing an appearance of an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a system configuration according to an embodiment of the present invention.

FIG. 3 is a flowchart showing the processing contents of the embodiment of the present invention.

FIG. 4 is an explanatory diagram showing the contents of a control command conversion table.

[Explanation of symbols]

 1 Cap 2 Video Camera 3 Microphone 4 Information Processing Device 10 CPU 11 System Memory 16 HD 18 CRT

Claims (8)

[Claims] 1. An object is schematically displayed in a first notation on a display screen of a display device, a second notation is displayed to specify a position of the display screen, and a position is designated by moving the second notation. In the graphical user interface for inputting an instruction for the object of the first notation designated by, the position detecting means for detecting the position of the moving object without contact with the object, and the detected position and moving direction are made to correspond to each other. And a display control means for moving and displaying the second notation. 2. The graphical user interface according to claim 1, wherein input means for inputting a voice instruction for the object of the first notation, and voice recognition means for recognizing the instruction content from the input voice. And a graphical user interface further comprising information processing means for executing the recognized instruction content. 3. The graphical user interface according to claim 2, wherein the object of the first notation that matches the position of the second notation of claim 1 is a processing target of the information processing means. A graphical user interface featuring. 4. In the graphical user interface according to claim 3, a voice for instructing the processing target of the information processing means by the position of the second notation as the instruction for the object according to claim 2. And a graphical user interface characterized in that, when the type of processing is input to the input means by voice, the information processing means executes processing of the type indicated by the voice. 5. The graphical user interface according to claim 4, wherein the first notation according to claim 1 is a graphic, and an object schematically displayed by the graphic is a program. User interface. 6. The graphical user interface according to claim 5, wherein the type of the processing according to claim 4 is program activation. 7. The graphical user interface according to claim 4, wherein the first notation according to claim 1 is a graphic, and an object schematically displayed by the graphic is data composed of characters and / or images. A graphical user interface characterized by. 8. The graphical user interface according to claim 7, wherein the type of processing according to claim 4 is display of characters and / or images.