JPH10260773A - Information input method and device - Google Patents

Abstract

(57) [Summary] [Problem] As information input means to a computer,
It is to substitute the manual input using the line of sight. A line-of-sight information detecting unit detects a position and a movement of a line of sight poured on a display unit, and displays a line-of-sight position detected by an input processing and display control unit on a display unit. It is characterized in that information determined by a position is used as an input signal of a computer.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for inputting information to a computer, and more particularly to a method and apparatus for inputting information using voice and eyes.

[0002]

2. Description of the Related Art Conventional methods relating to a method of inputting information to a computer mainly use a method of inputting characters and numbers with a finger using a keyboard and a method of inputting with a mouse mainly in currently marketed computers. ing.
FIG. 10 shows the configuration of the prior art.

In FIG. 10, reference numeral 1 denotes input means such as a keyboard or a mouse, and 2 denotes a signal from the input means 1 and an information processing means 3 which is a final receiving side of the input signal.
Input processing and display control means for performing a transfer process to the display unit 4 and further performing control for displaying on the display unit 4. The input processing and display control means 2, the information processing means 3 and the display means 4 are so-called computer components. Keyboard input is for inputting characters or numbers by hitting a key with a finger, and mouse input is for pressing (clicking) a desired item on the screen once with a mouse button and pressing twice (double-clicking). ), While holding down and pulling (drag), "select",
Input control information such as "execute" and "drag".
To move the mouse without pressing the mouse button, simply move the pointer indicating the designated position on the display unit 4 by “moving”.
Becomes These methods require manual input using two types of input terminals, such as a keyboard and a mouse, while always looking at the screen or keyboard, and cannot be input in situations where hands cannot be used.

[0004]

As described above, in the prior art, a method of inputting information to a computer is limited.
That is, the input means 1 by manual input such as a keyboard and a mouse is required, and no voice or eyes are used. Also, since the functions of the mouse and keyboard are different,
Each of them needed to be used according to the situation.

It is an object of the present invention to provide a computer information input method and apparatus in which a line of sight and a voice are used as information input means for the computer, and a manual input is substituted by combining the line of sight and the voice. is there.

[0006]

According to a first aspect of the present invention, a gaze position on a display is detected, the detected gaze position is displayed on the display, and the gaze position and the movement of the gaze are displayed. The determined information is used as an input signal to the computer.

According to a second aspect of the present invention, a gaze position on a display is detected, the detected gaze position is displayed on the display, and information determined according to the gaze position and the movement of the gaze is transmitted to a computer. And the input voice is recognized, and information determined by the voice is used as an input signal to the computer.

According to a third aspect of the present invention, control signals for selecting, executing, and dragging the icon are made to correspond to the number of blinks and the interval when the line of sight stops on the icon. When the gaze position stops on the icon, when a blink is made, a control signal corresponding to the number and interval of the blinks is input to a computer, and the movement of the gaze position when the icon is dragged is followed. , To move the position of the dragged icon.

According to a fourth aspect of the present invention, when a keyboard is displayed on a display, a character at a line of sight is detected on the keyboard on the display, and a predetermined line of sight movement is detected, the detected character is detected. The character at the line of sight is used as input information to the computer.

According to a fifth aspect of the present invention, voices corresponding to a finite number of characters or commands are registered in advance, and when a voice is input, the input voice is registered in advance. Compared with a voice corresponding to a finite number of characters or commands, the closest character or command is used as input information to the computer.

According to a sixth aspect of the present invention, there is provided a visual line information detecting means (a visual line information detecting means 5 shown in FIG. 1) for detecting a position and a movement of a visual line poured on a display, and the visual line information detecting means. A gaze position display means (input processing and display control means 2 shown in FIG. 1) for displaying the detected gaze position on a display (display means 4 shown in FIG. 1), and the detected gaze position or movement. It has gaze-corresponding information input means (input processing and display control means 2 shown in FIG. 1) for inputting information determined accordingly as an input signal of a computer.

According to a seventh aspect of the present invention, there is provided a voice processing unit (voice processing unit 6 shown in FIG. 1) for recognizing a voice, and a voice for inputting information determined by the recognized voice as an input signal to the computer. Correspondence information input means (Fig. 1
And input control and display control means 2).

According to the present invention, a control signal for selecting, executing, and dragging the icon is made to correspond according to the number of blinks and the interval when the line of sight stops on the icon. Control signal input means (input processing and display control means 2 shown in FIG. 1) for inputting a control signal to be input to the computer, and the movement of the line-of-sight position when the icon is being dragged. Moving means for moving the position (input processing and display control means 2 shown in FIG. 1).

According to a ninth aspect of the present invention, a keyboard display means (input processing and display control means 2 shown in FIG. 1) for displaying a keyboard on a display, and a character at a line of sight is detected on the keyboard on the display. Then, when the movement of the predetermined line of sight is detected, the character corresponding to the detected line of sight is input to the computer by keyboard corresponding information input means (input processing and display control means 2 shown in FIG. 1). ).

According to a tenth aspect of the present invention, a comparison means (information processing means 3 shown in FIG. 1) for comparing a voice corresponding to a finite number of characters or commands registered in advance with an input voice. , Registration information input means (information processing means 3 shown in FIG. 1) for making the closest character or command input information to the computer.

[0016]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A feature of the present invention is to substitute the input of a mouse or a keyboard, which has conventionally been an input of a computer, with a line of sight and a voice and a line of sight. And a voice processing means for recognizing and processing the voice, and converting the generated voice into an input signal of a computer (information processing means).

Further, a line-of-sight detecting means for detecting the position of the line of sight poured on the display and collecting the moving distance, moving speed, gazing time, pupil diameter, number of blinks and the interval between blinks, and processing the collected result. Control processing means for inputting to the computer and controlling the display on the display (display means), detects the position of the line of sight and uses it to indicate the position on the display, and furthermore, the time during blinking By measuring the length (blink interval) and the number of blinks, the input at the designated position corresponds to selection of a command, execution or dragging, and a method and an apparatus for inputting information to a computer using a line of sight I will provide a.

The present invention differs from the prior art in that a computer user can input to the computer with his / her eyes and voice instead of the input means conventionally performed by a mouse or keyboard.

FIG. 1 shows an example of the configuration of an information input device using the present invention. 2 is an input processing and display control means, 3 is an information processing means, 4 is a computer display means (hereinafter referred to as a display), 5 is a line-of-sight information detecting means, and 6 is an audio processing means which is a target of information input. .

The line of sight is not used at all as input means in the task of keyboard input or mouse input, which is a conventional computer information input method. Also, the audio
It is hardly used as information input means of conventional computers. The line-of-sight information detecting means 5 includes line-of-sight information such as line-of-sight position, moving distance, moving speed, gaze time, pupil diameter,
The number of blinks and the interval between blinks are measured. The input processing and display control means 2 uses the line-of-sight position, the moving distance, the moving speed, the gaze time, the pupil diameter, the number of blinks, and the blink interval to determine whether or not the pointer is in a moving state or a moving state. It is determined which of the selection, the execution, and the drag is. The voice processing unit 6 that receives voice as input recognizes voice and converts the recognized voice into an input signal, thereby enabling voice input to the computer. This enables input to the computer even when the hand cannot be used.
Further, by using not only the line-of-sight input but also the voice input, fatigue caused by only the line-of-sight input can be reduced.

[0021]

【Example】

[Embodiment 1] A first embodiment of the present invention will be described. In the present embodiment, the line-of-sight information input replaces the input by a normal mouse by the configuration shown in FIG. 1 described above. In addition, the input by voice replaces the input by a normal keyboard.

In FIG. 1, 2 is input processing and display control means, 3 is information processing means, 4 is display means (display), 5 is visual line information detecting means, 6 is voice processing means, and 2 to 4 are diagrams. Corresponding to that shown in FIG. Although not shown, the line-of-sight information detecting means 5 includes an optical device, a video camera, and a mirror, and outputs infrared rays transmitted from the optical device to the pupil of the computer user.
Is reflected by a mirror placed beside the camera, and the reflected light is taken into the optical system again. The video camera is for displaying the display 4 seen by the subject (operator of the computer), and is for matching the position of the line of sight with the position of the screen.

Techniques for detecting the line-of-sight position are already known, and the literature includes Sakaguchi, Nakano and Yamamoto: "Detection of the driver's line of sight while driving a car", IEICE Technical Report PRU95-28, pp57-6.
4, 1995 and the like. Further, as an example of a commercialized gaze detection device, ASL (Applied Science Laborato)
ries), “Eye Tracking System 4250R”
Remote type optical system device ". In the present invention,
It is possible to use these techniques.

The accuracy of detection of the line-of-sight position is such that the error between the true line-of-sight position and the measured line-of-sight position near the true eyeball position is 1 degree or less, and the error at the periphery of the visual field is 2 degrees or less. Therefore, assuming that the distance to the display is 30 cm,

[0025]

## EQU1 ## 30 cm × tan 2 degrees = 1.047 cm That is, an error of plus or minus about 1 cm. Therefore, if the icon is about 2 cm in size, it is possible to detect whether the operator is gazing at the icon by detecting the line of sight. In addition, when the detected line-of-sight position is displayed on the display 4 with the line-of-sight pointer as described later, the feedback from the operator becomes possible, so that even a smaller icon can be instructed.

FIG. 2 is an image of a start icon and an end icon for performing voice and line-of-sight input. When the voice input activation icon is clicked, voice input becomes possible, and at the same time, a voice input end icon is displayed. Clicking the voice input end icon ends voice input. Similarly, by clicking the gaze input activation icon, gaze input becomes possible, the pointer becomes the gaze position on the display 4, and the gaze input end icon is displayed at the same time. Clicking on the gaze input end icon ends the gaze input. It is assumed that the voice input activation icon and the eye-gaze input activation icon appear on the screen when the computer is activated. By preparing these icons,
It can be used only when input by voice or eyes is required.

In order to convert the voice into an input, the voice input from the microphone input of the computer is processed by the voice processing means 6.
To the input processing and display control means 2 to convert voice into input information. Use a microphone that can be connected to the microphone input provided with the computer, and does not require a special microphone for voice input.

The voice input information mainly substitutes for the function of the keyboard, and is a sentence based on alphabet input or kana input, a numerical value based on keypad input, or the like.

Commands prepared in advance for voice input include commands such as "copy", "delete", and "rename" required for file management, and "input" and "conversion" required for editing file contents. , "Copy,""paste,""delete," and "replace / search." As for these commands, the commands can be executed by uttering them by voice.

For input of characters and numbers by voice input,
Use the "input" command. When the "input" command is executed, what is subsequently uttered is regarded as a numeral or character input, and is converted into the nearest character or numeral. Also,
It is assumed that words and commands can be registered by the user for characters and commands frequently used by the user.

Furthermore, for basic operations such as file operation and editing operation, there is already a voice recognition technology that enables these operations by voice input by using words registered in advance as voice commands. Therefore, the details are omitted.

The line-of-sight input information mainly substitutes for the mouse function, and enables functions such as pointer movement, selection, execution, and dragging.

In order to convert the line of sight into an input, the line of sight information obtained through the line of sight information detecting means 5 is sent to the input processing and display control means 2 and the line of sight position, moving distance, moving speed, gaze time, pupil diameter, From the number of times of flapping and the interval of flapping, it is determined whether or not the pointer is in the position or moving state, and which input control among “select”, “execute”, and “drag” is to be performed. The result is output to the display means 4 and passed to the information processing means 3.

The flow of information for using the line of sight as an alternative input of the mouse function is shown in the flowcharts of FIGS. Note that (1) to (21) indicate each step.

In FIG. 3, when the gaze input activation icon is activated (1), collection of gaze information on the pointer is started. First, calibration is performed as an initial setting to match the position of the line of sight with the position of the screen (2). In the calibration, the user gazes at several positions (five or nine) around the center and four corners on the screen according to the display on the screen, and registers the characteristics of the line of sight. This causes the computer to recognize the visual characteristics of the user. However, proofreading is required every time the computer is started, but if the same user uses the proofreader continuously, proofreading can be omitted.

Next, collection of eye gaze information for an actual computer input is started (3). At the start of the processing, the input state to the computer is "neutral state" in the figure (4).

The line of sight is always detected during the period in which the input by the line of sight is possible. However, there is a fixation micromotion shown below as a characteristic of the human gaze, and measures for this are performed as follows.

The fixational tremor is a small tremor in which the human eye is constantly moving, and even when he / she stares at one point, he / she is not aware of it but continues to make small tremor. Therefore, in the present invention, the term “stop of the line of sight” means that the line of sight is stopped when the moving speed of the line of sight is smaller than a predetermined constant value (threshold a). A specific value of the threshold value a is 5 degrees / second for reference. This value can be set by the user.

The above is the basis of the gaze position detection processing in the present invention, and this processing is always performed. Based on the above, a procedure for substituting a normal mouse input will be described with reference to FIGS.

In the present invention, on the display 4,
(A) selection of an icon or the like (implemented with a single click with a normal mouse), (b) execution of a process corresponding to the icon (executed with a double click with a normal mouse),
(C) Movement by dragging an icon or the like is realized as follows.

(A) Selection of Icon, etc. It is assumed that the user comes over an icon with a line of sight (5), blinks once (6), and selects that icon (9). At this time, the moving speed of the line of sight is
It is assumed that it is smaller, that is, it is considered as stopped. In addition, it is necessary to detect whether the blink is truly only once or the first one of the two blinks. However, after the above-mentioned blink, if there is no blink for a certain period of time, the blink is truly once. It can be solved by assuming that it is only blinking (7). The specific value of the fixed time b is, for example, 0.5 seconds, and can be set by the user (10). The icon etc. selected by this operation shall be kept selected,
It is assumed that the selection information is released when a blink is detected at a position other than the icon at the line of sight (13) to
(16).

(B) Executing the processing corresponding to the icon In the same manner as in (a), it is considered that the line of sight has come to a stop over a certain icon, and if blinking is performed twice within a certain period of time b, the icon is displayed. Corresponding processing or software execution (7), (8).

(C) Dragging icons and the like Stop gaze on the icon to be dragged,
It is assumed that the icon is dragged when the user blinks once to make the selected state (same as the processing in step (1) above) and further performs the next blink without moving the line-of-sight position for a certain period of time or more ( 11), (12). To move the dragged icon, bring the line of sight to the icon (17) and move the line of sight (1).
8), and move following the movement (19). However, when the movement of the line of sight is equal to or more than the threshold value c of the fixed speed, the movement of the icon following the line of sight is not performed. To give a specific value of the threshold value c for reference, it is assumed that the icon dragged once is released from the drag state when a blink is detected in a place other than the above-mentioned line of sight (20), ( 21) Until then, even if the line of sight comes to a position other than the icon, it is not released.

By performing such a process, when the dragged icon is moved by the movement of the line of sight, excessive concentration is not required.

The release of the drag is as described above. In addition, the blink detection uses the fact that pupil diameter data cannot be obtained as line-of-sight information when the pupil is closed by blinking.
If the time is 0 ms or less, it is determined that blinking has occurred.

With such a structure, a line of sight can be used instead of a mouse which is an input device of a computer, so that input can be performed without using a finger.
As the effect, the input terminal of the computer is not particularly required, the burden on the user interface of the user is small,
It is easy to use for users who have difficulty moving their fingers.

[Embodiment 2] FIG. 6 is a view for explaining a second embodiment of the present invention. It is not only used as a substitute for, but also used as a substitute for keyboard input. In this case, voice input is not performed. The configuration is the same as in FIG. 1, but no microphone input is used because no voice input is performed. When the gaze input activation icon is activated, collection of gaze information on the pointer is started. First, calibration is performed as an initial setting to match the position of the line of sight with the position of both surfaces. In the calibration, the user gazes at several positions (five or nine) around the center and four corners on the screen according to the display on the screen, and registers the characteristics of the line of sight. This causes the computer to recognize the visual characteristics of the user. However, proofreading is required every time the computer is started, but if the same user uses the proofreader continuously, proofreading can be omitted.

Next, collection of the line of sight for the actual computer input is started. There are icons on the screen that allow keyboard input. When the icons are activated, the keyboard appears on the screen from the state of FIG. 6A to the state of FIG. 6B. By selecting a key on the screen keyboard with a line of sight, keyboard input is enabled. In this case, instead of inputting by placing a finger on the keyboard terminal, which is the conventional usage, the function is substituted by placing the line of sight on the keyboard key on the screen. The gaze of keyboard input can be ended by selecting the end button on the top left of the keyboard displayed on the screen.

FIG. 7 is a flowchart showing the flow of information for using the line of sight as an alternative input of the keyboard function. In addition,
(31) to (37) show each step. When the computer is started, a pointer and a gaze input start icon are displayed at the center of the screen. When the gaze input activation icon is clicked (31), collection of the gaze on the pointer is started. First, calibration is performed (32). When the calibration is completed, the input information is analyzed. First, whether the gaze is stationary or moving is measured (33). If the moving speed (moving angle per unit time) v1 is smaller than the threshold value a, (3
4), it is determined that the line of sight is stationary at that position,
If it is larger than the threshold value a, it is determined that the moving state is established (3
7). In the moving state (No direction), the measurement time unit is:
The position is detected by measuring the moving angle and the moving distance. On the other hand, in the stationary state (Yes direction), it is detected whether or not the next blink has occurred. In the blinking state (Yes direction), the command at that position is executed (35), (36). On the other hand, when the user does not blink (the direction of No), the gaze position, the gaze time, the pupil diameter, the moving speed, and the moving distance are measured again. When both the mouse function and the keyboard function are replaced by the line of sight, the user only needs to perform calibration once.

With this structure, the contents of work that can be performed by a computer are expanded in that characters and numerals can be directly input without using a finger as compared with the first embodiment. The effect is that work equivalent to the current computer operation can be performed without touching a finger, and the user interface is less burdensome.

[Embodiment 3] FIG. 8 is a view for explaining a third embodiment of the present invention, wherein 1A is a keyboard of input means, 2 is input processing and display control means, and 3 is information processing means. Reference numeral 4 denotes a display unit, and reference numeral 5 denotes a line-of-sight information detection unit. The keyboard 1A is used in the same manner as the conventional usage, and inputs by a finger, and only the function of the mouse is replaced by the line of sight. With such a structure, the mouse function in the conventional computer work is replaced with the line of sight, and only two types of input devices, a keyboard and a mouse, which are conventionally used, can be input from the fingertip with the keyboard, and the subsequent input. Can be one type of input device using a fingertip, such as input by gaze. The effects include simplification of the input terminal and relief from the complexity of the user using the input terminal.

[Embodiment 4] FIG. 9 is a view for explaining a fourth embodiment of the present invention, wherein 1B is a mouse among input means, and 2 to 4 and 6 are the same as those of the embodiment of FIG. It is. Mouse 1
B is the same as the conventional usage, and is input by a finger, and only the function of the keyboard is replaced by voice. With such a structure, by replacing the keyboard function in the conventional computer work with voice, only the mouse input from the fingertips from the two types of input devices conventionally used, the keyboard and the mouse, the subsequent input Can be one type of input device using a fingertip, such as input by voice. The effects include simplification of the input terminal and relief from the complexity of using the input terminal by the user.

Commands prepared in advance for voice input include commands such as "copy", "delete", and "rename" required for file management, and "input" and "conversion" required for editing file contents. , "Copy,""paste,""delete," and "replace / search." As for these commands, the commands can be executed by uttering them by voice.

For inputting characters and numbers by voice input,
Use the "input" command. When the "input" command is executed, what is subsequently uttered is regarded as a numeral or character input, and is converted into the nearest character or numeral. Also,
It is assumed that words and commands can be registered by the user for characters and commands frequently used by the user.

[0055]

As described above, since the information input method and apparatus according to the present invention enable input to a computer by voice and eyes, the user interface of the computer becomes easier and the input terminal can be simplified. For example, there is an advantage that even a person who has a finger disorder or who is bedridden due to illness can operate a computer without operating a keyboard or a mouse.

Further, in the present invention, since both the line-of-sight information and the voice information are used in combination, it is effective to move around to indicate the location by inputting only the line-of-sight information. When inputting, the drawback is that fatigue builds up, and input using only voice information is effective for continuous input of letters and numbers, but it is inconvenient to show places in detail,
The above drawbacks can be complemented.

Further, it is inconvenient to indicate a place in detail by inputting only by voice, and it is difficult to draw an arbitrary figure. However, it is possible to easily draw a figure by using an eye-gaze input.

Further, with the eyes input alone, fatigue may be caused by repeatedly moving small places when characters and numbers are input from the keyboard on the screen. However, voice input is also used when inputting characters and numbers. This has the effect of reducing fatigue.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of an information input device according to the present invention.

FIG. 2 is an image diagram showing an example of a start icon and an end icon for voice and line-of-sight input displayed on a screen used in the embodiment of FIG. 1;

FIG. 3 is a flowchart for explaining an embodiment of an information input method according to the present invention.

FIG. 4 is a flowchart illustrating an embodiment of an information input method according to the present invention.

FIG. 5 is a flowchart for explaining an embodiment of an information input method according to the present invention.

FIG. 6 is an image diagram of a keyboard display when performing a keyboard input from a screen in the present invention.

FIG. 7 is a flowchart for explaining another embodiment of the information input method according to the present invention.

FIG. 8 is a block diagram showing a configuration of another embodiment of the information input device according to the present invention.

FIG. 9 is a block diagram showing a configuration of still another embodiment of the information input device according to the present invention.

FIG. 10 is a block diagram showing a configuration example of a conventional computer.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 input means 2 input processing and display control means 3 information processing means 4 display means 5 gaze information detecting means 6 information processing means

Claims (10)

[Claims] 1. A method for inputting information to a computer, comprising detecting a gaze position on a display, displaying the detected gaze position on the display, and determining information based on the gaze position and the movement of the gaze. An information input method for inputting information to a computer. 2. A method for inputting information to a computer, comprising detecting a gaze position on a display, displaying the detected gaze position on the display, and determining information based on the gaze position and the movement of the gaze. As an input signal to a computer, further recognizing the input voice, and using information determined by the voice as an input signal to the computer. 3. The method for inputting information to a computer according to claim 1, wherein the icon is selected, executed, and dragged in accordance with the number of blinks and the interval when the line of sight stops on the icon. The control signal for the eye is stopped, the line of sight stops on the icon, and when a blink is made, the control signal corresponding to the number and interval of the blink is input to the computer, and the icon is dragged. An information input method characterized by moving the position of the dragged icon in accordance with the movement of the line-of-sight position. 4. The information input method according to claim 1, wherein a keyboard is displayed on a display, a character at a line of sight is detected on the keyboard on the display, and a predetermined line of sight movement is detected. The character at the detected line-of-sight position is used as input information to a computer. 5. The information input method according to claim 2, wherein voices corresponding to a finite number of characters or commands are registered in advance, and if a voice is input, the input voice is registered in the pre-registered voice. An information input method characterized in that the closest character or command is used as input information to a computer as compared with a voice corresponding to a finite number of characters or commands. 6. A line-of-sight information detecting means for detecting a position and a movement of a line of sight poured on a display, and a line-of-sight position display means for displaying on the display the position of the line of sight detected by the line of sight information detecting means. Information input means for inputting information determined according to the detected line-of-sight position or movement as an input signal of a computer. 7. An information input device, comprising: a voice processing unit for recognizing a voice; and a voice corresponding information input unit for inputting information determined by the recognized voice as an input signal to the computer. 8. The information input device according to claim 6, wherein control signals for selecting, executing, and dragging the icon are supported according to the number of blinks and the interval when the line of sight stops on the icon. Control signal input means for inputting a corresponding control signal to the computer, and moving means for moving the position of the dragged icon in association with the movement of the line-of-sight position when the icon is dragged. An information input device, characterized in that: 9. The information input device according to claim 6, wherein a keyboard display means for displaying a keyboard on a display, and a character at a line of sight is detected on the keyboard on the display to detect a predetermined line of sight movement. Information input means for inputting the character at the detected line-of-sight position to the computer when the input is performed. 10. The information input device according to claim 7, wherein a voice corresponding to a finite number of characters or commands registered in advance is compared with the input voice, And a registration information input means for inputting information to a computer.