JP2015103152A - Electronic apparatus and visual line input method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an electronic apparatus and a visual line input method which allow a visual line of a user to select an icon even if a visual line of the user fluctuates.SOLUTION: A display unit 3 displays an image to which a plurality of blocks each including a GUI are set. A visual line detection unit 7 detects an input position of a visual line on a display surface of the display unit 3. A count unit 6 counts the number of movements of the visual line input position from the outside to the inside of any block. A program execution unit 4 executes processing corresponding to the GUI included in a block with a counted value of which is equal to or greater than a specified value.

Description

  The present invention relates to an electronic device and a line-of-sight input method.

  An apparatus for selecting an icon displayed on a screen by a line of sight is known. For example, in the apparatus described in Patent Document 1 (Japanese Patent Laid-Open No. 8-272517), a gaze position on the screen is detected, the gaze is in a gaze state, and the gaze is displayed on an icon displayed on the screen. If it is all gaze, a selection mark blinks on the icon.

JP-A-8-272517

  However, when the user's line of sight fluctuates, the icon may not be selected accurately.

  Therefore, an object of the present invention is to provide an electronic device and a line-of-sight input method capable of executing processing corresponding to a GUI such as an icon by the user's line of sight even when the line of sight of the user fluctuates.

  An electronic device according to an aspect of the present invention includes a display unit that displays an image. The image includes a plurality of blocks, and the blocks include a GUI. The electronic device is counted with a line-of-sight detection unit that detects a line-of-sight input position on the display surface of the display unit, and a count unit that counts the number of times the line-of-sight input position has moved from the outside of any block to the inside. An execution unit that executes processing corresponding to a GUI included in a block having a count value equal to or greater than a predetermined value.

  According to the present invention, even if the user's line of sight fluctuates, processing corresponding to a GUI such as an icon can be executed by the user's line of sight.

It is a figure showing the external appearance of the electronic device of embodiment of this invention. It is a figure showing the main structures of the electronic device of 1st Embodiment. It is a figure for demonstrating the block contained in the image displayed on a display part. It is a figure for demonstrating the block of embodiment. It is a figure showing the example of the count value of each block. It is a figure showing the example of the time when a eyes | visual_axis is input into a block. It is a figure showing the example of a detection effective zone and a reset zone. It is a figure showing another example of a detection effective zone and a reset zone. It is a flowchart showing the procedure of the icon selection based on the gaze time by a prior art. It is a flowchart showing the procedure of the icon selection based on the frequency | count that the locus | trajectory of the eyes | visual_axis entered in the block by 1st Embodiment. It is a flowchart showing the procedure of the icon selection based on the frequency | count that the locus | trajectory of the eyes | visual_axis entered in the block by 2nd Embodiment. It is a figure for demonstrating the change of the detection effective zone by 2nd Embodiment. It is a flowchart showing the procedure of the icon selection based on the frequency | count that the locus | trajectory of the eyes | visual_axis entered in the block by 3rd Embodiment. It is a flowchart showing the procedure of icon selection based on the frequency | count that the locus | trajectory of the eyes | visual_axis entered in the block by 4th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating an appearance of an electronic apparatus according to an embodiment of the present invention. As shown in FIG. 1, the electronic device is a smartphone and includes a housing 12. A display unit 3 such as a liquid crystal display is disposed on the surface of the housing 12. A touch panel 16 is provided on the upper surface of the display unit 3. A speaker 18 is provided on the surface side near one end of the housing 12 in the Y-axis direction, and a microphone 20 is provided on the surface side near the other end in the Y-axis direction. Further, a call key 22 for starting a call, a call end key 24 for ending a call, and a menu key 26 for displaying a menu screen are provided in the vicinity of the surface side of the other end in the Y-axis direction. An infrared camera 30 is provided on the left side of the speaker 18, and an infrared LED 32 is provided on the left side of the microphone 20.

  In the embodiment of the present invention, a mobile phone such as a smartphone will be described as an example of an electronic device. This includes electronic devices such as special support devices used in

FIG. 2 is a diagram illustrating a main configuration of the electronic apparatus 1 according to the first embodiment.
The electronic device 1 includes an antenna 2, a wireless communication unit 8, a display unit 3, a program execution unit 4, an icon selection unit 5, a count unit 6, and a line-of-sight detection unit 7. The line-of-sight detection unit 7 includes an infrared LED 32 and an infrared camera 30.

  The wireless communication unit 8 transmits and receives signals to and from other communication devices such as a base station through the antenna 2.

  The display unit 3 displays an image including a plurality of icons. A plurality of blocks are set in the image displayed on the display unit 3. Each block includes an icon which is a kind of GUI (Graphical User Interface).

  FIG. 3 is a diagram for explaining blocks included in an image displayed on the display unit 3. The image includes an area 91 including icons and an area 92 other than that. An area 91 including icons is divided into a plurality of blocks. In FIG. 3, an area 91 including icons is divided into 20 blocks of 4 in the vertical direction (Y-axis direction) and 4 in the horizontal direction (X-axis direction). Each block is specified by a block BLi (i = 1 to 20) as shown in FIG.

  The line-of-sight detection unit 7 includes an infrared LED 32 that emits infrared light and an infrared camera 30. The line-of-sight detection unit 7 detects the user's pupil and infrared reflected light from the captured image captured by the infrared camera 30, so that the user's line of sight on the image displayed on the display unit 3 (hereinafter, display image) is detected. The input position (hereinafter referred to as the line-of-sight position) is detected. More specifically, the line-of-sight detection unit 7 detects infrared light traveling from the position of reflected light in the captured image toward the pupil. The line-of-sight detection unit 7 detects the line-of-sight position on the display screen depending on which position on the display screen the line-of-sight vector indicates.

  The count unit 6 counts the number of times the user's line-of-sight position has moved from the outside of the block to the inside of the block for each block.

  The count unit 6 counts, for each block, the number of times that the user's line-of-sight position has moved from the outside of the block to the inside of the block (that is, the number of times the line-of-sight locus has entered the block). In the example of FIG. 5, the count number of the block BL2 is “1”, the count number of the block BL3 is “1”, the count number of the block BL5 is “1”, the count number of the block BL6 is “6”, The count number of the block BL7 is “1”, the count number of the block BL9 is “1”, and the count number of the block BL10 is “2”.

  FIG. 6 is a diagram illustrating the locus of the line of sight. In the example of FIG. 6, the line-of-sight position is detected by the block BL6 at time t0. Next, at time t1, the user's line-of-sight position moves to block BL10. Further, at time t2 to t12, the user's line-of-sight position moves to blocks BL9, BL5, BL6, BL2, BL3, BL7, BL6, BL10, BL6, BL2, BL6.

  The counting unit 6 sets the block in which the user's line-of-sight position is first detected as the reference block B, and sets the reference block B and one or more blocks near the reference block B as the detection effective zone D. To do. For example, the counting unit 6 sets blocks adjacent to the reference block B in the vertical, horizontal, and diagonal directions as blocks near the reference block B. The counting unit 6 performs counting only for the blocks included in the detection effective zone D.

  The counting unit 6 sets one or more blocks remote from the reference block in the reset zone R. For example, the count unit 6 sets a block that is separated by 3 blocks in the X direction and a block that is separated by 3 blocks in the Y direction in the reset zone R. When the user's line-of-sight position moves to a block included in the reset zone R, the count unit 6 initializes the count values of all the blocks BL1 to BL20.

FIG. 7 is a diagram illustrating an example of the detection effective zone D and the reset zone R.
As shown in FIG. 7, when the reference block B is the block BL6, the blocks BL1, BL2, BL3, BL5, BL7, BL9, BL10, BL11 adjacent to the block BL6 are set as the detection effective zone D. Also, the blocks BL17 to BL20 that are separated by three blocks in the Y direction are set as the reset zone R.

FIG. 8 is a diagram illustrating another example of the detection effective zone D and the reset zone R.
As shown in FIG. 8, when the reference block B is the block BL1, the blocks BL2 and BL6 adjacent to the block BL1 are set as the detection effective zone D. Also, the blocks BL4, BL8, BL12, BL16, and BL20 that are separated by 3 blocks in the X direction and the blocks BL13 to BL20 that are separated by 3 blocks in the Y direction (BL16 and BL20 overlap) are set as the reset zone R. The

  The icon selection unit 5 selects one of a plurality of icons based on the count result. More specifically, the icon selection unit 5 selects an icon included in a block whose count value by the count unit 6 is a predetermined value or more.

The program execution unit 4 executes a program corresponding to the selected icon.
In the present embodiment, a method for selecting an icon according to how long the user's line-of-sight position stays in the block according to the prior art (hereinafter referred to as an icon selection method based on gaze time), and wandering of the line of sight are considered. The icon is selected by using two methods of selecting an icon according to the number of times that the user's line of sight trajectory has passed through the block (hereinafter referred to as an icon selection method based on the number of times the line of sight has passed). In other words, the two methods are executed in parallel, the icon previously selected by either method is notified to the program execution unit 4, and the program corresponding to the icon is executed.

  FIG. 9 is a flowchart showing a procedure for selecting an icon based on the gaze time according to the prior art.

  In step S901, the count unit 6 sets all the blocks BL1 to BL20 to the detection effective zone D.

  In step S902, when the line-of-sight position remains in any block of the detection effective zone D (the block is referred to as a block *) for a predetermined time, the process proceeds to step S903.

  In step S903, the icon selection unit 5 selects an icon included in the block *.

  FIG. 10 is a flowchart showing an icon selection procedure based on the number of times the line-of-sight locus enters the block according to the first embodiment.

  In step S101, the count unit 6 sets all the blocks BL1 to BL20 to the detection effective zone D. The count unit 6 initializes the count values of all the blocks BL1 to BL20 to 0, and stops the timers of all the blocks BL1 to BL20. Further, the count unit 6 cancels the setting when the reference block B is set.

  In step S102, when the line-of-sight position is detected for the first time in any block of the detection effective zone D, the process proceeds to step S103.

  In step S <b> 103, the count unit 6 sets the block in which the line-of-sight position is detected for the first time as the reference block B.

  In step S104, the count unit 6 sets the reference block B and one or more blocks in the vicinity of the reference block B as the detection effective zone D as shown in FIG.

  In step S105, the counting unit 6 sets one or more blocks remote from the reference block B in the reset zone R as shown in FIG.

  In step S106, the count unit 6 increments the count value of the reference block B.

  In step S107, the count unit 6 starts a timer for the block set in the reference block B.

  In step S108, when the line-of-sight position has moved to any block in the detection effective zone D (the block is referred to as a block *), the process proceeds to step S109, and when such a condition is not satisfied, The process proceeds to step S114.

  In step S109, the count unit 6 increments the count value of the block *.

  In step S110, when the count value of the block * is “1”, the process proceeds to step S111, and when such a condition is not satisfied, the process proceeds to step S112.

  In step S111, the count unit 6 starts a timer for block *.

  In step S112, if the count value of any block (the block is referred to as block **) is greater than or equal to the predetermined value CT, the process proceeds to step S113, and if such a condition is not satisfied, the process is performed. Returns to step S108.

  In step S113, the icon selection unit 5 selects an icon included in the block **.

  In step S104, when the timer value of any block (the block is referred to as block ***) is equal to or greater than the predetermined value TM, the process proceeds to step S115, and when such a condition is not satisfied, the process is performed. Proceed to step S116.

  In step S115, the count unit 6 excludes the block *** from the detection valid zone D, and the process returns to step 108.

  If the line-of-sight position has moved to any block in the reset zone R in step S116, the process returns to step S101 (that is, all blocks are set to the detection effective zone D, and the count value of all blocks is set to 0). Initialization is performed, the timers of all blocks are stopped, and the setting of the reference block B is canceled). If such a condition is not satisfied, the process returns to step S108.

  As described above, according to the present embodiment, since an icon included in a block with a large number of times the line of sight has entered is selected, even if the user's line of sight fluctuates, the icon is selected based on the user's line of sight. be able to.

[Second Embodiment]
The second embodiment is different from the first embodiment in the method of setting the detection effective zone D.

  The counting unit 6 according to the present embodiment specifies the maximum value among the count values of a plurality of blocks, and detects one or more blocks having a count value that is greater than or equal to a predetermined value from the specified maximum value. Exclude from D.

  FIG. 11 is a flowchart showing an icon selection procedure based on the number of times the line-of-sight locus enters the block according to the second embodiment.

  In step S201, the count unit 6 sets all the blocks BL1 to BL20 to the detection effective zone D. The count unit 6 initializes the count values of all the blocks BL1 to BL20 to 0, and stops the timers of all the blocks BL1 to BL20.

  In step S202, when the line-of-sight position has moved to any block in the detection effective zone D (this block is referred to as a block *), the process proceeds to step S203.

  In step S203, the count unit 6 increments the count value of the block *.

  In step S204, when the count value of the block * is “1”, the process proceeds to step S205, and when such a condition is not satisfied, the process proceeds to step S206.

  In step S205, the count unit 6 starts a timer for block *.

  In step S206, if the count value of any block (the block is referred to as block **) is greater than or equal to the predetermined value CT, the process proceeds to step S207, and if such a condition is not satisfied, the process is performed. Advances to step S208.

  In step S207, the icon selection unit 5 selects an icon included in the block **.

  In step S208, if the difference between the count value of any block (the block is referred to as block ***) and the maximum count value is the predetermined value SA, the process proceeds to step S209, and such If the condition is not satisfied, the process proceeds to step S210.

  In step S209, the count unit 6 excludes the block *** from the detection valid zone D, and the process returns to step 202.

  In the second embodiment, the detection effective zone D changes as the count value of each block changes with time. A change in the detection effective zone D according to the second embodiment will be described with reference to FIG.

  As shown in FIG. 12A, all blocks BL1 to BL20 are set in the detection effective zone D in the initial state.

FIG. 12B shows the progress of the count values of the blocks BL1 to BL20.
As shown in FIG. 12C, the difference from the maximum count value (here, “5”) is equal to or greater than a predetermined value SA (here, “4”) (here, “0”). And blocks BL1, BL4, BL8, and BL12 to BL20 having “1”) are excluded from the detection effective zone D.

  In the process of counting the number of times the line of sight has entered each block, it is estimated that the block with a small count value does not include the icon that the user wants to select. In this embodiment, by excluding these blocks from the detection effective zone and narrowing down selection candidate icons, it is possible to more quickly and efficiently select an icon based on the line of sight.

[Third Embodiment]
The third embodiment is different from the first embodiment in the procedure from when the line-of-sight position moves from outside the reset zone R to the reset zone R until the count value is actually reset.

  When the user's line-of-sight position moves from the outside of the reset zone R to the reset zone R a predetermined number of times, the selection unit according to the present embodiment resets the count values of all blocks and resets the reference block B.

  FIG. 13 is a flowchart showing an icon selection procedure based on the number of times the line-of-sight locus enters the block according to the third embodiment.

  In step S301, the count unit 6 sets all the blocks BL1 to BL20 to the detection valid zone D, initializes the count values of all the blocks BL1 to BL20 to 0, and similarly to the first embodiment, The timer of BL20 is stopped. Further, the count unit 6 initializes a reset count value indicating the number of times that the user's line-of-sight position has moved from the outside of the reset zone R to the reset zone R to 0. Further, the count unit 6 cancels the setting when the reference block B is set.

  Since the processing of steps S102 to S116 is the same as that of the first embodiment, description thereof will not be repeated.

  In step S116, if the line-of-sight position has moved to any block in the reset zone R, the process proceeds to step S302. If such a condition is not satisfied, the process returns to step S108.

  In step S302, the count unit 6 increments the reset counter value.

  In step S303, if the reset count value is equal to or greater than the predetermined number RC, the process returns to step S301 (that is, all blocks are set in the detection valid zone D, and the count value and reset count value of all blocks are 0). The timer of all blocks is stopped and the setting of the reference block B is canceled). If such a condition is not satisfied, the process returns to step S108.

  As described above, according to the present embodiment, when the user's line-of-sight position moves from the outside of the reset zone R to the reset zone R a predetermined number of times, the user includes the icons included in the blocks near the reference block B. It can be presumed that the user does not want to select or has not yet decided which icon to select. In such a case, the icon desired by the user can be selected quickly and efficiently by resetting the count values of all blocks and re-setting the reference block from the beginning.

[Fourth Embodiment]
The fourth embodiment is different from the first and third embodiments in the procedure from when the line-of-sight position moves from outside the reset zone R to the reset zone R until the count value is actually reset. .

  The selection unit according to the present embodiment moves the user's line-of-sight position from the outside of the reset zone R to the reset zone R, and if the user's line-of-sight position does not move outside the reset zone R even after a predetermined time has elapsed, The block count value is reset, and the reference block B is set again.

  FIG. 14 is a flowchart showing an icon selection procedure based on the number of times the line-of-sight locus enters the block according to the fourth embodiment.

  Since the processing in steps S101 to S116 is the same as that in the first embodiment, description thereof will not be repeated.

  If the line-of-sight position has moved to any block in the reset zone R in step S116, the process proceeds to step S401. If such a condition is not satisfied, the process returns to step S108.

  If the line-of-sight position moves outside the reset zone R within the predetermined time TR in step S401, the process returns to step S108, and if such a condition is not satisfied, the process returns to step S101.

  As described above, according to the present embodiment, the user's line-of-sight position moves to the outside of the reset zone R even if the user's line-of-sight position moves from the outside of the reset zone R to the reset zone R and exceeds a predetermined time. If not, it can be estimated that the user does not want to select an icon included in a block near the reference block B, or has not yet decided which icon to select. In such a case, the icon desired by the user can be selected quickly and efficiently by resetting the count values of all blocks and setting the reference block again.

(Modification)
The present invention is not limited to the above embodiment, and includes, for example, the following modifications.

(1) Selection of GUI In the embodiment of the present invention, an icon is included in each block, an icon included in the block is selected according to the user's line-of-sight position, and a program corresponding to the icon is executed. It is not limited to. A GUI may be included in each block, a GUI included in the block may be selected according to the user's line-of-sight position, and a program corresponding to the GUI may be executed. In addition to icons, the GUI includes menu items, buttons, widgets, and other display objects that can be operated.

(2) Selection Method Combining Gaze Time and Number of Passes of Gaze Trajectory In the embodiment of the present invention, an icon selection method based on gaze time (first method) and icon selection based on the number of passes of the gaze trajectory Two methods (second method) are executed in parallel, the icon previously selected by any of the methods is notified to the program execution unit 4, and the program corresponding to the icon is executed. However, the present invention is not limited to this.

  Weighted addition of the line-of-sight residence time X of each block according to the first method and the count value Y of each block according to the second method to obtain an evaluation value Z of each block, and the evaluation value Z is equal to or greater than a predetermined value The icon corresponding to the block may be selected.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  DESCRIPTION OF SYMBOLS 1 Electronic device, 2 Antenna, 3 Display part, 4 Program execution part, 5 Icon selection part, 6 Count part, 7 Eye-gaze detection part, 8 Wireless communication part, 32 Infrared LED, 30 Infrared camera.

Claims (8)

A display unit for displaying an image, a plurality of blocks are set in the image, and the block includes a GUI.
A line-of-sight detection unit that detects an input position of the line of sight on the display surface of the display unit;
A counting unit that counts the number of times the line-of-sight input position has moved from the outside to the inside of any block;
An execution unit that executes processing corresponding to a GUI included in a block in which the counted value is equal to or greater than a predetermined value;
An electronic device.   If the count value of each block does not become a predetermined value or more after a predetermined time has elapsed since the first line-of-sight input position was detected in each block, the count unit The electronic device according to claim 1, wherein the count value of each block counted by is returned to an initial value. The counting unit sets a block in which the line-of-sight input position is first detected as a reference block, sets the reference block and one or more blocks in the vicinity of the reference block as a detection effective zone,
The electronic device according to claim 1, wherein the count unit performs the count for each block included in the detection effective zone. The selection unit sets one or more blocks remote from the reference block as a reset zone,
The electronic device according to claim 3, wherein when the line-of-sight input position moves to the reset zone, the count unit resets the count values of all blocks and resets the reference block. The selection unit sets one or more blocks remote from the reference block as a reset zone,
The said selection part resets the said count value, and resets the setting of the said reference block, when the input position of the said line of sight moves to the said reset zone from the reset zone a predetermined number of times. Electronics. The selection unit sets one or more blocks remote from the reference block as a reset zone,
When the line-of-sight input position moves from the outside of the reset zone to the reset zone, and the line-of-sight input position does not move outside the reset zone even after a predetermined time has elapsed, The electronic apparatus according to claim 3, wherein the reference block is set again.   The counting unit performs the counting for each block included in the detection effective zone, the counting unit specifies a maximum value among the count values of the plurality of blocks, and a difference from the specified maximum value is The electronic device according to claim 1, wherein one or more blocks having a count value equal to or greater than a predetermined value are excluded from the detection effective zone. An eye-gaze input method for an electronic device including a display unit for displaying an image,
A plurality of blocks are set in the image, and the block includes a GUI.
Detecting an input position of a line of sight on the display surface of the display unit;
Counting the number of times the line-of-sight input position has moved from the outside of any block to the inside;
Executing a process corresponding to a GUI included in a block having a counted value equal to or greater than a predetermined value;
A line-of-sight input method.

Images