JPH0736610A - Presentation image control device - Google Patents

Abstract

(57) [Abstract] [Purpose] To provide a presentation image control device capable of automatically controlling an operation state of the image presentation device so as to match an operator's request. An image presentation device 1 for presenting an external image to an operator or a camera device for recording an external image, an eye movement detection device 20 for detecting the movement of the operator's eyeball, and an eye movement potential signal detected are used. The image presentation device or the presentation state control device 40 for controlling the camera device is provided, and the image presentation device 1 or the camera device is dimmed, zoomed, and the night-vision camera and the normal camera are switched based on the operator's eye movement. Presentation image control device that automatically controls.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a presentation image control device used for various simulation devices, remote control devices, virtual reality devices and the like.

[0002]

2. Description of the Related Art A presentation image control device is a device for controlling a presentation state such as dimming or zooming of an image presented to an operator, for example, a video camera image or a computer graphics image. As a conventional image presentation device, there is, for example, one shown in FIG. Figure 1
FIG. 9 is a top view of an image presentation device (or a camera device, hereafter represented by an image presentation device) 1 such as a video display device and the operator's head 2. FIG. 19 shows a state in which the operator is looking at the presented image with the image presentation device 1 applied to his or her right eye. In the conventional device as above,
The operator manually controls the image presenting device while looking at the image presented on the image presenting device to adjust the brightness, enlargement, reduction, etc. of the image to meet the operator's request. Also, a device has been developed in which the operator's line-of-sight direction is detected and the light of the illumination device is applied to the line-of-sight direction.

[0003]

As described above, in the conventional image presenting apparatus, the operator manually operates the image presentation state, so that a part of the operation of the operator's hand is There is a problem that the operability is poor because it is limited. Further, even in the case of illuminating light in the direction of the operator's line of sight, the illumination state is simply controlled automatically, and the operator still has to manually control the presentation image.

The present invention has been made in order to solve the problems of the prior art as described above, and it is possible to automatically control the operating state of the image presenting apparatus so as to meet the operator's request. An object is to provide an image control device.

[0005]

In order to achieve the above object, the present invention is constructed as described in the claims. That is, in the invention described in claim 1, an image presenting device for presenting an external image to the operator or a camera device for converting the external image into an electric signal, and an eye movement detecting device for detecting the movement of the eyeball of the operator. And a presentation state control device that controls the image presentation device or the camera device using the detected eye movement potential signal. The eye movement detecting device, for example, as described in claim 2, of the eye movement potential signal detected by the vertical component detecting electrodes attached to the upper and lower sides of the eye and the horizontal component detecting electrodes attached to the left and right of the eye. The open state, the closed state, and the attention state of the eye are detected based on the vertical component and the horizontal component, and the image presentation device is, for example, an external camera image or computer graphics image providing device. Yes, the presentation state control device, for example, based on the detection result of the eye movement detection device, dimming the presentation image, switching between night-vision camera and normal camera, zooming, video on / off, recording on / off. At least one presentation state control of OFF is performed. The above eye movement also includes eyelid opening / closing movement. That is, it is collectively referred to as eye movement including the opening and closing of the eyelids.

[0006]

As described above, in the present invention, the eye movement of the operator is detected, and various states of the presented image are controlled in accordance with the detected eye movement potential signal. The manual operation by the operator can be omitted. An example of the correspondence between the above-mentioned eye movement and the presentation image control is shown, for example, as described in Examples below, when it is determined that the eye is strongly opened, the presentation image is controlled to be bright, and the eye is strongly closed. When it is determined that the state is the state, the presenting image is controlled to be dark. Further, the strong eye opening state and the eye closed state as described above, in the eye movement detection device, the eye movement detected by the vertical component detection electrodes attached to the top and bottom of the eyeball and the horizontal component detection electrodes attached to the left and right of the eyeball. It can be detected based on a vertical component and a horizontal component of the potential signal (for example, FIGS. 7 to 10 described later).

[0007]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings.
1A and 1B are views showing an embodiment of the present invention. FIG. 1A is a perspective view of the appearance of the apparatus, and FIG. 1B is a block diagram. In FIG. 1, reference numeral 1 denotes an image presenting device for presenting an external image to an operator, such as a video display device or a video camera device.
Further, 20 is an eye movement detecting device (details will be described later) for detecting the eye movement of the operator, 21 is a signal detecting unit in the eye movement detecting device 20, and 22 is also a signal processing unit. Also, 40
Is a presentation state control device (details described later) that controls the image presentation device 1 using the detected eye movement potential signal.

Next, FIG. 2 is another embodiment of the present invention, (a) is a perspective view of the appearance of the apparatus, and (b) is a block diagram. This embodiment is an example in which the present invention is applied to a device that presents a three-dimensional stereoscopic image. In FIG. 2, 10 is a head-mounted three-dimensional stereoscopic image presentation device that presents an external image to the operator, 20 is an eye movement detection device that detects the eye movement of the operator, 21 is a signal detection unit, and 22 is signal processing. Reference numeral 30 denotes a stereo camera type image providing device, which sends a signal thereof to the head-mounted 3D stereoscopic image presentation device 10 to provide an external image to the operator. Reference numeral 40 is a head-mounted three-dimensional stereoscopic image presentation device 10 using the detected eye movement potential signal.
Alternatively, the presentation state control device controls the image providing device 30.

Next, FIG. 3 is a perspective view of a structural example of the head-mounted three-dimensional stereoscopic image presentation device 10. In FIG. 3, reference numerals 11 and 11 'denote small-sized display devices (hereinafter referred to as small-sized televisions) for presenting an external image obtained from the image providing device 30 to an operator, such as a CRT or a liquid crystal display. Reference numerals 12, 12 ′ and 14 are mirrors for optically guiding the image of the small television 11 to the eyes of the operator, 13 is a convex lens, and 21 is a signal detection unit. The images radiated from the two small televisions 11 and 11 'are mirror 1 respectively.
After being rotated by 90 ° at 2, 12 ′ and passing through the convex lens 13, they are combined by the mirror 14 and rotated by 90 ° so as to enter the operator's eye. Therefore, this device is a mirror 1
4 is mounted so that the operator's eyes are positioned in front of 4, and at that time, the operator's eyes are mounted on the operator's face such that the position of the operator's eyes is between the plurality of signal detection units 21. The details of the electrode mounting portion of the signal detecting unit 21 will be described later with reference to FIG.

Next, FIG. 4 is a side view of a structural example of the signal detecting section 21 in the eye movement detecting device 20. In FIG. 4, 211 is a signal deriving electrode for detecting an eye movement potential signal, 212 is a suction plate for fixing the electrode to the operator, 213 is the image providing device 1 or the head-mounted three-dimensional stereoscopic image presentation device 10. When the operator puts on the electrode 21
This is a spring for ensuring that 1 is accurately fixed.

Next, FIG. 5 is a view showing a mounting position of the signal derivation electrode 211 on the operator's face. In FIG. 5, reference numeral 230 denotes the operator's eye position, and 231 denotes the nose position. Further, there are three types of signal derivation electrodes 211: a vertical component detection electrode 232, a ground electrode 233, and a horizontal component detection electrode 234. And the vertical component detection electrode 23
2 is attached above and below the eye, for example, directly above the eyebrow and about 20 mm below the eye, the ground electrode 233 is attached between the eyebrows, and the horizontal component detection electrode 234 is on the left and right of the eye, for example, about 15 mm from the outer corners of both eyes.
Attach to the outside.

Next, the operation will be described. This example is an example for a person to work under a strong light ray or extreme darkness, which is difficult for the human to see directly. FIG. 6 is a diagram showing an operator's eyeball (eyelid) state depending on the brightness of the presented image. In FIG. 6, (a) shows an eye open state that is stronger than normal eye opening, and (b) shows an eye closed state that is stronger than normal eye opening. That is (a)
Is a state in which the operator focuses his attention on the object in an extremely dark work environment, and (b) assumes a state in which the operator closes his eyes in the extremely bright work environment or strongly narrows his eyes. . Moreover, (c) is a normal attention state. In the above situation, the eye movement potential signals detected by the signal deriving electrode 211 are shown in FIGS.
Shown in. 7 is a normal state, FIG. 8 is a strongly opened state, FIG. 9 is a strongly closed state, and FIG. 10 is an eye potential diagram showing an eye movement potential signal in a focused state. In each of FIGS. 7 to 10, the horizontal component (horizon) of the eye movement potential signal is displayed in order from the top.
tal), the result of filtering horizontal components (af
ter filtering), a vertical component (vertical), and a result (afterfiltering) of filtering processing of the vertical component. The sampling frequency of the signal is 1kHz,
The horizontal axis represents time in units of sec, and the vertical axis represents the normalized potential. Focusing on the vibration component that vibrates finely,
It is confirmed that the vibrating component appears only in the vertical component in the strongly opened state of FIG. 8, whereas the vibrating component appears in both the vertical and horizontal components in the strongly closed state of FIG.
Moreover, this vibration component is a component that does not appear in the electrooculogram (FIG. 7) during normal eye movement. Therefore,
By detecting this vibration component, the intention of the operator can be accurately determined. Further, focusing on the blink component appearing in the vertical component, it is confirmed that the blink component is stopped during the attention time in the attention state of FIG. 10 compared to the normal state of FIG. 7.

In the following, focusing on the above points, the signal processing unit 2
The means for determining the eyeball state of the operator performed in 2 will be described. FIG. 11 is a signal processing flowchart for the open / closed eye determination. In FIG. 11, the eye movement potential signal detected for each of the vertical and horizontal components is first cut out by the data cutout 221 by 512 ms, for example. Next, the cut-out signal is converted into a 10 to 100 Hz component by the bandpass filter 222, and the baseline fluctuation component is removed. By this processing, the eye movement potential signal is converted into an extremely characteristic signal as shown in (after filtering) of FIGS. 8 and 9. Next, the fast Fourier transformer 22
The frequency is decomposed by 3. Next, in step 225, it is determined whether or not there is a vibration component in the vertical component in the frequency component obtained by the frequency decomposition. Since the frequency of this vibration component is about 50 to 100 Hz component, if the vibration component is included in half of the cutout section by setting 50 Hz as the threshold value, about 250 ms after the start of a predetermined eyeball state.
It can be determined later. Where 50-100
If there is no Hz component, after shifting while cutting the segment movement while overlapping with a shift width of 32 ms,
Return to the data cutout 221. On the other hand, if there is a 50-100 Hz component in step 225, step 2
At 26, it is determined whether or not there is a vibration component in the horizontal component.
Here, when there is no 50-100 Hz component, it is determined that the eye is in the strong eye open state shown in FIG. 6A, and when there is a 50-100 Hz component, it is determined that the eye is in the strong eye closed state shown in FIG. 6B. As described above, when it is determined that the eye opening state is strong, the presentation image of the image providing apparatus 1 or the head-mounted three-dimensional stereoscopic image presentation apparatus 10 (hereinafter, represented by the image presentation apparatus 1) is brightened and the eyes are strongly closed. When it is determined that the state is the state, the presenting image is controlled to be dark. By repeatedly performing the above processing in real time, it is possible to perform the light control of the image presented to the image providing apparatus 1. It should be noted that, in the case of malfunction or when the control by the eye movement becomes unnecessary, the manual switch can always return to the initial state.

Next, FIG. 12 is a block diagram of another construction for the eye open / closed eye determination similar to the signal processing of FIG. 12
In FIG. 11, the bandpass filter 222 is a filter for removing the baseline fluctuation similar to that in FIG. The determination circuit 224 is a circuit that determines the presence or absence of the 50 Hz to 100 Hz component by comparing it with a predetermined threshold value. Then, by a circuit that combines two AND circuits and a NOT circuit, when there are both 50 Hz to 100 Hz components in the vertical component and 50 Hz to 100 Hz components in the horizontal component, there is a closed eye condition, and there is a vertical component and no horizontal component. It is determined that the eyes are open.

Next, FIG. 13 is a flow chart showing the process of blinking component detection. In FIG. 13, in step 227, the baseline fluctuation is removed by a bandpass filter. Further, in step 228, it is determined whether or not there is a 0.1 to 10 Hz component including a blink component with a predetermined threshold value.
Further, in step 229, when the blinking component is not detected for about 2 seconds, it is determined to be the attention state.

An embodiment in which the image presenting apparatus 1 or the image providing apparatus 30 is controlled to control the presented image based on the signal processing result of the open / closed eye and the attention state as described above will be described below. 14 is a block diagram of an embodiment for performing dimming control, FIG. 15 is a block diagram of an embodiment for performing night vision camera control, FIG. 16 is a block diagram of an embodiment for performing zooming control, and FIG. 17 is computer graphics. It is a block diagram of an example which performs screen control. First, in the embodiment of FIG. 14, the electronic switch 41 is operated by the eye opening signal and the eye closing signal determined by the eye movement detecting device 20. Then, when the eye opening signal is given, the electronic switch 41 operates so as to raise the dimming by one step, and the dimming raising control circuit 42 operates to raise the brightness of the image presentation device 1. Further, when the eye closing signal is given, the electronic switch 41 operates so as to lower the dimming by one step, and the dimming lowering control circuit 43 operates to lower the brightness of the image presentation device 1. It should be noted that it is also possible to control the switching of television / video images, ON / OFF of recording, etc. by using the eye opening signal and the eye closing signal as described above.

Next, in the embodiment of FIG. 15, the image providing apparatus 30 is provided with a night-vision stereo camera 31 and a normal stereo camera 32. Then, the electronic switch 44 transmits the image information of the night-vision stereo camera 31 to the image presentation device 1 when the eye-opening signal of the eye movement detecting device 20 is turned on, and the image information of the normal stereo camera 32 is imaged when the eye-closing signal is turned on. The switching control is performed so that the presentation device 1 is notified. As a result, when the eyes are naturally opened in a dark place, the camera system becomes a night-vision camera for easy viewing, and when the eyes are naturally closed, the camera system is switched to the normal camera.

Next, in the embodiment shown in FIG. 16, when the eye-opening signal of the eye movement detecting device 20 is output, the electronic switch 45 is switched and the zoom control circuit 46 operates to operate the stereo camera (not shown) of the image providing device 30. No) enters the zoom operation. Further, when the eye-closing signal of the eye movement detecting device 20 is output, the electronic switch 45 is switched and the wide control circuit 4
6 operates and the stereo camera enters wide operation.
As a result, when the operator wants to pay attention and the eyes open, the camera system naturally enters the zoom, and when the operator wants to return to the original, the eyes switch to wide by narrowing the eyes. You can do it.

Next, in the embodiment of FIG. 17, the electronic switch 49 is set by the attention signal output from the eye movement detecting device 20 and the origin for computer graphics is set via the CG origin moving circuit 49-1. Move.
At this time, the one-shot timer 48 measures the elapsed time of 5 seconds, and after the elapsed time, the CG origin return circuit 49-
Return to the origin via 2. This enables computer graphics (CG) screen control.

FIG. 18 is a perspective view of another embodiment of the present invention, showing one embodiment of zooming control in remote operation. In FIG. 18, reference numeral 100 is a head-mounted three-dimensional stereoscopic image presentation device that presents an external image to the operator, and reference numeral 200 is an eye movement detection device that detects the eye movement of the operator. Further, 300 is a stereo camera type image providing device,
The external image thus detected is provided to the operator via the head-mounted 3D stereoscopic image presentation device 100. 400
Is a presentation state control device that controls the head-mounted three-dimensional stereoscopic image presentation device 100 or the image providing device 300 using the detected eye movement potential signal, and 500 is a work arm of a robot that performs remote operation. Further, reference numeral 600 is a work arm operating device for remotely operating the work arm 500 by an operator. In the present embodiment, for example, the dimming control shown in FIGS. 11 to 13, that is, the presented image is brightened when it is determined that the eye is strongly opened, and the presented image is darkened when the eye is strongly closed. By combining the control operation described above and the zoom / wide control shown in FIG. 16, it is possible to automatically control the image of the work target performed by the work arm to an optimum state.

Next, in the above-described embodiments, the control of dimming, zooming, etc. is explained in response to the natural movement of the eyeball, but the specific movement of the eyeball and the specific control are corresponded. It is also possible to control automatically. That is, as described above, it is possible to detect the opening and closing of the eye and the state of attention from the vertical and horizontal components of the eye movement potential signal, so that specific control can be performed by the movement of the eyeball of the operator. For example, recording is started when the eye is strongly opened, and recording is stopped when the eye is strongly closed. On / off can be controlled automatically. In this way, when the operator performs a predetermined eye movement, a predetermined specific control corresponding to the movement (for example, turning on / off the image) is performed.
Off, recording on / off, etc.) can be performed automatically.

[0022]

As described above, according to the present invention, a device for detecting the eye movement potential signal of the operator is provided, and on the basis of it, dimming of the presented image, switching to the night-vision camera, zooming, Since the presentation state control such as on / off of the recording is performed, the effect that the operator can omit the manual operation for performing the presentation state control of the presentation image is obtained. Also, judging from the eyeball state of the operator, when the operator wants to see the details, the presented image is brightened or presented by zooming, and in the opposite case, the presented image is darkened or displayed wide, It is possible to control the presented image based on the natural behavior of the operator. Further, when the head-mounted 3D stereoscopic image presentation device is used, the operator does not directly see the object but indirectly through the head-mounted 3D stereoscopic image presentation device. While maintaining the original sense of distance, it is possible to obtain high-value-added visual information that is impossible with direct vision.

[Brief description of drawings]

1A and 1B are diagrams showing an embodiment of the present invention, in which FIG. 1A is a perspective view of the appearance of an apparatus, and FIG.

2A and 2B are views of another embodiment of the present invention, in which FIG. 2A is a perspective view of the appearance of the apparatus, and FIG.

FIG. 3 is a perspective view of a configuration example of a head-mounted 3D stereoscopic image presentation device 10.

FIG. 4 is a side view of a configuration example of a signal detection unit 21.

FIG. 5 is a diagram showing a mounting position of a signal derivation electrode 211 on the operator's face.

FIG. 6 is a diagram showing an operator's eyeball (eyelid) state depending on the brightness of a presented image.

FIG. 7 is an eye potential diagram showing an eye movement potential signal in a normal state.

FIG. 8 is an eye potential diagram showing an eye movement potential signal in a strongly opened state.

FIG. 9 is an ocular potential diagram showing an eye movement potential signal in a strongly closed state.

FIG. 10 is an eye potential diagram showing an eye movement potential signal in a state of attention.

FIG. 11 is a signal processing flowchart for opening / closing eye determination.

FIG. 12 is a block diagram of another configuration of the open / closed eye determination.

FIG. 13 is a flowchart showing a process of blink component detection.

FIG. 14 is a block diagram of an embodiment for performing dimming control.

FIG. 15 is a block diagram of an embodiment for performing night-vision camera control.

FIG. 16 is a block diagram of an embodiment for performing zooming control.

FIG. 17 is a block diagram of an embodiment for performing computer graphics screen control.

FIG. 18 is a perspective view of another embodiment of the present invention.

FIG. 19 is a top view of an example of a conventional image presentation device.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Image presentation device 10 ... Head-mounted 3D stereoscopic image presentation device 11, 11 '... Small display device (small television) 12, 12', 14 ... Mirror 13 ... Convex lens 42 ... Dimming control circuit 20 ... Eye movement detection device 43 ... Dimming control circuit 21 ... Signal detection unit 44 ... Electronic switch 22 ... Signal processing unit 45 ... Electronic switch 30 ... Stereo camera type image providing device 46 ... Zoom control circuit 31 ... Night vision stereo camera 48 ... One-shot timer 32 ... Normal stereo camera 49 ... Electronic switch 40 ... Presentation state control device 49-1
… CG origin movement circuit 41… Electronic switch 49-2
... CG origin return circuit 211 ... Signal deriving electrode 230 ...
Position of operator's eyes 212 ... Adsorption plate 231 ...
Position of operator's nose 213 ... Spring 232 ...
Vertical component detection electrode 222 ... Bandpass filter 233 ...
Grounding electrode 223 ... Fast Fourier transformer 234 ...
Horizontal component detection electrode 224 ... Judgment circuit 100 ... Head-mounted three-dimensional stereoscopic image presentation device 200 ...
Eye movement detecting device 300 ... Stereo camera type image providing device 400 ...
Presentation state control device 500 ... Robot work arm 600 ...
Work arm operating device

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification number Office reference number FI technical display location H04N 5/00 A 5/232 Z 7/18 U

Claims (2)

[Claims] 1. An image presenting device for presenting an external image to an operator or a camera device for converting the external image into an electrical signal, an eye movement detecting device for detecting the movement of the operator's eyeball, and a detected eye movement potential signal. And a presentation state control device that controls the image presentation device or the camera device by using the presentation image control device. 2. The eye movement detecting apparatus according to claim 1, wherein the vertical component and horizontal component of the eye movement potential signal detected by the vertical component detecting electrodes mounted on the upper and lower sides of the eyeball and the horizontal component detecting electrodes mounted on the right and left sides of the eyeball. Based on the open, closed,
For detecting the state of attention, the image presentation device is an external camera image or a computer graphics image providing device, the presentation state control device, based on the detection result of the eye movement detection device, At least one presentation state control of dimming a presented image, switching between a night-vision camera and a normal camera, zooming, video on / off, and recording on / off is performed. 1. The presentation image control device according to 1.