JPH0583246B2 - - Google Patents

Description

[Detailed description of the invention]

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight display device, and more particularly to a line-of-sight display device that displays the movement of the line of sight of an observed person having a wide field of view.

[0002]

[Prior Art and Problems to be Solved by the Invention] Conventionally, in analyzing the movement of the line of sight of a driver who is driving or piloting a car or aircraft, it has been necessary to analyze the influence of the movement of the driver's head on the movement of the line of sight. Only the movement of the driver's line of sight due to the movement of the driver's eyeballs was analyzed without taking into account the Therefore, it is necessary to accurately detect movement of the line of sight by taking into account not only eye movement but also head movement, and the applicant of the present application has already proposed this method.

[0003] The detected movement of the line of sight is displayed superimposed on an image of the external situation seen from the driver's seat. Although it is preferable that the external situation be photographed by one camera, it is generally difficult to photograph the external situation with one camera because the driver's field of view is very wide. Even if it is possible to photograph the driver's entire field of view using a fisheye lens, distortion will occur in the periphery of the image. Therefore, even if the line of sight is displayed superimposed on the image, it is difficult to analyze the movement of the line of sight.

[0004] This invention was made in order to solve the above-mentioned problems, and provides a line of sight display device that can display the movement of the line of sight of an observed person with a wide field of view on an undistorted background image. The purpose is to

[0005]

[Means for Solving the Problems] A visual line display device according to the invention of claim 1 includes: an eyeball movement detecting means for detecting the movement of the eyeball of an observed person; and a head movement detecting means for detecting the movement of the observed person's head. The motion detection means calculates the movement of the observed person's line of sight by processing the output data from the eye movement detection means and the head movement detection means, and divides the observed person's visual field into multiple parts. out of a plurality of image signals captured by a plurality of image capture means in response to the movement of the observed person's line of sight obtained by a plurality of image capture means that separately image each part, and a line of sight calculation means. a video selection means for selecting some of the video signals, and displaying the selected video based on the video signal selected by the video selection means, and displaying the selected video on the displayed video by the line-of-sight calculation means. and display means for displaying the movement of the line of sight.

[0006] The line of sight display device according to the invention of claim 2 includes: an eye movement detection means for detecting the movement of the eyeball of the observed person; a head movement detection means for detecting the movement of the head of the observed person; A line-of-sight calculation means calculates the movement of the observed person's line of sight by processing the output data from the motion detection means and the head movement detection means, and the visual field of the observed person is divided into a plurality of parts, and each part is A plurality of photographing means for photographing, a video composition means for combining a plurality of images photographed by the plurality of photographing means so that they can be displayed on one screen, and displaying the video composited by the video composition means. and display means for displaying the movement of the line of sight obtained by the line of sight calculation means on the displayed image.

[0007]

[Effect]

In the line-of-sight display device according to the first aspect of the invention, the image selection means selects some of the images to be displayed by the display means according to the movement of the observer's line of sight. The display means displays the selected image, and also displays the movement of the line of sight obtained by the line of sight calculation means on the image. Therefore, the movement of the observer's line of sight can be displayed on an undistorted background image.

[0008] In the line-of-sight display device according to the invention of claim 2, the video compositing means composites the plurality of images taken by the plurality of photographing means so that they can be displayed on one screen. The display means displays the synthesized image, and the movement of the line of sight obtained by the line of sight calculation means is displayed on the image. Therefore, the movement of the observer's line of sight can be displayed on an undistorted background image.

[0009]

Embodiment FIG. 1 is a block diagram of a gaze movement display device showing an embodiment of the present invention. Referring to FIG. 1, this eye movement display device includes a head movement detection device 6 that detects the head movement of an exerciser using a magnetic sensor, and an eye movement detection device 5 that detects the movement of the exerciser's eyeballs. , a gaze point processing device 7 that calculates gaze data of the exerciser by processing the detected head movement data and eyeball movement data; an image processing device 8 that performs image processing of the determined gaze data; It includes a superimposer 9 that superimposes an image-processed line-of-sight signal on a video signal showing an external situation photographed from a seat, and a display device 10 connected to the output of the superimposer 9. This gaze movement display device further includes, for example, four cameras C1 to C4 for photographing the external situation visible from the driver's seat, and cameras C1 to C4 in response to a switching signal SW from the gaze point processing device 7. and an image switching device 3 for selecting a video signal photographed by the camera. Therefore, the superimposer 9 superimposes the line-of-sight signal on the selected video signal indicating the external situation output from the image switching device 3.

[0010] FIG. 2 shows the four cameras C shown in FIG.
FIG. 2 is a layout diagram showing the arrangement of vehicles 1 to C4 in the driver's seat. Referring to FIG. 2, the camera C1 captures an image corresponding to the left field of view including the left side door mirror 21 as seen from the driver 2. Camera C2 is for driver 2
Shoot an image of the front field of view as seen from the front. Camera C
3 shoots an image of the right visual field including the right side door mirror 23 as seen from the driver 2. The camera C4 captures an image including the rearview mirror 22, an indicator 24 such as a speedometer, a shift lever (not shown), and the like. That is, the cameras C1 to C4 are respectively arranged at positions where they can photograph the above-mentioned surrounding situation that can be seen by the driver 2.

[0011] FIG. 5 is a block diagram of the head movement detection device 6 shown in FIG. Referring to FIG. 5, this head motion detection device 6 includes an excitation coil 61 that generates a reference magnetic field, a drive circuit 65 for AC driving the excitation coil 61, and a head motion detection device 6 that is attached to the head of an observed person. A sensor coil 62 , a detection circuit 63 that detects the voltage induced in the sensor coil 62 according to the movement of the head of the person to be observed, and an arithmetic process based on the voltage detected by the detection circuit 63 . and a CPU 64 for determining the head movement of the person. CPU
64 also controls a drive circuit 65. The excitation coil 61 has three coils wound in three axes (X, Y, Z) directions.
It consists of two coils. Each coil is supplied with driving power supply voltages generated from the drive circuit 65 and having different frequencies. Therefore, a magnetic field based on three axes is generated by the excitation coil 61, and the person to be observed is present in this magnetic field. The sensor coil 62 also has (X, Y, Z)
It is equipped with three detection coils wound in each direction. When the head of the person to be observed moves, a starting force is induced in each coil in the sensor coil 62 according to the movement. By detecting the electromotive force generated in each coil by the detection circuit 63, a signal corresponding to the movement of the head of the person to be observed is obtained. The CPU64 is
By performing arithmetic processing on the basis of this signal, head motion data H indicating the movement of the observed person's head is obtained and output.

[0012] FIG. 3 is a schematic diagram showing the direction of each movement parameter of the head movement of the driver 2, who is the observed person. The aforementioned sensor coil 6 is placed on the forehead of the driver 2.
2 is installed. The parameter Hx shown in Figure 3,
Hy, Hz, Hθ, Hφ and Hψ each indicate the movement of the head movement in the direction shown in this figure. That is, Hx indicates horizontal parallel motion of the head, Hy indicates forward-backward motion, and Hz indicates vertical parallel motion. In addition, Hθ indicates the movement in the direction of tilting the neck left and right, Hφ indicates the rotational movement of the neck in the vertical direction,
Hψ indicates the rotational movement of the head in the left-right direction.

[0013] FIG. 4 is a schematic diagram showing how head motion is detected in the driver's seat. Referring to FIG. 4, an excitation coil 61 is mounted at a fixed location near the driver's seat, such as at the bottom of the steering wheel. The sensor coil 62 is attached to the driver's forehead. As the driver's head moves, the relative positional relationship between the excitation coil 61 and the sensor coil 62 changes, and head movement is detected.

[0014] FIG. 6 is a schematic diagram showing the detection principle in the eye movement detection device 5 shown in FIG. As an example, this device uses a scleral reflection method that utilizes the difference in reflectance between the white and iris of the corneal surface. As other methods, a corneal reflection method and a search coil method can also be applied. Referring to FIG.
Sensors 71 and 72 are arranged in front of the left and right eyes E1 and E2, respectively. A light emitting element 73 is arranged at the center of each sensor 71 and 72. As the light emitting element 73, ±
A light emitting diode with an infrared projection of about 21° is used. A light receiving element 74 is provided on both sides of the light emitting element 73. As the light receiving element 74, a photodiode with sharp directivity of approximately ±10° is used. The light projected from the light emitting element 73 toward the eyeball is reflected at the white and iris portions of the eye. The whites of the eyes and the iris have different reflectances, so if we amplify this difference in reflectance and take the difference, we can get the horizontal (left and right) eye movements as an output, and if we add the sum, we can get the output in the vertical (up and down) direction. Eye movement is obtained as output.

[0015] The positions of the sensors 71 and 72 relative to the eyeball are different in the horizontal direction and the vertical direction, with the horizontal sensor 71 detecting reflected light in the center relative to the top and bottom of the eyeball, and the vertical sensor 72 detecting reflected light downward. and are arranged so as to detect the reflected light. A horizontal sensor 71 is placed on one eye E1, and a vertical sensor 72 is placed on the other eye E2. If these are used simultaneously, two-dimensional eye movement can be detected. A method of determining the moving speed, moving direction, moving distance, gaze time, etc. of the eyeball using such a detection principle is disclosed in, for example, Japanese Patent Laid-Open No. 126140/1983. Therefore, data E indicating the driver's eye movement is obtained from the eye movement detection device 5 shown in FIG. 1 and is provided to the gaze point processing device 7.

[0016] FIG. 7 is a schematic diagram for explaining the movement of the line of sight due to head movement. Referring to FIG.
Generally, when a visual object moves, the eyeball moves to follow the visual object. Instead of moving the eyeballs, it is also possible to track the visual object by moving the head. Usually, both movements are used together. Head movements include parallel movements caused by the orientation of the feet and spine, and rotational movements realized by the neck, spine, hips, legs, and the like. This head movement is converted into an eyeball rotation angle by a method described later, and a head movement corrected eyeball rotation angle α shown in FIG. 7 is obtained. Figure 1
In the gaze point processing device 7 shown in , line of sight data is expressed by a combination of the head movement corrected eyeball rotation angle α and the eyeball rotation angle θ. Here, the sum of the rotation angles α and θ is collectively defined as the line of sight.

[0017] The processing in the gaze point processing device 7 shown in FIG. 1 will be described below. The gaze point processing device 7 is
Head movement data H and eye movement data E are received. As horizontal motion components of data H (Hx, Hy,
Hz) is given, and (Hφ,
Hθ, Hψ) are given. The line of sight changes depending on the horizontal component (Hx, Hy, Hz) of the head movement, and the following equation is used to convert this into an eyeball rotation angle (Ex, Ey). Here, D is the distance between the subject and the observation target.

[0018]

[Math. 1] Ex=180/π・tan ^-1 Hx/(D+Hy) ...(1) Ey=180/π・tan ^-1 Hz/(D+Hy) ...(2)

[0019] When the head is tilted by Hθ toward the left shoulder or the right shoulder, the coordinates of the eye movement system rotate. Therefore, the eye movement coordinate system (Xe,
Ye) in the coordinate system (Xe′,
Ye′). Therefore, the following formula is used.

【0020】

[Mathematical 2]
Yh) is expressed by the following equation from equations (1) and (2).

[0021]

[Math. 3] Yh=Ex+Hψ...(5) Yh=Ey+Hφ...(6) Therefore, the movement of the line of sight (Xv, Yv) considering the movement of the head can be calculated from equations (3) to (6). It is expressed by the following formula.

[0022]

[Mathematical 4] It is possible to reproduce normal eye movements. The above-described processing is performed in the gaze point processing unit 7 shown in FIG. 1, and gaze data indicating the movement of the gaze is provided to the image processing device 8.

[0023] In addition to detecting the eye movement described above, the gaze point processing device 7 generates a switching signal SW based on the movement of the eye movement. That is, since the position of the line of sight of the driver 2, that is, the position of the point of gaze is obtained through the above-mentioned arithmetic processing, the camera that is photographing the situation including the position of the point of gaze is selected.
For example, if the horizontal angle of the line of sight is ψ and the vertical angle is φ, then camera C1
It is selected in the range of 30° and −20°<φ<20°. Camera C2 is -30°<ψ<30° and -20°<φ
Selected in the range <20°. Camera C3 is
It is selected in the range of ψ>+30° and −20°<φ<20°. Camera C4 is -20°≦ψ≦20° and φ<
-20° or φ>20°.
Therefore, in response to the switching signal SW, the image switching device 3 shown in FIG. The superimposer 9 is provided with a video signal of the superimposer 9.

[0024] The image processing device 8 is the gaze point processing device 7.
Image processing is performed on the line-of-sight data given from the display device 10, and a line-of-sight signal necessary to display the line of sight on the screen of the display device 10 is output, and the signal is provided to the superimposer 9. The superimposer 9 superimposes the given line-of-sight signal on the video signal given from the image switching device 3. The superimposed video signal is applied to the display device 10, and the movement of the driver's line of sight is displayed on the selected background image on the screen of the display device 10.
8 to 10 illustrate the display device 10 shown in FIG.
It is a screen diagram showing a display example in . FIG. 8 shows a display when the line of sight moves within the video shot by camera C2 shown in FIG. FIG. 9 shows the display when the line of sight moves to the left side door mirror 21. Since the line of sight is on the left side of the image shown in FIG. 8 just before moving, the left side image and the image obtained by camera C1 are displayed simultaneously, and the movement of the line of sight is displayed above them. Similarly, a display example when the line of sight moves to the right side door mirror 23 is shown in FIG.

FIG. 11 is a block diagram of a gaze movement display device showing another embodiment of the present invention. Figure 11
Referring to , this gaze movement display device differs from the device shown in FIG. 1 in that it includes an image synthesizing device 4 instead of the image switching device 3 shown in FIG. The image synthesis device 4 includes the four cameras C1 to C4 described above.
connected to receive video signals from the The image synthesis device 4 displays four images on one screen of the display device 10.
Video signals are synthesized so that images obtained from the two cameras C1 to C4 can be displayed simultaneously.
The combined video signal is given to the superimposer 9. The superimposer 9 superimposes the line-of-sight signal on the combined video signal and provides the superimposed signal to the display device 10. An example of how an image is displayed on the display device 10 is shown in FIG.
2. As shown in FIG. 12, images obtained by four cameras C1 to C4 are displayed on one screen, and the movement of the line of sight is displayed on the screen.

[0026] In this way, the four cameras C1 to C
4, and the movement of the driver's line of sight is displayed on the captured image. Therefore, the movement of the driver's line of sight can be displayed on an undistorted background image, making it easier to analyze the movement of the driver's line of sight.

[0027]

Effects of the Invention As described above, according to the invention of claim 1, a plurality of photographing means for photographing each part of the visual field of the observed person are provided, and the line of sight is placed on the image selected by the image selection means. is displayed, so a line-of-sight display device that can display the movement of the observer's line of sight on an undistorted background image was obtained.

[0028] According to the invention of claim 2, since the image synthesis means is provided to synthesize each part of the visual field of the observed person obtained by the plurality of photographing means so that they can be displayed on one screen, A line-of-sight display device capable of displaying the movement of the observed person's line of sight on an undistorted background image was obtained.

[Brief explanation of drawings]

FIG. 1 is a block diagram of a gaze movement display device showing an embodiment of the present invention.

FIG. 2 is a layout diagram showing the arrangement of the four cameras shown in FIG. 1 in a driver's seat.

FIG. 3 is a schematic diagram showing the direction of each motion parameter of the driver's head motion.

FIG. 4 is a schematic diagram showing head motion detection in the driver's seat.

FIG. 5 is a block diagram of the head motion detection device shown in FIG. 1.

6 is a schematic diagram showing a detection principle in the eye movement detection device shown in FIG. 1. FIG.

FIG. 7 is a schematic diagram for explaining the movement of the line of sight during head movement.

FIG. 8 is a screen diagram showing a display example of the display device shown in FIG. 1;

9 is a screen diagram showing another display example of the display device shown in FIG. 1. FIG.

10 is a screen diagram showing still another example of the display device shown in FIG. 1. FIG.

FIG. 11 is a block diagram of a gaze movement display device showing another embodiment of the present invention.

12 is a screen diagram showing a display example of the display device shown in FIG. 11. FIG.

[Explanation of symbols]

3 Image switching device 4 Image synthesis device 5 Eye movement detection device 6 Head movement detection device 7 Gaze point processing device 10 Display device.

Claims (2)

[Claims] 1. A line-of-sight display device that displays movement of the line of sight of an observed person having a wide field of view, comprising: eye movement detection means for detecting movement of the observed person's eyeballs; and head movement of the observed person. a head movement detection means for detecting the movement of the subject; a line of sight calculation means for calculating the movement of the observed person's line of sight by processing output data from the eye movement detection means and the head movement detection means; a plurality of photographing means that divides an observer's visual field into a plurality of parts and photographs each part; and a plurality of photographing means that respond to the movement of the observed person's line of sight obtained by the line of sight calculation means. video selection means for selecting some of the plurality of video signals shot by the video selection means; displaying the selected video based on the video signals selected by the video selection means; and display means for displaying the movement of the line of sight obtained by the line of sight calculation means on the image. 2. A line-of-sight display device that displays the movement of the line of sight of an observed person having a wide field of view, comprising: eye movement detection means for detecting the movement of the observed person's eyeballs; and a head of the observed person. a head movement detection means for detecting the movement of the subject; and a line of sight calculation means for calculating the movement of the observed person's line of sight by processing output data from the eye movement detection means and the head movement detection means; A plurality of photographing means for dividing the field of view of the observed person into a plurality of parts and photographing each part, and combining the plurality of images photographed by the plurality of photographing means so that they can be displayed on one screen. and display means for displaying the image synthesized by the image compositing means and displaying the movement of the line of sight obtained by the line of sight calculation means on the displayed image. Display device.