JPH03111028A - Gazing direction predicting device - Google Patents

Abstract

PURPOSE:To execute the prediction of a gazing direction by predicting the movement of a mobile object by utilizing reversely a follow-up characteristic which an eyeball motion has at the time of gazing a moving image, and also, predicting the movement of the mobile object by extracting the natural movement of an observer's head part. CONSTITUTION:The gazing direction predicting device is provided with an input part 11, a movement component extraction processing part 12, a first prediction processing part 13, a head part operation detecting part 14, a second prediction processing part 15, and a gazing direction prediction processing part 16. The device is provided with a first prediction processing part 13 for executing a prediction by using the movement component extraction processing part 12 for extracting a scene whose movement is comparatively remarkable in the moving image, and a second prediction processing part 15 for predicting the direction in which a head part moves and the speed of the movement from the head part operation detecting part 14 for detecting the smooth operation of the head part. By a (n-1)-th and an n-th frame image input parts 12-1, 12-2 of the movement component extraction processing part 12, an (n-1)-th frame image and an n-th frame are accumulated successively in a buffer, and an inter-frame difference is extracted by a difference image generating part 12-3. A moving vector component extracting part 12-4 extracts the side of the area of a low correlation from in a difference image, and the moving vector of the mobile object is found by the moving vector component extracting part 12-4, and displayed by an image display part 12-5.

Description

[Detailed Description of the Invention] [Field of Industrial Application] The present invention provides a method of moving the direction of gaze by inversely utilizing the visual mechanism inherent in the eyeballs that follows relatively remarkable scenes of movement in moving images. The present invention relates to a gaze direction prediction device that predicts the final gaze direction by using both or one of the prediction of the direction of motion and the prediction of the motion direction in the natural motion process of two heads.

(Prior art) Fig. 7 is a diagram illustrating a state in which a command is given to determine the target position based on the aviator's gaze direction.The aviator's gaze direction is detected by the line-of-sight direction detector 3, and As a result of detecting the direction of gaze on the radar L of
Many manual operations cannot be used, and visual operations are indispensable, as shown in Figure 1. This is an example in which the detected gaze direction is effectively used.

FIG. 8 is a diagram illustrating a system for projecting on a display a landscape obtained according to the direction of gaze. Landscape 4 is
The information is input from the camera 5 and displayed on the display screen 8. At this time, the gaze direction of the subject 7 is detected by the gaze direction detector 6. As a result, the output from the camera 5 equipped with a variable focal length mechanism and the detection result from the gaze direction detector 6 are linked, and the same effect as when viewing a natural landscape is reproduced on the display screen 8. Ru.

[Problem to be solved by the invention]

Even in the case of the configuration shown in FIG. 7 above, since the conventional case did not have a prediction mechanism, there still remains a problem in realizing prompt target display results. Since the variable mechanism has a control delay, the test subject is required to move very slowly, and rapid movement cannot be expected.

As described above, the conventional gaze direction detection device did not have a prediction mechanism for predicting the gaze direction, and therefore there was a problem in effectively realizing a state of interlocking with a mechanism having a control system.

The present invention solves the problems of the conventional method described above, and in order to compensate for the time delay inherent in the control system when controlling peripheral devices etc. by the direction of gaze, the present invention aims to solve the problems of the conventional method. The purpose of this paper is to predict the direction of gaze by making inverse use of the following mechanism and by predicting the direction of motion in the natural motion process of one head.

(Means for Solving the Problems) In order to achieve the above object, the present invention has a processing section that predicts the eye movement direction from the processing result from the motion component extraction processing section from the moving image, and a camera input section. The main feature is that it has two prediction processing units, a processing unit that predicts the direction of movement of the head from the processing results from the head motion detection unit, and predicts the gaze direction of one person.

This differs from the prior art in that a prediction device is added to the gaze direction device.

〔Example〕

Figure 1 shows the configuration of an embodiment of the present invention with reference numeral 11 in Figure 0.
12 is an input section, and 12 is a motion component extraction processing section.

13 is a first prediction processing unit, and 14 is a head motion detection unit.

Reference numeral 15 represents a second prediction processing unit, and reference numeral 16 represents a gaze direction prediction processing unit.6 In the case of the present invention, two means are used to predict the gaze direction. A first prediction processing section 13 performs prediction using a motion component extraction processing section 12 that extracts significant scenes, and the other one uses a head motion detection section 14 that detects smooth motion of one head. Second prediction processing unit 15 that predicts moving direction and moving speed
It has a configuration that has the following.

FIG. 2 shows the configuration of an embodiment of the motion component extraction processing section. In FIG. 12-4 is a differential image generation unit, 12-4 is a motion vector component extraction unit,
12-5 represents an image display section, which sequentially stores the (n-1)th frame image and the nth frame image among the continuously captured images in a buffer, and calculates the difference between the frames as a difference image. It is extracted in the generation unit 12-3. The motion vector component extraction unit 12-4 divides the difference image into a region with high correlation and a region with low correlation, and extracts the region with low correlation. In this way, the motion vector of the moving object is extracted from several regions by the motion vector component extraction unit 12-
4, the result is displayed on the image display section 12-5.

FIGS. 3A and 3B are explanatory diagrams for extracting the maximum motion vector within an image. The motion vector component of the object 25 on each frame 22, 23, 24 captured at a predetermined time interval is determined by the position 126 of the moving object in the (n-1)th frame and the nth The direction and magnitude of the motion component 127 can be easily calculated from the position 128 in the frame. Generally, each frame includes a plurality of motion vectors, and the motion vector component extraction unit 12-4 extracts the maximum motion vector from each frame.

FIG. 4 is an explanatory diagram showing how a person's eyeballs follow a moving object. When the moving object 25 moves in the direction that the eyeball is viewing (FIG. 4(A)), the eyeball 2
6, the eyeballs 26 rotate in opposite directions as shown by arrows. When the moving object 25 crosses in front of the eyeball 26 (FIG. 4(B)), the 1wi sphere 26 rotates in the same direction. In the case where the moving object 25 moves on the line of sight vector of one eyeball 26 (FIG. 4(C)).

One of the eyeballs 26 rotates in accordance with the moving direction of the object 25, and the other eyeball 26 does not rotate and the focusing mechanism of the eyeball is adjusted.

Based on this understanding of the eye movement mechanism, we can utilize the tracking characteristics of eye movement for moving objects in reverse.

Gaze direction can be predicted. The first prediction processing unit I3 processes this process.

FIG. 5 is a diagram illustrating a typical smooth movement process of the head. This smooth movement of the head 27 shows the case where the 11 part 27 only makes a translational movement (Fig. 5 (A)) and the case where it makes a rotational movement (Fig. 5 (B)). This tends to occur as a natural movement when observing.

FIGS. 6(A) and 6(B) are diagrams showing how the amount of movement of the head is detected. A movement direction prediction processing section 30 predicts the moving direction of the head 27, which is captured by the input camera 28 and detected by the head movement amount detection processing section 29 based on image processing. In the processing unit 30, linear prediction is performed from information such as the past movement speed and each speed of the head movement. The second prediction processing unit 15 processes this process.

Based on the prediction from the first prediction processing section 13 and the prediction from the second prediction processing section 15, the gaze direction prediction processing section 1
6 detects the gaze direction in real time.

〔Effect of the invention〕

As explained above, according to the second aspect of the present invention, the movement of a moving object is predicted by inversely utilizing the tracking characteristics of eyeball movements when gazing at a moving image.

Furthermore, the direction of gaze can be predicted by predicting the movement of the moving object by extracting the natural movement of the observer's head.

[Brief explanation of drawings]

Fig. 1 shows the configuration of an embodiment of the present invention, Fig. 2 shows the configuration of an embodiment of the motion component extraction processing section, and Figs. 3 (A) and (B) are explanatory diagrams for extracting the maximum motion vector in an image. , 4th
The figure is an explanatory diagram showing how a person's eyeballs follow a moving object, Figure 5 is a diagram explaining a typical smooth movement process of the head, and Figure 6 is an illustration of the amount of head movement. A diagram showing detection status. Figure 7 shows a command to determine the target position based on the direction of the pilot's gaze. FIG. 8 is a diagram illustrating a system for displaying a landscape obtained according to the direction of gaze on a display. 2: motion component extraction processing section; 13: -th prediction processing section; 14: head motion detection section; 15: second prediction processing section; 16: gaze direction prediction processing section 12-; n-1) Frame image input unit. 12-2 F nth frame image input unit, 12-1 difference image generation unit, 12-4: motion vector component extraction unit, 12-
! lz image display section.

Claims (3)

[Claims] (1) A gaze direction prediction device that predicts the direction in which an observer gazes at a moving object is equipped with a motion component extraction processing section that extracts a vector component indicating the maximum motion from within a moving image, and the motion component extraction processing section A gaze direction prediction device comprising a first prediction processing unit that predicts a gaze direction based on the movement of an eyeball that follows the movement of a moving object having the extracted maximum motion vector component. (2) In a gaze direction prediction device that predicts the direction in which an observer gazes at a moving object, a head motion detection unit that performs image processing on a head image input by the input unit to determine the direction of movement of the observer's head. A gaze direction prediction device comprising: a second prediction processing unit that predicts a gaze direction based on the amount of movement of the head. (3) A gaze direction prediction device for predicting the direction in which an observer gazes at a moving object, comprising both the first prediction processing section and the second prediction processing section. .