JP2025123769A - Gaze guidance method and gaze guidance program - Google Patents

Abstract

To definitely guide a line of sight.SOLUTION: An information processing apparatus 1 is configured to: display, on a display apparatus 2, an image of a three-dimensional virtual space area seen from a viewpoint of a user from among three-dimensional virtual space areas; detect the position of a user's line of sight in the image and display a predetermined line of sight position image 4 at the position of the line of sight in the image; determine whether the movement of the user's line of sight satisfies a predetermined movement condition in a state where the line of sight position image 4 is displayed; and display, if the movement condition is satisfied, the line of sight position image 4 shifted by a predetermined amount from the position of the line of sight toward a target position 6a based on the relative positional relationship between the direction of the user's line of sight and the target position 6a set in a virtual space.SELECTED DRAWING: Figure 1

Description

The present invention relates to a gaze guidance method and a gaze guidance program.

In recent years, virtual reality (VR) technology has been applied to a variety of fields. For example, images showing a three-dimensional virtual space that recreates a workplace are displayed on a head-mounted display (HMD) worn by the user. By depicting the situation in the virtual space from the user's perspective based on the orientation and position of the user's head, the user can experience the feeling of being in the workspace. This type of technology is being used in training and education to acquire skills and improve them.

Furthermore, the following proposals have been made regarding technology for displaying images in virtual space. For example, an image processing device has been proposed that displays a guide image on a displayed image to guide the user in the direction they should look. Also proposed is a gaze guidance device that synthesizes and displays a motion stimulus image for guiding the gaze in a peripheral visual field that is preset at the edge of a displayed image.

Japanese Patent Application Laid-Open No. 2019-114822 Japanese Patent Application Laid-Open No. 2023-160623

One way to guide the user's gaze in a specific direction while a virtual space is displayed is to display an explicit image for guiding the gaze, like the guide image mentioned above. While this method can guide the gaze relatively reliably, it may reduce the user's sense of immersion, so it is not suitable for applications where you want to guide the gaze naturally. On the other hand, it is difficult to reliably guide the gaze using methods that implicitly guide the gaze.

In one aspect, the present invention aims to provide a gaze guidance method and a gaze guidance program that can reliably guide the gaze.

One proposal provides a gaze guidance method in which a computer displays on a display device an image of an area in a three-dimensional virtual space seen from the user's viewpoint, detects the user's gaze position in the image, displays a predetermined gaze position image at the gaze position in the image, determines whether the movement of the user's gaze satisfies predetermined movement conditions while the gaze position image is displayed, and if the movement conditions are satisfied, shifts the gaze position image a predetermined amount from the gaze position toward the target position based on the relative positional relationship between the direction of the user's gaze and a target position set in the virtual space.

In one proposal, a gaze guidance program is provided that causes a computer to perform processing similar to the gaze guidance method described above.

On the one hand, it can reliably guide the eye.

1A and 1B are diagrams illustrating a configuration example and a processing example of a display control system according to a first embodiment; FIG. 10 is a diagram illustrating an example of the configuration of a work training system according to a second embodiment. FIG. 2 is a diagram illustrating an example of the configuration of processing functions included in an information processing apparatus. FIG. 10 is a diagram illustrating an example of gaze guidance processing. 10 is a flowchart showing the overall processing procedure for gaze guidance. FIG. 10 illustrates a first example of advance preparation processing. FIG. 10 illustrates a second example of the advance preparation process. 10 is a flowchart illustrating an example of a procedure for advance preparation processing. FIG. 10 is a diagram illustrating an example of horizontal gaze guidance. 10A and 10B are diagrams illustrating an example of processing for guiding the user's gaze in accordance with the movement of the user's position and gaze direction. 10 is a flowchart illustrating an example of a procedure for a gaze guidance process.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
First Embodiment
1 is a diagram showing an example of the configuration and processing of a display control system according to a first embodiment. The display control system shown in FIG.

The information processing device 1 is a computer device such as a personal computer or a server device. The information processing device 1 renders an image showing a three-dimensional virtual space and displays the image on the display device 2 for the user to view. In displaying this image, the information processing device 1 acquires the direction and position of the user's head in the virtual space, and renders an image of the area of the virtual space seen from the user's viewpoint based on this direction and position. If the direction or position of the user's head changes, the displayed content of the image changes accordingly. The display device 2 may be, for example, an HMD worn on the user's head, or a general display placed on a desk, etc.

The processing of the information processing device 1 will be described below.
As described above, the information processing device 1 displays an image of an area of virtual space seen from the user's viewpoint. At the same time, the information processing device 1 detects the user's gaze position in the displayed image. The user's gaze position is detected, for example, using a dedicated sensor for detecting the user's gaze. Alternatively, the forward direction of the user's head may be detected as the gaze position. In the former case, the gaze position may fluctuate in the displayed image, whereas in the latter case, the gaze position is located at a fixed position (for example, the center of the image) in the displayed image.

The information processing device 1 displays a predetermined gaze position image at the gaze position in the displayed image. Image 3a shown in Figure 1 is an example of an image depicting an area of virtual space as seen from the user's viewpoint. As an example, a dot-like gaze position image 4 is displayed at the gaze position on this image 3a. As in this example, it is desirable that the gaze position image be a small image that is reliably visible to the user, but does not significantly impede the user's ability to view the image.

The information processing device 1 displays a gaze position image at the detected gaze position and determines whether the movement of the user's gaze satisfies a predetermined movement condition. This movement condition is a condition for making the user aware that the displayed gaze position image is the user's gaze position.

For example, buildings 5a to 5c are displayed in image 3b shown in Figure 1. When the user's gaze captures all of buildings 5a to 5c (when the gaze position moves and covers all of buildings 5a to 5c), it is determined that the movement conditions are met. In this case, for example, each time the user's gaze position is located on one of buildings 5a to 5c, the image of the corresponding building is highlighted, and when the gaze position is located on all of buildings 5a to 5c and the movement conditions are met, the user is notified of this.

When the information processing device 1 determines that the above movement conditions are met, it executes gaze guidance processing to guide the user's gaze toward a target position set in the virtual space. In this gaze guidance processing, the information processing device 1 displays a gaze position image that is shifted a predetermined amount from the detected gaze position toward the target position based on the relative positional relationship between the direction of the user's gaze and the target position in the virtual space.

In image 3c shown in Figure 1, target position 6a is set on the side of bridge pier 6. Furthermore, in image 3c, target position 6a is located to the lower right of the detected gaze position. In this case, gaze position image 4 is displayed, for example, at a position shifted a predetermined amount to the lower right from the detected gaze position. Enlarged image 3d shown in Figure 1 is an enlarged view of a rectangular area centered on the gaze position in image 3c. In this enlarged image 3d, gaze position image 4a when displayed at the gaze position is shown by a dashed line, and it can be seen that gaze position image 4 is displayed shifted to the lower right from gaze position image 4a at the gaze position.

In the gaze guidance process shown in image 3c and enlarged image 3d above, no explicit image is displayed to guide the gaze. This allows the user's gaze to be naturally guided in the direction of the target position 6a. However, unless the user is fully aware that the displayed gaze position image 4 indicates the user's gaze position, it is difficult to reliably guide the gaze.

Therefore, before starting the gaze guidance process, the information processing device 1 displays gaze position image 4 at the detected gaze position, as shown in image 3b, and has the user move their gaze until a specified movement condition is met. This makes the user fully aware that gaze position image 4 indicates the user's gaze position. Then, when the above-mentioned gaze guidance process is executed after this, which shifts gaze position image 4 from the gaze position, the gaze is guided to the position of gaze position image 4 in an attempt to maintain the recognition once formed by the user's brain that "gaze position image 4 is the gaze position." As a result, the user's gaze can be reliably guided.

Second Embodiment
Next, a case where the processing of the information processing device 1 is applied to a work training system will be described.

Figure 2 is a diagram showing an example configuration of a work training system according to the second embodiment. The work training system includes an information processing device 100 and an HMD unit 200. Note that the information processing device 100 is an example of the information processing device 1 in Figure 1, and the HMD unit 200 is an example of the display device 2 in Figure 1.

This work training system allows a user wearing an HMD unit 200 to view images showing a work site, allowing them to receive work training and education. The information processing device 100 stores three-dimensional data showing a three-dimensional virtual space that recreates the work site, draws an image showing the virtual space based on the three-dimensional data, and displays the image on the HMD unit 200.

The HMD unit 200 is worn on the user's head. The HMD unit 200 includes a display 201, a direction/position sensor 202, and a gaze sensor 203. The display 201 is a non-transparent HMD that displays images based on image data transmitted from the information processing device 100. A display 201 may be provided for each of the right and left eyes. The direction/position sensor 202 is a sensor for detecting the direction (direction pointing forward) and position of the user's head, and includes, for example, an acceleration sensor and an angular velocity sensor (gyro). The gaze sensor 203 is a sensor for detecting the direction of the user's gaze. The gaze sensor 203 is, for example, a near-infrared camera. In this case, the near-infrared camera captures images of the light reflection points on the cornea and the eyeball, and the direction and position of the gaze are estimated from the fluctuations in these images.

Next, the hardware configuration of the information processing device 100 will be described using Figure 2. The information processing device 100 is realized, for example, as a computer as shown in Figure 2. The information processing device 100 includes a processor 101, a RAM (Random Access Memory) 102, a HDD (Hard Disk Drive) 103, a GPU (Graphics Processing Unit) 104, an input interface (I/F) 105, a reader 106, and a communication interface (I/F) 107.

The processor 101 (processor circuit) provides overall control of the information processing device 100. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device). The processor 101 may also be a combination of two or more elements of a CPU, MPU, DSP, ASIC, or PLD.

RAM 102 is used as the main storage device of information processing device 100. RAM 102 temporarily stores at least a portion of the OS (Operating System) program and application programs executed by processor 101. RAM 102 also stores various data necessary for processing by processor 101.

The HDD 103 is used as a secondary storage device for the information processing device 100. The HDD 103 stores the OS program, application programs, and various data. Note that other types of non-volatile storage devices, such as an SSD (Solid State Drive), can also be used as the secondary storage device.

A display 111 is connected to the GPU 104. The GPU 104 displays images on the display 111 in accordance with instructions from the processor 101. Examples of the display 111 include a liquid crystal display and an organic EL (ElectroLuminescence) display.

An input device 112 is connected to the input interface 105. The input interface 105 transmits signals output from the input device 112 to the processor 101. Examples of the input device 112 include a keyboard and a pointing device. Examples of pointing devices include a mouse, touch panel, tablet, touchpad, and trackball.

A portable recording medium 113 is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 113 and transmits it to the processor 101. Examples of the portable recording medium 113 include an optical disc and a semiconductor memory.

The communication interface 107 transmits and receives data to and from other devices. In the present embodiment, for example, the communication interface 107 performs wireless communication, enabling wireless communication with the HMD unit 200. As another example, the information processing device 100 and the HMD unit 200 may be connected via a communication cable such as a USB (Universal Serial Bus).

The processing functions of the information processing device 100 can be realized by the above hardware configuration.
3 is a diagram showing an example of the configuration of processing functions of the information processing device 100. The information processing device 100 includes a storage unit 120, a direction/position detection unit 131, a gaze detection unit 132, a virtual space display unit 133, and a gaze guidance processing unit 134.

The memory unit 120 is a storage area allocated in a storage device provided in the information processing device 100, such as the RAM 102 or HDD 103. The memory unit 120 stores three-dimensional data 121. The three-dimensional data 121 is data representing the three-dimensional shape of a virtual space that is a virtualized work site that is the subject of work training or education. The three-dimensional data 121 includes, for example, point cloud data that represents the three-dimensional coordinates of multiple feature points set in the virtual space, and texture image data that is applied to each polygon based on the point cloud data. The virtual space may be a virtualized space of a real work site, or a space that represents a fictitious work site created for training or education purposes.

The processing of the direction/position detection unit 131, gaze detection unit 132, virtual space display unit 133, and gaze guidance processing unit 134 is realized, for example, by the processor 101 executing a predetermined program.

The direction/position detection unit 131 detects the direction and position of the user's head based on the detection results of the direction/position sensor 202. Note that the direction and position of the user's head may be detected in response to user operation, instead of calculation based on actual measurements by the direction/position sensor 202. For example, the user may virtually change the direction and position of their own head in the up, down, left, and right directions in the virtual space using a stick-shaped input device or a key input device.

The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203 .
The virtual space display unit 133 draws an image showing the virtual space of the work site based on the three-dimensional data 121 and the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200.

In this image display, the direction and position of the user's head are associated with the coordinates of the virtual space, and the situation in the virtual space as seen from the user's perspective is reproduced on the image based on these directions and positions. For example, three-dimensional objects in the virtual space are placed on the image in a posture and position that corresponds to the direction and position of the user's head. Furthermore, if the direction or position of the user's head changes, the posture and position of the objects on the image also change accordingly. This allows the user to experience as if they were actually present at the work site, visually observing the environment as they move around the site.

The virtual space display unit 133 may render the image of the virtual space as a panoramic image, a celestial sphere image, or a celestial sphere image centered on the head, or as a part of any of these images (for example, an image of a certain field of view centered in front of the head).

By displaying images such as those described above using the virtual space display unit 133, users can gain a detailed understanding of the situation at a work site without actually going to the actual work site. This allows users to receive training and education on work at the work site from a location other than the work site, such as an office. It is also possible to provide three-dimensional data showing an appropriate work site for training and education, and have users receive training and education on that work site.

The information processing device 100 may detect the movement of the user's hands, for example, to allow the user to virtually experience work at a work site.
In this embodiment, an object that the user should pay attention to is set in the virtual space. Then, the information processing device 100 has the user perform the task of finding the set object as a training or educational task. The gaze guidance processing unit 134 performs gaze guidance processing to guide the user's gaze toward the object based on the relative position of the object with respect to the direction of the user's gaze.

Hereinafter, the above-mentioned object to which the line of sight is guided may be referred to as the "target object." Data relating to the target object set in the virtual space is included, for example, in the three-dimensional data 121.

In order to enable users to reliably and efficiently master the above tasks and skills in a virtual space, it is effective to provide training and education while maintaining a sense of self-control. For example, rather than passive learning such as showing users images showing the work process, it is important to provide learning that gives users the feeling that they are performing the work of their own volition.

For example, a user with low skill in finding the target object described above will not be able to easily find the target object in the virtual space. In such cases, by guiding the user's gaze in the direction of the target object through processing by the gaze guidance processing unit 134, the user can be given the sensation that they have found the target object themselves. This can enhance the effectiveness of the user's acquisition and improvement of skills.

One possible method of guiding the gaze when displaying images in a virtual space is to explicitly display images such as arrows to guide the gaze. However, this method may reduce the sense of immersion in the video content, and as a result, may also reduce the user's sense of control. Therefore, while it is desirable to implicitly guide the gaze, this poses the problem of making it difficult to reliably guide the gaze in the desired direction.

In this embodiment, the following method is first used to implicitly guide the gaze. The gaze guidance processing unit 134 constantly displays a gaze position image representing the user's gaze position on an image depicting the virtual space. The gaze guidance processing unit 134 then displays the gaze position image shifted a predetermined amount from its original position in the direction of the target object. With this method, the user is less likely to realize that their gaze has been guided, making it possible to naturally guide their gaze in the direction of the target object.

FIG. 4 is a diagram illustrating an example of the gaze guidance process.
First, the display of a gaze position image showing the gaze position will be described. As described above, the virtual space display unit 133 renders an image showing the situation of the virtual space as seen from the user's viewpoint based on the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200. The image 210 shown in FIG. 4 is an example of an image of a portion of the virtual space rendered in this manner. The image 210 shows the situation of a certain field of view area of the virtual space, centered in front of the user's head. Therefore, the center of the image 210 indicates the direction forward of the user's head.

The gaze guidance processing unit 134 also detects the gaze position (position of the point of gaze) in the image 210 based on the detection result of the gaze detection unit 132. The gaze position is the position where the detected gaze direction intersects with the screen. The gaze guidance processing unit 134 displays a gaze position image 211 at the gaze position in the image 210. The gaze position image 211 is displayed as a dot-like image (point image) centered on the gaze position, as shown in Figure 4, for example. This dot-like image is sufficiently visible to the user, but has a small radius that does not significantly affect the visibility of the background virtual space, and is drawn in a specified color.

Next, we will explain gaze guidance. When a target object is set and gaze guidance processing begins, the gaze position image 211 is displayed at a position shifted a predetermined amount in the direction of the target object from the original gaze position. The gaze guidance processing unit 134 first performs gaze guidance in the horizontal direction, and then performs gaze guidance in the vertical direction as necessary.

The reason for initially guiding the gaze only horizontally is to reduce the likelihood of symptoms known as "VR sickness." Because the human sense of balance is primarily controlled by the anterior cingulate cortex in the inner ear, humans are more sensitive to vertical movement than horizontal movement in the image they are viewing. For this reason, humans are generally more susceptible to sickness when vertical movement occurs in the image they are viewing than when horizontal movement occurs. The gaze guidance processing unit 134 guides the gaze only horizontally as much as possible, minimizing vertical guidance, thereby reducing the likelihood of the user feeling sick.

In Figure 4, an example of a user's task is assumed to be inspecting cracks on a bridge set up in a virtual space, with the location of the crack on a bridge pier 212 set as the target object 213. An image of the target object 213 is captured in image 210. In this example, the target object 213 is located to the upper right of the user's line of sight. Therefore, the line of sight guidance processing unit 134 first performs processing to guide the user's line of sight to the right.

Figure 4 shows an enlarged image 210a of the image 210, with the detected gaze position at its center. In this enlarged image 210a, a gaze position image 211 indicating the detected original gaze position is indicated by a dotted line. As shown in this enlarged image 210a, the gaze guidance processing unit 134 first displays the gaze position image 211a shifted a predetermined amount to the right from the gaze position. This guides the user's gaze to the right.

Furthermore, the gaze guidance processing unit 134 performs processing to guide the user's gaze vertically, for example, when the horizontal coordinate of the gaze position reaches the vicinity of the position of the target object 213 but the vertical coordinate does not reach the vicinity of the position of the target object 213. The enlarged image 210b shown in FIG. 4 illustrates an example of a state in which the horizontal coordinate of the gaze position reaches the vicinity of the position of the target object 213 but the vertical coordinate does not reach the vicinity of the position of the target object 213. The center of the enlarged image 210b is the detected gaze position, and the enlarged image 210b shows a gaze position image 211b indicating the original gaze position by a dotted line. In this enlarged image 210b, the target object 213 is located to the upper right of the user's gaze position, so the gaze guidance processing unit 134 displays the gaze position image 211 shifted upward by a predetermined amount from the gaze position. This guides the user's gaze upward.

Note that the example in Figure 4 illustrates a case where the target object 213 is included in the user's field of view (i.e., included in the displayed image). However, even if the target object 213 is outside the user's field of view, gaze guidance processing is performed to shift the gaze position image 211 from the gaze position based on the relative positional relationship between the gaze position and the target object 213.

The above-described gaze guidance process can naturally guide the user's gaze in the direction of the target object 213 without displaying an explicit image for guiding the gaze. This prevents a decrease in the sense of immersion in the video content, and as a result, prevents a decrease in the user's sense of self-operation. This makes it possible to efficiently acquire and improve work skills. On the other hand, the above-described gaze guidance process has the problem that it is difficult to reliably guide the gaze unless the user is fully aware that the displayed gaze position image indicates the user's gaze position.

Figure 5 is a flowchart showing the overall processing procedure for gaze guidance. To solve the above problem, the gaze guidance processing unit 134 performs a preparatory process (step S11) to make the user fully aware that the gaze position image indicates the user's gaze position, before performing the gaze guidance process (step S12) in which the gaze position image is displayed shifted from the gaze position.

In the advance preparation process (step S11), the gaze guidance processing unit 134 displays a gaze position image at the detected gaze position and has the user move their gaze until a predetermined movement condition is met. This makes the user fully aware that the gaze position image indicates the user's gaze position. Then, when the gaze guidance process (step S12) described above is executed, which shifts the gaze position image from the gaze position, the gaze is guided to the position of the gaze position image in an attempt to maintain the recognition once formed by the user's brain that "the gaze position image is the gaze position." As a result, it is possible to reliably guide the user's gaze in the direction of the target object, even though an image for guiding the gaze is not explicitly displayed.

An example of the advance preparation process will be described below.
FIG. 6 is a diagram illustrating a first example of the advance preparation process. One example of a movement condition that can be applied is that the user visually recognizes one or more predetermined objects set in the virtual space. The image 220 shown in FIG. 6 shows four buildings 221a to 221d as examples of objects. The gaze guidance processing unit 134 determines that the movement condition is satisfied when the user's gaze captures all of buildings 221a to 221d (when the gaze position moves and is located on all of buildings 221a to 221d). The gaze position being located on the object may be determined, for example, when the gaze position overlaps the area of the object, or when the gaze image 222 overlaps a portion of the area of the object.

In the case of FIG. 6, the gaze guidance processing unit 134 provides visual feedback to the user, for example, each time the user's gaze position is positioned on one of buildings 221a to 221d. For example, the image of the building on which the gaze position is positioned is highlighted. For example, the highlighting method may change the color of the building or display the building's outline in bold. Alternatively, the name of the building on which the gaze position is positioned may be displayed. Furthermore, auditory feedback may be provided to the user each time the user's gaze position is positioned on one of buildings 221a to 221d. For example, a sound may be output indicating the name of the building on which the gaze position is positioned. Such visual or auditory feedback stimulates the user's brain, enabling the user to form a strong awareness that the gaze position image is the user's own gaze position.

The gaze guidance processing unit 134 may determine that the movement conditions are met not when the gaze captures all of the specified objects as described above, but, for example, when the gaze captures a certain number of the multiple set objects. Alternatively, the gaze guidance processing unit 134 may determine that the movement conditions are met when the gaze captures more than a certain number of the multiple set objects and a certain amount of time has passed since the start of the advance preparation process. Furthermore, the objects to be viewed may be selected and set from objects placed in the workspace, or may be objects (such as dedicated markers) that are specifically provided to be viewed during the advance preparation process.

Figure 7 shows a second example of advance preparation processing. Another example of a movement condition is that a predetermined task must be completed by the user's line of sight. Figure 7 shows an example in which a task of drawing a predetermined shape is executed by line of sight movement.

In image 230 shown in Figure 7, the user's gaze image 231 moves from the upper left to the upper right, lower right, and lower left before returning to its original position, thereby drawing a rectangle. The gaze guidance processing unit 134, for example, displays movement trajectories 232a-232d in accordance with this movement of the gaze image 231, allowing the user to visually recognize the figure drawn by the gaze movement. The gaze guidance processing unit 134 determines that the movement condition has been met when it determines that a rectangle has been drawn.

The gaze guidance processing unit 134 may also draw a figure by moving the gaze to a number of pre-set objects in a specified order, as shown in Figure 6. Furthermore, the gaze guidance processing unit 134 can measure the degree to which the user recognizes that the gaze position image is the user's own gaze position by evaluating task performance. Therefore, the gaze guidance processing unit 134 may evaluate the time and accuracy required to draw the figure as task performance, and determine that the movement condition has been met when the degree of recognition estimated from the evaluation results is equal to or greater than a predetermined value.

As described above, by having the user perform a specific task by having the gaze position image move in accordance with the movement of the gaze, the user's brain is stimulated and it becomes possible for the user to form a strong awareness that the gaze position image is the user's own gaze position.

As other examples of the movement condition, it is possible to apply a condition that the line of sight moves for a certain period of time or more, or that the line of sight moves horizontally by a certain angle or more, for example.
8 is a flowchart showing an example of the procedure of the advance preparation process. The process in FIG. 8 corresponds to step S11 in FIG.

[Step S21] The virtual space display unit 133 draws an image showing the virtual space as seen from the user's viewpoint based on the three-dimensional data 121 and the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200.

[Step S22] The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203. The gaze guidance processing unit 134 detects the gaze position in the displayed image based on the direction and position of the user's head and the detected gaze direction. The gaze guidance processing unit 134 draws a gaze position image at the detected gaze position.

[Step S23] The line-of-sight guidance processing unit 134 determines whether the line-of-sight movement conditions are met. If the movement conditions are not met, processing proceeds to step S21. Note that steps S21 to S23 are executed, for example, at the image frame rate. On the other hand, if the movement conditions are met, the advance preparation processing ends.

Next, we will explain the gaze guidance process. In the following explanation, the X and Z axes are taken as the horizontal direction in the virtual space, and the Y axis is taken as the vertical direction. The Z axis is taken as pointing forward from the user's head, the X axis is taken to the user's right, and the Y axis is taken as pointing upward from the user.

Figure 9 is a diagram showing an example of gaze guidance in the horizontal direction. In Figure 9, as an example, the virtual space is depicted as a spherical image. In this case, the gaze position and the target position serving as the guidance target are expressed as coordinates on a spherical surface centered on the user's head. In the following explanation, the coordinates of gaze position Pe are defined as (xe, ye, ze), and the coordinates of target position Pt are defined as (xt, yt, zt).

In the following explanation, a specific point within the image area in which the target object is drawn is set as the target position. For example, the target position is set to the center point of the rectangular area inscribed by the target object.

Figure 9 shows the situation when looking downward from above the user's head. In the state at the top of Figure 9, as an example, the detected gaze position Pe is on the Z axis, and the target position Pt is located to the right and rear of the user's head. In this case, since the target position Pt is located relatively to the right of the gaze position Pe, the gaze position image is displayed shifted a predetermined amount to the right of the gaze position Pe.

In the lower part of Figure 9, the center point of the gaze position image during gaze guidance processing is shown as Pe'. The center point Pe' of the gaze position image is set, for example, at a position shifted by an angle θ1 from the detected gaze position Pe, with the position of the user's head as the center. In the example in the lower part of Figure 9, the center point Pe' is set at a position shifted by an angle θ1 to the right from the gaze position Pe, and the gaze position image is displayed at this center point Pe'. On the other hand, although not shown, if the target position Pt is located relatively to the left of the gaze position Pe, the center point Pe' of the gaze position image is set at a position shifted by an angle θ1 to the left from the gaze position Pe.

Furthermore, for vertical gaze guidance, if the target position Pt is located relatively higher than the gaze position Pe, the center point Pe' of the gaze position image is set to a position shifted upward from the gaze position Pe by an angle θ2. On the other hand, if the target position Pt is located relatively lower than the gaze position Pe, the center point Pe' of the gaze position image is set to a position shifted downward from the gaze position Pe by an angle θ2. Here, with regard to angles θ1 and θ2, θ1 = θ2 may be satisfied. However, as mentioned above, humans are more sensitive to vertical image movement than horizontal image movement, so by making θ1 > θ2, the user is less likely to feel sick.

In the above explanation, the shift amount of the gaze position image from the detected gaze position Pe is defined by angles θ1 and θ2, but as another example, the shift amount may be defined by the number of pixels in the displayed image.

Next, a guidance end condition for ending the gaze guidance process and a vertical guidance start condition for starting the gaze guidance in the vertical direction will be described.
The gaze guidance processing unit 134 terminates the gaze guidance processing, for example, when the horizontal and vertical coordinates of the gaze position Pe remain near the target object for a certain period of time. Specifically, for example, when both conditions of the following formulas (1-1) and (1-2) are satisfied continuously for a period of time T1, the gaze guidance processing unit 134 determines that the guidance termination condition has been met and terminates the gaze guidance processing.
xt-Δx<xe<xt+Δx (1-1)
yt-Δy<ye<yt+Δy...(1-2)
Furthermore, the gaze guidance processing unit 134 starts vertical gaze guidance, for example, when the horizontal coordinate of the gaze position Pe stays near the target object for a certain period of time but the vertical coordinate of the gaze position Pe is not located near the target object. Specifically, for example, when the condition of the following formula (2) is satisfied continuously for a period of time T2 but the condition of formula (1-2) is not satisfied, the gaze guidance processing unit 134 determines that the vertical guidance start condition has been met and starts vertical gaze guidance.
xt-Δxh<xe<xt+Δxh (2)
Between the above equations (1-1), (1-2) and (2), Δx, Δy, Δxh and Δyh may be set so that, for example, Δx≧Δxh and Δy≧Δyh.

10 is a diagram illustrating an example of a process for guiding the user's gaze in response to a change in the user's position and gaze direction. In FIG. 10, an example of guiding the user's gaze in the horizontal direction is illustrated.
10, a bridge 241 and a pier 242 are placed in a virtual space, and a target position Pt is set on the side of the pier 242. In this virtual space, it is assumed that the user's head position moves from Ph1 to Ph4. The arrows shown at the head positions Ph1 to Ph4 indicate the user's line of sight.

At head position Ph1, the target position Pt is located relatively to the right of the gaze position Pe, so the center point Pe' of the gaze position image is set to a position shifted to the right from the gaze position Pe by an angle θ1. As a result, the user's viewpoint is guided to the right, and the user's head moves to head position Pt2.

At head position Ph2, the target position Pt is located relatively to the right of the gaze position Pe, so the center point Pe' of the gaze position image is set to a position shifted to the right from the gaze position Pe by an angle θ1. As a result, the user's viewpoint is guided to the right, and the user's head moves to head position Pt3.

At head position Ph3, the target position Pt is located relatively to the left of the gaze position Pe, so the center point Pe' of the gaze position image is set to a position shifted to the left of the gaze position Pe by an angle θ1. As a result, the user's viewpoint is guided to the left, and the user's head moves to head position Pt4.

At head position Ph4, the relationship between gaze position Pe and target position Pt satisfies the guidance termination condition using the above-mentioned equations (1-1) and (1-2). In this case, the gaze guidance process ends, and the center point Pe' of the gaze position image is positioned at gaze position Pe. Note that when the guidance termination condition is satisfied, not only does the gaze guidance process end, but the display of the gaze position image may also end.

Note that while the above equations (1-1), (1-2), and (2) show the conditions when the target position is set to a single point, the target position may also be a three-dimensional area with lengths in both the horizontal and vertical directions. In this case, for example, the minimum and maximum X coordinate values xt1 and xt2, and the minimum and maximum Y coordinate values yt1 and yt2 in this three-dimensional area may be used to set the guidance end condition and vertical guidance start condition as follows:

The guidance termination condition is when both of the following formulas (1-1a) and (1-2a) are satisfied continuously for a time T1.
xt1-Δx<xe<xt2+Δx...(1-1a)
yt1-Δy<ye<yt2+Δy...(1-2a)
The vertical guidance start condition is when the condition of the following formula (2a) is continuously satisfied for a time T2, but the condition of formula (1-2a) is not satisfied.
xt1-Δxh<xe<xt2+Δxh (2a)
11 is a flowchart showing an example of the procedure for the gaze guidance process. The process in FIG. 11 corresponds to the process in step S12 in FIG.

[Step S31] The virtual space display unit 133 draws an image showing the virtual space as seen from the user's viewpoint based on the three-dimensional data 121 and the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200.

[Step S32] The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203. The gaze guidance processing unit 134 detects the gaze position in the displayed image based on the direction and position of the user's head and the detected gaze direction. The gaze guidance processing unit 134 also detects the target position for gaze guidance.

The gaze guidance processing unit 134 detects the horizontal and vertical relative positions of the target position with respect to the gaze position. In this detection, if the gaze position and target position do not match, it detects whether the target position is located to the right or left of the gaze position and the difference in the horizontal coordinates of each position. It also detects whether the target position is located above or below the gaze position and the difference in the vertical coordinates of each position.

[Step S33] The gaze guidance processing unit 134 calculates the drawing position of the gaze position image based on the detection result of the relative position in the horizontal direction in step S32, and draws the gaze position image at that drawing position. If the target position is located to the right of the gaze position, the gaze position image is drawn at a position to the right of the gaze position by an angle θ1 centered on the user's head. On the other hand, if the target position is located to the left of the gaze position, the gaze position image is drawn at a position to the left of the gaze position by an angle θ1 centered on the user's head. This guides the user's gaze horizontally.

[Step S34] The gaze guidance processing unit 134 determines whether the gaze movement status satisfies the guidance end condition based on the detection results of each relative position in the horizontal and vertical directions in step S33 at the most recent time T1. If the guidance end condition is not satisfied, processing proceeds to step S35. If the guidance end condition is satisfied, the gaze guidance processing ends. Note that even in the latter case, the virtual space display unit 133 may continue drawing the image in the virtual space.

[Step S35] The gaze guidance processing unit 134 determines whether the vertical guidance start condition is met based on the result of detecting the relative position in the horizontal direction in step S33 at the most recent time T2 and the result of detecting the relative position in the vertical direction in the most recent step S33. If the vertical guidance start condition is not met, processing proceeds to step S31. In this case, the loop processing of steps S31 to S35 is executed, for example, at a frame cycle. On the other hand, if the vertical guidance start condition is met, processing proceeds to step S36.

[Step S36] The virtual space display unit 133 draws an image showing the virtual space as seen from the user's viewpoint based on the three-dimensional data 121 and the direction and position of the user's head, and displays the image on the display 201 of the HMD unit 200.

[Step S37] The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203. The gaze guidance processing unit 134 detects the gaze position in the displayed image based on the direction and position of the user's head and the detected gaze direction. The gaze guidance processing unit 134 also detects the target position for gaze guidance. The gaze guidance processing unit 134 detects the horizontal and vertical positions of the target position relative to the gaze position using a procedure similar to step S32.

[Step S38] The gaze guidance processing unit 134 calculates the drawing position of the gaze position image based on the detection result of the relative position with respect to the vertical direction in step S37, and draws the gaze position image at that drawing position. If the target position is located above the gaze position, the gaze position image is drawn at a position above the gaze position by an angle θ2 centered on the user's head. On the other hand, if the target position is located below the gaze position, the gaze position image is drawn at a position below the gaze position by an angle θ2 centered on the user's head. This guides the user's gaze vertically.

Then, processing proceeds to step S34, where a determination is made as to whether the gaze movement status satisfies the guidance termination condition based on the results of the detection of the relative positions in the horizontal and vertical directions in step S37 at the most recent time T2. Note that the loop processing of steps S34 to S38 is executed, for example, at a frame cycle.

In addition, in step S38, processing may be performed to shift the gaze position image from the gaze position not only vertically but also horizontally. At least when Δxh > Δx is set, it is desirable that gaze guidance in the horizontal direction be performed in this way in step S38.

According to the second embodiment described above, natural gaze guidance can be reliably achieved while maintaining a sense of immersion and self-control. This allows users to efficiently acquire and improve the skills of the task being trained or learned.

In the second embodiment described above, the direction of the user's gaze was calculated based on the detection results of the gaze sensor 203. However, the direction of the user's gaze may also be the direction forward of the user's head. In this case, the gaze position in the displayed image will be the center of the image.

Furthermore, at least a portion of the above processing performed by the information processing device 100 may be executed by a processor (not shown) provided in the HMD unit 200. For example, if the HMD unit 200 has all of the processing functions shown in FIG. 3, the above processing is realized by the HMD unit 200 alone.

Furthermore, while the second embodiment above describes gaze guidance in VR, the processing of the second embodiment can also be applied to gaze guidance in mixed reality and augmented reality.

The processing functions of the devices (e.g., information processing devices 1, 100) described in each of the above embodiments can be implemented by a computer. In this case, a program describing the processing details of the functions that each device should have is provided, and the processing functions are implemented on the computer by executing the program. The program describing the processing details can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and sold. The program can also be stored in the storage device of a server computer, and then transferred from the server computer to other computers via a network.

A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing in accordance with the program. Note that the computer can also read the program directly from a portable recording medium and execute processing in accordance with that program. The computer can also execute processing in accordance with the received program each time a program is transferred from a server computer connected via a network.

1 Information processing device 2 Display device 3a to 3c Images 3d Enlarged image 4, 4a Gaze position images 5a to 5c Building 6 Bridge pier 6a Target position

Claims (5)

The computer
displaying on a display device an image of an area of the three-dimensional virtual space seen from the user's viewpoint;
Detecting a gaze position of the user in the image and displaying a predetermined gaze position image at the gaze position in the image;
determining whether a movement of the user's gaze satisfies a predetermined movement condition while the gaze position image is displayed;
When the movement condition is satisfied, the gaze position image is displayed by shifting it by a predetermined amount from the gaze position toward the target position based on a relative positional relationship between the gaze direction of the user and the target position set in the virtual space.
Eye guidance method. the movement condition indicates that the user's line of sight has moved to each of one or more predetermined objects arranged in the virtual space;
The method for guiding the gaze according to claim 1. the movement condition indicates that a predetermined task has been performed by a movement of the gaze position image accompanying a gaze movement of the user.
The method for guiding the gaze according to claim 1. When the movement condition is satisfied, the gaze position image is displayed shifted horizontally from the gaze position;
when a predetermined horizontal position condition indicating that the horizontal coordinate of the gaze position has reached the vicinity of the target position in the horizontal direction is satisfied but a predetermined vertical position condition indicating that the vertical coordinate of the gaze position has reached the vicinity of the target position in the vertical direction is not satisfied, the gaze position image is displayed shifted vertically from the gaze position;
The method for guiding the gaze according to claim 1. On the computer,
displaying on a display device an image of an area of the three-dimensional virtual space seen from the user's viewpoint;
Detecting a gaze position of the user in the image and displaying a predetermined gaze position image at the gaze position in the image;
determining whether a movement of the user's gaze satisfies a predetermined movement condition while the gaze position image is displayed;
When the movement condition is satisfied, the gaze position image is displayed by shifting it by a predetermined amount from the gaze position toward the target position based on a relative positional relationship between the gaze direction of the user and the target position set in the virtual space.
A gaze guidance program that executes the process.