WO2026009727A1 - Information processing device, information processing system, information processing program, and information processing method - Google Patents

Abstract

This information processing device has an acquisition means for acquiring information about placement of a real object in a real space, and an image acquisition means for acquiring a captured image by capturing an image of the real space in a predetermined direction, wherein, when a position where a virtual object is placed in a mixed reality space is a first position where at least a part of the virtual object is positioned behind the real object or is in a covered position inside the real object, the information processing device performs control to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is a position in front of the real object when viewed from the predetermined direction.

Description

Information processing device, information processing system, information processing program, and information processing method

This disclosure relates to an information processing device that displays virtual objects to a user in a virtual space or mixed reality space.

In recent years, mixed reality (MR) technology has become known as a technology that seamlessly blends the real world and the virtual world in real time.

One example of MR technology is an MR system that uses a video see-through HMD (Head Mounted Display). In an MR system, a camera built into the HMD captures an image of a subject that roughly matches the subject observed from the user's pupil position. The captured image is then superimposed with CG (Computer Graphics, virtual objects), creating a composite image that is presented to the user, allowing the user to experience an MR space (mixed reality space). One technique required to make virtual objects appear as if they actually exist is called occlusion. Occlusion is an image processing technique that calculates the anteroposterior relationship between virtual and real objects, and renders virtual objects behind real objects as hidden or missing. For example, Patent Document 1 discloses an image processing technique that represents the anteroposterior relationship between virtual and real objects.

Japanese Patent Application Laid-Open No. 2020-30748

However, when occlusion is applied, depending on the placement of virtual objects, they may be hidden by real objects, reducing their visibility and impairing the user experience in MR space.

This disclosure has been made in light of these points, and aims to improve the user experience in MR spaces for users of information processing devices.

One aspect of the present disclosure is an information processing device comprising: an acquisition means for acquiring information regarding the placement of real objects in real space; an image acquisition means for acquiring a captured image of the real space in a predetermined direction; and a control means for controlling the generation of an image of a mixed reality space that is an image obtained by combining the captured image with a virtual object; wherein the control means controls the generation of an image of the mixed reality space in which, when the position at which the virtual object is placed in the mixed reality space is a first position where at least a portion of the virtual object is behind the real object or embedded inside the real object, the image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object as viewed from the predetermined direction.

This disclosure makes it possible to improve the user experience of information processing devices in MR spaces.

FIG. 2 is a diagram illustrating the configuration of an information processing device according to the first embodiment. FIG. 2 is a diagram illustrating a real space in which a user exists in the first embodiment. 1 is a diagram illustrating a virtual space viewed by a user via an HMD in a first embodiment. FIG. This is a diagram showing a bird's-eye view of the mixed reality space in which the user 202 exists, assuming a scene in the first embodiment when switching from the virtual space 301 in Figure 3 to an image of the mixed reality space, leaving the virtual window 305. FIG. 4B is a diagram showing a scene in which the part of the image that the user views through the HMD in the scene of FIG. 4A is also displayed that is hidden in the wall in the first embodiment. 4B is a diagram showing a scene in which the portion of the image that the user views through the HMD in the scene of FIG. 4A is not displayed, the portion being hidden behind a wall, in the first embodiment. FIG. 4B and 4C in the first embodiment, the virtual window is translated forward and reduced in size in mixed reality space so that the appearance of the virtual window from the user does not change. 5B is a diagram illustrating an image of a mixed reality space generated by a control unit A101 in the scene of FIG. 5A according to the first embodiment. FIG. 4 is a flowchart illustrating a detailed operation of the information processing device according to the first embodiment. 10 is a flowchart illustrating a detailed operation of the information processing device according to the second modification of the first embodiment. FIG. 10 is a diagram illustrating an example of a confirmation screen that is displayed after a portion of a virtual object that is buried in a wall is displayed, and that asks the user whether or not to rearrange the virtual object, in a second modification of the first embodiment. FIG. 10 is a diagram illustrating an example of a confirmation screen that is displayed before a portion of a virtual object that is embedded in a wall is displayed, and that asks the user whether or not to rearrange the virtual object, in a second modification of the first embodiment. FIG. 10 is a diagram illustrating a scene in which a virtual window is displayed indoors as a virtual object on a large screen in the second embodiment. 9A and 9B are diagrams illustrating a case in which the virtual object rearrangement method of the first embodiment is applied to a virtual window 903 in the situation of FIG. 9 according to the second embodiment. 9A and 9B are diagrams illustrating a case where the virtual object rearrangement method according to the second embodiment is applied to a virtual window 903 in the situation of FIG. 9 according to the second embodiment. FIG. 10 is a diagram illustrating a scene in which a virtual window is displayed indoors as a virtual object on a large screen in the second embodiment. 12A and 12B are diagrams illustrating a case in which the virtual object rearrangement method of the first embodiment is applied to a virtual window 1203 in the situation of FIG. 12 according to the second embodiment. 12A and 12B are diagrams illustrating a case where the virtual object rearrangement method according to the second embodiment is applied to a virtual window 1203 in the situation of FIG. 12 according to the second embodiment. FIG. 11 is a diagram illustrating a scene in which three virtual windows are displayed indoors in MR mode in the third embodiment. FIG. 10 is a diagram illustrating a scene in which three virtual windows have been rearranged according to the rearrangement method of the first embodiment in the third embodiment. FIG. 15 is a diagram illustrating a scene in which the entire screen is rearranged using a virtual window 1505 as a reference for rearrangement in the third embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the disclosure according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the disclosure, and multiple features may be combined arbitrarily. Furthermore, in the attached drawings, the same reference numbers are used for identical or similar components, and duplicate explanations will be omitted.

(First embodiment)
In this embodiment, a use case will be described in which a user wears a head-mounted display (HMD), which is an information processing device worn on the head, and experiences a mixed reality space. The user watches videos of movies, sports, etc. on a virtual display (virtual window) arranged in the mixed reality space. The description will also assume a situation in which the virtual window (virtual object) is repositioned when at least a portion of the virtual window (virtual object) is embedded in a wall.

<Internal configuration of information processing device>
The internal configuration of an information processing device according to a first embodiment of the present disclosure will be described below with reference to Fig. 1. In this embodiment, the information processing device A100 will be described assuming that it is an HMD.

The information processing device A100 is composed of a control unit A101, a storage unit A102, a memory A103, an input unit A104, an output unit A105, a sensor A106, and a communication unit A107.

The control unit A101 is a CPU that controls each component of the information processing device A100. The control unit A101 combines an image captured by an imaging sensor (camera) with a virtual object to generate a composite image (an image of mixed reality space). The control unit A101 also generates an image of a virtual space, which is an image that does not include real space and is composed only of CG, based on the image captured by the imaging sensor. For example, the control unit A101 identifies a floor or real objects from the captured image and determines the placement of the floor and virtual objects in the virtual space according to the position of the real objects. The control unit A101 may also obtain information on the distances from the two cameras to real objects as distance information and determine the placement of virtual objects based on the distance information. The control unit A101 controls the generated image of mixed reality space and image of virtual space to be displayed on a display such as a liquid crystal panel or an organic EL panel. Furthermore, if an HMD is assumed as the information processing device A100, the control unit A101 controls the generation of an image for the right eye and an image for the left eye and displays them on the respective displays. The control unit A101 estimates the position or orientation of the camera (the position or orientation of the information processing device A100) from the captured image. In this embodiment, the position and orientation is estimated using a method for estimating the camera's self-position and orientation, such as SLAM (Simultaneous Localization and Mapping). The position and orientation estimation unit 104 may estimate its self-position using LiDAR SLAM (LiDAR; Light Detection and Ranging) that uses a laser, in addition to Visual SLAM, which estimates its self-position from camera images. The self-position may also be estimated using Depth SLAM, which uses a ToF sensor (ToF; Time of Flight). Instead of the control unit A101 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

The storage unit A102 is an electrically erasable and recordable non-volatile storage medium such as an SSD (Solid State Drive) or flash memory, and stores programs executed by the control unit A101, as well as databases and user settings.

Memory A103 is, for example, RAM (Random Access Memory) and is used as a buffer memory for temporarily storing various data, a working area for control unit A101, etc.

The input unit A104 is used to input instructions to the information processing device A100. The input unit A104 includes, for example, a power button for instructing the information processing device A100 to power on/off, and operation buttons for instructing screen transitions. Note that the input unit A104 does not necessarily have to be built into the information processing device A100. It may also be configured so that input is made via the communication unit A107, which will be described later.

The output unit A105 may be, for example, a display (display unit) viewed by the user, a GUI (Graphical User Interface) display for interactive operation, or a light-emitting device such as an LED (Light Emitting Diode). The output unit A105 may also include a sound output device such as a speaker, and a vibration device that provides a haptic effect to the user. The output unit A105 may also be configured to output via the communication unit A107, which will be described later.

The sensor A106 is, for example, an imaging sensor that captures images of the surroundings, or a sensor such as LiDAR (Light Detection and Ranging) or ToF (Time of Flight) that measures the surrounding conditions. It is also, for example, a sensor such as an IMU (Inertial Measurement Unit) or geomagnetic sensor that measures the attitude and position of the information processing device A100. Note that the sensor A106 does not necessarily have to be built into the information processing device A100.

The communication unit A107 is, for example, a NIC (Network Interface Card) that incorporates a serial bus or parallel bus for connecting to other devices, a connector (RJ45) for connecting to Ethernet, and a communication IC. It is also a communication unit for wireless connection with a controller or external device. The communication unit A107 outputs a weighted wireless signal from an antenna and demodulates the wireless signal received by the antenna to achieve short-range wireless communication in accordance with the IEEE 802.15 standard (also known as Bluetooth (registered trademark)). The communication unit A107 may be wired, such as a USB cable (registered trademark), or wireless, such as Wi-Fi (Wireless Fidelity) (registered trademark).

The terminal configuration has been described assuming that the information processing device A100 is an HMD, but the terminal configuration described above is merely an example, and the information processing device A100 does not have to be an HMD, and the terminal configuration may also be different. Note that the information processing device A100 may also be an information processing device such as a PC (personal computer), tablet terminal, or smartphone connected to an HMD. It may also be an information processing system equipped with each of the components of the information processing device A100 described above.

<Explanation of situations where virtual objects need to be rearranged>
A method for determining whether a virtual object should be rearranged in this embodiment will be described below.

To determine whether a virtual object should be relocated, information about the virtual object's position and size, as well as spatial information such as the position and size of surrounding walls, ceilings, furniture, etc., is used.

Information about the position and size of the virtual object is obtained by the system or application that controls the display of the virtual object. The control unit A101 obtains information about the position and size of the virtual object from the system or application.

In addition, spatial information such as the position and size of surrounding walls, ceilings, furniture, etc. in real space must be acquired by capturing images of the surroundings and calculating them through spatial recognition. One method of spatial recognition is to measure the distance to an object by emitting laser light and measuring the reflected light, then perform calculations through mapping. Another method involves extracting feature points from images captured by a camera and combining them with the camera's movement to determine depth, and then making inferences based on the determined depth. Other known methods include using machine learning such as Deep Learning to estimate depth from images captured by a camera and then making calculations based on this. The control unit A101 uses these methods to acquire spatial information about real space.

The control unit A101 uses the position and size information of the virtual object and the spatial information of the real space to calculate the front-to-back relationship between the virtual object and the real object in the mixed reality space as seen from the HMD. In other words, it determines which object is in front in the mixed reality space as seen from the camera's imaging direction.

If the virtual object is in front, it will not be obscured by surrounding walls, ceilings, furniture, etc., so it can be determined that there is no need to reposition the virtual object.

If the virtual object is behind the virtual object, or if the virtual object is obscured by the surrounding walls, ceiling, furniture, etc., it is determined that at least part of the virtual object will become invisible and that it needs to be relocated. In other words, it is determined that relocation is necessary if the virtual object interferes with the surrounding real objects such as walls, ceiling, furniture, etc. Or, it is determined that relocation is necessary if at least part of the virtual object is embedded in the surrounding real objects such as walls, ceiling, furniture, etc.

The method for determining whether a virtual object should be rearranged described above is merely an example, and different methods for determining whether a virtual object should be rearranged may be used.

The real space in which a user exists in the first embodiment of the present disclosure will now be described with reference to Figure 2. Figure 2 illustrates a bird's-eye view of the real space in which user 202 exists, with user 202 existing in room 201 separated by walls. The room 201 in which user 202 exists is assumed to be furnished with multiple interior items (real objects) including a television 203, a sofa 204, a table 205, a shelf 206, and a sofa 207. The user 202 is assumed to be sitting on sofa 207 and facing the television 203.

Below, with reference to Figure 3, the virtual space viewed by the user via the HMD in the first embodiment of the present disclosure will be described. Figure 3 illustrates a scene in which an avatar 302, representing the user 202, exists in virtual space 301, and shows a bird's-eye view of the virtual space in which avatar 302 exists. In virtual space 301, it is possible to represent the space as if it were expanding independently of real objects. In Figure 3, it is assumed that virtual space 301 is a space larger than area 303, which corresponds to the area surrounded by the walls of room 201 in the real space in Figure 2. Furthermore, a virtual object 304 and a virtual window 305 are arranged in virtual space 301. Virtual object 304 is a virtual object superimposed on sofa 207 in room 201 in Figure 2. Furthermore, virtual window 305 is a virtual object whose width is larger than room 201 (area 303) as seen from the user 202. In the virtual space 301, the user 202 can watch videos of movies, sports, etc. in a virtual window 305 that is larger than the display that can be placed in the room 201.

Imagine a situation in which, while viewing such a virtual space, user 202 switches from the image of the virtual space to an image of mixed reality space, in which the virtual window 305 appears as if it were placed in real space.

Below, with reference to Figures 4A, 4B, and 4C, we will explain the screen that appears when switching from an image in virtual space to an image in mixed reality space in the first embodiment of the present disclosure.

Figure 4A is a bird's-eye view of the mixed reality space in which user 202 exists, imagining a scene when switching from virtual space 301 in Figure 3 to an image of mixed reality space, leaving virtual window 305. In mixed reality space 401, virtual window 305 is placed in the real space at the same position and size as it was in virtual space 301, and mixed reality space 401 is imagined to be expressed. As virtual window 305 is wider than room 201 when viewed from the user, it extends beyond the walls of room 201. In other words, both ends of virtual window 305 are positioned so that they are embedded in the walls of room 201.

Figure 4B shows the image viewed by the user through the HMD in the scene of Figure 4A. To accurately represent the positional relationship between the virtual object and the real object, both ends of virtual window 305 are positioned so that they are embedded in the walls of room 201. Therefore, of virtual window 305, virtual window 402 is positioned within the mixed reality space. Furthermore, virtual windows 403 and 404, which correspond to both ends of virtual window 305, are positioned so that they are embedded in the walls in the mixed reality space. In Figure 4B, virtual windows 403 and 404, which are embedded in the walls, are also displayed. A face is reflected in virtual window 402, but part of the face is also displayed in virtual window 403, which is embedded in the wall. Note that virtual object 412, which represents the shadow of virtual window 402, is also positioned in the image of mixed reality space 401 so that the user can see the depth of virtual window 402 in mixed reality space 401. Similarly, virtual object 413, which is the shadow of virtual window 403, and virtual object 414, which is the shadow of virtual window 404, are also placed in the image of mixed reality space 401. That is, control unit A101 combines virtual window 402, virtual window 403, virtual window 404, and virtual object 412, virtual object 413, and virtual object 414, which correspond to the shadows, with the captured image. In this way, an image of mixed reality space 401 is generated. In this way, if an image of mixed reality space 401 is generated so that virtual window 403 and virtual window 404, which are embedded in the wall, are also displayed, it may be difficult for the user to get the feeling that virtual window 305 is actually present in real space.

4C is a diagram showing the image viewed by the user through the HMD in the scene of FIG. 4A. In FIG. 4B, virtual windows 403 and 404, which correspond to both ends of virtual window 305 embedded in the wall of the real space, are displayed in the image of mixed reality space 401. In FIG. 4C, the virtual object embedded in the wall is not displayed. Here, the only virtual window displayed is virtual object 402, so virtual object 412, which corresponds to the shadow of virtual object 402, is also displayed. A face is reflected in virtual window 402, but part of the face is embedded in the wall, so that part of the face is not visible. In other words, control unit A101 combines virtual window 402, virtual object 412, which corresponds to the shadow, with the captured image to generate an image of mixed reality space 401. If the image of mixed reality space 401 is generated in this manner, when the user switches from the virtual space to the mixed reality space, they will no longer be able to view the entire virtual window 305 that they were viewing when the image of the virtual space was being displayed, which may impair the user experience. For example, if video content is displayed in the virtual window 305, parts of the content that are hidden in the wall may be missed.

<Method for rearranging virtual objects>
4B or 4C , the virtual object is displayed so that it is in front of the real object. That is, occlusion is adjusted so that the virtual object is not hidden by the real object. Here, according to the physical laws of real space, if the size of a virtual object in mixed reality space is fixed, the size of the virtual object will be represented as increasing from the user's perspective as it gets closer to the user. That is, although the size of the virtual object in mixed reality space does not change, when an image of the mixed reality space is generated, by increasing the size of the virtual object with the same angle of view in the mixed reality space, the virtual object will appear to be getting closer to the user.

However, if the size of the virtual object increases when adjusting occlusion, the area in which real space is displayed in the image of the mixed reality space viewed by the user will decrease.

Therefore, when a virtual object is displayed in front of a real object, the size of the virtual object in mixed reality space is reduced. In other words, when an image in mixed reality space is generated, the virtual object is displayed in front of the real object without changing its size in the composite image. When a virtual object is displayed in mixed reality space in this way, the size of the virtual object will be the same before and after adjusting the positional relationship between the real object and virtual object when viewed from the same position. Here, before adjusting the positional relationship between the real object and virtual object, the virtual object is farther from the user, and after adjusting the positional relationship between the real object and virtual object, the virtual object is closer to the user. However, when viewing the virtual object from a position the same distance from each virtual object before and after adjusting the positional relationship between the virtual objects, the size of the virtual object will be smaller after adjusting the positional relationship of the virtual objects.

In other words, when viewed from the perspective of a user in the same position, the apparent size of the virtual object remains the same, but the size of the virtual object in the mixed reality space becomes smaller after the positional relationship of the virtual object has been adjusted. Similarly, when it comes to the size of the virtual object synthesized by control unit A101, the size of the virtual object in the synthesized image remains the same, but the size of the virtual object in the mixed reality space becomes smaller after the positional relationship of the virtual object has been adjusted.

<Example of rearrangement>
Hereinafter, a screen on which virtual objects are rearranged in an image of mixed reality space according to the first embodiment of the present disclosure will be described with reference to FIGS. 5A and 5B.

Figure 5A illustrates a scenario in which, when a virtual object is displayed as in Figures 4B and 4C, the virtual window is translated forward and shrunk in mixed reality space so that the way the virtual window appears to the user does not change. Figure 5A also illustrates a bird's-eye view of the mixed reality space in which user 202 exists. In mixed reality space 501, the virtual object is moved from the position of virtual window 502, which was placed so that it was embedded in the wall, to the position of virtual window 503 in the foreground, and is shrunk so that its apparent size to the user does not change. If the virtual object is moved closer to the user at its original size, its apparent size to the user will increase.

By shrinking the virtual object and bringing it closer to the user, the apparent size to the user can be maintained. In this example, we are imagining a bird's-eye view of the scene, so the size of the virtual object changes before and after it is repositioned.

5B is an example of a screen viewed by user 202 in the scene described in FIG. 5A. That is, FIG. 5B is an image of the mixed reality space generated by control unit A101. FIG. 5B assumes a scene in which a virtual object is moved closer to the user at the same position and orientation as the user in FIG. 4B or 4C. In FIG. 5B, in mixed reality space 501, virtual window 503 is positioned closer than the virtual window in FIG. 4B or 4C and does not interfere with the wall of the room. That is, at least a portion of the virtual window is not embedded in the wall. Furthermore, before the rearrangement, the shadow of virtual window 503 was located at the position of virtual window shadow 512, but in this mixed reality space image, virtual window shadow 512 is not composited, and virtual window shadow 513 is drawn according to the position after the rearrangement. In FIG. 5B, a virtual window of the same size as the virtual window in FIG. 4B or 4C is composited on the image of the mixed reality space generated by control unit A101. The positional relationship in mixed reality space has changed. If the distance between the virtual window and the user in Figure 4B is the same as the distance between the virtual window and the user in Figure 5B, the size of the virtual window seen by the user will be larger in Figure 4B.

By shrinking a virtual object in mixed reality space and moving it towards the front, it is possible to make a virtual object that was previously hidden by a wall no longer hidden by the wall. In addition, by adjusting the shrinking ratio and movement distance, it is possible to prevent the appearance of the virtual object from changing before and after rearrangement due to perspective. Because the appearance of the virtual object does not change even after rearrangement, this method reduces the degradation of the user experience caused by rearrangement of virtual objects compared to other rearrangement methods that change the appearance of the virtual object.

However, if the user's angle of view does not change, i.e., if the user is in the same position and facing the same direction, the control unit A101 will essentially composite a virtual object of the same size with the captured image before and after rearrangement when generating an image in mixed reality space.

In this way, the method for rearranging virtual objects in this embodiment is performed based on values such as information about the position and size of the virtual object and information about the surrounding space.

In this embodiment, a method for rearranging a virtual object has been described in which the virtual object is scaled down in mixed reality space and moved closer to the user so that the appearance of the virtual object does not change.

Note that any affine transformation or other different method may be used as long as the rearrangement method minimizes degradation of the user experience. For example, the size of the virtual object may not be changed in mixed reality space, and an image of the virtual object from the user's viewpoint may be generated and composited with the captured image.

<Virtual object rearrangement flow>
Hereinafter, the detailed operation of the information processing device according to this embodiment will be described with reference to FIG.

FIG. 6 is a flowchart of the information processing device A100 in this embodiment. This flowchart starts when a system or app that controls the display of virtual objects is launched.

In step S601, the control unit A101 determines whether to end the processing based on the status of each unit of the information processing device A100. For example, if an end operation is performed via the input unit A104, the control unit A101 ends the processing of this flowchart. If it determines that the processing should continue, the control unit A101 proceeds to step S602.

In step S602, the control unit A101 photographs the surroundings via the sensor A106 and acquires the image. Photographing the surroundings means, for example, capturing images with a camera or measuring distances with laser light using LiDAR. The captured information is recorded in memory A103. Here, the control unit A101 activates the camera to acquire captured images in real time, and then proceeds to step S603.

In step S603, the control unit A101 recognizes the real space (spatial recognition) based on the surrounding data captured in step S602. The result of this spatial recognition is spatial information such as the positions and sizes of real objects such as surrounding walls, ceilings, and furniture. Once the control unit A101 has acquired the spatial information, it proceeds to step S604.

In step S604, the control unit A101 obtains information about the display position and size of the virtual object from the memory A103, compares it with the surrounding space information calculated in step S603, and determines whether the virtual object needs to be rearranged. If the control unit A101 determines that the virtual object needs to be rearranged, it proceeds to step S605. If the control unit A101 determines that the virtual object does not need to be rearranged, it proceeds to step S606. While the image of the virtual space is being displayed, the control unit A101 determines that the virtual object does not need to be rearranged because there is no interference with real objects. Furthermore, while the image of the mixed reality space is being displayed, the control unit A101 determines that the virtual object needs to be rearranged if there is interference with real objects or if the virtual object is hidden behind a real object.

In step S605, the control unit A101 updates the display position and size information of the virtual object stored in memory A103 according to the virtual object rearrangement method described above, and rearranges the virtual object. If at least a portion of the virtual object is embedded in (buried in) the real object, the control unit A101 rearranges the virtual object as if it were in front of the real object.

In step S606, the control unit A101 combines the virtual object read from memory A103 with the image captured by the camera in step S602 to generate an image in mixed reality space. Alternatively, the control unit A101 generates an image in virtual space from the virtual object read from memory A103. Once the control unit A101 has generated an image in mixed reality space or an image in virtual space, which is an image for display, it proceeds to step S607.

In step S607, the control unit A101 displays the display image generated in step S606 via the output unit A105, and then proceeds to step S601.

As described above, the information processing device A100 of this embodiment determines whether or not a virtual object needs to be rearranged based on the spatial information obtained by capturing an image of the surroundings and understanding the spatial relationship between the virtual object and the image. Furthermore, if rearrangement is necessary, the information processing device A100 of this embodiment rearranges the virtual object by shrinking it in mixed reality space and moving it forward.

As a result, when the angle of view is the same (same position and same viewing direction), the way a virtual object appears to the user will not change even if the virtual object is repositioned. This means that virtual objects that were previously invisible due to the surrounding environment can be repositioned to a position where they are visible, while minimizing any degradation to the user experience.

If the virtual object is a two-dimensional virtual object such as a virtual screen, the thickness of the screen may not be reduced. In other words, the vertical and horizontal lengths of the screen may be reduced before and after rearrangement, but the thickness may remain constant before and after rearrangement. Also, if the virtual object is a three-dimensional virtual object, it may be reduced three-dimensionally. Also, virtual objects that cast shadows on virtual objects may not be displayed. Furthermore, the shadowed portion may be displayed in accordance with the reduction ratio and movement amount of the virtual object.

When switching from an image in virtual space to an image in mixed reality space, virtual objects that are only displayed in virtual space may not be subject to rearrangement, but virtual objects that continue to be displayed in the image in mixed reality space may be subject to rearrangement.

<Modification 1 of the First Embodiment>
As described above, by following the flow in FIG. 6 , when it is necessary to rearrange a virtual object, an image of the mixed reality space in which the virtual object has been rearranged is displayed without displaying an image in which the virtual object and the real object interfere with each other.

In a modified example, an image in which a virtual object and a real object are interfering with each other, as shown in Figures 4B and 4C, may be displayed, and then an image in mixed reality space in which the virtual object has been rearranged may be displayed. In this case, in step S604 of Figure 6, the control unit A101 determines whether rearrangement of the virtual object is necessary and whether an image in mixed reality space in which a virtual object and a real object are interfering with each other has been displayed once.

If the control unit A101 determines that rearrangement of the virtual object is necessary and that an image of the mixed reality space in which the virtual object and real object are interfering has been displayed once, it proceeds to step S605. On the other hand, if it does not determine that rearrangement of the virtual object is necessary and that an image of the mixed reality space in which the virtual object and real object are interfering has been displayed once, it proceeds to step S606. This allows the user to visually recognize once that the virtual object and real object are interfering, and can predict that the virtual object will be rearranged. Instead of determining whether an image of the mixed reality space in which the virtual object and real object are interfering has been displayed once, it may also determine whether an image of the mixed reality space in which the virtual object and real object are interfering has been displayed for a predetermined period of time.

<Modification 2 of the First Embodiment>
In another modification 2, after displaying an image in which a virtual object and a real object interfere with each other, if it is determined that the virtual object needs to be rearranged, a screen may be displayed to confirm with the user whether or not to rearrange the virtual object. The flow in this case will be described with reference to FIG. 7 .

FIG. 7 is a flowchart of the information processing device A100 in this embodiment. This flowchart starts when a system or application that controls the display of virtual objects is launched. Here, explanations of steps S601 to S603 and step S606, which are the same as those in the flow of FIG. 6, will be omitted.

In step S604, the control unit A101 obtains information about the display position and size of the virtual object from memory A103, compares it with the surrounding space information calculated in step S603, and determines whether the virtual object needs to be rearranged. Furthermore, in step S702, which will be described below, the control unit A101 determines whether the user has instructed not to rearrange the virtual object. If there has been an instruction not to rearrange the virtual object, the control unit A101 determines that the virtual object does not need to be rearranged, and if there has been no instruction not to rearrange the virtual object, the control unit A101 determines that the virtual object needs to be rearranged. If the control unit A101 determines that the virtual object needs to be rearranged, the process proceeds to step S701. If the control unit A101 determines that the virtual object does not need to be rearranged, the process proceeds to step S606.

In step S701, the control unit A101 determines whether or not the user has instructed the user to rearrange the virtual objects. If the control unit A101 determines that the user has instructed the user to rearrange the virtual objects, it proceeds to step S605; if the control unit A101 does not determine that the user has instructed the user to rearrange the virtual objects, it proceeds to step S702. For example, if the user instructs the user to rearrange the virtual objects on a confirmation screen generated in step S703 below to ask the user whether or not to rearrange the virtual objects, it is determined that the user has instructed the user to rearrange the virtual objects. Furthermore, if the user does not instruct the user to rearrange the virtual objects on a confirmation screen generated in step S703 below to ask the user whether or not to rearrange the virtual objects, it is not determined that the user has instructed the user to rearrange the virtual objects.

Here, with reference to Figures 8A and 8B, we will explain an example of a confirmation screen that asks the user whether or not to rearrange virtual objects.

In FIG. 8A, the portion of the virtual object that is embedded in the wall is also composited with the mixed reality space image. For example, the image of FIG. 8A may be displayed after the image of FIG. 4B is displayed. Alternatively, the image of FIG. 8A may be displayed without first displaying the image of FIG. 4B. In the mixed reality space image 801 of FIG. 8A, a confirmation screen 802 that asks the user whether or not to rearrange the virtual object is further composited with the mixed reality space image of FIG. 4B. In addition, a shadow virtual object 812 of the confirmation screen 802 and a virtual object 813 indicating the user's designated position are composited into the mixed reality space image 801. In addition, the confirmation screen 802 displays a button that allows the user to select whether or not to rearrange the virtual window that is embedded in the wall. The user can instruct to rearrange the virtual object by moving the virtual object 813 to the "Yes" position and performing a confirm operation. In addition, the user can instruct not to rearrange the virtual object by moving the virtual object 813 to the "No" position and performing a confirm operation.

In Figure 8B, if a virtual object has a portion that is embedded in a wall, that virtual object is not composited into the image of the mixed reality space. Here, a confirmation screen 802 that asks the user whether or not to rearrange the virtual object, a virtual object 812 that is the shadow of the confirmation screen 802, and a virtual object 813 that indicates the position indicated by the user are composited into the captured image. For example, the image of Figure 8A may be displayed after the image of the mixed reality space of Figure 4B is displayed. Alternatively, the image of Figure 8A may be displayed without first displaying the image of the mixed reality space of Figure 4B.

In step S702, the control unit A101 determines whether the user has instructed not to rearrange the virtual objects. If the control unit A101 determines that the user has instructed not to rearrange the virtual objects, it proceeds to step S606. If the control unit A101 does not determine that the user has instructed not to rearrange the virtual objects, it proceeds to step S703. For example, if the user instructs not to rearrange the virtual objects on a confirmation screen generated in step S703 below to confirm whether the user will rearrange the virtual objects, it is determined that the user has instructed not to rearrange the virtual objects. Also, if the user does not instruct not to rearrange the virtual objects on a confirmation screen generated in step S703 below to confirm whether the user will rearrange the virtual objects, it is not determined that the user has instructed not to rearrange the virtual objects. In other words, if the user has not yet instructed either to rearrange or not to rearrange the virtual objects on the confirmation screen generated in step S703 below to confirm whether the user will rearrange the virtual objects, it proceeds to step S703.

Furthermore, if the control unit A101 determines that the user has instructed not to rearrange the virtual objects, it determines in the subsequent step S604 that there is no need to rearrange the virtual objects.

For example, on the screens of Figures 8A and 8B, the user can move virtual object 813 to the "Yes" position and perform a confirm operation to instruct the rearrangement of the virtual object. Also, the user can move virtual object 813 to the "No" position and perform a confirm operation to instruct the rearrangement of the virtual object. Also, when the user has not yet performed a confirm operation, this corresponds to a case where the user has not yet instructed the rearrangement or the user has not instructed the rearrangement.

In step S703, the control unit A101 generates a display image on which a confirmation screen for the user is superimposed. The control unit A101 combines the virtual object read from memory A103, the virtual object corresponding to the confirmation screen for the user, and the captured image captured by the camera in step S602 to generate an image of mixed reality space. After generating the image of mixed reality space, which is the display image, the control unit A101 proceeds to step S607. For example, an image of mixed reality space like the ones shown in Figures 8A and 8B above is generated as the display image. Note that when it is determined that virtual objects need to be rearranged, if a display image is generated on which a confirmation screen for asking the user whether or not to rearrange the virtual objects is superimposed, the virtual objects that need to be rearranged may or may not be displayed. Furthermore, when virtual objects that need to be rearranged are also displayed, parts of the virtual objects that are hidden by or embedded in real objects may not be displayed, or the hidden or embedded parts may be displayed transparently. For example, if a virtual object that requires rearrangement is also displayed, but parts of the virtual object that are hidden by or contained within a real object are not to be displayed, an image in mixed reality space is generated in which a confirmation screen is also superimposed on the image in Figure 4C.

In addition, if there is a virtual window hidden by a real object, i.e., if there is a virtual window placed behind a real object, a confirmation screen may be displayed to confirm whether or not to relocate the virtual window hidden by the real object.

This allows a decision to be made as to whether or not to reposition the virtual object in response to a user instruction when the virtual object is positioned in a position where at least a portion of the virtual object is obscured by a real object.

Second Embodiment
In the first embodiment, a method for rearranging a virtual object by shrinking it in mixed reality space without changing the appearance of the virtual object and moving it closer to the user has been described. However, depending on the user's position and the arrangement of walls, ceilings, furniture, etc., the rearrangement may result in the virtual object being reduced too much or being moved too far forward, making it too close.

If the virtual object is made too small, it may become difficult to perform touch operations, gaze input, or ray input. Also, if the virtual object is too close, it may block your view and make it impossible to see the real object in front of you.

In this embodiment, we will explain the operation of the information processing device A100 (HMD) when restrictions are placed on the reduction ratio and movement amount when rearranging a virtual object. Note that this embodiment has much in common with the first embodiment, so we will focus on the parts unique to this embodiment.

The method for determining whether to rearrange a virtual object in this embodiment is described below.

Similar to the first embodiment, whether to rearrange a virtual object is determined using spatial information such as the position and size of the virtual object and the positions and sizes of surrounding walls, ceilings, furniture, etc. First, spatial information such as the positions and sizes of the virtual object and surrounding walls, ceilings, furniture, etc. is used to calculate which of the virtual object or real object is in front as viewed from the HMD (or user). In other words, it is calculated which of the virtual object or real object is in front as viewed from the imaging position.

If the virtual object is closer than the real object, it can be determined that there is no need to reposition the virtual object, since it will not be obscured by surrounding walls, ceilings, furniture, etc.

Next, if the virtual object is behind the real object, or if the virtual object is obscured by surrounding walls, ceilings, furniture, etc., a judgment is made based on the values of the virtual object's reduction ratio and movement amount. If an attempt is made to relocate a virtual object in front of a real object and both the reduction ratio and movement amount values upon relocation do not exceed the limits (thresholds), it is determined that the virtual object will be relocated. If an attempt is made to relocate a virtual object in front of a real object and both the reduction ratio and movement amount values upon relocation exceed the limits (thresholds), it is determined that the virtual object will not be relocated. In other words, if the virtual object is reduced more than the reduction ratio limit or moved more than the movement amount limit upon relocation, it is determined that the virtual object will not be relocated to the intended position. Note that if the virtual object is reduced more than the reduction ratio limit or moved more than the movement amount limit upon relocation, it will only be reduced until the reduction ratio reaches its limit and moved until both the movement amount values reach their limits.

Here, "the reduction ratio value has reached its limit" means that the virtual object has been reduced to the smallest possible size in the mixed reality space within the set limits. Furthermore, "the movement amount value has reached its limit" means that the virtual object has been positioned closest to the HMD within the set limits.

In the method for determining whether to rearrange a virtual object described above, if the virtual object is obscured by a real object, or if the values for the reduction ratio and movement amount reach their limits, the virtual object is rearranged to a position where the values for the reduction ratio and movement amount reach their limits. In this way, by shrinking the virtual object to its limit and moving it forward, it may be possible to display part of the virtual object in front of the surrounding walls, ceiling, furniture, etc. Alternatively, the visibility of the virtual object may be improved compared to before it was rearranged.

The method for determining whether a virtual object should be rearranged described above is merely an example, and different methods for determining whether a virtual object should be rearranged may be used.

This section explains how to rearrange virtual objects in this embodiment. Examples of rearrangement methods that impose limits on the reduction ratio and movement amount when rearranging virtual objects are given here.

<First Example of Second Embodiment>
As a first example, a method for rearranging a virtual object when a limit is placed on the reduction ratio will be described below using Figures 9, 10, and 11. Here, a limit is placed on the reduction ratio, but no limit is placed on the amount of movement. Therefore, the virtual object is placed in front of the real object after being reduced to the reduction ratio limit.

Figure 9 is a diagram illustrating a scene in which a virtual window is displayed as a virtual object on a large screen indoors. Virtual window 903 is a virtual object that is treated as a target for rearrangement in this embodiment. In Figure 4, which was used for explanation in the first embodiment, the user is using the HMD while sitting on a sofa, and there is a sufficient distance to the wall in front of them. However, in Figure 9 of this embodiment, the user is using the HMD while sitting at a counter table, and the distance to the wall in front of them is short. In this situation, the virtual window 903 is displayed as if it exists behind the wall of the real object in front of the user.

FIG. 10 is a diagram illustrating the application of the virtual object rearrangement method of the first embodiment to virtual window 903 in the situation of FIG. 9. Through rearrangement, virtual window 903 is moved so that it is closer to the front wall, and its size is reduced to approximately one-seventh of its original size. In other words, in FIG. 10, it is placed at the position of virtual window 1003. If the virtual object shrinks too much like this, it may become difficult to perform touch operations, gaze input, ray input, and other inputs.

FIG. 11 is a diagram illustrating the application of the virtual object rearrangement method of the second embodiment to a virtual window 903 in the situation of FIG. 9. Here, a case is described in which the lower limit of the virtual object reduction rate is set to half of the original size. The virtual window 903 is still moved so that it is closer to the front wall through rearrangement, but its size is only reduced to half of its original size. The virtual object rearrangement method of this embodiment changes the appearance of the virtual object, but it can prevent the virtual object from shrinking so much that it becomes difficult to perform touch operations, gaze input, ray input, etc.

<Second Example of Second Embodiment>
As a second example, a method for rearranging a virtual object when a limit is placed on the amount of movement will be described below with reference to Figures 12, 13, and 14. Here, a limit is placed on the amount of movement, but the reduction ratio is adjusted to match the arrangement of the real object.

Figure 12 is a diagram illustrating a scene in which a virtual window is displayed indoors on a large screen as a virtual object. A virtual window 1203 is a virtual object that is treated as a target for rearrangement in this embodiment, and is placed in a mixed reality space 1201. Figure 12 shows a bird's-eye view of the mixed reality space 1201 in which a user 1202 exists.

In Figure 4, which was used for explanation in the first embodiment, the user is using the HMD while sitting on a sofa, with a sufficient distance between the left and right walls. However, in Figure 12 of this embodiment, the user is using the HMD in a small room such as a bathroom, with the left and right walls close by.

FIG. 13 is a diagram illustrating the application of the virtual object rearrangement method of the first embodiment to a virtual window 1203 in the situation of FIG. 12. The virtual window 1303 is a virtual object rearranged in this embodiment, and is placed in a mixed reality space 1301. FIG. 13 is a diagram looking down from above at the mixed reality space 1301 in which the user 1202 exists. The virtual window 1203 is moved so that it is closer to the left and right walls and the toilet paper holder, and its size is reduced accordingly, and it is placed at the position and size of the virtual window 1303. If the virtual object is rearranged in this way too close to the user, there is a risk that the virtual object will block the user's view and make it impossible to see the real object in front of them.

14 is a diagram illustrating the application of the virtual object rearrangement method of the second embodiment to virtual window 1203 in the situation of FIG. 12. Here, the case is described where the upper limit of the virtual object's movement is set to half the distance from the user. Virtual window 1403 is a virtual object rearranged in this embodiment, and is placed in mixed reality space 1401. FIG. 14 is a diagram illustrating a top-down view of mixed reality space 1401 in which user 1202 exists. Through rearrangement, virtual window 1203 is moved only to a position half the distance from the user, which corresponds to the upper limit of the movement, and is placed at the position of virtual window 1303. Furthermore, the size of virtual window 1203 is reduced to a size that will not be obscured by the left and right walls or the toilet paper holder, and is placed at the size of virtual window 1303. With the virtual object rearrangement method of this embodiment, the virtual object appears differently, but it is possible to prevent the virtual object from blocking the view and obscuring the real object in front of the user.

As in the two examples described above, the method for rearranging a virtual object in the second embodiment can be determined from information about the position and size of the virtual object, information about the surrounding space, and values such as the reduction ratio and movement amount limit at the time of rearrangement.

The above-mentioned rearrangement of virtual objects is performed, for example, by the processing in step S605 in Figures 6 and 7 described in the first embodiment.

In the second embodiment, a limit (threshold) is set for the amount of movement from the original virtual object position, but this is not limited to this. Because there is a possibility that the virtual object will block the view and make the real object in front of the user invisible, a limit (threshold) may be set for the position at which the virtual object is rearranged, depending on the distance from the HMD or the user. In other words, even if the virtual object is rearranged, the position at which the virtual object is rearranged will be farther away from the HMD or the user than a predetermined distance.

The above-described method for rearranging virtual objects is merely one example, and the method for rearranging virtual objects, as well as the restrictions on the reduction ratio and movement amount during rearrangement, may be different.

As described above, in the second embodiment, we have explained the operation of the information processing device A100 when limits are placed on the reduction ratio and movement amount when rearranging a virtual object.

By using the method for determining whether to rearrange a virtual object and the method for rearranging a virtual object described in the second embodiment, it is possible to prevent input to the virtual object from becoming difficult and to prevent the virtual object from blocking the view.

(Third embodiment)
In the first and second embodiments, the explanation has been given on the assumption that only one virtual object is displayed. However, depending on the use case, it may be possible to simultaneously display and use multiple virtual objects.

In such use cases, if only the virtual objects that need to be rearranged are rearranged, their relative positions to other virtual objects will change, confusing the user and worsening the user experience. Furthermore, in use cases where multiple virtual objects are displayed, there may be virtual objects that should be excluded from rearrangement in consideration of the user experience.

In this embodiment, we will explain a method for rearranging virtual objects while minimizing degradation of the user experience when multiple virtual objects are displayed simultaneously.

Note that this embodiment has many aspects in common with the first and second embodiments, so the following explanation will focus on the aspects unique to this embodiment.

The following describes the method for determining whether a virtual object is to be rearranged in this embodiment.

In the first and second embodiments, the explanations have been given on the assumption that there is always one virtual object that appears and that it is always subject to rearrangement. In this embodiment, the explanation will be given on the assumption that multiple virtual objects are displayed, and that among these there are virtual objects that should not be subject to rearrangement.

There are several possible methods for determining whether a virtual object is a target for rearrangement. For example, the user can set whether the virtual object is a target for rearrangement. The user determines for each virtual object whether the virtual object is a target for rearrangement, and this is recorded as a user setting in memory A103. The information processing device A100 can determine whether the virtual object is a target for rearrangement by checking the user setting recorded in memory A103.

Another method is to make the determination based on the type of virtual object. In many cases, the user experience does not deteriorate even if a virtual window is reduced in size when it is relocated, but reducing a 3DCG object displayed at life-size can have an impact on the user experience. For this reason, this method determines that if the type of virtual object is a virtual window, it is a candidate for relocation, but if it is 3DCG, it is not a candidate for relocation.

Another method is to make a judgment based on the situation in which the virtual object is placed. An image of the virtual space is displayed, and whether the virtual object is to be subject to relocation is determined based on whether it was placed in VR mode, which normally does not include real space in the image, or MR mode, which displays an image in mixed reality space. In the image of mixed reality space, the virtual object is placed as if it exists in real space. When a virtual object is placed in MR mode, it is likely that the virtual object is placed using the real object as the starting point, and if the virtual object is relocated, it may move to a position that does not match the user's intention. For this reason, one method of determination is to treat virtual objects placed in VR mode as subjects to relocation, and not virtual objects placed in MR mode.

Several specific examples have been given of methods for determining whether a virtual object is a target for rearrangement, but it is also possible to use one of these methods, or to use a combination of multiple methods. Furthermore, the above-mentioned methods are merely examples, and different methods for determining whether a virtual object is a target for rearrangement may also be used.

<Method for determining whether a virtual object should be repositioned>
A method for determining whether a virtual object should be rearranged in this embodiment will be described below.

First, from among the multiple virtual objects being displayed, the virtual object to be treated as the target for rearrangement is identified using the method described above for determining whether the virtual object is the target for rearrangement.

Next, select one virtual object from the virtual objects to be rearranged to use as the basis for rearrangement. Any virtual object can be selected as the basis for rearrangement, but the behavior during rearrangement will change depending on the virtual object selected. How the behavior during rearrangement changes will be explained using an example in the explanation of how to rearrange virtual objects below.

A determination is made as to whether the virtual object selected as the basis for rearrangement should be rearranged. The method for determining whether the virtual object should be rearranged is the same as that used in the first or second embodiment. If no restrictions are placed on the reduction ratio or movement amount when rearranging the virtual object, the method used in the first embodiment is used; if restrictions are placed, the method used in the second embodiment is used.

If it is determined that the virtual object selected as the basis for rearrangement should be rearranged, it is determined that all virtual objects, including other virtual objects, should be rearranged. If it is determined that the virtual object selected as the basis for rearrangement does not need to be rearranged, it is determined that all virtual objects, including other virtual objects, do not need to be rearranged.

The method for determining whether a virtual object should be rearranged described above is merely an example, and different methods for determining whether a virtual object should be rearranged may be used.

The method for rearranging virtual objects in the third embodiment will be described below with reference to Figures 15, 16, and 17.

Figure 15 is a diagram illustrating a scene in which three virtual windows are displayed indoors in MR mode. Figure 15 is a top-down view of mixed reality space 1501 in which user 1502 is present and, from left to right, virtual window 1503, virtual window 1504, and virtual window 1505 are located. Virtual window 1503, virtual window 1504, and virtual window 1505 are all treated as candidates for rearrangement.

In explaining the rearrangement method of this embodiment, for comparison, we will explain the case where the rearrangement method of the first embodiment is applied under the same conditions. Here, for each of the multiple virtual objects, it is determined whether there are any parts that are hidden by real objects and cannot be seen, or whether there are any buried parts, and then a decision is made as to whether to perform rearrangement.

FIG. 16 is a diagram illustrating a scene in which three virtual windows have been rearranged according to the rearrangement method of the first embodiment. FIG. 16 is a top-down view of mixed reality space 1601 in which user 1502 is present and in which, from left to right, virtual window 1603, virtual window 1504, and virtual window 1605 are located. In FIG. 16, virtual window 1503 and virtual window 1505 have been rearranged and are now located at the positions of virtual window 1603 and virtual window 1605, respectively.

Since the left side of virtual window 1503 is embedded in the wall, it is shrunk slightly and moved forward so that it is no longer embedded in the wall. Because virtual window 1503 is rearranged to the position of virtual window 1603, it becomes approximately two-thirds its size and the distance from the user becomes approximately two-thirds of its original distance. Next, virtual window 1504 does not need to be rearranged because it is not embedded in the wall, so it is not shrunk or moved. Furthermore, virtual window 1505 has its right side embedded in the wall and is hidden by shelf 1606 in the foreground. Therefore, it needs to be shrunk even more than virtual window 1503 and moved even further forward. Because virtual window 1505 is rearranged to the position of virtual window 1605, it becomes approximately half its size and the distance from the user becomes approximately half of its original distance.

As described above, when the rearrangement method of the first embodiment is applied individually to each virtual object, if the user is in the same position before and after the rearrangement, the size of the virtual object appears to remain unchanged. However, when the rearrangement method of the first embodiment is applied individually to each virtual object, three virtual objects that should have been at the same depth are rearranged to different positions. If the user intentionally places the virtual objects with the same depth, this type of rearrangement method can be a factor that degrades the user experience.

<Relocation Method in the Third Embodiment>
Therefore, in this embodiment, the rearrangement method of other virtual objects is determined in accordance with the rearrangement method of the virtual object selected as the reference.

Figure 17 is a diagram illustrating a scene where an entire rearrangement has been performed using virtual window 1505 as the rearrangement reference. Figure 17 is a top-down view of mixed reality space 1701 in which user 1502 is present and in which, from left to right, virtual window 1703, virtual window 1704, and virtual window 1705 are located. In Figure 17, virtual window 1503, virtual window 1504, and virtual window 1505 have been rearranged and are now located at the positions of virtual window 1703, virtual window 1704, and virtual window 1705, respectively.

Since the left side of virtual window 1505 is hidden by the wall, it is shrunk so that it is no longer hidden by the wall, and is moved forward and placed in the position of virtual window 1705. After being rearranged, virtual window 1505 becomes approximately half its size, and the distance from the user is also approximately half of its original distance.

Virtual windows 1503 and 1504 are moved forward while shrinking by the same amount as virtual window 1505, in accordance with the shrink ratio and movement amount when virtual window 1505 is rearranged, and are thereby placed at the positions of virtual windows 1703 and 1704, respectively.

Here, even if virtual window 1504 is selected as the reference, virtual window 1503 and virtual window 1505 will not be repositioned and will remain embedded in the wall. Also, even if virtual window 1503 is selected as the reference, virtual window 1505 will be repositioned and will no longer be embedded in the wall, but will instead be embedded in shelf 1606. Therefore, virtual window 1505 is selected as the reference.

As described above, in this embodiment, by determining the rearrangement method for other virtual objects in accordance with the rearrangement method for the virtual object selected as the reference, it is possible to maintain the relative positional relationships between virtual objects before and after rearrangement.

The above-mentioned rearrangement of virtual objects is performed, for example, by the processing in step S605 in Figures 6 and 7 described in the first embodiment.

As described above, this embodiment has described the operation of the information processing device A100 when multiple virtual objects are displayed simultaneously.

By using the method described in this embodiment, the relative positional relationships between virtual objects can be maintained before and after rearrangement, allowing virtual objects to be rearranged while minimizing degradation of the user experience.

(Fourth embodiment)
In the first, second, and third embodiments, methods for rearranging virtual objects according to the respective conditions have been described. However, in order to apply these rearrangement methods while minimizing degradation of the user experience, it is considered desirable to perform the rearrangement gradually over a certain period of time.

If the HMD completes the repositioning of a virtual object within one frame of updating the screen, the user will not notice that the virtual object has been repositioned, and the position of the virtual object will differ from the user's perception. This could lead to a poor user experience.

In this embodiment, we will therefore explain a method for rearranging virtual objects with animation to minimize degradation of the user experience.

Note that this embodiment has many aspects in common with the first to third embodiments, so the following explanation will focus on the aspects unique to this embodiment.

The following describes a method for rearranging virtual objects with animation in this embodiment.

The method for rearranging virtual objects in this embodiment is basically the same as the methods described in the first, second, and third embodiments. The difference is that rearrangement is performed over time while animating.

For example, suppose the user operates the HMD to switch scenes from VR mode to MR mode.

In this embodiment, when it is determined that a virtual object needs to be relocated due to a scene change, the virtual object is relocated with animation in time with the scene change. The animation may be linear (constant speed), ease-in (gradual acceleration), ease-out (gradual deceleration), ease-in-out (acceleration and deceleration), etc., as long as it conveys to the user that the virtual object is gradually changing in size or moving. Furthermore, by matching the time it takes for the animation to the time it takes for the scene change, the relocation can be performed more naturally.

The above-described method of rearranging virtual objects with animation is merely one example, and different methods of rearranging virtual objects with animation may be used.

As described above, this embodiment has described the operation of the information processing device A100 when rearranging virtual objects with animation to prevent a deterioration in the user experience.

This allows the user to recognize that the virtual object has been repositioned, and there is no discrepancy between the actual position of the virtual object and the user's perception, so virtual objects can be repositioned without deteriorating the user experience.

For example, in FIG. 6, in step S604, if an animation in which a virtual object is being rearranged is in progress, the control unit A101 determines that the virtual object needs to be rearranged until the rearrangement is complete. Furthermore, in step S605, if an animation in which a virtual object is being rearranged is in progress, the control unit A101 rearranges the virtual object in accordance with the animation. Furthermore, in step S606, the control unit A101 combines the virtual object with a captured image of real space in accordance with the arrangement of the virtual object rearranged in step S605, and generates an image of mixed reality space as an image to be displayed.

The above describes preferred embodiments of the present disclosure, but the present disclosure is not limited to these embodiments, and various modifications and variations are possible within the scope of its essence.

In the first, second, third, and fourth embodiments, it is assumed that the virtual object is shrunk when it is rearranged. However, depending on the size of the virtual object, it may be sufficient to simply move the virtual object closer. Therefore, the position of the virtual object may be adjusted so that it is placed in front of the real object without shrinking the virtual object.

(Other embodiments)
The present disclosure can also be realized by executing the following process: That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and a computer (or a control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage media storing the program constitute the present disclosure.

The present disclosure has been described in detail above based on preferred embodiments, but the present disclosure is not limited to these specific embodiments, and various forms that do not deviate from the gist of the disclosure are also included in the present disclosure. Parts of the above-described embodiments may be combined as appropriate.

Note that each functional unit in each of the above embodiments (variations) may or may not be separate hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of a single functional unit may be realized by separate hardware. Two or more functions of a single functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

The present disclosure can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

Furthermore, in each of the examples described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs, ASICs, FPGAs, programmable logic devices, etc.).

The disclosure of this embodiment includes the following configuration, method, and program.

[Configuration 1]
an acquisition means for acquiring information about the arrangement of real objects in a real space;
an image acquisition means for acquiring a captured image of the real space in a predetermined direction;
a control means for controlling the generation of an image in a mixed reality space, which is an image obtained by combining the captured image and a virtual object;
the control means controls to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction, when the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object.

[Configuration 2]
the control means controls the information processing device to generate, in the mixed reality space, an image of a first mixed reality space in which the virtual object is placed at the first position, and then generate, in the mixed reality space, an image of a second mixed reality space in which the virtual object is placed at the second position.

[Configuration 3]
the control means controls the information processing device to generate, after generating the image of the first mixed reality space, an image of a third mixed reality space in which the virtual object is placed at a position between the first position and the second position, until generating the image of the second mixed reality space.

[Configuration 4]
4. The information processing device according to any one of configurations 1 to 3, wherein, when generating an image of the mixed reality space, the control means controls the virtual object in the mixed reality space to be reduced from a size of the virtual object at the first position and placed at the second position.

[Configuration 5]
5. The information processing apparatus according to configuration 4, wherein the control means reduces the virtual object based on the size of the virtual object in the mixed reality space at the first position.

[Configuration 6]
The information processing device according to configuration 5, wherein the control means generates an image in the mixed reality space in which the virtual object is placed at the second position at a position and size that match the first position and the second position as seen by the user.

[Configuration 7]
the control means controls a display unit to display the generated image of the mixed reality space, and after displaying an image of a first mixed reality space in which the virtual object is located at the first position in the mixed reality space, controls a display unit to display an image of a second mixed reality space in which the virtual object is located at the second position in the mixed reality space.

[Configuration 8]
The information processing device according to configuration 7, wherein the control means controls to display a screen that prompts a user to select whether or not to move the virtual object after displaying the image of the first mixed reality space, and when the user selects to move the virtual object, controls to generate an image of the second mixed reality space.

[Configuration 9]
9. The information processing apparatus according to configuration 8, wherein the control means displays an image of the first mixed reality space when the user selects not to move the virtual object.

[Configuration 10]
10. The information processing device according to any one of configurations 1 to 9, wherein the control means controls a display unit to display the generated image of the mixed reality space, and controls a display unit to not display an image of a first mixed reality space in which the virtual object is located at the first position in the mixed reality space, but to display an image of a second mixed reality space in which the virtual object is located at the second position in the mixed reality space.

[Configuration 11]
The information processing device according to configuration 10, wherein the control means controls to display a screen that prompts a user to select whether or not to move the virtual object before displaying the image of the first mixed reality space, and when the user selects to move the virtual object, controls to display the image of the second mixed reality space.

[Configuration 12]
12. The information processing apparatus according to configuration 11, wherein the control means controls the display of the image of the first mixed reality space when the user selects not to move the virtual object.

[Configuration 13]
13. The information processing device according to any one of configurations 1 to 12, wherein the control means generates an image of the mixed reality space in which the virtual object is placed at the second position by performing a predetermined affine transformation on the virtual object.

[Configuration 14]
The information processing device according to any one of configurations 1 to 13, wherein the second location is a location closer to the first location than a predetermined distance or a location farther from the user's location than a predetermined distance.

[Configuration 15]
15. The information processing device according to any one of configurations 4 to 14, wherein when the virtual object is reduced, the virtual object is not reduced below a predetermined size.

[Configuration 16]
16. The information processing device according to any one of configurations 1 to 15, wherein, when a plurality of virtual objects exist in the mixed reality space and at least a part of a first virtual object among the plurality of virtual objects is at the first position, the control means determines positions to rearrange the plurality of virtual objects based on the second position in the mixed reality space at which the first virtual object is to be arranged, and controls to generate an image of the mixed reality space.

[Configuration 17]
17. The information processing apparatus according to any one of configurations 1 to 16, wherein the control means controls whether or not to place the virtual object at the second position depending on the type of the virtual object.

[Control method]
an acquisition step of acquiring information about the arrangement of real objects in a real space;
an image acquisition step of acquiring a captured image of the real space in a predetermined direction;
a control step of controlling the generation of an image in a mixed reality space, which is an image obtained by combining the captured image and a virtual object;
and when the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object, the control step controls to generate an image of the mixed reality space where the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction.

[program]
A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 17.

[system]
an acquisition device that acquires information about the arrangement of real objects in real space;
an image capture device that captures an image of the real space in a predetermined direction;
a control device that controls the generation of an image in a mixed reality space that is an image obtained by combining the captured image and a virtual object,
the control device controls to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the specified direction, when the position of the virtual object in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object.

The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present invention. Therefore, the following claims are appended to clarify the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2024-107716, filed July 3, 2024, the entire contents of which are incorporated herein by reference.

Claims (21)

an acquisition means for acquiring information about the arrangement of real objects in a real space;
an image acquisition means for acquiring a captured image of the real space in a predetermined direction;
a control means for controlling the generation of an image in a mixed reality space, which is an image obtained by combining the captured image and a virtual object;
the control means controls to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the predetermined direction, when the position at which the virtual object is placed in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object. 2. The information processing device according to claim 1, wherein the control means controls to generate, in the mixed reality space, an image of a first mixed reality space in which the virtual object is placed at the first position, and then to generate, in the mixed reality space, an image of a second mixed reality space in which the virtual object is placed at the second position. 3. The information processing device according to claim 2, wherein the control means controls the generation of an image of a third mixed reality space in which the virtual object is placed at a position between the first position and the second position, after generating the image of the first mixed reality space and before generating the image of the second mixed reality space. The information processing device according to claim 1 , wherein, when generating an image of the mixed reality space, the control means controls the virtual object to be reduced in size in the mixed reality space from the size at the first position and placed at the second position. The information processing apparatus according to claim 4 , wherein the control means reduces the virtual object based on a size of the virtual object in the mixed reality space at the first position. The information processing device according to claim 5, wherein the control means generates an image in the mixed reality space in which the virtual object is placed at the second position at a position and size that match the first position and the second position as seen by the user. the control means controls a display unit to display the generated image of the mixed reality space;
2. The information processing device according to claim 1, wherein the information processing device controls to display, in the mixed reality space, an image of a first mixed reality space in which the virtual object is placed at the first position, and then, in the mixed reality space, an image of a second mixed reality space in which the virtual object is placed at the second position. 8. The information processing device according to claim 7, wherein the control means controls to display a screen that prompts a user to select whether or not to move the virtual object after displaying the image of the first mixed reality space, and when the user selects to move the virtual object, controls to generate an image of the second mixed reality space. The information processing apparatus according to claim 8 , wherein the control means displays the image of the first mixed reality space when the user selects not to move the virtual object. the control means controls a display unit to display the generated image of the mixed reality space;
2. The information processing device according to claim 1, wherein the information processing device controls the display of an image of a second mixed reality space in which the virtual object is located at the second position in the mixed reality space, without displaying an image of a first mixed reality space in which the virtual object is located at the first position in the mixed reality space. 11. The information processing device according to claim 10, wherein the control means controls to display a screen that prompts a user to select whether or not to move the virtual object before displaying the image of the first mixed reality space, and when the user selects to move the virtual object, controls to display the image of the second mixed reality space. The information processing apparatus according to claim 11 , wherein the control means controls the display of the image of the first mixed reality space when the user selects not to move the virtual object. The information processing apparatus according to claim 1 , wherein the control means generates an image of the mixed reality space in which the virtual object is placed at the second position by performing a predetermined affine transformation on the virtual object. The information processing apparatus according to claim 1 , wherein the second location is a location closer to the first location than a predetermined distance or a location farther from the user's location than a predetermined distance. The information processing apparatus according to claim 4 , wherein when the virtual object is reduced, the virtual object is not reduced below a predetermined size. 2. The information processing device according to claim 1, wherein, when a plurality of virtual objects exist in the mixed reality space and at least a part of a first virtual object among the plurality of virtual objects is at the first position, the control means controls to determine positions to rearrange the plurality of virtual objects based on the second position at which the first virtual object is to be arranged in the mixed reality space, and generate an image of the mixed reality space. The information processing apparatus according to claim 1 , wherein the control means controls whether or not to place the virtual object at the second position depending on the type of the virtual object. The information processing apparatus according to claim 1 , wherein the real object is a real object that is stationary in the real space. an acquisition step of acquiring information about the arrangement of real objects in a real space;
an image acquisition step of acquiring a captured image of the real space in a predetermined direction;
a control step of controlling the generation of an image in a mixed reality space, which is an image obtained by combining the captured image and a virtual object;
and when the position at which the virtual object is placed in the mixed reality space is a first position at which at least a part of the virtual object is behind the real object or embedded inside the real object, the control step controls to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object as viewed from the predetermined direction. A program for causing a computer to function as each of the means of the information processing device described in claim 1. an acquisition device that acquires information about the arrangement of real objects in real space;
an image capture device that captures an image of the real space in a predetermined direction;
a control device that controls the generation of an image in a mixed reality space that is an image obtained by combining the captured image and a virtual object,
the control device controls to generate an image of the mixed reality space in which the position of the virtual object in the mixed reality space is a second position that is in front of the real object when viewed from the specified direction, when the position at which the virtual object is placed in the mixed reality space is a first position where at least a part of the virtual object is behind the real object or embedded inside the real object.