KR102872425B1 - Extended reality device and method for controlling the same - Google Patents

Abstract

The present invention relates to an XR device and a control method thereof, and more specifically, is applicable to the fields of 5G communication technology, robotics technology, autonomous driving technology, and AI (Artificial Intelligence) technology.

Description

XR device and method for controlling the same {EXTENDED REALITY DEVICE AND METHOD FOR CONTROLLING THE SAME}

The present invention relates to an XR device and a control method thereof, and more specifically, is applicable to the fields of 5G communication technology, robotics technology, autonomous driving technology, and AI (Artificial Intelligence) technology.

VR (Virtual Reality) technology provides real-world objects and backgrounds as CG (Computer Graphic) images only, AR (Augmented Reality) technology provides virtual CG images on top of real-world object images, and MR (Mixed) technology is a computer graphics technology that provides virtual objects by mixing and combining them in the real world. VR, AR, MR, etc. mentioned above are all sometimes simply referred to as XR (extended reality) technology.

In particular, XR technology has recently been used to provide virtual services to users. For example, XR devices installed in stores or at home provide virtual fitting services through screens.

Therefore, users want to quickly select the clothes they want while the virtual fitting service is provided.

However, XR devices provided according to conventional technology had the problem that it was virtually impossible for users to quickly select the clothes they wanted because they did not provide a separate keyboard or the like.

KR10-2017-0133455 (published on December 5, 2017)

One embodiment of the present invention seeks to provide a solution for a user to more quickly select a desired outfit when an XR device provides a virtual fitting service.

Another embodiment of the present invention provides a technology for filtering clothes of a main group through a first interface in a virtual fitting service and accurately selecting specific clothes of a subgroup through a second interface.

Another embodiment of the present invention seeks to address the critical power saving issue in XR devices.

However, the scope of the present invention is not limited to the aforementioned purposes, and the scope of the present invention may be expanded to other purposes that can be inferred by a person skilled in the art based on the entire contents of this specification.

In order to achieve the above-described objects and other purposes, a method for controlling an XR device providing a virtual fitting service according to an embodiment of the present invention comprises the steps of: storing information on at least one virtual clothing in a memory; receiving at least one voice command from a user around the XR device; extracting information on clothing belonging to a specific main group from among information on at least one virtual clothing stored in the memory based on the received at least one voice command; recognizing a gesture of the user; and extracting information on clothing belonging to a subgroup from among clothing belonging to the specific main group based on the recognized gesture.

And, an XR device providing a virtual fitting service according to another embodiment of the present invention includes a memory that stores information on at least one virtual garment, a microphone that receives at least one voice command from a user around the XR device, a controller that extracts information on garments belonging to a specific main group from among information on at least one virtual garment stored in the memory based on the received at least one voice command, and a camera that recognizes a gesture of the user.

The above controller is designed to extract information on clothes belonging to a sub-group from among clothes belonging to the specific main group, for example, based on the recognized gesture.

According to one embodiment of the present invention, when an XR device provides a virtual fitting service, a solution is provided for a user to select a desired outfit more quickly.

According to another embodiment of the present invention, a technology is provided for filtering clothes of a main group through a first interface in a virtual fitting service and accurately selecting specific clothes of a subgroup through a second interface.

And, another embodiment of the present invention solves an important power saving problem in XR devices.

However, the scope of the present invention is not limited to the aforementioned technical effects, and the scope of the present invention may be expanded to other technical effects that can be inferred by a person skilled in the art based on the entire contents of this specification.

Figure 1 illustrates a case where an XR device according to one embodiment of the present invention is implemented as an HMD type.
FIG. 2 illustrates a case where an XR device according to one embodiment of the present invention is implemented as an AR glass type.
Figure 3 illustrates a case where one embodiment of the present invention is applied to a clothing-related device.
FIG. 4 illustrates a process in which an XR device according to one embodiment of the present invention provides a virtual fitting service.
Figure 5 illustrates components of an XR device to which the flow chart illustrated in Figure 4 is applied.
Figures 6 and 7 illustrate problems of virtual fitting services according to the prior art.
FIG. 8 illustrates a process in which an XR device according to one embodiment of the present invention quickly detects specific clothing using two different interfaces.
Figures 9 to 11 illustrate a process in which an XR device according to one embodiment of the present invention primarily filters virtual clothes based on voice recognition.
FIGS. 12 to 14 illustrate a process of secondarily filtering virtual clothes based on gesture recognition by an XR device according to an embodiment of the present invention.
FIGS. 15 to 17 illustrate a process in which an XR device according to one embodiment of the present invention quickly detects specific virtual clothes using two different interfaces, namely voice recognition and gesture recognition.
Figure 18 illustrates an inactive area and an active area within a screen defined by an XR device according to one embodiment of the present invention.
And, FIGS. 19 to 21 illustrate a process of arranging a plurality of virtual clothes based on the area information defined in FIG. 18.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

The following examples of the present invention are intended only to illustrate the invention and are not intended to limit or restrict the scope of the invention. Anything readily inferred by a specialist in the technical field of the invention from the detailed description and examples of the invention is construed as falling within the scope of the invention.

The above detailed description should not be construed as limiting in any respect, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

The extended reality described in this specification is a general term for virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR technology provides real-world objects and backgrounds only as CG images, AR technology provides virtual CG images over images of actual objects, and MR technology is a computer graphics technology that mixes and combines virtual objects with the real world.

MR technology is similar to AR in that it presents both real and virtual objects simultaneously. However, while AR uses virtual objects to complement real objects, MR uses virtual and real objects on an equal footing.

XR technology can be applied to HMD (Head-Mount Display), HUD (Head-Up Display), mobile phones, tablet PCs, laptops, desktops, TVs, digital signage, etc., and devices to which XR technology is applied can be called XR devices.

However, recently, rather than clearly distinguishing between VR, AR, and MR technologies, they are often referred to as XR (extended reality) technologies. Therefore, embodiments of the present invention are applicable to all VR, AR, MR, and XR technologies.

Meanwhile, hardware-related element technologies applicable to VR, AR, MR, and XR technologies include wired/wireless communication technologies, input interface technologies, output interface technologies, and computing device technologies. Furthermore, software-related element technologies include tracking and alignment technologies, voice recognition technologies, interaction and user interface technologies, location-based service technologies, search technologies, and AI (Artificial Intelligence) technologies.

In particular, embodiments of the present invention seek to solve at least one of the following problems: communication problems with other devices, inefficient memory usage problems, slow data processing speeds due to inconvenient UX/UI, video problems, audio problems, motion sickness, or other problems, by using the aforementioned HW/SW related element technologies.

FIG. 1 illustrates a case where an XR device according to one embodiment of the present invention is implemented as an HMD type. Various embodiments described below can also be implemented as the HMD type illustrated in FIG. 1.

The XR device (100a) of the HMD type illustrated in Fig. 1 includes a communication unit (110), a control unit (120), a memory unit (130), an I/O unit (140a), a sensor unit (140b), and a power supply unit (140c). In particular, the communication unit (110) within the XR device (10a) communicates with a mobile terminal (100b) via wired or wireless means.

Also, Fig. 2 illustrates a case where an XR device according to an embodiment of the present invention is implemented as an AR glass type. Various embodiments described below can also be implemented as the AR glass type illustrated in Fig. 2.

As illustrated in FIG. 2, AR glasses may include a frame, a control unit (200), and an optical display unit (300).

The frame may have the form of glasses worn on the face of the user's (10) body as shown in FIG. 2, but is not necessarily limited thereto, and may also have the form of goggles or the like worn in close contact with the face of the user (10).

Such a frame may include a front frame (110) and first and second side frames.

The front frame (110) has at least one opening and can extend in a first horizontal direction (x), and the first and second side frames can extend in a second horizontal direction (y) intersecting the front frame (110) and can extend parallel to each other.

The control unit (200) can generate an image or a video of a series of images to be shown to the user (10). The control unit (200) may include an image source that generates an image and a plurality of lenses that diffuse and converge light generated from the image source. The image generated by the control unit (200) can be output to the optical display unit (300) through a guide lens (P200) positioned between the control unit (200) and the optical display unit (300).

Such a control unit (200) may be fixed to one of the first and second side frames. For example, the control unit (200) may be fixed to the inside or outside of one of the side frames, or may be integrally formed by being built into the inside of one of the side frames.

The optical display unit (300) can perform a role of displaying an image generated by the control unit (200) to the user (10), and can be formed of a translucent material to display the image to the user (10) while allowing the external environment to be viewed through the opening.

Such an optical display unit (300) may be inserted into and fixed to an opening included in a front frame (110), or may be positioned on the back surface of the opening (i.e., between the opening and the user (10)) and fixed to the front frame (110). In the present invention, as an example, a case in which the optical display unit (300) is positioned on the back surface of the opening and fixed to the front frame (110) is illustrated.

As shown in Fig. 2, when an image is input from the control unit (200) to the incident area (S1) of the optical display unit (300), the image light is emitted through the optical display unit (300) to the output area (S2) of the optical display unit (300), so that the image generated by the control unit (200) can be shown to the user (10).

Accordingly, the user (10) can view the external environment through the opening of the frame (100) and simultaneously view the image generated by the control unit (200).

Figure 3 illustrates a case where one embodiment of the present invention is applied to a clothing-related device.

Embodiments of the present specification can be applied not only to XR devices but also to various clothing-related devices.

Further explanation in relation to one embodiment of the present invention, as illustrated in FIG. 3, when a user (100) approaches a clothing-related device (e.g., styler, air dresser, etc.), the user's approach is recognized through a camera or sensor installed in the clothing-related device.

In addition, the display (200) installed on the front of the clothing-related device displays a recognized user-related avatar, and additionally displays a graphic in the form of a virtual wearing of clothing, hat, etc. desired by the user (100). As illustrated in FIG. 3, the actual user (100) is not wearing a hat, but it can be confirmed that the avatar displayed on the display (200) is wearing a virtual hat. Furthermore, the display (200) can also function as a mirror when the user is not recognized.

Lastly, in FIG. 3, it is assumed that the display (200) is exposed on the front of the clothing-related device, but it is also within the scope of the present invention to design the display to be located inside when the user (100) opens the door of the clothing-related device.

Meanwhile, although the specification describes XR devices and various multimedia devices as examples, the present invention is sufficient for any device with a display function.

FIG. 4 illustrates a process in which an XR device according to one embodiment of the present invention provides a virtual fitting service.

As illustrated in FIG. 4, an XR device according to an embodiment of the present invention provides a virtual fitting service, and in particular, it can solve the problems of the prior art described above in that a coarse search uses voice recognition technology, while a fine search uses gesture recognition technology.

First, the XR device stores information about at least one virtual garment in its memory (S410). Of course, receiving information from an external server rather than internal memory also falls within the scope of the present invention.

The XR device receives at least one voice command from a user around the XR device (S420).

The XR device extracts information on clothes belonging to a specific main group from information on at least one virtual clothes stored in the memory based on at least one voice command received (S430).

The XR device recognizes the user's gesture (S440), and based on the recognized gesture, extracts information on clothes belonging to a sub-group from among clothes belonging to the specific main group (S450).

A specific solution for extracting information on clothes belonging to the main group at step S430 and then extracting information on clothes belonging to the sub-group at step S450 will be described in detail in FIG. 8 below.

And, although not shown in FIG. 4, the at least one voice command includes, for example, a first voice command or a second voice command.

As described above, a feature of one embodiment of the present invention is that it quickly detects the clothes desired by the user (even without a keyboard, etc.) by optimizing voice recognition technology and gesture recognition technology. However, if the XR device continuously performs voice recognition without any trigger conditions, there is a problem that unnecessary power consumption occurs. Therefore, when the first voice command or the second voice command is received, the XR device changes the operation of the voice recognition engine stored in the memory to a disabled state.

Furthermore, if gesture recognition is designed to occur only within the necessary range, the power waste problem of XR devices can be additionally solved.

For example, when the XR device receives the first voice command (e.g., “add lettering”), the XR device is designed to recognize both hands of the user. Furthermore, the XR device searches for clothes containing any letters within the area formed by the first and second hands, and displays at least one of the clothes containing the random letters. This will be described in more detail below with reference to FIG. 15.

On the other hand, if the XR device receives the second voice command (e.g., “color fine-tuning”), the XR device displays an option for fine-tuning the color of the clothes. Furthermore, the XR device is designed to recognize only one hand of the user within a certain distance from the location of the displayed option. Then, the XR device searches for clothes of a specific color based on the motion of the recognized one hand. In this regard, this will be described in more detail below with reference to FIG. 17. That is, referring to FIGS. 15 and 17, since the number of hands to be recognized as gestures is predetermined based on keywords recognized by voice recognition, there is a technical effect of minimizing unnecessary motion tracking.

Furthermore, an XR device according to an embodiment of the present invention classifies the screen of the XR device into an active area and an inactive area based on information about clothes extracted through the various interfaces (e.g., voice recognition/gesture recognition).

In particular, the XR device is further designed so that only the virtual clothing in the active area of the screen changes in response to the user's gesture, while the virtual clothing in the inactive area of the screen does not change in response to the user's gesture. Therefore, the technical effect of providing the user with the option to more quickly select the desired virtual clothing is expected. This will be described later with reference to FIGS. 18 to 21.

Meanwhile, while the invention of the method is mainly described in FIG. 4, it can also be implemented in an XR device. This will be described later with reference to FIG. 5.

Figure 5 illustrates components of an XR device to which the flow chart illustrated in Figure 4 is applied. Of course, those skilled in the art may implement the XR device illustrated in Figure 5 using other devices of the previous Figures 1 to 3, and this also falls within the scope of the present invention.

An XR device providing a virtual fitting service according to an embodiment of the present invention includes a memory (510), a microphone (520), a camera (530), a screen (540), a voice recognition engine (550), and a controller (500), as illustrated in FIG. 5. In particular, the controller (500) controls the overall function of the XR device and the operation of each component module. However, the component modules illustrated in FIG. 5 are merely examples, and deleting, adding, or modifying some modules to implement the technical idea of the present invention also falls within the scope of the present invention.

The memory (510) stores information of at least one virtual garment, and the microphone (520) is designed to receive at least one voice command from a user around the XR device.

The controller (500) extracts information on clothes belonging to a specific main group from among information on at least one virtual clothes stored in the memory (510), based on at least one voice command received.

The camera (530) recognizes the user's gesture. Furthermore, the controller (500) extracts information on clothes belonging to a subgroup from among clothes belonging to the specific main group based on the recognized gesture. That is, as described above, the first filtering is designed to be performed based on voice recognition technology, and the second filtering is designed to be performed based on gesture recognition technology, thereby providing the advantage of allowing the user to quickly select the desired clothes without the aid of a keyboard.

Figures 6 and 7 illustrate problems of virtual fitting services according to the prior art.

As shown in (a) of Fig. 6, a virtual fitting service can also be provided through signage (611) using conventional technology.

For example, a virtual avatar corresponding to a user (610) is displayed, and a desired top (e.g., a shirt) (612) can be confirmed. However, according to the prior art, when the user (610) selects a top (612), there is a problem in that the type of clothing worn on the lower body (613) is not displayed. One embodiment of the present invention seeks to solve this problem as well.

Furthermore, if a user does not like the top (612) and wants to select a different virtual outfit, there is a problem in that the user must scroll (621) through a complex screen (620) to find the desired outfit, as shown in (b) of Fig. 6. In particular, if there are a large number of outfits that can be tried on on the virtual avatar, manually scrolling through them takes too much time.

Furthermore, the signage shown in Figure 6 is typically installed in public places, making it virtually impossible to use a separate keyboard. One embodiment of the present invention seeks to address all of these issues.

Meanwhile, to solve the aforementioned problems of the prior art, the upper garment (710) and lower garment (720) of the clothing are designed to be displayed together, as shown in Fig. 7. Accordingly, the user can check the overall feel of the clothing in advance.

That is, even when a user selects either a top or bottom, the remaining parts (bottoms or tops) are displayed with the default clothing. Alternatively, the XR device can display recommended clothing for the unselected parts, which is determined based on the user's previous history.

FIG. 8 illustrates a process in which an XR device according to one embodiment of the present invention quickly detects specific clothing using two different interfaces.

According to one embodiment of the present invention, as illustrated in FIG. 8, a user can quickly check the clothes he or she wants by using two different types of interfaces, namely, voice recognition and gesture recognition.

For example, voice recognition can be used to express various things (e.g. jeans, skirts, hats, etc.), and gesture recognition can be used to clearly express ambiguous dimensions and locations that are difficult to express with voice recognition.

As illustrated in Figure 8, in Step 1, a coarse filter is used based on voice recognition technology. This means that clothing belonging to a main group is extracted from the numerous clothing items stored in memory and on the server. For example, suppose a user gives a voice command for a white suit shirt. However, the user may want to adjust the specific size, and voice recognition alone cannot quickly accommodate this.

Therefore, as illustrated in Figure 8, in Step 2, a fine filter is used based on gesture recognition technology. This means that clothing in a subgroup is extracted from the main group. For example, if a user makes a gesture to adjust the size of a specific part of a shirt, the XR device can ultimately recommend a specific piece of clothing optimized for the user.

Figures 9 to 11 illustrate a process of primarily filtering virtual clothing based on voice recognition by an XR device according to an embodiment of the present invention. Figures 9 to 11 relate to embodiments of extracting clothing from a main group using the coarse filter illustrated in Figure 8.

As illustrated in (a) of Fig. 9, when a user near the XR device says “jeans and a white t-shirt” (910), the XR device recognizes this. At this time, the aforementioned microphone and voice recognition engine are activated. Then, the XR device displays a virtual avatar (911) wearing a random pair of jeans and a white t-shirt on the screen. Therefore, there is an advantage in that the user can simultaneously check the virtual avatar by putting on the desired top and bottom.

Furthermore, as illustrated in (b) of FIG. 9, when a user around the XR device says “Add baseball jumper” (920), the XR device recognizes this. Then, the XR device displays a virtual avatar (921) wearing a random baseball jumper on top of the existing clothing on the screen.

However, a user who sees the screen shown in (b) of Fig. 9 may not like the combination of jeans, a white T-shirt, and a baseball jumper. At this time, as shown in (c) of Fig. 9, if a user near the XR device says “take off the jumper” (930), the “baseball jumper” is deleted from the virtual avatar (931). Even if the user says “jumper” instead of “baseball jumper,” the XR device has the advantage of being able to recognize the user’s intention by referring to the existing history.

In addition, the embodiments illustrated in FIG. 9 have the advantage of allowing users to quickly check the look and feel, as coarse search through voice recognition is possible first.

Meanwhile, as shown in (a) of Fig. 10, it is assumed that the XR device is displaying a virtual avatar (1011) wearing a white T-shirt.

At this time, if a user around the XR device says “No. Change it to a shirt” (1010), the XR device displays a virtual avatar (1012) wearing a shirt, as shown in (b) of FIG. 10. Unlike the embodiments of FIG. 9 described above, since there is no color designation, the design maintains the same color as the previous top, the T-shirt (1011). Therefore, the user does not need to mention the color, thereby improving the data processing speed.

Furthermore, as illustrated in (c) of FIG. 10, when a user around the XR device says “matching tie” or “matching tie recommendation” (1020), the XR device recognizes this. Then, the XR device displays a virtual avatar (1021) wearing a tie appropriate for an existing outfit on the screen. At this time, the recommended tie may refer only to the purchase history of the recognized user, or a tie that has sold a lot in the same situation may be automatically recommended, all of which fall within the scope of the present invention.

And, in the previously described embodiments of FIGS. 9 and 10, it was assumed that the user specified both the type of upper and lower garments.

On the other hand, as illustrated in FIG. 11, a user around the XR device may also specify only “jeans” (1100) corresponding to the bottom without specifying the top. However, one embodiment of the present invention is designed so that even if a user specifies only one of the top or bottom, the other unspecified clothing item is automatically recommended.

That is, as illustrated in FIG. 11, the XR device is designed to not only display virtual clothes (1101) corresponding to the jeans mentioned by the user along with a virtual avatar, but also to display recommended clothes (1102) for tops that have not been specified by the user. Tops sold together with jeans within a specific period are automatically recommended. In particular, filtering only the quantity sold within a specific distance from the point where the XR device is located or within the same administrative district by using the location information of the XR device (EX: GPS information, etc.) also falls within the scope of the present invention.

Meanwhile, the aforementioned FIGS. 9 to 11 are based on coarse search through voice recognition, but voice commands have a problem in that there is a high possibility of error in detailed specifications. For example, it is impossible to accurately determine the user's actual intention for any clothing item with voice alone, such as "I wish the sleeves were a little longer (or 3 cm)?", "I wish the V-neck was a little deeper (or 2 cm)?", "I wish the check pattern was a little thicker?", "I wish there was a stripe pattern slightly below the left chest?". This is because the voice engine's recognition of the language is insufficient. To solve this problem, one embodiment of the present invention switches to gesture recognition for specific filtering.

In this regard, the drawings below will be described later.

FIGS. 12 to 14 illustrate a process of secondarily filtering virtual clothes based on gesture recognition by an XR device according to an embodiment of the present invention.

That is, after a base search is completed using voice recognition, it is designed to enable a fine search using gesture recognition. In particular, when voice recognition is performed for specific keywords, that is, when a keyword that is a trigger condition for entering gesture recognition mode is voice recognized, the XR device according to an embodiment of the present invention has a technical effect of minimizing unnecessary power consumption by disabling the microphone and voice recognition engine for voice recognition.

Furthermore, even during the fine search process through gesture recognition, it is expected that there will be a technological effect that can further reduce unnecessary power consumption by distinguishing and defining whether to recognize only one hand or two hands depending on specific situations.

First, when the user says “detailed adjustments to the white tee,” the XR device that recognizes this displays a virtual line (1201) that moves along the outline of the white tee worn by the virtual avatar, as shown in (a) of Fig. 12.

The XR device, which recognizes the user's gesture of moving his hand to the right, displays a virtual line (1202) moved around the neck as shown in (b) of Fig. 12.

Furthermore, the XR device, which recognizes the user's gesture of moving his hand back to the right, displays a virtual line (1203) moved around the right shoulder of the white T-shirt, as shown in (c) of FIG. 12.

Finally, the XR device, which recognizes the user's gesture of moving his hand back to the right, displays a virtual line (1204) moved around the right side of the white tee as shown in (d) of FIG. 12.

As mentioned above, a single hand gesture is sufficient for a user to select a specific part of a garment. More specifically, two hands can be used to adjust the size of the selected part.

First, as shown in (d) of Fig. 12, it is assumed that a virtual line is displayed on the side of the T-shirt.

At this time, as shown in (a) of Fig. 13, it is possible for the user (1310) to adjust the size of the virtual clothes by moving two hands (1320) left and right at his/her side.

In response to the user gesture illustrated in (a) of Fig. 13, the XR device displays a feedback graphic image (1330) that enlarges or reduces a specific portion of a virtual garment, as illustrated in (b) of Fig. 13. Furthermore, the XR device is designed to search for actual garments that correspond to the determined garment size. Therefore, one embodiment of the present invention has the unique effect of enabling rapid searching for specific garments that are loose or tight in specific areas, even among similar garments.

The embodiment illustrated in FIG. 13 described previously will be described in more detail with reference to FIG. 14 below, as follows.

It is assumed that the virtual T-shirt (1410) illustrated in (a) of Fig. 14 is selected by voice recognition and displayed on the screen of the XR device. At this time, in response to the first gesture (1420) of the XR device, a virtual garment (1430) with a changed side size of the T-shirt is displayed, as illustrated in (b) of Fig. 14.

And, in response to the second gesture (1440) of the XR device, a virtual garment (1450) with the side size of the T-shirt changed again is displayed as shown in (c) of FIG. 14.

However, there may not be a T-shirt with a side size corresponding to the user gesture. To address this issue, if there is no T-shirt with a side size corresponding to the second gesture (1440) described above, the previous virtual clothing (1430) of FIG. 14 (b) is displayed instead of FIG. 14 (c). Therefore, the user has the advantage of receiving feedback that there is no T-shirt with the corresponding side size.

FIGS. 15 to 17 illustrate a process in which an XR device according to one embodiment of the present invention quickly detects specific virtual clothes using two different interfaces, namely voice recognition and gesture recognition.

As shown in (a) of Fig. 15, when a user says a specific keyword, “add lettering,” the XR device first filters multiple T-shirts containing lettering (letters or numbers, etc.) into a main group through a coarse search.

And, in response to the specific keyword “add lettering”, the XR device no longer runs the microphone and voice recognition engine for voice recognition and switches to motion (gesture) recognition mode.

For example, as shown in (b) of FIG. 15, both hands (1520, 1530) used by the user to designate an area are detected, and only clothes in a subgroup containing lettering in the area composed of the two hands (1520, 1530) are subjected to a fine search again. In particular, not only is there a feature of one embodiment of the present invention in that it converts from a voice recognition mode to a gesture recognition mode for a specific keyword, but it is also within the scope of the present invention to determine whether to recognize only one hand or both hands depending on a specific keyword even in the gesture recognition mode.

Alternatively, a detailed search can be performed only on clothes in a subgroup that contain lettering around the location pointed to by one hand (1540).

Meanwhile, a user who has identified clothes with added lettering according to the embodiments illustrated in FIG. 15 may want to search again for clothes with the lettering removed.

Considering such needs, when a user says “remove filter” (1610) as shown in (a) of FIG. 16, he/she can select one (1630) of the two letterings (1620, 1630) displayed on the virtual clothes and make a gesture (1640) to remove it from the virtual clothes, as shown in (b) of FIG. 16.

The XR device, recognizing this, searches for virtual clothes containing only one letter (1621) and displays them on the screen.

And, an embodiment of displaying the color of virtual clothes in detail will be described below with reference to Fig. 17.

As shown in (a) of Figure 17, if a user says “Change to red” (1710) for a specific piece of clothing, or as shown in (b) of Figure 17, if the user says “Make it a little brighter” (1720), it is virtually impossible for the XR device to accurately recognize the user’s intention. This is because there are many different shades of red.

To solve this problem, as shown in (c) of Fig. 17, when the user says “color detail adjustment” (1730) and the XR device recognizes it, the voice recognition mode is stopped and automatically switched to gesture recognition mode.

However, unlike previous embodiments, an option (1740) for adjusting the color in detail is displayed as an intermediate step in switching from voice recognition mode to gesture recognition mode.

And, by designing to track and recognize only one hand (1741) around the displayed option (1740), there is an advantage in terms of power consumption when tracking both hands.

Meanwhile, the existing drawings describe a process of first sorting virtual clothes in the main group using coarse search, and secondarily sorting virtual clothes in the subgroup using fine search. However, despite this process, there is a high possibility that a large number of virtual clothes will be detected as a result. An embodiment of more efficiently arranging a large number of sorted virtual clothes within the screen of an XR device will be described in the drawings below.

Figure 18 illustrates an inactive area and an active area within a screen defined by an XR device according to one embodiment of the present invention.

As illustrated in FIG. 18, an XR device according to one embodiment of the present invention includes a face area, an inactive area, and an active area. The face area may reflect the face of a user located around the XR device. A camera and a facial recognition algorithm are used at this time.

In response to the user's gesture, the virtual clothing in the active area of the XR device's screen changes. Conversely, the virtual clothing in the inactive area of the XR device's screen does not change in response to the user's gesture. Therefore, the user has the advantage of being able to selectively select the top or bottom while maintaining a full view of the entire scene.

Furthermore, embodiments that allow the user to check the overall look and feel while displaying as many virtual clothes as possible by prioritizing each area in its entirety or in part in each area are described below in FIGS. 19 to 21.

And, FIGS. 19 to 21 illustrate a process of arranging a plurality of virtual clothes based on the area information defined in FIG. 18.

As illustrated in Figure 19, when a user says "jeans" (1910), the XR device recognizes this and displays all filtered virtual jeans without encroaching on the currently active area. However, this design has the drawback of making it difficult to confirm the overall feel of wearing an upper garment (e.g., a T-shirt).

To address this, various virtual jeans are displayed, with both the inactive area (the virtual T-shirt top) and the active area (the jeans bottoms) visible, as shown in the left and middle drawings of Figure 20. Therefore, the user has the advantage of being able to see the overall appearance of the top and bottoms.

However, if the number of virtual jeans found exceeds a threshold value (which changes depending on the screen size of the XR device, the number of virtual clothing images, and their sizes), the entire area starting from the inactive area is filled with virtual jeans, as shown in the right drawing of FIG. 20. In other words, giving priority to the active area over the inactive area is also a feature of one embodiment of the present invention.

Meanwhile, if the screen size of the XR device is large, it will take a lot of time for the user to select the specific virtual clothes he or she wants from among the displayed virtual clothes.

To address this, as illustrated in Fig. 21, the virtual jeans that the user is most likely to select are positioned at a radius (2114) from the center point of the XR screen. The remaining virtual jeans are positioned at other radii (2113, 2112, 2111) from the center point of the XR screen. For reference, the virtual jeans that the user is most likely to select are designed to be closest to the center point of the XR screen.

And, as a method of determining the possibility of user selection, the jeans of the most frequently purchased brands are placed in a radius (2114) by referring to the previous purchase history through user facial recognition.

110: Communication unit
120: Control unit
130: Memory unit
140a: I/O unit
140b: Sensor unit
140c: Power supply unit

Claims (15)

In a method for controlling an XR device that provides a virtual fitting service,
A step of storing information of at least one virtual garment in memory;
A step of receiving at least one voice command from a user around the XR device;
A step of extracting information on clothes belonging to a specific main group from information on at least one virtual clothes stored in the memory based on at least one voice command received;
A step of recognizing the user's gesture;
A step of extracting information on clothes belonging to a sub-group among clothes belonging to the specific main group based on the recognized gesture; and
A step of classifying the screen of the XR device into an active area and an inactive area based on the information of the extracted clothes,
Based on the information of the extracted clothes, the step of classifying the screen of the XR device into an active area and an inactive area is as follows:
A control method for an XR device that determines the possibility of the user selecting clothes and places clothes that the user is likely to select closest to the center of the screen among the inactive areas. In the first paragraph,
A method for controlling an XR device, wherein the at least one voice command comprises a first voice command or a second voice command. In the second paragraph,
A step of changing the operation of the voice recognition engine stored in the memory to a disenable state when the first voice command or the second voice command is received
A method for controlling an XR device, characterized in that it further includes: In the third paragraph,
A step of recognizing both hands of the user when the first voice command is received;
A step of searching for clothes containing any letters within an area formed by the first hand and the second hand; and
A step of displaying at least one of the clothes containing the above arbitrary letters.
A method for controlling an XR device, characterized in that it further includes: In paragraph 4,
When the second voice command is received, a step of displaying an option for adjusting the color of the clothes in detail;
A step of recognizing only one hand of the user within a certain distance from the location of the displayed option;
A step of searching for clothes of a specific color based on the recognized motion of one hand; and
A step of displaying at least one of the clothes of the above specific color
A method for controlling an XR device, characterized in that it further includes: delete In the first paragraph,
Only the virtual clothes in the active area within the above screen are changed in response to the user's gesture,
A method for controlling an XR device, characterized in that the virtual clothes in the inactive area of the screen do not change in response to the user's gesture. For XR devices that provide virtual fitting services,
A memory storing information about at least one virtual garment;
A microphone for receiving at least one voice command from a user surrounding the XR device;
A controller that extracts information on clothes belonging to a specific main group from information on at least one virtual clothes stored in the memory based on at least one voice command received; and
Including a camera that recognizes the user's gestures,
The above controller,
Based on the recognized gesture, information on clothes belonging to a subgroup is extracted from among clothes belonging to the specific main group, and based on the information on the extracted clothes, the screen of the XR device is classified into an active area and an inactive area.
An XR device that determines the possibility of the user selecting clothes and places clothes that the user is likely to select closest to the center of the screen among the inactive areas. In paragraph 8,
An XR device, wherein the at least one voice command comprises a first voice command or a second voice command. In paragraph 9,
The above controller,
An XR device characterized in that, when the first voice command or the second voice command is received, the voice recognition engine stored in the memory changes its operation to a disenable state. In Article 10,
The above controller,
When the above first voice command is received, both hands of the user are recognized,
Search for clothes containing any letters within the area formed by the first hand and the second hand, and
An XR device characterized by controlling to display at least one of the clothes containing the above arbitrary letters. In Article 11,
The above controller,
When the second voice command is received, display options for adjusting the color of the clothes in detail;
Recognize only one hand of the user within a certain distance from the location of the displayed option,
Based on the recognized motion of one hand, clothes of a specific color are searched, and
An XR device characterized by controlling to display at least one of the clothes of the above specific color. delete In paragraph 8,
Only the virtual clothes in the active area within the above screen are changed in response to the user's gesture,
An XR device characterized in that the virtual clothes in the inactive area of the screen do not change in response to the user's gesture. In paragraph 8,
The above XR device,
An XR device characterized by a styler product, signage, mobile device or TV.