WO2026023964A1 - Dispositif à porter sur soi, procédé et support de stockage non transitoire lisible par ordinateur pour traiter une entrée de simulation - Google Patents

Dispositif à porter sur soi, procédé et support de stockage non transitoire lisible par ordinateur pour traiter une entrée de simulation

Info

Publication number
WO2026023964A1
WO2026023964A1 PCT/KR2025/010220 KR2025010220W WO2026023964A1 WO 2026023964 A1 WO2026023964 A1 WO 2026023964A1 KR 2025010220 W KR2025010220 W KR 2025010220W WO 2026023964 A1 WO2026023964 A1 WO 2026023964A1
Authority
WO
WIPO (PCT)
Prior art keywords
input
electronic device
simulation
user
tracking data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/KR2025/010220
Other languages
English (en)
Korean (ko)
Inventor
정준식
김보성
김지호
이은주
임동건
정원영
김성오
염동현
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR1020240155718A external-priority patent/KR20260014444A/ko
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of WO2026023964A1 publication Critical patent/WO2026023964A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Definitions

  • the following descriptions relate to a wearable device, a method, and a non-transitory computer-readable recording medium for processing simulation input.
  • the electronic devices are being developed that provide augmented reality (AR) and/or virtual reality (VR) services that display computer-generated information in conjunction with external objects in the real world.
  • the electronic devices may be wearable devices that can be worn by a user.
  • the electronic devices may be AR glasses and/or head-mounted devices (HMDs).
  • the wearable device may include a display assembly including displays respectively arranged toward the two eyes of a user when worn, an input circuit configured to acquire a specified type of tracking data of the user, at least one processor including a processing circuit, and a memory storing instructions, the memory including one or more storage media.
  • the instructions when individually or collectively executed by the at least one processor, may cause the wearable device to receive a simulated input of the specified type based on a first center point of a field of view (FoV) within a three-dimensional virtual space while at least partially limiting the execution of a function corresponding to the input based on the specified type of tracking data.
  • FoV field of view
  • the instructions when individually or collectively executed by the at least one processor, may cause the wearable device to identify the specified type of input at a second location adjusted from the first location where the simulated input was input, based on a difference between the first center point and a second center point of another FoV based on the pose of the user.
  • the instructions when individually or collectively executed by the at least one processor, may cause the wearable device to perform the function corresponding to the input identified at the second location based on the simulated input.
  • a method is disclosed.
  • the method can be performed on a wearable device comprising a display assembly including displays respectively arranged to face the two eyes of a user when worn, and an input circuit configured to receive a specified type of input from the user.
  • the method can include: receiving a simulated input of the specified type based on a first center point of a FoV within a three-dimensional virtual space while at least partially limiting the execution of a function corresponding to the input based on the tracking data of the specified type.
  • the method can include identifying the specified type of input at a second location adjusted from a first location where the simulated input was input, based on a difference between the first center point and a second center point of another FoV based on a pose of the user.
  • the method can include performing the function corresponding to the input identified at the second location based on the simulated input.
  • a non-transitory computer-readable storage medium can store a program including instructions.
  • the instructions when individually or collectively executed by at least one processor of a wearable device, the wearable device including a display assembly including displays respectively arranged toward two eyes of a user when worn, and an input circuit configured to obtain a specified type of tracking data of the user, cause the wearable device to receive a simulation input of the specified type based on a first center point of a FoV within a three-dimensional virtual space while at least partially limiting the execution of a function corresponding to the input based on the specified type of tracking data.
  • the instructions when individually or collectively executed by the at least one processor, may cause the wearable device to identify the specified type of input at a second location adjusted from the first location where the simulated input was input, based on a difference between the first center point and a second center point of another FoV based on the pose of the user.
  • the instructions when individually or collectively executed by the at least one processor, may cause the wearable device to perform the function corresponding to the input identified at the second location based on the simulated input.
  • FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.
  • FIG. 2A illustrates an example of a perspective view of a wearable device, according to one embodiment.
  • FIG. 2b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.
  • FIG. 3A illustrates an example of an appearance of a wearable device according to one embodiment.
  • FIG. 3b illustrates an example of an appearance of a wearable device according to one embodiment.
  • FIG. 4 illustrates an example block diagram of an electronic device according to one embodiment.
  • Figure 5 illustrates an example of a structure of multiple layers according to one embodiment.
  • FIG. 6A illustrates an example of a structure in which an electronic device according to one embodiment receives simulation input entered through a client.
  • FIG. 6b illustrates an example of a structure in which an electronic device according to one embodiment receives a simulation input entered through an input device.
  • FIG. 7 illustrates an example structure in which an electronic device according to one embodiment receives simulation input through a tester application.
  • FIG. 8 illustrates an example of a structure for evaluating the performance of a tester application by an electronic device according to one embodiment.
  • FIG. 9 illustrates an example of a situation in which an electronic device according to one embodiment switches between a simulation mode and a normal mode.
  • FIG. 10A illustrates an example of a situation in which an electronic device according to one embodiment processes input in a simulation mode and a normal mode.
  • Figure 10b illustrates examples of simulation input and actual input according to one embodiment.
  • FIG. 11 illustrates an example of a situation in which an electronic device according to one embodiment records input in normal mode and then inputs it to a simulation input connection.
  • Figure 12 illustrates an example of a situation in which a record file is processed according to one embodiment.
  • FIG. 13A illustrates an example of an electronic device according to one embodiment of the present invention that identifies a user's gaze within a virtual space based on the position of the user's eyes.
  • FIG. 13b illustrates an example of an electronic device according to an embodiment of the present invention showing an operation of identifying a gaze within a virtual space based on the position of a user's eyes in spaces.
  • Figure 14 illustrates an example of a 2D image according to one embodiment.
  • FIG. 15 illustrates an example of an electronic device converting coordinates within a 2D image into coordinates within a 3D virtual space according to an embodiment.
  • FIG. 16 illustrates an example of a field of view (FoV) within a 3D virtual space shown to a user according to one embodiment.
  • FoV field of view
  • FIG. 17 illustrates an example of an electronic device according to one embodiment changing the FoV within a 3D virtual space displayed to a user based on simulation input.
  • FIG. 18 illustrates an example of an electronic device synchronizing between simulation input and general input according to one embodiment.
  • Figure 19 is a flowchart illustrating the operation of an electronic device according to one embodiment.
  • FIG. 20A illustrates an example of an electronic device according to one embodiment of the present invention that identifies a user's gaze within a virtual space based on the position of the user's eyes.
  • FIG. 20b illustrates an example of an electronic device according to an embodiment of the present invention that identifies a user's gaze within a virtual space based on the position of the user's eyes.
  • FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.
  • an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network).
  • the electronic device (101) may communicate with the electronic device (104) via the server (108).
  • the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197).
  • the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added.
  • some of these components e.g., the sensor module (176), the camera module (180), or the antenna module (197) may be integrated into one component (e.g., the display module (160)).
  • the processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations.
  • the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134).
  • the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121).
  • a main processor (121) e.g., a central processing unit or an application processor
  • an auxiliary processor (123) e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor
  • the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function.
  • the auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.
  • the auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state.
  • the auxiliary processor (123) e.g., an image signal processor or a communication processor
  • the auxiliary processor (123) may include a hardware structure specialized for processing artificial intelligence models.
  • the artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)).
  • the learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above.
  • the artificial intelligence model can include a plurality of artificial neural network layers.
  • the artificial neural network can be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above.
  • the artificial intelligence model can additionally or alternatively include a software structure.
  • the memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101).
  • the data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto.
  • the memory (130) can include volatile memory (132) or non-volatile memory (134).
  • the program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).
  • the input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101).
  • the input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).
  • the audio output module (155) can output audio signals to the outside of the electronic device (101).
  • the audio output module (155) can include, for example, a speaker or a receiver.
  • the speaker can be used for general purposes, such as multimedia playback or recording playback.
  • the receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.
  • the display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101).
  • the display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device.
  • the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.
  • the audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).
  • an external electronic device e.g., electronic device (102)
  • speaker or headphone directly or wirelessly connected to the electronic device (101).
  • the sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status.
  • the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
  • the interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)).
  • the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
  • HDMI high definition multimedia interface
  • USB universal serial bus
  • SD card interface Secure Digital Card
  • connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)).
  • the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
  • the haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations.
  • the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
  • the camera module (180) can capture still images and videos.
  • the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.
  • the power management module (188) can manage power supplied to the electronic device (101).
  • the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).
  • PMIC power management integrated circuit
  • a battery (189) may power at least one component of the electronic device (101).
  • the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
  • the communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel.
  • the communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication.
  • the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module).
  • a wireless communication module (192) e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module
  • GNSS global navigation satellite system
  • wired communication module (194) e.g., a local area network (LAN) communication module, or a power line communication module.
  • the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)).
  • a first network (198) e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)
  • a second network (199) e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)
  • a computer network e.g., a
  • the wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).
  • subscriber information e.g., an international mobile subscriber identity (IMSI)
  • the wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology).
  • the NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)).
  • eMBB enhanced mobile broadband
  • mMTC massive machine type communications
  • URLLC ultra-reliable and low-latency communications
  • the wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate.
  • a high-frequency band e.g., mmWave band
  • the wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna.
  • the wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)).
  • the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, a loss coverage (e.g., 664 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for realizing URLLC.
  • a peak data rate e.g., 20 Gbps or more
  • a loss coverage e.g., 664 dB or less
  • U-plane latency e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip
  • the antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device).
  • the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB).
  • the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna.
  • another component e.g., a radio frequency integrated circuit (RFIC)
  • RFIC radio frequency integrated circuit
  • the antenna module (197) may form a mmWave antenna module.
  • the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.
  • a first side e.g., a bottom side
  • a plurality of antennas e.g., an array antenna
  • At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).
  • peripheral devices e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).
  • commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199).
  • Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101).
  • all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service.
  • One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101).
  • the electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request.
  • cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example.
  • the electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example.
  • the external electronic device (104) may include an Internet of Things (IoT) device.
  • the server (108) may be an intelligent server utilizing machine learning and/or a neural network.
  • the external electronic device (104) or the server (108) may be included in the second network (199).
  • the electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.
  • FIG. 2A illustrates an example of a perspective view of a wearable device (200), according to one embodiment.
  • FIG. 2B illustrates an example of one or more hardware elements disposed within a wearable device (200), according to one embodiment.
  • the wearable device (200) of FIGS. 2A and 2B may correspond to the electronic device (101) of FIG. 1.
  • the wearable device (200) may include at least one display (250) and a frame supporting the at least one display (250).
  • the wearable device (200) can be worn on a part of a user's body.
  • the wearable device (200) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (200).
  • AR augmented reality
  • VR virtual reality
  • MR mixed reality
  • the wearable device (200) can output a virtual reality image to the user through at least one display (250) in response to a designated gesture of the user acquired through the motion recognition camera (240-2) of FIG. 2B.
  • At least one display (250) within a wearable device (200) may provide visual information to a user.
  • at least one display (250) may include a transparent or translucent lens.
  • At least one display (250) may include a first display (250-1) and/or a second display (250-2) spaced apart from the first display (250-1).
  • the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.
  • At least one display (250) may form a display area on a lens to provide a user wearing the wearable device (200) with visual information contained in external light passing through the lens, along with other visual information distinct from the visual information.
  • the lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens.
  • the display area formed by at least one display (250) may be formed on the second surface (232) among the first surface (231) and the second surface (232) of the lens.
  • At least one display (250) may display a virtual reality image to be combined with a real screen transmitted through the external light.
  • the virtual reality image output from at least one display (250) can be transmitted to the user's eyes through one or more hardware (e.g., optical devices (282, 284), and/or at least one waveguide (233, 234)) included in the wearable device (200).
  • a wearable device (200) may include waveguides (233, 234) that diffract light transmitted from at least one display (250) and relayed by optical devices (282, 284) and transmit the diffracted light to a user.
  • the waveguides (233, 234) may be formed based on at least one of glass, plastic, or polymer.
  • Nanopatterns may be formed on at least a portion of the exterior or interior of the waveguides (233, 234).
  • the nanopatterns may be formed based on a grating structure having a polygonal and/or curved shape. Light incident on one end of the waveguides (233, 234) may be propagated to the other end of the waveguides (233, 234) by the nanopatterns.
  • the waveguides (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror).
  • DOE diffractive optical element
  • HOE holographic optical element
  • the waveguides (233, 234) may be arranged within the wearable device (200) to guide a screen displayed by at least one display (250) to the user's eyes.
  • the screen may be transmitted to the user's eyes based on total internal reflection (TIR) occurring within the waveguides (233, 234).
  • TIR total internal reflection
  • the wearable device (200) may analyze an object included in a real image collected through a capturing camera (240-1), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250).
  • the virtual object may include at least one of text and an image regarding various information related to the object included in the real image.
  • the wearable device (200) may analyze the object based on a multi-camera, such as a stereo camera.
  • the wearable device (200) may execute time-of-flight (ToF) and/or simultaneous localization and mapping (SLAM) supported by the multi-camera.
  • a user wearing the wearable device (200) may view an image displayed on at least one display (250).
  • the frame may be formed as a physical structure that allows the wearable device (200) to be worn on the user's body.
  • the frame may be configured so that, when the user wears the wearable device (200), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes.
  • the frame may support at least one display (250).
  • the frame may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.
  • the frame may include a region (220) that at least partially contacts a part of the user's body when the user wears the wearable device (200).
  • the region (220) of the frame that contacts a part of the user's body may include a region that contacts a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (200) makes contact with.
  • the frame may include a nose pad (210) that contacts a part of the user's body. When the wearable device (200) is worn by the user, the nose pad (210) may contact a part of the user's nose.
  • the frame may include a first temple (204) and a second temple (205) that contact a part of the user's body that is distinct from the part of the user's body.
  • the frame may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of an ear of the wearer, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of an ear opposite the ear.
  • a first rim (201) that surrounds at least a portion of the first display (250-1)
  • a second rim (202) that surrounds at least a portion of the second display
  • the first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear.
  • the temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B.
  • the first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204).
  • the second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205).
  • the wearable device (200) can identify an external object (e.g., a user's fingertip) touching the frame and/or a gesture performed by the external object by using a touch sensor, a grip sensor, and/or a proximity sensor formed on at least a portion of a surface of the frame.
  • an external object e.g., a user's fingertip
  • the wearable device (200) may include hardwares that perform various functions (e.g., hardwares described above based on the block diagram of FIG. 1).
  • the hardwares may include a battery module (270), an antenna module (275), optical devices (282, 284), speakers (292-1, 292-2), microphones (294-1, 294-2, 294-3), a light-emitting module (not shown), and/or a printed circuit board (290).
  • the various hardwares may be arranged within a frame.
  • the microphones (294-1, 294-2, 294-3) of the wearable device (200) may be arranged on at least a portion of the frame to acquire a sound signal.
  • a first microphone (294-1) arranged on the nose pad (210), a second microphone (294-2) arranged on the second rim (202), and a third microphone (294-3) arranged on the first rim (201) are illustrated in FIG. 2B, but the number and arrangement of the microphones (294) are not limited to the embodiment of FIG. 2B.
  • the wearable device (200) may identify the direction of the sound signal by using a plurality of microphones arranged on different portions of the frame.
  • the optical devices (282, 284) can transmit a virtual object transmitted from at least one display (250) to the wave guides (233, 234).
  • the optical devices (282, 284) can be projectors.
  • the optical devices (282, 284) can be positioned adjacent to at least one display (250) or can be included within at least one display (250) as a part of the at least one display (250).
  • the first optical device (282) can correspond to the first display (250-1), and the second optical device (284) can correspond to the second display (250-2).
  • the first optical device (282) can transmit light output from the first display (250-1) to the first waveguide (233), and the second optical device (284) can transmit light output from the second display (250-2) to the second waveguide (234).
  • the camera (240) may include an eye tracking camera (ET CAM) (240-1), a motion recognition camera (240-2), and/or a recording camera (240-3).
  • the recording camera, the eye tracking camera (240-1), and the motion recognition camera (240-2) may be positioned at different locations on the frame and may perform different functions.
  • the eye tracking camera (240-1) may output data representing the gaze of a user wearing the wearable device (200).
  • the wearable device (200) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (240-1).
  • An example in which the gaze tracking camera (240-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (240-1) may be positioned solely toward the user's left eye, or may be positioned toward both eyes.
  • the capturing camera (240-3) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content.
  • the capturing camera can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250).
  • the at least one display (250) can display a single image in which information about the actual image or background including the image of the specific object acquired using the capturing camera is superimposed on a virtual image provided through optical devices (282, 284).
  • the capturing camera can be placed on a bridge (203) disposed between the first rim (201) and the second rim (202).
  • the gaze tracking camera (240-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (200) and thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (200) looks straight ahead, the wearable device (200) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250).
  • the gaze tracking camera (240-1) can be configured to capture an image of the user's pupil to determine the user's gaze.
  • the gaze tracking camera (240-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light.
  • the gaze tracking camera (240-1) can be positioned at positions corresponding to the user's left and right eyes.
  • the gaze tracking camera (240-1) may be positioned within the first rim (201) and/or the second rim (202) to face the direction in which the user wearing the wearable device (200) is positioned.
  • the gesture recognition camera (240-2) can provide a specific event on a screen provided on at least one display (250) by recognizing the movement of the user's entire body or a part of the user's body, such as the user's torso, hands, or face.
  • the gesture recognition camera (240-2) can recognize the user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (250).
  • the processor can identify the signal corresponding to the gesture and perform a designated function based on the identification.
  • the gesture recognition camera (240-2) can be disposed on the first rim (201) and/or the second rim (202).
  • the camera (240) included in the wearable device (200) is not limited to the gaze tracking camera (240-1) and the motion recognition camera (240-2) described above.
  • the wearable device (200) can identify an external object included in the FOV using a photographing camera (240-3) positioned toward the user's FOV.
  • the wearable device (200) can identify an external object based on a sensor for identifying the distance between the wearable device (200) and the external object, such as a depth sensor and/or a time of flight (ToF) sensor.
  • the camera (240) positioned toward the FOV can support an autofocus function and/or an optical image stabilization (OIS) function.
  • the wearable device (200) may include a camera (240) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (200).
  • FT face tracking
  • the wearable device (200) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and/or an external object within the FOV) being photographed using the camera (240).
  • the light source may include an infrared wavelength LED.
  • the light source may be disposed on at least one of the frame and hinge units (206, 207).
  • the battery module (270) may supply power to electronic components of the wearable device (200).
  • the battery module (270) may be disposed within the first temple (204) and/or the second temple (205).
  • the battery module (270) may be a plurality of battery modules (270).
  • the plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205).
  • the battery module (270) may be disposed at an end of the first temple (204) and/or the second temple (205).
  • the antenna module (275) can transmit signals or power to the outside of the wearable device (200), or receive signals or power from the outside.
  • the antenna module (275) can be electrically and/or operatively connected to the communication module (190) of FIG. 1.
  • the antenna module (275) can be positioned within the first temple (204) and/or the second temple (205).
  • the antenna module (275) can be positioned close to one surface of the first temple (204) and/or the second temple (205).
  • the speakers (292-1, 292-2) may output audio signals to the outside of the wearable device (200).
  • the audio output module may be referred to as a speaker.
  • the speakers (292-1, 292-2) may be positioned within the first temple (204) and/or the second temple (205) so as to be positioned adjacent to the ears of a user wearing the wearable device (200).
  • the wearable device (200) may include a second speaker (292-2) positioned within the first temple (204) and thus adjacent to the user's left ear, and a first speaker (292-1) positioned within the second temple (205) and thus adjacent to the user's right ear.
  • the light-emitting module may include at least one light-emitting element.
  • the light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state to visually provide information regarding a specific state of the wearable device (200) to the user. For example, when the wearable device (200) requires charging, the wearable device (200) may repeatedly emit red light at a designated time.
  • the light-emitting module may be disposed on the first rim (201) and/or the second rim (202).
  • a wearable device (200) may include a printed circuit board (PCB) (290).
  • the PCB (290) may be included in at least one of the first temple (204) and the second temple (205).
  • the PCB (290) may include an interposer disposed between at least two sub-PCBs.
  • One or more hardware components included in the wearable device (200) may be disposed on the PCB (290).
  • the wearable device (200) may include a flexible PCB (FPCB) for interconnecting the hardware components.
  • a wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and/or an acceleration sensor for detecting a posture of the wearable device (200) and/or a posture of a body part (e.g., a head) of a user wearing the wearable device (200).
  • Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and/or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis).
  • the gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU).
  • IMU inertial measurement unit
  • the wearable device (200) may identify a user's motion and/or gesture performed to execute or terminate a specific function of the wearable device (200) based on the IMU.
  • FIGS. 3A and 3B illustrate an example of an exterior appearance of a wearable device (300) according to one embodiment.
  • the wearable device (300) of FIGS. 3A and 3B may be included in the electronic device (101) of FIG. 1.
  • An example of an exterior appearance of a first side (310) of a housing of the wearable device (300) according to one embodiment is illustrated in FIG. 3A, and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B.
  • a first surface (310) of a wearable device (300) may have a form attachable to a body part of a user (e.g., the face of the user).
  • the wearable device (300) may further include a strap for fixing to a body part of a user, and/or one or more temples (e.g., the first temple (204) and/or the second temple (205) of FIGS. 2A and 2B).
  • a first display (350-1) for outputting an image to a left eye among the user's two eyes, and a second display (350-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310).
  • the wearable device (300) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (350-1) and the second display (350-2).
  • the wearable device (300) may include cameras (340-1, 340-2) for photographing and/or tracking the user's two eyes adjacent to the first display (350-1) and the second display (350-2), respectively.
  • the cameras (340-1, 340-2) may be referred to as ET cameras.
  • the wearable device (300) may include cameras (340-3, 340-4) for photographing and/or recognizing the user's face.
  • the cameras (340-3, 340-4) may be referred to as FT cameras.
  • a camera e.g., cameras (340-5, 340-6, 340-7, 340-8, 340-9, 340-10)
  • a sensor e.g., depth sensor (330)
  • the cameras may be disposed on the second surface (320) to recognize an external object different from the wearable device (300).
  • the wearable device (300) can obtain images and/or videos to be transmitted to each of the user's eyes.
  • the camera (340-9) can be placed on the second face (320) of the wearable device (300) to obtain an image to be displayed through the second display (350-2) corresponding to the right eye among the two eyes.
  • the camera (340-10) can be placed on the second face (320) of the wearable device (300) to obtain an image to be displayed through the first display (350-1) corresponding to the left eye among the two eyes.
  • the wearable device (300) may include a depth sensor (330) disposed on the second surface (320) to identify a distance between the wearable device (300) and an external object. Using the depth sensor (330), the wearable device (300) may obtain spatial information (e.g., a depth map) for at least a portion of the FOV of a user wearing the wearable device (300).
  • a depth sensor 330
  • the wearable device (300) may obtain spatial information (e.g., a depth map) for at least a portion of the FOV of a user wearing the wearable device (300).
  • a microphone may be placed on the second side (320) of the wearable device (300) to acquire sound output from an external object.
  • the number of microphones may be one or more depending on the embodiment.
  • the wearable device (300) may have a form factor for being worn on a user's head.
  • the wearable device (300), while worn on the head, may provide a user experience based on augmented reality, virtual reality, and/or mixed reality.
  • the wearable device (300) and a server connected to the wearable device (300) may provide an on-demand service that provides video of a location and/or place selected by the user, and/or a metaverse service.
  • the wearable device (300) can display frames acquired through cameras (340-9, 340-10) on the first display (350-1) and the second display (350-2), respectively.
  • the wearable device (300) can provide a user with a mixed user experience of real objects and virtual objects (e.g., video see-through (VST)) by combining virtual objects within frames displayed through the first display (350-1) and the second display (350-2) and including real objects.
  • the wearable device (300) can change the virtual objects based on information acquired by the cameras (340-1, 340-2, 340-3, 340-4, 340-5, 340-6, 340-7, 340-8) and/or the depth sensor (330).
  • the wearable device (300) may stop displaying the virtual object based on detecting a motion for interacting with the real object. By stopping displaying the virtual object, the wearable device (300) may prevent the visibility of the real object from being reduced as the visual object corresponding to the real object is occluded by the virtual object.
  • FIG. 4 illustrates an example of a block diagram of an electronic device according to an embodiment.
  • the electronic device (400) of FIG. 4 may be an example of the electronic device (101) of FIG. 1, the wearable device (200) of FIGS. 2A and 2B, and the wearable device (300) of FIGS. 3A and 3B.
  • the electronic device (400) may include at least one of a processor (410), a memory (415), a display (420), a camera (425), a sensor (430), or a communication circuit (440).
  • the processor (410), the memory (415), the display (420), the camera (425), the sensor (430), and the communication circuit (440) may be electrically and/or operatively connected to each other by electronic components such as a communication bus (402).
  • operatively coupled hardware may mean that a direct connection or an indirect connection is established between the hardwares, either wired or wireless, such that a second hardware is controlled by a first hardware among the hardwares.
  • the electronic device (400) may include only some of the hardware components illustrated in FIG. 4.
  • the processor (410) of the electronic device (400) may include hardware for processing data based on one or more instructions.
  • the hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and/or an application processor (AP).
  • ALU arithmetic and logic unit
  • FPU floating point unit
  • FPGA field programmable gate array
  • CPU central processing unit
  • AP application processor
  • the processor (410) may have a single-core processor structure, or a multi-core processor structure such as a dual core, a quad core, or a hexa core.
  • the memory (415) of the electronic device (400) may include hardware components for storing data and/or instructions input and/or output to the processor (410) of the electronic device (400).
  • the memory (415) may include, for example, volatile memory such as random-access memory (RAM) and/or non-volatile memory such as read-only memory (ROM).
  • RAM random-access memory
  • ROM read-only memory
  • the volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM).
  • DRAM dynamic RAM
  • SRAM static RAM
  • PSRAM pseudo SRAM
  • the non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multi media card (eMMC).
  • PROM programmable ROM
  • EPROM erasable PROM
  • EEPROM electrically erasable PROM
  • flash memory hard disk, compact disc, solid state drive (SSD), and embedded multi media card (eMMC).
  • SSD solid state drive
  • eMMC embedded multi media card
  • a display (420) of an electronic device (400) can output visualized information to a user of the electronic device (400).
  • the display (420) can be controlled by a processor (410) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user.
  • the display (420) can include a flat panel display (FPD) and/or electronic paper.
  • the FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and/or one or more light emitting diodes (LEDs).
  • the LEDs can include organic LEDs (OLEDs).
  • the camera (425) of the electronic device (400) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and/or brightness of light.
  • the plurality of optical sensors included in the camera (425) may be arranged in the form of a two-dimensional array.
  • the camera (425) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array.
  • photographic data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425).
  • video data captured using the camera (425) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (425) according to a frame rate.
  • the camera (425) may further include a flash light that is positioned toward the direction in which the camera (425) receives light and outputs light toward the direction.
  • the electronic device (400) may include, as an example of a camera (425), a first camera (425-1) and a second camera (425-2) positioned facing different directions.
  • the first camera (425-1) may be referred to as a motion recognition camera
  • the second camera (425-2) may be referred to as an eye tracking camera.
  • the electronic device (400) may identify a position, shape, and/or gesture of a hand using an image (426-1) of the first camera (425-1).
  • the electronic device (400) may identify a direction of a gaze of a user wearing the electronic device (400) using an image (426-2) of the second camera (425-2).
  • a sensor (430) of an electronic device (400) may generate electrical information that may be processed by a processor (410) and/or a memory (415) of the electronic device (400) from non-electronic information related to the electronic device (400).
  • the information may be referred to as sensor data.
  • the sensor (430) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and/or a time-of-flight (ToF) sensor for detecting a geographic location of the electronic device (400), and an inertial measurement unit (IMU) for detecting a physical motion of the electronic device (400).
  • GPS global positioning system
  • ToF time-of-flight
  • IMU inertial measurement unit
  • the communication circuit (440) of the electronic device (400) may include hardware components for supporting transmission and/or reception of electrical signals between the electronic device (400) and an external electronic device (e.g., a remote controller for controlling the electronic device (400).
  • the communication circuit (440) may include, for example, at least one of a modem (MODEM), an antenna, and an optical/electronic (O/E) converter.
  • the communication circuit (440) may support transmission and/or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and/or 6G.
  • the electronic device (400) may include output means for outputting information in a form other than a visualized form.
  • the electronic device (400) may include a speaker for outputting an acoustic signal.
  • the electronic device (400) may include a motor for providing haptic feedback based on vibration.
  • one or more instructions (or commands) representing operations and/or actions to be performed on data by the processor (410) of the electronic device (400) may be stored in the memory (415) of the electronic device (400).
  • a set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and/or an application.
  • an application when installed in the electronic device (e.g., the electronic device (400)), it may mean that one or more instructions provided in the form of an application are stored in the memory (415), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the electronic device (400)).
  • the electronic device (400) may execute one or more instructions stored in the memory (415) to perform operations of the electronic device (400) described below.
  • programs installed in the electronic device (400) may be classified into one of different layers, including an application layer (460), a framework layer (470), and/or a hardware abstraction layer (HAL) (450), based on the target.
  • an application layer 460
  • a framework layer 470
  • HAL hardware abstraction layer
  • programs e.g., drivers
  • the hardware of the electronic device (400) e.g., display (420), camera (425), sensor (430), and/or communication circuit (440)
  • programs e.g., eye tracker (471), gesture tracker (472), motion tracker (473), and/or position tracker (474)
  • Programs classified into the framework layer (470) may provide an executable API (application programming interface) based on other programs.
  • programs designed to target users controlling electronic devices (400) may be classified within the application layer (460).
  • programs classified within the application layer (460) may include at least one of an application for playing and/or streaming video (461), an application for video conferencing (462), an application for viewing media content (e.g., images and/or videos) in memory (415) (463), or an application for call connection (464).
  • the embodiment is not limited thereto.
  • programs classified within the application layer (460) may call APIs to cause execution of functions supported by programs classified within the framework layer (470).
  • the electronic device (400) may display one or more visual objects on the display (420) for performing interaction with a user based on the execution of an application (461, 462, 463, 464).
  • a visual object may refer to an object that can be placed within a screen for transmitting and/or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and/or a table.
  • a visual object may be referred to as a visual guide, a virtual object, a visual element, a user interface (UI) element, a view object, and/or a view element.
  • UI user interface
  • the applications (461, 462, 463, 464) may include an XR application for providing an extended reality (XR) service and/or a general application for providing a general 2D screen.
  • the XR application may be configured to output images with binocular parallax to display the application's screen in a 3D virtual space.
  • the general application may be configured to output a single 2D image.
  • the framework layer (470) may be referred to as an XR framework layer in the sense that it includes one or more programs for providing an XR service.
  • the framework layer (470) may include programs designed to target at least one of the hardware abstraction layer (450) and/or the application layer (460), such as an eye-gaze tracker (471), a gesture recognizer (472), a motion tracker (473), and/or an external space recognizer (474).
  • the programs included in the framework layer (470) may provide an API that is executable (or callable) based on other programs.
  • the framework layer (470) may include a platform layer (e.g., 510 of FIG. 5), a recognition service layer (e.g., 520 of FIG. 5), and/or a sensor service layer (e.g., 530 of FIG. 5).
  • the platform layer (510) may include a platform for supporting a virtual space service (e.g., a composition presentation manager (CPM) (730) of FIG. 7).
  • the platform layer (510) may include a platform for performing processing for displaying an image of a general application within a three-dimensional virtual space (e.g., a space flinger (SpF) (730) of FIG. 7).
  • the electronic device (400) may process information related to the gaze of a user wearing the electronic device (400) based on the execution of the gaze tracker (471) within the framework layer (470). For example, the electronic device (400) may obtain an image (426-2) including the user's eyes from the second camera (425-2). Based on the position and/or direction of the pupil included in the image (426-2), the electronic device (400) may identify the direction of the user's gaze.
  • the gaze tracker (471) may be referred to as an eye tracking (ET) module and/or a gaze tracking module.
  • the electronic device (400) may identify a motion of a designated body part including a hand based on the execution of a gesture tracker (472) within the framework layer (470). For example, the electronic device (400) may obtain an image (426-1) and/or a video including the body part from a first camera (425-1). Based on the motion and/or posture of the designated body part as represented by the image (426-1) and/or the video, the electronic device (400) may identify a gesture performed by the designated body part.
  • the electronic device (400) may determine that a pinch gesture is performed by the hand.
  • the gesture tracker (472) may be referred to as a hand tracking (HaT) module and/or a gesture tracking module.
  • the electronic device (400) can identify the motion of the electronic device (400) based on the execution of the motion tracker (473) within the framework layer (470).
  • the motion of the electronic device (400) can be related to the motion of the user's head.
  • the electronic device (400) can identify an orientation of the electronic device (400) that substantially matches the orientation of the head.
  • the electronic device (400) can identify the motion of the electronic device (400) based on sensor data of a sensor (430) including an IMU.
  • the electronic device (400) can identify the pose of the electronic device (400) using the sensor (430) based on the execution of the position tracker (474).
  • the electronic device (400) can identify the 6 degrees of freedom pose (6 DOF pose) of the electronic device (400) using data acquired using the camera (425) and the IMU based on the execution of the position tracker (474).
  • the position tracker (474) may be referred to as a head tracking (HeT) module.
  • the electronic device (400) may further include a program (e.g., an external space recognizer) designed to target the hardware of the electronic device (400) within the framework layer (470).
  • a program e.g., an external space recognizer
  • the electronic device (400) may obtain information about an external space in which the electronic device (400) is included or adjacent to the electronic device (400) based on the execution of the external space recognizer.
  • the electronic device (400) may obtain the information using a camera (425) and/or a sensor (430).
  • the electronic device (400) may identify a virtual space mapped to the external space based on the information acquired based on the external space recognizer.
  • the electronic device (400) may identify a position and/or direction of the electronic device (400) within the external space. For example, based on the execution of an external space recognizer and/or a motion tracker (473), the electronic device (400) may perform simultaneous localization and mapping (SLAM) to recognize the external space and the location of the electronic device (400) within the external space.
  • the space recognizer may be referred to as a scene understanding (SU) module.
  • FIG. 5 A specific example of a layer structure for an XR service including the above-described application layer (460), hardware abstraction layer (450), and framework layer (460) will be described in FIG. 5.
  • the layers for the XR service may be operated by the processor (410) of the electronic device (400).
  • the modules (or components) included in the layers for the XR service will be described as performing operations.
  • the elements (e.g., layers and/or modules) in the memory described below may be in a logically separated state.
  • the present invention is not limited thereto.
  • Figure 5 illustrates an example of a structure of multiple layers according to one embodiment.
  • programs installed in the electronic device (400) can be classified into one of the platform layer (510), the recognition layer (520), and the sensor service layer (530).
  • the electronic device (400) can operate based on the platform layer (510), the recognition layer (520), and the sensor service layer (530).
  • the platform layer (510) may be configured for XR services.
  • the platform layer (510) may include a platform (e.g., an Android platform) for supporting XR services.
  • the platform layer (510) may include an XR runtime module (511).
  • the XR runtime module (511) may be referred to as an OpenXR runtime module.
  • the XR runtime module (511) may be used to provide at least one of a pose prediction function, a frame timing function, and/or a spatial input function to a user through the electronic device (400).
  • the XR runtime module (511) may be used to perform rendering for an XR service to a user.
  • an application e.g., unity or an OpenXR native application
  • the interface (512) can be used for data exchange between the platform layer (510) and the perception service layer (520).
  • the interface (512) can be referenced as OpenPX and/or PPAL (perception platform abstract layer).
  • the interface (512) can be used for a perception client and a perception service.
  • the platform layer (510) can save perception data from the interface (512) (e.g., OpenPX) as a proto file.
  • the platform layer (510) can generate simulated input perception data based on the proto file.
  • the platform layer (510) can convert eye tracking information (or eye tracking data) transmitted through the interface (512) into a proto file in XRinputLogger.
  • the recognition service layer (520) may include a service module (521), a recognition plug-in layer (522), a sensor management module (523), a playback module (524), and/or an external data management module (525).
  • the recognition service layer (520) may include at least one of the service module (521), the recognition plug-in layer (522), the sensor management module (523), the playback module (524), and/or the external data management module (525).
  • at least some of the service module (521), the recognition plug-in layer (522), the sensor management module (523), the playback module (524), and the external data management module (525) may be omitted.
  • the service module (521) may be used to manage data (e.g., gesture information) obtained from a plurality of recognition modules included in the recognition plug-in layer (522).
  • data e.g., gesture information
  • the service module (521) may be referenced as SxrDataService.
  • the service module (521) may be used to exchange data (e.g., gesture information) between multiple recognition modules.
  • the service module (521) may interface with an upper layer (e.g., the platform layer (510) or the XR runtime module (511)).
  • the service module (521) may exchange data with the upper layer (e.g., the platform layer (510) or the XR runtime module (511)) through an interface (512).
  • the interface (512) may be referred to as OpenPX.
  • the service module (521) may support not only OpenPX but also the OpenXR Extension.
  • the recognition plug-in layer (522) may include a plurality of recognition modules.
  • the plurality of recognition modules may be referred to as a plurality of perception solutions.
  • the plurality of recognition modules may include at least one of a head tracking (HeT) module (522-1), a scene understanding (SU) module (522-2), a hand tracking (HaT) module (522-3), an eye tracking (ET) module (522-4), and a face tracking (FT) module (522-5).
  • HeT head tracking
  • SU scene understanding
  • HaT hand tracking
  • ET eye tracking
  • FT face tracking
  • Each of the plurality of recognition modules included in the recognition plug-in layer (522) may include a common interface for connection (or interworking) with the sensor management module (523).
  • Each of the plurality of recognition modules may include a common interface for connection (or interworking) with the sensor management module (523).
  • a plurality of recognition modules included in the recognition plug-in layer (522) may be configured in a plug-in structure. For example, some of the plurality of recognition modules may be replaced with other modules regardless of the sensor service layer (530) and the platform layer (510), which are lower layers of the recognition service layer (520).
  • the sensor management module (523) may be used to provide (or transmit) data to each of a plurality of recognition modules through a common interface.
  • the sensor management module (523) may be used to separate (or remove) the dependency between the sensor service layer (530), which is a lower layer, and the recognition plug-in layer (522), which is an upper layer.
  • the sensor management module (523) may be referred to as SxrSensorSeviceManger.
  • the sensor management module (523) may support various modules (or sensor services) of the sensor service layer (530).
  • the plurality of recognition modules may not directly interface with the sensor service layer (530).
  • the plurality of recognition modules may receive data (e.g., sensor data) through the sensor management module (523). Therefore, even if a module of the sensor service layer (530) is changed, the plurality of recognition modules may not be affected.
  • the sensor management module (523) may further include a load balancing module.
  • the load balancing module may identify data provided from the sensor service layer (530).
  • the load balancing module may identify at least some of the plurality of perception modules based on the data provided from the sensor service layer (530).
  • the load balancing module may provide data to at least some of the identified perception modules.
  • the load balancing module may distribute data to the plurality of perception modules based on the status of the plurality of perception modules and/or the status of the electronic device (400).
  • the load balancing module may filter the data provided to the plurality of perception modules based on the status of the plurality of perception modules and/or the status of the electronic device (400).
  • the load balancing module may be configured independently from the sensor management module (523).
  • the load balancing module may be referred to as SxrPerceptionLoadBalancer.
  • the playback module (524) may be used to provide a stored dataset to at least one of the plurality of recognition modules in real time through playback.
  • the dataset may be stored through the playback module (524) based on a specified standard.
  • the dataset may include first data acquired from the sensor service layer (530) as well as second data (e.g., virtual object data or synthetic data) acquired based on the first data acquired from the service layer (530).
  • the first data may be referred to as sensor data.
  • the second data may be referred to as virtual data.
  • the sensor data (or first data) included in the dataset may represent actual input acquired from the plurality of recognition modules (522-1, 522-2, 522-3, 522-4, 522-5).
  • virtual data (or second data) included in the dataset may represent simulation inputs obtained from modules other than the plurality of recognition modules (522-1, 522-2, 522-3, 522-4, 522-5).
  • a dataset may include virtual data portions between sensor data portions.
  • a dataset may include sensor data portions between virtual data portions.
  • operations for coordinate correction, data timing synchronization, data interpolation, and/or center coordinate point unification may be performed.
  • operations for coordinate correction, data timing synchronization, data interpolation, and/or center coordinate point unification between the sensor data and virtual data may be performed.
  • Operations for coordinate correction may be described with reference to FIGS. 13A and 13B.
  • Operations for data timing synchronization and/or data interpolation may be described with reference to FIG. 18.
  • Operations for center coordinate point unification may be described with reference to FIGS. 14 to 17.
  • the electronic device (400) may receive data from an external electronic device.
  • the data received from the external electronic device may include first data obtained from a service layer included in the external electronic device and/or second data obtained based on the first data.
  • the electronic device (400) may perform playback (or a playback function) using the data received from the external electronic device.
  • the electronic device (200) may transmit the result of performing the playback (or the playback function) to the external electronic device.
  • the electronic device (400) may be used to process the data obtained from the external electronic device on its behalf.
  • the electronic device (400) may receive data obtained from at least one sensor of the external electronic device.
  • the electronic device (400) may obtain information (e.g., information on a 6-degree-of-freedom pose) through the playback module (524) (or multiple recognition modules).
  • the electronic device (400) may transmit the obtained information to the external electronic device.
  • External electronic devices can provide XR services based on the acquired information.
  • the playback module (524) may perform playback (or a playback function) based on at least one of the first data and the second data. In some embodiments, the playback module (524) may perform playback by combining (or mixing) real-time data (e.g., runtime data) and pre-stored data.
  • real-time data e.g., runtime data
  • playback may refer to a function that utilizes data (or gesture information) stored according to an operation of the electronic device (400).
  • playback may refer to a function that identifies a value for the performance of an XR service by comparing gesture information acquired based on a specified operation related to the XR service with reference gesture information according to the specified operation.
  • playback may refer to a function for obtaining performance information of an XR service provided to a user of an electronic device (400).
  • the playback module (524) may identify information (e.g., gesture information) about a user who has performed a specified action (e.g., a mission) regarding an XR service.
  • the playback module (524) may identify reference information about the specified action.
  • the reference information may refer to information for determining the completion of performance of the specified action.
  • the playback module (524) may identify the similarity between the information about the user who has performed the specified action and the reference information. Based on the similarity, the playback module (524) may identify whether the performance of the action specified by the user has been completed.
  • the external data management module (525) may be used to manage data acquired through an external electronic device (e.g., a smartwatch, a smartphone, or a tablet PC) (or at least one sensor of the external electronic device) connected to the electronic device (400).
  • the external data management module (525) may use data acquired from the external electronic device to improve the accuracy of multiple recognition modules.
  • the external data management module (525) may use data acquired from the external electronic device to correct data (or gesture information) acquired from multiple recognition modules.
  • the external data management module (525) may not be included in the recognition service layer (520).
  • the sensor service layer (530) may be used to control at least one sensor (e.g., a camera, an IMU, a time of flight (ToF) sensor).
  • the sensor service layer (530) may be used to provide a service for accessing at least one sensor.
  • the sensor service layer (530) may include at least one of a module for a VR service (e.g., QVRservice), a module for an XR service (e.g., SxrSensorService), a sensor API (e.g., android sensor API), and a sensor hardware abstraction layer (sensor HAL).
  • a VR service e.g., QVRservice
  • a module for an XR service e.g., SxrSensorService
  • a sensor API e.g., android sensor API
  • sensor hardware abstraction layer sensor hardware abstraction layer
  • the sensor management module (523) may provide sensor data to the recognition plug-in layer (522) through a common interface.
  • the sensor management module (523) may provide sensor data to each of a plurality of recognition modules through substantially the same interface.
  • the sensor management module (523) may provide sensor data according to the operation of the recognition module to the recognition module without changing the configuration information of the recognition plug-in layer (522) based on changing (or modifying) the configuration information (e.g., configuration file) regarding the sensor management module (523).
  • the sensor management module (523) may identify sensor data for at least one recognition module based on the operation of at least one recognition module among a plurality of recognition modules. The sensor management module (523) may provide the identified sensor data to the at least one recognition module.
  • the sensor management module (523) may obtain camera data and IMU data through at least one of a module for VR service, a module for XR service, a sensor API, and a sensor hardware abstraction layer in the sensor service layer (530).
  • the sensor management module (523) may provide the camera data and IMU data to the head tracking module (522-1).
  • the camera data and IMU data may be obtained through different modules.
  • the sensor management module (523) can identify stored camera data and stored posture data.
  • the sensor management module (523) can provide the camera data and posture data to the environment recognition module (522-2).
  • the service module (521) may be configured to eliminate dependency on a higher layer of the recognition plugin layer (522).
  • the higher layer of the recognition plugin layer (522) may include a platform layer (510) (e.g., Android XR) and/or an application layer (e.g., the application layer (460) of FIG. 4).
  • the service module (521) may be configured to integrate and manage information (e.g., gesture information or tracking data) obtained from multiple recognition modules.
  • information e.g., gesture information or tracking data
  • the service module (521) may convert information (e.g., gesture information or tracking data) according to the requirements of the upper layer without changing a plurality of recognition modules, and then provide the converted information to the upper layer.
  • information e.g., gesture information or tracking data
  • the service module (521) can obtain information on a 6-degree-of-freedom pose from the head tracking module (522-1).
  • the information on the 6-degree-of-freedom pose obtained from the head tracking module (522-1) can be configured in a quaternion format.
  • a higher layer e.g., the platform layer (510)
  • the service module (521) can change (or convert) the information on the 6-degree-of-freedom pose configured in a quaternion format into information on the 6-degree-of-freedom pose configured in an axis-angle representation format.
  • the service module (521) can provide the information on the 6-degree-of-freedom pose configured in an axis-angle representation format to the higher layer (e.g., the platform layer (510)).
  • the service module (521) can change (or convert) information about a 6-degree-of-freedom pose configured in an axis-angle representation format into information about a 6-degree-of-freedom pose configured in a quaternion format and provide it to an upper layer.
  • the service module (521) may obtain information about hand movements from the hand tracking module (522-3). The information about hand movements may be obtained based on the movements of a first number of joints.
  • a higher layer e.g., the platform layer (510)
  • the service module (521) may perform either a joint interpolation procedure or a simplification procedure.
  • the service module (521) may support the structure of the joints required by the higher layer based on performing either a joint interpolation procedure or a simplification procedure.
  • the interface (526) may be used for data exchange between the platform layer (510) and the perception service layer (520).
  • the interface (526) may be referenced as OpenPX and/or PPAL.
  • the interface (526) may be used for the perception client and the perception service.
  • FIG. 6A illustrates an example of a structure in which an electronic device according to one embodiment receives simulation input entered through a client.
  • FIG. 6B illustrates an example of a structure in which an electronic device according to one embodiment receives simulation input entered through an input device.
  • FIGS 6a and 6b can be explained with reference to Figures 1 to 5.
  • a client (610) may generate an input for an electronic device (400) worn by a user.
  • the client (610) may be an external electronic device (e.g., a smartwatch, a smartphone, or a tablet PC) (or at least one sensor of the external electronic device) connected to the electronic device (400).
  • the client (610) may be in a state where a screen representing a three-dimensional virtual space generated by the electronic device (400) is being received from the electronic device (400) (e.g., two two-dimensional images considering binocular disparity and/or one two-dimensional image representing a three-dimensional virtual space).
  • the screen representing the three-dimensional virtual space generated by the electronic device (400) may be being shared through the display of the client (610).
  • the screen displayed on the client (610) may not be substantially identical to the FoV (field of view) displayed on the electronic device (400).
  • the screen displayed on the client (610) may be a two-dimensional projection of the FoV displayed on the electronic device (400).
  • the screen displayed on the client (610) may be substantially identical to the FoV displayed on the electronic device (400).
  • the input to the electronic device (400) generated by the client (610) may be distinguished from the input generated through the recognition plug-in layer (522) and/or the sensor service layer (530). In one embodiment, the input to the electronic device (400) generated by the client (610) may be an input managed by the external data management module (525).
  • the input to the electronic device (400) generated by the client (610) may be a simulated input.
  • the simulated input may be distinguished from an actual input.
  • the simulated input may be an input identified through a module other than the plurality of recognition modules (522-1, 522-2, 522-3, 522-4, 522-5) (e.g., the playback module (524), the external data management module (525), the simulated input API (620)).
  • the simulated input may be an input for a three-dimensional virtual space converted from an input for a two-dimensional space.
  • the actual input may be an input generated through the electronic device (400) and/or a controller connected to the electronic device (400).
  • the actual input may be input converted from recognition data (e.g., tracking data for a head, hand, eye, and/or controller) acquired via an actual camera (425) and/or sensor (430).
  • the actual input may be input identified via a recognition plug-in layer (522) and/or a sensor service layer (530).
  • the actual input may be input identified via a plurality of recognition modules (522-1, 522-2, 522-3, 522-4, 522-5).
  • the actual input may be input identified via the recognition plug-in layer (522).
  • the actual input may be input identified via a camera (425) and/or a sensor (430).
  • the client (610) can generate an input corresponding to at least one of the plurality of recognition modules (522-1, 522-2, 522-3, 522-4, 522-5).
  • the client (610) can generate an input corresponding to at least one of the plurality of recognition modules (522-1, 522-2, 522-3, 522-4, 522-5) via an input device (650) including an eye tracking device (651), a head tracking device (653), a hand tracking device (655), and/or a controller device (657).
  • the client (610) may generate a simulation input within a three-dimensional virtual space displayed by the electronic device (400) through the display (420).
  • the simulation input may include, but is not limited to, input representing a head posture (e.g., input based on head tracking), input representing a hand posture (e.g., input based on hand tracking), input representing an eye position (e.g., input based on eye tracking), and/or input based on a gesture by a controller and/or input representing the pushing of a button.
  • the simulation input may have a format (e.g., tracking data) of camera data, sensor data, and/or inertial measurement unit (IMU) data output from the sensor service layer (530) and/or the sensor management module (523).
  • the simulation input may have a format of data input to the recognition plug-in layer (522).
  • the simulation input may have a data format of cognitive data (e.g., tracking data for head, hand, eye, and/or controller).
  • the client (610) may transmit simulation input to the electronic device (400) based on calling the simulation input API (620).
  • the simulation input API (620) may include an input transformation module (625). In one embodiment, the simulation input API (620) may provide functions for interacting with a client (610) external to the electronic device (400). In one embodiment, the simulation input API (620) may be an API that allows the client (610) to invoke or call a function for inputting simulation input to the platform layer (510).
  • the input conversion module (625) may convert a simulation input into an input for a three-dimensional virtual space of the electronic device (400).
  • the simulation input may be an input for a two-dimensional plane.
  • the input for the two-dimensional plane may be an input including positions for two mutually perpendicular coordinates.
  • the input for the three-dimensional virtual space may be an input including positions for three mutually perpendicular coordinates.
  • the present invention is not limited thereto.
  • the simulation input may be an input for a three-dimensional virtual space.
  • an operation of converting an input for a two-dimensional plane into an input for a three-dimensional virtual space of the electronic device (400) may be described with reference to FIGS. 14 and 15.
  • simulation input from the client (610) can be provided to the platform layer (510) via a simulation input API (620) and via a simulation input connection (630).
  • the service module (521) (or the interface (526) within the service module (521)) can transmit at least one of the simulation input and/or the real input to the platform layer (510) depending on the operating mode (e.g., simulation mode and/or normal mode) of the electronic device (400).
  • the simulation input connection (630) and/or the real input connection (640) can dynamically connect the recognition plug-in layer (522) and/or the simulation input API (620) related to the real input to the platform layer (510) via the service module (521) (or the interface (526) within the service module (521)).
  • the simulation mode may refer to a mode in which simulation inputs are input to the platform layer (510) via a simulation input connection (630). In one embodiment, the simulation mode may refer to a mode in which a sequence of simulation inputs (or a scenario) (or a script) is played via the simulation input connection (630). In one embodiment, the normal mode may refer to a mode in which tracking data (or input) of the recognition plug-in layer (522) is input to the platform layer (510) via a real input connection (640).
  • the platform layer (510) can receive simulation input and/or real input via the simulation input connection (630) and/or the real input connection (640).
  • the platform layer (510) can sequentially perform reception of simulation input via the simulation input connection (630) and reception of real input via the real input connection (640).
  • the platform layer (510) can receive real input via the real input connection (640) after (or before) receiving simulation input via the simulation input connection (630).
  • the platform layer (510) can perform reception of simulation input via the simulation input connection (630) and reception of real input via the real input connection (640) in parallel (or substantially simultaneously).
  • the platform layer (510) can receive simulation input via a simulation input connection (630) and actual input via a actual input connection (640) substantially simultaneously.
  • a three-dimensional virtual space displayed on the electronic device (400) may change based on the simulation input. For example, based on a simulation input related to head tracking, the field of view (FoV) within the three-dimensional virtual space that the electronic device (400) displays on the display (420) may change. For example, based on a simulation input related to hand tracking being an input for selecting a virtual object, the electronic device (400) may execute a visual object within the three-dimensional virtual space.
  • the FoV at which the electronic device (400) displays the display (420) may change depending on the simulated input and the real input. For example, if the simulated input is an input related to a specific module (e.g., hand tracking and/or a controller) and the real input is an input related to other modules, while the FoV at which the electronic device (400) displays the display (420) changes depending on the simulated input, the electronic device (400) may execute a visual object within a three-dimensional virtual space depending on the simulated input.
  • a specific module e.g., hand tracking and/or a controller
  • the electronic device (400) may render an area corresponding to the user's gaze within the FoV more clearly and render an area not corresponding to the gaze less clearly depending on the real input. Since the electronic device (400) processes the simulation input obtained through the simulation input API (620), the electronic device (400) can ensure consistent operation even if the user interface for each application changes, and maintenance of the electronic device (400) and/or the application can be easy.
  • the electronic device (400) can control the electronic device (400) through simulated input received from an external device (e.g., client (610)) and/or actual input from the user. Accordingly, even in the absence of actual input from the user, the electronic device (400) can be controlled through the external device, thereby controlling the electronic device (400) to execute an application according to a desired scenario, and externally controlling the functions of the application executed through the electronic device (400).
  • an external device e.g., client (610)
  • the electronic device (400) can be controlled from an external device (client (610)). Therefore, users who have difficulty using the electronic device (400) can learn how to control the electronic device (400) by having the external device control the electronic device (400) while viewing the three-dimensional virtual space of the electronic device (400). In addition, even if an error occurs in the electronic device (400), the external device (client (610)) can remotely identify an abnormality in the electronic device (400) and remotely resolve any issues that can be resolved.
  • the electronic device (400) can process the progress of the game through simulation input remotely received from an external device (e.g., client (610)).
  • the electronic device (400) can share the progress of the game (or progress screen) with the external device (e.g., client (610)) and receive simulation input for the progress of the game from the external device (e.g., client (610)).
  • the user of the electronic device (400) can progress the game through the help (or simulation input) of the external device (e.g., client (610)).
  • the electronic device (400) can remotely share a screen displayed via FoV with an external device (e.g., a client (610)).
  • the electronic device (400) can receive simulation inputs related to the FoV of the screen being shared from the external device (e.g., a client (610)) (e.g., an input for selecting a virtual object within the FoV, an input for changing the FoV).
  • a user of the electronic device (400) can view and control the same screen together with a user of the external device (e.g., a client (610)).
  • the electronic device (400) can process the specific task (e.g., document work) through a simulation input remotely received from an external device (e.g., client (610)).
  • the electronic device (400) can share the progress (or progress screen) of the specific task (e.g., document work) with the external device (e.g., client (610)) and receive a simulation input (e.g., input for typing) for the specific task (e.g., document work) from the external device (e.g., client (610)).
  • a user of the electronic device (400) can jointly complete the specific task (e.g., document work) through the help (or simulation input) of the external device (e.g., client (610)).
  • FIG. 7 illustrates an example structure in which an electronic device according to one embodiment receives simulation input through a tester application.
  • Fig. 7 can be explained with reference to Figs. 1 to 5. Compared to Figs. 6a and 6b, Fig. 7 can illustrate a situation in which simulation input is received according to a sequence (or script) (or scenario) of simulation inputs generated by a tester application (710), rather than the simulation input being received from a client (610).
  • the application layer (460) may be connected to a space flinger (SpF) (730) and/or a CPM (720).
  • the SpF (730) may be connected to the CPM (720) and the application layer (460).
  • the electronic device (400) can execute applications (461, 462, 463, 464) within the application layer (460). In one embodiment, the electronic device (400) can execute the CPM (720) in response to the execution of an XR application among the applications (461, 462, 463, 464).
  • the electronic device (400) can provide image information provided from the XR application to the CPM (720).
  • the dual image information can include two pieces of image information that take binocular parallax into account.
  • the dual image information can include first image information for the user's left eye and second image information for the user's right eye for rendering in a three-dimensional virtual space.
  • the electronic device (400) can generate a composite image by merging image layers via the CPM (720).
  • the electronic device (400) can transmit the generated composite image to a display buffer.
  • the composite image can be displayed on the display (420) of the electronic device (400).
  • the electronic device (400) may execute the CPM (720) in response to the execution of a general application among the applications (461, 462, 463, 464).
  • the general application may be configured to output image information for displaying a two-dimensional image.
  • the general application may provide a two-dimensional image.
  • the electronic device (400) may execute the SpF (730) to provide dual image information to the CPM (720). For example, based on the execution of the SpF (730), the electronic device (400) may receive application-related information from the general application.
  • the application-related information may include image information representing a two-dimensional image of the application.
  • the application-related information (563) may be obtained through a spatializer API.
  • the electronic device (400) can identify information about the location and size of the area where the general application is to be rendered (hereinafter, “location information”).
  • location information information about the location and size of the area where the general application is to be rendered
  • the electronic device (400) can generate dual image information that takes into account the user’s binocular disparity through the image information and location information.
  • the electronic device (400) can provide the dual image information (565) to the CPM (720).
  • the electronic device (400) may execute a tester application (710).
  • the tester application (710) may generate inputs for visual objects (e.g., three-dimensional objects and/or two-dimensional objects in three-dimensional space) displayed through the currently executing application.
  • the electronic device (400) may generate a plurality of simulation inputs according to a script (or scenario) defined in the tester application (710).
  • the tester application (710) may, through the input transformation module (625), generate an input directed at a visual object based on coordinates for the visual object (e.g., a three-dimensional object and/or a two-dimensional object in three-dimensional space) displayed through the currently running application.
  • a visual object e.g., a three-dimensional object and/or a two-dimensional object in three-dimensional space
  • the tester application (710) can identify positional information of a virtual object within a two-dimensional and/or three-dimensional virtual space. In one embodiment, the tester application (710) can identify absolute positional information of a virtual object within a two-dimensional and/or three-dimensional virtual space.
  • the tester application (710) can identify relative position information of a virtual object. In one embodiment, the tester application (710) can identify relative position information of a virtual object based on the user's field of view (FoV) based on the user's posture (e.g., the posture of the user's head).
  • FoV field of view
  • the tester application (710) can generate simulation input within a 3D virtual space based on relative position information of the virtual object.
  • the simulation input within the 3D virtual space can be eye input, head input, hand input, and/or controller input directed toward the virtual object within the 3D virtual space.
  • the tester application (710) can obtain coordinates for a 2D virtual object and/or a 3D virtual object currently being displayed in real time.
  • the tester application (710) can change the coordinates obtained in real time to 3D coordinates based on the current head tracking (HeT) module, hand tracking (HaT) module, eye tracking (ET) module, and/or controller.
  • the tester application (710) can generate simulation input based on the changed 3D coordinates and transmit the generated simulation input to the simulation input connection (630).
  • the tester application (710) can pass the simulation input to the CPM (720) via the simulation input connection (630). In one embodiment, the tester application (710) can pass the simulation input to an input manager (e.g., a gaze manager, a head manager, a hand manager, and/or a controller manager) within the CPM (720) via the simulation input connection (630). In one embodiment, the CPM (720) can generate raw data for eye tracking using 3D coordinates in 3D space that have been converted from 2D coordinates by the tester application (710). For example, the CPM (720) can improve the accuracy of the simulation input by generating raw data for eye tracking through a space conversion process.
  • an input manager e.g., a gaze manager, a head manager, a hand manager, and/or a controller manager
  • the CPM (720) can generate raw data for eye tracking using 3D coordinates in 3D space that have been converted from 2D coordinates by the tester application (710). For example, the CPM (720) can improve the accuracy of the simulation input by generating
  • the CPM (720) can receive simulated input and/or real input via the simulated input connection (630) and/or the real input connection (640).
  • the 3D virtual space displayed on the electronic device (400) can be changed based on the simulated input. For example, based on the simulated input related to head tracking, the FoV within the 3D virtual space that the electronic device (400) displays on the display (420) can be changed.
  • the electronic device (400) can execute a visual object within the 3D virtual space.
  • the electronic device (400) can generate simulation inputs through the tester application (710) and generate virtual recognition data (or tracking data) from the generated simulation inputs. Accordingly, the electronic device (400) can control the execution of the application according to the scenario defined in the tester application (710) when the application within the application layer (460) is executed, so that the developer can easily verify the operation of the application and analyze its performance.
  • FIG. 8 illustrates an example of a structure for evaluating the performance of a tester application by an electronic device according to one embodiment.
  • the input transformation module (625) can transform an input for a two-dimensional plane into a simulation input (810) in a three-dimensional virtual space.
  • the input transformation module (625) can transform the coordinates of the input for the two-dimensional plane into coordinates in a three-dimensional virtual space.
  • the input transformation module (625) can transform the coordinates of the input toward a virtual object in a two-dimensional plane into the coordinates of the input toward a virtual object in a three-dimensional virtual space.
  • the electronic device (400) can convert the simulation input (810) input to the CPM (720) into simulation tracking data of the input module through spatial transformation. For example, if the simulation input (810) is an input for eye gaze, the electronic device (400) can convert the simulation input (810) into eye tracking data.
  • the electronic device (400) can identify actual tracking data of an input module for identifying an actual input (820) input to the CPM (720).
  • the actual input (820) may be an input directed toward a virtual object within a three-dimensional virtual space. For example, if the actual input (820) is an input for eye gaze, the electronic device (400) can identify actual tracking data of the ET module (522-4) for identifying the actual input (820).
  • the electronic device (400) can compare simulated tracking data with actual tracking data. The smaller the difference between the simulated tracking data and the actual tracking data, the better the performance of the input conversion module (625) can be evaluated.
  • the electronic device (400) may update the conversion algorithm of the input conversion module (625) so that the difference between the simulated tracking data and the actual tracking data is reduced.
  • the conversion algorithm of the input conversion module (625) may be an algorithm that converts an input on a two-dimensional plane into a simulated input (810) in a three-dimensional virtual space.
  • the electronic device (400) can evaluate the performance of the input conversion module (625) by comparing the tracking data for input other than the input within the CPM (720).
  • FIG. 9 illustrates an example of a situation in which an electronic device, according to one embodiment, switches between simulation mode and normal mode.
  • FIG. 10a illustrates an example of a situation in which an electronic device, according to one embodiment, processes input in simulation mode and normal mode.
  • FIG. 10b illustrates examples of simulated input and actual input, according to one embodiment.
  • FIGS. 9 to 10b may be described with reference to FIGS. 1 to 8.
  • scripts (911, 913, 915) may be executed by the electronic device (400) during periods (930, 950) operating in simulation mode.
  • scripts (911, 913, 915) may define a sequence of simulation inputs.
  • scripts (911, 913, 915) may be computer programming codes configured to input simulation inputs to the CPM (720) according to specified conditions.
  • the electronic device (400) may execute scripts (911, 913, 915) based on receiving a start command (e.g., startSimulation) requesting execution of scripts (911, 913, 915) through a simulation mode.
  • a start command e.g., startSimulation
  • the electronic device (400) may execute script (911) based on receiving a start command generated at a time point (901).
  • input of simulation input may be permitted. Allowing input of simulation input may include that the simulation input connection (630) connects the CPM (720) and the simulation input API (620). For example, referring to the structure (1001) of FIG.
  • the simulation input connection (630) may connect the CPM (720) and the simulation input API (620), and the actual input connection (640) may selectively connect the CPM (720) and the recognition plug-in layer (522).
  • the actual input connection (640) may selectively connect the CPM (720) and the recognition plug-in layer (522), which may include connecting some of the input modules (e.g., the ET module (522-4)) to the CPM (720), and not connecting other some of the input modules (e.g., the HaT module (522-3)) to the CPM (720).
  • the electronic device (400) may allow input of actual input through the input module during a time period (935) from the time when the execution of the script (911) is completed until the execution of the next script (913) is completed, based on receiving a start command generated at time point (901). Allowing input of actual input through the input module may include that the actual input connection (640) connects the CPM (720) and the recognition plug-in layer (522). According to an embodiment, the electronic device (400) may not allow input of actual input through the input module during a time period (960) from the time when the execution of the script (911) is initiated until the execution of the script (913) is completed, based on receiving a start command generated at time point (901). For example, the electronic device (400) may allow input of only simulated input during the time period (960). For example, the time interval (960) may be a time interval during which the electronic device (400) operates in simulation mode.
  • the electronic device (400) may stop execution of running scripts (911, 913, 915) based on receiving a stop command (e.g., stopSimulation) requesting to stop execution of scripts (911, 913, 915) through simulation mode.
  • a stop command e.g., stopSimulation
  • the electronic device (400) may operate in normal mode (or real-time mode) after receiving the stop command (e.g., stopSimulation) and may switch to simulation mode during runtime.
  • the electronic device (400) may switch between normal mode and simulation mode in real time after booting of the electronic device (400).
  • the electronic device (400) may stop execution of the script (913) based on receiving a stop command generated at time point (903). For example, the electronic device (400) may switch from simulation mode to normal mode during a time interval (940) until the next start command based on receiving a stop command generated at time point (903). For example, the electronic device (400) may not allow input of simulation input during the time interval (940) while operating in normal mode. Disallowing input of simulation input may include the simulation input connection (630) disconnecting the CPM (720) and the simulation input API (620).
  • the simulation input connection (630) may selectively connect the CPM (720) and the simulation input API (620), and the actual input connection (640) may connect the CPM (720) and the recognition plug-in layer (522).
  • the simulation input connection (630) may selectively connect the CPM (720) and the simulation input API (620) to connect some types of input (e.g., input based on hand tracking) among a plurality of types to the CPM (720) and not connect other types of input modules (e.g., input based on eye tracking) to the CPM (720).
  • the electronic device (400) can connect the CPM (720) (or the XRHeadDevice (1040) of the CPM (720)) to a simulation head input connection (1041) and/or a real head input connection (1045).
  • the electronic device (400) can, during normal mode, connect the XRHeadDevice (1040) to the real head input connection (1045) and optionally connect the XRHeadDevice (1040) to the simulated head input connection (1041).
  • the electronic device (400) can perform head tracking using real input while the XRHeadDevice (1040) is connected to the real head input connection (1045).
  • the electronic device (400) can perform head tracking using real input and/or simulated input while the XRHeadDevice (1040) is connected to the real head input connection (1045) and optionally connect the XRHeadDevice (1040) to the simulated head input connection (1041).
  • the electronic device (400) can perform head tracking using real input and simulated input simultaneously.
  • the electronic device (400) can perform head tracking on an input selected from among actual input or simulated input.
  • the electronic device (400) can, during simulation mode, connect the XRHeadDevice (1040) to the simulation head input connection (1041), and optionally, connect the XRHeadDevice (1040) to the real head input connection (1045).
  • the electronic device (400) can perform head tracking via simulated input while the XRHeadDevice (1040) is connected to the simulated head input connection (1041).
  • the electronic device (400) can perform head tracking via real input and/or simulated input while the XRHeadDevice (1040) is connected to the real simulated head input connection (1041), and optionally, connect the XRHeadDevice (1040) to the head input connection (1045).
  • the electronic device (400) can connect the CPM (720) (or the XRHandDevice (1050) of the CPM (720)) to a simulated hand input connection (1051) and/or a real hand input connection (1055).
  • the electronic device (400) can, during normal mode, connect the XRHandDevice (1050) to a real hand input connection (1055), and optionally, connect the XRHandDevice (1050) to a simulated hand input connection (1051).
  • the electronic device (400) can, during simulation mode, connect the XRHandDevice (1050) to a simulated hand input connection (1051), and optionally, connect the XRHandDevice (1050) to a real hand input connection (1055).
  • the electronic device (400) can connect the CPM (720) (or the XREyeDevice (1060) of the CPM (720)) to the simulated eye input connection (1061) and/or the real eye input connection (1065).
  • the electronic device (400) can, during normal mode, connect the XREyeDevice (1060) to the real eye input connection (1065), and optionally, connect the XREyeDevice (1060) to the simulated eye input connection (1061).
  • the electronic device (400) can, during simulation mode, connect the XREyeDevice (1060) to the simulated eye input connection (1061), and optionally, connect the XREyeDevice (1060) to the real eye input connection (1065).
  • the electronic device (400) can connect the CPM (720) (or the XRControllerDevice (1070) of the CPM (720)) to the simulation controller input connection (1071) and/or the actual controller input connection (1075).
  • the electronic device (400) can, during normal mode, connect the XRControllerDevice (1070) to the actual controller input connection (1075), and optionally, connect the XRControllerDevice (1070) to the simulation controller input connection (1071).
  • the electronic device (400) can, during simulation mode, connect the XRControllerDevice (1070) to the simulation controller input connection (1071), and optionally, connect the XRControllerDevice (1070) to the actual controller input connection (1075).
  • the electronic device (400) may execute a script (915) based on receiving a start command generated at a point in time (905). During the time period during which the script (915) is executed, input of simulation input may be permitted. Allowing input of simulation input may include the simulation input connection (630) connecting the CPM (720) and the simulation input API (620).
  • the electronic device (400) can selectively allow real input during the simulation mode, or can selectively allow simulated input during the normal mode. Accordingly, the user of the electronic device (400) can obtain assistance from an external device when using the electronic device (400) during the normal mode.
  • the three-dimensional virtual space of the electronic device (400) can be controlled together with an external device that provides a screen representing the three-dimensional space of the electronic device (400) (e.g., image information considering binocular disparity or two-dimensional image information).
  • the electronic device (400) can control the three-dimensional virtual space of the electronic device (400) using real input and simulated input that are received sequentially (or substantially simultaneously).
  • operations for coordinate correction, data timing synchronization, data interpolation, and/or center coordinate point unification may be performed.
  • the input used by the electronic device (400) to control the three-dimensional virtual space of the electronic device (400) is switched between actual input and simulated input, operations for coordinate correction, timing synchronization, data interpolation, and/or center coordinate point unification between the actual input and simulated input may be performed.
  • the operation for coordinate correction may be described with reference to FIGS. 13A and 13B.
  • the operation for data timing synchronization and/or data interpolation may be described with reference to FIG. 18.
  • the operation for center coordinate point unification may be described with reference to FIGS. 14 to 17.
  • FIG. 11 illustrates an example of a situation in which an electronic device according to one embodiment records input in normal mode and then inputs it to a simulation input connection.
  • FIG. 12 illustrates an example of a situation in which a recording file is processed according to one embodiment.
  • Figures 11 and 12 can be explained with reference to Figures 1 to 5. Compared to Figures 6a and 6b, Figures 11 and 12 can illustrate a situation in which, rather than receiving simulation input from a client (610), actual input is recorded and then the recorded actual input is utilized as simulation input.
  • the electronic device (400) can generate a log file (1110) based on input input to the CPM (720) via an actual input connection (640).
  • the electronic device (400) may generate tracking data identified through a tracking module (1210) (e.g., at least one of the plurality of recognition modules (522-1, 522-2, 522-3, 522-4, 522-5) of FIG. 5 ).
  • the tracking data may be data acquired through the ET module (522-4).
  • the tracking data may include information about a tracking mode, information about the left eye, and/or information about the right eye.
  • the information about the left eye and/or the information about the right eye may include eye status, eye direction, eye position, and gaze information for foveation.
  • the present invention is not limited thereto.
  • the information about the left eye and/or the information about the right eye may include information about the dominant eye and the auxiliary eye.
  • the information about the left eye and/or the information about the right eye may include coordinate information in which fine eye (or pupil) movements are filtered out to stabilize eye coordinates (or pupil coordinates). Coordinate information in which fine eye (or pupil) movements are filtered out may be referred to as filtered gaze.
  • the filtered gaze may be coordinate information in which eye tremor (or coordinate variation) is stabilized by applying a filter (e.g., a normalization filter) to the eye coordinates to reduce eye coordinate tremor caused by pupil tremor.
  • the information about the left eye and/or the information about the right eye may include coordinate information in which eye (or pupil) movements are not filtered.
  • Coordinate information in which eye (or pupil) movements are not filtered may be referred to as unfiltered gaze.
  • the unfiltered gaze may be coordinate information in which eye coordinate tremor caused by pupil tremor is not reduced.
  • unfiltered gazes may be used to reduce side effects caused by stabilization of eye coordinates by filtered gazes.
  • the electronic device (400) can input tracking data into an input device (1220) within the CPM (720) via a physical input connection (640).
  • the electronic device (400) can input tracking data into an input device (1220) (e.g., a device that processes eyes, a head, a hand, and/or a controller) associated with the tracking data within the CPM (720) via a physical input connection (640).
  • an input device (1220) e.g., a device that processes eyes, a head, a hand, and/or a controller
  • the tracking data can be input into an eye device (e.g., an XREyeDevice).
  • the electronic device (400) may transmit tracking data input into the input device (1220) to the tracking record module (1230).
  • the tracking record module (1230) may record the tracking data from the time when recording is started.
  • the tracking record module (1230) may record the tracking data according to a specified cycle from the time when recording is started.
  • the electronic device (400) may store a realignment signal (or recenter signal) based on the gaze looking at the FoV when recording the tracking data.
  • the realignment signal may represent the FoV displayed through the display (420) according to the head pose and/or gaze in a three-dimensional space.
  • the electronic device (400) may convert the tracking data recorded through the input recording module (1240) based on a proto (or protocol buffer).
  • the proto may be an algorithm for serializing structured data.
  • the electronic device (400) may store the tracking data converted through the input recording module (1240) as a recording file (1110). For example, if the tracking data is data acquired through the ET module (522-4), the recording file (1110) may be data converted based on the proto, including information about the tracking mode, information about the left eye, and/or information about the right eye.
  • the electronic device (400) can reproduce a simulation input based on a recording file (1110). In one embodiment, the electronic device (400) can realign the FoV through a recenter signal to reproduce the simulation input. In one embodiment, the electronic device (400) can adjust (or realign) the FoV within a three-dimensional space being displayed through the display (420) of the electronic device (400) to correspond to the FoV when recording the tracking data that generated the recording file (1110) begins. In one embodiment, the electronic device (400) can adjust (or realign) the FoV within the three-dimensional space being displayed through the display (420) without any movement of the user (e.g., head movement, gaze movement).
  • the electronic device (400) can adjust (or realign) the FoV within the three-dimensional space being displayed through the display (420) without any movement of the user (e.g., head movement, gaze movement).
  • the electronic device (400) can input simulation input based on the log file (1110) into the CPM (720) via the simulation input connection (630). In one embodiment, the electronic device (400) can generate simulation input (or tracking data) by inversely transforming data stored in the log file (1110) based on the protocol.
  • the electronic device (400) can input simulation input to an input device (1220) within the CPM (720) via a simulation input connection (630).
  • the electronic device (400) can input simulation input to an input device (1220) (e.g., a device that processes eyes, a head, a hand, and/or a controller) associated with tracking data within the CPM (720) via a simulation input connection (630).
  • an input device (1220) e.g., a device that processes eyes, a head, a hand, and/or a controller
  • the tracking data can be input to an eye device (e.g., an XREyeDevice).
  • the electronic device (400) can transmit tracking data input into the input device (1220) to the input manager (1250).
  • the input manager (1250) can process input based on the tracking data. For example, based on a simulation input related to head tracking, the electronic device (400) can change the FoV within the 3D virtual space in which the display (420) is displayed. For example, based on the simulation input related to hand tracking being an input for selecting a virtual object, the electronic device (400) can execute a visual object within the 3D virtual space.
  • FIG. 13A illustrates an example of an electronic device according to an embodiment of the present invention, which identifies a user's gaze within a virtual space based on the position of the user's eyes.
  • FIG. 13B illustrates an example of an electronic device according to an embodiment of the present invention, which identifies a user's gaze within a virtual space based on the position of the user's eyes.
  • FIGS 13a and 13b can be explained with reference to Figures 1 to 12.
  • the electronic device (400) must generate eye tracking data such that the user (1300) has a gaze (1340) directed toward a virtual object (1330) within the FoV (1301) at a time when the user (1300) should look at the virtual object (1330) within the FoV (1301).
  • the electronic device (400) may convert the coordinates of the virtual object (1330) within the virtual space and the coordinates of the eyes (1310, 1315) within the cognitive space into the same coordinate system to generate tracking data representing the gaze (1340) toward the virtual object (1330).
  • the coordinates (1351) of the eyes (1310, 1315) within the cognitive space (1350) may be coordinates based on the camera (425-2) that acquires the image (1305) that captures the eyes (1310, 1315).
  • the coordinates (1351) of the eyes (1310, 1315) within the cognitive space (1350) may be the coordinates of the pupils (1320, 1325) of the eyes (1310, 1315).
  • the electronic device (400) may convert coordinates (1351) of eyes (1310, 1315) into coordinates (1361) in stage space (1360) based on head posture (1365).
  • Stage space (1360) may be coordinates of a fixed three-dimensional virtual space.
  • Stage space (1360) may be coordinates of a three-dimensional virtual space based on a fixed guardian (or anchor).
  • the electronic device (400) can convert coordinates (1363) of a virtual object (1330) in a 3D virtual space into coordinates (1373) in a tracker space (1370) based on coordinates (1361) of the eyes (1310, 1315) in the stage space (1360). In one embodiment, the electronic device (400) can convert coordinates (1373) of a virtual object (1330) in the tracker space (1370) based on coordinates (1361) of the eyes (1310, 1315) in the stage space (1360) and coordinates (1363) of the virtual object (1330) in the stage space (1360). In one embodiment, the tracker space (1370) can be a space based on the sensor (430) (or head IMU sensor).
  • the electronic device (400) can convert the coordinates of the virtual object (1330) in the tracker space (1370) to coordinates (1383) in the view space (or FoV) (1380) based on the offset between the center positions of the eyes (1310, 1315) from the sensor (430) (or head IMU sensor). In one embodiment, the electronic device (400) can convert the coordinates of the virtual object (1330) in the tracker space (1370) to coordinates (1383) in the view space (or FoV) (1380) by subtracting the offset from the coordinates (1373) of the virtual object (1330) in the tracker space (1370).
  • the electronic device (400) can transform coordinates (1383) of a virtual object (1330) within a view space (or FoV) (1380) into coordinates (1393) within a perception space (1390).
  • the electronic device (400) can generate coordinates (1393) of the virtual object (1330) within a perception space (1390) by multiplying the coordinates (1383) of the virtual object (1330) within the view space (or FoV) (1380) by a specified transformation matrix (e.g., a matrix for rotation about a specified axis (e.g., a 180 degree rotation) and/or a matrix for subtracting an offset from the center positions of the eyes (1310, 1315) to the camera (425-2).
  • the cognitive space (1350) and the cognitive space (1390) in FIG. 13b may represent the same cognitive space.
  • the electronic device (400) can obtain information indicating the line of sight (1340) (or the direction (1391) of the eyes (1310, 1315)) by subtracting the coordinates (1393) of the virtual object (1330) within the cognitive space (1390) and the coordinates (1351) of the eyes (1310, 1315) within the cognitive space (1390).
  • Fig. 14 illustrates an example of a 2D image according to an embodiment.
  • Fig. 15 illustrates an example of an electronic device converting coordinates within a 2D image into coordinates within a 3D virtual space according to an embodiment.
  • Fig. 16 illustrates an example of a field of view (FoV) within a 3D virtual space shown to a user according to an embodiment.
  • Fig. 17 illustrates an example of an electronic device changing a FoV within a 3D virtual space shown to a user according to a simulation input according to an embodiment.
  • FoV field of view
  • a client (610) may generate an input (1440) for a two-dimensional image (1400).
  • the client (610) may generate an input (1440) directed toward a virtual object (1410) among virtual objects (1410, 1420, 1430) within the two-dimensional image (1400) and select the virtual object (1410).
  • the two-dimensional image (1400) may represent an FoV within a three-dimensional virtual space generated by the electronic device (400).
  • the electronic device (400) may receive a simulation input from a client (610). For example, the electronic device (400) may receive an input (1440) regarding a two-dimensional image (1400) from the client (610). For example, the electronic device (400) may receive an input (1440) pointing toward a virtual object (1410) among virtual objects (1410, 1420, 1430) in the two-dimensional image (1400) and selecting the virtual object (1410).
  • the electronic device (400) can convert a simulation input from a client (610) into an input (1540) in an FoV (1505) in a three-dimensional virtual space (1501).
  • the electronic device (400) can convert an input (1440) for a two-dimensional image (1400) into an input (1540) directed toward a virtual object (1510) among virtual objects (1510, 1520, 1530) in the FoV (1505) and select the virtual object (1510).
  • the present invention is not limited thereto.
  • the electronic device (400) can receive an input (1540) in an FoV (1505) in a three-dimensional virtual space (1501) from the client (610).
  • the electronic device (400) can generate an input (1440) for a two-dimensional image (1400) based on executing a tester application (710).
  • the electronic device (400) can generate an input (1540) in a FoV (1505) in a three-dimensional virtual space (1501) based on executing a tester application (710).
  • the electronic device (400) can receive a sequence of multiple simulation inputs based on playing a recording file (1110).
  • the electronic device (400) can generate an input (1440) for a two-dimensional image (1400) based on receiving the sequence of multiple simulation inputs.
  • the electronic device (400) can generate an input (1540) in a FoV (1505) in a three-dimensional virtual space (1501) based on receiving the sequence of multiple simulation inputs.
  • the electronic device (400) may determine to play back the input (1540) while providing the user with the FoV (1605) within the three-dimensional virtual space (1501). For example, the electronic device (400) may determine to play back the input (1540) while the user views the FoV (1605) from a different gaze than the gaze for the FoV (1505). For example, the electronic device (400) may determine to switch from normal mode to simulation mode while the user views the FoV (1605). For example, the electronic device (400) may determine to play back the input (1540) based on a user's request for playing back the input (1540).
  • the electronic device (400) may determine, based on determining the reproduction of the input (1540), whether the center point (or FoV) toward which the gaze of the user is directed within the three-dimensional virtual space when the input (1540) is acquired is substantially the same as the center point (or FoV) toward which the current user's gaze is directed within the three-dimensional virtual space. In one embodiment, the electronic device (400) may change the image displayed within the user's FoV (1605) based on whether the FoV of the user is different from the FoV of the user when the input (1540) is acquired.
  • the electronic device (400) may change the image displayed within the user's FoV (1605) based on a determination to play back the input (1540). For example, the electronic device (400) may rearrange the 3D virtual space (1501) so that an image for the FoV (1505) is displayed in the FoV (1605) while the user faces the FoV (1605) based on a playback request of the input (1540). For example, the electronic device (400) may rearrange the 3D virtual space (1501) so that an image for the FoV (1505) is displayed in the FoV (1605) while the user faces the FoV (1605).
  • the present invention is not limited thereto.
  • the electronic device (400) can guide the user's gaze movement from FoV (1605) to FoV (1505) so that the user's gaze is directed toward FoV (1505).
  • the electronic device (400) may display an image for the FoV (e.g., 1505 of FIG. 15) in the FoV (1605) based on determining playback of the input (1540). In one embodiment, the electronic device (400) may gradually replace (e.g., fade in, fade out) the image displayed in the FoV (1605) with the image for the FoV (1505). In one embodiment, the electronic device (400) may replace the image displayed in the FoV (1605) with the image for the FoV (1505) through a visual effect such that the FoV (1605) moves to the FoV (1505).
  • the electronic device (400) may display an image for the FoV (e.g., 1505 of FIG. 15) in the FoV (1605) based on determining playback of the input (1540). In one embodiment, the electronic device (400) may gradually replace (e.g., fade in, fade out) the image displayed in the FoV (1605) with the image for the FoV (1505). In one embodiment, the electronic device (400) may replace the image displayed
  • a visual effect such as FoV(1605) moving to FoV(1505) may be achieved by sequentially changing the image displayed at FoV(1605) to images representing multiple FoVs between FoV(1605) and FoV(1505) without the user changing the posture of their head.
  • the electronic device (400) can reproduce the input (1540) based on the display of the image for the FoV (1505) in the FoV (1605). In one embodiment, the electronic device (400) can reproduce the input (1540) after the 3D virtual space (1501) is rearranged so that the image for the FoV (1505) is displayed in the FoV (1605). For example, the electronic device (400) can move the user's gaze toward the virtual object (1510) among the virtual objects (1510, 1520, 1530) in the FoV (1605) and select the virtual object (1510) based on the display of the input (1540). In one embodiment, while playing back the input (1540) (or during simulation mode), the electronic device (400) may limit the user's actual input. For example, while playing back the input (1540) (or during simulation mode), the electronic device (400) may limit some of the plurality of input modules (e.g., a head tracking module) from connecting to the platform layer (510) via the actual input connection (640).
  • the electronic device (400) may limit some of the plurality
  • the electronic device (400) can reproduce the input (1540) without rearranging the three-dimensional virtual space (1501).
  • the electronic device (400) can adjust (or change) the position of the input (1540) to be reproduced within the FoV (1605).
  • the electronic device (400) can identify the position of the input (1540) within the FoV (1605) based on the coordinates of the input (1540) within the FoV (1505) and the distance between the FoVs (1605) and the FoVs (1505) in the three-dimensional virtual space.
  • the electronic device (400) can reproduce the input (1540) at the position within the identified FoV (1605).
  • the electronic device (400) can select a virtual object (1510) that is not visible within the FoV (1605) by executing an input (1540).
  • FIG. 18 illustrates an example of an electronic device synchronizing between simulation input and general input according to one embodiment.
  • the inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) in the simulation mode (1810) may be simulation inputs obtained from a start time (1811) according to a specified period (1815).
  • the inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) received in the simulation mode (1810) may be inputs of some types (e.g., inputs based on hand tracking) among a plurality of types.
  • inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) may be a sequence of two or more inputs acquired during a specified first time.
  • inputs (1821, 1822, 1823, 1824, 1825) may be actual inputs acquired from a start time (1831) according to a designated cycle (1835).
  • Inputs (1821, 1822, 1823, 1824, 1825) received in normal mode (1830) may be inputs of multiple input modules. Accordingly, processing actual inputs of multiple input modules may require more time than processing some types of simulation inputs. In order to smoothly process actual inputs of multiple input modules, the sampling cycle may be longer than the sampling cycle for simulation inputs.
  • inputs (1821, 1822, 1823, 1824, 1825) sampled (or received) according to the normal mode (1830) may be acquired with a longer period than inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) sampled (or received) according to the simulation mode.
  • the electronic device (400) when processing real inputs and simulation inputs simultaneously, the electronic device (400) must synchronize the simulation inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) acquired in a shorter cycle and the real inputs (1821, 1822, 1823, 1824, 1825).
  • mode (1850) may be a mode in which the electronic device (400) simultaneously processes inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) according to the simulation mode (1810) and inputs (1821, 1822, 1823, 1824, 1825) according to the normal mode (1830).
  • the electronic device (400) may determine inputs (1801, 1861, 1804, 1862, 1807) synchronized with inputs (1821, 1822, 1823, 1824, 1825) obtained according to the cycle (1835) according to the normal mode (1830).
  • the electronic device (400) can receive a request to play back a simulation input. In one embodiment, the electronic device (400) can receive the request to play back the simulation input during a different playback time that is distinct from the playback time of the simulation input. In one embodiment, the electronic device (400) can receive the request to play back the simulation input at a different time from the time at which the simulation input was recorded. In one embodiment, the electronic device (400) can determine which of the inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) to play back during a different playback time (or at a different playback speed).
  • the playback speed of the simulation inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) may be different from or the same as the playback speed of the actual inputs (1821, 1822, 1823, 1824, 1825).
  • the playback speed of the simulation inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) may be referred to as the first playback speed.
  • the playback speed of the actual inputs (1821, 1822, 1823, 1824, 1825) may be referred to as the second playback speed.
  • the electronic device (400) when processing the first playback speed of the simulation inputs to be slower than the second playback speed of the actual input, can generate interpolated simulation inputs corresponding to the second playback speed by interpolating (or upsampling) data in the simulation inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807). In one embodiment, when processing the first playback speed of the simulation inputs to be faster than the second playback speed of the actual input, the electronic device (400) can generate interpolated simulation inputs corresponding to the second playback speed by interpolating (or downsampling) data in the simulation inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807).
  • the time interval at which the simulation inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807) are processed may be increased.
  • the user may experience the FoV switching at a slower speed.
  • the user may experience applying an input (or gesture) to a virtual object within the FoV at a slower speed.
  • the electronic device (400) proceeds with a game through the simulation inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807), the user may experience the game proceeding at a slower speed, and the difficulty of the game perceived by the user may be reduced. Additionally, when the electronic device (400) performs a specific task (e.g., an assembly task) through simulation inputs (1801, 1802, 1803, 1804, 1805, 1806, 1807), the user may experience the specific task being performed at a slower speed, and the user may be able to understand the specific task more precisely.
  • a specific task e.g., an assembly task
  • the electronic device (400) may provide the acquired (or synchronized) simulation input to the platform layer (510) together with the actual input at a time corresponding to the acquisition of the actual input.
  • the electronic device (400) may provide the input (1821) to the platform layer (510) via the actual input connection (640), and simultaneously provide the input (1801) synchronized to the input (1821) to the platform layer (510) via the simulation input connection (630).
  • the electronic device (400) may provide the input (1823) to the platform layer (510) via the actual input connection (640), and simultaneously provide the input (1804) synchronized to the input (1823) to the platform layer (510) via the simulation input connection (630).
  • the electronic device (400) can provide an input (1825) to the platform layer (510) via a real input connection (640), and simultaneously provide an input (1807) synchronized to the input (1825) to the platform layer (510) via a simulated input connection (630).
  • the electronic device (400) may provide a simulation input interpolated from two or more simulation inputs around the time at which the actual input is acquired to the platform layer (510) together with the actual input. For example, based on the absence of a simulation input synchronized to the input (1822), the electronic device (400) may provide a simulation input (1861) interpolated from two or more simulation inputs (1802, 1803) around the time at which the input (1822) is acquired to the platform layer (510) together with the input (1822).
  • the electronic device (400) may provide the platform layer (510) with the input (1824) a simulation input (1862) that interpolates two or more simulation inputs (1805, 1806) around the time point at which the input (1824) is acquired.
  • the present invention is not limited thereto. If there is no simulation input synchronized to the time point at which the actual input is acquired, the electronic device (400) may provide the simulation input acquired (or synchronized) at the closest time point to the platform layer (510) together with the actual input.
  • the electronic device (400) may provide the simulation input (1803) that is closest to the time point at which the input (1822) is acquired to the platform layer (510) together with the input (1822). For example, the electronic device (400) may provide the platform layer (510) with the closest simulation input (1806) to the time at which the input (1824) was acquired, based on the absence of a simulation input synchronized to the input (1824).
  • Figure 19 is a flowchart illustrating the operation of an electronic device according to one embodiment.
  • Fig. 19 can be explained with reference to Figs. 1 to 18.
  • the electronic device (400) may receive a simulation input.
  • the simulation input may be an input identified through a module (e.g., a playback module (524), an external data management module (525), a simulation input API (620)) other than a plurality of recognition modules (e.g., 522-1, 522-2, 522-3, 522-4, 522-5 of FIG. 5).
  • the simulation input may be an input for a three-dimensional virtual space converted from an input for a two-dimensional space.
  • the actual input may be an input generated through the electronic device (400) and/or a controller connected to the electronic device (400).
  • the actual input may be input converted from recognition data (e.g., tracking data for a head, hand, eye, and/or controller) acquired via an actual camera (425) and/or sensor (430).
  • the actual input may be input identified via a recognition plug-in layer (522) and/or a sensor service layer (530).
  • the actual input may be input identified via a plurality of recognition modules (522-1, 522-2, 522-3, 522-4, 522-5).
  • the actual input may be input identified via the recognition plug-in layer (522).
  • the actual input may be input identified via a camera (425) and/or a sensor (430).
  • the electronic device (400) may receive simulation input while the electronic device (400) operates in simulation mode, but is not limited thereto.
  • the electronic device (400) may receive simulation input based on a simulation input connection (e.g., 630 of FIG. 6A) being connected to a CPM (e.g., 720 of FIG. 7) while the electronic device (400) operates in normal mode.
  • a simulation input connection e.g., 630 of FIG. 6A
  • CPM e.g., 720 of FIG.
  • the simulation input may be an input received by the electronic device (400) from an external device (e.g., the electronic device (102) of FIG. 1 and/or the client (610) of FIGS. 6A and 6B) via the communication circuit (440).
  • the simulation input may be an input via a designated type of input device (e.g., an eye tracking device (651)) among a plurality of input devices (651, 653, 655, 657).
  • the simulation input may be an input via the client (610) described with reference to FIGS. 6A and 6B.
  • the simulation input may be an input for a two-dimensional image generated through the tester application (710) described with reference to FIG. 7.
  • the simulation input may be an input for a three-dimensional virtual space converted from an input for a two-dimensional planar space.
  • the simulation input may be an input for a three-dimensional virtual space converted from an input included in a script (or scenario) (or a sequence of inputs) through the input transformation module (625).
  • the simulated input may be an input recorded from the actual input described with reference to FIGS. 11 and 12.
  • the simulated input may be an input based on tracking data of a specified type (e.g., eye, head, hand, and/or controller) in a virtual space received via an input circuit (e.g., camera (425) and/or sensor (430)).
  • the simulated input may be an input based on tracking data recorded in memory (415) from a specified point in time (or recording point in time).
  • the simulated input may be an input based on tracking data recorded in memory (415) together with data representing the user's pose (or FoV according to the pose) and/or the center point of the FoV at the specified point in time (or recording point in time).
  • the simulated input may be generated based on the stored tracking data and the data representing the center point of the FoV according to the pose.
  • the electronic device (400) can receive a simulation input of a specified type (e.g., eye tracking) based on a first center point of the FoV within a three-dimensional virtual space while at least partially limiting the execution of a function corresponding to the input based on a specified type of tracking data (e.g., eye tracking) (or, during a simulation mode).
  • a specified type e.g., eye tracking
  • the simulated input may be received while at least partially limiting the execution of a function corresponding to the input based on a specified type of tracking data.
  • the input based on the specified type of tracking data may be one of: input for head tracking, input for hand tracking, input for eye tracking, or input for face tracking.
  • the receiving of the simulated input while at least partially limiting the execution of the function corresponding to the input based on the specified type of tracking data may include that the reception of other types of inputs via the input circuit (e.g., the camera (425) and/or the sensor (430)) is not limited.
  • the receiving of the simulated input while at least partially limiting the execution of the function corresponding to the input based on the specified type of tracking data may include that the electronic device (400) may receive other types of inputs via the input circuit (e.g., the camera (425) and/or the sensor (430)) to the platform layer (510).
  • the receipt of a simulation input while at least partially limiting the execution of a function corresponding to the input based on a specified type of tracking data may include the electronic device (400) being able to perform a function corresponding to another input based on the receipt of another input.
  • the electronic device (400) may convert the simulation input into a specified type of input based on the virtual space provided to the user. In one embodiment, the electronic device (400) may convert the simulation input into an input for the three-dimensional virtual space of the electronic device (400) through the input conversion module (625).
  • the electronic device (400) may adjust the position where the simulation input is input based on the difference between the center point of the FoV based on the user's pose and the center point of the FoV according to the simulation input.
  • the present invention is not limited thereto.
  • the electronic device (400) may change the FoV based on the user's pose to the FoV according to the simulation input based on the difference between the center point of the FoV based on the user's pose and the FoV according to the simulation input.
  • the electronic device (400) may perform a function corresponding to a specified type of input.
  • the electronic device (400) may perform a function corresponding to an input converted from a simulation input at a position adjusted based on the simulation input.
  • the electronic device (400) may change the FoV within the 3D virtual space in which the display (420) is displayed. For example, based on a simulation input related to hand tracking being an input for selecting a virtual object, the electronic device (400) may execute a visual object within the 3D virtual space.
  • FIG. 20A illustrates an example of an operation of an electronic device according to an embodiment of the present invention to identify a user's gaze within a virtual space based on the position of the user's eyes.
  • FIG. 20B illustrates an example of an operation of an electronic device according to an embodiment of the present invention to identify a user's gaze within a virtual space based on the position of the user's eyes.
  • the electronic device (400) can identify coordinates of a target object (2030).
  • the electronic device (400) can identify coordinates of a target object (2030) generated through a tester application (710).
  • the electronic device (400) can identify 2D coordinates of a target object (2030) within a 2D panel (2020) (or 2D space) generated through the tester application (710).
  • the 2D panel (2020) can represent a 2D space rendered by the tester application (710).
  • the electronic device (400) can convert the 2D coordinates of the target object (2030) into 3D coordinates.
  • the electronic device (400) can convert the 2D coordinates of the target object (2030) within the 2D panel (2020) (or 2D space) into 3D coordinates within the 3D space (2010).
  • the electronic device (400) can convert the 2D coordinates of the target object (2030) into 3D coordinates within the 3D space (2010) based on the position information of the 2D panel (2020).
  • the electronic device (400) can convert the 2D coordinates of the target object (2030) into 3D coordinates within the 3D space (2010) by adding the coordinates in the depth direction among the coordinates within the 3D space of the 2D panel (2020) to the 2D coordinates of the target object (2030).
  • the electronic device (400) can convert the 2D coordinates of the input device (2040) into 3D coordinates.
  • the electronic device (400) can convert the 2D coordinates of the input device (2040) into 3D coordinates within a 3D space (2010).
  • the input device (2040) is an eye tracking device
  • the electronic device (400) can convert the 2D coordinates of the input device (2040) into 3D coordinates within a 3D space (2010) through the coordinate conversion method described with reference to FIGS. 13A and 13B.
  • the electronic device (400) can calculate a direction between the 3D coordinates of the target object (2030) and the 3D coordinates of the input device (2040).
  • the electronic device (400) can calculate a direction between the 3D coordinates of the target object (2030) and the 3D coordinates of the input device (2040) within a 3D space (2010).
  • the electronic device (400) can calculate a direction from the input device (2040) to the target object (2030) through the 3D coordinates of the target object (2030) and the 3D coordinates of the input device (2040) within the same 3D space (2010).
  • the electronic device (400) may perform an action (e.g., input, and/or gesture) (e.g., turning the head and/or gaze, input to a virtual object (e.g., drag, pinch)) of the input device (2040) with respect to the 3D coordinates of the target object (2030).
  • an action e.g., input, and/or gesture
  • a virtual object e.g., drag, pinch
  • the wearable device (400) may include a display assembly (420) including displays respectively arranged toward the two eyes of the user when worn, an input circuit (425, 430) configured to acquire a specified type of tracking data of the user, at least one processor (410) including a processing circuit, and a memory (415) storing instructions and including one or more storage media.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to receive a simulated input of the specified type based on a first center point of a field of view (FoV) within a three-dimensional virtual space while at least partially limiting execution of a function corresponding to the input based on the specified type of tracking data.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to identify the specified type of input at a second location adjusted from the first location where the simulation input was input, based on a difference between the first center point and a second center point of another FoV based on the pose of the user.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to perform the function corresponding to the input identified at the second location based on the simulation input.
  • the wearable device (400) may include another input circuit (425, 430) configured to receive a different type of input specified by the user.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to receive, through the other input circuit (425, 430), the different type of input specified while at least partially limiting the execution of a function corresponding to the input based on the tracking data of the different type.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to perform, based on the receipt of the different input, a different function corresponding to the different input.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to receive an input for a two-dimensional planar space.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to receive a simulation input by converting the input for the planar space into the simulation input for the virtual space through the input conversion module (625).
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to obtain, through the input circuit (425, 430), the tracking data of the specified type toward a virtual object displayed at a specified location within the virtual space.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to convert the simulation input converted through the input conversion module (625) into other tracking data.
  • the instructions, when individually or collectively executed by the at least one processor (410) may cause the wearable device (400) to update an input conversion algorithm of the input conversion module (625) such that a difference between the tracking data and the other tracking data is reduced.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to record, in the memory (415), the specified type of tracking data in the virtual space received via the input circuit (425, 430).
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to store, in the memory (415), data representing the first center point of the FoV according to different poses of the user when recording of the specified type of tracking data begins.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to generate the simulation input based on the stored tracking data and the data representing the first center point while at least partially limiting the execution of a function corresponding to the input based on the specified type of tracking data.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to receive the generated simulation input.
  • the above-mentioned specified type of tracking data may be tracking data for the gaze of the user's two eyes.
  • the instructions, when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to identify a location of a virtual object within the virtual space.
  • the instructions, when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to identify locations of the user's two eyes within the cognitive space based on cameras photographing the user's two eyes.
  • the instructions, when individually or collectively executed by the at least one processor (410) may cause the wearable device (400) to convert a location of the virtual object within the virtual space into a location of the virtual object within the cognitive space.
  • the instructions, when individually or collectively executed by the at least one processor (410) may cause the wearable device (400) to identify the gaze based on the position of the virtual object within the cognitive space and the positions of the two eyes of the user.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to change the other FoV based on the pose of the user so that the other FoV represents the FoV based on the first center point while at least partially limiting the execution of a function corresponding to an input based on the tracking data of the specified type.
  • the above-mentioned type of input may be one of input for head tracking, input for hand tracking, input for eye tracking, or input for face tracking.
  • the above instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to receive the simulation input of the specified type from an external electronic device via the communication circuit.
  • the simulated input may be a sequence of two or more inputs acquired during a specified first time period.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to, based on a request to reproduce the simulated input acquired during the first time period during a second time period different from the first time period, while at least partially limiting the execution of a function corresponding to the input based on the specified type of tracking data: transform the two or more inputs acquired at points in time within the first time period into two or more other inputs acquired at other points in time within the second time period.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to receive the specified type of input based on the two or more other inputs at each cycle during which it receives an input through the input circuit (425, 430) within the second time period.
  • the method can be performed in a wearable device (400) including a display assembly (420) having displays respectively arranged toward the two eyes of a user when worn, and an input circuit (425, 430) configured to receive a specified type of input from the user.
  • the method can include: receiving a simulated input of the specified type based on a first center point of a field of view (FoV) in a three-dimensional virtual space while at least partially limiting the execution of a function corresponding to the input based on the tracking data of the specified type.
  • FoV field of view
  • the method can include identifying the specified type of input at a second location adjusted from a first location where the simulated input was input based on a difference between the first center point and a second center point of another FoV based on a pose of the user.
  • the method can include performing the function corresponding to the input identified at the second location based on the simulated input.
  • the electronic device (400) may include another input circuit (425, 430) configured to receive a different type of input specified by the user.
  • the method may include: receiving, through the other input circuit (425, 430), the different type of input specified while at least partially limiting the execution of a function corresponding to the input based on the tracking data of the different type.
  • the method may include performing, based on the receipt of the different input, a different function corresponding to the different input.
  • the method may include an operation of receiving an input for a two-dimensional planar space.
  • the method may include an operation of receiving the simulation input by converting the input for the planar space into the simulation input for the virtual space through an input conversion module (625).
  • the method may include an operation of obtaining the tracking data of the specified type toward a virtual object displayed at a specified location within the virtual space through the input circuit (425, 430).
  • the method may include an operation of converting the simulation input converted through the input conversion module (625) into other tracking data.
  • the method may include an operation of updating an input conversion algorithm of the input conversion module (625) so that a difference between the tracking data and the other tracking data is reduced.
  • the method may include an operation of recording the specified type of tracking data in the virtual space received through the input circuit (425, 430) into a memory (415).
  • the method may include an operation of storing data representing the first center point of the FoV according to different poses of the user when recording of the specified type of tracking data starts in the memory (415).
  • the method may include an operation of generating the simulation input based on the stored tracking data and the data representing the first center point while at least partially limiting execution of a function corresponding to an input based on the specified type of tracking data.
  • the method may include an operation of receiving the generated simulation input.
  • the tracking data of the above-mentioned specified type may be tracking data for the gaze of the two eyes of the user.
  • the method may include an operation of identifying a position of a virtual object within the virtual space.
  • the method may include an operation of identifying positions of the two eyes of the user within a cognitive space based on cameras that capture the two eyes of the user.
  • the method may include an operation of converting a position of the virtual object within the virtual space into a position of the virtual object within the cognitive space.
  • the method may include an operation of identifying the gaze based on the position of the virtual object within the cognitive space and the positions of the two eyes of the user.
  • the method may include an operation of changing the other FoV based on the pose of the user so that the other FoV represents the FoV based on the first center point, while at least partially limiting the execution of a function corresponding to the input based on the tracking data of the specified type.
  • the above-mentioned type of input may be one of input for head tracking, input for hand tracking, input for eye tracking, or input for face tracking.
  • the method may include an operation of receiving the simulation input of the specified type from an external electronic device through a communication circuit of the wearable device (400).
  • the above simulation input may be a sequence of two or more inputs acquired during a specified first time period.
  • the method may include an operation of: converting the two or more inputs acquired at points in time within the first time period into two or more different inputs acquired at different points in time within the second time period, based on a request for reproducing the simulation input acquired during the first time period during a second time period different from the first time period, while at least partially limiting the execution of a function corresponding to the input based on the specified type of tracking data.
  • the method may include an operation of receiving the specified type of input based on the two or more different inputs at each cycle of receiving the input through the input circuit (425, 430) within the second time period.
  • a non-transitory computer readable storage medium as described above can store a program including instructions.
  • the instructions when executed individually or collectively by at least one processor (410) of a wearable device (200), comprising a display assembly (420) including displays respectively arranged toward two eyes of a user when worn, and an input circuit (425, 430) configured to obtain a specified type of tracking data of the user, can cause the wearable device (400) to receive a simulated input of the specified type based on a first center point of a field of view (FoV) within a three-dimensional virtual space while at least partially limiting execution of a function corresponding to the input based on the specified type of tracking data.
  • a processor when executed individually or collectively by at least one processor (410) of a wearable device (200), comprising a display assembly (420) including displays respectively arranged toward two eyes of a user when worn, and an input circuit (425, 430) configured to obtain a specified type of tracking data of the user, can cause the wearable device (400) to
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to identify the specified type of input at a second location adjusted from the first location where the simulation input was input, based on a difference between the first center point and a second center point of another FoV based on the pose of the user.
  • the instructions when individually or collectively executed by the at least one processor (410), may cause the wearable device (400) to perform the function corresponding to the input identified at the second location based on the simulation input.
  • Electronic devices may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.
  • first,” “second,” or “first” or “second” may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order).
  • a component e.g., a first component
  • another component e.g., a second component
  • functionally e.g., a third component
  • module used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit.
  • a module may be an integral component, or a minimum unit or part of such a component that performs one or more functions.
  • a module may be implemented in the form of an application-specific integrated circuit (ASIC).
  • ASIC application-specific integrated circuit
  • Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)).
  • a processor e.g., a processor (120)
  • the machine e.g., an electronic device (101)
  • the one or more instructions may include code generated by a compiler or code executable by an interpreter.
  • the machine-readable storage medium may be provided in the form of a non-transitory storage medium.
  • 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.
  • the method according to various embodiments disclosed in this document may be provided as included in a computer program product.
  • the computer program product may be traded as a product between a seller and a buyer.
  • the computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., by download or upload) through an application store (e.g., Play Store TM ) or directly between two user devices (e.g., smart phones).
  • an application store e.g., Play Store TM
  • at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.
  • each component e.g., a module or a program of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components.
  • one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added.
  • a plurality of components e.g., a module or a program
  • the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration.
  • the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Optics & Photonics (AREA)
  • Ophthalmology & Optometry (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

Est divulgué un dispositif à porter sur soi. Le dispositif à porter sur soi peut comprendre : un ensemble d'affichage comprenant des unités d'affichage qui font respectivement face aux deux yeux d'un utilisateur lorsqu'il est porté ; et un circuit d'entrée configuré pour acquérir un type désigné de données de suivi de l'utilisateur. Le dispositif à porter sur soi peut recevoir le type désigné d'entrée de simulation sur la base d'un premier point central d'un FoV dans un espace virtuel tridimensionnel pendant que l'exécution d'une fonction correspondant à une entrée basée sur le type désigné de données de suivi est au moins partiellement limitée. Le dispositif à porter sur soi peut identifier le type désigné d'entrée à une seconde position, ajustée à partir d'une première position à laquelle l'entrée de simulation a été effectuée, sur la base de la différence entre le premier point central et un second point central d'un autre FoV basé sur une pose de l'utilisateur. Le dispositif à porter sur soi peut effectuer la fonction, correspondant à l'entrée identifiée au niveau de la seconde position, sur la base de l'entrée de simulation.
PCT/KR2025/010220 2024-07-23 2025-07-11 Dispositif à porter sur soi, procédé et support de stockage non transitoire lisible par ordinateur pour traiter une entrée de simulation Pending WO2026023964A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
KR10-2024-0097374 2024-07-23
KR20240097374 2024-07-23
KR10-2024-0126302 2024-09-16
KR20240126302 2024-09-16
KR10-2024-0155718 2024-11-05
KR1020240155718A KR20260014444A (ko) 2024-07-23 2024-11-05 시뮬레이션 입력을 처리하는 웨어러블 장치, 방법, 및 비-일시적 컴퓨터 판독 가능 기록 매체

Publications (1)

Publication Number Publication Date
WO2026023964A1 true WO2026023964A1 (fr) 2026-01-29

Family

ID=98566134

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2025/010220 Pending WO2026023964A1 (fr) 2024-07-23 2025-07-11 Dispositif à porter sur soi, procédé et support de stockage non transitoire lisible par ordinateur pour traiter une entrée de simulation

Country Status (1)

Country Link
WO (1) WO2026023964A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190141696A (ko) * 2017-04-19 2019-12-24 매직 립, 인코포레이티드 웨어러블 시스템을 위한 멀티모달 임무 실행 및 텍스트 편집
KR102437276B1 (ko) * 2020-10-15 2022-08-30 (주)이머시브캐스트 바디 움직임 기반의 클라우드 vr 장치 및 방법
KR20220155071A (ko) * 2021-05-14 2022-11-22 (주)이머시브캐스트 적응적인 fov 렌더링 방법
KR20240028262A (ko) * 2022-08-24 2024-03-05 삼성전자주식회사 카메라 모듈을 제어하는 웨어러블 전자 장치 및 그 동작 방법
KR20240030885A (ko) * 2022-08-29 2024-03-07 삼성전자주식회사 전자 장치의 입력에 기반하여 웨어러블 장치를 제어하기 위한 전자 장치 및 그 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190141696A (ko) * 2017-04-19 2019-12-24 매직 립, 인코포레이티드 웨어러블 시스템을 위한 멀티모달 임무 실행 및 텍스트 편집
KR102437276B1 (ko) * 2020-10-15 2022-08-30 (주)이머시브캐스트 바디 움직임 기반의 클라우드 vr 장치 및 방법
KR20220155071A (ko) * 2021-05-14 2022-11-22 (주)이머시브캐스트 적응적인 fov 렌더링 방법
KR20240028262A (ko) * 2022-08-24 2024-03-05 삼성전자주식회사 카메라 모듈을 제어하는 웨어러블 전자 장치 및 그 동작 방법
KR20240030885A (ko) * 2022-08-29 2024-03-07 삼성전자주식회사 전자 장치의 입력에 기반하여 웨어러블 장치를 제어하기 위한 전자 장치 및 그 방법

Similar Documents

Publication Publication Date Title
WO2024117649A1 (fr) Dispositif vestimentaire pour afficher un contenu multimédia sur la base d'une forme de préhension par rapport à un objet externe, et procédé associé
WO2026023964A1 (fr) Dispositif à porter sur soi, procédé et support de stockage non transitoire lisible par ordinateur pour traiter une entrée de simulation
WO2025244289A1 (fr) Dispositif à porter sur soi et procédé d'affichage d'interface utilisateur, et support de stockage non transitoire lisible par ordinateur
WO2026023845A1 (fr) Dispositif à porter sur soi, procédé et support de stockage non transitoire lisible par ordinateur pour se connecter à un dispositif électronique externe
WO2025070993A1 (fr) Dispositif portable, procédé et support de stockage non transitoire lisible par ordinateur pour entrée de geste
WO2025225926A1 (fr) Dispositif portable, procédé et support de stockage non transitoire lisible par ordinateur pour générer un mouvement global d'avatar
WO2025023480A1 (fr) Dispositif électronique, procédé et support de stockage lisible par ordinateur pour changer un écran sur la base d'une commutation d'espace virtuel
WO2025023479A1 (fr) Dispositif électronique, procédé et support de stockage lisible par ordinateur pour afficher un objet visuel associé à une application dans un espace virtuel
WO2024181695A1 (fr) Dispositif électronique et procédé de fourniture de fonction associée à une réalité étendue
WO2026034748A1 (fr) Dispositif à porter sur soi, procédé, ainsi que et support d'enregistrement non transitoire lisible par ordinateur permettant de reconnaître des données d'un dispositif électronique en tant qu'entrée
WO2026029319A1 (fr) Dispositif électronique, procédé et support de stockage non transitoire lisible par ordinateur pour interagir avec un dispositif portable
WO2025037953A1 (fr) Dispositif électronique et procédé de gestion et d'affichage d'avatars de multiples fournisseurs
WO2025023451A1 (fr) Dispositif portable, procédé et support de stockage non transitoire lisible par ordinateur pour interaction avec le regard de l'utilisateur
WO2025216484A1 (fr) Dispositif portable, procédé et support de stockage non transitoire lisible par ordinateur permettant de modifier un schéma pour l'affichage d'un avatar
WO2026059040A1 (fr) Dispositif portable, procédé et support de stockage non transitoire lisible par ordinateur pour afficher un écran sur la base de la vue de l'utilisateur
WO2026043146A1 (fr) Dispositif électronique pouvant être porté sur la tête, procédé et support de stockage non transitoire lisible par ordinateur pour afficher une représentation visuelle correspondant à la main
WO2026043081A1 (fr) Dispositif portable, procédé et support de stockage non transitoire lisible par ordinateur pour acquérir une image en fonction du regard d'un utilisateur
WO2026095355A1 (fr) Dispositif électronique, procédé, et support de stockage non transitoire lisible par ordinateur pour commander un écran affiché sur une unité d'affichage
WO2026034811A1 (fr) Dispositif portable, procédé et support de stockage non transitoire lisible par ordinateur pour afficher un objet visuel correspondant à un objet externe
WO2025042048A1 (fr) Dispositif portable et procédé de réglage de luminosité d'environnement, et support d'enregistrement non transitoire lisible par ordinateur
WO2025110582A1 (fr) Appareil et procédé de mesure de distance à un objet externe
WO2025048141A1 (fr) Dispositif à porter sur soi et procédé d'identification de l'emplacement d'un objet cible
WO2025155001A1 (fr) Dispositif habitronique et procédé d'affichage de réalité étendue, et support d'enregistrement non transitoire lisible par ordinateur
WO2024215173A1 (fr) Dispositif électronique, procédé et support de stockage lisible par ordinateur pour afficher un écran correspondant à la taille d'un objet externe sur un écran
WO2025150657A1 (fr) Dispositif électronique et procédé de fourniture de contenu en perspective à la troisième personne

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25845060

Country of ref document: EP

Kind code of ref document: A1