EP4445248A1 - Restitution différée sur des dispositifs de réalité étendue (xr) - Google Patents

Restitution différée sur des dispositifs de réalité étendue (xr)

Info

Publication number
EP4445248A1
EP4445248A1 EP22916836.4A EP22916836A EP4445248A1 EP 4445248 A1 EP4445248 A1 EP 4445248A1 EP 22916836 A EP22916836 A EP 22916836A EP 4445248 A1 EP4445248 A1 EP 4445248A1
Authority
EP
European Patent Office
Prior art keywords
server
mode
media
immersive
pose information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP22916836.4A
Other languages
German (de)
English (en)
Other versions
EP4445248A4 (fr
Inventor
Christopher Anthony Peri
Eric Yip
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of EP4445248A1 publication Critical patent/EP4445248A1/fr
Publication of EP4445248A4 publication Critical patent/EP4445248A4/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1069—Session establishment or de-establishment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • H04L65/612—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for unicast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • H04L65/613—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for the control of the source by the destination
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80—Responding to QoS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/141—Setup of application sessions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/10—Architectures or entities
    • H04L65/1063—Application servers providing network services

Definitions

  • This disclosure generally relates to extended reality (XR) devices and processes. More specifically, this disclosure relates to an XR device and a method for deferred rendering on the XR device.
  • XR extended reality
  • 5G mobile communication technologies define broad frequency bands such that high transmission rates and new services are possible, and can be implemented not only in “Sub 6GHz” bands such as 3.5GHz, but also in “Above 6GHz” bands referred to as mmWave including 28GHz and 39GHz.
  • 6G mobile communication technologies referred to as Beyond 5G systems
  • terahertz bands for example, 95GHz to 3THz bands
  • IIoT Industrial Internet of Things
  • IAB Integrated Access and Backhaul
  • DAPS Dual Active Protocol Stack
  • 5G baseline architecture for example, service based architecture or service based interface
  • NFV Network Functions Virtualization
  • SDN Software-Defined Networking
  • MEC Mobile Edge Computing
  • multi-antenna transmission technologies such as Full Dimensional MIMO (FD-MIMO), array antennas and large-scale antennas, metamaterial-based lenses and antennas for improving coverage of terahertz band signals, high-dimensional space multiplexing technology using OAM (Orbital Angular Momentum), and RIS (Reconfigurable Intelligent Surface), but also full-duplex technology for increasing frequency efficiency of 6G mobile communication technologies and improving system networks, AI-based communication technology for implementing system optimization by utilizing satellites and AI (Artificial Intelligence) from the design stage and internalizing end-to-end AI support functions, and next-generation distributed computing technology for implementing services at levels of complexity exceeding the limit of UE operation capability by utilizing ultra-high-performance communication and computing resources.
  • FD-MIMO Full Dimensional MIMO
  • OAM Organic Angular Momentum
  • RIS Reconfigurable Intelligent Surface
  • VR and AR multimedia typically require a user to wear a corresponding VR or AR headset, where the user is presented with a virtual world or augmented features localized into the real world such that the augmented features appear to be a part of the real world.
  • This disclosure relates to deferred rendering on extended reality (XR) devices.
  • a method for deferred rendering on an XR device includes establishing a transport session for content on the XR device with a server. The method also includes performing a loop configuration for the content based on the transport session between the XR device and the server. The method further includes providing pose information based on parameters of the loop configuration to the server. The method also includes receiving pre-rendered content based on the pose information from the server. In addition, the method includes processing and displaying the pre-rendered content on the XR device.
  • an XR device in a second embodiment, includes a transceiver configured to communicate with a server and at least one processing device operably coupled to the transceiver.
  • the at least one processing device is configured to establish a transport session for content on the XR device with the server.
  • the at least one processing device is also configured to perform a loop configuration for the content based on the transport session between the XR device and the server.
  • the at least one processing device is further configured to provide pose information based on parameters of the loop configuration to the server.
  • the at least one processing device is also configured to receive pre-rendered content based on the pose information from the server.
  • the at least one processing device is configured to process and display the pre-rendered content on the XR device.
  • a non-transitory machine readable medium contains instructions that when executed cause at least one processor to establish a transport session for content on an XR device with a server.
  • the non-transitory machine readable medium also contains instructions that when executed cause the at least one processor to perform a loop configuration for the content based on the transport session between the XR device and the server.
  • the non-transitory machine readable medium further contains instructions that when executed cause the at least one processor to provide pose information based on parameters of the loop configuration to the server.
  • the non-transitory machine readable medium also contains instructions that when executed cause the at least one processor to receive pre-rendered content based on the pose information from the server.
  • the non-transitory machine readable medium contains instructions that when executed cause the at least one processor to process and display the pre-rendered content on the XR device.
  • various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium.
  • application and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code.
  • computer readable program code includes any type of computer code, including source code, object code, and executable code.
  • computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.
  • ROM read only memory
  • RAM random access memory
  • CD compact disc
  • DVD digital video disc
  • a "non-transitory” computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals.
  • a non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device.
  • phrases such as “have,” “may have,” “include,” or “may include” a feature indicate the existence of the feature and do not exclude the existence of other features.
  • the phrases “A or B,” “at least one of A and/or B,” or “one or more of A and/or B” may include all possible combinations of A and B.
  • “A or B,” “at least one of A and B,” and “at least one of A or B” may indicate all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.
  • first and second may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another.
  • a first user device and a second user device may indicate different user devices from each other, regardless of the order or importance of the devices.
  • a first component may be denoted a second component and vice versa without departing from the scope of this disclosure.
  • the phrase “configured (or set) to” may be interchangeably used with the phrases “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of” depending on the circumstances.
  • the phrase “configured (or set) to” does not essentially mean “specifically designed in hardware to.” Rather, the phrase “configured to” may mean that a device can perform an operation together with another device or parts.
  • the phrase “processor configured (or set) to perform A, B, and C” may mean a generic-purpose processor (such as a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (such as an embedded processor) for performing the operations.
  • Examples of an "electronic device” may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch).
  • PDA personal digital assistant
  • PMP portable multimedia player
  • MP3 player MP3 player
  • a mobile medical device such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch.
  • Other examples of an electronic device include a smart home appliance.
  • Examples of the smart home appliance may include at least one of a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washer, a drier, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (such as SAMSUNG HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or speaker with an integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a gaming console (such as an XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.
  • a television such as SAMSUNG HOMESYNC, APPLETV, or GOOGLE TV
  • a smart speaker or speaker with an integrated digital assistant such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON
  • an electronic device include at least one of various medical devices (such as diverse portable medical measuring devices (like a blood sugar measuring device, a heartbeat measuring device, or a body temperature measuring device), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, a sailing electronic device (such as a sailing navigation device or a gyro compass), avionics, security devices, vehicular head units, industrial or home robots, automatic teller machines (ATMs), point of sales (POS) devices, or Internet of Things (IoT) devices (such as a bulb, various sensors, electric or gas meter, sprinkler, fire alarm, thermostat, street light, toaster, fitness equipment, hot water tank, heater, or boiler).
  • MRA magnetic resource
  • an electronic device include at least one part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or various measurement devices (such as devices for measuring water, electricity, gas, or electromagnetic waves).
  • an electronic device may be one or a combination of the above-listed devices.
  • the electronic device may be a flexible electronic device.
  • the electronic device disclosed here is not limited to the above-listed devices and may include any other electronic devices now known or later developed.
  • the term "user” may denote a human or another device (such as an artificial intelligent electronic device) using the electronic device.
  • FIGURE 1 illustrates an example network configuration including an electronic device in accordance with this disclosure
  • FIGURE 2 illustrates example use cases for extended reality (XR) devices in accordance with this disclosure
  • FIGURES 3A and 3B illustrate an example technique for rendering immersive media by an XR device in accordance with this disclosure
  • FIGURES 4A and 4B illustrate an example technique for rendering immersive media using server assistance in accordance with this disclosure
  • FIGURES 5A and 5B illustrate an example technique for using a media session loop between a user equipment (UE) and a server in accordance with this disclosure
  • FIGURES 6A and 6B illustrate an example environment for device functions related to pose information delivery configuration in accordance with this disclosure
  • FIGURE 7 illustrates an example technique for pose information delivery configuration and frame recycling decisions by a UE in accordance with this disclosure
  • FIGURE 8 illustrates an example graphical representation of object safe boundary description metadata in accordance with this disclosure
  • FIGURE 9 illustrates an example system for efficiently communicating with a remote computing system and an immersive device in accordance with this disclosure
  • FIGURE 10 illustrates an example comprehensive computer vision system in accordance with this disclosure
  • FIGURE 11 illustrates an example software stack for an immersive device in accordance with this disclosure
  • FIGURE 12 illustrates an example method for deferred rendering on an immersive device that is tethered to an electronic device in accordance with this disclosure
  • FIGURE 13 illustrates another example method for deferred rendering on an immersive device in accordance with this disclosure.
  • FIGURES 1 through 13 described below, and the various embodiments of this disclosure are described with reference to the accompanying drawings. However, it should be appreciated that this disclosure is not limited to these embodiments and all changes and/or equivalents or replacements thereto also belong to the scope of this disclosure.
  • VR and AR multimedia typically require a user to wear a corresponding VR or AR headset, where the user is presented with a virtual world or augmented features localized into the real world such that the augmented features appear to be a part of the real world.
  • Multimedia content processing can include various functions (such as authoring, pre-processing, post-processing, metadata generation, delivery, decoding, and rendering) of VR, AR, and mixed reality (MR) contents.
  • VR, AR, and MR are generally referred to collectively as extended reality (XR).
  • XR contents can include two-dimensional (2D) videos, 360o videos, and three-dimensional (3D) media represented by point clouds and meshes.
  • Multimedia contents can include scene descriptions, dynamic scene descriptions, dynamic scene descriptions supporting timed media, and scene description formats (such as Graphics Language Transmission Format or "glTF,” Moving Picture Experts Group or "MPEG,” and ISO Base Media File Format or "ISOBMFF" file formats).
  • the multimedia contents can include support for immersive contents and media, split rendering between AR glasses, split rendering between a tethered device and a cloud/edge server, etc.
  • Various improvements in media contents can include rendering resource optimization that considers pose information, content properties, re-projection, etc.
  • Various improvements in media contents can also include hardware resource optimization that considers operating modes between an application, a remote computer/server, and an XR device.
  • a scene description is typically represented by a scene graph, such as in a format using glTF or Universal Scene Description (USD).
  • a scene graph describes objects in a scene, including their various properties like their locations, textures, and other information.
  • a glTF scene graph expresses this information as a set of nodes that can be represented as a node graph.
  • the exact format used for glTF is the JavaScript Object Notation (JSON) format, meaning that a glTF file is stored as a JSON document.
  • JSON JavaScript Object Notation
  • a specific challenge in immersive media rendering is related to the form factor of XR devices, such as AR devices that typically resemble a pair of glasses. Due to this type of form factor, design restrictions on weight, bulkiness, and overheating related to portability and comfort can affect the overall battery life and capabilities of the devices. Unfortunately, high processing requirements for rendering and displaying immersive contents conflict with battery-life expectations of consumers, especially for glasses-type wearable devices that can be worn even when a fully-immersive XR experience is not required. In other words, the processing capabilities for some XR devices can be limited in order to extend the battery life of the XR devices.
  • Existing technologies for AR glasses are often derived from VR headsets, which do not have the same limits in processing powers and battery lives.
  • compensation for processing can be provided using off-device rendering, such as when rendering operations are performed by a tethered smartphone or other tethered device, on a server, or in the cloud/server.
  • current pose information of AR glasses can be sent to a remote or external rendering entity.
  • pose information can be sent at a relatively high frequency (such as up to 1 KHz or more).
  • the rendering entity uses the latest pose information in order to render the latest media frame.
  • the rendered frame is sent to the AR glasses and corrected using the latest pose information to compensate for the latency between the rendering and the presentation of the frame.
  • the pose information can be redundant, such as when the motion of the AR glasses is minimal and a new rendered frame is unnecessary or when properties of the immersive content allow for re-projection by the AR glasses.
  • Many current immersive devices also have a number of sensors and solutions that allow for performing operations using six degrees of freedom (DoF) while maintaining a high frame rate. These operations may support head, hand, and eye tracking; full mapping of an environment; artificial intelligence (AI)-based object and face recognition; and body detection. Many of these sensors represent optical-based sensors, which can consume quite a bit of power. These sensors and the processing powers needed to support them place significant loads on the batteries of XR devices, such as wireless AR devices. In addition, running these systems generate significant heat, which in turns requires additional cooling solutions.
  • DoF degrees of freedom
  • Optimizing rendering resources can include providing pose information delivery configuration modes, frame rendering and delivery conditions and decisions (including the use of re-projection algorithms), and multi-split rendering modes (depending on the device configuration and service).
  • Optimizing hardware resources can include providing operational modes, computer vision system optimizations, and operational mode engine decisions.
  • techniques for defining and communicating modes of operation for an XR device are provided such that each hardware/software mode can optimize its functionality to allow for efficient operation while still maintaining performance key performance indexes (KPIs) that are expected for the current operational mode. This can include efficient operations related to head poses, hand poses, eye tracking, device tracking, sensor frequency, etc.
  • KPIs performance key performance indexes
  • this disclosure provides procedures and call flows for pose delivery configuration modes, XR device operation procedures for pose triggers and frame recycling (re-projection), and media description properties and metadata that enable pose modes, frame recycling, and multi-split rendering modes.
  • this disclosure also specifies hardware resource optimization operational modes for different XR use cases, XR device software and hardware stacks for operational mode decisions, and component-based computer vision systems supporting multiple operational modes.
  • This disclosure enables support for pose information delivery configuration modes, conditional and selective frame recycling by an immersive media non-rendering entity, device operation procedures and media description metadata properties, hardware resource optimization operational modes, XR device software and hardware stacks to support operational modes, and multi-component computer vision systems to support operational modes.
  • FIGURE 1 illustrates an example network configuration 100 including an electronic device in accordance with this disclosure.
  • the embodiment of the network configuration 100 shown in FIGURE 1 is for illustration only. Other embodiments of the network configuration 100 could be used without departing from the scope of this disclosure.
  • an electronic device 101 is included in the network configuration 100.
  • the electronic device 101 can include at least one of a bus 110, a processor 120, a memory 130, an input/output (I/O) interface 150, a display 160, a communication interface 170, and a sensor 180.
  • the electronic device 101 may exclude at least one of these components or may add at least one other component.
  • the bus 110 includes a circuit for connecting the components 120-180 with one another and for transferring communications (such as control messages and/or data) between the components.
  • the processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs).
  • the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processor unit (GPU).
  • the processor 120 is able to perform control on at least one of the other components of the electronic device 101 and/or perform an operation or data processing relating to communication or other functions. As described below, the processor 120 may be used to perform one or more functions related to deferred rendering of XR content.
  • the memory 130 can include a volatile and/or non-volatile memory.
  • the memory 130 can store commands or data related to at least one other component of the electronic device 101.
  • the memory 130 can store software and/or a program 140.
  • the program 140 includes, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and/or an application program (or "application”) 147.
  • At least a portion of the kernel 141, middleware 143, or API 145 may be denoted an operating system (OS).
  • OS operating system
  • the kernel 141 can control or manage system resources (such as the bus 110, processor 120, or memory 130) used to perform operations or functions implemented in other programs (such as the middleware 143, API 145, or application 147).
  • the kernel 141 provides an interface that allows the middleware 143, the API 145, or the application 147 to access the individual components of the electronic device 101 to control or manage the system resources.
  • the application 147 may include one or more applications that, among other things, perform one or more functions related to deferred rendering of XR content. These functions can be performed by a single application or by multiple applications that each carries out one or more of these functions.
  • the middleware 143 can function as a relay to allow the API 145 or the application 147 to communicate data with the kernel 141, for instance.
  • a plurality of applications 147 can be provided.
  • the middleware 143 is able to control work requests received from the applications 147, such as by allocating the priority of using the system resources of the electronic device 101 (like the bus 110, the processor 120, or the memory 130) to at least one of the plurality of applications 147.
  • the API 145 is an interface allowing the application 147 to control functions provided from the kernel 141 or the middleware 143.
  • the API 145 includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.
  • the I/O interface 150 serves as an interface that can, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic device 101.
  • the I/O interface 150 can also output commands or data received from other component(s) of the electronic device 101 to the user or the other external device.
  • the display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display.
  • the display 160 can also be a depth-aware display, such as a multi-focal display.
  • the display 160 is able to display, for example, various contents (such as text, images, videos, icons, or symbols) to the user.
  • the display 160 can include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.
  • the communication interface 170 is able to set up communication between the electronic device 101 and an external electronic device (such as a first electronic device 102, a second electronic device 104, or a server 106).
  • the communication interface 170 can be connected with a network 162 or 164 through wireless or wired communication to communicate with the external electronic device.
  • the communication interface 170 can be a wired or wireless transceiver or any other component for transmitting and receiving signals.
  • the electronic device 101 further includes one or more sensors 180 that can meter a physical quantity or detect an activation state of the electronic device 101 and convert metered or detected information into an electrical signal.
  • one or more sensors 180 can include one or more cameras or other imaging sensors, which may be used to capture images of scenes.
  • the sensor(s) 180 can also include one or more buttons for touch input, one or more microphones, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red-green-blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasound sensor, an iris sensor, or a fingerprint sensor.
  • a gesture sensor e.g., a gyroscope or gyro sensor
  • an air pressure sensor e.g., a gyroscope or gyro sensor
  • a magnetic sensor or magnetometer e.gyroscope or gy
  • the sensor(s) 180 can further include an inertial measurement unit, which can include one or more accelerometers, gyroscopes, and other components.
  • the sensor(s) 180 can include a control circuit for controlling at least one of the sensors included here. Any of these sensor(s) 180 can be located within the electronic device 101.
  • the first external electronic device 102 or the second external electronic device 104 can be a wearable device or an electronic device-mountable wearable device (such as an HMD).
  • the electronic device 101 can communicate with the electronic device 102 through the communication interface 170.
  • the electronic device 101 can be directly connected with the electronic device 102 to communicate with the electronic device 102 without involving with a separate network.
  • the electronic device 101 can also be an augmented reality wearable device, such as eyeglasses, that include one or more cameras.
  • the wireless communication is able to use at least one of, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (5G), millimeter-wave or 60 GHz wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol.
  • the wired connection can include, for example, at least one of a universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS).
  • the network 162 or 164 includes at least one communication network, such as a computer network (like a local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.
  • the first and second external electronic devices 102 and 104 and the server 106 each can be a device of the same or a different type from the electronic device 101.
  • the server 106 includes a group of one or more servers.
  • all or some of the operations executed on the electronic device 101 can be executed on another or multiple other electronic devices (such as the electronic devices 102 and 104 or server 106).
  • the electronic device 101 when the electronic device 101 should perform some function or service automatically or at a request, the electronic device 101, instead of executing the function or service on its own or additionally, can request another device (such as electronic devices 102 and 104 or server 106) to perform at least some functions associated therewith.
  • the other electronic device (such as electronic devices 102 and 104 or server 106) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device 101.
  • the electronic device 101 can provide a requested function or service by processing the received result as it is or additionally.
  • a cloud computing, distributed computing, or client-server computing technique may be used, for example. While FIGURE 1 shows that the electronic device 101 includes the communication interface 170 to communicate with the external electronic device 104 or server 106 via the network 162 or 164, the electronic device 101 may be independently operated without a separate communication function according to some embodiments of this disclosure.
  • the server 106 can include the same or similar components as the electronic device 101 (or a suitable subset thereof).
  • the server 106 can support to drive the electronic device 101 by performing at least one of operations (or functions) implemented on the electronic device 101.
  • the server 106 can include a processing module or processor that may support the processor 120 implemented in the electronic device 101.
  • the server 106 may be used to perform one or more functions related to deferred rendering of XR content.
  • FIGURE 1 illustrates one example of a network configuration 100 including an electronic device 101
  • the network configuration 100 could include any number of each component in any suitable arrangement.
  • computing and communication systems come in a wide variety of configurations, and FIGURE 1 does not limit the scope of this disclosure to any particular configuration.
  • FIGURE 1 illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.
  • FIGURE 2 illustrates example use cases 200 for XR devices in accordance with this disclosure.
  • the XR devices are represented by AR glasses, although XR devices of other forms may be used here.
  • user equipment (UE) 202 and a server 204 can exchange pose information 206 and rendered media 208.
  • the UE 202 may represent one or more electronic devices of FIGURE 1, such as the electronic device 101.
  • the server 204 may represent the server 106 of FIGURE 1.
  • the UE 202 can include standalone AR glasses 210 that can directly engage in network communications with the server 204.
  • the UE 202 can include tethered AR glasses 210 and a separate device containing a network modem enabling suitable connectivity between the AR glasses 210 and the server 204, such as a mobile smartphone or other tethered electronic device 212.
  • the AR glasses 210 can include a network modem enabling the AR glasses 210 to connect to the server 204 via a network connection without the use of any tethered electronic device 212.
  • pose information 206 is sent from the AR glasses 210 to the server 204 over the network connection.
  • the server 204 can use the latest pose information 206 to render immersive 3D media as 2D frames before encoding and sending the 2D rendered frames to the AR glasses 210.
  • the AR glasses 210 may not contain a network modem and instead may be connected to a tethered electronic device 212, such as via Bluetooth or Wi-Fi.
  • the tethered electronic device 212 contains a network modem enabling the tethered electronic device 212 to connect to the server 204 via a network connection.
  • pose information 206 from the AR glasses 210 is passed to the tethered electronic device 212, which forwards the pose information 206 to the server 204.
  • rendered media 208 from the server 204 is received by the tethered electronic device 212 and forwarded to the AR glasses 210.
  • the tethered electronic device 212 can also be additionally or exclusively used to render immersive media, in which case the pose information 206 from the AR glasses 210 may be sent only to the tethered electronic device 212 and may not be required by or forwarded to the server 204.
  • FIGURE 2 illustrates examples of use cases 200 for XR devices
  • XR devices may have any other suitable form factors
  • tethered XR devices may be used with any other suitable external components
  • XR devices may be used in any other suitable media rendering process and are not limited to the specific processes described above.
  • FIGURES 3A and 3B illustrate an example technique 300 for rendering immersive media by an XR device in accordance with this disclosure.
  • the technique 300 may, for example, be performed to provide immersive media to one or more XR devices such as the electronic device 101, which may represent the AR glasses 210.
  • a rendering system can include an immersive application 302, an immersive runtime 304, an immersive scene manager 306, media access functions 308 including a media client 310 and a media session handler 312, a network application function (AF) 314, a network application server (AS) 316, and an immersive application provider 318 including a scene server 320.
  • the immersive application 302 can represent at least one software application that integrates audio-visual content into a real-world environment.
  • the immersive runtime 304 can represent a set of functions that integrates with a platform to perform common operations, such as accessing controller or peripheral states, getting current and/or predicted tracking positions, performing general spatial computing, and submitting rendered frames to a display processing unit.
  • the scene manager 306 can support immersive rendering and scene graph handling functionalities.
  • the media access functions 308 can represent a set of functions that enable access to media and other immersive content-related data that is used by the immersive scene manager 306 or the immersive runtime 304 in order to provide an immersive experience.
  • the media access functions 308 can be divided into user data for the media client 310 and control data for the media session handler 312.
  • the network AF 314, network AS 316, and immersive application provider 318 can represent components used to provide a 5G Media Downlink Streaming (5GMSd) service in this example, although other services or mechanisms may be used to provide content.
  • 5GMSd 5G Media Downlink Streaming
  • scene content can be ingested by the network AS 316 in operation 322.
  • a service announcement can be triggered by the immersive application 302 in operation 324.
  • service access information (including media client entry) or a reference to the service access information can be provided through an M8d interface.
  • Desired immersive media content can be selected by the immersive application 302 in operation 326, and service access information can be acquired or updated in operation 328.
  • the immersive application 302 can initialize the scene manager with an entry point, which can be a scene description, in operation 330.
  • the media client 310 can establish a transport session for receiving the entry point or scene description in operation 332, and the media client can request and receive a full scene description in operation 334.
  • the immersive scene manager 306 can process the entry point or scene description in operation 336.
  • the immersive scene manager 306 can request creation of a new immersive session from the immersive runtime 304 in operation 338, and the immersive runtime 304 can create a new immersive session in operation 340.
  • Operations 342-356 describe an immersive media delivery pipeline that can be used to receive and render immersive scene and immersive scene updates.
  • the media client 310 and/or the immersive scene manager 306 can provide quality of service (QoS) information to the media session handler 312 in operation 342.
  • QoS quality of service
  • the media session handler 312 can share information with the network AF 314, in some cases including desired QoS information, in operation 344.
  • the network AF 314 may request QoS modifications to the PDU sessions.
  • a subprocess 346 can establish transport sessions and can receive and process delivery manifests and includes operations 348-352.
  • the media client 310 can establish one or more transport sessions to acquire delivery manifest information in operation 348.
  • the media client 310 can request and receive delivery manifests from the network AS 316 in operation 350, and the media client 310 can process the delivery manifests in operation 352.
  • the media client 310 can determine a number of needed transport sessions for media acquisition.
  • the media client 310 can be expected to be able to use the delivery manifest information to initialize media pipelines for each media stream.
  • the immersive scene manager 306 and media client 310 can configure rendering and delivery media pipelines in operation 354.
  • a subprocess 356 can provide latest pose information and can request, receive, and render media objects of the immersive scene in operations 358-370.
  • the media client 310 can establish one or more transport sessions to acquire the immersive media content in operation 358.
  • the latest pose information can be acquired by the immersive scene manager 306 and shared to the media client 310 in operation 360, and the media client 310 can request the immersive media data according to the delivery manifest processed in operation 362.
  • the request can include pose information, such as for viewpoint-dependent streaming.
  • the media client 310 can receive the immersive media data and can trigger one or more media rendering pipelines in operation 364.
  • the triggering of the media rendering pipeline(s) can include registration of immersive content accordingly into the real world.
  • the media client 310 can decode and process the media data in operation 366.
  • the media client 310 may also perform decryption.
  • the media client 310 can pass the media data to the immersive scene manager 306 in operation 368, and the immersive scene manager 306 can render the media and can pass the rendered media to the immersive runtime 304 in operation 370.
  • the immersive runtime 304 can perform further processing, such as registration of the immersive content into the real world and pose correction.
  • FIGURES 3A and 3B illustrate one example of a technique 300 for rendering immersive media by an XR device
  • various changes may be made to FIGURES 3A and 3B.
  • various operations in FIGURES 3A and 3B may overlap, occur in parallel, occur in a different order, or occur any number of times.
  • FIGURES 4A and 4B illustrate an example technique 400 for rendering immersive media using server assistance in accordance with this disclosure.
  • a rendering system can include many of the same components described above with respect to FIGURES 3A and 3B.
  • the immersive scene manager 306 has been replaced by an immersive lightweight scene manager 406.
  • the immersive lightweight scene manager 406 can represent a scene manager that is capable of handling a limited set of immersive media or 3D media.
  • the immersive lightweight scene manager 406 can require some form of pre-rendering by another element, such as an edge server or cloud server.
  • the technique 400 here can include the same operations 322-336 as described above, which are combined in FIGURES 4A and 4B for simplicity.
  • the network AS 316 can be selected and edge processes can be instantiated in operation 422.
  • the immersive lightweight scene manager 406 can send the scene description and the device capabilities to the network AS 316.
  • the network AS 316 can derive an edge application server (EAS) key performance index (KPI) and can select a new network AS 316 based on the new KPI.
  • the edge processes are started and a new entry point URL can be provided to the immersive lightweight scene manager 406.
  • the immersive lightweight scene manager 406 can derive the EAS KPIs from the scene description and the device capabilities.
  • the immersive lightweight scene manager 406 can also request the network AF 314 to provide a list of suitable network AS 316.
  • the immersive lightweight scene manager 406 can request a lightweight scene description in operation 424.
  • the edge processes derive the lightweight scene description from a full scene description and can provide the lightweight scene description to the immersive lightweight scene manager 406.
  • the lightweight scene manager 406 can process the simplified entry point or lightweight scene description in operation 426.
  • the operations 338-354 can be performed similarly in FIGURES 4A and 4B as in FIGURES 3A and 3B and are omitted here for simplicity.
  • the media client 310 can establish one or more transport sessions to acquire the immersive media content in operation 428.
  • the network AS 316 can initiate and start a media session in operation 430, and the media session can include a stateful session loop 402 specific to the UE 202.
  • the stateful session loop 402 can include operations 432-438.
  • the immersive lightweight scene manager 406 can acquire the latest pose information and share the pose information to the media client 310 in operation 432, and the media client 310 can send the latest pose information to the network AS 316 in operation 434.
  • the network AS 316 can perform pre-rendering of the media based on the latest received pose information and any original scene updates in operation 436.
  • the pre-rendering may include decoding and rendering of immersive media and encoding the rendered media.
  • the rendered media can be rendered 2D media.
  • the network AS 316 can send the pre-rendered media to the media client 310 in operation 438.
  • the pose information can be sent from the UE 202 to the server periodically during the media session loop, regardless of whether the pose information is used instantly or not during the pre-rendering operation. Pre-rendering can also be performed regardless of UE decisions or specific information related to the pose information.
  • the media client 310 can decode and process the media data in operation 440.
  • the media client 310 can perform decryption.
  • the media client 310 can pass the media data to the immersive lightweight scene manager 406 in operation 442.
  • the immersive lightweight scene manager 406 can render the media and can pass the rendered media to the immersive runtime 304 in operation 444.
  • the immersive runtime 304 can perform further processing, such as composition, pose correction, and registration of the immersive content into the real world.
  • FIGURES 4A and 4B illustrate one example of a technique for rendering immersive media using server assistance
  • various changes may be made to FIGURES 4A and 4B.
  • various operations in FIGURES 4A and 4B may overlap, occur in parallel, occur in a different order, or occur any number of times.
  • FIGURES 5A and 5B illustrate an example technique 500 for using a media session loop between a UE and a server in accordance with this disclosure.
  • a rendering system can include the same components described above with respect to FIGURES 4A and 4B.
  • the technique 500 here can include the same operations 322-336 as described above, which are combined in FIGURES 5A and 5B for simplicity. Additional operations described above for FIGURES 4A and 4B are also included in FIGURES 5A and 5B and not described here.
  • a media session loop 502 can include a loop configuration where the immersive runtime 304 configures properties of a newly-created media session loop 502 or loop reconfiguration where the immersive runtime 304 reconfigures properties of the media session loop 502 in operation 522.
  • properties for the media session loop 502 can include a pose information delivery configuration, a media session loop setting, a frame recycling flag, etc.
  • the pose information delivery configuration can include an offline mode, a periodic mode, a trigger mode, etc.
  • the offline mode can cause pose information to not be sent to the server 204.
  • Split-rendering may not be performed or pose information may not be necessary for split-rendering.
  • the periodic mode can cause pose information to be periodically sent from the UE 202 to the rendering entity or server.
  • a frequency of the pose information delivery can be set by the UE 202 through this parameter.
  • the trigger mode can cause pose information to be sent when triggered by the UE 202. Example conditions for triggering delivery of pose information are described in greater detail below with reference to FIGURE 7.
  • the media session loop setting can be used to control whether the UE 202 sends pose information to the server 204 using any of the pose information delivery configurations and whether the UE 202 receives pre-rendered media from the server 204.
  • the relationship between the receipt of pose information and the rendering of a current frame by a server 204 can be implementation-specific.
  • the media session loop setting can include a send pose variable (0,1) to indicate whether to send pose information and a receive media variable (0,1) to indicate whether to receive pre-rendered media from the server 204.
  • the frame recycling flag can indicate that a UE 202 is performing frame recycling.
  • FIGURES 5A and 5B illustrate one example of a technique 500 for using a media session loop between a UE and a server
  • various changes may be made to FIGURES 5A and 5B.
  • various operations in FIGURES 5A and 5B may overlap, occur in parallel, occur in a different order, or occur any number of times.
  • FIGURES 6A and 6B illustrate an example environment 600 for device functions related to pose information delivery configuration in accordance with this disclosure.
  • the environment 600 can include a UE 602, a cloud/edge server 604, and an immersive application provider 606.
  • the UE 602 may represent the electronic device 101, which may represent the AR glasses 210 or UE 202 described above.
  • the cloud/edge server 604 may represent the server 106, 204 described above.
  • the immersive application provider 606 may represent the immersive application provider 318 described above.
  • the UE 602 can include hardware, such as one or more sensors 608, one or more cameras 610, one or more user inputs 612, at least one display 614, and one or more speakers 616.
  • the UE 602 can also include software, such as immersive runtime 618, lightweight scene manager 620, media access functions 622, and an immersive application 624. These functions may represent the corresponding functions in FIGURES 3A through 5 described above.
  • the UE 602 can include 5G connectivity or other network connectivity provided through an embedded 5G modem and other 5G system components or other networking components.
  • the immersive runtime 618 is local to the UE 602 and uses data from the sensors 608 and other components, such as audio inputs and video inputs.
  • the immersive runtime 618 may be assisted by a cloud/edge application for spatial localization and mapping provided by a spatial computing service.
  • the immersive runtime 618 can control tracking and sensing functions and capturing functions in addition to immersive runtime functions.
  • the tracking and sensing functions can include inside-out tracking for six DoF user position, eye tracking, and hand tracking, such as by using the sensors 608 and cameras 610.
  • the capturing functions can include vision camera functions for capturing a surrounding environment for vision-related functions and media camera functions for capturing scenes of objects for media data generation.
  • the vision and media camera functions may be mapped to the same camera 610 or separate cameras 610.
  • at least one external camera 610 can be implemented on one or more other electronic devices tethered to the UE 602 or can exist as at least one stand-alone device connected to the UE 602.
  • Functions of the immersive runtime 618 can include vision engine/simultaneous localization and mapping (SLAM) functions 626, pose correction functions 628, sound field mapping functions 630, etc.
  • the vision engine/SLAM functions 626 can represent functions that process data from the sensors 608 and cameras 610 to generate information about a surrounding environment of the UE 602.
  • the vision engine/SLAM functions 626 can include functions for spatial mapping to create a map of a surrounding area, localization to establish a position of a user and objects with the surrounding area, reconstructions, semantic perception, etc.
  • the sensors 608 can include microphones for capturing audio sources including environmental audio source and user audio.
  • the pose correction functions 628 can represent functions for pose correction that stabilize immersive media when a user moves.
  • the stabilization can be performed using asynchronous time warping (ATW) or late stage re-projection (LSR).
  • the sound field mapping functions 630 can convert signals captured by the UE 602 into semantical concepts, such as by using artificial intelligence (AI) or machine learning (ML). Specific examples here can include object recognition and object classification.
  • the lightweight scene manager 620 can be local to the immersive device but main scene management and composition may be performed on the could/edge server 604.
  • the lightweight scene manager 620 can include a basic scene handler 632 and a compositor 634.
  • the basic scene handler 632 can represent functions that support management of a scene graph, which represents an object-based hierarchy for a geometry of a scene and can regulate interaction with the scene.
  • the compositor 634 can represent functions for compositing layers of images at different levels of depth for presentation.
  • the lightweight scene manager 620 can also include immersive media rendering functions.
  • the immersive media rendering functions can include generation of monoscopic display or stereoscopic display eye buffers from visual content using GPUs.
  • Rendering operations may be different depending on a rendering pipeline of the immersive media.
  • the rendering operations may include 2D or 3D visual/audio rendering, as well as pose correction functionalities.
  • the rendering operations may also include audio rendering and haptic rendering.
  • the media access functions 622 can include tethering and network interfaces for immersive content delivery.
  • AR glasses 210 or other XR device can be tethered through non-5G connectivity, 5G connectivity, and a combination of non-5G and 5G connectivity.
  • the media access functions 622 can include a media session handler 636 and a media client 638. These functions may represent the corresponding functions in FIGURES 3A through 5 described above.
  • the media session handler 636 can include services on the UE 602 that connect to system network functions in order to support media delivery and QoS requirement for media delivery.
  • the media client 638 can include scene description delivery functions 640, content delivery functions 642, and basic codec functions 644.
  • the scene description delivery functions 640 can provide digital representations and delivery of scene graphs and XR spatial descriptions.
  • the content delivery functions 642 can include connectivity and protocol frameworks to deliver immersive media content and provide functionality, such as synchronization, encapsulation, loss and jitter management, bandwidth management, etc.
  • the basic codec functions 644 can include one or more codecs to compress the immersive media provided in a scene.
  • the basic codec functions 644 can include 2D media codecs, immersive media decoders (to decode immersive media as inputs to an immersive media renderer and may include both 2D and 3D visual/audio media decoder functionalities), and immersive media encoders for providing compressed versions of visual/audio immersive media data.
  • the display 614 can include an optical see-through display to allow the user to see the real world directly through a set of optical elements.
  • AR and MR displays can add virtual content by displaying additional light on the optical elements on top of the light received from the real world.
  • the speakers 616 can allow rendering of audio content to enhance the immersive experience.
  • the immersive application 624 can make use of XR functionalities on the UE 602 and the network to provide an immersive user experience.
  • the immersive runtime 618 can perform frame recycling for immersive media processing.
  • Frame recycling can involve using a previously-rendered frame to estimate or produce a subsequent frame for rendering, such as by using techniques such as late stage re-projection (LSR).
  • LSR late stage re-projection
  • Several factors may be considered for enabling frame recycling, which can include determining differences between adjacent frames based on pose information for motion of a user.
  • some immersive contents consumed by the UE 602 may contain scene properties that allow for frame recycling.
  • Frame recycling can be considered when a difference between adjacent frames is sufficiently small that re-projection techniques do not result in occlusion holes and do not generate significant artifacts in the next frame produced by frame recycling.
  • Scene properties can include static scene volume, scene camera safe volumes, or object safe boundaries.
  • the UE 602 can determine which of the lightweight scene manager 620 and the immersive application 624 can perform the frame recycling decision.
  • FIGURES 6A and 6B illustrate one example of an environment 600 for device functions related to pose information delivery configuration
  • various changes may be made to FIGURES 6A and 6B.
  • the number and placement of various components of the environment 600 can vary as needed or desired.
  • the environment 600 may be used in any other suitable media rendering process and is not limited to the specific processes described above.
  • FIGURE 7 illustrates an example technique 700 for pose information delivery configuration and frame recycling decisions by a UE in accordance with this disclosure.
  • the technique 700 may, for example, be used by any of the user equipment described above, such as the electronic device 101, which may represent the AR glasses 210 or UE 202 or 602.
  • the electronic device 101 which may represent the AR glasses 210 or UE 202 or 602.
  • the UE 602 is used, although any other suitable user equipment may be used here.
  • the technique 700 includes operations for pose information delivery confirmation 702 and frame recycling decisions 704.
  • the pose information delivery confirmation 702 is performed when the trigger mode is activated in order to confirm a pose trigger.
  • the entry point can be a scene description, such as a glTF file or any kind of manifest, that contains content information.
  • the content information may describe a location of immersive content for accessing by the media access functions 622 and metadata describing properties of the content, such as one or more objects in a scene.
  • the metadata can include static scene volume descriptions, scene camera safe volume descriptions, per-object safe boundary descriptions, etc.
  • different modes for pose information delivery confirmation can be configured depending on a service use case.
  • StaticSceneConfigBox extends FullBox('stat', 0, 0) ⁇
  • bit(6) reserved 0;
  • StaticSceneSampleEntry represents static metadata or metadata that can change non-frequently, which is defined in the sample entry of the timed metadata track.
  • StaticSceneSample can define the metadata that exists inside each timed metadata sample and may change per sample or frame.
  • a location of each static scene volume can be changed per sample or frame and is described by centre_x, centre_y, and centre_z.
  • dynamic_scene_range_flag and dynamic_safe_range_flag are set to one, a size of the static scene volume and the safe range, respectively, may change over time.
  • the value of dynamic_scene_range_flag is set to zero or one to indicate whether a size of static scene volumes in the content does not change with time.
  • the value of dynamic_safe_range_flag is set to zero or one to indicate whether a size of safe range volumes in the content changes with time.
  • the value of num_volumes can indicate a number of static scene volumes in the content.
  • the values of x_range, y_range, and z_range each define a distance in a respective direction of the x, y, and z axes of the scene volume where contents are static.
  • the value of radius defines a safe range in or around a static scene range.
  • the value of sample_persistence defines a number of samples after a current sample for which syntax values defined in StaticSceneSample are applicable.
  • the values of centre_x, centre_y, and centre_z define a center of a static scene volume in each of the x, y, and z axes directions from an origin defined by the scene description for the content.
  • the scene safe volume paths description may be provided as an extension on the camera elements in the glTF file. Bounding volumes may each define a camera path within a scene that allows for frame recycling, indicates that mesh objects viewed along a path are static, and indicates that rendered frames can be recycled. Examples of syntax and semantics for scene safe volume paths description metadata are shown below in Table 1.
  • per-object safe boundary description metadata may be provided as an extension defined on mesh objects in a glFT file or other file for each mesh object. Examples of syntax and semantics for per-object safe boundary description metadata are shown below in Table 2.
  • the immersive runtime 618 can send latest pose information 706 to the lightweight scene manager 620. If the pose information delivery configuration is set to periodic mode, the lightweight scene manager 620 can send the pose information 706 to the media access function 622, which forwards the pose information to the cloud/edge server 604. The frequency of sending the pose information 706 between the UE 602 and the cloud/edge server 604 or between the lightweight scene manager 620 and the media access function 622 can be dependent on the configuration indicated by the periodic mode parameter, which may be different than a frequency between the immersive runtime 618 and the lightweight scene manager 620.
  • the immersive application 624 or the lightweight scene manager 620 can perform the frame recycling decision 704. In some cases, the frame recycling decision 704 can be performed based on content-related metadata parsed by the lightweight scene manager 620 and device-related factors, such as device status, operational modes, or other hardware-related factors provided by the immersive application 624.
  • a detailed report 708 of the frame recycling decision 704 can be provided from the lightweight scene manager 620 and/or the immersive application 624 to the media access function 622.
  • the detailed report 708 can be forwarded from the media access function 622 to the cloud/edge server 604.
  • the detailed report 708 can include results and factors of the frame recycling decisions 704.
  • the cloud/edge server 604 does not need to send a processed next frame.
  • the detailed report 708 can also include an estimated probability or classification for whether frame recycling may be possible for other future frames. In some cases, the estimated probability or classification can depend on pose motion vectors and location with a scene for the UE 602, where the pose motion vectors and locations with the scene can be based on the content metadata available in the entry point.
  • the UE 602 can proceed with a first option 710 when deciding to frame recycle and a second option 712 when deciding not to frame recycle.
  • the first option 710 can be performed based on the immersive application 624 and/or lightweight scene manager 620 deciding that frame recycling can be performed in the frame recycling decision 704.
  • the first option 710 includes operations 714 and 716.
  • the immersive application 624 and the lightweight scene manager 620 can send a notification to the immersive runtime 618 in operation 714.
  • the notification can include any information to indicate the frame to be recycled as the next frame.
  • the immersive runtime 618 can reuse a previous frame or frames in order to create a next frame or frames for rendering in operation 716.
  • the recycled frame or frames can be determined based on an implemented algorithm discussed in this disclosure.
  • An example implementation can include a late-stage re-projection or other re-projection algorithm that may use additional media related information, such as depth information.
  • the second option 712 can be performed based on the immersive application 624 and/or lightweight scene manager 620 deciding that frame recycling cannot be performed in the frame recycling decision 704.
  • the second option 712 includes operations 718-724.
  • the lightweight scene manager 620 can send the latest pose to the cloud/edge server 604 via the media access function 622 in operation 718.
  • the transmission of pose information can be based on the pose delivery mode, such as in trigger mode.
  • the cloud/edge server 604 can use the pose information during remote pre-rendering.
  • the pre-rendered frame is compressed or encoded and sent to the UE 602.
  • the pre-rendered frame is received and decoded by the media access function 622 in operation 720.
  • the media access function 622 passes the pre-rendered frame to the immersive runtime 618 in operation 722, and the immersive runtime can perform pose correction on the frame based on the latest pose information to compensate for any motion due to photon latencies in operation 724.
  • FIGURE 7 illustrates one example of a technique 700 for pose information delivery configuration and frame recycling decisions by a UE
  • various changes may be made to FIGURE 7.
  • various operations in FIGURE 7 may overlap, occur in parallel, occur in a different order, or occur any number of times.
  • FIGURE 8 illustrates an example graphical representation 800 of object safe boundary description metadata in accordance with this disclosure.
  • the object safe boundary description metadata may be used as described above.
  • a safe boundary 802 can be identified for each object, such as through the per-object safe boundary description metadata from Table 2. Areas marked as “not safe” indicate areas where frame recycling is not feasible, and areas marked as "safe” indicate areas where frame recycling may be feasible. Each object can have different "safe” and "not safe” areas. To determine a safe area, the per-object safe boundary description metadata can be reviewed for each object in the scene.
  • FIGURE 8 illustrates one example of a graphical representation 800 of object safe boundary description metadata
  • object safe boundary description may have any other suitable size and shape.
  • FIGURE 9 illustrates an example system 900 for efficiently communicating with a remote computing system and an immersive device in accordance with this disclosure.
  • a rendering system 900 can include the UE 602, the cloud/edge server 604, and an immersive application 902 (which may represent the immersive application 302 or 624 described above).
  • the rendering system 900 can determine which component to run for a use case employed at the time.
  • an immersive device, the cloud/edge server 604, and the UE 602 may not know the use case.
  • not all compute devices and UEs are the same, so simply saying whether to turn off or turn on a particular sensor or service may not be practicable.
  • the rendering system 900 can determine efficient communication with both the remote cloud/edge server 604 and the UE 602 and classes of services used for the most-optimal way to support a use case class.
  • the rendering system 900 is extensible enough to allow almost any remote components, including tethered devices or cloud-based devices, to work with almost any immersive device.
  • the UE 602, the cloud/edge server 604, and immersive application 902 can work together in a specified configuration, which can be called an "operational mode.” Example operational modes are shown in Table 3.
  • operational modes can be dynamic and can be changed by systems subscribing to state information. Read and write permissions for the operational modes can be managed by a developer. Each system subscribed to the operational modes can have a listener to check for operational mode changes. For example, an HMD optical tracking system may lose six DoF tracking due to poor conditions and revert to three DoF tracking. The HMD service that is subscribed to an operational mode can change the capabilities from an operational mode that supports six DoF head tracking to an operational mode that supports three DoF head tracking. Each operational mode may have a minimum KPI. For example, "mode 4" in Table 3 may require a certain level of accuracy for detecting surfaces. If the rendering system 900 cannot meet this KPI, the rendering system 900 may be prevented from operating in mode 4.
  • FIGURE 9 illustrates one example of a system 900 for efficiently communicating with a remote computing system and an immersive device
  • various changes may be made to FIGURE 9.
  • the number and placement of various components of the system 900 can vary as needed or desired.
  • the system 900 may be used in any other suitable media rendering process and is not limited to the specific processes described above.
  • FIGURE 10 illustrates an example comprehensive computer vision system 1000 in accordance with this disclosure.
  • the computer vision system 1000 may, for example, be used by any of the user equipment described above, such as the electronic device 101, which may represent the AR glasses 210 or UE 202 or 602.
  • the comprehensive computer vision system 1000 includes a computer vision (CV) system 1001.
  • the CV system 1001 can include sensing units 1002 and software modules 1004 to provide various levels of tracking and scene comprehension capabilities.
  • the sensing units 1002 can include a camera 1006, a depth sensor 1008, and an IMU 1010. These sensing units 1002 may, for instance, represent different sensors 180 of the electronic device 101.
  • the software modules 1004 can include a three DoF tracking function 1012, a DOF tracking function 1014, a SLAM function 1016, a plane detection function 1018, a surface reconstruction function 1020, and an object reconstruction function 1022.
  • the CV system 1001 can register with an operational mode engine 1024, which uses an operational mode provider list 1026 that supports various modes of operation.
  • the CV system 1001 can also register itself in a listener list 1028 for operational mode changes.
  • the CV system 1001 can turn on all sensing units 1002 to enable the software modules 1004 to perform the necessary functions.
  • the operational mode engine 1024 decides to change to another operational mode (such as Desktop AR in Table 3) that does not use full comprehension and tracking, the CV system 1001 can turn off the camera 1006 and the depth sensor 1008 but keep the IMU 1010 running to provide three DoF tracking capability, which can be adequate for this operational mode.
  • Various modifications to the sensing units 1002 and software modules 1004 used in each operational mode can be made in order to support proper execution in each operational mode.
  • FIGURE 10 illustrates one example of a comprehensive computer vision system 1000
  • various changes may be made to FIGURE 10.
  • the number and placement of various components of the comprehensive computer vision system 1000 can vary as needed or desired.
  • the comprehensive computer vision system 1000 may be used in any other suitable media rendering process and is not limited to the specific processes described above.
  • FIGURE 11 illustrates an example software stack 1100 for an immersive device in accordance with this disclosure.
  • the software stack 1100 may, for example, be used by any of the user equipment described above, such as the electronic device 101, which may represent the AR glasses 210 or UE 202 or 602.
  • an operational mode engine 1102 is part of an XRIO service, although other services may also be supported.
  • the operational mode engine 1102 is responsible for moving immersive data to an immersive runtime/renderer 1104.
  • the operational mode engine 1102 is the central decision-making entity that controls what operational mode the system functions in at any given time.
  • the operational mode engine 1102 can take requests from an immersive application 1106 or the immersive runtime/renderer 1104 to set particular operational modes of the system if possible.
  • a media app can request a specific operational mode and (if system conditions permit) the operational mode engine 1102 can set the mode for the system 1100.
  • the operational mode engine 1102 is also responsible for setting appropriate operational modes of the system 1100 based on the performance/system load and available power (such as battery level).
  • the operational mode engine 1102 further publishes a set operational mode to the immersive runtime/renderer 1104 and the immersive application 1106 so that the immersive application 1106 can adjust the user's experience accordingly. For example, if the immersive application 1106 is requesting operational mode 7 (Outdoor MR) but the current system state is running under critical battery or high load (low performance), the operational mode engine 1102 can decide to only support up to mode 5 (Room AR) based on the system conditions. The decision by the operational mode engine 1102 can be communicated to the immersive application 1106 and immersive runtime/renderer 1104, which can adjust the user experience accordingly and inform the user.
  • operational mode engine 1102 can decide to only support up to mode 5 (Room AR) based on the system conditions.
  • the decision by the operational mode engine 1102 can be communicated to the immersive application 1106 and immersive runtime/renderer 1104, which can
  • the operational mode engine 1102 can be aware of what hardware modules 1108 and/or functions are available on any given system and can control power for certain hardware modules. For example, if the immersive application 1106 has requested "mode 1" (HUD), the operational mode engine 1102 can ensure that all unused hardware modules 1108 (such as cameras 1110, sensors 1112, etc.) are turned off to save power.
  • the sensors can include one or more depth sensors, one or more inertial measurement unit (IMU) sensors, one or more gyroscopic sensors, one or more accelerometers, one or more magnetometers, etc.
  • the operational mode engine 1102 can also inform the immersive application 1106 whether certain operational modes are not available based on a particular hardware 1108. As a non-limiting example, the operational mode can be determined based on an availability of at least one camera, at least one depth sensor, and at least one IMU. Table 4 shows example ways in which hardware resource optimization can be used to define possible pose information delivery configuration modes.
  • FIGURE 11 illustrates one example of a software stack 1100 for an immersive device
  • various changes may be made to FIGURE 11.
  • the number and placement of various components of the software stack 1100 can vary as needed or desired.
  • the software stack 1100 may be used in any other suitable media rendering process and is not limited to the specific processes described above.
  • FIGURE 12 illustrates an example method 1200 for deferred rendering on an immersive device that is tethered to an electronic device in accordance with this disclosure.
  • the method 1200 of FIGURE 12 is described as being performed using the AR glasses 210 and the tethered electronic device 212 of FIGURE 2.
  • the method 1200 may be used with any other suitable electronic device(s) and in any other suitable system(s).
  • the tethered electronic device 212 can access updated images at step 1202.
  • the updated images may include renders of red, green, and blue (RGB) frames and depth frames using a last known head pose. Head-locked images can be treated separately.
  • the head pose is not updated unless triggered by the AR glasses 210, and an update of head pose can start a new rendering process.
  • the tethered electronic device 212 determines whether a scene delta is set or equal to true at step 1204.
  • the scene delta indicates that image content has changed in a scene (such as by at least a specified amount or percentage).
  • the tethered electronic device 212 can pause for a preset time (such as about 16 ms or other time) and call for a new render after the pause. If adequate content has change in the scene, the tethered electronic device 212 transfers an image delta to the AR glasses 210 at step 1206. The image delta can indicate changes between adjacent frames. The tethered electronic device 212 can also transfer one or more new frames to the AR glasses 210. For example, a listener can be invoked to check for new frames from the tethered electronic device 212. If a request for a new frame is made, the most recent head pose can be sent to the tethered electronic device 212.
  • a preset time such as about 16 ms or other time
  • the AR glasses 210 can calculate a head pose limit at step 1208.
  • the AR glasses 210 can calculate a range of motion that is considered "safe" for reusing a previously-rendered frame.
  • the range of motion may be calculated based on an amount of head pose (HP) translation and head pose rotation that can support a desired image quality based purely on image re-projection.
  • HP head pose
  • One example calculation could be performed as follows.
  • Headpose_rotation_delta (deg) FOV (deg) of rendering camera / 1+(preset/distance (m) from POV).
  • the AR glasses 210 perform one or more tests on RGB and depth data at step 1210.
  • the tests can include determining whether content is within range limits, depth density is at a preset level, near/far content depth is within limits, content depth continuity is within limits, etc.
  • the AR glasses 210 can check if changes from depth point to adjacent depth points are not beyond a preset ratio. For example, if an average depth difference between a test depth point and the eight adjacent depth points are above set limits, an exception can be called. In some cases, the AR glasses 210 can continue the process even if one or more tests fail to be within preset limits. Thus, if at least one of the tests is determined to fail, the process continues to step 1220.
  • the AR glasses 210 can determine whether a head pose is within a range relative to a head posed used to calculate a head pose range at step 1212. If not, the AR glasses 210 can perform re-projection and display functions at step 1214. If sprite animations exist, the AR glasses 210 can update image data and depth data. The AR glasses 210 can determine whether one or more new frames are available at step 1216. When one or more new frames are available, the AR glasses 210 can load the new frame(s) and perform step 1206. A time/frame limit can be used here to request new frames regardless of the whether new frames are available. The AR glasses 210 can access head pose delta information at step 1218.
  • the head pose delta can be determined by comparing information from one or more sensors at times of adjacent frames. When the user's head does not move between the times of adjacent frames, the head pose delta is zero.
  • the head pose delta can be defined based on either three or six DoF based on the operational mode, and the head pose delta can have a value that combines each of the DoF or a value for each DoF.
  • the combined value for head pose delta can be used to determine whether an aggregate movement is within a threshold, and individual values for individual DoFs can be used to determine whether a single DoF exceeds a threshold.
  • the threshold can be different for the combined value and the individual values, and the individual thresholds can be different for different DoFs.
  • the AR glasses 210 can perform re-projection and display functions at step 1220.
  • the AR glasses 210 can automatically request a new frame by checking for a companion device update in operation 1216. Failing the tests can indicate that head pose data should be triggered for sending to the tethered electronic device 212.
  • the AR glasses 210 can determine whether an anchor exists at a position and where content has not been updated based on the head pose at step 1222. For example, for each head pose and associated image and depth data set, an anchor or anchor view can be stored.
  • An anchor or anchor view is a view that can be reprojected or adjusted from when a difference between a current head pose and a head pose corresponding to the anchor or anchor view is within one or more movement thresholds.
  • a set of anchor views can be created to allow a user to have a large range of motion without calling for an updated frame from the server or rendering system.
  • the AR glasses 210 can request a new frame from the tethered electronic device 212.
  • the AR glasses 210 can load image and depth delta and update a sprint frame at step 1224.
  • the image and depth data can be used when an anchor point exists corresponding to the latest head pose. Image data and depth data can be loaded from the anchor or anchor view corresponding to the associated head pose.
  • an anchor view corresponding to a head pose that exceeds the thresholds for movement from a previous head pose can be used for re-projecting and displaying in operation 1214.
  • FIGURE 12 illustrates one example of a method 1200 for deferred rendering on an XR device
  • various changes may be made to FIGURE 12.
  • steps in FIGURE 12 may overlap, occur in parallel, occur in a different order, or occur any number of times.
  • FIGURE 13 illustrates another example method 1300 for deferred rendering on an XR device in accordance with this disclosure.
  • the method 1300 of FIGURE 13 is described as being performed using the electronic device 101 of FIGURE 1.
  • the method 1300 may be used with any other suitable electronic device(s) and in any other suitable system(s).
  • the electronic device 101 establishes a transport session for content on the XR device with a server 106 at step 1302. Transport sessions can provide immersive content from the server 106 to the electronic device 101.
  • the electronic device 101 selects an operational mode at step 1304.
  • the selected operational mode can be partially used for the loop configuration.
  • the operational mode can include at least one of: a HUD mode, a 2D mode, a media mode, a desktop mode, a room MR mode, an area MR mode, and an outside MR mode.
  • the operational mode can be selected based on data from at least one of a camera, a depth sensor, and an IMU.
  • a transport session can be a layered coding transport (LCT) channel uniquely identified by a transport session identifier.
  • LCT layered coding transport
  • a transport session can carry a media component.
  • a transport session can carry one or more objects that are typically related to a representation of a media component.
  • the electronic device 101 provides pose information based on parameters of the loop configuration to the server 106 at step 1308.
  • the parameters of the loop configuration can include at least one of a pose delivery mode, a media session loop setting, and a frame recycling flag.
  • the pose delivery mode can include an offline mode where the pose information is not sent to the server 106, a periodic mode where the pose information is periodically sent to the server 106, and a trigger mode where the pose information is sent only when triggered by the XR device.
  • the media session loop setting can include a first variable to indicate future transmission of pose information to the server 106 and a second variable to indicate pre-rendering of the content by the server 106.
  • the electronic device 101 receives pre-rendered content based on the provided pose information at step 1310.
  • the pre-rendered content can be ignored or not sent based on the pose information indicating frame recycling.
  • the pre-rendered content can always be transmitted from the server 106 to the tethered electronic device 212, and the tethered electronic device 212 can perform additional processing based on an updated head pose received from the AR glasses 210.
  • the tethered electronic device 212 can determine whether to transmit the content to the AR glasses 210 or wait until receiving a request for the content from the AR glasses 210.
  • the electronic device 101 can process and display the content on the XR device at step 1312.
  • the content can be a recycled frame.
  • the content can be a new frame received from the server.
  • the content can be the anchor view with modifications for movement within the at least one associated threshold of the anchor view
  • FIGURE 13 illustrates one example of another method 1300 for deferred rendering on an XR device
  • various changes may be made to FIGURE 13.
  • steps in FIGURE 13 may overlap, occur in parallel, occur in a different order, or occur any number of times.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • User Interface Of Digital Computer (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

La divulgation concerne un système de communication 5G ou 6G pour prendre en charge un débit supérieur de transmission de données. Un procédé de restitution différée sur un dispositif de réalité étendue (XR) consiste à établir une session de transport pour un contenu sur le dispositif XR avec un serveur. Le procédé consiste également à exécuter une configuration de boucle pour le contenu sur la base de la session de transport entre le dispositif XR et le serveur. Le procédé consiste en outre à fournir des informations de pose sur la base de paramètres de la configuration de boucle, au serveur. Le procédé consiste également à recevoir un contenu pré-restitué sur la base des informations de pose en provenance du serveur. De plus, le procédé consiste à traiter et à afficher le contenu pré-restitué, sur le dispositif XR.
EP22916836.4A 2022-01-01 2022-12-30 Restitution différée sur des dispositifs de réalité étendue (xr) Pending EP4445248A4 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202263295859P 2022-01-01 2022-01-01
US202263338575P 2022-05-05 2022-05-05
US18/048,352 US20230215075A1 (en) 2022-01-01 2022-10-20 Deferred rendering on extended reality (xr) devices
PCT/KR2022/021716 WO2023128695A1 (fr) 2022-01-01 2022-12-30 Restitution différée sur des dispositifs de réalité étendue (xr)

Publications (2)

Publication Number Publication Date
EP4445248A1 true EP4445248A1 (fr) 2024-10-16
EP4445248A4 EP4445248A4 (fr) 2025-04-16

Family

ID=86992018

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22916836.4A Pending EP4445248A4 (fr) 2022-01-01 2022-12-30 Restitution différée sur des dispositifs de réalité étendue (xr)

Country Status (4)

Country Link
US (1) US20230215075A1 (fr)
EP (1) EP4445248A4 (fr)
CN (1) CN118475901A (fr)
WO (1) WO2023128695A1 (fr)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230260240A1 (en) * 2021-03-11 2023-08-17 Quintar, Inc. Alignment of 3d graphics extending beyond frame in augmented reality system with remote presentation
US12045940B2 (en) * 2021-11-03 2024-07-23 Tencent America LLC Method for streaming dynamic 5G AR/MR experience to 5G devices with updatable scenes
US20240161225A1 (en) * 2022-11-11 2024-05-16 Qualcomm Incorporated Communicating Pre-rendered Media
US20250063079A1 (en) * 2023-08-15 2025-02-20 Nokia Technologies Oy Seamless split rendering session relocation between split rendering servers supporting extended reality applications in mobile communication networks
US12608421B2 (en) * 2023-09-13 2026-04-21 Disney Enterprises, Inc. System and method for large asset deployment for extended reality devices
CN119946024A (zh) * 2023-11-06 2025-05-06 中兴通讯股份有限公司 一种渲染方法、通信节点及存储介质

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AUPM701394A0 (en) * 1994-07-22 1994-08-18 Monash University A graphical display system
US7558320B2 (en) * 2003-06-13 2009-07-07 Microsoft Corporation Quality control in frame interpolation with motion analysis
KR100962557B1 (ko) * 2009-02-05 2010-06-11 한국과학기술원 증강현실 구현 장치 및 증강현실 구현 방법
US9443355B2 (en) * 2013-06-28 2016-09-13 Microsoft Technology Licensing, Llc Reprojection OLED display for augmented reality experiences
US10962780B2 (en) * 2015-10-26 2021-03-30 Microsoft Technology Licensing, Llc Remote rendering for virtual images
US9928660B1 (en) * 2016-09-12 2018-03-27 Intel Corporation Hybrid rendering for a wearable display attached to a tethered computer
US10204395B2 (en) * 2016-10-19 2019-02-12 Microsoft Technology Licensing, Llc Stereoscopic virtual reality through caching and image based rendering
US11037200B2 (en) * 2016-12-16 2021-06-15 United States Postal Service System and method of providing augmented reality content with a distribution item
US20180324229A1 (en) * 2017-05-05 2018-11-08 Tsunami VR, Inc. Systems and methods for providing expert assistance from a remote expert to a user operating an augmented reality device
US10311833B1 (en) * 2018-03-27 2019-06-04 Seiko Epson Corporation Head-mounted display device and method of operating a display apparatus tracking an object
US11127214B2 (en) * 2018-09-17 2021-09-21 Qualcomm Incorporated Cross layer traffic optimization for split XR
US11181862B2 (en) * 2018-10-31 2021-11-23 Doubleme, Inc. Real-world object holographic transport and communication room system
US11307410B2 (en) * 2020-02-06 2022-04-19 Varjo Technologies Oy Display apparatus and method incorporating adaptive pose locking

Also Published As

Publication number Publication date
US20230215075A1 (en) 2023-07-06
EP4445248A4 (fr) 2025-04-16
WO2023128695A1 (fr) 2023-07-06
CN118475901A (zh) 2024-08-09

Similar Documents

Publication Publication Date Title
WO2023128695A1 (fr) Restitution différée sur des dispositifs de réalité étendue (xr)
US12550228B2 (en) Systems and methods for collaborative edge computing
US10823958B2 (en) Electronic device for encoding or decoding frames of video and method for controlling thereof
WO2018194306A1 (fr) Système et procédé d'utilisation d'application bidimensionnelle dans un environnement de réalité virtuelle tridimensionnelle
US11843755B2 (en) Cloud-based rendering of interactive augmented/virtual reality experiences
US10404960B2 (en) System and method to prevent boundary artifacts
US12217368B2 (en) Extended field of view generation for split-rendering for virtual reality streaming
CN110704324B (zh) 应用调试方法、装置及存储介质
US12192673B2 (en) Accurate optical flow interpolation optimizing bi-directional consistency and temporal smoothness
CN112527174B (zh) 一种信息处理方法及电子设备
Braud et al. DiOS—An extended reality operating system for the metaverse
WO2018226080A1 (fr) Système et procédé pour la mise à l'échelle transparente et dynamique d'un affichage de contenu
CN112527222A (zh) 一种信息处理方法及电子设备
KR20190055064A (ko) 화각 가상 현실 스트리밍을 위한 세션 제어 지원을 위한 방법 및 장치
CN110321228B (zh) 原生方法的调用方法、装置、设备及计算机可读存储介质
WO2022250452A1 (fr) Système et procédé de commande de modes de fonctionnement pour dispositifs xr pour optimisation de performance
CN106658103A (zh) 一种分享视频流数据的方法、源设备及电子设备
US12538191B2 (en) Methods and apparatus for application service relocation for multimedia edge services
CN118056353A (zh) 通过眼睛跟踪进行运动模糊补偿
US20230324714A1 (en) Intelligent actuated temple attachments
WO2024244964A1 (fr) Procédé de connexion de dispositif et appareil associé
WO2021249562A1 (fr) Procédé de transmission d'informations, dispositif associé, et système
CN104698728A (zh) 微型投影装置及其操作方法和操作装置
US20250310216A1 (en) Eyewear experience hub for network resource optimization
KR20250010061A (ko) 네트워크 슬라이싱을 이용한 미디어 서비스들에 대한 네트워크 지원 스트림 동기화를 위한 방법 및 장치

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20240708

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20250313

RIC1 Information provided on ipc code assigned before grant

Ipc: H04L 67/141 20220101ALI20250307BHEP

Ipc: H04L 67/131 20220101ALI20250307BHEP

Ipc: G02B 27/01 20060101ALI20250307BHEP

Ipc: G06Q 50/10 20120101ALI20250307BHEP

Ipc: G06T 19/00 20110101ALI20250307BHEP

Ipc: G06F 3/01 20060101AFI20250307BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20251210

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED