EP4619845A1 - Techniques d'affichage de contenu de réalité étendue sur la base de paramètres relatifs à un opérateur - Google Patents

Techniques d'affichage de contenu de réalité étendue sur la base de paramètres relatifs à un opérateur

Info

Publication number
EP4619845A1
EP4619845A1 EP23828281.8A EP23828281A EP4619845A1 EP 4619845 A1 EP4619845 A1 EP 4619845A1 EP 23828281 A EP23828281 A EP 23828281A EP 4619845 A1 EP4619845 A1 EP 4619845A1
Authority
EP
European Patent Office
Prior art keywords
content
operator
geometric feature
virtual geometric
orientation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP23828281.8A
Other languages
German (de)
English (en)
Inventor
Sundar Murugappan
Pratap Ganapathy
Mark GARIBALDI
Michael Jones
Sida LI
Richard Mahoney
Govinda PAYYAVULA
Kelly PRINCE
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intuitive Surgical Operations Inc
Original Assignee
Intuitive Surgical Operations Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intuitive Surgical Operations Inc filed Critical Intuitive Surgical Operations Inc
Publication of EP4619845A1 publication Critical patent/EP4619845A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements

Definitions

  • the present disclosure relates generally to electronic devices and more particularly relates to techniques for displaying extended reality content based on operator related parameters.
  • Computer-assisted electronic devices are being used more and more often. This is especially true in industrial, entertainment, educational, and other settings. As medical examples, the medical facilities of today have large arrays of computer-assisted devices being found in operating rooms, interventional suites, intensive care wards, emergency rooms, and/or the like. Many of these computer-assisted devices are moveable, and may be capable of autonomous or semi-autonomous motion. It is also known for personnel to control the motion and/or operation of moveable computer-assisted devices using one or more input devices located at a user control system. As a specific example of electronic systems comprising computer-assisted devices, minimally invasive, robotic telesurgical devices permit surgeons to operate on patients from bedside or remote locations. Telesurgery refers generally to surgery performed using surgical devices where the surgeon uses some form of remote control, such as a servomechanism, to manipulate surgical instrument movements rather than directly holding and moving the instruments by hand.
  • remote control such as a servomechanism
  • Extended reality (XR) systems are oftentimes used in conjunction with computer- assisted devices to assist in the performance of tasks at worksites.
  • XR devices include augmented reality (AR) devices and virtual reality (VR) devices.
  • AR refers to a view of the physical environment with an overlay of one or more computergenerated graphical elements, including mixed reality (MR) environments in which physical objects and computer-generated elements can interact.
  • MR mixed reality
  • VR refers to a virtual environment that includes computer-generated elements.
  • an XR device can present data about an operating environment of a computer-assisted device, graphical elements for entertainment, or visual guidance during operation of the computer-assisted device, among other things.
  • Some conventional techniques for displaying XR content cause the XR content to be displayed in less optimal locations in some instances, thereby decreasing visibility of the XR content, increasing the time needed to view or interact with the XR content, decreasing the efficiency of operations performed with an XR device, reducing operator enjoyment or causing operator discomfort, requiring the operator to reposition him or herself, and/or the like.
  • an extended reality (XR) system includes an XR device and a processor system.
  • the processor system is configured to: determine an operator position of an operator portion of an operator based on first sensor data, determine a first object position of an object portion of an object based on second sensor data, determine a first feature position for a virtual geometric feature based on the first object position, determine a first content position based on the operator position while using the virtual geometric feature located at the first feature position to constrain the first content position, and cause the XR device to display an XR content based on the first content position.
  • a method of causing an extended reality (XR) device to display XR content includes: determining an operator position of an operator portion of an operator based on first sensor data, determining a first object position of an object portion of an object based on second sensor data, determining a first feature position for a virtual geometric feature based on the first object position, determining a first content position based on the operator position while using the virtual geometric feature located at the first feature position to constrain the first content position, and causing the XR device to display the XR content based on the first content position.
  • XR extended reality
  • Figure 1 is a simplified diagram including an example of a computer-assisted device and an extended reality (XR) device, according to various embodiments.
  • XR extended reality
  • Figure 2 is a perspective view illustrating an XR device, according to various embodiments.
  • Figure 3 illustrates the XR module of Figure 1 in greater detail, according to various embodiments.
  • Figure 6 illustrates an example of displaying XR content, according to various embodiments.
  • Figure 7 illustrates another example of displaying XR content, according to various embodiments.
  • Figure 8 illustrates another example of displaying XR content, according to various embodiments.
  • spatially relative terms such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like-may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures.
  • These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features.
  • position refers to the location of an element or a portion of an element in space (e.g., three degrees of translational freedom in three-dimensional (3D) space, which can be described as along Cartesian x-, y-, and z-coordinates).
  • orientation refers to the rotational placement of an element or a portion of an element in space (e.g., three degrees of rotational freedom in 3D space, such as rotations about Cartesian z-, y-, and z- axes, rotation sets such as roll, pitch, and yaw, and the like).
  • proximal refers to a direction toward the base of the kinematic series
  • distal refers to a direction away from the base along the kinematic series.
  • aspects of this disclosure are described in reference to electronic systems and computer-assisted devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, manually manipulated, and/or the like.
  • Example computer-assisted systems include those that comprise robots or robotic devices.
  • aspects of this disclosure are often described in terms of an embodiment using a medical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.
  • inventive aspects disclosed herein may be embodied and implemented in various ways, including in medical and non-medical embodiments, and including with robotic and, as applicable, non-robotic embodiments.
  • Embodiments described for da Vinci® Surgical Systems are merely exemplary, and are not to be considered as limiting the scope of the inventive aspects disclosed herein.
  • the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems.
  • the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and/or the like.
  • Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.
  • FIG. 1 is a simplified diagram of an electronic system 100, according to various embodiments.
  • electronic system 100 includes, without limitation, a computer- assisted device 110 and an extended reality (XR) device 130.
  • Computer-assisted device 110 includes a repositionable structure, the repositionable structure comprising a manipulator arm 120 configured to support an instrument 122.
  • Figure 1 shows the repositionable structure as comprising one manipulator arm 120 configured to support one instrument 122, this is for illustrative purposes.
  • the repositionable structure of a computer-assisted device can comprise only a manipulator arm, a manipulator arm and other mechanical structures physically coupled to the manipulator arm, one or multiple (e.g., zero, one, two, three, four, or any suitable number) of manipulator arms, one or more repositionable hardware components other than manipulator arms, a combination or portion of any of the foregoing, and/or the like. Further, each manipulator arm or other repositionable component can be configured to support one instrument, or a plurality of instruments.
  • instrument 122 is an imaging instrument, a manipulation instrument such as graspers or scissors, a fastening instrument such as a stapler, an irrigation instrument, a suction instrument, an energy application instrument, an instrument with multiple functions, or any other appropriate instrument.
  • instrument 122 is an imaging instrument such as a monoscopic or stereoscopic camera, a still or video camera, an endoscope, a hyperspectral device, an infrared or ultrasonic device, an ultrasonic device, a fluoroscopic device, and/or the like.
  • instrument 122 is a medical instrument, such as a medical endoscope, forceps, clip appliers, a gripper, a retractor, a cautery instrument, a suction instrument, a suturing device, a stapling device, a cutting device, and/or the like.
  • instrument 122 includes an end effector capable of performing one or multiple tasks, such as grasping a material (e.g., tissue of a patient in a medical example) located in a workspace and delivering energy to material (e.g., delivering electrocautery energy to a patient in a medical example).
  • the energy includes ultrasonic, radio frequency, electrical, magnetic, thermal, light, and/or other types of energy.
  • the manipulator arm 120 includes one or more joints and links and is configured to support the instrument 122.
  • instrument 122 is, during use, inserted into a workspace (e.g., anatomy of a patient or cadaver, a veterinary subject, an anatomical model, and/or the like in some medical examples) through a cannula, access port, and/or the like to perform a procedure.
  • a workspace e.g., anatomy of a patient or cadaver, a veterinary subject, an anatomical model, and/or the like in some medical examples
  • computer-assisted device 110 is a teleoperated device.
  • the teleoperated device is a teleoperated medical device, such as a telesurgical device, that can be found in an operating room and/or an interventional suite.
  • computer-assisted device 110 is a follower device that is teleoperated by being controlled by one or more leader devices (not shown), such as one or more input devices designed to be contacted and manipulated by an operator (not shown).
  • the one or more input devices may be mechanically grounded (kinematically grounded by mechanical structures) or mechanically ungrounded (not kinematically grounded by mechanical structures).
  • leader-follower systems systems that include a leader device and a follower device are referred to as leader-follower systems, and also sometimes referred to in literature as master-slave systems where the leader is termed the “master” and the follower is termed the “slave.”
  • computer-assisted device 110 is a teleoperated follower device comprising a repositionable structure (e.g., a manipulator arm 120), and the follower device is controlled to move and articulate in response to manipulation of leader device(s) by an operator, the computer-assisted device 110 “follows” the leader input device(s) through teleoperation. The operator is then able to perform tasks at a worksite using the manipulator arm 120 and/or instrument 122 (if supported by the manipulator arm 120).
  • the repositionable structure e.g., a manipulator arm
  • the repositionable structure may or may not be configured to support an instrument (e.g., instrument 122).
  • XR device 130 comprises an augmented reality (AR) device or a virtual reality (VR) device.
  • AR augmented reality
  • VR virtual reality
  • XR device 130 is described in greater detail below in conjunction with Figure 2.
  • XR device 130 is used in conjunction with computer-assisted device 110.
  • XR device 130 could be used to present instructional content on how to operate computer-assisted device 110.
  • XR device 130 could be used to present content that provides guidance during operation of computer-assisted device 110.
  • XR device 130 has displayed AR content 136 next to repositionable structure with a manipulator arm 120 of computer-assisted device 110.
  • AR content 136 can be provided in any manner appropriate for visual AR content, such as a visual overlay.
  • AR content 136 can be rendered for display by XR device 130, by computer-assisted device 110, or by any other device or devices.
  • XR content e.g., AR content 136
  • a registration transform 134 between XR device 130 and computer-assisted device 110 is established.
  • Registration transform 134 provides a geometric relationship between a frame of reference related to XR device 130 (such as reference frame 132 fixed relative to the XR device 130 or another frame of reference) and a frame of reference related to the computer- assisted device 110 (such as reference frame 124 fixed relative to computer-assisted device 110 or another frame of reference).
  • Each of the one or more processors of control system 140 is an integrated circuit for processing instructions.
  • the one or more processors can be one or more cores or micro-cores of a processor, a central processing unit (CPU), a microprocessor, a field- programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and/or the like.
  • control system 140 also includes one or more input devices (not shown), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device.
  • control system 140 further includes one or more output devices (not shown), such as a display device (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, organic LED display (OLED), projector, or other display device), a printer, a speaker, external storage, or any other output device.
  • a display device e.g., a liquid crystal display (LCD), a plasma display, touchscreen, organic LED display (OLED), projector, or other display device
  • a printer e.g., a printer, a speaker, external storage, or any other output device.
  • control system 140 is connected to or be a part of a network.
  • the network can include multiple nodes.
  • Control system 140 is implemented on one node or on a group of nodes.
  • control system 140 can be implemented on a node of a distributed system that is connected to other nodes.
  • control system 140 can be implemented on a distributed computing system having multiple nodes, where different functions and/or components of control system 140 are located on a different node within the distributed computing system.
  • one or more elements of the aforementioned control system 140 can be located at a remote location and connected to the other elements over a network.
  • Some embodiments include one or more components of a teleoperated medical system such as a da Vinci® Surgical System, commercialized by Intuitive Surgical, Inc. of Sunnyvale, California, U.S.A.
  • a teleoperated medical system such as a da Vinci® Surgical System, commercialized by Intuitive Surgical, Inc. of Sunnyvale, California, U.S.A.
  • da Vinci® Surgical Systems are merely examples and are not to be considered as limiting the scope of the features disclosed herein.
  • different types of teleoperated systems having computer-assisted devices comprising follower devices configured to be placed at worksites can use the features described herein.
  • non-teleoperated systems can also make use of features described herein.
  • FIG. 2 is a perspective view illustrating head-mounted XR device 130 in greater detail, according to various embodiments.
  • XR device 130 includes a body 205 and a head mount 210.
  • Body 205 includes one or more electronic display elements of an electronic display 230.
  • Body 205 also includes a sensor system 240 that acquires sensor data associated with the physical environment external to XR device 130, which can also be external to any objects (such as computer-assisted devices) in the physical environment.
  • Sensor system 240 can include any technically feasible sensor or sensors, such monoscopic and stereoscopic optical systems, ultrasonic systems, depth cameras such as cameras using time-of-flight sensors, LIDAR (Light Detection and Ranging) sensors, stereo RGB (red, green, blue) sensors, RGB-D depth-sensors, etc.
  • Figure 2 shows a head-mounted XR device, other XR devices may be used in other embodiments. Examples of other types of XR devices include appropriately configured tablets, smart-phones, projectors, etc.
  • sensor system 240 is shown in Figure 2 as included in XR device 130, a sensor system used to provide sensor data associated with the physical environment external to XR device 130 (which can also include data external to one or more objects in the physical environment) can be provided in any appropriate location.
  • a sensor system used to provide sensor data associated with the physical environment external to XR device 130 (which can also include data external to one or more objects in the physical environment) can be provided in any appropriate location.
  • part or all of such a sensor system can be alternatively or additionally located elsewhere in the physical environment, including mounted on walls, ceilings, or stands, or coupled to a computer-assisted device.
  • body 205 includes electronic display 230 and an optics block 235 that together provide image light to a target location of body 205 where an eye 202 of an operator may be positioned.
  • body 205 also includes one or more other sensors, such as one or more imaging devices (e.gncy one or more imaging sensors for tracking eye 202), accelerometers, and/or angular velocity sensors (which can be part of inertial measurement units (IMUs)), position sensors, and/or other sensors.
  • imaging devices e.g. one or more imaging sensors for tracking eye 202
  • accelerometers e.g. one or more accelerometers, and/or angular velocity sensors (which can be part of inertial measurement units (IMUs)
  • IMUs inertial measurement units
  • Electronic display 230 is configured to display rendered images that are viewable by the operator.
  • electronic display 230 includes a single electronic display or multiple electronic displays (e.gitch a display for each eye of an operator).
  • Examples of the electronic display 230 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a QOLED, a QLED, some other display, or some combination thereof.
  • Optics block 235 includes optical elements that can be used to adjust an orientation of image light emitted from electronic display 230 such that electronic display 230 appears at particular virtual image distances from the operator.
  • XR device 130 operates as an AR device that presents computer-generated media to an operator using electronic display 230 that augments views of a physical, real-world environment visible to the operator through electronic display 230.
  • Examples of computer-generated media presented by XR device 130 include one or more images, video, audio, or some combination thereof.
  • XR device 130 operates as a VR device, or some combination of an AR device and a VR device, such as a device that permits switching between AR and VR environments.
  • sensor system 240 can capture images of the physical environment and display the captured images along with computer-generated elements, such as AR content 136, which is also sometimes referred to as “video see through.”
  • AR devices include Microsoft HoloLens®, Google Glass®, and Meta 2®.
  • MR devices include Microsoft HoloLens 2®, Samsung Odyssey+®, HP Reverb®, and Oculus Quest 2®.
  • VR devices include Oculus Rift®, Samsung Gear VR®, HTC Vive®, and Google Daydream View®.
  • Figure 2 merely shows an example configuration of an XR device.
  • the techniques for displaying XR content that are disclosed herein are usable with other configurations and/or types of XR devices.
  • the other configurations and/or types of XR devices include head-mounted XR devices, hand-held XR devices, and XR devices that are placed in the environment, among other things.
  • Examples of alternative configurations and/or types of XR devices include optical head-mounted displays (HMDs), mobile devices mobile phones, tablet computers, etc.), fully immersive projection systems, etc.
  • HMDs optical head-mounted displays
  • mobile devices mobile phones mobile phones, tablet computers, etc.
  • fully immersive projection systems etc.
  • Some conventional XR devices display XR content at a position that is fixed relative to the position of a portion of an object. In such cases, the XR content is rendered at the same position relative to the portion of the object even when an operator viewing the XR content moves. In some cases, the displayed position of XR content moves along with a portion of an object such that the displayed position stays fixed relative to the portion of the object.
  • FIG. 3 illustrates XR module 170 of Figure 1 in greater detail, according to various embodiments.
  • XR module 170 includes, without limitation, an operator position evaluation module 306, an object position evaluation module 308, and an overlay module 310.
  • operator position evaluation module 306 receives sensor data 302 that is acquired by a sensor system and determines the position of a portion of an operator (“operator portion”).
  • the portion of the operator can be the head, chest, or any other suitable portion of the operator.
  • the sensor data used to determine the position of the portion of the operator can be acquired by eye tracking sensors, gyroscopes, cameras, and/or any other suitable sensor device(s) that are mounted on the XR device or elsewhere in the environment.
  • operator position evaluation module 306 also determines an orientation metric associated with the portion of the operator, such as a direction of view of the head of the operator.
  • object position evaluation module 308 receives sensor data 304 that is acquired by a sensor system (e.g., sensor system 240) and determines the position of a portion of an object (e.g., an object portion). In some embodiments, the object position evaluation module 308 also determines an orientation metric associated with the portion of the object.
  • the sensor system that acquires sensor data 304 can be the same as, or different from, the sensor system that acquires sensor data 302.
  • the sensor data 304 can include kinematic data associated with one or more joints and/or links of the repositionable structure.
  • kinematic data can be acquired by joint sensors that transmit positions and orientations of joints of the repositionable structure to record movements thereof, shape sensors that monitor the shape of an optical fiber to determine the pose of the repositionable structure located at one end of the optical fiber relative to a frame of reference located at the other end of the fiber, and/or in any other technically feasible manner.
  • the portion of the object is a portion of a computer-assisted device that includes a repositionable structure
  • techniques disclosed herein are also applicable to cases in which the object is not a computer-assisted device, or is a computer-assisted device that does not include a repositionable structure.
  • overlay module 310 generates XR content and causes the XR content to be displayed via an XR device (e.g., XR device 130).
  • the XR content includes AR and/or VR content in some embodiments. Any suitable XR content, such as the AR content 136 described above in conjunction with Figure 1, can be generated.
  • the XR content includes text, graphical images, animations or videos, and/or virtual controls.
  • the XR content is used to entertain, aid or instruct, present virtual controls, and/or otherwise provide input for or output to an operator.
  • the XR content could include instructional content on how to operate a computer-assisted device (e.g., computer-assisted device 110).
  • the XR content could include content that provides data related to a procedure being performed by a computer-assisted device (e.g., computer-assisted device 110), such as previously captured images, models, real-time captured images, data about the functioning of the computer-assisted device, communications from others, tutorials or videos, guidance during operation of the computer-assisted device, etc.
  • a computer-assisted device e.g., computer-assisted device 110
  • data related to a procedure being performed by a computer-assisted device e.g., computer-assisted device 110
  • a computer-assisted device 110 such as previously captured images, models, real-time captured images, data about the functioning of the computer-assisted device, communications from others, tutorials or videos, guidance during operation of the computer-assisted device, etc.
  • overlay module 310 determines a position with which to display XR content (e.g., a content position), and optionally an orientation with which the XR content is displayed (e.g., a content orientation), based on (1) the position of a portion of an operator (also referred to herein as an “operator position”), and optionally the orientation of the operator (also referred to herein as an “operator orientation”), determined by operator position evaluation module 306; and (2) the position of the portion of the object (also referred to herein as an “object position”), and optionally the orientation of the portion of the object (also referred to herein as an “object orientation”), determined by object position evaluation module 308.
  • a position with which to display XR content e.g., a content position
  • an orientation with which the XR content is displayed e.g., a content orientation
  • overlay module 310 first determines, based on the position and optionally the orientation of the portion of the object, a position of a virtual geometric feature (also referred to herein as a “feature position”), and optionally an orientation of the virtual geometric feature (also referred to herein as a “feature orientation”), that has a known relationship with respect to a portion of the object. Then, overlay module 310 determines the position (and optionally orientation) of the XR content based on the position (and optionally orientation) of the virtual geometric feature and the position (and optionally orientation) of the portion of the operator, as discussed in greater detail below in conjunction with Figures 4-8.
  • overlay module 310 applies a registration transform to map the determined position (and optionally orientation) of the XR content in a reference frame of the object to a corresponding position (and optionally orientation) in a reference frame of XR device 130. Then, overlay module 310 generates (e.g., renders) XR content for display at the corresponding position (and optionally orientation) in the reference frame of XR device 130. Thereafter, overlay module 310 transmits, to XR device 130, a display signal 312 that causes XR device 130 to display the XR content at the corresponding position (and optionally orientation) in the reference frame of XR device 130.
  • overlay module 310 generates content for display to an operator to enhance a view of the physical environment, which is sometimes also referred to as “optical see through.”
  • content generated by overlay module 310 is combined with image data depicting the physical environment to generate a composite image for display to the operator, which is sometimes also referred to as “video see through.”
  • the image data is captured by one or more imaging devices in sensor system 240, or elsewhere.
  • display signal 312 is generated based on the composite image.
  • Figure 4 illustrates a simplified diagram of a method for displaying XR content based on one or more operator-related parameters, according to various embodiments.
  • One or more of the processes 402-410 of method 400 can be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine readable media that when executed by one or more processor systems (e.g., the processor system 150 in control system 140) cause the one or more processors to perform one or more of the processes 402-410.
  • processor systems e.g., the processor system 150 in control system 140
  • method 400 can be performed by one or more modules, such as XR module 170.
  • method 400 can include additional processes, which are not shown.
  • method 400 begins at process 402, where the position of a portion of an operator is determined based on sensor data acquired by a first sensor system. Examples of the first sensor system are described above in conjunction with Figure 3.
  • the portion of the operator is a head of the operator.
  • the sensor data includes data acquired by cameras, gyroscopes, eye tracking sensors, and/or any other suitable sensor device(s) that are mounted on an XR device (e.g., XR device 130) or elsewhere in the environment.
  • an orientation of the portion of the operator is also determined from the sensor data.
  • the orientation of the portion of the operator can be the direction of view of the operator, or derived from the direction of view of the operator, etc.
  • the direction of view can be an eye gaze direction, or a proxy or estimation of the gaze direction based on some other parameter(s).
  • a proxy for the direction of view can be a direction of a head of the operator, such as a direction that the nose or mouth or face is facing, or estimated from such direction of the head.
  • the direction of view can be represented by a gaze direction vector associated with the direction of view.
  • the gaze direction vector originates from the head of the operator and extends in a direction that the operator is looking, such as from the center of a line segment connecting the pupils of the operator and in the direction the operator has directed the pupils.
  • the position for a virtual geometric feature is determined based on the position of the portion of the object.
  • the virtual geometric feature can have any suitable (1) shape and/or size, and (2) relationship with respect to the portion of the object.
  • Examples of virtual geometric features include a portion of a line (“line portion,” such as a line segment) or a line (e.g., an infinite line), a portion of a spline (“spline portion”), a spline (e.g., an infinite spline), a portion of a surface (“surface portion”) or an entire surface of any shape and size, a complex feature combining any number of any of the foregoing, and/or the like.
  • the position of a virtual geometric feature could be offset by a predefined amount (e.g., by a predefined distance or other offset parameter) relative to the position of a portion of an object while being parallel to an axis of the portion of the object.
  • the virtual geometric feature could be a spline that circumscribes a portion of an object and is extruded into a surface that circumscribes the portion of the object.
  • the virtual geometric feature could change from being a certain offset away from a portion of the object to being further away or closer to the portion of the object.
  • the shape of a virtual geometric feature could change, such as from a line to a spline or vice versa, based on the position of a portion of the operator.
  • the depth with which XR content is displayed on a virtual geometric feature could depend on a distance of the portion of the operator from the portion of the object.
  • the virtual geometric feature can have any suitable orientation.
  • the virtual geometric feature has a fixed or dynamic orientation relative to an orientation of the portion of the object or an axis associated with the portion of the object. For example, when the virtual geometric feature is a line, the line could be vertical, horizontal, slanted, etc.
  • XR content is caused to be displayed based on the position of the portion of the operator and the position for the virtual geometric feature.
  • the XR content includes text, graphical images, animations or videos, and/or virtual controls that are used to entertain, aid or instruct, present virtual controls, and/or otherwise provide input for or output to an operator.
  • the XR content is constrained by the virtual geometric feature located at the position for the virtual geometric feature. In some embodiments, the XR content is constrained to be coincident with the virtual geometric feature.
  • causing the XR content to be displayed comprises: (1) applying a registration transform to map a position of the XR content that is coincident with the virtual geometric feature, in a reference frame of the object, to a corresponding position in a reference frame of an XR device (e.g., XR device 130); and (2) transmitting one or more control signals to the XR device that cause the XR device to display the XR content at the corresponding position in the reference frame of the XR device.
  • an XR device e.g., XR device 130
  • the XR content can be constrained by the virtual geometric feature in any technically feasible manner.
  • a gaze direction vector associtaed with the operator changes which DOFs of the operator are used to constrain the XR content to the virtual geometric feature. For example, when the gaze direction vector is parallel to the ground or within 45 degrees of plane parallel to the ground, then upward- downward motions of the head of the operator can be mapped to motions of the XR content along a vertical line that is the virtual geometric feature.
  • the gaze direction vector when pointed downward or within 45 degrees of downward, then sideways motions of the head of the operator can be be mapped to motions of the XR content along a horizontal line that is the virtual geometric feature, and the gaze direction vector is used to determine the position of the XR content.
  • XR content is displayed based on a virtual geometric feature without being coincident to the virtual geometric feature.
  • a size, position, and/or an orientation of the XR content can be adjusted to reduce an amount of the XR content that overlaps, underlays, or intersects the portion of the object.
  • the reduction provided by such adjustment e.g., adjusted position, adjusted orientation, and/or adjusted size is in comparison to the unadjusted position, orientation, and/or size of the XR content.
  • the reduction can be partial where some overlapping/underlaying of the XR content or intersection of the XR content with the portion of the object would still exist with the adjustment.
  • the reduction can be complete where the overlapping/underlaying of the XR content or the intersection of the XR content with the portion of the object is eliminated or reduced to an extent not perceptible by the operator.
  • the XR content can be moved in front of the portion of the object or to an offset position that reduces an amount of the XR content that overlaps, underlays, or intersects the portion of the object.
  • spatial mapping can be used to identify objects (which can be represented as virtual meshes) of interest in a scene, and in conjunction with a registration, place a 3D model of the portion of the object overlaid with the meshes. Then, XR content can be placed in front of, or offset from, the 3D model taking occlusion meshes into account. When the XR content is placed in front of the portion of the object or at the offset position, the adjusted XR content can still be coincident to the virtual geometric feature, or no longer coincident to the virtual geometric feature.
  • a depth and/or size of the XR content can be adjusted to reduce the amount to which the XR content overlaps/underlays or intersects the portion of the object. As noted, the reduction can be partial or complete. In some embodiments, XR content is permitted to be displayed in front of, behind, and/or in a manner that intersects one or more portions of an object.
  • the orientation of the XR content relative to the virtual geometric feature is controlled based on an orientation of the portion of the operator relative to the portion of the object.
  • the orientation of the XR content e.g., text boxes, menus, images, videos, and/or the like
  • the XR content can be controlled so that the XR content always faces the operator in a field of view of the operator, which is also referred to herein as “billboarding.”
  • the XR content is displayed perpendicular to a gaze direction vector, the vector associated with a direction of view of the operator.
  • additional rules are applied to determine how to display the XR content.
  • XR content is displayed relative to other XR content and/or to objects in the physical environment according to predefined rules. For example, a rule could prioritize animations to be displayed in at the center of a field of view, while other elements (e.g., text) are permitted to be displayed away from the center of the field of view and/or out of the field of view.
  • some XR content could be displayed to fit a field of view of the operator, i,eerne to be displayed entirely within the field of view of the operator, so that the operator has a full view of the content. In such cases, the z-position (depth) and scaling (up or down) of the XR content can also be optimized so that the content is as big as possible.
  • the position of the XR content relative to the virtual geometric feature is controlled based on the position of the head of the operator, and optionally the gaze direction vector (e.g., a vector associated with a direction of view of the operator) or another orientation metric associated with the head of the operator.
  • Figure 5 illustrates an example process 408 of the method 400 of Figure 4 in greater detail, according to various embodiments.
  • the portion of the operator is the head of the operator, the virtual geometric feature is used to constrain the XR content by requiring coincidence, and other considerations are described further below. It should be understood that Figure 5 presents just one example of process 408, and other embodiments may use other portions of the operator, other virtual geometric features, other types of constraints, etc.
  • XR content is displayed coincident to the virtual geometric feature based on the intersection of the gaze direction vector and the surface associated with the virtual geometric feature.
  • the gaze direction vector begins from a position associated with the head of the operator (e.g., the center of a line between eyes of the operator) and intersects the surface associated with the virtual geometric feature.
  • the virtual geometric feature is a vertical line
  • a height (or x-y position) of the XR content along the line is determined by intersecting a gaze direction vector with a planar surface that includes the line and is perpendicular to the gaze direction vector.
  • a position of the XR content along the spline is determined by intersecting a gaze direction vector with a non-planar surface that circumscribes the object portion and is generated by either extruding the spline in a direction corresponding to an orientation axis of the object portion, or by extruding or sweeping between the spline and another spline (e.g., a sweep surface between two rail curves).
  • a closest point on the virtual geometric feature to the gaze direction vector is determined.
  • the closest point is determined by (1) projecting the virtual geometric feature onto a plane that is perpendicular to the gaze direction vector, and (2) determining a point of the projected virtual geometric feature that is closest to the gaze direction vector.
  • the gaze direction vector is associated with a direction of view of the operator.
  • XR content is caused to be displayed coincident to the virtual geometric feature based on the closest point.
  • the XR content is displayed at or near the closest point.
  • the displayed position and/or orientation of the XR content can also account for operator preference in some embodiments.
  • the position and/or pitch of the XR content is adjusted based on operator preference that is specified in any technically feasible manner (e.g., via a user interface). For example, when the position of XR content along a vertical line segment (i.eembroidered the virtual geometric feature) is determined based on the eye gaze height of an operator, the position of the XR content could be further adjusted to be higher or lower than the eye gaze height based on operator preference. As another example, an offset of XR content from an axis associated with a portion of an object can be adjusted based on operator preference.
  • the displayed position and/or orientation of the XR content is also based on a control point associated with the XR content.
  • the control point can be at any suitable position.
  • the control point could be positioned at a corner (e.gsten the upper-left comer) of the XR content, along an edge a center point of a left edge) of the XR content, a center of the XR content, or at any other position relative to the XR content (e.g., within the XR content or located near the XR content).
  • the control point is positioned based on the intersection point determined at process 504 or the closest point determined at process 506, described above in conjunction with Figure 5.
  • method 400 returns to process 402, where the position of the portion of the operator is determined based on additional sensor data acquired by the first sensor system. For example, when the portion of the operator and/or the portion of the object moves, the processes of method 400 are repeated to update the displayed position (and optionally orientation) of the XR content.
  • method 400 is described primarily with respect to a single operator, in some embodiments, processes similar to those of method 400 are performed to display XR content for multiple operators.
  • the XR content is displayed based on the positions (and optionally view directions or other orientation metrics) of portions of the multiple operators and constrained by the virtual geometric feature.
  • the head positions and directions of view of the multiple operators are averaged, and method 400 is performed for the averaged head positions and directions of view.
  • XR content is displayed at a different position (and optionally orientation) determined according to method 400 for each of the multiple operators.
  • a substantially vertical line segment 602 is shown for illustrative purposes, similar processes can be performed to display XR content coincident to a horizontal or slanted line segment based on a gaze direction vector and move the XR content along the horizontal or slanted line segment left to right or vice versa in the x-axis and/or z-axis directions for a horizontal line segment) according to the head motion of an operator.
  • Figure 7 illustrates another example of displaying XR content, according to various embodiments.
  • a spline 702 that circumscribes a portion of computer-assisted device 110, shown as a columnar beam of computer-assisted device is the virtual geometric feature in this example.
  • Figure 8 illustrates another example of displaying XR content, according to various embodiments.
  • spline 702 is again the virtual geometric feature in this example.
  • gaze direction vector 802 does not intersect surface 704 that is generated by extruding spline 702 along an axis associated with the columnar beam of computer-assisted device 110.
  • XR content 806 is displayed coincident to spline 702 based on a closest point on spline 702 to gaze direction vector 802. The closest point is shown as point 804.
  • the disclosed techniques display XR content based on the position (and optionally orientation) of a portion of an operator and the position (and optionally orientation) of a portion of an object.
  • the XR content is displayed in positions (and optionally orientations) that are more easily viewable, are more easily interacted with, are more ergonomic, increase visibility or accessibility of the XR content, help increase the efficiency of operations performed with the XR device, reduce operator discomfort, reduce the need for operator repositioning, and/or the like.
  • the displayed XR content can include content to entertain, aid or instruct, present virtual controls, and/or otherwise provide input for or output to the operator.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Des techniques d'affichage de contenu de réalité étendue (XR) selon l'invention comprennent les étapes ci-après. Un système de réalité étendue (XR) comprend un dispositif XR et un système de processeur. Le système de processeur est configuré pour : déterminer une position d'opérateur d'une partie d'opérateur d'un opérateur sur la base de premières données de capteur, déterminer une première position d'objet d'une partie d'objet d'un objet sur la base de secondes données de capteur et déterminer une première position d'élément pour un élément géométrique virtuel sur la base de la première position d'objet. Le système de processeur est également configuré pour déterminer une première position de contenu sur la base de la position d'opérateur tout en utilisant l'élément géométrique virtuel situé au niveau de la première position d'élément pour limiter la première position de contenu et amener le dispositif XR à afficher un contenu XR sur la base de la première position de contenu.
EP23828281.8A 2022-11-15 2023-11-14 Techniques d'affichage de contenu de réalité étendue sur la base de paramètres relatifs à un opérateur Pending EP4619845A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202263425618P 2022-11-15 2022-11-15
PCT/US2023/037284 WO2024107455A1 (fr) 2022-11-15 2023-11-14 Techniques d'affichage de contenu de réalité étendue sur la base de paramètres relatifs à un opérateur

Publications (1)

Publication Number Publication Date
EP4619845A1 true EP4619845A1 (fr) 2025-09-24

Family

ID=89308207

Family Applications (1)

Application Number Title Priority Date Filing Date
EP23828281.8A Pending EP4619845A1 (fr) 2022-11-15 2023-11-14 Techniques d'affichage de contenu de réalité étendue sur la base de paramètres relatifs à un opérateur

Country Status (3)

Country Link
EP (1) EP4619845A1 (fr)
CN (1) CN120476366A (fr)
WO (1) WO2024107455A1 (fr)

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10410422B2 (en) * 2017-01-09 2019-09-10 Samsung Electronics Co., Ltd. System and method for augmented reality control

Also Published As

Publication number Publication date
WO2024107455A1 (fr) 2024-05-23
CN120476366A (zh) 2025-08-12

Similar Documents

Publication Publication Date Title
JP7802047B2 (ja) ロボット操作手術器具の位置を追跡システムおよび方法
US11244508B2 (en) Augmented reality display for surgical procedures
EP4137033B1 (fr) Système et procédé de restauration de vue
JP7851549B2 (ja) コンピュータ支援インターベンション用の空間認識ディスプレイ
JP7532649B2 (ja) 外科用ロボットによる内視鏡の制御
US20250032207A1 (en) System and method for assisting tool exchange
US20250387172A1 (en) System and method of displaying images from imaging devices
US20260061620A1 (en) Imaging device control in viewing systems
US20230186574A1 (en) Systems and methods for region-based presentation of augmented content
Zinchenko et al. Virtual reality control of a robotic camera holder for minimally invasive surgery
WO2022125699A1 (fr) Commande de dispositif d'imagerie par l'intermédiaire de multiples modalités d'entrée
US20240335245A1 (en) Techniques for adjusting a field of view of an imaging device based on head motion of an operator
US20250143819A1 (en) Techniques for repositioning a computer-assisted system with motion partitioning
EP4619845A1 (fr) Techniques d'affichage de contenu de réalité étendue sur la base de paramètres relatifs à un opérateur
US20250366927A1 (en) Visual guidance for repositioning a computer-assisted system
US20250390171A1 (en) Techniques for updating a registration transform between an extended-reality system and a computer-assisted device
US12616540B2 (en) Imaging device control via multiple input modalities
US20250205900A1 (en) Setting and using software remote centers of motion for computer-assisted systems
WO2025151641A1 (fr) Techniques de mise à jour d'un ancrage de réalité étendue partagé sur la base d'interactions d'opérateur avec un dispositif assisté par ordinateur
WO2024211671A1 (fr) Détermination automatisée de paramètres de déploiement pour un système assisté par ordinateur
Staub et al. Setup of a Scientific Research Platform for Robot-Assisted Minimally Invasive Heart Surgery Scenarios

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20250530

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)