WO2015123771A1

WO2015123771A1 - Gesture tracking and control in augmented and virtual reality

Info

Publication number: WO2015123771A1
Application number: PCT/CA2015/050120
Authority: WO
Inventors: Dhanushan Balachandreswaran; Reuben Anthony FERNANDEZ
Original assignee: Sulon Technologies Inc
Current assignee: Sulon Technologies Inc
Priority date: 2014-02-18
Filing date: 2015-02-18
Publication date: 2015-08-27
Anticipated expiration: 2016-08-18

Abstract

Tracking of location, motion and orientation of user objects within a physical environment for display to a user equipped with a head mounted display is described. Further, tracking of biometric information of the user is described. A processor maps the substantially real-time location and orientation of the user objects to a virtual map of the physical environment, and further maps gestures performed by the user, as detected by biometric sensors.

Description

GESTURE TRACKING AND CONTROL IN AUGMENTED

AND VIRTUAL REALITY

TECHNICAL FIELD

[0001] The following relates generally to biometric tracking systems for augmented reality and virtual reality applications, and more specifically to biometric tracking systems for tracking a user's position and gestures.

BACKGROUND

[0002] The range of applications for augmented reality (AR) and virtual reality (VR) visualization has increased with the advent of wearable technologies and 3-dimensional (3D) rendering techniques. AR and VR exist on a continuum of mixed reality visualization.

SUMMARY

[0003] In one aspect, a system for gesture tracking and control for augmented and virtual reality applications is provided, the system comprising: (a) a user object coupled to a user, the user object comprising: (i) a biometric module comprising a biometric sensor for determining biometric information for a user; biometric information comprising gesture information; and (ii) a location, motion and orientation module for determining location, motion and orientation information for the user; and (b) a processor in communication with the user object, the processor configured to: (i) obtain the location, motion and orientation information for the user object; (ii) obtain the biometric information for the user object; (iii) determine location and orientation of the user object from the location, motion and orientation information; and (iv) determine one or more gestures being performed by the user from the biometric information.

[0004] In another aspect, a method for gesture tracking and control for augmented and virtual reality applications is provided, the method comprising: (a) providing a user object to be coupled to a user, the user object comprising: (i) a biometric module comprising a biometric sensor for determining biometric information for a user; biometric information comprising gesture information; and (ii) a location, motion and orientation module for determining location, motion and orientation information for the user; and (b) configuring a processor to be in communication with the user object, the processor configured to: (i) obtain the location, motion and orientation information for the user object; (ii) obtain the biometric information for the user object; (iii) determine location and orientation of the user object from the location, motion and orientation information; and (iv) determine one or more gestures being performed by the user from the biometric information.

[0005] These and other aspects are contemplated and described herein. It will be appreciated that the foregoing summary sets out representative aspects of systems and methods for tracking user objects within a physical environment to assist skilled readers in understanding the following detailed description.

DESCRIPTION OF THE DRAWINGS

[0006] A greater understanding of the embodiments will be had with reference to the Figures, in which:

[0007] Fig. 1 illustrates in schematic form a system for tracking user objects in a physical environment occupied by a plurality of users equipped with head mounted displays;

[0008] Fig. 2 illustrates an exemplary configuration of biometric tracking systems upon a user equipped with an HMD;

[0009] Fig. 3 illustrates an exemplary configuration of a head mounted display comprising components of a biometric tracking system;

[0010] Fig. 4 illustrates an exemplary configuration of a peripheral comprising a biometric tracking system;

[001 1] Fig. 5 is a flowchart illustrating a method of processing biometric information relating to a gesture;

[0012] Fig. 6 is a flowchart illustrating a method of processing biometric information relating to a user's vital signs; and,

[0013] Fig. 7 is a flowchart illustrating a method of processing information from location, motion and orientation systems as well as biometric information.

DETAILED DESCRIPTION

[0014] It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the Figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practised without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.

[0015] It will be appreciated that various terms used throughout the present description may be read and understood as follows, unless the context indicates otherwise: "or" as used throughout is inclusive, as though written "and/or"; singular articles and pronouns as used throughout include their plural forms, and vice versa; similarly, gendered pronouns include their counterpart pronouns so that pronouns should not be understood as limiting anything described herein to use, implementation, performance, etc. by a single gender. Further definitions for terms may be set out herein; these may apply to prior and subsequent instances of those terms, as will be understood from a reading of the present description.

[0016] It will be appreciated that any module, unit, component, server, computer, terminal or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and nonremovable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Further, unless the context clearly indicates otherwise, any processor or controller set out herein may be implemented as a singular processor or as a plurality of processors. The plurality of processors may be arrayed or distributed, and any processing function referred to herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified. Any method, application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.

[0017] The present disclosure is directed to systems and methods for augmented reality (AR). However, the term "AR" as used herein may encompass several meanings. In the present disclosure, AR includes: the interaction by a user with real physical objects and structures along with virtual objects and structures overlaid thereon; and the interaction by a user with a fully virtual set of objects and structures that are generated to include renderings of physical objects and structures and that may comply with scaled versions of physical environments to which virtual objects and structures are applied, which may alternatively be referred to as an "enhanced virtual reality". Further, the virtual objects and structures could be dispensed with altogether, and the AR system may display to the user a version of the physical environment which solely comprises an image stream of the physical environment. Finally, a skilled reader will also appreciate that by discarding aspects of the physical environment, the systems and methods presented herein are also applicable to virtual reality (VR) applications, which may be understood as "pure" VR. For the reader's convenience, the following may refer to "AR" but is understood to include all of the foregoing and other variations recognized by the skilled reader.

[0018] Existing techniques for tracking users and peripherals in AR applications generally lack accuracy and exhibit increasing error over time. This is typically caused by existing tracking techniques incorporating inertial measurement units and relying on those units for continuous tracking of orientation. Since the measurement has error, the error compounds over time. Thus, it has typically been impractical to use these devices for determining user or peripheral orientation.

[0019] The following provides a system and method for gesture tracking and control in augmented reality and virtual reality applications. The system provides user objects, which include biometric tracking systems that comprises a biometric tracker in communication with a processor. The biometric tracker is configured to obtain biometric information of a user and comprises one or more location, motion and orientation tracking modules. The processor is configured to obtain the biometric information and location, motion and/or orientation information from the biometric tracker. The processor processes the biometric and location, motion and/or orientation information in order to determine characteristics of the user at the user object. In various embodiments described herein, the user object is a wearable device worn by a user. In some of these embodiments, the wearable device comprises an electromyogram, electrocardiogram and/or photoplethysmogram.

[0020] Embodiments of biometric tracking ("BT") systems are provided herein for use in AR applications for tracking location, motion and/or orientation ("LMO") information and biometric information of users. The BT systems may be provided in head mounted displays ("HMDs") worn by users, as well as in wearable peripherals ("peripherals") coupled to the user. User associated HMDs and peripherals are collectively referred to as "user objects".

[0021] Embodiments of BT systems are provided comprising location, motion and orientation modules ("LMO modules") for tracking location, motion and/or orientation information of a user object, and further comprising biometric modules comprising biometric sensors for tracking biometric information of the user at the user object. LMO information may be provided to a processor for processing in order to determine the LMO of the user object. The LMO of the user object may further be processed according to a predetermined kinematic relationship between the user object and an associated virtual object in an augmented reality or virtual reality application, to determine the position and orientation of the virtual object associated with the user object.

[0022] BT systems further comprise biometric tracking modules. The biometric tracking modules generate biometric information corresponding to the user at the user object. The biometric information may include, for example, information related to the gestures, heart rate, or other biometric information. The biometric information may be provided to a processor for processing in order to track, for example, gestures performed by a user, or to determine a user's vital signs, such as the user's heart rate. In the case of gestures, for example, during processing, biometric information may be compared to a library of gestures to determine a particular gesture performed by a user. As an illustrative example, a user object may be an armband worn on the user's forearm, the armband comprising an electromyogram and the armband being in data communication with the processor. The electromyogram is operable to obtain electrical activity of muscles in the forearm. Resulting, movement of the forearm, wrist, hand, fingers or thumb will be measureable by the user object and can be communicated to the processor. The processor may have access to a database of known gestures, such as wrist or finger movements, and their associated musculatory response. The processor can compare the obtained biometric information to the known gestures to determine a most likely gesture having been made by a user.

[0023] Examples of the use of such gesture information include providing user input to augmented reality applications. The corresponding location, motion and orientation information can be used to map additional motion information relating to tracked gestures to a virtual map.

[0024] The BT systems described herein may provide sufficient LMO information and biometric information to avoid the traditional requirement of using physical controllers to receive user input or provide user tracking. [0025] Referring now to Fig. 1 an exemplary scenario is shown in which multiple users occupy a physical environment. The users are equipped with HMDs 12 and peripherals 13. Each HMD and peripheral may be equipped with a BT system to provide LMO information and biometric information to a processor. The singular "processor" is used herein, but it will be appreciated that the processor may be distributed amongst the components occupying the physical environment, within the physical environment or in a server 14 in network communication with a network 17 accessible from the physical environment. For example, the processor may be distributed between the HMDs 12 and a console 11 , or over the Internet via the network 17. The HMD 12 may communicate with the peripherals 13, or the HMD 12 and peripherals 13 may communicate directly with console 1 1 or server 14 located over a network 17 accessible from the physical environment, as shown. Communication between the processor and BT systems of the user objects may be via suitable wired or wireless communication. Each BT system may comprise a communication module configured to transmit LMO information and biometric information for the system to the processor, or communication of information may be routed through HMDs, as shown in Fig. 1. In various embodiments, the processor can access a gesture library 19 which stores information relating to predetermined gestures, as described in more detail below. As illustrated, the gesture library 19 may be communicatively linked with server 14.

[0026] Referring now to Fig. 2, shown therein is a user 1 having disposed thereon various user objects, including an HMD 12 and peripherals 13. The user objects in this case are wearable devices, wherein each peripheral is an armband, wristband, legband, etc. Other suitable configurations are contemplated herein.

[0027] As illustrated in the exemplary peripheral shown in Fig 4, peripherals 13 comprise BT systems for tracking LMO information and biometric information.

[0028] As shown in Fig. 2, the user 1 may be equipped with an HMD 12, as well as peripherals 13 disposed upon her hands and feet.. Each peripheral 13 may comprise a BT system for providing LMO information and biometric information for the limb or body part upon which it is disposed.

[0029] Referring now to Fig. 3, an exemplary HMD 12 configured as a helmet is shown; however, other configurations are contemplated. The HMD 12 may comprise a processor 130 for generating a rendered image stream comprising CGI. The processor 130 shown in Fig. 3 is shown mounted within the HMD 12; however, as previously described, the processor may be located separately from the HMD 12. The processor may communicate with the following components of the HMD 12: (i) a scanning system for scanning the physical environment surrounding the HMD 12; (ii) an LMO module 21 for determining LMO information for the HMD;

(iii) a biometric module 22 comprising at least one biometric sensor for providing biometric information, which may include biometric information relating to gesture tracking of the HMD;

(iv) an imaging system, such as, for example, a camera system comprising one or more cameras 123, to capture a physical image stream of the physical environment; (v) a display system 121 for displaying to a user of the HMD 12 the physical image stream and/or the rendered image stream; (vi) a power management system 113 for distributing power to the components; (vii) a sensory feedback system comprising, for example, haptic feedback devices 120, for providing sensory feedback to the user; and (vii) an audio system 124 with audio input and output to provide audio interaction. The HMD 12 may further comprise a wireless communication system 126 having, for example, antennae, to communicate with other components in an AR and/or VR system, such as, for example, other HMDs, peripherals, actors, a gaming console, or a router. In various embodiments described in more detail below, the LMO module 21 of the HMD comprises a transmitter or receiver of a magnetic tracking system. As further described below, in alternate embodiments, the LMO module 21 comprises a tracking marker configured to be tracked by a camera to determine LMO information of the HMD.

[0030] Referring now to Fig. 4, shown therein is an illustrative embodiment of a peripheral 13 comprising a BT system 15, the components and functionality of which will be described in more detail below. The BT system 15 comprises an LMO module 21 for providing LMO information and a biometric module 22 for providing biometric information. The LMO module 21 may provide LMO information of the peripheral in a physical environment, or may provide LMO information for the peripheral relative to the HMD 12. The biometric module 22 may provide biometric information to the processor for detecting gestures performed by the user, or to detect the user's vital signs, such as a user's heartbeat.

[0031] According to the illustrative embodiment of the BT system 15, a user 1 wears the peripheral 13 comprising the BT system 15 on their forearm. As the user moves her hand and/or arm, the LMO module 21 provides real-time LMO information which the processor may process to map the user's motions to a virtual map. The LMO module 21 may comprise a transmitter/receiver of a magnetic tracking system, or may comprise a marker imaged by an associated camera in an active tracking system, either of which may provide LMO information for the peripheral to the processor via a communication module. The BT system 15 receives biometric information from a biometric sensor of the biometric module 22, such as an electromyographic sensor, and provides the biometric information to the processor to determine any gestures performed by the user. As the user flexes their finger 16, the processor may determine that biometric information detected by the biometric sensor corresponds to a predetermined gesture stored in a gesture library, such as a trigger pulling gesture associated with flexing of the index finger 16. The processor may further receive information from the gesture library relating to the predetermined gesture, such as associated motion and user input information. Upon detecting the trigger pulling gesture and receiving the associated user input information, the processor may provide user input to an AR or VR application, such as to control a virtual gun to fire. Motion information received for the predetermined gesture may cause the processor to map the user's finger motion to the virtual map based on an approximate finger motion expected for the trigger pulling gesture.

[0032] Further, the processor may process the LMO information for the user object to determine a position and orientation of a virtual object associated with the user object in an AR application - such as a virtual gun. The processor may correlate the LMO information of a given peripheral 13 to a virtual object by way of a predetermined kinematic relationship between the peripheral and the virtual object. For example, where a virtual gun is associated with the peripheral 13 in an AR application, the LMO information may be processed so that any virtual bullet fired from the virtual gun is represented in a rendered image stream according to the determined position and orientation of the virtual gun at a time of firing, along the trajectory 18. For example, if the LMO module of the peripheral 13 is a magnetic transmitter and the HMD comprises a corresponding magnetic receiver, the magnetic receiver is preferably located near the cameras 123 at a fixed distance therefrom. The magnetic receiver provides the relative distance and orientation of the peripheral relative to the receiver. Since the offset between the receiver and the cameras is known, the processor may transform the relative position and orientation from relative to the receiver to a relative position and orientation relative to the camera. The processor may then map motions of the peripheral 13 to the virtual map so that the peripheral 13 may be rendered as a virtual object having motions corresponding to the peripheral's 13 motions within the physical environment.

[0033] Accordingly, LMO information from the peripheral 13 may be used to determine the position and orientation of a virtual object associated with the peripheral in an AR application. It will be understood that the position and orientation of a virtual object may be determined based on what, if any, virtual object is currently associated with the peripheral in the AR and VR application, and that various virtual objects are contemplated. For example, a virtual object may comprise a flashlight, and the processor may process LMO information from the peripheral to determine and render the orientation of the flashlight and its emitted light, and update the orientation of the flashlight in real time.

[0034] Various embodiments and components of the LMO module 21 for providing LMO information of user objects to the processor will now be described.

[0035] An LMO module 21 may comprise one or more of the following: (i) a position sensor configured to track a position or orientation, the position sensor comprising, for example, one or more of a magnetic, ultrasound, radio-frequency (RF), LiDAR or radar type of sensor, a tracking maker with an associated camera, whether alone or in combination; and (ii) a motion sensor such as an inertial measurement unit configured to track motions, the motion sensor comprising, for example, one or more of the following: an accelerometer, a gyroscope, a magnetometer, whether alone or in combination. The sensors in the LMO module may be, for example, MEMS- type sensors. Certain sensor configurations may provide all LMO information as a single suite. For example, a magnetic position tracking system alone may provide real-time location, positioning and orientation information.

[0036] Where a user object comprises an LMO module having more than one position sensor or motion sensor, readings from more than one sensor may be processed together to provide a more accurate reading of LMO for the user object. By way of example, readings from an inertia measuring unit may be combined with measurements from a magnetic tracking system to improve accuracy of determined position and orientation.

[0037] In embodiments, such as for magnetic position tracking, an LMO module 21 comprises a receiver or a transmitter of a receiver- transmitter pair for determining relative LMO information of the receiver with respect to the transmitter (or vice versa). In such embodiments, the receiver of the receiver-transmitter pair may be mounted to an HMD worn by the user or may be located elsewhere in the physical environment nearby the user. The transmitter may be coupled to the user object and may transmit a signal comprising information which can be processed for determining the position of the transmitter relative to the receiver. The receiver may be configured to provide information relating to a detected signal from the transmitter to the processor. The transmitter and receiver repeatedly and continuously transmit and receive, respectively, signals to provide substantially real-time LMO information for the LMO module.

[0038] For magnetic tracking, the transmitter may be a magnetic source and the receiver may be a magnetic receiver/antenna. The pair can be configured so that the receiver is able to detect the field of the transmitter, and the receiver may communicate characteristics of the detected field to the processor for determining LMO information of the receiver relative to the transmitter. Where a user is outfitted with an HMD and a peripheral configured with matched transmitter- receiver pairs, the processor may process signals detected by the receiver to provide LMO information of the peripheral relative to the HMD. The processor may use the LMO information to map the substantially real-time location and orientation of the peripheral relative to the position of the HMD.

[0039] In such embodiments, the magnetic receiver is preferably mounted to the HMD and the magnetic source is preferably mounted to (or otherwise disposed upon) the peripheral. The receiver mounted to the HMD may provide information to the processor from a plurality of transmitters mounted to a plurality of peripherals, without requiring additional wireless or wired communications systems. The transmitters may vary transmission characteristics (such as frequency or modulation), to facilitate identification by the receiver of the source transmitter of transmitted information. By way of example, each magnetic source provides a transmission having different magnetic field characteristics. Alternatively, the receiver may be mounted to the peripheral while the transmitter may be mounted to the HMD, and the LMO information for the HMD and peripheral may be provided to the processor from the peripheral via wired or wireless communication.

[0040] In alternate embodiments, an LMO module on a user object comprises a tracking marker for tracking by a camera to determine LMO information of the user object. Active tracking may determine LMO information for a user object by: (1) mounting a visual marker on the LMO module of a user object; (2) imaging the marker with a camera; (3) providing the camera's captured images to a processor for processing; and (4) if the processor detects a marker in the captured images, the processor may determine the position and orientation of the marker, and its associated user object from an image of the marker in each captured image. Accordingly, as a user object comprising a marker moves in 3D space within the camera's field of view (FOV), the user object's LMO can continue to be tracked as the camera continues to capture additional images and sends them to the processor for processing to determine additional LMO information of the user object. The camera may be oriented to have the user object within its field of view. Markers may include light emitters, such as active LEDs or infrared markers. In addition, the LMO module of the user object may further comprise an inertial measurement unit (IMU) to provide orientation measurements of the user object to the processor and to be combined by the processor with LMO information determined from the captured images. The IMU may allow for more accurate measurements of location, motion and orientation of user objects.

[0041] In embodiments comprising active tracking, where markers comprise coloured light emitters, such as LEDs, markers can be independently controlled by signals from the processor to change in color to reflect a specific assigned controller number for each of a plurality of user objects, such as peripherals. For example, if a user wears four peripherals, an LED color can be provided for each peripheral (e.g. blue, red, green, yellow, etc.). This may allow for more accurate tracking. The use of different LED marker colours is not limited to controller identity, but could extend to other applications requiring tracking of different user objects.

[0042] In embodiments, the LMO module of a peripheral coupled to a particular HMD may provide LMO information as values or vectors relative to the location and orientation of the HMD at a point in time. For example, the HMD and peripheral may comprise a magnetic position tracking transmitter/receiver pair, configured to provide location and orientation information for the peripheral relative to the HMD. The processor may map the location and orientation of the peripheral with reference to the HMD's location and orientation.

[0043] Various embodiments and components of the biometric module 22 will now be described.

[0044] Biometric modules comprise biometric sensors configured to gather biometric information from the user and provide that information to a processor for various uses in AR and VR applications. Biometric sensors may comprise sensors for use in tracking a user's gestures, such as myography sensors, including, for example, electromyography (EMG), mechanomyography (MMG) or phonomyography sensors, to measure the user's gestures and body movements.

[0045] Tracked gestures may be mapped to a virtual map, processed to provide an associated user input or may control functionality of an HMD, such as activating or deactivating the HMD. Biometric sensors may further comprise sensors for measuring a user's vital signs, such as a user's heart rate. Biometric sensors for measuring a user's heart rate may include sensors for providing an electrocardiogram (ECG) or a photoplethysmogram (PPG).

[0046] It will be appreciated that biometric sensor readings may be processed having regard to where a biometric sensor is mounted on a user. Accordingly, sensor readings provided by a biometric sensor mounted to a peripheral on a user's calf, may be processed having regard to the fact that they relate to muscle activity of the calf. [0047] In embodiments, EMG sensors may measure a user's muscle activity at the surface of the skin to provide sensor readings. A processor may correlate the sensor readings to predetermined gestures. Predetermined gestures may relate to measured biometric information such as a contraction or stretching of individual or a combination of muscles. While performing a gesture, activated muscles can be determined and measured.

[0048] Sensors for providing an ECG may include electrodes which measure the electrical activity of the heart to provide sensor signals. Specifically the sensors generate sensor signals relating to detected electrical discharges on the skin caused by heart activity.

[0049] Sensors for providing a PPG may include pulse oximeters. These PPGs generate sensor signals relating to measured volumetric changes of the human body such as lung displacement, or in a specific example, volumetric changes in arteries which can be used to calculate heartbeat. One method of measuring heart rate comprises utilizing light-emitting diodes (LED) to illuminate a user's skin. The amount of light that is transmitted or reflected to a photodiode can be correlated to pulse rate.

[0050] Biometric information from biometric sensors for use in tracking gestures may be processed by a processor according to the blocks 300 illustrated in Fig. 5 in order to correlate measured sensor readings from biometric sensors to predetermined gestures stored in a gesture library. At block 302 sensor readings may be received by a processor from a biometric sensor for use in tracking a user's gestures, such as a myographic sensor. At block 304, the sensor readings may be processed to provide a clean gesture signal. More specifically, the processing may comprise digital signal processing (DSP), noise reduction, amplification, elimination of motion artifacts and filtering. The processing can be done through hardware or software filters, rectifiers, or other techniques known to those of skill in the art of signal processing. Such filters can be built or programmed for filtering of specific frequencies, or for providing other signal processing. At block 306 the processor may compare and analyze the clean gesture signal to pre-determined gestures stored in a gesture library. At block 308 the processor may determine the particular predetermined gesture relating to the clean gesture signal, and the processor may retrieve information relating to the predetermined gesture, such as associated motion information, and associated user input information. At block 310, the particular predetermined gesture and its associated motion and user input information may be processed for use in various AR applications. As illustrated, the steps performed in relation to the blocks 300 may then repeat. The processor may repeatedly call the biometric sensors to provide sensor readings. [0051] Referring specifically to blocks 306, 308 and 310, the gesture library may comprise a plurality of predetermined gestures, and associated information for each gesture of the plurality of gestures, such as associated motion information, and user input information. Motion information may provide information for mapping a gesture performed by a user to a virtual map - e.g. for mapping a finger moving through a trigger pull gesture to a virtual map. User input information may identify user input associated with a gesture. By way of example, a trigger pull gesture may be identified, and user input information associated with the trigger pull gesture may be processed at block 310 in order to cause a virtual gun to be actuated. The gesture library may be expanded as needed. Some gestures that may be stored in the gesture library include tapping a finger, giving a thumbs up, closing a fist, opening and flexing open a hand and pulling a trigger (i.e. curling an index finger).

[0052] Any predetermined gesture determined to have been performed by the user, and its associated user input, may be processed in relation to dynamic game rules, and in relation to characteristics of an AR or VR environment. For example, pulling a trigger may serve as user input to cause a user to actuate a virtual gun, if a gun is held. Flexing open a hand may cause a user to drop a user's weapon, if a weapon is held. Closing a fist may cause a user to pick up a weapon, if the user is near a virtual weapon in a virtual environment, and the user does not currently hold a weapon. Conversely, closing a fist may cause a user to punch if the user is not near a weapon in the virtual environment. It will be understood that various predetermined gesture may correlate to various user inputs and game mechanics, and that the above examples are merely illustrative.

[0053] Predetermined gestures may be associated with different user inputs depending on which peripheral they are detected from. By way of example, a user could wear one peripheral which may be tracked to permit a user to open, close and navigate a virtual game menu. While the user's may wear another peripheral to track user inputs relating to gameplay.

[0054] Biometric information from biometric sensors for use in tracking a user's vital signs may be processed by a processor according to the blocks 320 as illustrated in Fig. 6. At block 322 sensor readings may be received from a biometric sensor for use in tracking a vital signs, such as the user's heart rate. At block 304, the sensor readings may be processed to provide a clean vital sign signal. More specifically, the processing may comprise digital signal processing (DSP), noise reduction, amplification, elimination of motion artifacts and filtering. The processing can be done through hardware or software filters, rectifiers, or other techniques known to those of skill in the art of signal processing. Such filters can be built or programmed for filtering of specific frequencies, or for providing other signal processing. At block 306 the processor may process the clean vital sign signal to determine a particular vital sign, such as a user's current heart rate. At block 308, the processor may further process the user's vital sign for use in various AR and VR applications. As illustrated, the steps performed in relation to the blocks 300 may then repeat. The processor may repeatedly call the biometric sensors to provide sensor readings.

[0055] Referring specifically to block 308, by way of example, vital signs comprising a user's heart rate may be provided to a display of an HMD. Further, vital signs may be processed and provided to a feedback system to provide physical feedback, visual feedback or audio feedback. For example, biometric information indicating that a user has a rapid heart rate may be processed and provided to a feedback system to generate pulsing haptic feedback correlating to measured pulses of the user's heart rate.

[0056] It will be understood that the biometric sensors need not be restricted to locations as illustrated in Fig. 2, but may be positioned in various locations for measurement of specific biometric data. By way of example, biometric sensors may be positioned proximally to particular muscular tissue that is desired to be measured for muscular activity for detecting a particular gesture.

[0057] Further, a biometric module may comprise any combination of biometric sensors. By way of example, a user may be provided with a biometric module comprising both an EMG and a PPG sensor. The user may be playing a horror game, and wearing a peripheral on one of their forearms comprising an EMG and a PPG sensor. For example, the EMG could track a user's gesture of holding a fist which may be detected and processed such that the user holds up a virtual flashlight. If the EMG detects that the user opens their hand, the flashlight could turn off. Owing to the scary nature of horror games, the user's virtual avatar may be in a dark and eerie setting, creating a sense of fear. The peripheral's PPG may measure the user's heart rate. As the user's heart rate increases, the heart rate could be processed and provided to a feedback system so that an audio pulse is played indicating the user's heart beat through an audio system of the HMD, adding to the eerie setting of the gameplay environment, and further immersing the user within the AR application. Accordingly, various biometric sensors can be provided in combination to provide different inputs or user interaction. It will also be understood that different biometric sensors may be provided on different peripherals to provide the same or additional functionality; for example, a user may be wearing a peripheral comprising an EMG sensor on one arm, and a second peripheral comprising a PPG sensor on the user's other arm. [0058] Referring now to Figs. 7, blocks 340 illustrate steps performed in processing information from a biometric tracking system 15 for tracking a user object's LMO, gestures performed by the user, and the user's vital signs. The biometric system 15 comprises an LMO module 21 and a biometric module 22. The biometric module 22 comprises biometric sensors for gesture tracking and biometric sensors for tracking a user's vital signs.

[0059] At block 342 the LMO module 31 receives LMO sensor readings relating to the position and orientation of a user object. At block 352, a processor receives the LMO sensor readings and processes the readings to determine the LMO of the user object. At block 344, the biometric module receives biometric information from biometric sensors for gesture tracking, such as myographic sensors. At block 354, the biometric information is processed to determine gestures performed by the user in proximity to the user object. The processor may receive information associated with the determined gestures, such as associated user input information for providing a user input associated with the gesture to AR applications, and associated motion information for use in mapping any determined gestures to a virtual map. The steps performed in relation to block 354 are further described above in relation to Fig. 5. At block 346, the biometric module retrieves further biometric information from further biometric sensors for tracking a user's vital signs. At block 356, the processor processes the further biometric information to determine a current vital sign of the user, such as the user's heart rate. The steps performed in relation to block 356 are further described above in relation to Fig. 6. At block 360, the processor processes the determined LMO of the user objects, determined gestures and the user's vital signs to generate outputs, for use as inputs for AR applications. The processor may further process the outputs to generate a rendered image stream comprising computer generated imagery ("CGI") with respect to the user objects. The processor may process the LMO information to determine the LMO of a virtual object associated with the user object in an AR application according to a predetermined kinematic relationship. The processor may process the information associated with the determined gestures to map additional motion information relating to a determined gesture to a virtual map or to provide user input to AR applications. The processor may then transmit a rendered image stream to the display system of an HMD for display to user thereof. The steps described in relation to blocks 340 may be repeated, as the biometric system 15 may continuously poll the LMO module and biometric module.

[0060] In some embodiments, at block 362, the outputs may be provided to a feedback system for providing feedback to the user relating to the outputs. In embodiments, at least one user object, such as a peripheral, may comprise a feedback system. The feedback system may be configured to provide physical, audible, or visual feedback to a user in response to detected gestures or vital signs. Physical feedback may include vibrations and micro-shocks. Visual feedback may include activation of blinking LEDs. Audible feedback may include an audible notification. By way of example, the physical feedback could be activated in response to detection of a gesture indicating that a user has pulled a trigger. Visual feedback could be activated in response to particular game parameters, as determined by the processor. By way of example, visual feedback could be provided to user when a particular task is completed in an AR/VR application. The feedback system may not necessarily be solely activated in relation to outputs from the processor, for example the feedback system may be configured to provide an indication to the user when the HMD is being powered on or off.

[0061] In aspects, the foregoing systems and methods may enable a user equipped with a wearable LMO module, and optionally biometric modules, to perform exercises, the motions of which may be mapped and tracked for analysis by, for example, the user's trainer.

[0062] In further aspects, the foregoing systems and methods may enable a user to use gesture controls to interact with a processor, for example to initiate AR or VR gameplay in a physical environment.

[0063] Although the foregoing has been described with reference to certain specific embodiments, various modifications thereto will be apparent to those skilled in the art without departing from the spirit and scope of the invention as outlined in the appended claims. The entire disclosures of all references recited above are incorporated herein by reference.

Claims

CLAIMS We claim:

1. A system for gesture tracking and control for augmented and virtual reality applications, the system comprising:

a) a user object coupled to a user, the user object comprising:

i) a biometric module comprising a biometric sensor for determining biometric information for a user; biometric information comprising gesture information; and

ii) a location, motion and orientation module for determining location, motion and orientation information for the user; and

b) a processor in communication with the user object, the processor configured to: i) obtain the location, motion and orientation information for the user object; ii) obtain the biometric information for the user object;

iii) determine location and orientation of the user object from the location, motion and orientation information; and

iv) determine one or more gestures being performed by the user from the

biometric information.

2. The system of claim 1 , wherein the processor generates a virtual object having a virtual object position and orientation associated with the user object.

3. The system of claim 2, wherein determining the position and orientation of a virtual object comprises:

a) obtaining the location and orientation of the user object;

b) determining a virtual object associated with the user object;

c) obtaining a predetermined kinematic relationship between the user object and the virtual object; and

d) determining the location, motion and orientation of the virtual object from the location and orientation of the user object and the kinematic relationship.

4. The system of claim 1 , wherein the biometric sensor comprises a myography sensor for measuring a user's muscle activity.

5. The system of claim 1 , wherein the processor determines the gestures by:

a) obtaining a gesture signal from the biometric sensor;

b) comparing the gesture signal to one or more predetermined gestures stored in a gesture library in communication with the processor; and

c) determining a particular predetermined gesture relating to the gesture signal.

6. The system of claim 5, wherein the gesture signal is filtered prior to comparison to the

predetermined gestures.

7. The system of claim 5, wherein the processor is configured to translate the gesture to a user input, and upon determining the gesture has been performed, the processor actuates the user input in the augmented or virtual reality application.

8. The system of claim 5, wherein the gesture information associated with the particular

predetermined gesture comprises gesture motion information, and upon receiving the gesture motion information, the processor processes the gesture motion information for mapping the particular predetermined gesture performed by the user to a virtual map.

9. The system of claim 1 , wherein the user object is a head mounted display.

10. The system of claim 1 , wherein the user object is a wearable peripheral controller.

1 1. A method for gesture tracking and control for augmented and virtual reality applications, the method comprising:

a) providing a user object to be coupled to a user, the user object comprising:

b) configuring a processor to be in communication with the user object, the processor configured to:

i) obtain the location, motion and orientation information for the user object; ii) obtain the biometric information for the user object; iii) determine location and orientation of the user object from the location, motion and orientation information; and

iv) determine one or more gestures being performed by the user from the

biometric information.

12. The method of claim 11 , wherein the processor further generates a virtual object having a virtual object position and orientation associated with the user object.

13. The method of claim 12, wherein determining the position and orientation of a virtual object comprises:

a) obtaining the location and orientation of the user object;

b) determining a virtual object associated with the user object;

14. The method of claim 11 , wherein the biometric sensor comprises a myography sensor for measuring a user's muscle activity.

15. The method of claim 11 , wherein the processor determines the gestures by:

a) obtaining a gesture signal from the biometric sensor;

16. The method of claim 15, wherein the gesture signal is filtered prior to comparison to the predetermined gestures.

17. The method of claim 15, wherein the processor is configured to translate the gesture to a user input, and upon determining the gesture has been performed, the processor actuates the user input in the augmented or virtual reality application.

18. The method of claim 15, wherein the gesture information associated with the particular predetermined gesture comprises gesture motion information, and upon receiving the gesture motion information, the processor processes the gesture motion information for mapping the particular predetermined gesture performed by the user to a virtual map.

19. The method of claim 11 , wherein the user object is a head mounted display.

20. The method of claim 11 , wherein the user object is a wearable peripheral controller.