CN110456902A - Track user movements to control skeletal models in computer systems - Google Patents

Abstract

A system having a sensor module and a computing device. Each sensor module has an inertial measurement unit attached to the user's part. An artificial neural network is used to make predictions about the orientation of the user's parts based on the measurements of the inertial-based modules. For example, an artificial neural network can be trained to predict orientations measured using an optical tracking system based on orientations measured using an inertial measurement unit and/or to predict some stiffness in the kinematic chain based on orientation measurements of other rigid parts of the kinematic chain Orientation measurements of the site. For example, the sensor modules may include different subsets that share a common sensor module; and the artificial neural network may be used to combine different predictions made individually for the common sensor module from the different subsets.

Description

Track user movements to control skeletal models in computer systems

related application

This application claims U.S. Patent Application Serial No. 15/973,137, filed May 7, 2018, and entitled "Tracking User Movements to Control a Skeleton Model in a Computer System" and filed June 1, 2018, and entitled Priority to US Patent Application Serial No. 15/996,389 for "Motion Predictions of Overlapping Kinematic Chains of a Skeleton Model used to Control a Computer System," the entire disclosure of which is hereby incorporated by reference.

This application is related to the following applications: U.S. Patent Application Serial No. 15/868,745, filed January 11, 2018, and entitled "Correction of Accumulated Errors in Inertial Measurement Units Attached to a User"; U.S. Patent Application Serial No. 15/864,860, entitled "Tracking Torso Leaning to Generate Inputs for Computer Systems"; filed December 19, 2017, and entitled "Calibration of Inertial Measurement Units Attached to Arms of a User and to a Head Mounted Device" U.S. Patent Application Serial No. 15/847,669; U.S. Patent Application Serial No. 15 filed November 20, 2017 and entitled "Calibration of Inertial Measurement Units Attached to Arms of a User to Generate Inputs for Computer Systems" /817,646; U.S. Patent Application Serial No. 15/813,813, filed November 15, 2017, and entitled "Tracking Torso Orientation to Generate Inputs for Computer Systems"; filed October 24, 2017, and entitled "Tracking Finger U.S. Patent Application Serial No. 15/792,255, filed October 18, 2017, and entitled "Tracking Arm Movements to Generate Inputs for Computer Systems"; and U.S. Patent Application Serial No. 15/492,915, filed April 20, 2017, and entitled "Devices for Controlling Computers based on Motions and Positions of Hands." The entire disclosures of the aforementioned related applications are hereby incorporated by reference.

technical field

At least a portion of the present disclosure relates generally to computer input devices, and more particularly but not limited to virtual reality and/or augmented/ Input device for mixed reality applications.

Background technique

US Patent Application Publication No. 2014/0028547 discloses a user control device with combined inertial sensors for detecting movement of the device for pointing and selection in real or virtual three-dimensional space.

US Patent Application Publication No. 2015/0277559 discloses a ring mounted touch screen with a wireless transceiver that wirelessly transmits commands generated from events on the touch screen.

US Patent Application Publication No. 2015/0358543 discloses a motion capture device having multiple inertial measurement units for measuring motion parameters of a user's fingers and palm.

US Patent Application Publication No. 2007/0050597 discloses a game controller with an acceleration sensor and a gyroscope sensor. US Patent Application No. D772,986 discloses a decorative design for a wireless game controller.

Chinese Patent Application Publication No. 103226398 discloses a data glove using micro-inertial sensor network technology, wherein each micro-inertial sensor is an attitude reference system with a three-axis micro-electromechanical system (MEMS) micro-gyroscope packaged in a circuit board, Three-axis micro-acceleration sensor and three-axis geomagnetic sensor. Other data gloves are disclosed in US Patent Application Publication No. 2014/0313022 and US Patent Application Publication No. 2012/0025945.

US Patent Application Publication No. 2016/0085310 discloses techniques for tracking hand or body poses from image data, where the best candidate pose from a pool of candidate poses is selected as the current tracked pose.

US Patent Application Publication No. 2017/0344829 discloses an action detection scheme using a recurrent neural network (RNN), where joint positions are applied to the recurrent neural network (RNN) to determine action labels representing the actions of entities depicted in video frames .

The disclosures of the patent documents discussed above are hereby incorporated by reference.

Description of drawings

Embodiments are shown by way of example and not limitation in the figures of the drawings, wherein like reference numerals refer to like elements.

Figure 1 illustrates a system for tracking user movement according to one embodiment.

Figure 2 illustrates a system for controlling computer operations, according to one embodiment.

Figure 3 illustrates a skeletal model that can be controlled by tracking user movements, according to one embodiment.

4 and 5 illustrate a method of training a recurrent neural network (RNN) and using the RNN to predict movement measurements of one tracking system based on movement measurements of another tracking system, according to one embodiment.

6 and 7 illustrate a method of training a recurrent neural network (RNN) and using the RNN to predict movement measurements for omitted tracking devices based on the remaining tracking devices, according to one embodiment.

8 and 9 illustrate a method of tracking user movement using an artificial neural network (ANN) according to one embodiment.

Figure 10 shows the results from different artificial neural networks combined using a Bidirectional Long Short-Term Memory (BLSTM) network, according to one embodiment.

FIG. 11 illustrates another technique for combining results from different artificial neural networks for kinematic chains with overlapping sites, according to one embodiment.

Figure 12 illustrates a method for training multiple artificial neural networks for multiple kinematic chains with overlapping parts, according to one embodiment.

Figure 13 illustrates a method for predicting motion measurements for overlapping parts of multiple kinematic chains as modules using separate artificial neural networks, according to one embodiment.

Figure 14 illustrates a method for using a skeletal model with multiple artificial neural networks for multiple kinematic chains, according to one embodiment.

Detailed ways

The following description and drawings are illustrative and should not be construed as limiting. Numerous specific details are described to provide a thorough understanding. However, in certain instances, well-known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure are not necessarily references to the same embodiment; and, such references mean at least one.

At least some embodiments disclosed herein allow tracking of a reduced number of user's parts using microelectromechanical systems (MEMS) inertial measurement units (IMUs) and use of artificial neural networks to control computer systems with more than a reduced number of articulated A skeletal model of the parts in , where each part can be considered rigid and movable relative to the other parts through rotations at the joints.

A kinematic chain is an assembly of rigid parts connected by joints. The user's skeletal model or user's parts may be constructed as a collection of rigid parts connected by articulations in a manner corresponding to the user's bones or groups of bones (which may be considered rigid parts).

For example, the head, torso, upper left and right arms, left and right forearms, palms, finger phalanges, metacarpal bones of the thumb, thighs, calves, and feet can be considered and knuckles) the rigid part of the connection.

A skeletal model of the user may be constructed based on rigid models of body parts and corresponding joints of the user's parts; and the relative positions and/or orientations of the rigid parts collectively represent the pose of the user and/or the skeletal model. The skeletal model of the user may be used to control the presentation of the user's avatar, to recognize the user's gesture input, and/or to author a virtual reality or augmented reality presentation of the user.

Figure 1 illustrates a system for tracking user movement according to one embodiment.

1 shows various parts of the user, such as the user's torso (101), the user's head (107), the user's upper arms (103 and 105), the user's forearms (112 and 114), and the user's hands (106 and 108). ).

In the application shown in Figure 1, the user's hands (106 and 108) are considered rigid parts that can move around the user's wrist. In other applications, the movement of the user's palm and finger bones relative to the finger joints can also be tracked (eg, to use the relative positions between the fingers of the hand and the palm of the hand to determine gestures made by the user).

In Figure 1, the user wears several sensor devices (111, 113, 115, 117 and 119) that track the orientation of parts of the user that are considered or identified as rigid in the application.

In the application shown in Figure 1, the rigid parts of the user are movable relative to the user's torso (101) and relative to each other. Examples of rigid parts include the head (107), upper arms (103 and 105), forearms (112 and 114), and hands (106 and 108). Joints such as neck, shoulder, elbow and/or wrist connect rigid parts of the user to form one or more kinematic chains. The kinematic chain can be modeled in the computing device (141) to control the application.

In order to track the relative position/orientation of the rigid parts in the kinematic chain, a tracking device can be attached to each individual rigid part in the kinematic chain to measure its orientation.

In general, one of a number of systems known in the art can be used to track the position and/or orientation of rigid features in the reference system (100). Some of the systems may use one or more cameras to take images of a rigid site marked with optical markers and analyze the images to calculate the position and/or orientation of the site. Some of the systems may track a rigid site based on signals (such as radio frequency signals, infrared signals, ultrasonic signals) sent from or received at a tracking device attached to the rigid site . The signal may correspond to a signal received in the tracking device and/or a signal transmitted from the tracking device. Some of the systems may use an inertial measurement unit (IMU) to track the position and/or orientation of the tracking device.

In FIG. 1 , sensor devices (111, 113, 115, 117, and 119) are used to track some of the rigid parts (e.g., 107, 103, 105, 106, 108) in one or more kinematic chains, but Sensor devices are omitted from other rigid locations (101, 112, 114) in the kinematic chain(s) to reduce the number of sensor devices used and/or to improve user experience by wearing a reduced number of sensor devices.

The computing device (141) has a predictive model (141) that is trained to generate an estimate of the user's location (101, 112, 114, 107, 103, 105, 106 and/or 108) of the predicted measurements.

For example, the predictive model (141) can be implemented using an artificial neural network in the computing device (141) to based on rigid parts (107, 103, 105) with attached sensor devices (111, 113, 115, 117 and 119) , 106, 108) to predict the orientation measurements of the rigid parts (101, 112, 114) of the sensor device have been omitted.

Furthermore, the artificial neural network can be trained to predict what will be detected by another system (e.g. Optical tracking system) measurement results of the orientation of the rigid parts (107, 103, 105, 106, 108).

The sensor devices (111, 113, 115, 117, 119) communicate their movement measurements to the computing device (141), which computes data from the attached sensor devices (111, 113, 115, 117 and 119) The obtained measurements are applied as input to an artificial neural network trained as discussed further below to calculate or predict the orientation of the rigid parts (107, 103, 105, 106, 108, 101, 112, 114).

In some embodiments, each of the sensor devices (111, 113, 115, 117, and 119) communicates its measurements directly to the computing device (141) in a manner independent of the operation of the other sensor devices.

Alternatively, one of the sensor devices (111, 113, 115, 117, and 119) may serve as a base unit that receives measurements from one or more other sensor devices and sends bundled and/or combined measurements to the computing device (141). In some cases, an artificial neural network is implemented in the base unit and used to generate predicted measurements that are communicated to the computing device (141).

Preferably, a wireless connection via a personal area wireless network (e.g., a Bluetooth connection) or a local area wireless network (e.g., a Wi-Fi connection) is used to facilitate communication from the sensor devices (111, 113, 115, 117, and 119) to Communication of a computing device (141).

Alternatively, wired connections may be used to facilitate communication between some of the sensor devices (111, 113, 115, 117 and 119) and/or with the computing device (141).

For example, a hand module (117 or 119) attached to or held in a user's corresponding hand (106 or 108) may receive motion measurements of a corresponding arm module (115 or 113) and send the corresponding hand ( 106 or 108) and corresponding upper arm (105 or 103) motion measurements are sent to the computing device (141).

The hand (106), forearm (114) and upper arm (105) can be considered a kinematic chain for which the artificial neural network can be trained based on input from sensor devices (117 and 105) attached to the hand (106) and upper arm (105). 115) (without a corresponding device on the forearm (114)) to predict the orientation measurements generated by the optical tracking system.

Alternatively or in combination, the hand module (e.g., 117) can combine its measurements with those of the corresponding arm Movements to Generate Inputs for Computer Systems" US Patent Application Serial No. 15/787,555", the entire disclosure of which is hereby Incorporated herein by reference.

For example, the hand modules (117 and 119) and the arm modules (115 and 113) can be obtained via U.S. Patent Publication No. 5,199,480, filed April 20, 2017 and entitled "Devices for Controlling Computers based on Motions and Positions of Hands," respectively. The base unit (or game controller) and arm/shoulder modules discussed in , the entire disclosure of which is hereby incorporated by reference.

In some embodiments, the head module (111) is configured as a base unit that receives motion measurements from the hand modules (117 and 119) and arm modules (115 and 113) and bundles the measurement data for sending to the computing device (141). In some cases, computing device (141) is implemented as part of head module (111). The head module (111) may also use an artificial neural network trained for the corresponding kinematic chain to determine the orientation of the torso (101) from the orientation of the arm modules (115 and 113) and/or the orientation of the head module (111) , the corresponding kinematic chain includes upper arms (103 and 105), torso (101) and/or head (107).

In order to determine the orientation of the torso (101), the hand modules (117 and 119) are optional in the system shown in Figure 1 .

Furthermore, in some cases, the head module (111) is not used to track the orientation of the user's torso (101).

Typically, the measurements of the sensor devices (111, 113, 115, 117 and 119) are calibrated to align with a common reference system, such as the coordinate system (100).

After calibration, the user's hands, arms (105, 103), head (107) and torso (101) are movable relative to each other and relative to the coordinate system (100). The measurements of the sensor devices (111, 113, 115, 117 and 119) provide the orientation of the user's hands (106 and 108), upper arms (105, 103) and head (107) relative to the coordinate system (100). The computing device (141) uses the predictive model (116) based on the current orientation of the upper arms (105, 103), the current orientation of the user's head (107), and/or the current orientation of the user's hands (106 and 108) and their orientation history. to calculate, estimate or predict the current orientation of the torso (101) and/or forearms (112 and 114).

Alternatively or in combination, the computing device (141) can also be used, for example, as disclosed in U.S. Patent Application Serial No. 15/787,555 filed on October 18, 2017 and entitled "Tracking Arm Movements to Generate Inputs for Computer Systems" The technique for calculating the orientation of the forearm from the orientation of the hands (106 and 108) and upper arms (105 and 103), the entire disclosure of which application is hereby incorporated by reference.

At least some embodiments disclosed herein allow, without the need for additional sensor modules to be attached to the torso (101) and forearms (112 and 114), based on the orientation of the upper arms (105 and 103), the orientation of the head (107) and and/or the orientation of the hands (106 and 108) to determine or estimate the orientation of the torso (101) and/or forearms (112 and 114).

Figure 2 illustrates a system for controlling computer operations, according to one embodiment. For example, the arm modules (115 and 113) may be attached to the upper arms (105 and 103), respectively, the head module (111) to the head (107) and/or the hand module (Fig. 117) in the manner shown in FIG. and 119) to implement the system of FIG. 2 .

In Figure 2, the head module (111) and arm module (113) have microelectromechanical systems (MEMS) inertial measurement units (IMUs) (121 and 131) that measure motion parameters and determine the head (107) and upper arm (103) orientation.

Similarly, the hand modules (117 and 119) may also have an IMU. In some applications, the hand modules (117 and 119) measure the orientation of the hands (106 and 108) and do not separately track the movement of the fingers. In other applications, the hand modules (117 and 119) have separate IMUs for measurements of the orientation of the palm of the hand (106 and 108) and the orientation of at least some of the phalanges of at least some of the fingers on the hand (106 and 108) . An example of a hand module can be found in U.S. Patent Serial No. 15/792,255, filed October 24, 2017, and entitled "Tracking FingerMovements to Generate Inputs for Computer Systems," the entire disclosure of which is incorporated herein by reference. into this article.

Each of the IMUs (131 and 121) has a set of sensor assemblies capable of determining the movement, position and/or orientation of the respective IMU along multiple axes. Examples of this component are: MEMS accelerometers, which measure the projection of acceleration (the difference between an object's true acceleration and the acceleration due to gravity); MEMS gyroscopes, which measure angular velocity; and magnetometers, which measure The magnitude and direction of the magnetic field. In some embodiments, the IMU uses a combination of sensors in three and two axes (eg, no magnetometer).

The computing device (141) has a predictive model (116) and a motion processor (145). Measurements from the IMUs (eg, 131, 121) of the head module (111), arm modules (eg, 113 and 115) and/or hand modules (eg, 117 and 119) are used in the prediction module (116) , to generate predicted measurements of at least some of the parts without attached sensor modules, such as the torso (101) and forearms (112 and 114). The predicted measurements and/or measurements of the IMU (eg, 131, 121) are used in the motion processor (145).

The motion processor (145) has a skeletal model (143) of the user (eg, as shown in Figure 3). The motion processor (145) controls the movement of the parts of the skeletal model (143) according to the movement/orientation of the corresponding parts of the user. For example, hand (106) as measured by the IMUs of the hand modules (117 and 119), arm modules (113 and 115), head module (111) sensor modules and/or predicted by the prediction module (116) based on IMU measurements. and 108), forearms (112 and 114), upper arms (103 and 105), torso (101), head (107) are used to set the orientation of corresponding parts of the skeleton model (143).

Since the torso (101) has no separately attached sensor modules, the movement/orientation of the torso (101) is predicted using a predictive model (116) utilizing sensor measurements from sensor modules on the kinematic chain including the torso (101). For example, the predictive model (116) can be trained with motion patterns comprising a kinematic chain of the head (107), torso (101) and upper arms (103 and 105), and can be used for head (107), torso (101) The orientation of the torso (101) is predicted using the movement history of the upper arms (103 and 105) and the current orientation of the head (107) and upper arms (103 and 105).

Similarly, since the forearm (112 or 114) does not have a separately attached sensor module, the predictive model (116) is used to predict the forearm (112 or 114) movement/orientation. For example, the predictive model (116) can be trained with motion patterns including a kinematic chain of the hand (106), forearm (114), and upper arm (105), and can be used to (105) motion history and the current orientation of the hand (106) and upper arm (105) to predict the orientation of the forearm (114).

The skeletal model (143) is controlled by the motion processor (145) to generate input for an application (147) running in the computing device (141). For example, the skeletal model (143) may be used to model the arms (112, 114, 105, and 103), hands (106, and 108) of a user of the computing device (141) in video games, virtual reality, mixed reality, or augmented reality, etc. ), the movement of the avatar/model of the head (107) and torso (101) is controlled.

Preferably, the arm module (113) has a microcontroller (139) for processing sensor signals from the IMU (131) of the arm module (113) and for sending the motion/orientation parameters of the arm module (113) to the computing A communication module (133) of a device (141). Similarly, the head module (111) has a microcontroller (129) for processing sensor signals from the IMU (121) of the head module (111) and for sending motion/orientation parameters of the head module (111) to the computing A communication module (123) of a device (141).

Optionally, the arm module (113) and head module (111) have LED indicators (137 and 127), respectively, to indicate the operating status of the modules (113 and 111).

Optionally, the arm modules (113) each have a tactile actuator (138) to provide tactile feedback to the user.

Optionally, the head module (111) has a display device (127) and/or buttons and other input devices (125), such as touch sensors, microphones, cameras and the like.

In some embodiments, the head module (111) is replaced by a module similar to the arm module (113) and attached to the head (107) via straps or fixed to the head mounted display device.

In some applications, the hand module (119) may be implemented with a module similar to the arm module (113) and attached to the hand via the hand or via a strap. Optionally, the hand module (119) has buttons and other input devices, such as touch sensors, joysticks, and the like.

For example, U.S. Patent Application Serial No. 15/792,255, filed October 24, 2017, and entitled "Tracking Finger Movements to Generate Inputs for Computer Systems," filed October 18, 2017, and entitled "Tracking Arm Movements to Generate Inputs for Computer Systems," and/or U.S. Patent Application Serial No. 15/787,555, filed April 20, 2017, and entitled "Devices for Controlling Computers based on Motions and Positions of Hands" The handheld modules disclosed in 15/492,915, the entire disclosure of which is hereby incorporated by reference, may be used to implement the hand modules (117 and 119).

When the hand module (e.g., 117 or 119) tracks the orientation of the palm and the selected set of phalanges, the motion pattern of the kinematic chain of the hand captured in the prediction mode (116) can be used in the prediction model (116) to predict Orientation of other phalanges not wearing sensor device.

Figure 2 shows a hand module (119) and an arm module (113) as an example. In general, applications for tracking the orientation of the torso (101) typically use two arm modules (113 and 115) as shown in Figure 1 . The head module (111) may optionally be used to further improve tracking of the orientation of the torso (101). The hand modules (117 and 119) may also be used to provide additional input and/or to predict/calculate the orientation of the user's forearms (112 and 114).

Typically, an IMU (eg, 131 or 121 ) in a module (eg, 113 or 111 ) generates acceleration data from an accelerometer, angular velocity data from a gyrometer/gyroscope, and/or orientation data from a magnetometer. The microcontrollers (139 and 129) perform pre-processing tasks such as filtering sensor data (e.g., to block sensors that are not used in a particular application), applying calibration data (e.g., to correct mean cumulative error), transforming motion/position/orientation data in three axes into quaternions, and packaging the preprocessed results into data packets (e.g., using data compression techniques) for use with reduced bandwidth requirements and / or communication time to send it to the host computing device (141).

Each of the microcontrollers (129, 139) may include memory that stores instructions that control the operation of the respective microcontroller (129 or 139) to perform the primary processing and processing of sensor data from the IMU (121, 131) Controlling the operation of communication modules (123, 133) and/or other components such as LED indicators (137), tactile actuators (138), buttons and other input devices (125), display devices (127), etc.

The computing device (141) may include one or more microprocessors and memory storing instructions for implementing the motion processor (145). The motion processor (145) may also be implemented via hardware such as an Application Specific Integrated Circuit (ASIC) or Field Programmable Gate Array (FPGA).

In some cases, one of the modules (111, 113, 115, 117, and/or 119) is configured as a primary input device; and the other module is configured as a secondary input device that is connected to the computer via the primary input device device (141). The secondary input device may use the microprocessor of its connected primary input device to perform some of the pre-processing tasks. A module that communicates directly with the computing device (141) is considered a primary input device even when the module does not have a secondary input device connected to the computing device via the primary input device.

In some cases, the computing device (141) specifies the type of input data requested and the conditions and/or frequency of the input data; and the modules (111, 113, 115, 117, and/or 119) under the conditions and/or Or report the requested input data according to the frequency specified by the computing means (141). Different reporting frequencies may be specified for different types of input data (eg, accelerometer measurements, gyroscope/gyrometer measurements, magnetometer measurements, position, orientation, velocity).

In general, the computing device (141) may be a data processing system such as a mobile phone, desktop computer, laptop computer, head-mounted virtual reality display, personal media player, tablet computer, or the like.

Figure 3 illustrates a skeletal model that can be controlled by tracking user movements, according to one embodiment. For example, the skeletal model of Figure 3 may be used in the motion processor (145) of Figure 2 .

The skeletal model shown in Figure 3 includes a torso (232) and upper left and right arms (203 and 205), which are movable relative to the torso (232) via shoulder joints (234 and 241). The skeletal model may also include forearms (215 and 233), hands (206 and 208), neck, head (207), legs and feet. In some cases, hand (206) includes a palm connected via joints (eg, 244) to phalanges (eg, 245) of the fingers and metacarpals of the thumb.

The position/orientation of the rigid parts of the skeletal model shown in FIG. 3 is controlled by the measured orientation of the corresponding parts of the user shown in FIG. 1 . For example, the orientation of the skeletal model's head (207) is configured according to the orientation of the user's head (107) measured using the head module (111); and the orientation of the skeletal model's hand (206) is configured according to the orientation of the user's hand (106) measured using the hand module (117); and so on.

The predictive model (116) may have multiple artificial neural networks trained for different motion patterns of different kinematic chains.

For example, a clavicle kinematic chain may include upper arms (203 and 205), torso (232) represented by clavicles (231), and optionally head (207) connected by shoulder joints (241 and 234) and neck. The clavicle kinematic chain may be used to predict the orientation of the torso (232) based on the motion history of the clavicle kinematic chain and the current orientation of the upper arms (203 and 205) and head (207).

For example, a forearm kinematic chain may include upper arm (205), forearm (215) and hand (206) connected by elbow joint (242) and wrist joint (243). The forearm kinematic chain can be used to predict the orientation of the forearm (215) based on the motion history of the forearm kinematic chain and the current orientation of the upper arm (205) and hand (206).

For example, the hand kinematic chain may include the palm of the hand (206), the phalanges (245) of the fingers on the hand (206), and the metacarpal bones of the thumb on the hand (206) connected by joints in the hand (206). The hand kinematic chain can be used to predict the orientation of the phalanges and metacarpals based on the motion history of the hand kinematic chain and the current orientation of the palm and a subset of the phalanges and metacarpals tracked using the IMU in the hand module (eg, 117 or 119).

For example, a trunk kinematic chain may include a clavicle kinematic chain, and also include forearms and/or hands and legs. For example, a leg kinematic chain may include feet, calves, and thighs.

Supervised machine learning techniques can be used to train the artificial neural network of the predictive model (116) to predict the orientation of parts in the kinetic chain based on the orientation of other parts in the kinetic chain, so that parts with predicted orientations do not have to wear separate sensors device to track its orientation.

In addition, supervised machine learning techniques can be used to train the artificial neural network of the predictive model (116) to predict motion that can be measured using one tracking technique based on the orientation of parts in the kinematic chain measured using another tracking technique The orientation of the sites in the chain.

For example, a tracking system as shown in FIG. 2 uses an IMU (eg, 111, 113, ...,) to measure the orientation of a module (111, 113, ..., 119). Inertial based sensors provide a good user experience, have fewer constraints on sensor usage, and can be implemented in a computationally efficient manner. However, in some cases, inertial-based sensors may not be as accurate as some tracking methods and may have drift errors and/or errors that accumulate through time integration.

For example, an optical tracking system may use one or more cameras to track the position and/or orientation of optical markers in the camera's field of view. When the optical marker is within the camera's field of view, images captured by the camera may be used to calculate the position and/or orientation of the optical marker and thus the orientation of the site marked with the optical marker. However, optical tracking systems may not be as user-friendly as inertial-based tracking systems and may be more expensive to deploy. Furthermore, the position and/or orientation of the optical marker cannot be determined by the optical tracking system when the optical marker is outside the camera's field of view.

The artificial neural network of the predictive model (116) can be trained to predict measurements produced by the optical tracking system based on measurements produced by the inertial-based tracking system. Thus, drift errors and/or cumulative errors in the inertial-based measurements may be reduced and/or suppressed, which reduces the need for recalibration of the inertial-based tracking system.

Figure 4 shows a method for training a recurrent neural network (RNN) (307). For example, the method of FIG. 4 may be used to generate the predictive model of FIG. 1 and/or FIG. 2 (116).

In Figure 4, two tracking systems (301 and 302) are used to track/measure human motion (303). For example, the inertial-based system of FIG. 2 can be used as tracking system A (301); and an optical tracking system can be used as tracking system B (302).

For example, a person may wear a sensor device (111, 113, 115, 117, and 119) that includes an IMU (eg, 121, 131, . . . ) for an inertial-based system. To train the recurrent neural network (RNN) (307), optical markers can be attached to the person for the optical tracking system (eg, 302).

Optionally, optical markers may be integrated on sensor devices (111, 113, 115, 117, and 119) to track motion for training of a predictive model (116) comprising a recurrent neural network (RNN) (307) ( 303).

Optionally, additional optical markers are attached to certain parts of the user not wearing the IMU-containing sensor device. For example, as shown in FIG. 1 , the user's forearms (112 and 114) and torso (101) do not have an attached IMU for measuring their orientation via an inertial-based system (e.g., 301); however, the user's The forearms (112 and 114) and torso (101) may have optical markers to measure their orientation using an optical tracking system (eg, 302).

In general, the optical tracking system (eg, 302 ) can be replaced with another tracking system that makes orientation measurements based on infrared signals, ultrasonic signals, radio frequency identification tag (RFID) signals, or the like. Furthermore, a combination of tracking systems can be used as tracking system B (302) to obtain the most accurate measurement of body motion (303) B (306). Tracking system B (302) is used to measure the orientation of at least some of the parts of the person not measured by tracking system A (301) and optionally the parts of the person measured by tracking system A (301).

After a person wears the sensor modules and optical markers for tracking system A (301) and tracking system B (302), the person can perform multiple action sequences involving various motion patterns of the kinematic chain.

The sequence may start with a common calibration pose such as the pose shown in FIG. 1 . may be found in U.S. Patent Application Serial No. 15/847,669, filed December 19, 2017, and entitled "Calibration of Inertial Measurement Units Attached to Arms of a User and to a Head Mounted Device," filed November 20, 2017 An example of a calibration posture is found in U.S. Patent Application Serial No. 15/817,646, entitled "Calibration of Inertial Measurement Units Attached to Arms of a User to Generate Inputs for Computer Systems," the entire disclosure of which is hereby incorporated by reference This article.

Tracking systems A and B (301 and 302) can be used to simultaneously measure/track parts of a person such as head (107), arms (103, 105, 112, 114) and hands (106 and 108) and torso (101) position and/or orientation to generate measurements A and B respectively (305 and 306).

Supervised machine learning techniques may be used to train a recurrent neural network (RNN) (307) based on measurements A (305) generated by a tracking system A (301) (e.g., an inertial-based system as shown in FIG. 2 ). to predict measurements B (306) generated by tracking system B (302) (eg, an optical tracking system, another tracking system, or a combination of tracking systems). Supervised machine learning techniques adjust the parameters in the recurrent neural network (RNN) (307) so that the actual measurement B (306) is different from that made using measurement A (305) as input to the recurrent neural network (RNN) (307). The difference between the predictions is minimized. A recurrent neural network (RNN) (307) with tuned parameters provides an RNN model (309), which can be used as the predictive model (116) in FIG. 1 and/or FIG. 2 .

A recurrent neural network (RNN) (307) may include a network of long short-term memory (LSTM) cells to selectively remember the history of states based on which predictions are made.

A sequence of actions performed by a person performing a human movement (303) may be sampled at predetermined time intervals to obtain measurements (305 and 306) for training a recurrent neural network (RNN) (307).

In some cases, some action sequences are repeated multiple times and/or at different speeds; and scaling and/or double-exponential smoothing of the time measurements can be applied to the input parameters to align the sequence of data sets and /or normalize the time scale.

After the RNN model (309) has been trained with the ability to predict the measurements B (306) generated using tracking system B (302), it is no longer necessary to use tracking system B (302). For example, the RNN model may be developed (309) at a manufacturing facility and/or at a developer's facility. A user of a sensor module (eg, 111 , 113 , . . . , 119 ) does not need to wear an optical marker or other device used in tracking system B ( 302 ), as shown in FIG. 5 .

Figure 5 illustrates a method for predicting movement measurements of one tracking system based on movement measurements of another tracking system using an RNN, according to one embodiment.

In FIG. 5, tracking system A (301) (e.g., the inertial-based system shown in FIG. 2) is used to track a user without tracking system B (302) (e.g., an optical tracking system). Movement (304). Measurements A (315) of user motion (304) measured using tracking system A (301) (e.g., the inertial-based system shown in FIG. 2 ) are used in the RNN model (309) to generate pairs of Predictions (316) of measurements B generated by system B (302) are tracked. The prediction (316) of measurement B may be provided to the motion processor (145) to control the skeletal model (143) (e.g., as shown in Figure 3), as if the tracking system B (302) was used to obtain the measurement Same result.

For example, an RNN model (309) with LSTM units can be trained to use orientation measurements of parts of a kinematic chain generated by a sensor module with an IMU to predict orientation measurements generated by an optical tracking system for a kinematic chain.

For example, sensor modules (113 and 119) are attached to the upper arm (103) and hand (119) in a forearm kinematic chain using an IMU (e.g., 131) from the sensor modules (113 and 119) Generated measurement results. The RNN model (309) predicts orientation measurements to be generated by the optical tracking system not only for the forearm (112) but also for the upper arm (103) and hand (108) in the forearm kinematic chain from the IMU measurements of the user motion (304) sequence. The predicted orientation measurements are used in the motion processor (145) to configure the corresponding forearm kinematic chain of the skeleton (143) including the forearm (233), upper arm (203) and hand (208).

In some cases, it is not necessary to use a tracking system implemented with one technology (eg, an optical tracking system) to calibrate measurements of a tracking system implemented with another technology (eg, an inertial-based tracking system). The predictive model ( 116 ) can be trained to predict the orientation of regions lacking sensor modules without using multiple training techniques, as shown in FIG. 6 .

Figure 6 shows a method for training a recurrent neural network (RNN) (337) to predict measurements of missing sensor modules.

In FIG. 6, the body motion (303) of a person wearing both tracking device A (331) and tracking device B (332) is measured. For example, tracking device A ( 331 ) corresponds to head module ( 111 ), arm modules ( 113 and 115 ), and hand modules ( 117 and 119 ), as shown in FIG. 1 and/or FIG. 2 . In Figure 1, the forearms (112 and 114) and torso (101) lack sensor modules.

To train a recurrent neural network (RNN) (337) to predict missing sensor module measurements, the method of FIG. 6 uses tracking device B (332), which is attached to the forearms (112 and 114) and additional IMU module. Additional IMU modules can be attached to the forearms (112 and 114) and torso (101) similar to the arm module (113) via arm straps, straps, and/or otherwise to attach the module to the performing body Corresponding position on the moving person.

Supervised machine learning techniques can be applied to a recurrent neural network (RNN) (337) that uses measurements A (335) generated by tracking device A (331 ) to predict measurements B ( 336). Supervised machine learning adjusts the parameters of the recurrent neural network (RNN) (337) so that measurement B (336) matches the prediction made by applying measurement A (335) as input to the recurrent neural network (RNN) (337) minimize the difference between them. A Recurrent Neural Network (RNN) (337) and its trained parameters provide an RNN model (339) that can be used to make predictions about the measurements of tracking device B (332). Therefore, when using the RNN model (339), the tracker B (332) can be omitted, as shown in FIG. 7 .

FIG. 7 illustrates a method for predicting movement measurements of missing sensor modules based on movement measurements of attached sensor modules using an RNN trained with the method of FIG. 6 , according to one embodiment.

In FIG. 7, tracking device A (331) (e.g., 111, 113, 115, 117, and 119 in FIGS. 1 and/or 2) is used to track the user's User Movement (304). Measurements A (345) of user motion (304) measured using tracking devices A (331) (e.g., 111, 113, 115, 117, and 119) are used as input to the RNN model (339) to generate pairs of Prediction (346) of measurements B (332) generated by device B (332) as if tracking device B (332) was used. Measurement A (345) and prediction for measurement B (346) may be provided to motion processor (145) to control skeletal model (143) (e.g., as shown in FIG. 3 ) as if tracking device A were used (331) and tracking device B (332) are both the same.

For example, an RNN model (339) with an LSTM unit can be trained to use orientation measurements of the remaining parts of the kinematic chain generated by the sensor module with the IMU to predict one or more parts of the kinematic chain generated by the sensor module with the IMU. Orientation measurements are generated such that the sensor module with the IMU can be omitted for one or more parts of the kinematic chain.

For example, sensor modules (113 and 119) are attached to the upper arm (103) and hand (119) in a forearm kinematic chain using an IMU (e.g., 131) from the sensor modules (113 and 119) Generated measurement results. The RNN model (339) predicts from the IMU measurements for the sequence of user motions (304) the orientation measurements that would be generated by a sensor module with an IMU (e.g., similar to the arm module (113)), as if such a sensor module were attached To the forearm (112) in the forearm kinematic chain. The predicted orientation measurements (346) for the forearm (112) and the measurements for the upper arm (103) and hand (119) are used together in the motion processor (145) to map the orientation of the forearm (233), upper arm (203) ) and the corresponding forearm kinematic chain of the skeleton (143) of the hand (208).

Since the prediction (346) can be obtained from the use of the sensor modules (113 and 119) without actually using the sensor module to track the orientation of the forearm (112), the user performing the user motion (304) does not have to be on the forearm (112) Wear an additional sensor module on the top. Thus, the user experience is improved; and the cost of the tracking system for the user is reduced.

Figure 8 shows a method for training an artificial neural network to predict orientation measurements.

The method of FIG. 8 includes attaching ( 401 ) a tracking device to at least one kinematic chain of a person including a first tracking device detached from a second tracking device on the one or more kinematic chains.

For example, the first tracking device is an arm module (115) on the upper arm (105) and a hand module (117) on the hand (106); and one or more second tracking devices include the forearm (114) on the forearm kinematic chain ), the forearm kinematic chain includes an upper arm (105), a forearm (114) and a hand (106) connected via an elbow joint and a wrist joint. A tracking device on the forearm (114) separates the arm module (115) from the hand module (117) on the forearm kinematic chain. The arm module (115) and hand module (117) include IMUs to track their orientation in an inertial-based tracking system, and may have optical markers to measure their orientation separately using the optical tracking system. The tracking device on the forearm (114) may be an optical marker used to measure its orientation in an optical tracking system, and may optionally include an IMU to track its orientation in an inertial-based tracking system. The tracking device on the forearm (114) may be implemented in the same manner as the arm module (115) as it enables tracking in both inertial-based and optical tracking systems.

The method of Figure 8 also includes performing (403) a sequence of actions (303) involving at least one kinematic chain. The sequence of actions (303) starts with a common calibration pose (eg, as shown in Figure 1 or another pose). Actions may be designed to simulate typical actions in applications (147) such as virtual reality games, augmented reality applications, and the like.

The method of FIG. 8 also includes recording (405) the orientations of the first tracking device and the second tracking device in a sequence wherein the first system (301) (e.g., an inertial-based tracking system) and the second system (302) (eg, an optical tracking system) both to track the orientation of a first tracking device, and a second system (302) is used to track the orientation of one or more second tracking devices. Optionally, the orientation of one or more second tracking devices can also be tracked using the first system (301) (e.g., an inertial-based tracking system), and used where the second system (302) is not capable of measuring a or the orientation of multiple second tracking devices. For example, the orientation determined from the IMU attached to the forearm (114) can be used to generate Orientation measurements of the forearm (114). For example, orientation measurements of the forearm (114) obtained from IMU measurements of the forearm (114) may be obtained via measurements from the optical tracking system when optical markers are visible before and/or after ambiguous positions in the motion sequence. is calibrated so that the orientation of the forearm at this ambiguous location is calculated with substantially the same improved accuracy as the measurements from the optical tracking system.

The method of FIG. 8 also includes training (407) the artificial neural network (e.g., 307) to predict the orientation of the first tracking device based on the orientation of the first tracking device measured by the first system (301) (e.g., an inertial-based tracking system). The orientations of the first tracking device and the second tracking device are measured by a second system (302) (eg, an optical tracking system).

In some cases, separate artificial neural networks are used to train different kinematic chains individually. The trained network can be used individually for each kinematic chain to improve computational efficiency. Alternatively, the artificial neural network can be trained on the motion model of the whole skeleton (143) for general application.

A trained artificial neural network can be retrained using reinforcement learning techniques to improve its predictive accuracy in some sports scenarios.

FIG. 9 illustrates a method for tracking user movement using an artificial neural network trained using the method of FIG. 8 .

The method of FIG. 9 includes attaching (411) a tracking device to at least one kinematic chain of the user that includes a first tracking device but does not include a tracking device at a location separate from the first tracking device in one or more kinematic chains second tracking device; perform (413) a sequence of actions (304) involving at least one kinematic chain starting from a calibration pose; use the first system (301) instead of the second system (302) to generate (415) the first in the sequence Orientation measurements of a tracking device; applying (417) the orientation measurements to a previously trained (e.g., using the method of FIG. 9 ) artificial neural network to measuring to predict the orientation of the at least one kinematic chain measured by the second system (302); and generating (419) the predicted orientation measurement of the at least one kinematic chain by applying the orientation measurement to the artificial neural network.

For example, a computing system includes a plurality of sensor modules (eg, 111, 113, 115, 117, and/or 119) and a computing device (141). Each of the sensor modules has an inertial measurement unit (e.g., 121 or 113) and is attached to a part of the user (e.g., 107, 113, 115, 106, or 108) to generate motion identifying a sequence of orientations for the corresponding part of the user. data. The inertial measurement unit includes a microelectromechanical system (MEMS) gyroscope, and may also include a magnetometer and a MEMS accelerometer. The computing device provides as input the sequence of orientations measured by the sensor modules to the artificial neural network (e.g., 116), obtains as an output from the artificial neural network (e.g., 116) at least one orientation measurement of the user's part, using data obtained from the artificial neural network (e.g., 116). The orientation measurements obtained by the neural network configure or set the orientation of rigid parts in the kinematic chain of the skeletal model (143) representing the user, and control the application (147) according to the state of the skeletal model (143).

For example, the artificial neural network may be a recurrent neural network previously trained to make predictions that match orientation measurements generated using the optical tracking system. The recurrent neural network contains long short-term memory (LSTM) units to memorize a set of state histories derived from a sequence of input orientations in order to predict the current orientation of the kinematic chain.

Optical tracking is not used to track this part of the user (and other parts of the user) since the artificial neural network can predict orientation measurements generated using optical tracking technology.

For example, one of multiple sensor modules is used to track the orientation of the user's part; and an artificial neural network is used to improve the IMU-based measurements to eliminate drift and/or cumulative errors.

For example, the orientation of the user's part is not even tracked using a sensor module containing an inertial measurement unit, since its orientation can be predicted using orientation measurements of other parts of the user in the kinematic chain applied as input to an artificial neural network.

For example, a plurality of sensor modules (e.g., 111, 113, 115, 117, and/or 119) track rigid locations (e.g., 207, 203, 205, 206, and/or 208) in a kinematic chain with the skeletal model (143) The user's parts corresponding to a subset of (for example, 107, 103, 105, 106, and/or 108); and the rigid parts (for example, , 215, 223, or 232) separate a subset of rigid sites (eg, 207, 203, 205, 206, and/or 208) in the kinematic chain.

For example, an artificial neural network can be trained to predict orientation measurements generated using a separate tracking system; , 117, and/or 119) of the user's location (eg, 107, 103, 105, 106, and/or 108) to generate predicted orientation measurements as output.

For example, to train an artificial neural network, a set of sensor modules is attached to a person performing a plurality of motion sequences to generate a first orientation measurement and a second orientation measurement from the set of sensor modules. Supervised machine learning techniques are used to train the artificial neural network to predict the second orientation measurement based on the first orientation measurement.

For example, a first orientation measurement is measured using a first technique; a second orientation measurement is measured using a second technique; and the artificial neural network is trained based on the measurements generated using the first technique (e.g., IMU-based tracking) to predict measurements made using a second technique (eg, optical tracking).

For example, when an artificial neural network is found to have inaccurate predictions in some scenarios, reinforcement learning techniques may be used to further train the artificial neural network based on additional measurements made in conjunction with those scenarios.

In some cases, the second orientation measurement identifies the orientation of the plurality of sensor modules; and the first orientation measurement identifies the orientation of a subset of the plurality of sensor modules such that a tracking technique (e.g., IMU-based tracking) is used to derive Measurements of a subset of sensor modules in can be used with an artificial neural network to predict measurements of the entire collection of sensor modules using another tracking technique (eg, optical tracking).

In other cases, the first orientation measurement identifies the orientation of a first subset of the plurality of sensor modules; and the second orientation measurement identifies the orientation of a second subset of the plurality of sensor modules such that from a subset of sensor devices Measurements of can be used to predict measurements made by another subset of sensor devices.

The skeleton model (143) may include multiple ANN models. Each of the ANN models is trained to use measurements obtained with tracking system A (301) (e.g., an IMU-based system such as the system shown in FIG. 2) to predict what would have happened using tracking system B (302) ( For example, an optical tracking system) measures motion measurements of parts of a kinematic chain. However, when two ANN models are used to predict motion measurements of two kinematic chains with overlapping parts, the ANN models can generate different predictions for the same overlapping parts.

For example, a forearm ANN model can be used to predict motion/orientation measurements of a forearm chain comprising the hand (106), forearm (114) and upper arm (105). The clavicle ANN model can be used to predict motion/orientation measurements of the clavicle chain comprising the upper left arm (105), torso (101) and upper right arm (103) (and optionally the head (107)).

Using the forearm ANN model, the upper arm (105) is predicted to have a generally different orientation than the orientation of the upper arm (105) predicted using the clavicle ANN model.

To use both the forearm ANN model and the clavicle ANN model, an additional ANN can be used to account for the differences generated by the forearm ANN model and the clavicle ANN model, and predict the upper arm in a manner similar to that discussed below in connection with FIG. 10 (105 )orientation.

Figure 10 illustrates the use of a Bidirectional Long Short-Term Memory (BLSTM) network to combine results from different artificial neural networks, according to one embodiment.

In Fig. 10, a set of devices (A, ..., C, ..., E) is used to generate its orientation measurements at time instances 1, 2, ..., t, where time Instances are numbered sequentially for identification. Adjacent time instances may have a fixed, predetermined time interval, or a variable time interval depending on the speed of the motion.

In Figure 10, two RNN models (511 and 513) are used for motion measurements such as parts of the user and/or corresponding rigid parts of the skeletal model (143) representing the user in a virtual or augmented reality application orientation) predictions. One RNN model (511) is trained for prediction of motion measurements of kinematic chain X (e.g., forearm kinematic chain); and another RNN model (513) is trained for kinematic chain Y (e.g., clavicle kinematic chain) Prediction of motion measurements. For example, use the hand module (117) and arm module (115) to track the forearm kinematic chain to generate input measurements of its orientation for the corresponding RNN model (e.g., 511); and use the arm modules (113 and 115) and optionally The head module (111) tracks the clavicle kinematic chain to generate input measurements of its orientation for another corresponding RNN model (eg, 513). Tracking of the forearm kinematic chain and tracking of the clavicle kinematic chain can be performed using two sets of sensor devices (e.g., 115 and 117; and 113, 115, and 111) sharing a common sensor device, such as the arm module (115) . The forearm kinematic chain and clavicle kinematic chain share a common site (eg, upper arm (105)), whose orientation is measured using a common sensor device C, such as arm module (115).

According to the sequence of orientation measurements (501, . . . , 505), the RNN model (511) generates a portion of the predicted measurements (515) comprising the predicted orientation of the sensor device C (e.g., the arm module (115)) in common at time instances t−1 and t+1 (517). Device C (eg, arm module (115)) is also used to generate input (505) for kinematic chain Y (eg, clavicle kinematic chain).

In one embodiment, the devices (A, ..., C and E) provide orientation measurements (501, ..., 505, . . . ., 509). When the measurement at time instance t is provided to the RNN model (511 and 513), the RNN model (511 and 513) generates The predicted measurements of device C are shared (517 and 519). When input measurements at time instance t after time instance t-1 become available, the RNN model (511 and 513) may optionally update the prediction of the measurement at time instance t-1.

For example, when a sequence of input measurements is provided as input to the RNN model (511 and 513) for time instances 1, 2, ..., t-2, the RNN model (511 and 513) can generate The predicted measurements of device C are shared at time instance t-3 before t-2 and at time instance t-1 after the last time instance t-2. After the input measurements for time instances t-1 and t are additionally added to the RNN model (511 and 513), the RNN model (511 and 513) can optionally use the input measurements at time instances t-1 and t The prediction of the measurement at time instance t-1 is updated as a result.

Alternatively, predicted measurements of past time instances are stored and used as input to the BLSTM network (521). For example, a predicted orientation measurement of device C at time instance t-1 is generated based on a sequence of input measurements at time instances 1, 2, ..., t-2, which at time instances 1, 2, ..., the sequence of input measurements at t is stored and used in the BLSTM network (521) when allowing the RNN models (511 and 513) to generate predicted orientation measurements for device C at time instance t+1.

From the sequence of orientation measurements (505, ..., 509) of the kinematic chain Y (e.g., the clavicle kinematic chain) at time instances 1, 2, ..., t from the devices (e.g., 115, 111, and 113), The RNN model (513) generates portions of predicted measurements (519), including predicted orientations (517) of device C (eg, arm module (115)) at time instances t-1 and t+1.

Since device C (e.g., arm module (115)) is in both kinematic chain X (e.g., forearm kinematic chain) and kinematic chain Y (e.g., clavicle kinematic chain), at time instances t−1 and t+1 Different predicted measurements are generated for device C (515 and 519). The different predictions at time t-1 and t+1 are given as input to a Bidirectional Long Short-Term Memory (BLSTM) network (521) to generate the orientation.

A Bidirectional Long Short Term Memory (BLSTM) network (521) may be trained using supervised machine learning techniques.

For example, tracking system A (301) (e.g., an IMU-based tracking system) may be used to generate input measurements A (305), including input measurements for kinematic chain X (e.g., forearm kinematic chain) and for kinematic chain X (e.g., forearm kinematic chain). Input measurements for Y (eg, clavicle kinematic chain). Tracking system B (302) (e.g., an optical tracking system) may be used to generate desired measurements B (306), which include desired measurements for kinematic chain X (e.g., forearm kinematic chain) and for kinematic chain Y (eg, the clavicle kinematic chain) for the desired measurement. The RNN models (511 and 513) are trained separately to predict the desired measurement B (306) from the input measurement A (305). Furthermore, the output of the RNN model (511 and 513) is used as input to the BLSTM network (521), which is also trained as in the measurement result B (306) generated by the tracking system B (302). To predict the expected measurement results of device C.

In Figure 10, the BLSTM network (521) does not use the predicted orientation measure of device C at time instance t predicted from the RNN models (511 and 513) when generating the predicted orientation of device C at time instance t (523) result.

In other embodiments, the BLSTM network (521) may receive additional inputs such as an orientation measured by device C at time instance t, an orientation(s) measured by device C prior to time instance t, and/or orientations measured by other devices in the kinematic chain X and Y at time instance t and/or other time instances.

Figure 10 shows an example of combining two kinematic chains. The system can be extended to combine more than two kinematic chains. For example, the left forearm kinematic chain can be tracked using sensing devices (119, 113); the right forearm kinematic chain can be tracked using sensing devices (117 and 115); and the sensing devices (113, 115 and optionally 111) to track the clavicle kinematic chain. The left forearm kinematic chain and the clavicle kinematic chain share a common site (103) tracked using a common device (113); and the right forearm kinematic chain and the clavicle kinematic chain share a common site (105) tracked using a common device (115). RNN models for the left forearm and clavicle kinematic chains can generate different predicted orientation measurements for their shared device (113) and site (103); and RNN models for the right forearm and clavicle kinematic chains can generate different predicted orientation measurements for their The shared device (115) and site (105) generate different predicted orientation measurements. Two BLSTM networks (e.g., 521) can be used, one for combining different predictions for the shared device (113) in the left forearm kinematic chain and clavicle kinematic chain, and another for combining the right forearm kinematic chain and clavicle kinematic chain Different projections for shared devices (115) in the chain. The BLSTM network (eg, 521) generates the predicted orientations of the shared devices (113 and 115), respectively.

Alternatively, a single BLSTM network (e.g., 521) can be used to combine the different predictions for the shared device (113) in the left forearm and clavicle kinematic chains and the sharing in the right forearm and clavicle kinematic chains different predictions of the device (115) to generate the predicted orientations of the shared devices (113 and 115), respectively.

Figure 10 shows an example of combining a kinematic chain using a BLSTM network (521). In general, other artificial neural networks and/or recurrent neural networks may also be used.

FIG. 11 illustrates another technique for combining results from different artificial neural networks for kinematic chains with overlapping sites, according to one embodiment.

In Fig. 11, devices (A, ..., C, ..., E) generate input orientation measurements for RNN models (512 and 514) of different kinematic chains X and Y in a manner similar to that discussed in Fig. 10 result(501,...,505,...,509).

In FIG. 11 , the predicted orientation (535) includes the predicted orientation (537 and 539) for device C at time instance t, where device C is shared in kinematic chains X and Y. The mean (531) of the different predictions (537 and 539) from the RNN models (512 and 514) for the different kinematic chains (X and Y) is computed as the predicted orientation of device C at time instance t (533).

Preferably, when using two RNN models (512 and 514), the RNN models (512 and 514) are adjusted in order to reduce the discrepancy between different predictions (537 and 539) made for the same device C. Adjustments are made to the RNN model (512 and 514) so that the predictions (537 and 539) approximate the input orientation measurements generated by device C at time instance t.

For example, the forgetting rate of the RNN model (512 and 514) applied to the input measurements generated by device C (505) may be reduced so that the predicted orientation of device C approaches the orientation measured by device C.

For example, the forgetting rates of RNN models (512 and 514) applied to input measurements (e.g., 501 and 509) generated by devices other than device C may be increased so that device C's predicted orientation is closer to that generated by The orientation measured by device C.

For example, the weights of the RNN models (512 and 514) applied to the input measurements generated by device C (505) may be increased so that the predicted orientation of device C approaches the orientation measured by device C.

For example, the weights of the RNN models (512 and 514) applied to the input measurements (e.g., 501 and 509) generated by devices other than device C may be reduced so that the predicted orientation of device C is closer to that produced by device C C measured orientation.

In some cases, when the RNN models (512 and 514) are used together, the RNN models (512 and 514) are also trained to minimize the difference between the predicted measurements (537 and 539) of device C.

In some embodiments, the average (531 ) includes weights obtained using machine learning techniques to predict orientation measurements made using the optical tracking system. Optionally, the average (531) also receives the orientation measurements of device C at time instance t with weights obtained using supervised machine learning techniques.

The techniques of Figures 10 and 11 can be combined and used together. For example, the tuned RNN models (512 and 514) in FIG. 11 that produced reduced variance in predictions for shared device C (e.g., 537 and 539) could be used in FIG. 10 in place of the separately trained RNN models ( 511 and 513); and the BLSTM network (521) can be used to generate a predicted orientation (523) from the predicted orientation (517 and 519) that is compatible with the RNN model trained separately (511 and 513) generated predictions with reduced variance compared to those. Furthermore, the output of the average (531) can be provided as input to the BLSTM network (521).

In one embodiment, two RNN models (511 and 512) are generated for the kinematic chain X. When tracking kinematic chain X without tracking kinematic chain Y, the RNN model ( 511 ) trained without considering the difference between device C's predictions between kinematic chains X and Y is used. When tracking both kinematic chain X and kinematic chain Y, an RNN model trained and/or tuned to reduce the difference between device C's predictions between kinematic chains X and Y is used (512).

Alternatively, the RNN models (512 and 514) may be used individually after the RNN models (512 and 514) have been trained to reduce the variance between the different predictions (537 and 539) of the shared device C. For example, when tracking kinematic chain X without tracking kinematic chain Y, the RNN model is used (512); and when kinematic chain Y is not tracked, the RNN model is used (514).

Figure 12 illustrates a method for training multiple artificial neural networks for multiple kinematic chains with overlapping parts, according to one embodiment. For example, the method of FIG. 12 may be used to generate the RNN model (511 and 513) and the BLSTM network (521) of FIG. 10 and/or the RNN model (512 and 514) of FIG. 11 .

The method of Figure 12 includes training (551) a first RNN (eg, 511 or 512) to predict orientation measurements of a first kinematic chain (eg, forearm kinematic chain). Predictions are made using measurements A (305) from tracking system A (301) (eg, an IMU-based tracking system) to match measurements B from tracking system B (302) (eg, an optical tracking system).

The method of FIG. 12 also includes training (553) a second RNN (e.g., 511 or 512) in a manner similar to the training (551) of the first RNN (e.g., 511 or 512) to predict Forearm kinematic chain) shares orientation measurements of a second kinematic chain (eg, clavicle kinematic chain) of at least one site (eg, upper arm).

The method of FIG. 12 also includes training (555) a third RNN (e.g., 521) to predict a first kinematic chain (e.g., For example, orientation measurements of at least one site (eg, forearm) shared between a forearm kinematic chain) and a second kinematic chain (eg, clavicle kinematic chain).

Optionally, the method of FIG. 12 may also include training the first RNN (for example, 511 or 512) and the second RNN (for example, 511 or 512) to reduce the first RNN (for example, 511 or 512) and the second RNN Predicted differences between (eg, 511 or 512 ) for at least one site (eg, upper arm) shared between a first kinematic chain (eg, forearm kinematic chain) and a second kinematic chain (eg, clavicle kinematic chain).

When the first RNN (e.g., 511 or 512) and the second RNN (e.g., 511 or 512) are trained to reduce the The training of the third RNN (eg, 521 ) can be skipped when the predictions for shared parts in the chain) differ; and the average (531 ) can be used to combine the different predictions (eg, 537 and 539 ).

Figure 13 illustrates a method for predicting motion measurements for overlapping parts of multiple kinematic chains as modules using separate artificial neural networks, according to one embodiment. For example, the method of FIG. 13 may be used in the system shown in FIG. 10 .

The method of FIG. 13 includes: receiving (561) sensor measurements from a plurality of motion sensing devices (e.g., 111, 113, 115, 117, 119); 113, 115) sensor measurements are applied (563) as input to a first RNN (e.g., 511 or 512) to obtain 105)) first predicted measurements of a first kinematic chain; sensor measurements from a second subset of motion sensing devices (e.g., 117 and 115, and/or 113 and 119) are applied as input (565 ) in the second RNN (513 or 514) to obtain a set of parts with a second set (for example, hand (106), forearm (114) and upper arm (105); and/or hand (108), forearm (112) and upper arm (103)) second predicted measurements of the second kinematic chain; and combining at least a portion of the first predicted measurements (e.g., 517 or 537) with the second predicted measurements (e.g., 519 or 539) A portion is applied (567) to a third RNN (e.g., 521) to obtain a third predicted measurement (e.g., , 523).

Figure 14 illustrates a method for using a skeletal model with multiple artificial neural networks for multiple kinematic chains, according to one embodiment. For example, the method of FIG. 14 may be used in the systems shown in FIG. 10 or FIG. 11 .

The method of FIG. 14 includes receiving ( 581 ) sensor measurements from a plurality of motion sensing devices (eg, 111 , 113 , 115 , 117 , and/or 119 ).

If (583) the device tracks motion of a first kinematic chain (e.g., head (107), torso (101), upper arms (105 and 103)), the method of FIG. An RNN (511 or 512) obtains (585) the predicted motion measurements (eg, 517 and/or 537).

If (587) the device tracks the motion of a second kinematic chain (e.g., hand (106), forearm (114), and upper arm (105); or hand (108), forearm (112), and upper arm (103)), then Fig. The method of 14 also includes obtaining (589) predicted motion measurements (519 and/or 539) from the second RNN (513 or 514) using at least a portion of the sensor measurements.

If (591) the device tracks both the first kinematic chain and the second kinematic chain, the method of FIG. 14 also includes using predicted motion measurements (e.g., 517 and 519 ) obtain ( 593 ) from a third RNN (eg, 521 ) predicted motion measurements ( 523 ) of common parts of the first strand and the second strand.

In some cases, in response to determining that the device tracks both the first and second kinematic chains, the first RNN (511) and the second RNN are adjusted to reduce difference between the forecasts.

For example, a system may include: a plurality of sensor modules (eg, 111, 113, 115, 117, and/or 119); and a computing device (141) coupled to the plurality of sensor modules.

Each respective sensor module (e.g., 111, 113, 115, 117, or 119) has an inertial measurement unit (e.g., 121, 131, ...) and is attached to the user's part to generate an orientation identifying the user's part Sequence of motion data.

The plurality of sensor modules includes a first subset (eg, 111, 113, 115) and a second subset (eg, 115 and 117) that share common sensor modules between the first and second subsets (eg, , 115).

the computing means (141) provides as input to the first artificial neural network (511 or 512) the orientation measurements (eg, 501, ..., 505) generated by the first subset (eg, 111, 113, 115), and obtaining as an output from the first artificial neural network (511 or 512) at least one first orientation measurement ( For example, 517 or 537).

The computing means (141) also provides orientation measurements (e.g., 505, ..., 509) generated by the second subset (e.g., 115 and 117) as input to the second artificial neural network (513 or 154), and At least one second orientation measurement (eg, 519 or 539) of a common part of the user (eg, 105) is obtained as an output from the second artificial neural network (513 or 514).

The computing device (141) based on at least one first orientation measurement (517 or 537) of the user's common part (eg, 105) and at least one second orientation measurement (519 or 539) of the user's common part (eg, 105) ) combination (eg, 521 or 531) to generate a predicted orientation measurement (eg, 523) of the user's common part (eg, 105).

For example, the first artificial neural network and the second artificial neural network may be recurrent neural networks comprising long short-term memory (LSTM) units; and a third artificial neural network comprising at least bidirectional long short-term memory (BLSTM) units (e.g., 521) to perform the combination. Alternatively, this may be performed via averaging a first orientation measurement (537) of the user's common part (eg, 105) and a second orientation measurement (539) of the user's common part (eg, 105). combination.

For example, predicting predicted orientation measurements (523) of a common part for a first time instance t; at least one first orientation measurement (517) of a common part (e.g., 105) of a user comprising: generated by a first artificial neural network ( 511 or 512) predicted orientation measurements of a common site (eg, 105) at a second time instance t−1 before the first time instance t; and predicted by the first artificial neural network (511 or 512) at an orientation measurement of the common part (e.g., 105) at a third time instance subsequent to the one time instance; and at least one second orientation measurement (519) of the user's common part (e.g., 105) comprising: an orientation measurement of the common site (e.g., 105) predicted by the network (513 or 514) at a second time instance t−1 prior to the first time instance t; and a second artificial neural network (e.g., 513 or 514) Predicted orientation measurements of the common site (eg, 105 ) at a third time instance t+1 after the first time instance t.

In some cases, the measurements made for time instance t−1 are updated in the artificial neural network (e.g., 511, 514) using measurements (501, ..., 505, ..., 509) at subsequent time instances t-1. Prediction.

For example, the predicted orientation measurements (523 or 533) of a common site (e.g., 105) to be measured using an optical tracking system (which is used to train the artificial neural network used in the prediction) are predicted such that The optical tracking system can be removed from the system after the artificial neural network has been trained.

The computing device (141) may have a skeletal model (143). The tracked movement of the user controls the movement of the corresponding parts of the skeletal model (143). For example, a first subset (e.g., 111, 113, and 115) tracks the user's first kinematic chain (e.g., head (107), torso (101), and upper arms (103 and 105)) to control the motion of the skeletal model (143). movement of the corresponding kinematic chain (e.g., 207, 232, 203, and 205); and a second subset (e.g., 115 and 117) tracks the user's second chain (e.g., hand (106), forearm (114), and upper arm ( 105)) to control the movement of the corresponding kinematic chains (206, 215, 205) of the skeletal model (143).

In some cases, the orientation of the first part (e.g., torso (101)) in the user's kinematic chain is not tracked using any inertial measurement unit attached to the first part (e.g., torso (101)); Use any inertial measurement unit attached to the second part (e.g., forearm (114)) to track the orientation of the second part (e.g., forearm (114)) in the user's kinematic chain; the first artificial neural network (e.g., , 511 or 512) to predict the orientation of the first part (e.g., torso (101)) from the orientation measurements generated by the first subset (e.g., 111, 113, 115); and the second artificial neural network (e.g., 513 or 514) Predict the orientation of the second part (eg, the forearm (114)) from the orientation measurements generated by the second subset (eg, 115 and 117).

Each inertial measurement unit (eg, 121 , 131 , . . . ) may include a microelectromechanical system (MEMS) gyroscope and optionally a magnetometer and MEMS accelerometer.

The artificial neural network (eg, 511, 512, 513, 514, 521) may be trained to track the user's motion using a separate tracking system not present in the system used to control the skeletal model (141) in the computing device. For example, a separate tracking system may be an optical tracking system that uses one or more cameras to determine the orientation of optical markers. To generate the training data set, optical markers may be placed on sensor modules similar to those used to track the user's movements.

For example, multiple sensor modules can be attached to a person performing an exercise to generate a training data set. In a manner similar to tracking the motion of the user, the plurality of sensor modules includes a first subset of sensor modules (e.g., 111, 113 and 115) and a second subset of sensor modules (eg, 117 and 115) for tracking the orientation of the person's second kinematic chain (106, 114 and 105). The first kinematic chain and the second kinematic chain have a common part of the person (eg, 105).

In addition to using the sensor modules (e.g., 111, 113, 115, 117) to measure multiple orientation sequences of the sensor modules during the multiple motion sequences performed by the person, a separate tracking system is used during the multiple motion sequences performed by the person independently of the The orientation of the sensor modules (eg, 111 , 113 , 115 and/or 117 ) is measured using the measurements of the sensor modules.

A first artificial neural network (e.g., 511 or 512) is trained using supervised machine learning techniques to predict a first motion generated from a separate tracking system using orientation measurements from a first subset (e.g., 111, 113, 115) Orientation measurements of chains (eg, 107, 101, 103, and 105).

Similarly, a second artificial neural network (e.g., 513 or 514) is trained using supervised machine learning techniques to use orientation measurements from a second subset (e.g., 115 and 117) to predict a second ANN generated from a separate tracking system. Orientation measurements of the kinematic chain (eg, 106, 114, and 105).

A third artificial neural network may be trained using supervised machine learning techniques to generate orientation measurements (e.g., 537) based on first predicted orientation measurements (e.g., 537) of a common part of a person (e.g., 105) generated from the first artificial neural network (e.g., 511 or 512). ) and a second predicted orientation measurement (eg, 539) of a common part of a person (eg, 105) generated by a second artificial neural network (eg, 513 or 514) to predict a common part of a person (eg, 105) Orientation measurement results.

Optionally, the first artificial neural network and the second artificial neural network may also be trained (e.g. using unsupervised machine learning techniques) to reduce the human-specific Differences between predictions for common sites (eg, 105 ).

The present disclosure includes methods and apparatus for performing these methods, including data processing systems for performing these methods, and computer-readable media containing instructions that, when executed on a data processing system, cause the system to perform the methods.

For example, computing device (141), arm modules (113, 115), and/or head module (111) may be implemented using one or more data processing systems.

A typical data processing system may include interconnects (eg, buses and system core logic) that interconnect the microprocessor(s) and memory. A microprocessor is typically coupled to a cache memory.

The interconnect interconnects the microprocessor(s) and memory together and via the I/O controller(s) to the input/output (I/O O) device. I/O devices may include display devices and/or peripheral devices such as mice, keyboards, modems, network interfaces, printers, scanners, cameras, and other devices known in the art. In one embodiment, some of the I/O devices, such as printers, scanners, mice, and/or keyboards, are optional when the data processing system is a server system.

The interconnect may include one or more buses connected to each other through various bridges, controllers, and/or adapters. In one embodiment, the I/O controller includes a USB (Universal Serial Bus) adapter for controlling USB peripherals and/or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

The memory may include one or more of: ROM (Read Only Memory), volatile RAM (Random Access Memory), and non-volatile memory, such as hard drives, flash memory, and the like.

Volatile RAM is typically implemented as dynamic RAM (DRAM), which requires continuous power in order to refresh or maintain data in the memory. Non-volatile memory is typically a magnetic hard drive, magnetic optical drive, optical drive (eg, DVD RAM), or other type of memory system that maintains data even after the system is powered down. Non-volatile memory can also be random access memory.

Nonvolatile memory can be a local device that is directly coupled to the remaining components in the data processing system. Nonvolatile memory remote from the system, such as a network storage device coupled to the data processing system through a network interface such as a modem or Ethernet interface, also can be used.

In the present disclosure, some functions and operations are described as being performed by or caused by software codes to simplify the description. However, such expressions are also used to designate that the execution of code/instructions by a processor (eg, a microprocessor) results in the functionality.

Alternatively or in combination, the functions and operations described herein may be implemented using dedicated circuitry with or without software instructions, such as using an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). Embodiments may be implemented using hardwired circuitry without or in combination with software instructions. Thus, the technology is neither limited to any specific combination of hardware circuitry and software nor to any specific source for the instructions executed by the data processing system.

Although an embodiment may be implemented in fully functional computers and computer systems, the various embodiments can be distributed in various forms as computing products and can be distributed on a computer regardless of the particular type of machine or computer being used to actually effect the distribution. how computer readable media are used.

At least some of the disclosed aspects can be embodied, at least in part, in software. That is, information contained in memory (such as ROM, volatile RAM, non-volatile memory, cache memory, or remote storage) may be executed in a computer system or other data processing system in response to its processor (such as a microprocessor). sequence of instructions to execute the technique.

The routines executed to implement the embodiments may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions called a "computer program". A computer program typically includes one or more sets of instructions at various times in various memories and storage devices in a computer, and which, when read and executed by one or more processors in a computer, cause The computer performs operations necessary for performing elements involved in various aspects.

The machine-readable medium can be used to store software and data which, when executed by the data processing system, cause the performance of the various methods. Executable software and data may be stored in various locations including, for example, ROM, volatile RAM, non-volatile memory, and/or cache memory. Portions of this software and/or data may be stored in any of these storage devices. Additionally, data and instructions can be obtained from centralized servers or peer-to-peer networks. Different portions of data and instructions may be obtained from different centralized servers and/or peer-to-peer networks at different times and in different communication sessions or in the same communication session. Data and instructions can be obtained in bulk before the application is executed. Alternatively, portions of data and instructions may be obtained dynamically, just in time, as they are needed for execution. Thus, there is no requirement that the data and instructions reside entirely on the machine-readable medium at any particular instance of time.

Examples of computer readable media include, but are not limited to, non-transitory, recordable and non-recordable types of media such as volatile and nonvolatile memory devices, read only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, magnetic disk storage media, optical storage media (eg, compact disk read only memory (CD ROM), digital versatile disk (DVD), etc.), and the like. A computer readable medium may store instructions.

Instructions may also be embodied in digital and analog communication links for electrical, optical, acoustic or other forms of propagated signals, such as carrier waves, infrared signals, digital signals, and the like. However, a propagated signal (such as a carrier wave, infrared signal, digital signal, etc.) is not a tangible, machine-readable medium and is not configured to store instructions.

In general, machine-readable media include media that are provided (i.e., storage and and/or any mechanism for transmitting) information.

In various embodiments, hard-wired circuitry may be used in conjunction with software instructions to implement the techniques. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source for instructions executed by a data processing system.

In the foregoing specification, the disclosure has been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications may be made therein without departing from the broader spirit and scope as set forth in the following claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims (20)

1. a kind of system, the system comprises:

Multiple sensor modules, the corresponding sensor module of each of the multiple sensor module have Inertial Measurement Unit And the position of user is attached to generate the exercise data of the orientation sequence at the position for identifying the user；And

Computing device is coupled to the multiple sensor module and is configured as:

The orientation sequence measured by each corresponding sensor module is fed as input to artificial neural network Network；

Obtain the orientation measurement at the position of the user as the output from the artificial neural network；And

By according to rigid in the kinematic chain from the orientation measurement that the artificial neural network obtains to skeleton pattern Property position orientation configured control application, the skeleton pattern have by joint connect it is multiple rigidity positions.

2. system according to claim 1, wherein instructed using the orientation measurement generated using optical tracking system Practice the artificial neural network；The artificial neural network is the recurrence mind comprising at least one shot and long term memory (LSTM) unit Through network.

3. system according to claim 2, wherein the orientation at the position of the user without using optical tracking come with It track and is tracked without using sensor module comprising Inertial Measurement Unit；And the multiple sensor module pair with it is described The position of the corresponding user of the subset at the rigid position in kinematic chain tracks；And the subset at the rigidity position It is separated in the kinematic chain by the rigid position, the rigidity position and the use tracked without using sensor module The position at family is corresponding.

4. system according to claim 1, wherein the artificial neural network is trained to using optical tracking system The orientation measurement of generation is predicted；And the artificial neural network, which provides, to be directed to quilt by the optical tracking system The position of the user of the multiple sensor module is attached and the orientation measurement of prediction that generates is as output.

5. a kind of method, which comprises

It receives and is identified described in the user from multiple Inertial Measurement Units at the position for being attached to the user connected as joint The exercise data of the orientation sequence at position；

It is provided the orientation sequence at the position of the user measured by the Inertial Measurement Unit as input To artificial neural network；

Obtain the orientation measurement at the position of the user as the output from the artificial neural network；

According to the orientation measurement obtained from the artificial neural network come the rigidity in the kinematic chain to skeleton pattern The orientation at position is configured, and the skeleton pattern has the multiple rigid positions connected by joint；And

Application is controlled based on the state of the kinematic chain.

6. according to the method described in claim 5, wherein, the artificial neural network is to remember (LSTM) unit with shot and long term Recurrent neural network；The position of the user does not have the inertia measurement for measuring the attachment of the orientation at the position Unit；The artificial neural network also exports the prediction at the position of the user tracked using the Inertial Measurement Unit Orientation measurement；And the orientation measurement of the prediction generated by the artificial neural network is to by the inertia The accumulated error in the orientation sequence that measuring unit is measured is corrected.

7. a kind of method, which comprises

Multiple sensor modules are attached to people；

Multiple orientation sequences of the sensor module are measured during the people executes multiple motion sequences, are taken with generating first To measurement result and second orientation measurement result；And

It is described to be predicted based on the first orientation measurement result using supervision machine learning art training artificial neural network Second orientation measurement result.

8. according to the method described in claim 7, wherein, the artificial neural network is to remember (LSTM) unit with shot and long term Recurrent neural network；And the method also includes:

The other measurement result to the people is generated using the multiple sensor module；And

Using the other measurement result and intensified learning technology come to previously based on the first orientation measurement result and institute Second orientation measurement result is stated to be trained using the artificial neural network of supervision machine learning art training.

9. according to the method described in claim 7, wherein, measuring the first orientation measurement result using the first technology；And The second orientation measurement result is measured using the second technology；And the second orientation measurement result identifies the multiple sensing The orientation of device module；And the first orientation measurement result identifies the orientation of the subset of the multiple sensor module.

10. according to the method described in claim 7, wherein, the first orientation measurement result identifies the multiple sensor die The orientation of first subset of block；And the second orientation measurement result identifies the second subset of the multiple sensor module Orientation.

11. a kind of system, the system comprises:

Multiple sensor modules, the corresponding sensor module of each of the multiple sensor module have Inertial Measurement Unit And it is attached to the position of user, it is described to generate the exercise data of the orientation sequence at the position for identifying the user Multiple sensor modules include the first subset and second subset, and first subset and the second subset are shared in described first Common sensor module between subset and the second subset；

Computing device is coupled to the multiple sensor module and is configured as:

The orientation measurement generated by first subset is fed as input to the first artificial neural network；

Obtain as the output from first artificial neural network the user common sites at least one first Orientation measurement, the common sensor module are attached in the common sites of the user；

The orientation measurement generated by the second subset is fed as input to the second artificial neural network；

At least one second orientation for obtaining the common sites as the output from second artificial neural network is surveyed Measure result；And

By by described at least one first orientation measurement result described in the common sites and the common sites at least One second orientation measurement result is combined and generates the orientation measurement of the prediction of the common sites.

12. system according to claim 11, wherein first artificial neural network and second artificial neural network Network includes that shot and long term remembers (LSTM) unit；The combination is executed using third artificial neural network；The third is manually refreshing It include that two-way shot and long term remembers (BLSTM) unit through network；And the institute of the common sites is predicted for first time example State the orientation measurement of prediction；At least one described first orientation measurement result of the common sites includes:

It is described total by the second time instance before the first time example of first neural network prediction With the orientation measurement at position；And

It is described total by the third time instance after the first time example of first neural network prediction With the orientation measurement at position；And

At least one described second orientation measurement result of the common sites includes:

By second time instance place before the first time example of second neural network prediction State the orientation measurement of common sites；And

By the third time instance place after the first time example of second neural network prediction State the orientation measurement of common sites.

13. system according to claim 12, wherein first artificial neural network and second artificial neural network Network includes that shot and long term remembers (LSTM) unit；Via the first orientation measurement result to the common sites and described common The second orientation measurement result at position averagely executes the combination；To the institute of optical tracking system to be used measurement The orientation measurement for stating the prediction of common sites is predicted；First subset tracks the first movement of the user Chain is controlled with the movement of the correspondence kinematic chain to the skeleton pattern in the system；And the second subset tracks institute The second chain of user is stated, is controlled with the movement of the correspondence kinematic chain to the skeleton pattern in the system；And no The orientation of the first position in first kinematic chain of the user is tracked using Inertial Measurement Unit；It is surveyed without using inertia Measure the orientation of the second position in second kinematic chain of the unit to track the user；The first artificial neural network root The orientation of the first position is predicted according to the orientation measurement generated by first subset；And second people Artificial neural networks predict the orientation of the second position according to the orientation measurement generated by the second subset.

14. according to claim 1 or system described in 11, wherein the Inertial Measurement Unit includes MEMS (MEMS) Gyroscope；And the Inertial Measurement Unit further includes magnetometer and mems accelerometer.

15. a kind of method, which comprises

It receives and is identified described in the user from multiple Inertial Measurement Units at the position for being attached to the user connected as joint The exercise data of the orientation sequence at position, the multiple Inertial Measurement Unit include the first subset and second subset, described first Subset and second subset are shared in the common Inertial Measurement Unit between first subset and the second subset；

The orientation measurement generated by first subset is fed as input to the first artificial neural network；

Obtain at least one of the common sites of the user as the output from first artificial neural network First orientation measurement result, the common Inertial Measurement Unit are attached in the common sites of the user；

The orientation measurement generated by the second subset is fed as input to the second artificial neural network；

At least one second orientation for obtaining the common sites as the output from second artificial neural network is surveyed Measure result；And

By by described at least one first orientation measurement result described in the common sites and the common sites at least One second orientation measurement result is combined and generates the orientation measurement of the prediction of the common sites.

16. according to the method for claim 15, wherein the combination includes:

By described at least one first orientation measurement result described in the common sites and the common sites at least one Second orientation measurement result is fed as input to third artificial neural network；And

Obtain the orientation measurement of the prediction of the common sites as the output from the third artificial neural network As a result；

Wherein, first artificial neural network and second artificial neural network include shot and long term memory (LSTM) unit； And

The third artificial neural network includes two-way shot and long term memory (BLSTM) unit.

17. according to the method for claim 15, wherein the combination includes calculating as the described pre- of the common sites Described the second of the first orientation measurement results of the common sites of the orientation measurement of survey and the common sites The average value of orientation measurement.

18. a kind of method, which comprises

Multiple sensor modules are attached to people, the multiple sensor module includes the first kinematic chain for tracking the people Orientation the sensor module the first subset and the second kinematic chain for tracking the people orientation the sensing The second subset of device module, wherein the common sites of first kinematic chain and second kinematic chain with the people；

During the people executes multiple motion sequences, the multiple of the sensor module are measured using the sensor module It is orientated sequence；

During the people executes multiple motion sequences, the measurement result and use independently of the sensor module are individual Tracking system measures the orientation of the sensor module；

Using supervision machine learning art the first artificial neural network of training, to use the orientation from first subset to measure As a result the orientation measurement of first kinematic chain from the individual tracking system is predicted；

Using the supervision machine learning art the second artificial neural network of training, to use the orientation from the second subset Measurement result predicts the orientation measurement of second kinematic chain from the individual tracking system；And

Using supervision machine learning art training third artificial neural network, according to from first artificial neural network The orientation measurement and given birth to from second artificial neural network that the first of the common sites of the people generated is predicted At the people the common sites second prediction orientation measurement, to the orientation of the common sites of the people Measurement result is predicted.

19. according to the method for claim 18, wherein first artificial neural network and second artificial neural network Network includes shot and long term memory (LSTM) unit；And the third artificial neural network includes two-way shot and long term memory (BLSTM) Unit；

Wherein, in order to predict the orientations of the common sites of the people at first time example, the third artificial neuron Network receives following as input:

By the second time instance people before the first time example of first neural network prediction The common sites orientation；

By the third time instance people after the first time example of first neural network prediction The common sites orientation；

By second time instance place before the first time example of second neural network prediction State the orientation of the common sites of people；And

By the third time instance place after the first time example of second neural network prediction State the orientation of the common sites of people；And

Wherein, using first fortune at the first time example of first subset from the sensor module The orientation measurement of dynamic chain to by first neural network prediction the second time instance people's The orientation of the common sites is updated；And

Use second kinematic chain at the first time example of the second subset from the sensor module Orientation measurement to by first neural network prediction described in the second time instance people The orientation of common sites is updated.

20. according to the method for claim 18, further includes:

First artificial neural network and second artificial neural network are trained, with reduce it is following between the two Difference:

The orientation measurement of first prediction of the common sites of the people generated from first artificial neural network As a result；And

The orientation measurement of second prediction of the common sites of the people generated from second artificial neural network As a result.