TW201928604A - Methods and apparatuses for identifying gestures based on ultrasound data - Google Patents

Abstract

Aspects of the techniques described herein relate to methods and devices for identifying gestures based on ultrasound data. Performing gesture recognition may include obtaining ultrasound data corresponding to an anatomical gesture by a wearable device; and identifying the anatomical gesture based on the obtained ultrasound data. Interfacing with a computing device may include identifying a anatomical posture by using a wearable device using ultrasound data obtained by the wearable device; and causing the computing device to perform a specific function based on the anatomical posture recognized by the wearable device. . Training a wearable device to perform posture recognition may include obtaining ultrasonic data corresponding to an anatomical posture through the wearable device; obtaining non-ultrasonic data corresponding to the anatomical posture; and training one of the wearable devices to access A machine learning model to identify the anatomical pose based on correlating the non-ultrasonic data with the ultrasonic data.

Description

Method and device for identifying gestures based on ultrasonic data

The detailed system of specific examples is about the collection of ultrasonic data. Some aspects are related to identifying gestures based on ultrasound data.

This application is based on 35 USC § 119 (e). No. 62 / 609,600, which application is hereby incorporated by reference in its entirety.

Ultrasound devices can be used to collect ultrasound data using sound waves with a higher frequency than those audible by humans. When transmitting ultrasound pulses into tissues (for example, by using a probe), the sound waves are reflected from the tissue, which has different tissues that reflect different degrees of sound. These reflected sound waves can then be recorded. The strength (amplitude) of the acoustic signal and the time it takes for the wave to travel through the body can provide information about the tissue affected by the ultrasound.

According to one aspect, a method of training a wearable device to perform posture recognition includes obtaining ultrasonic data corresponding to an anatomical posture through the wearable device; obtaining non- ultrasonic data corresponding to the anatomical posture; and training by the A wearable device accesses a machine learning model to identify the anatomical posture based on correlating the non-ultrasonic data with the ultrasonic data.

In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more position sensing devices. In some specific examples, the one or more position sensing devices include one or more accelerometers, gyroscopes, magnetometers, compasses, and Global Positioning System (GPS). In some specific examples, the method further includes obtaining the non-ultrasonic data using the one or more position sensing devices. In some specific examples, the method further includes obtaining the non-ultrasonic data using an image capture device. In some specific examples, the image capturing device includes one of a camera, a smartphone, or a tablet device. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

According to another aspect, a system for training a wearable device to perform posture recognition includes a processing circuit configured to: obtain ultrasonic data corresponding to an anatomical posture from the wearable device; and obtain an anatomy corresponding to the anatomy Non-ultrasonic data of the posture; and training a machine learning model accessed by the wearable device to identify the anatomical posture based on correlating the non-ultrasonic data with the ultrasonic data.

In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more position sensing devices. In some specific examples, the one or more position sensing devices include one or more accelerometers, gyroscopes, magnetometers, compasses, and Global Positioning System (GPS). In some specific examples, the processing circuit is further configured to obtain the non-ultrasonic data using the one or more position sensing devices. In some specific examples, the processing circuit is further configured to obtain the non-ultrasonic data using an image capture device. In some specific examples, the image capturing device includes one of a camera, a smartphone, or a tablet device. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

According to another aspect, at least one non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by at least one processor, cause the at least one processor to: The device obtains ultrasound data corresponding to an anatomical pose; obtains non-ultrasonic data corresponding to the anatomical pose; and trains a machine learning model accessed by the wearable device based on correlating the non-ultrasonic data with the ultrasound data To identify the anatomical posture.

In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more position sensing devices. In some specific examples, the one or more position sensing devices include one or more accelerometers, gyroscopes, magnetometers, compasses, and Global Positioning System (GPS). In some specific examples, the at least one non-transitory computer-readable storage medium further stores processor-executable instructions that, when executed by at least one processor, causes the at least one processor to use the one or more position sensing The device obtains the non-ultrasonic data. In some specific examples, the at least one non-transitory computer-readable storage medium further stores processor-executable instructions, which when executed by the at least one processor, causes the at least one processor to obtain the non-transitory device using an image capture device Sonic data. In some specific examples, the image capturing device includes one of a camera, a smartphone, or a tablet device. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

According to another aspect, a method for performing posture recognition includes: obtaining ultrasonic data corresponding to an anatomical posture by a wearable device; and identifying the anatomical posture based on the obtained ultrasonic data.

In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more position sensing devices. In some specific examples, the wearable device further includes one or more ultrasonic transducers acoustically coupled to an anatomical feature on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, and they are selected to obtain the ultrasonic data from the anatomical features. In some specific examples, the wearable device further includes an ultrasonic gel pad configured to be placed between the one or more ultrasonic transducers and the anatomical feature on which the wearable device resides. . In some specific examples, the wearable device further includes one or more ultrasonic transducers disposed in an outward direction with respect to one of the first anatomical features on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, which is selected to obtain the ultrasound from a second anatomical feature on which one of the wearable devices does not reside. Sonic data. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

According to another aspect, an apparatus for performing posture recognition includes a wearable device configured to obtain ultrasonic data corresponding to an anatomical posture; and identifying the anatomical posture based on the obtained ultrasonic data. In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more position sensing devices. In some specific examples, the wearable device further includes one or more ultrasonic transducers acoustically coupled to an anatomical feature on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, and they are selected to obtain the ultrasonic data from the anatomical features. In some specific examples, the wearable device further includes an ultrasonic gel pad configured to be placed between the one or more ultrasonic transducers and the anatomical feature on which the wearable device resides. . In some specific examples, the wearable device further includes one or more ultrasonic transducers disposed in an outward direction with respect to one of the first anatomical features on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, which is selected to obtain the ultrasound from a second anatomical feature on which one of the wearable devices does not reside. Sonic data. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

According to another aspect, at least one non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by at least one processor, cause the at least one processor to obtain a anatomical posture from a wearable device Ultrasonic data; and identifying the anatomical posture based on the obtained ultrasonic data. In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more position sensing devices. In some specific examples, the wearable device further includes one or more ultrasonic transducers acoustically coupled to an anatomical feature on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, and they are selected to obtain the ultrasonic data from the anatomical features. In some specific examples, the wearable device further includes an ultrasonic gel pad configured to be placed between the one or more ultrasonic transducers and the anatomical feature on which the wearable device resides. . In some specific examples, the wearable device further includes one or more ultrasonic transducers disposed in an outward direction with respect to one of the first anatomical features on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, which is selected to obtain the ultrasound from a second anatomical feature on which one of the wearable devices does not reside. Sonic data. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

According to another aspect, a method of interfacing with a computing device includes identifying a anatomical posture by using a wearable device using ultrasound data obtained by the wearable device; and causing the computing device to be based on the wearable device The identified anatomical posture performs a specific function.

In some specific examples, the computing device includes one of a smartphone, a tablet device, a computer, a virtual reality system, or the wearable device itself. In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more ultrasonic transducers acoustically coupled to an anatomical feature on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, and they are selected to obtain the ultrasonic data from the anatomical features. In some specific examples, the wearable device further includes an ultrasonic gel pad configured to be placed between the one or more ultrasonic transducers and the anatomical feature on which the wearable device resides. . In some specific examples, the wearable device further includes one or more ultrasonic transducers disposed in an outward direction with respect to one of the first anatomical features on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, which is selected to obtain the ultrasound from a second anatomical feature on which one of the wearable devices does not reside. Sonic data. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

According to another aspect, a device interfacing with a computing device includes a wearable device configured to identify an anatomical posture using ultrasound data obtained by the wearable device; and causing the computing device to be based on the The anatomical posture recognized by the wearable device performs a specific function.

In some specific examples, the computing device includes one of a smartphone, a tablet device, a computer, a virtual reality system, or the wearable device itself. In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more ultrasonic transducers acoustically coupled to an anatomical feature on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, which is selected to obtain the ultrasonic data from the anatomical features. In some specific examples, the wearable device further includes an ultrasonic gel pad configured to be placed between the one or more ultrasonic transducers and the anatomical feature on which the wearable device resides. . In some specific examples, the wearable device further includes one or more ultrasonic transducers disposed in an outward direction with respect to one of the first anatomical features on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, which is selected to obtain the ultrasound from a second anatomical feature on which one of the wearable devices does not reside. Sonic data. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

According to another aspect, at least one non-transitory computer-readable storage medium stores processor-executable instructions that, when executed by at least one processor, cause the at least one processor to: The ultrasound data obtained by the wearing device identifies an anatomical posture; and causes a computing device to perform a specific function based on the anatomical posture recognized by the wearable device.

In some specific examples, the computing device includes one of a smartphone, a tablet device, a computer, a virtual reality system, or the wearable device itself. In some specific examples, the wearable device includes one of a wristwatch or a wristband. In some specific examples, the wearable device further includes an ultrasonic chip device. In some specific examples, the wearable device further includes one or more ultrasonic transducers acoustically coupled to an anatomical feature on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, which is selected to obtain the ultrasonic data from the anatomical features. In some specific examples, the wearable device further includes an ultrasonic gel pad configured to be placed between the one or more ultrasonic transducers and the anatomical feature on which the wearable device resides. . In some specific examples, the wearable device further includes one or more ultrasonic transducers disposed in an outward direction with respect to one of the first anatomical features on which the wearable device resides. In some specific examples, the one or more ultrasonic transducers are configured to operate in a frequency range, which is selected to obtain the ultrasound from a second anatomical feature on which one of the wearable devices does not reside. Sonic data. In some specific examples, the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Conventional ultrasound systems are large, complex, and expensive systems that are typically purchased only by large medical institutions with large financial resources. Recently, cheaper and less complicated ultrasound imaging devices have been introduced. Such imaging devices may include a monolithic ultrasonic transducer integrated on a single semiconductor die to form a monolithic ultrasonic device. The appearance of these ultrasonic chip devices is described in US Patent Application No. 15 entitled "UNIVERSAL ULTRASOUND DEVICE AND RELATED APPARATUS AND METHODS" filed on January 25, 2017 (and assigned to the assignee of this application) No. / 415,434, the application is incorporated herein by reference in its entirety. The reduced cost and increased portability of these new ultrasonic devices may make them significantly more accessible to the general public compared to conventional ultrasonic devices. In addition, the portability of these new ultrasonic devices makes them suitable for incorporation into wearable devices that can collect ultrasonic data.

The inventors have recognized that these wearable devices may be suitable for use in gesture recognition. Gestures made by, for example, a user's fingers, hands, wrists, and arms on a particular upper limb can cause changes in the muscles in that particular upper limb. These muscle changes can be reflected in the ultrasound data / images collected from that particular limb. The muscle changes caused by the posture made by one part of the upper limb can even be reflected in the ultrasound data / images collected from the other part of the upper limb. For example, extending a finger on a particular upper limb can cause changes in the muscles reflected in the ultrasound data / image collected at the wrist of that particular upper limb. Different postures can cause different changes in muscles, which in turn can cause differences in the ultrasound data / images collected when performing different postures. Differences in ultrasonic data / images collected when performing different poses can be used to distinguish between different poses. In other words, specific features of the ultrasound data / images can be used to identify the anatomical pose performed when the ultrasound data / images are collected. This description discusses a wearable device configured to collect ultrasonic data from a user and identify a gesture being performed by the user.

As used herein, an object configured to be incorporated into an anatomical structure is understood to mean that the object is configured to remain at or near the anatomical structure without externally applied force. For example, an ultrasonic chip device coupled to a watch or bracelet worn on a user's wrist can be considered a "wrist-worn".

As used herein, an “ultrasound-on-a-chip device” should be understood to mean an ultrasonic transducer (eg, a micro-electromechanical device) that is integrated with a semiconductor die containing integrated circuits Ultrasound transducer).

As used herein, "ultrasound data" shall be understood to mean any of the original acoustic data, the original acoustic data that has been converted into another form, and the ultrasonic images generated based on the original acoustic data.

It should be understood that the specific examples described herein may be implemented in any of a number of ways. Examples of specific implementations are provided below for illustrative purposes only. It should be understood that these specific examples and the features / capabilities provided may be used individually, all together, or in any combination of two or more than two, because aspects of the techniques described herein are in this respect Unlimited.

FIG. 1 shows an example process 100 for performing gesture recognition according to some specific examples described herein. The process 100 may be performed by, for example, a processing circuit in a wearable device. Examples of wearable devices will be discussed below with reference to FIGS. 7 to 15.

In act 102, the wearable device can obtain ultrasonic data corresponding to an anatomical posture. For example, a user can perform an anatomical posture by a part of a specific limb (for example, fingers, hands, wrists, arms), and at the same time use a wearable device residing on another limb from the same limb (in Collect ultrasound data at a location (or at the same location) that is different from the part of the limb performing the posture. In a specific example, the wearable device may include one or more ultrasonic transducers acoustically coupled to one of the anatomical features of a limb on which the wearable device resides, and the ultrasonic transducers may be The state operates at a frequency range selected to acquire ultrasound data from the anatomical feature on which the wearable device resides. For example, a user can perform a gesture with a finger on his / her left hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing the user's left wrist can be removed from the user Ultrasonic data indicating gestures made with the fingers of the user's left hand are collected in his left wrist. This ultrasonic data collection may use frequencies in the megahertz range, for example, to collect ultrasonic data from the anatomical features on which the wearable device resides. The ultrasonic transducers are acoustically coupled to a residing element (e.g., an ultrasonic gel pad) disposed between the ultrasonic transducer and the anatomical features on which the wearable device resides. Wearable device limb. In another specific example, the wearable device may include one or more ultrasonic transducers positioned outwardly relative to one of the anatomical features of the limb on which the wearable device resides, and the ultrasonic waves. The transducer may be configured to transmit ultrasonic signals in the air to collect ultrasonic data from an anatomical feature on a limb that does not have a wearable device resident thereon but on the same limb as the limb on which the wearable device resides. For example, a user can perform a gesture with the fingers on his / her left hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing away from the user's left wrist is self-useable The finger on the left hand of the person collects ultrasonic data indicating a gesture made with the finger on the left hand of the user. The ultrasound transducer may be configured to operate in one of a range of frequencies selected to acquire ultrasound data from a limb in the air. For example, this aerial ultrasound imaging can use frequencies in the kilohertz range. In a specific example of a wrist-mounted wearable device transducer configured to transmit ultrasonic signals in the air to collect ultrasonic data from a limb on which the wearable device resides, the wearable device may reside in On the dorsal or palm wrists. In these specific examples, certain postures can be more easily detected when the wearable device resides on the back-side wrist or palm-side wrist. For example, when the wearable device resides on the back side wrist, the posture of the hand or finger bent toward the back side wrist can be more easily detected, and when the wearable device resides on the palm side wrist, the hand or finger faces the palm The side wrist bending posture can be more easily detected. In another specific example, the wearable device may include one or more ultrasonic transducers positioned outwardly relative to one of the anatomical features of the limb on which the wearable device resides, and the ultrasonic waves. The transducer may be configured to transmit ultrasonic signals in the air to collect ultrasonic data from an anatomical feature on a limb opposite the limb on which the wearable device resides. For example, a user can perform a gesture with a finger on his / her right hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing away from the user's left wrist is self-useable The finger on the right hand of the person collects ultrasonic data indicating a gesture made with the finger on the right hand of the user. The user can point the ultrasonic transducer on the left wrist to the right hand to collect the ultrasonic data. The ultrasonic transducers can be configured to operate in a frequency range selected to obtain ultrasonic data from the anatomical features of another limb. For example, this aerial ultrasound imaging can use frequencies in the kilohertz range.

In act 104, the wearable device may identify an anatomical posture based on the obtained ultrasound data. As discussed above, different postures can cause different changes in muscles, which in turn can cause differences in the ultrasound data / images collected when performing different postures. In some specific examples, the wearable device may use a machine learning model (for example, a network with a learning function) to recognize an anatomical gesture based on the obtained ultrasonic data. For example, a processing circuit in a wearable device may be accessed and stored on an internal memory circuit or stored on another device such as a smartphone, tablet device, laptop, or remote server A machine learning model on the memory circuit to recognize anatomical gestures. When accessing a machine learning model on another device, the wearable device can transmit ultrasonic data to another device and receive an indication of a gesture that has been recognized using the machine learning model from the other device. Further examples of deep learning techniques will be discussed below with reference to FIG. 6. Based on ultrasound data is discussed further gesture of recognition can be found in McIntosh, Jess, who's "EchoFlex: Hand Gesture Recognition using Ultrasound Imaging" (about 2017 CHI session of the Conference of the Human Factors in Computing System records, ACM, 2017), the It is incorporated herein by reference in its entirety.

Example gestures that a wearable device can be configured to recognize involving finger movement include pointing with any finger or combination of fingers, pinching any two or more fingers together, and touching with any finger or combination of fingers (Where tapping can be on a solid surface or a tapping motion can be performed in the air), counting (that is, extending any number of fingers while counting, as is customary), fracturing (ie, holding a pinch Together or any combination of fingers in a fist and then extending the fingers outwards), or twisting the fingers. Example gestures that the wearable device can be configured to recognize involving hand or wrist movement include clapping, swiping, waving, or orienting the hand in a particular direction. Example gestures that the wearable device can be configured to recognize involving arm movement include push, pull, hit, throw, throw, flap, and swim. It should be understood that such movements may include movements performed on an object (such as pushing and pulling an object), or simulations of movements performed on an object without actually performing movements on the object (such as simulating pushing and pulling an object, Without actually pushing or pulling the object).

In some specific examples, the wearable device may be configured to detect a gesture performed on an object. For example, one gesture may be tapping a square in a grid, and another gesture may be tapping another square in a grid. The wearable device may be configured to cause an electronic device to perform different functions in response to identifying that the user is tapping on a different square. The user's hand / wrist / arm portion can be constrained during the execution of these gestures to assist in distinguishing the ultrasonic data corresponding to each gesture. For example, in an example where a square on a grid is tapped, the user's wrist may be fixed in a position during the tap.

In some specific examples, the wearable device may be configured to identify both gesture types and parameters. This parameter can be, for example, distance, direction, or number. As a specific example, the wearable device can be configured to distinguish between extending five fingers and extending four fingers, or moving a hand across a small distance and a hand across a large distance.

In some specific examples, the wearable device may be configured to recognize gestures based on ultrasound data and non-ultrasonic data. The non-ultrasonic data may be, for example, data about the location of the wearable device, and may be by a position sensing device in the wearable device such as an accelerometer, gyroscope, magnetometer, compass and / or Global Positioning System (GPS) device). The gesture may include identifying one or more components based on the ultrasound data and identifying one or more components based on the non-ultrasonic data. Such pose components identified based on non-ultrasonic data may include pose components that do not cause changes in muscles, which in turn cause changes in the ultrasound data / images collected by the wearable device when performing different pose components. For example, certain arm movements, such as the movement of an arm swinging around an elbow, may not cause changes in the ultrasound data / images collected by the wearable device coupled to the wrist. Therefore, for one example posture including finger movement and arm swing, finger movement can be identified based on ultrasonic data, and arm swing can be identified based on non-ultrasonic data such as information about the position of the wearable device. The combined recognition of finger movement and arm swing from ultrasonic data and non-ultrasonic data can be used together to identify the total posture.

It should be understood that the wearable device can be configured to recognize any gesture with any movement or combination of movements, and the gesture need not be constrained to a predefined type or gesture type or type. For example, the wearable device may be configured to recognize gestures created by a user.

FIG. 2 shows an example process 200 for interfacing with a computing device according to some specific examples described herein. The process 200 may be performed by, for example, a processing circuit in a wearable device. Examples of wearable devices will be discussed below with reference to FIGS. 7 to 15.

In act 202, the wearable device may identify an anatomical posture using ultrasonic data obtained from the wearable device. For example, a user can perform an anatomical posture by a part of a specific limb (for example, fingers, hands, wrists, arms), and at the same time use a wearable device residing on another limb from the same limb (in Collect ultrasound data at a location (or at the same location) that is different from the part of the limb performing the posture. In a specific example, the wearable device may include one or more ultrasonic transducers acoustically coupled to one of the anatomical features of a limb on which the wearable device resides, and the ultrasonic transducers may be The state operates at a frequency range selected to acquire ultrasonic data from an anatomical feature on which the wearable device resides. For example, a user can perform a gesture with a finger on his / her left hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing the user's left wrist can be removed from the user Ultrasonic data indicating gestures made with the fingers of the user's left hand are collected in his left wrist. This ultrasonic data collection may use frequencies in the megahertz range, for example, to collect ultrasonic data from the anatomical features on which the wearable device resides. The ultrasonic transducers are acoustically coupled to a residing element (e.g., an ultrasonic gel pad) disposed between the ultrasonic transducer and the anatomical features on which the wearable device resides. Wearable device limb. In another specific example, the wearable device may include one or more ultrasonic transducers positioned outwardly relative to one of the anatomical features of the limb on which the wearable device resides, and the ultrasonic waves. The transducer may be configured to transmit ultrasonic signals in the air to collect ultrasonic data from an anatomical feature on a limb that does not have a wearable device resident thereon but on the same limb as the limb on which the wearable device resides. For example, a user can perform a gesture with the fingers on his / her left hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing away from the user's left wrist is self-useable The finger on the left hand of the person collects ultrasonic data indicating a gesture made with the finger on the left hand of the user. The ultrasound transducer may be configured to operate in one of a range of frequencies selected to acquire ultrasound data from a limb in the air. For example, this aerial ultrasound imaging can use frequencies in the kilohertz range. In a specific example of a wrist-mounted wearable device transducer configured to transmit ultrasonic signals in the air to collect ultrasonic data from a limb on which the wearable device resides, the wearable device may reside in On the dorsal or palm wrists. In these specific examples, certain postures can be more easily detected when the wearable device resides on the back-side wrist or palm-side wrist. For example, when the wearable device resides on the back side wrist, the posture of the hand or finger bent toward the back side wrist can be more easily detected, and when the wearable device resides on the palm side wrist, the hand or finger faces the palm The side wrist bending posture can be more easily detected. In another specific example, the wearable device may include one or more ultrasonic transducers positioned outwardly relative to one of the anatomical features of the limb on which the wearable device resides, and the ultrasonic waves. The transducer may be configured to transmit ultrasonic signals in the air to collect ultrasonic data from an anatomical feature on a limb opposite the limb on which the wearable device resides. For example, a user can perform a gesture with a finger on his / her right hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing away from the user's left wrist is self-useable The finger on the right hand of the person collects ultrasonic data indicating a gesture made with the finger on the right hand of the user. The user can point the ultrasonic transducer on the left wrist to the right hand to collect the ultrasonic data. The ultrasonic transducers can be configured to operate in a frequency range selected to obtain ultrasonic data from the anatomical features of another limb. For example, this aerial ultrasound imaging can use frequencies in the kilohertz range.

As discussed above, different postures can cause different changes in muscles, which in turn can cause differences in the ultrasound data / images collected when performing different postures. In some specific examples, the wearable device may use a machine learning model (eg, a network with a learning function) to identify the anatomical pose based on the obtained ultrasonic data. For example, a processing circuit in a wearable device may be accessed and stored on an internal memory circuit or stored on another device such as a smartphone, tablet device, laptop, or remote server A machine learning model on the memory circuit to recognize anatomical gestures. When accessing a machine learning model on another device, the wearable device can transmit ultrasonic data to another device and receive an indication of a gesture that has been recognized using the machine learning model from the other device. Further examples of deep learning techniques will be discussed below with reference to FIG. 6. Further examples of gestures recognizable by the wearable device are discussed above with reference to FIG. 1. In addition, as discussed above with reference to FIG. 1, certain gestures may be identified based on ultrasound data and non-ultrasonic data.

In act 204, the wearable device may cause the computing device to perform a specific function based on the anatomical gesture recognized by the wearable device. The computing device may be, for example, a smartphone, a tablet device, a computer, a virtual reality system, or the wearable device itself. In order for another computing device to perform a specific function, the wearable device may use a wireless communication circuit (eg, BLUETOOTH, ZIGBEE, and / or WiFi wireless communication circuits) to transmit signals to the electronic device via a wireless network. In order for the wearable device to perform a specific function itself, the wearable device may generate an internal signal in the processing circuit of the wearable device based on the posture being performed. The wearable device can access a database (locally stored or stored on a remote device) containing the association between posture and function. After recognizing a gesture, the wearable device can find the gesture in the database and cause the computing device to perform a function associated with the gesture in the database.

In some specific examples, in response to identifying a particular gesture being performed, the wearable device may be configured to cause the electronic device to move a cursor on a display of the electronic device. For example, when a user moves his / her hand in a certain direction, the cursor can move across the display in that direction. In some specific examples, in response to identifying a particular gesture, the wearable device may be configured to enable the electronic device to control keyboard input of the electronic device. For example, a user may simulate a series of letters typed on a keyboard that may not actually exist, and based on identifying their gestures, the series of letters may be input to the electronic device. As another example, in response to detecting that the user performs a sign language gesture corresponding to a series of letters, the wearable device may be configured to input the series of letters to the electronic device. In some specific examples, in response to identifying a user performing a gesture corresponding to dancing (ie, dance movement), the wearable device may be configured to enable an avatar of the user on a display screen of the electronic device to be on the electronic device Perform the same gesture on the display screen. In some specific examples, in response to identifying a user performing a gesture corresponding to playing sports (eg, throwing a ball, swinging a baseball or tennis racket or a golf club, shooting a ball, etc.), the wearable device may be configured to enable the user to The avatar performs the same gesture on the display of the electronic device. In some specific examples, in response to identifying a user performing a gesture corresponding to playing a musical instrument (eg, pressing keys on a piano, drumming, plucking strings on a guitar, etc.), the wearable device may be configured such that The user's avatar performs the same gesture on the display screen of the electronic device. In some specific examples where the electronic device is a virtual reality device, in response to recognizing that the user performs a certain gesture, the wearable device may be configured to cause the electronic device to replicate that gesture in the virtual world, or perform a corresponding operation in the virtual world. A gesture of gesture.

In some specific examples, the wearable device may be configured to identify both gesture types and parameters. This parameter can be, for example, distance, direction, or number. In some specific examples, depending on the value of the parameter, the wearable device may be configured to cause the electronic device to perform different functions in response to recognizing the same gesture type. For example, in response to recognizing that the user is extending five fingers, the wearable device may be configured to increase the volume of an electronic device to a higher volume than when the wearable device recognizes that the user is extending four fingers. . As another example, in response to identifying that the user is moving his / her hand to the right, the wearable device may be configured to cause an electronic device to slide the display screen to the right, and in response to identifying that the user is moving his / her hand to the right / Her hand moves to the left, and the wearable device can be configured to cause the electronic device to slide the display screen to the left. As another example, in response to recognizing that the user is moving his / her hand over a large distance, the wearable device may be configured to cause an electronic device to scroll the screen down than the wearable device recognizes that the user is A small distance in the case of moving his / her hand a long distance.

In some specific examples, the wearable device may be configured to cause an electronic device to perform a specific function based on detecting the execution of a specific gesture. For example, in response to detecting that the user has raised five fingers, the wearable device may be configured to cause the electronic device to open a specific application. In some specific examples, the wearable device may be configured to cause an electronic device to perform a specific function based on detecting a transition from one posture to another. For example, in response to detecting that a user has switched from raising four fingers to raising five fingers, the wearable device can be configured to enable an electronic device to open an application, and in response to detecting use The user has switched from raising three fingers to raising five fingers, and the wearable device can be configured to enable the electronic device to open another application.

FIG. 3 shows an example process 300 for training a wearable device to perform gesture recognition according to some specific examples described herein. The process 300 may be, for example, a processing circuit in a wearable device and / or another device or devices that communicate with the wearable device (eg, a smartphone, tablet device, laptop, or one or more devices) Server). Examples of wearable devices will be discussed below with reference to FIGS. 7 to 15.

In act 302, the processing circuit may use the wearable device to obtain ultrasonic data corresponding to an anatomical posture. For example, a user can perform an anatomical posture by a part of a specific limb (for example, fingers, hands, wrists, arms), and at the same time use a wearable device residing on another limb from the same limb (in Collect ultrasound data at a location (or at the same location) that is different from the part of the limb performing the posture. In a specific example, the wearable device may include one or more ultrasonic transducers acoustically coupled to one of the anatomical features of a limb on which the wearable device resides, and the ultrasonic transducers may be The state operates at a frequency range selected to acquire ultrasound data from the anatomical feature on which the wearable device resides. For example, a user can perform a gesture with a finger on his / her left hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing the user's left wrist can be removed from the user Ultrasonic data indicating gestures made with the fingers of the user's left hand are collected in his left wrist. This ultrasonic data collection may use frequencies in the megahertz range, for example, to collect ultrasonic data from the anatomical features on which the wearable device resides. The ultrasonic transducers are acoustically coupled to a residing element (e.g., an ultrasonic gel pad) disposed between the ultrasonic transducer and the anatomical features on which the wearable device resides. Wearable device limb. In another specific example, the wearable device may include one or more ultrasonic transducers positioned outwardly relative to one of the anatomical features of the limb on which the wearable device resides, and the ultrasonic waves. The transducer may be configured to transmit ultrasonic signals in the air to collect ultrasonic data from an anatomical feature on a limb that does not have a wearable device resident thereon but on the same limb as the limb on which the wearable device resides. For example, a user can perform a gesture with the fingers on his / her left hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing away from the user's left wrist is self-useable The finger on the left hand of the person collects ultrasonic data indicating a gesture made with the finger on the left hand of the user. The ultrasound transducer may be configured to operate in one of a range of frequencies selected to acquire ultrasound data from a limb in the air. For example, this aerial ultrasound imaging can use frequencies in the kilohertz range. In a specific example of a wrist-mounted wearable device transducer configured to transmit ultrasonic signals in the air to collect ultrasonic data from a limb on which the wearable device resides, the wearable device may reside in On the dorsal or palm wrists. In these specific examples, certain postures can be more easily detected when the wearable device resides on the back-side wrist or palm-side wrist. For example, when the wearable device resides on the back side wrist, the posture of the hand or finger bent toward the back side wrist can be more easily detected, and when the wearable device resides on the palm side wrist, the hand or finger faces the palm The side wrist bending posture can be more easily detected. In another specific example, the wearable device may include one or more ultrasonic transducers disposed in an outward direction relative to one of the anatomical features of the limb on which the wearable device resides, and the ultrasonic waves The transducer may be configured to transmit ultrasonic signals in the air to collect ultrasonic data from an anatomical feature on one of the limbs opposite the limb on which the wearable device resides. For example, a user can perform a gesture with a finger on his / her right hand, and a wearable device that resides on the user's left wrist and has an ultrasonic transducer facing away from the user's left wrist is self-useable The finger on the right hand of the person collects ultrasonic data indicating a gesture made with the finger on the right hand of the user. The user can point the ultrasonic transducer on the left wrist to the right hand to collect the ultrasonic data. The ultrasonic transducers can be configured to operate in a frequency range selected to obtain ultrasonic data from the anatomical features of another limb. For example, this aerial ultrasound imaging can use frequencies in the kilohertz range.

In act 304, the processing circuit can obtain non-ultrasonic data corresponding to the anatomical posture. The non-ultrasonic data may be tags for gestures (eg, pointing, pinching, tapping, counting, spreading, twisting fingers, clapping, slamming, waving, orienting, pushing, pulling, hitting, throwing, throwing, flapping and swimming) . In order to obtain non-ultrasonic data, the wearable device or another electronic device may prompt the user by displaying a label of the gesture to the user, and collect a gesture corresponding to the gesture performed by the user in response to the prompt (discussed in action 302 above) ) Ultrasonic data. The processing circuit may then correlate the ultrasound data with the cue's gesture tag (ie, non-ultrasonic data). In some specific examples, the wearable device or another computing device can display images or videos of gestures for user execution. In some specific examples, another person may demonstrate a gesture for user performance. In a specific example where the processing circuit is external to the device prompting the user, the processing circuit may receive a gesture tag related to ultrasonic data via a wireless network.

In some specific examples, the wearable device may capture ultrasonic data (as discussed above) from a limb performing a posture, while another electronic device may capture non- ultrasonic data. For example, the user can perform the gesture with one limb on which the wearable device resides, and the user can grasp the electronic device with the hand on the other limb and use the electronic device to capture the non-superiority of the posture being performed Sonic data (for example, shown in Figure 4 below). The non-ultrasonic data may be, for example, a two-dimensional or three-dimensional image or video of an anatomical posture, wherein the image or video is captured using an imaging modality different from that of ultrasonic imaging. The imaging modality may be, for example, standard optical imaging, radar imaging, or laser imaging. In order to capture two- or three-dimensional images or videos, it is possible to use devices such as smartphones, cameras (that is, devices whose primary purpose is to capture optical images) or have imaging sensors (eg, standard camera sensors, radar imaging sensors Image capture device for tablet computer devices such as sensors or laser imaging sensors). In order to capture three-dimensional images / videos, in some specific examples, the electronic device may include a structured lighting projector and a standard camera laterally offset from at least the structured lighting projector. In some specific examples, in order to collect three-dimensional images / videos of the posture being performed, the electronic device may include multiple standard cameras and use multi-view stereo vision technology. The processing circuit may then correlate the two-dimensional or three-dimensional image / video with the ultrasound data. In a specific example where the processing circuit is external to the image capturing device, the processing circuit may receive a two-dimensional or three-dimensional image / video related to ultrasonic data from the image capturing device via a wireless network.

In some specific examples, in order to obtain a three-dimensional image of the posture being performed, the image capture device may be configured to capture a two-dimensional image of the posture being performed, and input the two-dimensional image to a machine learning model (for example, having a Learning Function Network). The correlation between the two-dimensional and three-dimensional images of the same pose that can be performed trains the machine learning model, and once trained, the two-dimensional image of the pose input can be transformed into the three-dimensional image of the pose. To assist in transforming the input two-dimensional image of the pose into the three-dimensional image of the pose, the machine learning model may use a three-dimensional model of the upper limb (or a portion thereof). The model can be used, for example, to infer the position and / or orientation of the upper limb in the three-dimensional position and / or orientation of the masked portion in the two-dimensional image.

In some specific examples, the non-ultrasonic data may include data about the location of the wearable device. For example, when a user performs an anatomical posture using the upper limb on which the wearable device resides, the position of the wearable device may change during the execution of the posture. The position information of the wearable device may be obtained from a position sensing device such as an accelerometer, gyroscope, magnetometer, compass, and / or global positioning system (GPS) device in the wearable device.

In some specific examples, non-ultrasonic data may include information about electrical muscle activity, such as data obtained by electromyography (EMG). Information about electrical muscle activity can be obtained from a surface electrode integrated into a wearable device (eg, if the wearable device is a wrist-bound device, the surface electrode can be integrated into the wristband and / or coupled to the wrist With one of the main modules).

The ultrasound data obtained in act 302 and the non-ultrasonic data obtained in act 304 may be considered training data. In act 306, the processing circuit may train a machine learning model (eg, a learning-enabled network) accessed by the wearable device to identify based on correlating the non-ultrasonic data with the ultrasonic data (ie, training data) Anatomical pose. Machine learning models may use, for example, one or more convolutional neural networks, one or more fully connected neural networks, random forests, support vector machines, linear classifiers, and / or other machine learning models. Referring further to FIG. 6, example machine learning techniques are discussed. In some specific examples, the machine learning model may be stored on a memory circuit in the wearable device. In other specific examples, the machine learning model may be stored on a memory circuit in another computing device (eg, a smartphone, tablet device, laptop, or one or more servers). In a specific example where the processing circuit is external to a device having a memory circuit storing a machine learning model, the processing circuit can access the memory circuit via a wireless network to train the machine learning model.

As discussed above, in some specific examples, the non-ultrasonic data may be a label of a gesture being performed. In these specific examples, a machine learning model may be trained on training data including correlations between ultrasound data collected when performing different anatomical poses and labels of different anatomical poses. Once trained, the machine learning model can be configured to accept ultrasound data as input and output one of the best guesses about the labels of the poses being performed when the ultrasound data is collected. In these specific examples, identifying a gesture includes identifying a tag that is performing a gesture based on the ultrasound data.

As also discussed above, in some specific examples, the non-ultrasonic data may be a two-dimensional or three-dimensional image or video of the posture being performed. In order to process images / videos as training data, in some specific examples, a machine learning model can be used to model the relationship between the ultrasound data and a parametric model of the upper limb, and a second machine learning model can be used Modeling the relationship between a parametric model of the upper limb and a particular posture. In detail, the first machine learning model may use a parameterized model of one of the upper limbs, in which the upper limb is modeled as a combination of parts (eg, fingers, hands, wrists, etc.), among which Each has one or more parameters (eg, location, orientation, etc.) that have a finite set of feasible values. (It should be noted that the parameters of the upper limb model are not the same as any weights that can be applied to the input or are not associated with edges in the neural network). By determining the value of a model parameter corresponding to the pose based on the image / video determination of the pose displayed in an image / video, the first machine learning model can be trained to classify the pose. For example, the first machine learning model may be trained on the correlation between the image / video of the pose and the corresponding model parameter values for the pose. Once the image / video has fitted the model parameter values, the correlation between the model parameter values representing the pose and the ultrasound data collected when performing the pose can be used as training data to train a second machine learning model. Once trained, the second machine learning model can accept the ultrasound data as an input and output one of the best guesses about the value of the model parameter representing the pose performed during the collection of the ultrasound data. In contrast to the specific examples discussed above, in this specific example, identifying a gesture includes identifying model parameter values that represent performing gestures based on ultrasound data.

In some specific examples, after training a machine learning model on training data including correlations between model parameter values representing a pose and ultrasonic data collected when the pose is performed, the machine learning model can identify that it is not a training set. Partial pose. In other words, the machine learning model can output a specific set of model parameter values representing a pose in response to the input ultrasonic data, even if the specific set of model parameter values is not part of the training data. For example, if the training data includes model parameter values representing three fingers extended and five fingers extended, the machine learning model can output model parameter values representing four fingers extended. Therefore, the number of poses that a machine learning model can recognize can increase beyond a particular pose included in the training set.

In some specific examples, capturing images / videos of the posture being performed can assist in generating training data. Instead of using the prompting method (that is, correlating the ultrasound data with a fixed number of posture tags used to prompt the user to perform a posture), the user can perform a series of continuous unprompted upper limb movements, and can Capture this series of moving 2D or 3D video and ultrasound data. Then, each frame of the video or a subset thereof may be correlated with corresponding ultrasonic data when fitting model parameter values, and used as training data. The number of video frames (each of which can potentially provide a training data point) may be greater than a fixed number of gesture labels used to prompt the user to perform a pose using the prompt method. Therefore, in this specific example, the amount of training data generated may be greater than the amount of training data generated in the specific example using the prompt method.

As discussed above, in some specific examples, non-ultrasonic data may include position data for a wearable device obtained by a position sensing device. This location data can also fit model parameter values, and can therefore be used in a similar manner as discussed above regarding images / videos. In addition, in a specific example where the non-ultrasonic data includes position data for a wearable device, the non-ultrasonic data and the ultrasonic data may be used together as training data for identifying a posture. For example, the training data may include a pose that includes one or more components related to ultrasound data and one or more components related to non-ultrasonic data. Such pose components related to non-ultrasonic data may include pose components that do not cause changes in muscles, which in turn cause changes in the ultrasound data / images collected by the wearable device when performing different pose components. For example, certain arm movements, such as the movement of an arm swinging around an elbow, may not cause changes in the ultrasound data / images collected by the wearable device coupled to the wrist. Therefore, for one example posture including finger movement and arm swing, finger movement may be related to ultrasonic data, and arm swing may be related to non- ultrasonic data such as information about the position of the wearable device.

In some specific examples, the machine learning model may be trained with data collected from a plurality of individuals, and the trained machine learning model may be preloaded onto the wearable device. In some specific examples, machine learning models may be trained with data collected from specific individuals who will use the wearable device. One combination of the two methods is also feasible. In detail, a machine learning model may be trained with data collected from a plurality of individuals, and then the machine learning model may be personalized based on training data from a specific individual who will use the wearable device. To achieve this personalization, techniques similar to those used to personalize machine learning models for speech recognition can be used. For example, training data can be aggregated based on the characteristics of the ultrasound data, and different machine learning models can be trained on different clusters. A cluster of machine learning models corresponding to the characteristics of the ultrasound data of a particular user can then be used.

FIG. 4 shows an example of training a wearable device to perform gesture recognition as discussed with reference to FIG. 3 according to some specific examples discussed herein. In detail, FIG. 4 shows a user performing an anatomical gesture with his / her left hand 402 on which the wearable device 404 resides. The wearable device 404 is configured to capture ultrasound data. The user further grasps an image capturing device 408 (a smartphone in the example of FIG. 4) having a sensor 410 in his / her right hand 406. With the image capturing device, the user captures Non-ultrasonic data of anatomical posture performed by left hand 402. As discussed above, the non-ultrasonic data may be, for example, a two-dimensional or three-dimensional image or video of an anatomical posture, where the image or video is captured using an imaging modality different from that of ultrasonic imaging. The imaging modality may be, for example, standard optical imaging, radar imaging, or laser imaging. The image capture device 408 shown in FIG. 4 is a smart phone, and in some specific examples, the image capture device may be another type of device, such as a camera (ie, a device whose main purpose is to capture optical images) or Tablet device. The sensor 410 may include, for example, a standard camera sensor, a radar imaging sensor, a laser imaging sensor, and / or a structured lighting projector. In order to capture three-dimensional images / videos, in some specific examples, the sensor 410 may include a structured lighting projector and a standard camera (which may be laterally offset from at least the structured lighting projector). In some specific examples, in order to collect three-dimensional images / videos of the posture being performed, the sensor 410 may include multiple standard cameras, and the image capture device 408 uses multi-view stereo vision technology.

Figure 5 shows one example of a "virtual cycle" 500 for continuously improving the performance of certain processes and systems described herein. Stage 502 shows a wearable device (eg, any of the wearable devices discussed herein). Each of these wearable devices may be associated with a different individual, a group of individuals, an institution, or a group of institutions, and may be different. Stage 504 shows that each of the wearable devices shown in stage 502 can be used to perform artificial intelligence (AI) assisted gesture recognition, where the AI recognizes a gesture performed based on ultrasonic data collected during the positive execution of the gesture. Stage 506 shows that the data generated in stage 504 can be used as a source of feedback for the AI model (as shown in stage 508). In detail, during the explicit training of the AI model, the user can be prompted to perform the pose according to the display of the pose classification, and the ultrasound data collected during the execution of the pose can be related to the pose classification in the prompt and can be used as The source of explicit feedback. During normal use of the wearable device, other sources of implicit feedback on whether the gesture has been correctly identified may be generated. For example, during normal use of a wearable device, if the user does not report an error after the device recognizes a gesture based on the collected ultrasound data, the ultrasound data can be associated with the recognized gesture because the gesture can be considered Correctly identified. If the user does report an error after the device recognizes a gesture (for example, by selecting a "report error" option), or repeating the gesture shortly after starting to perform the gesture, the ultrasound data can be identified with the original gesture An incorrect one indicates an association. In addition, during normal use of the wearable device, the wearable device can collect additional information about the anatomical posture while collecting standard ultrasound data (for example, using standard optical imaging, radar imaging, laser imaging, or aerial ultrasound imaging collection) Image). One of the learning-capable networks trained to recognize gestures based on additional data can be used to confirm whether a gesture has been correctly recognized based on the collected ultrasound data. In stage 508, the ultrasound data and feedback from stages 504 and 506 can be uploaded to one or more servers (eg, "cloud" configuration) and can be used to train AI models hosted on the cloud (eg, with Learning network or other machine learning models) to more accurately recognize gestures based on ultrasound data. The trained model can be downloaded from the cloud to the wearable device shown in stage 502 (eg, across a wired or wireless communication link) and used to perform AI-assisted gesture recognition more accurately in stage 504. Therefore, as more wearable devices are deployed and more data is collected and used to train models, wearable devices and AI-assisted gesture recognition can continue to improve as wearable devices are updated with recent AI training updates.

Aspects of the techniques described herein relate to the application of automated image processing techniques to analyze data and images, such as ultrasound data / images. In some specific examples, the ultrasound data / images can be analyzed to identify the pose performed when the ultrasound data / images are captured. In some specific examples, automated image processing techniques may include machine learning techniques such as deep learning techniques. Machine learning techniques may include techniques that search to identify patterns in a set of data points and use the identified patterns to predict new data points. These machine learning techniques may involve training (and / or building) a model using training data sets to make such predictions. A trained model can be used, for example, as a classifier that is configured to receive data points as input and provide as output an indication of the category to which the data points are likely to belong.

Deep learning techniques may include their machine learning techniques that make predictions using networks with learning capabilities (and, more specifically, neural networks). Neural networks typically include a collection of neural units (called neurons), each of which can be configured to receive one or more inputs and provide an output that is a function of the inputs. For example, a neuron can sum the inputs and apply a transfer function (sometimes called a "excitation function") to the summed inputs to produce an output. A neuron may apply a weight to each input to, for example, weight some inputs higher than others. Example transfer functions that can be used include step functions, piecewise linear functions, and sigmoidal curve functions. These neurons may be organized into a plurality of sequential layers each including one or more neurons. The plurality of sequential layers may include an input layer receiving input data for the neural network, an output layer providing output data for the neural network, and one or more hidden layers connected between the input layer and the output layer. Floor. Each neuron in the hidden layer may receive input from one or more neurons in a previous layer (such as an input layer) and provide output to one or more neurons in a subsequent layer (such as an output layer) .

The neural network can be trained using, for example, labeled training data. The labeled training data may include a set of instance inputs and a response associated with each input. For example, the training data may include a plurality of ultrasound data sets, each of which is labeled by performing a pose when capturing the ultrasound data. In this example, the ultrasound data can be provided to a neural network to obtain an output that can be compared with a label associated with each of the ultrasound images. One or more characteristics of a neural network, such as the interconnections between neurons in different layers (called edges) and / or the weights associated with edges, can be adjusted until the neural network has an input Most (or all) of them are correctly classified.

Once the training data has been created, the training data can be loaded into a database and used to train a neural network using deep learning techniques. Once the neural network has been trained, the trained neural network can be deployed to one or more host devices. It should be understood that a neural network can be trained with any number of sample data sets. For example, a neural network can be trained with as few as about seven sample data sets, but it should be understood that the more sample images used, the more robust the model data after training.

In some applications, a neural network may be implemented using one or more convolutional layers to form a convolutional neural network. FIG. 6 shows an example convolutional neural network configured to analyze data 602 (which may be ultrasound data / ultrasonic images) according to one of some specific examples disclosed herein. As shown, the convolutional neural network includes an input layer 604 to receive data 602, an output layer 608 to provide output, and a plurality of hidden layers 606 connected between the input layer 604 and the output layer 608. The plurality of hidden layers 606 include a convolution and aggregation layer 610 and a dense layer 612. The input layer 604 can receive input from a convolutional neural network. As shown in FIG. 6, the input to the convolutional neural network may be data 602.

The input layer 604 may be followed by one or more convolution and aggregation layers 610. A convolutional layer may include a set of filters that are spatially smaller (e.g., have a smaller width and / or height) than the input (e.g., data 602) to the convolutional layer. Each of the filters can be used to convolve the input of the convolutional layer to generate activation maps (eg, 2-dimensional activation maps) indicating the response of that filter at each spatial location. The convolutional layer can be followed by a collection layer, which downsamples the output of the convolutional layer to reduce its size. The pooling layer may use any of a variety of pooling techniques, such as the largest pooling and / or global average pooling. In some specific examples, down-sampling may be performed by the convolutional layer itself (eg, without a collection layer) using striding.

The convolution and aggregation layer 610 may be followed by a dense layer 612. The dense layer 612 may include one or more layers, each having one or more neurons that receive input from a previous layer (eg, a convolution or pooling layer) and provide output to a subsequent layer (eg, an output layer 608). The dense layer 612 can be described as "dense" because each of the neurons in a given layer can receive input from each neuron in the previous layer and provide output to each neuron in a subsequent layer . The dense layer 612 may be followed by an output layer 608 that provides the output of the convolutional neural network. The output may be, for example, an indication of a category from a set of categories to which the data 602 (or any portion of the data 602) belongs.

It should be understood that the convolutional neural network shown in FIG. 6 is only implemented as one example and other implementations may be used. For example, one or more layers may be added to or removed from the convolutional neural network shown in FIG. 6. Additional example layers that can be added to the convolutional neural network include: a corrective linear unit (ReLU) layer, a cushion layer, a cascade layer, and a superscale layer. The superscale layer can be configured to up-sample the input to that layer. A ReLU layer can be configured to apply a rectifier (sometimes called a ramp function) as a transfer function to the input. The padding layer can be configured to change the size of the input to the layer by filling one or more dimensions of the input. The concatenation layer can be configured to combine multiple inputs (eg, combine inputs from multiple layers) into a single output.

Convolutional neural networks can be used to perform any of the various functions described herein. For example, a convolutional neural network can be used to identify the performing gesture based on the ultrasound data collected when the gesture is performed. It should be understood that more than a single convolutional neural network may be used to perform a function. Further discussion of deep learning technology can be found in the US patent entitled "AUTOMATIC IMAGE ACQUISITION FOR ASSISTING A USER TO OPERATE AN ULTRASOUND DEVICE" filed on June 19, 2017 (and assigned to the assignee of this application) Application No. 15 / 626,423, which is incorporated herein by reference in its entirety. It should be further understood that other machine learning models such as the use of random forests, support vector machines, and linear classifiers may also be used to perform any of the functions described herein.

FIG. 7 shows another example of a wearable device 700 for ultrasound data collection configured to be worn on a user's wrist in accordance with certain specific examples disclosed herein. In Figure 7, the wearable device 700 is shown disassembled. The wearable device 700 can be worn by a user around the user's wrist, and includes a main module 702, an ultrasonic module 704, a coupling element 748, a first wristband 706, and a second wristband 708. It should be understood that as mentioned herein, a "wristband" may be any type of band configured to surround any part or all of the wrist.

The ultrasound module 704 includes an ultrasound chip device 710 and an ultrasound housing element 728. The main module 702 includes a printed circuit board (PCB) 720, a display screen 722, a battery 730, and main casing components 724 and 726. On the PCB 720 are a processing circuit 712, a memory circuit 714, a communication circuit 716, and a power management circuit 718. The first wristband 706 includes a plurality of holes 732 at a first end portion 734 thereof, each of which is located at a different distance from the first end portion 734 of the first wristband 706. The conductor 736 extends through the first wristband 706 and into the main housing elements 724 and 726 to electrically connect the ultrasound module 704 to the PCB 720. The second wristband 708 includes a buckle 738 at a first end portion 742 thereof. The buckle 738 includes a pin 740.

The ultrasonic chip device 710 includes a micromechanical ultrasonic transducer integrated with a semiconductor die including an integrated ultrasonic circuit. In some specific examples, the ultrasonic transducer may be formed on the same wafer as the ultrasonic circuit to form a monolithic ultrasonic device. In other specific examples, some parts of the ultrasonic circuit may be in a semiconductor wafer different from the transducer. The ultrasonic transducer may be a capacitive micromechanical ultrasonic transducer (CMUT). CMUT can be integrated with CMOS (Complementary Metal Oxide Semiconductor) circuits. For example, the CMUT may include a cavity formed in a CMOS wafer, wherein a cavity is overlaid on the film, and in some specific examples, the cavity is sealed. The electrodes may be provided with a self-covering cavity structure to build a transducer unit. The CMOS wafer may include an integrated circuit to which the transducer unit may be connected. The transducer unit and the CMOS wafer can be monolithic integrated, so an integrated ultrasonic transducer unit and integrated circuit are formed on a single substrate (CMOS wafer). A CMUT integrated with a CMOS circuit can be referred to as a CMOS Ultrasonic Transducer (CUT).

Ultrasonic transducers may be configured in a one-dimensional array or a two-dimensional array, and there may be 1024, 2048, 4096, 8192, 16384, or any other suitable number of transducer elements in the array. The transducer can be configured at 50 μm, 100 μm, 130 μm, 200 μm, 250 μm, or any other suitable spacing. The semiconductor chip can be 5 mm × 5 mm, 10 × 5 mm, 1 × 1 cm, 1.5 × 1 cm, 1.5 cm × 1.5 cm, 2 × 1 cm, 2 × 1.5 cm, 2 × 2 cm Or any other suitable size. In some specific examples, the ultrasonic wafer device 710 includes one transducer array having 2048 transducer elements arranged in a 64 × 32 array with a 130 μm pitch on a semiconductor die having a size of 10 × 5 mm. In some specific examples, the ultrasonic wafer device 710 includes one transducer array of 4096 transducer elements arranged in a 64 × 64 array with a 130 μm pitch on a semiconductor die having a size of 1 × 1 cm. The ultrasonic circuit in the ultrasonic chip device 710 may include a transmission circuit for transmitting a signal to a transmission beamformer in the ultrasonic chip device 710, which in turn drives the ultrasonic converter to transmit a pulsed ultrasonic signal to a user Inside the wrist. The pulsed ultrasound signal can be scattered back from structures (such as blood vessels) in the user's wrist to generate an echo back to the transducer. These echoes can then be converted into electrical signals, or ultrasonic data, by the transducer elements, and these electrical signals are received by a receiving circuit in the ultrasonic circuit. The electric signal representing the received echo is transmitted to a receiving beamformer in one of the ultrasonic chip devices 710, and it outputs ultrasonic data in response to the received echo. For a further description of an example of an ultrasonic device and an ultrasonic circuit, see US Patent Application No. 15 / 415,434, entitled "UNIVERSAL ULTRASOUND DEVICE AND RELATED APPARATUS AND METHODS".

In some specific examples, the ultrasonic converter in the ultrasonic chip device 710 may transmit ultrasonic waves having a frequency between approximately 5 MHz and 20 MHz in order to collect ultrasonic data from the wrist. In some specific examples, the ultrasound chip device 710 may transmit signals having frequencies up to approximately 21 MHz, 22 MHz, 23 MHz, 24 MHz, 25 MHz, 26 MHz, 27 MHz, 28 MHz, 29 MHz, 30 MHz,> 30 MHz, or Ultrasound of any suitable frequency. In some specific examples, the ultrasonic chip device 710 may emit ultrasonic waves having frequencies as low as approximately 4 MHz, 3 MHz, 2 MHz, 1 MHz, <1 MHz, or any suitable frequency.

The ultrasound chip device 710 is positioned in the ultrasound module 704 such that its longitudinal axis is parallel to the longitudinal axis of the first wristband 706. In some specific examples, it may be feasible to rotate the ultrasound module 704 to a desired orientation relative to the first wristband 706 before coupling the ultrasound module 704 to the first wristband 706.

The ultrasonic chip device 710 may transmit the collected ultrasonic data to the processing circuit 712 via the conductor 736. The ultrasound module 704 and the PCB 720 are electrically coupled to a conductor 736 extending through the first wristband 706 and into the main module 702. The conductor 736 may be, for example, in a flexible printed circuit board or cable.

The ultrasound housing element 728 and the first wristband 706 surround the ultrasound chip device 710. The ultrasonic housing element 728 has an acoustic lens 746, and the ultrasonic wave can propagate from the ultrasonic chip device 710 through the acoustic lens to the user's wrist. In some specific examples, the acoustic lens 746 is a simple opening in the ultrasonic housing element 728. When the wearable device 700 is assembled, the ultrasonic housing element 728 faces the wrist of the user. In some specific examples, the ultrasonic housing element 728 is a protrusion from the first wristband 706, which forms a cavity containing the ultrasonic wafer device 710.

The coupling element 748 is attached to the surface of the acoustic lens 746 facing the user's wrist. The coupling element 748 is configured to reduce the air gap between the ultrasound module 704 and the user's wrist, and establish an acceptable impedance matching coupling for ultrasound signal transmission and reception. In some specific examples, therefore, the coupling element 748 may be considered as an impedance matching strip, or an impedance matching coupler. Further examples of the coupling element 748 are described in more detail below with reference to FIG. 15.

In the main module 702, the PCB 720 is communicatively coupled to the display screen 722, for example, via internal wires in the main housing elements 724 and 726, and includes processing included in one or more semiconductor wafers on the PCB 720 The circuit 712, the memory circuit 714, the communication circuit 716, and the power management circuit 718. The processing circuit 712 may be configured to perform any of the functionality described herein (eg, processes 100, 200, and 300). The processing circuit 712 may include one or more processors (eg, computer hardware processors) and may be configured to execute one or more processor-executable instructions stored in the memory circuit 714. The memory circuit 714 may be used to store programs and data, and may include one or more storage devices such as non-transitory computer-readable storage media. The processing circuit 712 may control writing data to and reading data from the memory circuit 714 in any suitable manner. The processing circuit 712 is configured to receive ultrasonic data from the ultrasonic chip device 710 and includes a method for reconstructing the ultrasonic data to form an ultrasonic image (which may be a two-dimensional image, or when the ultrasonic chip device 710 includes a two-dimensional array) It can reconstruct the circuit for 3D images). The processing circuit 712 may also be configured to perform calculations based on ultrasonic data and / or ultrasonic images (which may be two-dimensional images, or three-dimensional images when the ultrasonic chip device 710 includes a two-dimensional array) (eg, Anatomical or physiological measurements). The processing circuit 712 may include specially programmed and / or dedicated hardware, such as an application specific integrated circuit (ASIC). For example, the processing circuit 712 may include one or more ASICs specifically designed for machine learning (eg, deep learning). ASICs specifically designed for machine learning can be used, for example, to accelerate the inference phase of neural networks. The processing circuit 712 also includes a control circuit configured to supply control signals transmitted via the conductor 736 to control operations of the ultrasonic chip device 710, such as operations of transmission and reception circuits. The control circuit is also configured to supply control signals to the display screen 722, the circuit on the PCB 720, and the ultrasonic chip device 710 to control its operation. The processing circuit 712 may include a field programmable gate array (FPGA).

The battery 730 is electrically connected to the PCB 720 and the display screen 722 to provide power to the circuits on the PCB 720 and the display screen 722. The battery 730 is also configured to supply power to the ultrasound chip device 710 via the conductor 736. The battery 730 may be any type of battery, such as a button battery (for example, a zinc air battery, type PR48, size A13), a lithium ion battery, or a lithium polymer battery. The battery 730 may be rechargeable. The power management circuit 718 is configured to manage the supply of power from the battery 730 to the PCB 720, the display screen 722, and to the ultrasonic chip device 710. The power management circuit 718 may be responsible for converting one or more input voltages from the battery 730 into a voltage required to perform the operation of the ultrasonic chip device 710, and also be responsible for managing the power consumption in the device ultrasonic chip device 710 in addition. For example, the power management circuit 718 may increase or decrease the input voltage as needed using a charge pump circuit or via some other DC-DC voltage conversion mechanism.

The communication circuit 716 is configured to wirelessly transmit data (eg, ultrasound data, ultrasound images, ultrasound data / image-based calculations) to an external device (such as an external host device), a workstation, or a server. The communication circuit 716 may include BLUETOOTH, ZIGBEE, and / or WiFi wireless communication circuits. In some specific examples, the communication circuit 716 may be configured to transfer data to an external device via a wired connection, such as a SERDES, DDR, USB, or MIPI wired connection.

The main module 702 can be configured as any type of electronic device and can perform functions not related to the collection of ultrasonic data. For example, the main module 702 can be configured as a smart watch or a smart wristband, and the display screen 722 can be configured to display any type of data, including time, email, instant messages, and / or the Internet network. The display screen 722 may be any type of display screen, such as a low power light emitting diode (LED) array, a liquid crystal display (LCD) array, an active matrix organic light emitting diode (AMOLED) display, or a quantum dot display. The display screen 722 may be curved. The main module 702 may include other sensors such as global positioning, gyroscope, magnetometer, accelerometer, barometer, blood alcohol content, glucose content, blood oxygenation volume, microphone, heart rate, ultraviolet, electromyography ( EMG) and skin electrical response sensors, and the display screen 722 can display data from these additional sensors. Some sensors may be inside the main module 702, while others may be integrated on the outer surfaces of the main housing elements 724 and 726. In some specific examples, the display screen 722 may not be present.

In some specific examples, the ultrasound module 704 is configured to wirelessly communicate with the main module 702. In these specific examples, the ultrasonic module 704 may include a wireless communication circuit configured to wirelessly communicate with the communication circuit 716 of the main module 702. The ultrasonic module 704 and the main module 702 can wirelessly transmit the ultrasonic data from the ultrasonic module 704 to the main module 702, and the control signal is from the main module 702 to the ultrasonic module 704. In some specific examples, the ultrasonic module 704 includes a battery, and does not draw power from the battery 730 in the main module 702. In a specific example in which the ultrasonic module 704 is in wireless communication with the main module 702 and has its own battery, the conductor 736 may not be present. In some specific examples, the ultrasonic module 704 can be charged from the main module 702 or the auxiliary charger or can be powered inductively by itself.

In some specific examples, the ultrasound module 704 may include an internal processing circuit 712, a memory circuit 714, a communication circuit 716, and / or a power management circuit 718. A part of the circuit may be integrated with the ultrasonic chip device 710. In these specific examples, the ultrasound module 704 may use circuits inside the ultrasound module 704 to perform image reconstruction and / or data transmission to external devices, and may not communicate with the main module 702. Therefore, the conductor 736 may not exist.

The main housing elements 724 and 726 enclose the PCB 720, the display screen 722, and the battery 730. The display screen 722 is positioned adjacent to the main housing element 724, which includes an opening 744 through which the display screen 722 can be seen. When the wearable device 700 is assembled, the main housing element 724 faces the user's wrist and the main housing element 726 faces away from the user's wrist. The main housing element 726 and the display screen 722 are positioned on a surface of the wearable device 700 opposite to the PCB 720, the battery 730, and the main housing element 724 (ie, the surface facing away from the user's wrist). In some specific examples, the main housing element 724 and 726 may be a single element. For example, a single main housing element may have a hinge so that the ultrasonic housing element can be opened, and the PCB 720, the display screen 722, and the battery 730 can be inserted inside. As another example, a single main housing element may have a PCB 720, a display screen 722, and a battery 730 into one of the slots.

The first wristband 706 is coupled to the first end portion 750 of the main housing element 724 at its second end portion 754. The second wristband 708 is coupled to a second end portion 752 of the main housing element 724 at a second end portion 756 thereof. The first wristband 706 and the second wristband 708 can be configured to be coupled to the main housing element via any coupling member (such as a clamp, a buckle, a screw, an adhesive, a magnetic, a Velcro, an interlocking fit, etc.) 724. In some specific examples, the main housing element 724 may include a plurality of pairs of lugs at each of its first end portion 734 and second end portion 736, wherein a spring lever bridges each pair of lugs, and a first wristband 706 And the second wristband 708 can loop around the spring rods. The first wristband 706 and the second wristband 708 can be made of any material, such as leather, fabric, plastic, and metal. The first wristband 706 and the second wristband 708 may have any shape and may be similar to conventional straps for wristwatches or bracelets.

By inserting the pin 740 into one of the plurality of holes 732, the wearable device 700 can be coupled to a user's wrist. Based on which of the plurality of holes 732 are used, the perimeter of the wearable device 700 can be adjusted so that the wearable device 700 fits around the wrist of the user. In some specific examples, the wearable device 700 may be coupled to a user's wrist using other mechanisms. For example, instead of the plurality of holes 732 and buckles 738, the first wristband 706 and the second wristband 708 may include a clamp, a buckle, a Velcro, a magnet, or an interlocking fit. In some specific examples, the wearable device 700 includes only one wristband, or more than two wristbands.

The ultrasound module 704 is configured to attach to the first wristband 706. In some specific examples, the ultrasound module 704 is attached to the first wristband 706 at a location that is not intended to move. For example, the ultrasonic module 704 can be positioned at a specific position on the first wristband 706, so that when the wearable device 700 is worn, the ultrasonic module 704 is positioned at a specific area of the user's wrist For example, one area of the gesture can be identified based on the ultrasound data). The ultrasound module 704 can be configured to attach to the first wristband 706 via any coupling member. For example, the ultrasonic module 704 may be attached to the first wristband 706 via the ultrasonic module and a complementary Velcro, magnet, or buckle on the first wristband 706. In some specific examples, the wearable device 700 is configured so that the position of the ultrasound module 704 on the first wristband 706 can be changed. In some specific examples, the first wristband 706 may include a plurality of discrete coupling points (eg, discrete magnets, discrete Velcro elements, discrete snap locations) along its length. In other specific examples, the first wristband 706 has a continuous coupling region along one of its length (eg, a continuous length of a magnetic material or a continuous length of a Velcro material). In some specific examples, the ultrasonic module 704 may include a clamp for clamping the ultrasonic module 704 to the first wristband 706. In other specific examples, the first wristband 706 may have a cavity into which the ultrasound chip device 710 is placed. In still other specific examples, the first wristband 706 includes a plurality of holes and the ultrasonic module 704 includes a pin, and the ultrasonic module 704 can be coupled by inserting the pin into one of the plurality of holes. To the first wristband 706.

In some specific examples, the main module 702 may not exist, and the PCB 720, the processing circuit 712, the memory circuit 714, the communication circuit 716, the power management circuit 718, and the battery 730 may be included in the ultrasonic module 704. In these specific examples, the first wristband 706 and the second wristband 708 may be a single continuous wristband.

FIG. 8 shows another example of a wearable device 800 for ultrasound data collection configured to be worn on a user's wrist according to some specific examples disclosed herein. The wearable device 800 can be worn by a user around the user's wrist. In Figure 8, the wearable device 800 is shown disassembled. The following description discusses the differences between the wearable device 800 and the wearable device 700.

The wearable device 800 lacks an ultrasound module 704. The ultrasonic chip device 710 is located in the main module 702. The main housing element 724 includes an acoustic lens 746, and the first wristband 706 lacks an internal conductor that interfaces with the ultrasonic module. The coupling element 748 is coupled to the user-facing surface of the main housing element 724. Depending on the manner in which the main module 702 is worn (for example, whether the main module 702 is worn on the dorsal or palm wrist), the ultrasonic chip device 710 can collect ultrasonic data from various muscles.

FIG. 9 shows another example of a wearable device 900 for ultrasound data collection configured to be worn on a user's wrist according to some specific examples disclosed herein. The wearable device 900 is configured as a wristband, and the user can physically couple the wristband to the wristband of his or her personal smart watch module and electrically connect it to the smart watch module. In some specific examples, the wearable device 900 can be configured as an interchangeable wristband, and the user can directly couple (physically and electrically) the interchangeable wristband to his or her personal smart watch. Module to replace the original wristband of the smartwatch. In Figure 9, the wearable device 900 is shown disassembled. The following description discusses the differences between the wearable device 900 and the wearable device 700.

The wearable device 900 lacks the main module 702. The ultrasound module 704 includes a printed circuit board (PCB) 920. On the PCB 920 are a processing circuit 712 and a memory circuit 714. Compared with the wearable device 800, the ultrasonic module 904 has an internal processing circuit 912 and a memory circuit 914, because the smart watch to which the wearable device 900 is intended to be coupled may not have the ability to interface with the ultrasonic chip device 710 and Processing of ultrasonic data and memory circuit.

The first wristband 706 includes a conductor 736 extending through the first wristband 706, which is connected to the wiring cable 958 at the second end portion 754 of the first wristband 706. The wiring cable 958 exits from the first wristband 706 through the opening 960 in the first wristband 706, and has a male connector 962 configured to connect to a complementary notch on the user's personal smartwatch . An example of plugging the male connector 962 into the smart watch will be further illustrated in FIG. 12. In some specific examples, the wearable device 900 may include a plate configured to be screwed into a user's personal smart watch at a complementary recess, and prevent the use of the male connector 962 in the wearable device 900 During the removal from the notch on the smart watch.

The conductor 736 and the connecting cable 958 electrically connect the ultrasonic module 704 to the user's smart watch. Therefore, the ultrasonic module 704 can be used as a component in a user's smart watch, and the ultrasonic module 704 itself need not include such components. For example, in FIG. 9, the ultrasound module 704 is configured to draw power from the battery of the smart watch to power the ultrasound chip device 710 and the circuit on the PCB 920. In addition, the ultrasonic module 704 is configured to transmit data (for example, ultrasonic data, ultrasonic images, calculations based on ultrasonic images) to the communication circuit in the smart watch via the conductor 736 and the connection cable 958 for For wireless transmission to external devices, such as external host devices, workstations, or servers. The user's personal smart watch may execute a mobile application ("app") configured to interface with the ultrasound module 704. The wiring cable 958 may be any type of wiring cable, such as a Lightning connector or a micro USB connector.

The wearable device 900 may be configured to be coupled to the longitudinal axis of the wristband of the smart watch along its longitudinal axis. The wearable device 900 may use any coupling member to couple to the wristband of the user's personal smart watch. For example, the first wristband 706 may include a pin configured to be inserted into a hole in a wristband of a user's smart watch. As another example, the wearable device 900 may be coupled to a wrist watch of a user's smart watch with screws, Velcro, adhesive, buckles, slots and grooves, or one or more magnets. In the specific example where the wearable device 900 is directly coupled to the smart watch module, instead of the wristband of the smart watch, the wearable device 900 can be configured to pass through Any coupling member such as Velcro, interlocking fitting, is coupled to the smart watch module.

In some specific examples, the ultrasound module 704 has an internal battery and is configured to not draw the battery from the user's smart watch. In some specific examples, the ultrasonic module 704 has a communication circuit inside the ultrasonic module 704 and is configured to not use the communication circuit in a user's smart watch. In some specific examples, the ultrasonic module 704 can transmit the ultrasonic data collected by the ultrasonic chip device 710 to the processing circuit in the user's smart watch, and can receive control from the user's smart watch. The control signal of the circuit is configured to reconstruct the ultrasonic data to form an ultrasonic image (which may be a two-dimensional image or a three-dimensional image when the ultrasonic chip device 710 includes a two-dimensional array). For example, an application on a user's smart watch may include instructions for processing a circuit to reconstruct ultrasonic data to form an ultrasonic image, and instructions for controlling a circuit to output a control signal for the ultrasonic chip device 710. . In these specific examples, the ultrasound module 704 may lack a processing circuit 712 and / or a memory circuit 714.

FIG. 10 shows another example of a wearable device 1000 for ultrasound data collection configured to be worn on a user's wrist according to some specific examples disclosed herein. The wearable device 1000 is configured as a wristband, and the user can physically couple the wristband to the wristband of his or her personal wrist device, which can be a standard analog watch module, a standard digital watch Modular or smart watch. In some specific examples, the wearable device 1000 may be configured as an interchangeable wristband, and the user may couple (physically and electrically) the interchangeable wristband to his or her personal wrist device, thereby Replace the original wrist strap of the wrist unit. In Fig. 10, the wearable device 1000 is shown disassembled. The following description discusses the differences between the wearable device 1000 and the wearable device 900.

The ultrasound module 704 includes a printed circuit board (PCB) 1020 and a battery 1030. On the PCB 1020 are a processing circuit 712, a memory circuit 714, a communication circuit 716, and a power management circuit 718. Therefore, in contrast to the wearable device 900, the ultrasound module 704 does not need to use components (for example, communication circuits, batteries) outside the ultrasound module 704 (for example, the individual of the user to which the wearable device 1000 is coupled Smart watch), because these components are already included in the ultrasound module 704. Therefore, the first wristband 706 lacks communication components (for example, a conductor inside the first wristband 706 and a wiring cable extending from the first wristband 706) to interface with a user's personal watch. The battery 1030 may be any type of battery, such as a button battery (for example, a zinc air battery, type PR48, size A13), a lithium ion battery, or a lithium polymer battery. The battery 1030 may be rechargeable. An example of coupling a wearable device 1000 to a wristband of a user's wrist device will be further illustrated in FIGS. 11A to 11G.

11A to 11G show an example of a wearable device for ultrasonic data collection, the wearable device being configured to be coupled to a user's wrist when the wearable device is assembled and worn. FIG. 11A shows the assembled wearable device 1000 and a user's personal wrist device 1100 (before coupling the wearable device 1000 to the wrist device 1100). The wearable device 1000 includes an ultrasonic module 704, a first wristband 706, an ultrasonic shell module 728, and an acoustic lens 746. The coupling element 748 is not shown in FIG. 11A. The wrist device 1100 includes a main module 1102, a first wristband 1106, and a second wristband 1108. FIG. 11B shows a wearable device 1000 coupled to a wrist device 1100. In detail, the first wristband 706 is coupled to the first wristband 1106 along its longitudinal axis. The wearable device 1000 is oriented so that the ultrasonic module 704 is away from the main module 1102 of the wrist device 1100. FIG. 11B also shows the sensor 1104 integrated on the surface of the main module 1102. For example, the sensor may be a blood alcohol content, a glucose content, a blood oxygenation amount, a microphone, a heart rate, an ultraviolet ray, an electromyography (EMG), and / or a skin electrical response sensor. 11C and 11D show a wearable device 1000 coupled to a wrist device 1100 when worn. FIG. 11C shows the dorsal wrist and FIG. 11D shows the palm wrist. The wearable device 1000 is oriented in the orientation of FIG. 11B, that is, the wearable device 1000 is oriented so that the ultrasonic module 704 is away from the main module 1102 of the wrist device 1100. FIG. 11E shows a side view of a wearable device 1000 coupled to the wrist device 1100 in the orientation of FIG. 11B, ie, the wearable device 1000 is oriented so that the ultrasonic module 704 is away from the main module 1102 of the wrist device 1100. FIG. 11F shows a wearable device 1000 coupled to a wrist device 1100 in an orientation different from that in FIG. 11B. In detail, the wearable device 1000 is coupled to the first wristband 1106 so that the ultrasonic module 704 is closest to the main module 1102 of the wrist device 1100. FIG. 11G shows a wearable device 1000 coupled to a wrist device 1100 when worn. The wearable device 1000 is oriented in the orientation of FIG. 11F, that is, the wearable device 1000 is oriented so that the ultrasonic module 704 is closest to the main module 1102 of the wrist device 1100.

FIG. 12 shows an example of a wearable device 900 when electrically coupled to a user's personal wrist device 1100. The wrist device 1100 further includes a recess 1212. The wearable device 900 includes a wiring cable 958 exiting from the first wristband 706 via an opening 960 in the first wristband 706. The connection cable 958 has a male connector 962 that is inserted into a complementary recess 1212 in the wrist device 1100. In some specific examples, as an alternative or supplement to the male connector 962, the wiring cable 958 includes a female connector, and the wrist device 1100 includes a female connector inserted into a male port (as an alternative or supplement to the female port 1212). In some specific examples, the clip on the wristband of the wrist device 1100 has a pin to which a connector (male or female) on the connection cable 958 can be electrically coupled.

In some specific examples, instead of or in addition to using a wristband to couple the ultrasound chip device to a user's wrist, other components such as adhesives or clamps may be used.

FIG. 13 shows another example of a wearable device 1300 for ultrasound data collection configured to be worn on a user's wrist according to some specific examples disclosed herein. The wearable device 1300 differs from the wearable device 700 in that the ultrasonic module 704 in the wearable device 1300 is coupled to the side of the first wristband 706 opposite to the ultrasonic module 704 in the wearable device 700. In other words, when the user wears the wearable device 900 on his / her wrist, the ultrasonic module 704 is coupled to the surface of the first wristband 706 that does not face the wrist of the user. The opening 746 in the ultrasonic housing element 728 faces away from the wrist of the user, and the ultrasonic transducer on the ultrasonic chip 710 is oriented to face the opening 746 and is configured to transmit ultrasonic signals through the opening 746.

Because the ultrasound transducer is facing away from the wrist of the user, the ultrasound module 704 is configured to transmit ultrasound signals away from the wrist of the user on which the wearable device 1300 resides. In addition to the wrist on which the wearable device 1300 resides, the wearable device 1300 can therefore also collect ultrasound data from other parts of the user's body while other parts of the user's body are performing a posture. For example, if the wearable device 1300 resides on the user's left wrist, the wearable device 1300 can use airborne ultrasound from the user's right hand, right finger, right wrist, right arm, left hand, left finger, and / or Collect information on the left arm. When collecting ultrasound data from the upper limb opposite to the limb on which the wearable device 1300 resides, the user can point the ultrasound transducer on the wearable device 1300 to the opposite limb. When collecting ultrasound data from the same limb as the limb on which the wearable device 1300 resides, the user can wear the wearable device 1300 on the back wrist or the palm wrist. In these specific examples, when the wearable device 1300 resides on the back-side wrist or palm-side wrist, certain gestures can be more easily detected. For example, when the wearable device 1300 resides on the back side wrist, the posture of the hand or finger bent toward the back side wrist can be more easily detected, and when the wearable device 1300 resides on the palm side wrist, the hand or finger The posture of the wrist bent toward the palm side can be more easily detected. The ability of the ultrasonic module 704 to collect ultrasonic data from other parts of the user's body may depend on the configuration of the ultrasonic module 704 to transmit ultrasonic energy, which has the ability to allow the ultrasonic energy to pass through without undue attenuation. Frequency at which air travels to other parts of the user's body (eg, in the kHz range). Because the ultrasonic signal travels through the air, the coupling element 748 is unnecessary.

FIG. 14 shows another example of a wearable device 1400 configured for ultrasound data collection configured to be worn on a user's wrist according to some specific examples disclosed herein. The difference between the wearable device 1400 and the wearable device 800 is that the ultrasonic chip 710 is located on the side of the first wristband 706 and the second wristband 708 opposite to the ultrasonic chip 710 in the wearable device 800. In other words, the ultrasonic chip 710 in the wearable device 1400 is located closest to the main housing element 726, rather than the main housing element 724. In addition, the wearable device 1400 differs from the wearable device 800 in that the main housing element 726 includes an opening 746 (instead of the main housing element 724 including an opening 746), and the display screen 722 and the opening 744 are missing. In addition, the ultrasonic transducer on the ultrasonic chip 710 faces the opening 746 and is configured to transmit ultrasonic signals through the opening 746. As with the wearable device 1300, because the ultrasound transducer faces away from the user's wrist in the wearable device 1400, the ultrasound module 704 is configured to be remote from the wrist of the user on which the wearable device 1400 resides Transmission of ultrasonic signals. In addition to the wrist on which the wearable device 900 resides, the wearable device 1400 may therefore be able to collect ultrasound data from other parts of the user's body while other parts of the user's body are performing a posture. For example, if the wearable device 1400 resides on the user's left wrist, the wearable device 1400 can use airborne ultrasound from the user's right hand, right finger, right wrist, right arm, left hand, left finger, and / or Collect information on the left arm. When collecting ultrasound data from the upper limb opposite to the limb on which the wearable device 1400 resides, the user can point the ultrasound transducer on the wearable device 1400 toward the opposite limb. When collecting ultrasound data from the same limb as the limb on which the wearable device 1400 resides, the user can wear the wearable device 1400 on the back wrist or palm wrist. In these specific examples, when the wearable device 1400 resides on the back-side wrist or palm-side wrist, certain gestures can be more easily detected. For example, when the wearable device 1400 resides on the back-side wrist, the posture of the hand or finger bent toward the back-side wrist can be more easily detected, and when the wearable device 1400 resides on the palm-side wrist, the hand or finger Bending the wrist towards the palm is easier to detect.

It should be understood that by increasing the length of the wristband in the wearable device discussed herein, the wearable device can be configured to incorporate into other parts of the upper limb, such as the forearm. Gathering information from one specific part of the upper limb, but not others, can be helpful because, depending on the muscles located at the different parts of the upper limb, it is easier to use ultrasound data collected from one specific part of the upper limb rather than the other And / or it may be possible to more accurately identify the gesture being performed.

It should be understood that although the wearable device discussed above may include an ultrasonic chip device with a two-dimensional array of ultrasonic transducers, the wearable device may not need to analyze the data collected from each ultrasonic transducer in the array to Identify a gesture being performed. In some specific examples, the wearable device may use data from a subset of the ultrasound transducers in the array to identify a performing gesture. It should also be understood that the wearable device may not need to analyze the ultrasound image (ie, the image reconstructed from the ultrasound data) to identify a performing gesture. In some specific examples, the wearable device may use raw acoustic data or raw acoustic data converted into non-image form to identify a gesture being performed. In other specific examples, the wearable device can convert the collected ultrasonic data into an ultrasonic image, and use the ultrasonic image to identify a gesture being performed.

In some specific examples, the wearable device may monitor posture changes using low-power technology. For example, the wearable device may continuously collect low-frequency broadband ultrasound data (eg, data from a subset of ultrasound transducers in an array). Low-frequency ultrasound data may be sufficient to detect muscle movement that occurs while still operating in an acceptable low power state. Once muscle movement is detected, high-frequency ultrasound data (eg, data collected from all ultrasound transducers in the array) can be collected and used to identify the posture being performed. As another example, other sensors (eg, an electromyography sensor in a wearable device) may be used to continuously monitor muscle movement while still operating in an acceptable low power state. Once a muscle movement is detected, high-frequency ultrasonic data (eg, data collected from all ultrasonic transducers in the array) can be collected and used to identify the posture being performed. In this example, ultrasonic data may not be collected until muscle movement is detected by other sensors. As another example, the wearable device may collect ultrasonic data in the air from the upper limb on which the wearable device resides (eg, using a subset of ultrasonic transducers in an outward-facing array). An ultrasonic beam pattern can cover all areas of interest (eg, hands). Once movement of the upper limb is detected, high-frequency ultrasound data (for example, data collected from all ultrasound transducers in the array) can be collected and used to identify the posture being performed.

As discussed above, in some specific examples, the ultrasound module may include a two-dimensional array of ultrasound transducers. In these specific examples, it may be feasible to use beamforming to focus the ultrasound signal along a particular direction, such as parallel to the longitudinal axis of the upper limb, perpendicular to the longitudinal axis of the upper limb, or diagonally to the longitudinal axis of the upper limb. In addition, changing the frequency of the ultrasonic signal makes it possible for the ultrasonic module to collect data from a structure located at a specific depth within the user's wrist. Muscles at a certain depth within the user's wrist can show more distinguishable changes for a given set of ultrasound data / images, and the frequency can therefore be selected to select the optimal depth.

As discussed above, the coupling element 748 is configured to reduce the air gap between the ultrasound module 704 and a user's wrist. In detail, the coupling element 748 is configured to be coupled to the acoustic lens 746 and establish an acceptable impedance matching coupling for ultrasonic signal transmission and reception. In some specific examples, therefore, the coupling element 748 may be considered as an impedance matching strip, or an impedance matching coupler. To reduce the air gap between the ultrasonic module 704 and the wrist of the user, the coupling element 748 may be configured to be flexible, so that the coupling element 748 fits the irregular surface of the user's wrist.

In some specific examples, the coupling element 748 includes a solid material and a liquid absorbed in the solid material to increase the flexibility of the coupling element 748. In some specific examples, the liquid includes a hydrophilic solution. In these specific examples, the coupling element 748 can be configured to be renewed by adding water to the coupling element 748 to reduce the drying of the coupling element and maintain an acceptable attachment of the coupling element to the user's wrist合 性。 Combined. For example, the coupling element 748 can be re-coupled with the water in the shower head by immersing the coupling element 748 in water or by allowing water to flow over the coupling element 748. In some specific examples, the coupling element 748 includes a porous sponge that stores water and slowly releases water, and can be renewed by adding water to the porous sponge. In some specific examples, the liquid includes a hydrophobic solution. In these specific examples, the coupling element 748 is configured to be renewed by oil, gel, or another hydrophobic consumable mouth to reduce the drying of the coupling element 748 and maintain the coupling element 748 and the user's wrist Acceptable fit. In some specific examples, the wearable device (specifically, the ultrasonic module 704 and the main module 702) is configured to be waterproof, so that, for example, when the coupling element 748 is renewed, the ultrasonic module 704 and the main module When the module 702 becomes wet, the ultrasonic module 704 and the main module 702 continue to function. For example, the ultrasonic casing element 728 and the main casing elements 724 and 726 may be waterproof casings.

In some specific examples, the coupling element 748 is configured as replaceable. For example, the coupling element 748 may include an adhesive layer between the coupling element 748 and the surface of the ultrasonic module 704, and to replace the coupling element 748, the user may peel the coupling element from the ultrasonic module 704 748 and attach another coupling element 748 to the ultrasonic module 704.

Materials used in the coupling element 748 may include a rubber material (which absorbs water), a rubberized coating material, a polysiloxane-based material, a gel-based material (for example, the coupling element 748 may include a gel pad ), Agar-based materials and room temperature vulcanized polysiloxane materials. In some specific examples, the coupling element 748 includes a rubber silicone material that is sufficiently flexible to maintain acceptable contact with the user's wrist without replacement. In some specific examples, the coupling element 748 may include a sponge-like material that is capable of absorbing liquid and water (for example, by spraying the coupling element 748 with water, by dipping the coupling element 748 in water, by showering Or take a bath and / or clean the coupling element 748 with water) to maintain the fit of the coupling element 748 to the user's wrist. In these specific examples, the sponge-like material can release the absorbed liquid at an acceptable low rate, so that the coupling element 748 needs to be renewed at an acceptable low frequency.

In some specific examples, the ultrasonic module 704 lacks a coupling element, and the user can wet the wrist area before data collection (for example, by immersing the wrist in water or allowing water to flow on the first) to Establish proper impedance matching couplings for ultrasonic signal transmission and reception. Therefore, the ultrasound module 704 can operate the ultrasound without gel. In these specific examples, the ultrasound module is configured to be waterproof. For example, the ultrasonic housing element 728 may be a waterproof housing.

FIG. 15 shows an example of an ultrasonic module 704 including a reservoir for re-coupling the element 748 according to some specific examples described herein. In FIG. 15, the ultrasonic module 704 includes an ultrasonic chip device 710, reservoirs 1502 and 1504, and a cover 1506. The reservoir 1502 includes a valve 1508 and a door 1510. The reservoir 1504 includes a valve 1512 and a door 1514. The ultrasonic housing element 728 includes openings 1516 and 1518.

The ultrasound housing element 728 and the first wristband 706 enclose the reservoirs 1502 and 1504, the ultrasound chip device 710, and the cover 1506. The hollow cover 1506 covers the ultrasonic wafer device 710 and forms a housing for the ultrasonic wafer device 710 together with the ultrasonic housing element 728. The coupling element 748 is attached to a surface of the ultrasonic housing element 728.

Valve 1508 opens to opening 1516 and valve 1512 opens to opening 1518. The reservoirs 1502 and 1504 contain a liquid or a gel. The valve 1508 is configured to release liquid or gel from the reservoir 1502 via the opening 1516 and into the coupling element 748. The valve 1512 is configured to release liquid or gel from the reservoir 1502 via the opening 1518 and into the coupling element 748.

The liquids or gels in the reservoirs 1502 and 1504 may be hydrophilic or hydrophobic. As discussed above, the reservoirs 1502 and 1504 are configured to re-couple the element 748 with a liquid or gel. In detail, the reservoirs 1502 and 1504 are configured to add a liquid or gel to the coupling element 748, which can absorb the liquid or gel. Adding a liquid or gel to the coupling element 748 can help reduce the drying of the coupling element 748 and maintain an acceptable fit of the coupling element 748 to the user's wrist.

Valves 1508 and 1512 can be activated mechanically or electrically. In some specific examples, the user may trigger the valves 1508 and 1512 to release liquid or gel from the reservoirs 1502 and 1504 into the coupling element 748. In some specific examples, the user may directly apply mechanical pressure to the ultrasonic module 704 or by applying mechanical pressure to another component (for example, the first wristband 706) coupled to the ultrasonic module 704 ) To apply mechanical pressure to the ultrasonic module 704, and the mechanical pressure may trigger the valves 1502 and 1512 to release at least a portion of the liquid or gel from the reservoirs 1502 and 1504 into the coupling element 748. For example, the mechanical pressure applied to the ultrasound module 704 can compress the reservoirs 1502 and 1504 and cause them to drain liquid or gel through the valves 1508 and 1512. In some specific examples, a user may apply mechanical pressure to a recess in the first wristband 706. In some specific examples, a user may place his or her fingers on sensors on the first wristband 706, and the sensors may transmit electrical signals to the valves 1508 and 1512 for self-collection The devices 1502 and 1504 release liquid or gel into the coupling element 748. In some specific examples, the user can activate a button (eg, a mechanical button or a virtual button), and activation of the button can transmit electrical signals to the valves 1508 and 1512 to release liquid or gel from the reservoirs 1502 and 1504 to Coupling element 748.

In some specific examples, the processing circuit may be configured to automatically trigger the valves 1508 and 1512 to release liquid or gel from the reservoirs 1502 and 1504 into the coupling element 748. The processing circuit may be a processing circuit 712 or a processing circuit in an external host device (eg, a smartphone, tablet device or laptop), a workstation, or a server. For example, the processing circuit may be configured to trigger the valves 1508 and 1512 to periodically release liquid or gel from the reservoirs 1502 and 1504 into the coupling element 748. In some specific examples, the processing circuit may be configured to trigger the valves 1508 and 1512 based on the detection that the coupling element 748 needs to be renewed with a liquid or gel to release the liquid or gel from the reservoirs 1502 and 1504 to Coupling element 748. In some specific examples, detecting that the coupling element 748 requires a new liquid or gel includes determining whether the current amount of the liquid or gel associated with the coupling element 748 is below a critical limit. In some specific examples, in order to detect that the coupling element 748 needs to be renewed with a liquid or gel 424, the processing circuit may be configured to analyze (continuously or periodically) the ultrasonic data collected by the ultrasonic chip device 710 It is determined whether the collected ultrasonic data exhibits a sign that the coupling element 748 is poorly attached to the user's wrist (eg, reduced image quality). In some specific examples, in order to detect that the coupling element 748 needs to be renewed with a liquid or a gel, the processing circuit may be configured to receive a signal from a moisture sensor in or adjacent to the coupling element 748. A signal indicating that the moisture content in or adjacent to the coupling element 748 is below a threshold moisture content. In some specific examples, the processing circuit may be configured to use other sensors, such as a capacitive sensor or a skin conductivity sensor, to detect that the coupling element 748 needs to be renewed. In some specific examples, the processing circuit may detect that the coupling element 748 needs to be renewed with a liquid or gel and generate a notification that the user needs to re-couple the element 748 with a liquid or gel. In some specific examples, the notification may be displayed on the display screen 722. In some specific examples, the notification displayed on the display screen 722 may include text, images, and / or video. In some specific examples, the notification may include audio output from the main module 702.

The door 1510 may be opened to expose the internal cavity of the reservoir 1502 and enable the reservoir 1502 to be refilled with liquid or gel. The door 1514 may be opened to expose the internal cavity of the reservoir 1504 and enable the reservoir 1504 to be refilled with liquid or gel. To refill the reservoirs 1502 and 1504, the user can remove the ultrasonic housing element 728 from the first wristband 706, thereby revealing the reservoirs 1502 and 1504. The user can open the doors 1510 and 1514 and then flow the liquid or gel over the reservoirs 1502 and 1504, soak the reservoirs 1502 and 1504 into the liquid or gel, or shower to add liquid to the reservoir器 1502 and 1504. The cover 1506 protects the ultrasonic wafer device 710 from damage during the refilling process, and the cover forms a case for the ultrasonic wafer device 710 and is waterproof. In some specific examples, the reservoirs 1502 and 1504 may be removable to allow a user to refill the reservoirs 1502 and 1504 without the risk of damaging the ultrasound chip device 710. In some specific examples, the gate 1510 may be any type of input port.

In some specific examples, the reservoirs 1502 and 1504 may be coupled together as a single part, and / or may be connected together such that the reservoirs 1502 and 1504 form a single reservoir. In some specific examples, one of the reservoirs 1502 and 1504 does not exist, or there may be more than two reservoirs. In some specific examples, a tube may connect the reservoirs 1502 and 1504 to the coupling element 748. In these specific examples, the reservoirs 1502 and 1504 may not be located adjacent to the coupling element 748. The reservoirs 1502 and 1504 in the ultrasonic chip device 710 located in the main module 702 may also be located in the main module 702. In specific examples, in some specific examples, the cover 1506 may not exist. In some specific examples, other components, such as valves, for refilling the reservoirs 1502 and 1504 may be included.

Other specific examples of a reservoir for re-coupling the element 748 with a liquid or gel are feasible, such as a valveless reservoir. For example, in some embodiments, the reservoir includes a sponge-like amorphous surface from which the gel can be extruded. In some specific examples, the reservoir includes a sponge-like material coupled to the coupling element 748 via a restriction, so that the reservoir can slowly release liquid or gel to re-couple the element 748.

The various aspects of the present invention can be used individually, in combination, or in various configurations specifically discussed in specific examples not described in the foregoing, and therefore their application is not limited to those illustrated in the foregoing description or illustrated in the drawings Details and configuration of the components. For example, the aspects described in one specific example can be combined with the aspects described in other specific examples in any manner.

The following exemplary specific examples are presented to further illustrate the principles of disclosure. These examples are illustrative and non-limiting.

Example 1 is directed to a system for training a wearable device to perform posture recognition, including: a memory circuit; and a processing circuit in communication with the memory circuit, the processing circuit is configured to: from the wearable device Obtaining ultrasonic data corresponding to an anatomical posture; obtaining non-ultrasonic data corresponding to the anatomical posture; and training a machine learning model accessed by the wearable device by correlating the non-ultrasonic data with the ultrasonic data Recognize the anatomical posture.

Example 2 is directed to the system of Example 1, wherein the wearable device includes one of a wristwatch or a wristband configured with an ultrasonic chip.

Example 3 is directed to the system of Example 2, wherein the wearable device further includes one or more position sensors.

Example 4 is directed to the system of Example 3, wherein the one or more position sensors include one or more accelerometers, gyroscopes, magnetometers, compasses, and Global Positioning System (GPS).

Example 5 is directed to the system of Example 3, wherein the processing circuit is further configured to obtain the non-ultrasonic data using the one or more position sensors.

Example 6 is directed to the system of Example 1, wherein the processing circuit is configured to obtain the non-ultrasonic data using an image capture device, and the image capture device further includes one of a camera, a smartphone, or a tablet device .

Example 7 is directed to the system of Example 1, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Example 8 is directed to a method of training a wearable device to perform posture recognition, including: obtaining ultrasonic data corresponding to an anatomical posture through the wearable device; obtaining non-ultrasonic data corresponding to the anatomical posture; and training A machine learning model accessible by the wearable device to identify the anatomical posture based on correlating the non-ultrasonic data with the ultrasonic data.

Example 9 is directed to the method of Example 8, wherein the wearable device includes one of a wristwatch or a wristband.

Example 10 is directed to the method of Example 9, wherein the wearable device further includes an ultrasonic chip device.

Example 11 is directed to the method of Example 10, wherein the wearable device further comprises one or more position sensors.

Example 12 is directed to the method of Example 11, and further includes obtaining the non-ultrasonic data using the one or more position sensors.

Example 13 is directed to the system of Example 12, wherein the one or more position sensors include one or more accelerometers, gyroscopes, magnetometers, compasses, and Global Positioning System (GPS).

Example 14 is directed to the method of Example 8, further comprising obtaining the non-ultrasonic data using an image capture device.

Example 15 is directed to the method of Example 14, wherein the image capturing device includes one of a camera, a smartphone, or a tablet device.

Example 16 is directed to the method of Example 8, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Example 17 is directed to a non-transitory computer-readable storage medium that stores processor-executable instructions that, when executed by at least one processor, cause the at least one processor to: from a wearable device Obtaining ultrasonic data corresponding to an anatomical posture; obtaining non-ultrasonic data corresponding to the anatomical posture; and training a machine learning model accessed by the wearable device based on the non-ultrasonic data and the ultrasonic data Correlation to identify the anatomical posture.

Example 18 is directed to the non-transitory computer-readable storage medium of Example 17, wherein the wearable device includes one of a wristwatch or a wristband.

Example 19 is directed to the non-transitory computer-readable storage medium of Example 18, wherein the wearable device further includes an ultrasonic chip device.

Example 20 is directed to the non-transitory computer-readable storage medium of Example 19, wherein the wearable device further includes one or more position sensors.

Example 21 is directed to the non-transitory computer-readable storage medium of Example 20, wherein the one or more position sensors include one or more accelerometers, gyroscopes, magnetometers, compasses, and Global Positioning System (GPS).

Example 22 is directed to the non-transitory computer-readable storage medium of Example 20, and further stores the non-ultrasonic data obtained by the at least one processor using the one or more position sensors when executed by the at least one processor. The processor can execute instructions.

Example 23 is a non-transitory computer-readable storage medium for Example 17, further storing processor-executable instructions that when executed by the at least one processor causes the at least one processor to use an image capture device to obtain the non-ultrasonic data .

Example 24 is directed to the non-transitory computer-readable storage medium of Example 23, wherein the image capture device includes one of a camera, a smartphone, or a tablet computer device.

Example 25 is directed to the non-transitory computer-readable storage medium of Example 17, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Example 26 is directed to a device for interfacing with a computing device, including: a wearable device configured to: identify an anatomical posture using ultrasonic data obtained by the wearable device; and base the computing device on The anatomical gesture recognized by the wearable device performs a specific function.

Example 27 is directed to the device of Example 26, wherein the computing device includes one of a smartphone, a tablet device, a computer, a virtual reality system, or the wearable device itself, and wherein the wearable device includes One of a wristwatch or a wristband.

Example 28 is directed to the device of Example 26, wherein the wearable device further comprises an ultrasonic chip device.

Example 29 is directed to the device of Example 28, wherein the wearable device further comprises one or more ultrasonic transducers acoustically coupled thereto on which an anatomical feature of the wearable device resides.

Example 30 is for the device of Example 29, wherein the one or more ultrasonic transducers are configured to operate at a frequency range selected to obtain the ultrasonic data from within the anatomical feature.

Example 31 is the device of Example 29, wherein the wearable device further includes an ultrasonic gel pad configured to be placed on the one or more ultrasonic transducers and the wearable device on which the wearable device resides. Between anatomical features.

Example 32 is the device of Example 28, wherein the wearable device further comprises one or more ultrasonic transducers disposed in an outward direction with respect to a first anatomical feature of the wearable device.

Example 33 is for the device of Example 32, wherein the one or more ultrasonic transducers are configured to obtain the ultrasonic data from a second anatomical feature selected to have no wearable device resident thereon. Operates in a frequency range.

Example 34 is the device of Example 32, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Example 35 is directed to at least one non-transitory computer-readable storage medium that stores processor-executable instructions that, when executed by at least one processor, cause the at least one processor to: The wearable device uses ultrasonic data obtained by the wearable device to identify an anatomical posture; and causes a computing device to perform a specific function based on the anatomical posture recognized by the wearable device.

Example 36 is directed to the media of Example 35, wherein the computing device includes one of a smartphone, a tablet device, a computer, a virtual reality system, or the wearable device itself.

Example 37 is directed to the media of Example 35, wherein the wearable device includes one of a wristwatch or a wristband.

Example 38 is directed to the media of Example 37, wherein the wearable device further includes an ultrasonic chip device.

Example 39 is directed to the media of Example 38, wherein the wearable device further comprises one or more ultrasonic transducers acoustically coupled thereto on which an anatomical feature of the wearable device resides.

Example 40 is directed to the media of example 39, wherein the one or more ultrasonic transducers are configured to operate at a frequency range selected to obtain the ultrasonic data from within the anatomical feature.

Example 41 is directed to the media of Example 39, wherein the wearable device further comprises an ultrasonic gel pad configured to be placed on the one or more ultrasonic transducers and the wearable device on which the wearable device resides. Between anatomical features.

Example 42 is directed to the media of Example 38, wherein the wearable device further comprises one or more ultrasonic transducers disposed in an outward direction with respect to a first anatomical feature of the wearable device.

Example 43 is for the media of example 42, wherein the one or more ultrasonic transducers are configured to obtain the ultrasonic data from a second anatomical feature selected to have no one of the wearable devices resident thereon. Operates in a frequency range.

Example 44 is directed to the media of Example 35, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Example 45 is directed to a method for interfacing with a computing device, including: identifying an anatomical posture by using a wearable device using ultrasound data obtained by the wearable device; and causing the computing device to be based on the wearable device identification The anatomical posture performs a specific function.

Example 46 is directed to the method of Example 45, wherein the computing device includes one of a smart phone, a tablet device, a computer, a virtual reality system, or the wearable device itself.

Example 47 is the method of Example 45, wherein the wearable device comprises one of a wristwatch or a wristband.

Example 48 is directed to the method of Example 47, wherein the wearable device further includes an ultrasonic chip device.

Example 49 is directed to the method of Example 48, wherein the wearable device further comprises one or more ultrasonic transducers acoustically coupled to an anatomical feature of the wearable device.

Example 50 is the method of Example 49, wherein the one or more ultrasonic transducers are configured to operate at a frequency range selected to obtain the ultrasonic data from within the anatomical feature.

Example 51 is the method directed to Example 49, wherein the wearable device further comprises an ultrasonic gel pad configured to be disposed on the one or more ultrasonic transducers and the residing device on which the wearable device resides. Between anatomical features.

Example 52 is the method of Example 48, wherein the wearable device further comprises placing one or more ultrasonic transducers in an outward direction with respect to a first anatomical feature of the wearable device.

Example 53 is the method of Example 52, wherein the one or more ultrasonic transducers are configured to obtain the ultrasonic data from a second anatomical feature selected to have no one of the wearable devices resident thereon. Operates in a frequency range.

Example 54 is directed to the method of Example 45, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Example 55 is directed to a device for performing posture recognition for a portable ultrasonic device, including: a wearable device configured to: obtain ultrasonic data corresponding to an anatomical posture; and based on the obtained ultrasonic The sonic data identifies the anatomical pose.

Example 56 is the device of Example 55, wherein the wearable device comprises one of a wristwatch or a wristband having an ultrasonic chip.

Example 57 is directed to the device of Example 55, wherein the wearable device further comprises one or more position sensors.

Example 58 is directed to the device of Example 57, wherein the wearable device further comprises one or more ultrasonic transducers acoustically coupled thereto on which an anatomical feature of the wearable device resides.

Example 59 is directed to the device of Example 58, wherein the one or more ultrasonic transducers are configured to operate at a frequency range selected to obtain the ultrasonic data from within the anatomical feature.

Example 60 is the device of Example 58, wherein the wearable device further comprises an ultrasonic gel pad configured to be placed on the one or more ultrasonic transducers and the device on which the wearable device resides. Between anatomical features.

Example 61 is the device of Example 57, wherein the wearable device further comprises one or more ultrasonic transducers disposed in an outward direction with respect to a first anatomical feature of the wearable device.

Example 62 is the device of Example 61, wherein the one or more ultrasonic transducers are configured to obtain the ultrasonic data from a second anatomical feature selected to have no wearable device thereon. Operates in a frequency range.

Example 63 is the device of Example 55, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Example 64 is directed to a method for performing posture recognition, including: obtaining ultrasonic data corresponding to an anatomical posture by a wearable device; and identifying the anatomical posture based on the obtained ultrasonic data.

Example 65 is the method of Example 64, wherein the wearable device comprises one of a wristwatch or a wristband.

Example 66 is directed to the method of Example 65, wherein the wearable device further comprises an ultrasonic chip device.

Example 67 is directed to the method of example 66, wherein the wearable device further comprises one or more position sensors.

Example 68 is directed to the method of Example 65, wherein the wearable device further comprises one or more ultrasonic transducers acoustically coupled to an anatomical feature of the wearable device.

Example 69 is the method of Example 68, wherein the one or more ultrasonic transducers are configured to operate at a frequency range selected to obtain the ultrasonic data from within the anatomical feature.

Example 70 is the method directed to Example 68, wherein the wearable device further comprises an ultrasonic gel pad configured to be placed on the one or more ultrasonic transducers and the wearable device on which the wearable device resides. Between anatomical features.

Example 71 is the method of Example 67, wherein the wearable device further comprises placing one or more ultrasonic transducers in an outward direction with respect to a first anatomical feature of the wearable device.

Example 72 is directed to the method of Example 71, wherein the one or more ultrasonic transducers are configured to obtain the ultrasonic data from a second anatomical feature selected to have no one of the wearable devices resident thereon. Operates in a frequency range.

Example 73 is directed to the method of Example 64, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Example 74 is directed to a non-transitory computer-readable storage medium that stores processor-executable instructions that, when executed by at least one processor, cause the at least one processor to: from a wearable device Obtaining ultrasonic data corresponding to an anatomical posture; and identifying the anatomical posture based on the obtained ultrasonic data.

Example 75 is directed to the media of Example 74, wherein the wearable device includes one of a wristwatch or a wristband.

Example 76 is directed to the media of Example 75, wherein the wearable device further includes an ultrasonic chip device.

Example 77 is directed to the media of example 76, wherein the wearable device further includes one or more position sensors.

Example 78 is the non-transitory computer-readable storage medium for Example B24, wherein the wearable device further comprises one or more ultrasonic transducers acoustically coupled to the anatomical feature of one of the wearable devices residing thereon .

Example 79 is directed to the media of example 79, wherein the one or more ultrasonic transducers are configured to operate at a frequency range selected to obtain the ultrasonic data from within the anatomical feature.

Example 80 is directed to the media of Example 78, wherein the wearable device further includes an ultrasonic gel pad configured to be placed on the one or more ultrasonic transducers and the one on which the wearable device resides. Between anatomical features.

Example 81 is directed to the media of example 77, wherein the wearable device further comprises one or more ultrasonic transducers disposed in an outward direction with respect to a first anatomical feature of the wearable device.

Example 82 is directed to the media of example 81, wherein the one or more ultrasonic transducers are configured to obtain the ultrasonic data from a second anatomical feature selected to have no one of the wearable devices resident thereon. Operates in a frequency range.

Example 83 is directed to the media of Example 74, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.

Unless clearly indicated to the contrary, the word "a" as used in this specification and the scope of the patent application should be understood to mean "at least one".

As used herein in the description and the scope of the patent application, the phrase "and / or" should be understood to mean "either or both" of the elements so conjoined, that is, in some cases, conjunctive existence and in Elements that are otherwise unconnected.

As used in this specification and the scope of the patent application, the phrase "at least one" with respect to a list of one or more elements shall be understood to mean that any one or more of those elements selected from the list of elements At least one element, but not necessarily including at least one of each element specifically listed in the element list, and does not exclude any combination of elements in the element list. This definition also allows elements other than the specifically identified elements in the list of elements referred to by the phrase "at least one" to be present, regardless of whether they are related or unrelated to those elements that are specifically identified.

The use of ordinal terms such as "first," "second," and "third" in the scope of a patent application to modify a claim element itself does not imply any priority, priority, or Sequential or chronological order of performing method actions. It is only used as a label to distinguish one claim element with a certain name from another element with the same name (but using ordinal terms) to distinguish those claim elements.

The terms "approximately" and "about" can be used to mean within ± 20% of the target value in some specific examples, within ± 10% of the target value in some specific examples, and within ± 10% of the target value in some specific examples Within 5%, and in some specific examples within ± 2% of the target value. The terms "approximately" and "about" may include target values.

Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including", "including" or "having", "containing", "concerned" and variations thereof in this article is intended to cover the items listed below and their equivalents and additional items.

After having described several aspects of at least one specific example above, it should be understood that those skilled in the art will readily think of various changes, modifications and improvements. These changes, modifications, and improvements are intended to be the object of this disclosure. Therefore, the foregoing description and drawings are by way of example only.

402‧‧‧left hand

404‧‧‧ Wearable device

406‧‧‧right hand

408‧‧‧Image capture device

410‧‧‧Sensor

500‧‧‧ "virtual cycle"

502‧‧‧stage

504‧‧‧stage

506‧‧‧stage

508‧‧‧stage

602‧‧‧ Information

604‧‧‧input layer

606‧‧‧hidden layer

608‧‧‧ output layer

610‧‧‧ Convolution and Pooling Layer

612‧‧‧ dense layer

700‧‧‧ wearable device

702‧‧‧Main Module

704‧‧‧ Ultrasonic Module

706‧‧‧First wristband

708‧‧‧Second wristband

710‧‧‧ Ultrasonic Chip Device

712‧‧‧Processing Circuit

714‧‧‧Memory circuit

716‧‧‧communication circuit

718‧‧‧Power Management Circuit

720‧‧‧Printed Circuit Board (PCB)

722‧‧‧display

724‧‧‧Main housing components

726‧‧‧Main housing components

728‧‧‧ Ultrasonic Shell Components

730‧‧‧ battery

732‧‧‧hole

734‧‧‧ the first end of the first wristband

736‧‧‧conductor

738‧‧‧Ring buckle

740‧‧‧pin

742‧‧‧ the first end of the second wristband

744‧‧‧ opening

746‧‧‧ Acoustic lens

748‧‧‧Coupling element

750‧‧‧ first end

752‧‧‧ second end

754‧‧‧ second end

756‧‧‧second end

800‧‧‧ wearable device

900‧‧‧ wearable device

912‧‧‧ Internal Processing Circuit

914‧‧‧Memory Circuit

920‧‧‧Printed Circuit Board (PCB)

958‧‧‧ wiring cable

960‧‧‧ opening

962‧‧‧Male connector

1000‧‧‧ wearable device

1020‧‧‧Printed Circuit Board (PCB)

1030‧‧‧ Battery

1100‧‧‧ wrist device

1102‧‧‧Main Module

1104‧‧‧Sensor

1106‧‧‧First wristband

1108‧‧‧Second wristband

1212‧‧‧Allow

1300‧‧‧ wearable device

1400‧‧‧ wearable device

1502‧‧‧Reservoir

1504‧‧‧Reservoir

1506‧‧‧ cover

1508‧‧‧valve

1510‧‧‧ gate

1512‧‧‧ Valve

1514‧‧‧ gate

1516‧‧‧ opening

1518‧‧‧ opening

Various aspects and specific examples will be described with reference to the following illustrative and non-limiting figures. It should be understood that the figures are not necessarily drawn to scale. Items that appear in multiple figures are indicated by the same or similar reference numerals in all figures in which they appear.

FIG. 1 shows an example process for performing gesture recognition according to some specific examples described herein;

2 illustrates an example process for interfacing with a computing device according to some specific examples described herein;

3 shows an example process for training a wearable device to perform gesture recognition according to some specific examples described herein;

4 shows an example of training a wearable device to perform posture recognition as discussed with reference to FIG. 3 according to some specific examples described herein;

Figure 5 shows an example of a "virtual cycle" for continuously improving the performance of certain processes and systems described herein;

Figure 6 shows an example convolutional neural network configured to analyze data according to some specific examples contained herein;

7 shows an example of a wearable device for ultrasound data collection configured to be worn on a user's wrist according to some specific examples contained herein;

8 shows another example of a wearable device for ultrasonic data collection configured to be worn on a user's wrist according to some specific examples contained herein;

9 shows another example of a wearable device for ultrasound data collection configured to be worn on a user's wrist according to some specific examples contained herein;

FIG. 10 shows another example of a wearable device for ultrasound data collection configured to be worn on a user's wrist according to some specific examples contained herein; FIG.

11A to 11G show an example of a wearable device for ultrasonic data collection, the wearable device being configured to be coupled to a user's wrist when the wearable device is assembled and worn;

FIG. 12 shows an example of the wearable device of FIG. 9 when electrically coupled to a user's personal wrist device; FIG.

13 shows another example of a wearable device for ultrasonic data collection configured to be worn on a user's wrist according to some specific examples contained herein;

14 shows another example of a wearable device for ultrasound data collection configured to be worn on a user's wrist according to some specific examples contained herein; and

FIG. 15 shows an example of the ultrasonic module of FIG. 7 including a reservoir for renewing the coupling element of FIG. 7 according to some specific examples contained herein.

Claims (25)

A system for training a wearable device to perform posture recognition, including: Memory circuit A processing circuit in communication with the memory circuit, the processing circuit being configured to: Obtaining ultrasonic data corresponding to the anatomical posture from the wearable device; Obtaining non-ultrasonic data corresponding to the anatomical posture; and By correlating the non-ultrasonic data with the ultrasonic data, a machine learning model accessed by the wearable device is trained to recognize the anatomical posture. The system of claim 1, wherein the wearable device comprises one of a wristwatch or a wristband configured with an ultrasonic chip. The system of claim 2, wherein the wearable device further comprises one or more position sensors. The system of claim 3, wherein the one or more position sensors include one or more accelerometers, gyroscopes, magnetometers, compasses, and global positioning systems (GPS). The system of claim 3, wherein the processing circuit is further configured to use the one or more position sensors to obtain the non-ultrasonic data. The system of claim 1, wherein the processing circuit is configured to obtain the non-ultrasonic data using an image capture device, the image capture device further comprising one of a camera, a smartphone, or a tablet device. The system of claim 1, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture. A method of training a wearable device to perform posture recognition includes: Obtaining ultrasonic data corresponding to an anatomical posture by the wearable device; Obtaining non-ultrasonic data corresponding to the anatomical posture; and Based on correlating the non-ultrasonic data with the ultrasonic data, a machine learning model accessible by the wearable device is trained to recognize the anatomical pose. The method of claim 8, wherein the wearable device comprises one of a wristwatch or a wristband. The method of claim 9, wherein the wearable device further comprises an ultrasonic chip device. The method of claim 10, wherein the wearable device further comprises one or more position sensors. The method according to claim 11, further comprising using the one or more position sensors to obtain the non-ultrasonic data. The method of claim 12, wherein the one or more position sensors include one or more accelerometers, gyroscopes, magnetometers, compasses, and global positioning systems (GPS). The method according to claim 8, further comprising using an image capture device to obtain the non-ultrasonic data. The method according to claim 14, wherein the image capturing device comprises one of a camera, a smartphone, or a tablet device. The method according to claim 8, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture. A non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor, causes the at least one processor to: Obtaining ultrasonic data corresponding to an anatomical posture from a wearable device; Obtaining non-ultrasonic data corresponding to the anatomical posture; and Based on correlating the non-ultrasonic data with the ultrasonic data, a machine learning model accessed by the wearable device is trained to recognize the anatomical pose. The at least one non-transitory computer-readable storage medium according to claim 17, wherein the wearable device includes one of a watch or a wristband. The at least one non-transitory computer-readable storage medium according to claim 18, wherein the wearable device further comprises an ultrasonic chip device. The at least one non-transitory computer-readable storage medium according to claim 19, wherein the wearable device further comprises one or more position sensors. The at least one non-transitory computer-readable storage medium of claim 20, wherein the one or more position sensors include one or more accelerometers, gyroscopes, magnetometers, compasses, and global positioning systems (GPS ). The at least one non-transitory computer-readable storage medium according to claim 20, further storing, when executed by the at least one processor, causing the at least one processor to use the one or more position sensors to obtain the non-transitory The processor of the sonic data can execute instructions. The at least one non-transitory computer-readable storage medium according to claim 17, further storing a processor that, when executed by the at least one processor, causes the at least one processor to use the image capture device to obtain the non-ultrasonic data. Execute the instruction. The at least one non-transitory computer-readable storage medium according to claim 23, wherein the image capture device comprises one of a camera, a smartphone, or a tablet device. The at least one non-transitory computer-readable storage medium according to claim 17, wherein the anatomical posture includes one of a hand posture, a finger posture, a wrist posture, and / or an arm posture.