EP4134924B1 - Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder - Google Patents

Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder

Info

Publication number
EP4134924B1
EP4134924B1 EP21190926.2A EP21190926A EP4134924B1 EP 4134924 B1 EP4134924 B1 EP 4134924B1 EP 21190926 A EP21190926 A EP 21190926A EP 4134924 B1 EP4134924 B1 EP 4134924B1
Authority
EP
European Patent Office
Prior art keywords
gesture
training
neural network
positional
network algorithm
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP21190926.2A
Other languages
English (en)
French (fr)
Other versions
EP4134924A1 (de
Inventor
Avik SANTRA
Souvik Hazra
Thomas Reinhold STADELMAYER
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Infineon Technologies AG
Original Assignee
Infineon Technologies AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Infineon Technologies AG filed Critical Infineon Technologies AG
Priority to EP21190926.2A priority Critical patent/EP4134924B1/de
Priority to US17/886,264 priority patent/US12307821B2/en
Priority to CN202210966956.1A priority patent/CN115705757A/zh
Publication of EP4134924A1 publication Critical patent/EP4134924A1/de
Application granted granted Critical
Publication of EP4134924B1 publication Critical patent/EP4134924B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/02—Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
    • G01S13/50—Systems of measurement based on relative movement of target
    • G01S13/58—Velocity or trajectory determination systems; Sense-of-movement determination systems
    • G01S13/583—Velocity or trajectory determination systems; Sense-of-movement determination systems using transmission of continuous unmodulated waves, amplitude-, frequency-, or phase-modulated waves and based upon the Doppler effect resulting from movement of targets
    • G01S13/584—Velocity or trajectory determination systems; Sense-of-movement determination systems using transmission of continuous unmodulated waves, amplitude-, frequency-, or phase-modulated waves and based upon the Doppler effect resulting from movement of targets adapted for simultaneous range and velocity measurements
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G01S7/415—Identification of targets based on measurements of movement associated with the target
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
    • G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G01S7/417—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section involving the use of neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/761—Proximity, similarity or dissimilarity measures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
    • G06V10/763—Non-hierarchical techniques, e.g. based on statistics of modelling distributions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/02—Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
    • G01S13/06—Systems determining position data of a target
    • G01S13/42—Simultaneous measurement of distance and other co-ordinates
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/02—Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
    • G01S13/50—Systems of measurement based on relative movement of target
    • G01S13/58—Velocity or trajectory determination systems; Sense-of-movement determination systems
    • G01S13/589—Velocity or trajectory determination systems; Sense-of-movement determination systems measuring the velocity vector
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/12—Acquisition of 3D measurements of objects

Definitions

  • Various examples of the disclosure are broadly concerned with recognizing gestures based on a radar measurement.
  • a computer-implemented method of training a variational auto-encoder neural network algorithm for predicting a gesture class of a gesture performed by an object of a scene, the gesture class being selected from a plurality of gesture classes includes obtaining multiple training sets of one or more training positional time spectrograms of a radar measurement of the scene including the object. Each one of the multiple training sets is associated with a respective ground-truth label indicative of a respective gesture class. Also, the computer-implemented method includes training the variational auto-encoder neural network algorithm based on the multiple training sets and the associated ground-truth labels.
  • the method of FIG. 8 includes multiple boxes 3105, 3110, 3115, and 3120 that together implement obtaining - box 3140 - input data for performing a gesture classification at box 3150.
  • the positional time spectrograms can be obtained by time gating the measurement data of the radar measurement based on one or more corresponding trigger events. These one or more trigger events can be associated with the gesture detection.
  • the gesture duration 250 per gestures is pre-set or initialized at 2 s. Within this time, the test person has to perform the gesture. Some gestures like the swipes are performed in a much shorter time period, and therefore, after recording, the start and end of a gesture is detected, based on the trigger events. Thus, the gesture duration 250 is refined. The data samples within the refined gesture duration 250 are preserved, whereas the remaining data samples are set to zero. The start of a gesture is detected, if for example the energy within 10 frames 45 increases over a threshold compared to the energy of the first frame 45 in the series. The end of gesture is similarly detected when a drop of energy larger then the threshold is detected, as the trigger event.
  • the measurement data is optionally preprocessed at box 3115, to obtain the positional time spectrograms.
  • the measurement data is optionally preprocessed at box 3115, to obtain the positional time spectrograms.
  • the measurement data is optionally preprocessed at box 3115, to obtain the positional time spectrograms.
  • spectrograms of the range, Doppler (velocity), azimuth and elevation are obtained from the measurement data 64.
  • Such spectrograms show the temporal progress of the respective physical observable and allow a unique identification of a specific gesture.
  • FIG. 11 and FIG. 12 - upper row - illustrate unfiltered positional time spectrograms 101-104; while FIG. 11 and FIG. 12 - lower row - illustrate filtered positional time spectrograms 101*-104* (the filtering will be explained in detail below).
  • N st and N ft are the number of chirps 48 and number of samples 49 per chirps 48 respectively (cf. FIG. 9 ).
  • a moving target indication (MTI) in form of an exponentially weighted moving average (EWMA) filter may be applied on the RDIs (cf. FIG. 13 : 7010).
  • x MTI is the MTI filtered RDI
  • x 1 is the RDI of the current time step
  • x avg is the average RDI of the filter.
  • a range and a Doppler vector can be extracted (cf. FIG 13 : 7020 and 7025; FIG. 14 : boxes 7120 and 7125).
  • the selected vectors - within the gesture duration 250 at which a gesture 501-510 is detected - are aggregated / concatenated and form the range and Doppler spectrograms respectively.
  • the range vectors and correspondingly the Doppler vectors are selected based on marginalization along each axis, they are appended across time to generate the range spectrogram and Doppler spectrogram respectively (cf. FIG. 14 : boxes 7130 and 7135).
  • the data of receiving antennas 1 and 3 and for the elevation angle the data of antennas 2 and 3 is used. Again, a concatenation of these vectors for each data frame 45 within the gesture duration 250 fields the respective time angle spectrogram (cf. FIG. 14 : box 7150).
  • FIG. 11 and FIG. 12 illustrates respective filtered positional time spectrograms 101*-104*.
  • the gesture class is predicted based on a comparison of the mean 144 of the distribution of the feature embedding 149 of the positional time spectrograms with one or more of the predefined regions 211-213 defined in the feature space 200 of the feature embedding 149 (cf. FIG. 6 ).
  • These predefined regions 211-213 can be obtained from the training of the VAENN (cf. FIG. 5 : box 3005). Next, techniques with respect to the training will be described.
  • FIG. 15 schematically illustrates aspects with respect to the training of the VAENN 111.
  • FIG. 15 illustrates a processing pipeline for implementing the training.
  • the processing pipeline can thus implement box 3005.
  • the training of the VAENN 111 is based on multiple training sets 109 of training positional time spectrograms 101-104, 101*-104* and associated ground-truth labels 107.
  • These training positional time spectrograms 101-104, 101*-104* can be obtained using the pre-processing described in connection with box 3115 of the method of FIG. 8 ; in particular, the UKF can be used to obtain the filtered positional time spectrograms 101*-104*.
  • the VAENN 111 receives, as input, the raw and filtered positional time spectrograms 101-104, 101*-104* (in FIG. 15 , only the raw spectrograms 101-104 are shown as input).
  • the ground-truth labels 107 denote the gesture class 520 of the gesture 501-510 captured by the respective positional time spectrograms 101-103.
  • a first loss 191 is based on a difference between the reconstructions 181-184 of the respective input positional time spectrograms 101-104 and data associated with the input positional time spectrograms 101-104. More specifically, in the illustrated example, the input (raw) positional time spectrograms 101-104 are filtered at box 7005, e.g., using an Unscented Kalman filter, to obtain respective filtered positional time spectrograms 101*-104* (cf. FIG. 11 and FIG. 12 ). These filtered positional time spectrograms 101*-104* are then compared with the reconstructions 181-184. For instance, a pixel-wise difference could be calculated (cf. Eq. 12). Accordingly, the VAENN 111 is trained to reconstruct filtered positional time spectrograms 101*-104*.
  • filtering can be helpful for training (cf. FIG. 5 : box 3005).
  • filtering may sometimes also be used for inference (cf. FIG. 5 : Box 3010) and then be executed as part of the pre-processing (cf. FIG. 8 : box 3115).
  • the VAENN 111 is trained to reconstruct the filtered positional time spectrograms 101*-104*. Then, during inference, it may not be required to implement the filtering. Not having to rely on the filtering during inference (by appropriately training the VAENN 111) makes the implementation of the gesture classification fast and robust.
  • an unscented Kalman filter may be applied to the positional time spectrograms.
  • the maximum value of each positional time spectrogram is extracted, which serves as the measurement vector for the UKF. Due to filtering, outliers and measurement errors are mitigated, but on the other hand also "micro" features are removed. Especially for the gestures finger-wave and finger-rub these micro features can be important since the hand is kept static and only small finger movements define the gesture.
  • filtering emphasizes the overall movement of the hand and removes outliers ( FIG. 11 and FIG. 12 : lower rows). Especially the angle estimation using only two antennas tend to have large variances in its results. Thus, the filtering is helpful to remove outliers.
  • class-specific (and thus generally desirable) "micro" features can also be filtered out. For instance, this is apparent when comparing the filtered elevation angle time spectrograms 104* for the gesture classes "circuit clockwise" and "finger wave" according to FIG. 11 and FIG. 12 : both spectrograms 104* have a comparable qualitative shape (peak - plateau - dip) - micro features distinguishing these spectrograms 104* are removed due to the filtering.
  • the unscented transformation - used in the UKF - tries to approximate the distribution of a random variable that undergoes a non-linear transformation.
  • ⁇ ⁇ ( ⁇ )
  • ⁇ represents both process model (.) and measurement model h(.).
  • L VAE ⁇ 1 L triplet stat + ⁇ 2 L MSE + ⁇ 3 L KL + ⁇ 4 L center stat
  • each gesture class Based on the class distributions of the feature embedding 149 of the VAENN 111 obtained for the training sets 109 of training positional time spectrograms 101-104 belonging to each gesture class, it is possible to determine the regions 211-213 in the feature space 200 used during gesture classification in the inference phase at box 3010. Each region 211-213 is thus associated with a respective gesture class 520.
  • regions 211-213 may be stored as parameters along with the VAENN 111 and then used during inference at box 3010. It can be decided whether the mean 144 of a certain instance of the feature embedding is inside or outside such regions 211-213.
  • VAENN can also be applied for achieving a robust gesture classification using other sensors such as vision, ultra-sonic sensors and any other sensor capable of receiving gesture feedback.
  • the raw data obtained from the radar sensor undergoes a preprocessing step (cf., e.g., box 3115) to obtain features relevant for the purpose of gesture classification.
  • a preprocessing step cf., e.g., box 3115
  • similar specific gesture feature extraction process can be performed for other sensor such as velocity and range information where applicable.
  • the possible gesture classification is not limited to just hand gesture but virtually any form of gesture feedback such as body pose or facial expressions.
  • a statistical distance is determined and considered in a loss for training the VAENN.
  • the disclosed embodiments are not limited to a statistical distance between a distribution and a point (mean), but can also be applied to a distance between two distributions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Psychiatry (AREA)
  • Human Computer Interaction (AREA)
  • Social Psychology (AREA)
  • Data Mining & Analysis (AREA)
  • Probability & Statistics with Applications (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • User Interface Of Digital Computer (AREA)
  • Radar Systems Or Details Thereof (AREA)

Claims (15)

  1. Computerimplementiertes Verfahren, das Folgendes umfasst:
    - Erhalten eines oder mehrerer Position-Zeit-Spektrogramme (101-104, 101*-104*) einer Radarmessung einer Szene (80), die ein Objekt (83) umfasst, und
    - auf der Grundlage der einen oder mehreren Position-Zeit-Spektrogramme (101-104, 101*-104*) und auf der Grundlage einer Merkmalseinbettung (149) eines variationalen Autoencoder-Neuralnetzwerkalgorithmus, (111) Vorhersagen einer Gestenklasse (520) einer von dem Objekt (83) ausgeführten Geste (501-510).
  2. Computerimplementiertes Verfahren nach Anspruch 1,
    wobei die Gestenklasse (520) auf der Grundlage eines Vergleichs eines Mittelwerts (144) einer Verteilung der Merkmalseinbettung (149) des variationalen Auto-Encoder-Neuronalnetzwerk-Algorithmus (111) mit einem oder mehreren Bereichen (211-213) vorhergesagt wird, die in einem Merkmalsraum (200) der Merkmalseinbettung (149) vordefiniert sind.
  3. Computerimplementiertes Verfahren nach Anspruch 2, das ferner umfasst:
    - Überwachen einer Clusterbildung der Mittelwerte (144) der Verteilungen der Merkmalseinbettung (149) des variationalen Autoencoder-Neuralnetzwerkalgorithmus (111), die für mehrere Sätze der einen oder mehreren Position-Zeit-Spektrogramme erhalten wurden, wobei die Clusterbildung außerhalb der einen oder mehreren vordefinierten Bereichen (211-213) liegt, und
    - auf der Grundlage der Überwachung der Clusterbildung, Bestimmen eines weiteren vordefinierten Bereichs im Merkmalsraum (200), um einen entsprechenden Cluster (214) zu umschließen.
  4. Computerimplementiertes Verfahren nach einem der vorstehenden Ansprüche,
    wobei das eine oder die mehreren Position-Zeit-Spektrogramme (101-104, 101*-104*) durch zeitliches Gating von Messdaten (64) der Radarmessung auf der Grundlage mindestens eines Triggerereignisses erhalten werden.
  5. Verfahren nach Anspruch 4,
    wobei das mindestens eine Triggerereignis einen Vergleich zwischen einer Änderungsrate einer durch die Messdaten (64) erfassten Positionsbeobachtungsgröße und mindestens einem vordefinierten Schwellenwert umfasst.
  6. Computerimplementiertes Verfahren nach Anspruch 4 oder 5,
    wobei das mindestens eine Triggerereignis eine Ausgabe eines Gestenerkennungsalgorithmus umfasst.
  7. Computerimplementiertes Verfahren nach einem der vorstehenden Ansprüche,
    wobei das eine oder die mehreren Position-Zeit-Spektrogramme (101-104, 101*-104*) aus der Gruppe ausgewählt werden, die aus einem Entfernung-Zeit-Spektrogramm, einem Geschwindigkeit-Zeit-Spektrogramm, einem Azimutwinkel-Zeit-Spektrogramm und einem Elevationswinkel-Zeit-Spektrogramm besteht.
  8. Computerimplementiertes Verfahren nach einem der vorstehenden Ansprüche,
    wobei das eine oder die mehreren Position-Zeit-Spektrogramme (101-104, 101*-104*) ein oder mehrere Roh-Position-Zeit-Spektrogramme (101-104) umfassen,
    wobei der Variations-Autoencoder-Neuronalnetzwerk-Algorithmus (111) darauf trainiert wurde, ein oder mehrere gefilterte Position-Zeit-Spektrogramme (101*-104*) zu rekonstruieren.
  9. Computerimplementiertes Verfahren zum Trainieren (3005) eines variationalen Auto-Encoder-Neuronalnetzwerk-Algorithmus (111) zum Vorhersagen einer Gestenklasse (520) einer Geste (501-510), die von einem Objekt (83) einer Szene (80) ausgeführt wird, wobei die Gestenklasse (520) aus einer Vielzahl von Gestenklassen (520) ausgewählt wird, wobei das computerimplementierte Verfahren umfasst:
    - Erhalten mehrerer Trainingssätze (109) aus einem oder mehreren Training-Position-Zeit-Spektrogrammen (101-104, 101*-104*) einer Radarmessung der Szene (80) mit dem Objekt (83), wobei jeder der mehreren Trainingssätze (109) einem jeweiligen Ground-Truth-Label (107) zugeordnet ist, das die jeweilige Gestenklasse (520) indiziert, und
    - Trainieren des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111) auf der Grundlage der mehreren Trainingssätze (109) und der zugehörigen Ground-Truth-Labels (107).
  10. Computerimplementiertes Verfahren nach Anspruch 9,
    wobei das Training des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111) mindestens einen Verlust (192) verwendet, der bestimmt ist auf der Grundlage mindestens eines statistischen Abstands zwischen einer Verteilung einer Merkmalseinbettung (149) des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111), die für einen ersten Trainingssatz der mehreren Trainingssätze (109) erhalten wurde, der einer ersten Gestenklasse (520) der mehreren Gestenklassen (520) zugeordnet ist, und mindestens einem Mittelwert (144) der mindestens einen Verteilung der Merkmalseinbettung (149) des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111), die für mindestens einen zweiten Trainingssatz der mehreren Trainingssätze (109) erhalten wurde, der mindestens einer der ersten Gestenklasse (520) oder einer zweiten Gestenklasse (520) der mehreren Gestenklassen (520) zugeordnet ist.
  11. Computerimplementiertes Verfahren nach Anspruch 10,
    wobei der mindestens eine Verlust (192) einen statistischen Abstand-Triplet-Verlust umfasst, der auf der Grundlage eines ersten statistischen Abstands und eines zweiten statistischen Abstands bestimmt wird,
    wobei der erste statistische Abstand zwischen der Verteilung der Merkmalseinbettung (149) des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111), die für einen Anker-Trainingssatz der mehreren Trainingssätze (109) erhalten wurde, und dem Mittelwert (144) der Verteilung der Merkmalseinbettung (149) des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111), die für einen positiven Trainingssatz der mehreren Trainingssätze erhalten wurde,
    wobei der zweite statistische Abstand zwischen der Verteilung der Merkmalseinbettung (149) des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111), die für den Anker-Trainingssatz erhalten wurde, und dem Mittelwert (144) der Verteilung der Merkmalseinbettung (149) des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111) liegt, die für einen negativen Trainingssatz (109) der mehreren Sätze erhalten wurde.
  12. Computerimplementiertes Verfahren nach Anspruch 10 oder 11,
    wobei der mindestens eine Verlust (192) einen statistischen Distanzzentrumverlust umfasst, der auf der Grundlage einer statistischen Distanz zwischen einer Klassenverteilung, die der ersten Gestenklasse (520) zugeordnet ist, und Mitteln der Verteilungen der Merkmalseinbettung (149) des variablen Auto-Encoder-Neuronalnetzwerk-Algorithmus (111) bestimmt wird, die für alle Trainingssätze der mehreren Trainingssätze (109) erhalten wurden, die der ersten Gestenklasse (520) zugeordnet sind.
  13. Computerimplementiertes Verfahren nach einem der Ansprüche 10 bis 12, wobei der statistische Abstand ein Mahalanobis-Abstand ist.
  14. Computerimplementiertes Verfahren nach einem der Ansprüche 9 bis 13,
    wobei das eine oder die mehreren Training-Position-Zeit-Spektrogramme (101-104) ein oder mehrere rohe Training-Position-Zeit-Spektrogramme (101-104) umfassen,
    wobei das Verfahren ferner umfasst:
    - Anwenden (7005) eines Unscented-Kalman-Filters auf das eine oder die mehreren rohen Training-Position-Zeit-Spektrogramme, um ein oder mehrere gefilterte Training-Position-Zeit-Spektrogramme (101*-104*) zu erhalten,
    wobei das Training des variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111) mindestens einen Rekonstruktionsverlust (191) verwendet, der auf einer Differenz zwischen einer Rekonstruktion (181-184) des einen oder der mehreren rohen Training-Position-Zeit-Spektrogramme, die von dem variationalen Autoencoder-Neuronalnetzwerk-Algorithmus (111) ausgegeben werden, und dem einen oder den mehreren gefilterten Training-Position-Zeit-Spektrogrammen (101*-104*) basiert.
  15. Computerimplementiertes Verfahren nach einem der vorstehenden Ansprüche, das ferner umfasst:
    - basierend auf Klassenverteilungen einer Merkmalseinbettung (149) des variationalen Autoencoder-Neuronalen-Netzwerk-Algorithmus (111), die für die Trainingssätze (109) erhalten wurden, die mit jeder der mehreren Gestenklassen (520) assoziiert sind, Bestimmen vordefinierter Bereiche (211-213) in einem Merkmalsraum (200) der Merkmalseinbettung (149) für die Gestenklassenvorhersage.
EP21190926.2A 2021-08-12 2021-08-12 Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder Active EP4134924B1 (de)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP21190926.2A EP4134924B1 (de) 2021-08-12 2021-08-12 Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder
US17/886,264 US12307821B2 (en) 2021-08-12 2022-08-11 Radar-based gesture classification using a variational auto-encoder neural network
CN202210966956.1A CN115705757A (zh) 2021-08-12 2022-08-11 使用变分自动编码器神经网络算法的基于雷达的手势分类

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP21190926.2A EP4134924B1 (de) 2021-08-12 2021-08-12 Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder

Publications (2)

Publication Number Publication Date
EP4134924A1 EP4134924A1 (de) 2023-02-15
EP4134924B1 true EP4134924B1 (de) 2025-10-01

Family

ID=77316831

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21190926.2A Active EP4134924B1 (de) 2021-08-12 2021-08-12 Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder

Country Status (3)

Country Link
US (1) US12307821B2 (de)
EP (1) EP4134924B1 (de)
CN (1) CN115705757A (de)

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4134924B1 (de) * 2021-08-12 2025-10-01 Infineon Technologies AG Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder
US12487599B2 (en) 2022-03-24 2025-12-02 Dell Products L.P. Efficient event-driven object detection at the forklifts at the edge in warehouse environments
US12530619B2 (en) * 2022-07-14 2026-01-20 Dell Products L.P. Feature-aware open set multi-model for trajectory classification in mobile edge devices
EP4310738A1 (de) * 2022-07-20 2024-01-24 Infineon Technologies AG Training eines maschinenlernalgorithmus unter verwendung erklärbarer künstlicher intelligenz
US12524886B2 (en) 2022-07-27 2026-01-13 Dell Products L.P. Object-driven event detection from fixed cameras in edge environments
US20240230841A9 (en) * 2022-10-21 2024-07-11 Texas Instruments Incorporated Frequency modulated continuous wave radar system with object classifier
EP4432162A1 (de) 2023-03-13 2024-09-18 Infineon Technologies Dresden GmbH & Co . KG Frühkommende neuronale netze zur radarverarbeitung
CN116524537A (zh) * 2023-04-26 2023-08-01 东南大学 一种基于cnn和lstm联合的人体姿态识别方法
CN117054803B (zh) * 2023-06-14 2026-02-17 南昌航空大学 一种含分布式光伏配电网接地故障辨识方法及系统
CN116482680B (zh) * 2023-06-19 2023-08-25 精华隆智慧感知科技（深圳）股份有限公司 一种身体干扰识别方法、装置、系统和存储介质
EP4650991A1 (de) 2024-05-15 2025-11-19 Infineon Technologies AG Datenerweiterung für objektspezifische kinematische beobachtungsbereiche aus radarmessdaten
CN119622679B (zh) * 2024-11-25 2025-10-21 重庆大学 一种基于多级内存增强自编码器持续身份认证方法
CN119439155A (zh) * 2024-11-28 2025-02-14 珠海城市职业技术学院 基于毫米波集成的人体行为识别方法、系统及电子设备
CN119690319B (zh) * 2024-12-19 2025-11-25 清华大学 基于敲击手势识别的符号输入方法及装置

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10699538B2 (en) * 2016-07-27 2020-06-30 Neosensory, Inc. Method and system for determining and providing sensory experiences
US10466772B2 (en) * 2017-01-09 2019-11-05 Infineon Technologies Ag System and method of gesture detection for a remote device
US11354823B2 (en) * 2017-07-11 2022-06-07 Deepmind Technologies Limited Learning visual concepts using neural networks
US10937438B2 (en) * 2018-03-29 2021-03-02 Ford Global Technologies, Llc Neural network generative modeling to transform speech utterances and augment training data
US10205457B1 (en) * 2018-06-01 2019-02-12 Yekutiel Josefsberg RADAR target detection system for autonomous vehicles with ultra lowphase noise frequency synthesizer
EP3608918B1 (de) * 2018-08-08 2024-05-22 Tata Consultancy Services Limited Parallele implementierung von tiefenneuronalen netzen zur klassifizierung von herztonsignalen
EP3716020B1 (de) * 2019-03-29 2023-03-01 Tata Consultancy Services Limited Systeme und verfahren zur dreidimensionalen (3d) rekonstruktion menschlicher gesten aus radargestützten messungen
US11487363B2 (en) * 2019-07-29 2022-11-01 Qualcomm Incorporated Gesture detection in interspersed radar and network traffic signals
US11640208B2 (en) * 2019-11-21 2023-05-02 Infineon Technologies Ag Gesture feedback in distributed neural network system
US11475910B2 (en) * 2020-02-11 2022-10-18 Purdue Research Foundation System and methods for machine anomaly detection based on sound spectrogram images and neural networks
US11774553B2 (en) * 2020-06-18 2023-10-03 Infineon Technologies Ag Parametric CNN for radar processing
EP4134924B1 (de) * 2021-08-12 2025-10-01 Infineon Technologies AG Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder

Also Published As

Publication number Publication date
US20230068523A1 (en) 2023-03-02
US12307821B2 (en) 2025-05-20
CN115705757A (zh) 2023-02-17
EP4134924A1 (de) 2023-02-15

Similar Documents

Publication Publication Date Title
EP4134924B1 (de) Radarbasierte gestenklassifizierung unter verwendung eines algorithmus eines neuronalen netzes mit variationalem auto-encoder
Arsalan et al. Character recognition in air-writing based on network of radars for human-machine interface
Kong et al. A survey of mmwave radar-based sensing in autonomous vehicles, smart homes and industry
Sun et al. Real-time radar-based gesture detection and recognition built in an edge-computing platform
Hazra et al. Short-range radar-based gesture recognition system using 3D CNN with triplet loss
Seyfioglu et al. DNN transfer learning from diversified micro-Doppler for motion classification
Dubey et al. A bayesian framework for integrated deep metric learning and tracking of vulnerable road users using automotive radars
CN111722706A (zh) 基于雷达网络的空中书写字符识别的方法和系统
US20220269926A1 (en) Radar-Based Object Tracking Using a Neural Network
US12529777B2 (en) Radar-based motion classification using one or more time series
Sun et al. Automatic radar-based gesture detection and classification via a region-based deep convolutional neural network
Guendel et al. Multipath exploitation for human activity recognition using a radar network
Arsalan et al. Air-writing with sparse network of radars using spatio-temporal learning
Arsalan et al. Radar trajectory-based air-writing recognition using temporal convolutional network
EP4036599A1 (de) Interagierender multimodellverfolgungsalgorithmus unter verwendung eines ruhezustandsmodells
Sun et al. Multi-feature encoder for radar-based gesture recognition
US20240103130A1 (en) Feature extraction for remote sensing detections
Li et al. Dynamic gesture recognition method based on millimeter-wave radar
Zhai et al. A multialignment task-adaptive method for MFR mode recognition on few-shot open-set learning
Wang et al. mPCT-LSTM: A lightweight human activity recognition model for 3D point clouds in millimeter-wave radar
Wang et al. Multi-human activity recognition based on sequential 4D point clouds using frequency-modulated continuous wave radar
Bai et al. Deep CNN for micromotion recognition of space targets
Kang et al. From classical approaches to recent advancements: A holistic review of acoustic detection for unmanned aerial vehicles
Chang et al. Efficient human action recognition with fine-grained spatiotemporal feature extraction from millimeter-wave point clouds
Ege Human activity classification with deep learning using FMCW radar

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: INFINEON TECHNOLOGIES AG

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230804

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602021039426

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06V0040200000

Ipc: G06V0010820000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G01S 13/00 20060101ALI20250324BHEP

Ipc: G06V 40/20 20220101ALI20250324BHEP

Ipc: G01S 7/41 20060101ALI20250324BHEP

Ipc: G01S 13/42 20060101ALI20250324BHEP

Ipc: G01S 13/58 20060101ALI20250324BHEP

Ipc: G06F 18/2413 20230101ALI20250324BHEP

Ipc: G06V 10/82 20220101AFI20250324BHEP

INTG Intention to grant announced

Effective date: 20250423

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

P01 Opt-out of the competence of the unified patent court (upc) registered

Free format text: CASE NUMBER: APP_32322/2025

Effective date: 20250703

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Ref country code: CH

Ref legal event code: F10

Free format text: ST27 STATUS EVENT CODE: U-0-0-F10-F00 (AS PROVIDED BY THE NATIONAL OFFICE)

Effective date: 20251001

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602021039426

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20251001

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1843329

Country of ref document: AT

Kind code of ref document: T

Effective date: 20251001

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251001

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251001

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20260101

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251001

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251001

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251001

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20260101

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20260201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251001

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20260202

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251001

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251001