EP4325450B1 - Verfahren und system zur analyse der augenbewegung - Google Patents

Verfahren und system zur analyse der augenbewegung

Info

Publication number
EP4325450B1
EP4325450B1 EP22216993.0A EP22216993A EP4325450B1 EP 4325450 B1 EP4325450 B1 EP 4325450B1 EP 22216993 A EP22216993 A EP 22216993A EP 4325450 B1 EP4325450 B1 EP 4325450B1
Authority
EP
European Patent Office
Prior art keywords
eye movement
scene video
gaze
coordinate
scene
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP22216993.0A
Other languages
English (en)
French (fr)
Other versions
EP4325450A1 (de
EP4325450C0 (de
Inventor
Qichao Zhao
Ran YANG
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kingfar International Inc
Original Assignee
Kingfar International Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kingfar International Inc filed Critical Kingfar International Inc
Publication of EP4325450A1 publication Critical patent/EP4325450A1/de
Application granted granted Critical
Publication of EP4325450B1 publication Critical patent/EP4325450B1/de
Publication of EP4325450C0 publication Critical patent/EP4325450C0/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/19—Sensors therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/60—Extraction of image or video features relating to illumination properties, e.g. using a reflectance or lighting model
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/809—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data
    • G06V10/811—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data the classifiers operating on different input data, e.g. multi-modal recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image

Definitions

  • the present application relates to the technical field of eye movement data analysis, and in particular to an eye movement analysis method, system, computer apparatus and computer-readable storage medium.
  • the eye movement tracking technology uses electronic and optical detection methods to acquire eye movement gaze data of the eye movement behavior of a target user, and then studies and analyzes human behavior and eye movement behavior.
  • AOI Area of Interest
  • the inventors have found that when studying and analyzing the eye movement behavior of a target user in a preset environment, since the shape and position of a research object in a scene video tend to change, a researcher needs to adjust the shape and position of an area of interest on a frame-by-frame basis to ensure that the area of interest always covers the research object in the scene video, and then maps the eye movement data of the target user into each frame image of the scene video; therefore, when the shape and position of the research object in the scene video change dynamically, it takes a lot of time to study the eye movement behavior of the target user.
  • Method for advanced visual analytics of eye tracking data of Panetta et al. describes a eye-tracking data visualization and analysis system that allows for automatic recognition of independent objects within field-of-vision, using deep-learning-based semantic segmentation, which recolors the fixated objects-of-interest by integrating gaze fixation information with semantic maps.
  • US 2021/350184A1 describes a training system for a deep neural network and method of training, which comprises receiving, from an eye-tracking system associated with a sensor, an image frame captured while an operator is controlling a vehicle; receiving, from the eye-tracking system, eyeball gaze data corresponding to the image frame; and iteratively training the deep neural network to determine an object of interest depicted within the image frame based on the eyeball gaze data.
  • a method includes determining eye tracking data associated with a user of a vehicle from a vehicle sensor and determining a field of vision of the user based on the eye tracking data.
  • the method includes determining object data associated with an object within the field of vision and identifying the object within the field of vision based on the object data to produce an object identification hit.
  • the method includes storing the object identification hit in memory.
  • the present application provides an eye movement analysis method and system in order to reduce the time required to study the eye movement behavior of a target user when the shape and position of the research object in the scene video change dynamically.
  • the present application provides an eye movement analysis method, using the following technical solution:
  • the following can be realized: acquiring a first scene video seen by a target user in a preset environment, and acquiring eye movement gaze data of the target user in the environment; based on the deep learning algorithm, performing semantic segmentation on the first scene video to realize the automatic division and identification of the eye movement area of interest of the first scene video so as to obtain the second scene video, and superposing the eye movement gaze data with the second scene video to obtain the corresponding gaze pixel point of the eye movement gaze data in the second scene video. Therefore, the eye movement data index, such as gaze times, gaze times, etc., of the target user gazing at the eye movement area of interest in the environment can be obtained and output.
  • the present application uses the deep learning algorithm, performs semantic segmentation on the first scene video, and automatically divides the eye movement area of interest, to a certain extent, researchers can avoid spending a lot of time adjusting the shape and size of the eye movement area of interest frame by frame, and further, the time required to study the eye movement behavior of the target user can be reduced when the shape and position of the research object in the scene video change dynamically.
  • the deep learning algorithm uses DeepLab, EncNet, SegNet, or PSPNet.
  • Performing semantic segmentation on the first scene video comprises:
  • semantic segmentation can assign a semantic tag to each pixel in the first scene video, indicating the category of each pixel; pixels of the same category having the same semantic tag are automatically drawn into the eye movement area of interest.
  • Superposing the eye movement gaze data with the second scene video to obtain a gaze pixel point corresponding to the eye movement gaze data in the second scene video includes:
  • the first coordinate system is one three-dimensional coordinate system, and the first coordinate system can take the scene camera as the origin of the coordinate, and the position of the gazed object can be represented by the three-dimensional coordinate point or the vector of the line of sight of the target user;
  • the second coordinate system is one two-dimensional coordinate system, and the second coordinate system can take the central point of the second scene video as the origin of coordinate, and after superposing the eye movement gaze data with the second scene video, the gazed object of the target user can be corresponding to one gaze pixel in the second scene video.
  • determining the gaze pixel point corresponding to each frame image in the second scene video and outputting, in combination with a time sequence, eye movement data index of the target user gazing at the eye movement area of interest include:
  • each pixel in the second scene video has a semantic tag; after a gazed object of a target user is corresponding to one gaze pixel in the second scene video, the semantic tag category of the gaze pixel can learn an eye movement area of interest gazed by the target user; by determining the gaze pixel on each frame image of the second scene video, an eye movement data index of the target user gazing at the eye movement area of interest can be output.
  • determining indexes of the eye movement area of interest comprise: the first gaze time, the number of visits, the total visit duration, the gaze times, the total gaze duration, and the average gaze duration.
  • the present application provides an eye movement analysis system, as defined in the subject-matter of claim 5.
  • the present application provides a computer apparatus, using the following technical solution: a computer apparatus, including memory, a processor, and a computer program stored on the memory and executable on the processor.
  • the processor when executing a program, realizes the method as in the first aspect.
  • the present application provides a computer-readable storage medium, using the technical solution as follows: the computer-readable storage medium stores a computer program capable of being loaded by a processor and executing the method of the first aspect.
  • the present application at least includes the following beneficial effects.
  • the following can be realized: acquiring a first scene video seen by a target user in a preset environment, and acquiring eye movement gaze data of the target user in the environment; based on the deep learning algorithm, performing semantic segmentation on the first scene video to realize the automatic division and identification of the eye movement area of interest of the first scene video so as to obtain the second scene video, and superposing the eye movement gaze data with the second scene video to obtain the corresponding gaze pixel point of the eye movement gaze data in the second scene video. Therefore, the eye movement data index, such as gaze times, gaze times, etc. of the target user gazing at the eye movement area of interest in the environment can be obtained and output.
  • the present application reduces the amount of data processing by researchers, thus reducing the time for research and analysis of eye movement behaviors of a target user in the preset environment.
  • an embodiment of the present application discloses an eye movement analysis method, comprising the steps of S 10-S40:
  • pixels having the same semantic tag are divided into the same eye movement area of interest.
  • the deep learning algorithm uses DeepLab, EncNet, SegNet, or PSPNet; likewise, the second scene video comprises multiple frames of second scene images, and the second scene images correspond to the first scene images on a one-to-one basis in a time sequence.
  • the position of the gazed object can be represented by a three-dimensional coordinate point or a vector of the line of sight of a target user.
  • the eye movement gaze data is converted into a coordinate system of the second scene video so that the gaze position of the eye movement gaze data is represented by the position of the gaze pixel point; in the present embodiment, each set of the eye movement gaze data corresponds to each frame of the second scene image of the second scene video on a one-to-one basis, and the gaze pixel points at which each set of eye movement gaze data gazes in the corresponding each frame of the second scene graph are determined.
  • the time sequence can be based on the frame rate and the number of frames of the second scene video, for example, the frame rate of the second scene video is 30 frames/second, which is 3000 frames in total, and then the second scene video having 100 seconds, and the residence time of each frame image in the second scene video being 1/30 second can be obtained; as shown in FIG. 4 , it is determined that a gaze pixel point corresponds to each frame image in the second scene video, and combining with the time sequence to output the eye movement data index of the target user gazing at the eye movement area of interest comprises sub-steps S401-S402:
  • the eye movement data index includes the first gaze time, the number of visits, the total visit duration, the gaze times, the total gaze duration, and the average gaze duration.
  • the semantic segmentation of the first scene video is performed by using the deep learning algorithm to achieve automatic division and identification of the eye movement area of interest of the first scene video and obtain the second scene video, so that, to a certain extent researchers can avoid spending a lot of time adjusting the shape and size of the eye movement area of interest frame by frame, and then the eye movement gaze data is superposed with the second scene video to obtain the gaze pixel points corresponding to the eye movement gaze data in the second scene video. Therefore, the output eye movement data index of the eye movement area of interest of the target user in the environment can be obtained; therefore, the present application can reduce the time for studying the eye movement behavior of a target user when the shape and position of a research object in a scene video change dynamically.
  • An embodiment of the present application also provides an eye movement analysis system.
  • an eye movement analysis system comprises a first acquisition module, a second acquisition module, a semantic segmentation module, a superposition module, and an output module.
  • the first acquisition module is configured for acquiring a first scene video seen by a target user in a preset environment;
  • the second acquisition module is configured for acquiring eye movement gaze data of the target user in an environment when the first acquisition module acquires the first scene video;
  • the semantic segmentation module is configured for receiving the first scene video, and performing semantic segmentation on the first scene video by using a deep learning algorithm so as to obtain a second scene video, wherein the second scene video is divided to have an eye movement area of interest;
  • the superposition module is configured for receiving the second scene video and the eye movement gaze data, and superposing the eye movement gaze data with the second scene video so as to obtain a gaze pixel point corresponding to the eye movement gaze data in the second scene video;
  • the output module is configured for determining the eye movement area of interest where the gaze pixel point is located, and outputting, in combination with the time sequence,
  • the first acquisition module acquires a first scene video seen by a target user in a preset environment, and at the same time, the second acquisition module acquires eye movement gaze data of the target user in the environment;
  • the semantic segmentation module based on the deep learning algorithm, performs semantic segmentation on the first scene video to realize the automatic division and identification of the eye movement area of interest of the first scene video so as to obtain the second scene video, and then the superposition module superposes the eye movement gaze data with the second scene video to obtain the corresponding gaze pixel point of the eye movement gaze data in the second scene video; therefore, the output module can output the eye movement data index of the target user gazing at the eye movement area of interest in the environment; since the present application uses the deep learning algorithm, performs semantic segmentation on the first scene video, and automatically divides the eye movement area of interest, to a certain extent, researchers can avoid spending a lot of time adjusting the shape and size of the eye movement area of interest frame by frame, and further, the time required to study the eye movement behavior of the target user
  • the superposition module comprises a acquiring unit, a conversion unit, and a corresponding unit; wherein the acquiring unit is configured for acquiring a first coordinate point corresponding to the eye movement gaze data in the first coordinate system; the conversion unit is configured for converting the first coordinate point into a second coordinate system of a second scene video to obtain a second coordinate point; the corresponding unit is configured for corresponding the second coordinate point to a pixel point of the second scene video, and obtaining a gaze pixel point corresponding to the eye movement gaze data in the second scene video.
  • the eye movement gaze data of the target user is converted into the coordinate system of the second scene video, and the eye movement gaze data corresponds to the gaze pixel point of the second scene video.
  • the eye movement analysis system of the present application is capable of implementing any method of the eye movement analysis methods described above, and the specific working process of the eye movement analysis system refers to the corresponding process in the above method embodiments.
  • An embodiment of the present application also provides a computer apparatus.
  • the computer apparatus includes a memory, a processor, and a computer program stored on the memory and executable on the processor.
  • the processor when executing a program, realizes the method as in the first aspect.
  • An embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program capable of being loaded by a processor and executing the method of the first aspect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Computational Linguistics (AREA)
  • Databases & Information Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Human Computer Interaction (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Claims (6)

  1. Verfahren zur Analyse von Augenbewegungen, umfassend:
    Erfassen eines ersten Szenenvideos, das von einem Zielbenutzer in einer voreingestellten Umgebung gesehen wird, und Erfassen von Blickbewegungsdaten des Zielbenutzers in der Umgebung;
    Durchführung einer semantischen Segmentierung des ersten Szenenvideos auf der Grundlage eines Deep-Learning-Algorithmus, um ein zweites Szenenvideo zu erhalten; wobei das zweite Szenenvideo so unterteilt wird, dass es einen interessierenden Bereich für die Augenbewegung aufweist;
    Zuweisen eines semantischen Tags zu jedem Pixel im ersten Szenenvideo;
    Aufteilen eines interessierenden Augenbewegungsbereichs auf der Grundlage des semantischen Tags; wobei Pixel mit demselben semantischen Tag demselben interessierenden Augenbewegungsbereich zugeordnet werden;
    Überlagern der Blickbewegungsdaten mit dem zweiten Szenenvideo, um einen Blickpixelpunkt zu erhalten, der den Blickbewegungsdaten im zweiten Szenenvideo entspricht; und
    Bestimmen des Blickpunkts, der jedem Einzelbild im zweiten Szenenvideo entspricht, und Ausgeben eines Blickbewegungsdatenindex des Zielbenutzers, der den interessierenden Blickbewegungsbereich betrachtet, in Kombination mit einer Zeitsequenz;
    dadurch gekennzeichnet, dass die voreingestellte Umgebung ein Forschungsobjekt enthält;
    wobei das Überlagern der Blickbewegungsdaten mit dem zweiten Szenenvideo, um einen Blickpunkt zu erhalten, der den Blickbewegungsdaten im zweiten Szenenvideo entspricht, Folgendes umfasst:
    Erfassen eines ersten Koordinatenpunkts, der den Blickbewegungsdaten in einem ersten Koordinatensystem entspricht, wobei das erste Koordinatensystem ein dreidimensionales Koordinatensystem ist und wobei das erste Koordinatensystem eine Szenenkamera, die das erste Szenenvideo aufnimmt, als Koordinatenursprung nimmt, eine vertikale Richtung als Koordinaten-Z-Achse nimmt, eine horizontale Aufnahmerichtung, auf die die Szenenkamera direkt ausgerichtet ist, als Koordinaten-X-Achse nimmt und eine Richtung senkrecht zur horizontalen Aufnahmerichtung der Szenenkamera als Koordinaten-Y-Achse nimmt;
    Umwandeln des ersten Koordinatenpunkts in ein zweites Koordinatensystem des zweiten Szenenvideos, um einen zweiten Koordinatenpunkt zu erhalten, wobei das zweite Koordinatensystem ein zweidimensionales Koordinatensystem ist und das zweite Koordinatensystem einen zentralen Pixelpunkt des zweiten Szenenvideos als Koordinatenursprung nimmt, eine Querrichtung der Pixel als Koordinaten-X-Achse nimmt und eine Längsrichtung der Pixel als Koordinaten-Y-Achse nimmt; und
    Zuordnen des zweiten Koordinatenpunkts zu einem Pixelpunkt im zweiten Szenenvideo, um den Blickpunkt zu erhalten, der den Blickbewegungsdaten im zweiten Szenenvideo entspricht.
  2. Verfahren zur Analyse von Augenbewegungen nach Anspruch 1, wobei der Deep-Learning-Algorithmus DeepLab, EncNet, SegNet oder PSPNet verwendet.
  3. Verfahren zur Analyse von Augenbewegungen nach Anspruch 1, wobei das Bestimmen des Blickpunktpixels, das jedem Einzelbild im zweiten Szenenvideo entspricht, und das Ausgeben, in Kombination mit einer Zeitsequenz, des Augenbewegungsdatenindex des Zielbenutzers, der auf den interessierenden Augenbewegungsbereich blickt, Folgendes umfasst:
    Bestimmen einer Reihenfolge und einer Anzahl, in der semantische Tags von Blickpixelpunkten in jedem Einzelbild des zweiten Szenenvideos in der Zeitsequenz erscheinen; und
    Berechnen und Ausgeben des Augenbewegungsdatenindex auf der Grundlage der Reihenfolge und der Anzahl, in der die semantischen Tags in der Zeitsequenz erscheinen.
  4. Verfahren zur Analyse von Augenbewegungen nach einem der Ansprüche 1 bis 3, wobei der Augenbewegungsdatenindex Folgendes umfasst: erste Blickzeit, Anzahl der Besuche, Gesamtbesuchsdauer, Blickzeiten, Gesamtblickdauer und durchschnittliche Blickdauer.
  5. Augenbewegungsanalysesystem, das Folgendes umfasst: ein erstes Erfassungsmodul, ein zweites Erfassungsmodul, ein semantisches Segmentierungsmodul, ein Überlagerungsmodul und ein Ausgabemodul, dadurch gekennzeichnet, dass
    das erste Erfassungsmodul zum Erfassen eines ersten Szenenvideos konfiguriert ist, das von einem Zielbenutzer in einer voreingestellten Umgebung gesehen wird;
    das zweite Erfassungsmodul so konfiguriert ist, dass es gleichzeitig Blickbewegungsdaten des Zielbenutzers in der Umgebung erfasst, wenn das erste Erfassungsmodul das erste Szenenvideo erfasst;
    das semantische Segmentierungsmodul so konfiguriert ist, dass es das erste Szenenvideo empfängt und eine semantische Segmentierung des ersten Szenenvideos auf der Grundlage eines Deep-Learning-Algorithmus durchführt, um ein zweites Szenenvideo zu erhalten, wobei das zweite Szenenvideo so unterteilt ist, dass jedem Pixel im ersten Szenenvideo ein semantisches Tag zugewiesen wird, und der interessierende Bereich der Augenbewegung auf der Grundlage des semantischen Tags unterteilt wird, wobei Pixel mit demselben semantischen Tag in denselben interessierenden Bereich der Augenbewegung unterteilt werden;
    das Überlagerungsmodul so konfiguriert ist, dass es das zweite Szenenvideo und die Blickbewegungsdaten empfängt und die Blickbewegungsdaten mit dem zweiten Szenenvideo überlagert, um einen Blickpixelpunkt zu erhalten, der den Blickbewegungsdaten im zweiten Szenenvideo entspricht;
    das Ausgabemodul so konfiguriert ist, dass es den Blickpunkt jedes Einzelbildes im zweiten Szenenvideo bestimmt und in Kombination mit einer Zeitsequenz einen Augenbewegungsdatenindex des Zielbenutzers ausgibt, der den interessierenden Augenbewegungsbereich betrachtet;
    wobei das Überlagerungsmodul eine Erfassungseinheit, eine Umwandlungseinheit und eine Zuordnungseinheit umfasst; wobei
    die Erfassungseinheit so konfiguriert ist, dass sie einen ersten Koordinatenpunkt erfasst, der den Blickbewegungsdaten in einem ersten Koordinatensystem entspricht, wobei das erste Koordinatensystem ein dreidimensionales Koordinatensystem ist und das erste Koordinatensystem eine Szenenkamera, die das erste Szenenvideo aufnimmt, als Koordinatenursprung nimmt, eine vertikale Richtung als Koordinaten-Z-Achse nimmt, eine horizontale Aufnahmerichtung, auf die die Szenenkamera direkt ausgerichtet ist, als Koordinaten-X-Achse nimmt und eine Richtung senkrecht zur horizontalen Aufnahmerichtung der Szenenkamera als Koordinaten-Y-Achse nimmt;
    die Umwandlungseinheit so konfiguriert ist, dass sie den ersten Koordinatenpunkt in ein zweites Koordinatensystem des zweiten Szenenvideos umwandelt, um einen zweiten Koordinatenpunkt zu erhalten, wobei das zweite Koordinatensystem ein zweidimensionales Koordinatensystem ist und das zweite Koordinatensystem einen zentralen Pixelpunkt des zweiten Szenenvideos als Koordinatenursprung nimmt, eine Queranordnungsrichtung der Pixel als Koordinaten-X-Achse nimmt und eine Längsanordnungsrichtung der Pixel als Koordinaten-Y-Achse nimmt;
    die Zuordnungseinheit so konfiguriert ist, dass sie den zweiten Koordinatenpunkt einem Pixelpunkt im zweiten Szenenvideo zuordnet, um den Blickpunkt zu erhalten, der den Blickbewegungsdaten im zweiten Szenenvideo entspricht.
  6. Computerlesbares Speichermedium mit einem darauf gespeicherten Computerprogramm, dadurch gekennzeichnet, dass das Computerprogramm so konfiguriert ist, dass es von einem Prozessor geladen wird, um das Verfahren nach einem der Ansprüche 1 bis 4 auszuführen.
EP22216993.0A 2022-08-19 2022-12-28 Verfahren und system zur analyse der augenbewegung Active EP4325450B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211001336.0A CN115661913A (zh) 2022-08-19 2022-08-19 一种眼动分析方法及系统

Publications (3)

Publication Number Publication Date
EP4325450A1 EP4325450A1 (de) 2024-02-21
EP4325450B1 true EP4325450B1 (de) 2025-10-29
EP4325450C0 EP4325450C0 (de) 2025-10-29

Family

ID=84688171

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22216993.0A Active EP4325450B1 (de) 2022-08-19 2022-12-28 Verfahren und system zur analyse der augenbewegung

Country Status (3)

Country Link
US (1) US12315178B2 (de)
EP (1) EP4325450B1 (de)
CN (1) CN115661913A (de)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117315763A (zh) * 2023-09-27 2023-12-29 哈尔滨工业大学建筑设计研究院有限公司 一种基于问卷调研和眼动追踪的调研方式在校园恢复性元素识别中的应用方法
CN118865384B (zh) * 2024-07-01 2025-02-18 重庆菲利信科技有限公司 一种用于自动驾驶场景图片的快速标注方法和装置
CN119067924B (zh) * 2024-08-08 2025-08-29 中山大学中山眼科中心 眼动视频数据处理方法、装置、计算机设备和存储介质
CN119148860B (zh) * 2024-11-11 2026-04-17 北京津发科技股份有限公司 人因智能用户注视分析方法、装置和系统、边缘计算设备
CN120452052B (zh) * 2025-07-09 2025-09-16 南航科技(广东横琴)有限公司 一种三维环境下的眼动注视点识别方法、系统及设备

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
RU2696042C2 (ru) * 2017-12-11 2019-07-30 Федеральное государственное бюджетное образовательное учреждение высшего образования "Московский государственный университет имени М.В. Ломоносова" (МГУ) Способ и система для регистрации движения глаз
CN108665496B (zh) * 2018-03-21 2021-01-26 浙江大学 一种基于深度学习的端到端的语义即时定位与建图方法
US20200064912A1 (en) 2018-08-22 2020-02-27 Ford Global Technologies, Llc Eye gaze tracking of a vehicle passenger
CN110276334A (zh) * 2019-06-28 2019-09-24 海马汽车有限公司 一种针对用户车辆使用情况的分析方法及系统
US11225266B2 (en) * 2019-08-20 2022-01-18 Toyota Motor Engineering & Manufacturing North America, Inc. Systems and methods for improving visual scanning behavior associated with controlling a vehicle
US11475249B2 (en) * 2020-04-30 2022-10-18 Electronic Arts Inc. Extending knowledge data in machine vision
US11604946B2 (en) * 2020-05-06 2023-03-14 Ford Global Technologies, Llc Visual behavior guided object detection
CN112949409A (zh) * 2021-02-02 2021-06-11 首都师范大学 基于感兴趣客体的眼动数据分析方法及装置、计算机设备
CN113391699B (zh) * 2021-06-10 2022-06-21 昆明理工大学 一种基于动态眼动指标的眼势交互模型方法
WO2023272453A1 (zh) * 2021-06-28 2023-01-05 华为技术有限公司 视线校准方法及装置、设备、计算机可读存储介质、系统、车辆
CN114120432B (zh) * 2021-11-17 2025-02-14 湖北大学 基于视线估计的在线学习注意力跟踪方法及其应用

Also Published As

Publication number Publication date
CN115661913A (zh) 2023-01-31
US12315178B2 (en) 2025-05-27
US20240062400A1 (en) 2024-02-22
EP4325450A1 (de) 2024-02-21
EP4325450C0 (de) 2025-10-29

Similar Documents

Publication Publication Date Title
EP4325450B1 (de) Verfahren und system zur analyse der augenbewegung
Moreno-Arjonilla et al. Eye-tracking on virtual reality: a survey
US11503998B1 (en) Method and a system for detection of eye gaze-pattern abnormalities and related neurological diseases
CN109086726B (zh) 一种基于ar智能眼镜的局部图像识别方法及系统
CN108919958B (zh) 一种图像传输方法、装置、终端设备及存储介质
US10831268B1 (en) Systems and methods for using eye tracking to improve user interactions with objects in artificial reality
Pfeiffer Measuring and visualizing attention in space with 3D attention volumes
US11010646B2 (en) Object tracking assisted with hand or eye tracking
KR101471488B1 (ko) 시선 추적 장치 및 방법
CN108427503A (zh) 人眼追踪方法及人眼追踪装置
US12299192B2 (en) Visual assistance
Mehrubeoglu et al. Real-time eye tracking using a smart camera
CN112099615A (zh) 注视信息确定方法、装置、眼球追踪设备及存储介质
Hibbard Virtual reality for vision science
CN117152841B (zh) 基于双流滑动注意力网络的学生注意力评价方法
WO2024228743A1 (en) Camera focus adjustment
CN112651270A (zh) 一种注视信息确定方法、装置、终端设备及展示对象
US20250173893A1 (en) Electronic device for detecting gaze of user and operating method thereof
CN112528713B (zh) 一种注视点估计方法、系统、处理器及设备
Parada et al. ExpertEyes: Open-source, high-definition eyetracking
CN113780414B (zh) 眼动行为分析方法、图像渲染方法、组件、装置和介质
WO2024198121A1 (zh) 一种跨平台的注视点实时追踪方法、装置及智能终端
CN116665288A (zh) 眼球追踪方法、装置和存储介质
Soundararajan et al. Study on Eye Gaze Detection Using Deep Transfer Learning Approaches.
Mehrubeoglu et al. Capturing reading patterns through a real-time smart camera iris tracking system

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20221228

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06V0040190000

Ipc: G06N0003046400

Ref country code: DE

Ref legal event code: R079

Ref document number: 602022023889

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06V0040190000

Ipc: G06N0003046400

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 3/0464 20230101AFI20250703BHEP

Ipc: G06N 3/0455 20230101ALI20250703BHEP

Ipc: G06V 40/18 20220101ALI20250703BHEP

Ipc: G06V 20/52 20220101ALI20250703BHEP

Ipc: G06V 40/19 20220101ALI20250703BHEP

Ipc: G06V 20/70 20220101ALI20250703BHEP

Ipc: G06V 10/25 20220101ALI20250703BHEP

Ipc: G06V 10/82 20220101ALI20250703BHEP

Ipc: G06V 10/80 20220101ALI20250703BHEP

Ipc: G06V 10/60 20220101ALI20250703BHEP

Ipc: G06N 3/08 20230101ALN20250703BHEP

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 3/0464 20230101AFI20250707BHEP

Ipc: G06N 3/0455 20230101ALI20250707BHEP

Ipc: G06V 40/18 20220101ALI20250707BHEP

Ipc: G06V 20/52 20220101ALI20250707BHEP

Ipc: G06V 40/19 20220101ALI20250707BHEP

Ipc: G06V 20/70 20220101ALI20250707BHEP

Ipc: G06V 10/25 20220101ALI20250707BHEP

Ipc: G06V 10/82 20220101ALI20250707BHEP

Ipc: G06V 10/80 20220101ALI20250707BHEP

Ipc: G06V 10/60 20220101ALI20250707BHEP

Ipc: G06N 3/08 20230101ALN20250707BHEP

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 3/0464 20230101AFI20250711BHEP

Ipc: G06N 3/0455 20230101ALI20250711BHEP

Ipc: G06V 40/18 20220101ALI20250711BHEP

Ipc: G06V 20/52 20220101ALI20250711BHEP

Ipc: G06V 40/19 20220101ALI20250711BHEP

Ipc: G06V 20/70 20220101ALI20250711BHEP

Ipc: G06V 10/25 20220101ALI20250711BHEP

Ipc: G06V 10/82 20220101ALI20250711BHEP

Ipc: G06V 10/80 20220101ALI20250711BHEP

Ipc: G06V 10/60 20220101ALI20250711BHEP

Ipc: G06N 3/08 20230101ALN20250711BHEP

INTG Intention to grant announced

Effective date: 20250723

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: F10

Free format text: ST27 STATUS EVENT CODE: U-0-0-F10-F00 (AS PROVIDED BY THE NATIONAL OFFICE)

Effective date: 20251029

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602022023889

Country of ref document: DE

U01 Request for unitary effect filed

Effective date: 20251126

U07 Unitary effect registered

Designated state(s): AT BE BG DE DK EE FI FR IT LT LU LV MT NL PT RO SE SI

Effective date: 20251202

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: AT

Payment date: 20260113

Year of fee payment: 4

U20 Renewal fee for the european patent with unitary effect paid

Year of fee payment: 4

Effective date: 20251230

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251029

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20260129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251029

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20260129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20260228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20251029