JP7638962B2 - 軸外カメラを使用した眼追跡および視線推定 - Google Patents

軸外カメラを使用した眼追跡および視線推定 Download PDF

Info

Publication number
JP7638962B2
JP7638962B2 JP2022510817A JP2022510817A JP7638962B2 JP 7638962 B2 JP7638962 B2 JP 7638962B2 JP 2022510817 A JP2022510817 A JP 2022510817A JP 2022510817 A JP2022510817 A JP 2022510817A JP 7638962 B2 JP7638962 B2 JP 7638962B2
Authority
JP
Japan
Prior art keywords
eye
task
data
neural network
layers
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
JP2022510817A
Other languages
English (en)
Japanese (ja)
Other versions
JP2022544696A (ja
Inventor
ジェンヤン ウー，
スリヴィグネシュ ラジェンドラン，
アズ， タレンス ヴァン
ジョエル ジマーマン，
ヴィジャイ バドリナラヤナン，
アンドリュー ラビノビッチ，
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Magic Leap Inc
Original Assignee
Magic Leap Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Magic Leap Inc filed Critical Magic Leap Inc
Publication of JP2022544696A publication Critical patent/JP2022544696A/ja
Application granted granted Critical
Publication of JP7638962B2 publication Critical patent/JP7638962B2/ja
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/197—Matching; Classification
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Molecular Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Optics & Photonics (AREA)
  • Image Analysis (AREA)
  • User Interface Of Digital Computer (AREA)
JP2022510817A 2019-08-19 2020-08-19 軸外カメラを使用した眼追跡および視線推定 Active JP7638962B2 (ja)

Applications Claiming Priority (7)

Application Number Priority Date Filing Date Title
US201962888953P 2019-08-19 2019-08-19
US62/888,953 2019-08-19
US201962926241P 2019-10-25 2019-10-25
US62/926,241 2019-10-25
US201962935584P 2019-11-14 2019-11-14
US62/935,584 2019-11-14
PCT/US2020/047046 WO2021034961A1 (fr) 2019-08-19 2020-08-19 Oculométrie et estimation du regard à l'aide d'une caméra désaxée

Publications (2)

Publication Number Publication Date
JP2022544696A JP2022544696A (ja) 2022-10-20
JP7638962B2 true JP7638962B2 (ja) 2025-03-04

Family

ID=74659542

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2022510817A Active JP7638962B2 (ja) 2019-08-19 2020-08-19 軸外カメラを使用した眼追跡および視線推定

Country Status (5)

Country Link
US (2) US12487664B2 (fr)
EP (1) EP4018288A4 (fr)
JP (1) JP7638962B2 (fr)
CN (1) CN114303117B (fr)
WO (1) WO2021034961A1 (fr)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11798288B2 (en) * 2021-03-16 2023-10-24 Toyota Research Institute, Inc. System and method for generating a training set for improving monocular object detection
US12254406B2 (en) 2021-06-22 2025-03-18 Google Llc Object observation prediction in images using encoder-decoder models
CN117373075B (zh) * 2023-09-05 2024-08-20 北京师范大学 基于眼部特征点和眼部区域分割结果的情感识别数据集
KR20250053502A (ko) * 2023-10-13 2025-04-22 한국전자통신연구원 외형 기반 시선 추정의 일반화를 위한 심층 신경망 학습 방법 및 이를 위한 장치
WO2025096467A1 (fr) * 2023-11-02 2025-05-08 Snap Inc. Correction de décalage de centre de pupille pour calculer le regard

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018035531A1 (fr) 2016-08-19 2018-02-22 Massachusetts Intitute Of Technology Systèmes et procédés pour fournir une gestion d'attribution visuelle
WO2018039269A1 (fr) 2016-08-22 2018-03-01 Magic Leap, Inc. Dispositif d'affichage à réalité augmentée pourvu de capteurs d'apprentissage en profondeur
WO2018063451A1 (fr) 2016-09-29 2018-04-05 Magic Leap, Inc. Réseau neuronal pour segmentation d'image d'œil et estimation de qualité d'image

Family Cites Families (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8896725B2 (en) * 2007-06-21 2014-11-25 Fotonation Limited Image capture device with contemporaneous reference image capture mechanism
CA2750287C (fr) 2011-08-29 2012-07-03 Microsoft Corporation Detection du regard dans un affichage transparent, pres de l'oeil et de realite mixte
US8913789B1 (en) 2012-01-06 2014-12-16 Google Inc. Input methods and systems for eye positioning using plural glints
US9874749B2 (en) 2013-11-27 2018-01-23 Magic Leap, Inc. Virtual and augmented reality systems and methods
US9857591B2 (en) 2014-05-30 2018-01-02 Magic Leap, Inc. Methods and system for creating focal planes in virtual and augmented reality
US10203762B2 (en) 2014-03-11 2019-02-12 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US9684827B2 (en) 2014-03-26 2017-06-20 Microsoft Technology Licensing, Llc Eye gaze tracking based upon adaptive homography mapping
US20170119298A1 (en) 2014-09-02 2017-05-04 Hong Kong Baptist University Method and Apparatus for Eye Gaze Tracking and Detection of Fatigue
JP2016106668A (ja) 2014-12-02 2016-06-20 ソニー株式会社 情報処理装置、情報処理方法、およびプログラム
US9864430B2 (en) 2015-01-09 2018-01-09 Microsoft Technology Licensing, Llc Gaze tracking via eye gaze model
JP2017211891A (ja) 2016-05-27 2017-11-30 ソニー株式会社 情報処理装置、情報処理方法および記録媒体
JP2019519859A (ja) 2016-06-29 2019-07-11 シーイング マシーンズ リミテッド 視線追跡を実行するシステム及び方法
CN106503654A (zh) * 2016-10-24 2017-03-15 中国地质大学（武汉） 一种基于深度稀疏自编码网络的人脸情感识别方法
KR102707594B1 (ko) 2016-11-11 2024-09-19 삼성전자주식회사 홍채 영역 추출 방법 및 장치
KR102707595B1 (ko) * 2016-12-01 2024-09-19 삼성전자주식회사 눈 검출 방법 및 장치
US11132543B2 (en) 2016-12-28 2021-09-28 Nvidia Corporation Unconstrained appearance-based gaze estimation
US10650552B2 (en) * 2016-12-29 2020-05-12 Magic Leap, Inc. Systems and methods for augmented reality
US10521661B2 (en) 2017-09-01 2019-12-31 Magic Leap, Inc. Detailed eye shape model for robust biometric applications
WO2019147677A1 (fr) * 2018-01-24 2019-08-01 Kaban Technologies Llc Suivi de regard à base de caméra d'évènement grâce à des réseaux neuronaux
WO2019157257A1 (fr) * 2018-02-08 2019-08-15 Cognizant Technology Solutions U.S. Corporation Système et procédé d'augmentation avec des pseudo-tâches dans un apprentissage multitâche profond
US11556741B2 (en) 2018-02-09 2023-01-17 Pupil Labs Gmbh Devices, systems and methods for predicting gaze-related parameters using a neural network
CN112040834A (zh) 2018-02-22 2020-12-04 因诺登神经科学公司 眼球跟踪方法及系统
US10948985B2 (en) 2018-03-27 2021-03-16 Nvidia Corporation Retina space display stabilization and a foveated display for augmented reality
US10558895B2 (en) 2018-03-30 2020-02-11 Tobii Ab Deep learning for three dimensional (3D) gaze prediction
CN112400148B (zh) 2018-06-22 2024-12-24 奇跃公司 使用离轴相机执行眼睛跟踪的方法和系统
CN108809522B (zh) * 2018-07-09 2021-09-14 上海大学 多码的深度学习译码器的实现方法
CN109274812B (zh) * 2018-08-13 2021-02-02 Oppo（重庆）智能科技有限公司 超声波手势控制电子装置的方法及相关产品
US11113842B2 (en) * 2018-12-24 2021-09-07 Samsung Electronics Co., Ltd. Method and apparatus with gaze estimation
IL264530B2 (en) * 2019-01-29 2024-07-01 Eyeway Vision Ltd Eye tracking device and a method thereof
KR102669768B1 (ko) * 2020-12-17 2024-05-29 진 랩스 인코포레이티드 동공 검출 및 시선 추적을 위한 이벤트 카메라 시스템

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018035531A1 (fr) 2016-08-19 2018-02-22 Massachusetts Intitute Of Technology Systèmes et procédés pour fournir une gestion d'attribution visuelle
WO2018039269A1 (fr) 2016-08-22 2018-03-01 Magic Leap, Inc. Dispositif d'affichage à réalité augmentée pourvu de capteurs d'apprentissage en profondeur
WO2018063451A1 (fr) 2016-09-29 2018-04-05 Magic Leap, Inc. Réseau neuronal pour segmentation d'image d'œil et estimation de qualité d'image

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Clement Playout; Renaud Duval; Farida Cheriet，A Novel Weakly Supervised Multitask Architecture for Retinal Lesions Segmentation on Fundus Images，IEEE Transactions on Medical Imaging，米国，ＩＥＥＥ，2019年03月20日，Volume: 38, Issue: 10，p.2434－p.2444，https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8672120
松井 涼，山下 隆義，藤吉 弘亘，Ｓｅｐａｒａｔｉｏｎ Ｍｕｌｔｉ－ｔａｓｋ Ｎｅｔｗｏｒｋｓによる顔器官点と顔属性の同時推定，電子情報通信学会技術研究報告，日本，2018年12月06日，Ｖｏｌ．１１８ Ｎｏ．３６２，p.39－p.44

Also Published As

Publication number Publication date
WO2021034961A1 (fr) 2021-02-25
EP4018288A1 (fr) 2022-06-29
CN114303117A (zh) 2022-04-08
JP2022544696A (ja) 2022-10-20
US12487664B2 (en) 2025-12-02
CN114303117B (zh) 2025-10-17
EP4018288A4 (fr) 2022-10-12
US20260056608A1 (en) 2026-02-26
US20220244781A1 (en) 2022-08-04

Similar Documents

Publication Publication Date Title
US11861873B2 (en) Event camera-based gaze tracking using neural networks
US12062297B2 (en) System and method for three-dimensional augmented reality guidance for use of equipment
CN115427758B (zh) 具有精确共享地图的交叉现实系统
JP7645041B2 (ja) 軸外カメラを使用して眼追跡を実施するための方法およびシステム
US12487664B2 (en) Eye tracking and gaze estimation using off-axis camera
CN105900141B (zh) 将闪光映射到光源
US11170212B2 (en) Sensor fusion eye tracking
CN112805750A (zh) 跨现实系统
Wu et al. Magiceyes: A large scale eye gaze estimation dataset for mixed reality
US20230036611A1 (en) Gaze estimation apparatus, gaze estimation method, model generation apparatus, and model generation method
KR20250175276A (ko) 시선 온라인 학습
KR102966829B1 (ko) 걸음걸이프린트 모방을 갖는 ar 데이터 시뮬레이션
CN121236811A (zh) 一种基于头部运动的移动眼动追踪方法及系统
Du Plessis Graphical processing unit assisted image processing for accelerated eye tracking

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20230725

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20240514

RD03 Notification of appointment of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7423

Effective date: 20240613

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20240617

RD04 Notification of resignation of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7424

Effective date: 20240626

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20240913

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20250121

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20250219

R150 Certificate of patent or registration of utility model

Ref document number: 7638962

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150