EP4577892A1 - Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage - Google Patents

Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage

Info

Publication number
EP4577892A1
EP4577892A1 EP23755433.2A EP23755433A EP4577892A1 EP 4577892 A1 EP4577892 A1 EP 4577892A1 EP 23755433 A EP23755433 A EP 23755433A EP 4577892 A1 EP4577892 A1 EP 4577892A1
Authority
EP
European Patent Office
Prior art keywords
depth camera
calibration
projection
user
interaction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP23755433.2A
Other languages
German (de)
English (en)
Inventor
Martin Seiler
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ameria AG
Original Assignee
Ameria AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ameria AG filed Critical Ameria AG
Publication of EP4577892A1 publication Critical patent/EP4577892A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures

Definitions

  • GLAMOS https://www.kickstarter.com/projects/300948436/glamos-bring-your-touchless- screens-to-life), sets out to use lidar technology to turn screens into interactive touch screens.
  • Touchjet https://www.touchjet.com/wave-lily/ aims at turning flat screen TVs into a tablet for collaboration and interactive presentations using a digital pen.
  • a computer-implemented touchless interaction enablement method may comprise projecting, using a projection means, a projection scene onto a projection target.
  • the method may comprise capturing, using a calibration depth camera, at least part of the projection scene and generating calibration data.
  • the calibration data may at least partially include content of the captured projection scene.
  • the method may comprise calibrating at least one interaction depth camera based at least in part on the calibration data, thereby enabling the at least one interaction depth camera to detect an input object for recognizing a desired touchless user input made by a user using the input object.
  • projection means may include any type of means capable of providing a projection scene, including but not limited to projectors or projector arrays.
  • the projection means may emit light for this purpose.
  • the depth cameras according to the present invention may in particular be infrared stereovision depth cameras.
  • the calibration depth camera may be placed at or near the projection means such that it can observe the projection scene.
  • the interaction depth camera may be placed at or near the projection means such that it can observe an area in which a user may, or is expected to, provide user-input through an input object.
  • the term "calibrating” is understood as a calculation and/or configuration process in which, in particular, the interaction depth camera is enabled to correctly recognize user-input.
  • calibration is performed in such a way that the calibration depth camera generates calibration data with which the interaction depth camera is calibrated.
  • one depth camera calibrates another depth camera.
  • the calibration data may be used to generate a spatial input area that is observed by the interaction depth camera.
  • the input area may be mapped to the projection scene such that an input object detected by the interaction depth camera in the input area produces an effect at a correct desired portion of the projection scene.
  • the input area may be a frustum of a cone or a frustum of a pyramid.
  • Calibration may further include calibration of the projection means.
  • the projection scene can be at least partially adjusted to the input area to increase the precision of user-input detection. This may be necessary in particular if the input area of the interaction depth camera cannot be fully adapted to the projection scene due to conflicting positions of the projection means and the interaction depth camera. By at least partially adjusting the projection scene, the conflict can be effectively resolved.
  • One aspect of the calibration may include the detection of the edges of the projection scene and the definition of an input area of the interaction depth camera corresponding to the detected edges of the projection scene.
  • user input is detected inside the input area. Outside the input area it may be provided that no user input is detected, although the input device in within a field of view of the interaction depth camera.
  • each depth camera may be modelled to observe a certain area which may have the form of a frustum of a cone or a frustum of a pyramid.
  • the projection means may be modelled as an inverse depth camera, so to say, providing content instead of observing, though light emitted by the projection means may be modelled as a frustum of a cone or a frustum of a pyramid.
  • a desired touchless user input may for example be a marking of certain positions on the projection scene, for example, in a similar way as a laser pointer does in PowerPoint presentations or other lectures.
  • a hovering cursor can be displayed overlaying the projection scene.
  • the cursor can have animations that show the recognition of the gesture, depending on the gesture. For example, a click gesture can cause the cursor to change size momentarily, or a circle can be displayed around the cursor.
  • desired touchless user input may be any types of gestures, such as swipe gestures, tap gestures, and other more complex gestures. It may further be provided that a user configures own, i.e. user-defined, gestures and assigns a specific intended user input to these gestures, which is recognized accordingly.
  • machine-readable calibration pattern may include all types of suitable patterns that can be used as a calculation basis for a calibration.
  • a machine-readable calibration pattern can be designed in such a way that it has uniquely distinctive sections that make it possible to derive the respective positions and any distortions of a projection scene.
  • a reference calibration pattern may be stored in a memory against which the captured calibration pattern is compared for calibration purposes, in particular to detect distortions of the projection scene and take them into account for calibration.
  • the method enables the provision of a very user-friendly and intuitive operation and/or interaction with a user interface. It is particularly advantageous that even a commercially available projector can be equipped to provide touchless user interaction by using depth cameras as proposed in the method of the invention. In addition to the high user-friendliness and the intuitive operation, the hygiene of the user interaction is further increased, since no physical touches are required for a user interaction.
  • detecting the input object includes determining a pointing direction and/or a position of the input object, using the interaction depth camera. Determining a pointing direction may be performed based on the shape of the input object. For example, if an elongated input object, such as a pen or a finger, is used, the main axis of the elongated input object may be determined as the pointing direction towards the projection scene.
  • detecting the input object may comprise determining a set of 3D-points corresponding to the input object, using the interaction depth camera.
  • the method may comprise recognizing a touchless user input based on a position and/or movement of the input object detected by the interaction depth camera.
  • the method may comprise augmenting, using the at least one projection means, a cursor into the projection scene based on the determined touchless user input, wherein the cursor is visually perceivable by the user.
  • Movement of the input object may include a change of a pointing direction of the input object and/or successive positions of a moving input object.
  • a cursor in particular as a hovering cursor, other hardware, for example a laser pointer in presentations, can be omitted.
  • a hovering cursor even has better visibility than a laser pointer, as it is visible even in difficult lighting conditions.
  • a user can immediately see how the position of his input object, for example his finger, is interpreted by the system. This allows him to interact in a targeted manner and make precise inputs. User-friendliness is thus further enhanced by augmenting a hovering cursor.
  • the method may provide that the projection scene includes a machine-readable calibration pattern, and wherein calibrating at least one interaction depth camera comprises comparing the captured calibration pattern to a predefined reference calibration pattern.
  • a machine-readable calibration pattern is a chessboard, among others.
  • the machine-readable calibration pattern can be hidden or placed in the projection scene in such a way that it does not interfere with the user experience.
  • it can be placed at the edge or in a particularly light color so that it is as unnoticeable as possible.
  • the method may provide that calibrating the at least one interaction depth camera comprises defining a spatial input area for recognizing of touchless user input, preferably wherein recognizing of touchless user input is only performed within the spatial input area.
  • a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to the first aspect of the present invention.
  • All technical implementation details and advantages described with respect to the first and/or second and/or third and/or fourth and/or fifth aspect of the present invention are self- evidently mutatis mutandis applicable for the sixth aspect of the present invention and vice versa.
  • Fig. 5 A schematic depiction illustrating the operation of a method according to embodiments of the present invention, including an illustration of a field of view of a calibration depth camera, a field of view and input area of the interaction depth camera, and an input object performing a first gesture.
  • Fig. 7c A schematic side view of a first exemplary embodiment of a touchless interaction enabled apparatus, configured to perform a method according to embodiments of the present invention.
  • Fig. 8a A schematic perspective view of a second exemplary embodiment of a touchless interaction enabled apparatus, configured to perform a method according to embodiments of the present invention.
  • Fig. 10b A schematic front view of a second exemplary embodiment of a retrofitting assembly according to embodiments of the present invention.
  • Fig. 10c A schematic side view of a second exemplary embodiment of a retrofitting assembly according to embodiments of the present invention.
  • FIG. 11 A schematic illustration of an exemplary input object according to embodiments of the present invention.
  • Fig. 12 A first photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
  • Fig. 13 A second photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
  • Fig. 14 A third photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
  • Fig. 15 A fourth photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
  • Fig. 16 A fifth photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
  • Fig. 17 A sixth photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
  • Fig. 18 A seventh photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
  • Fig. 19 A eighth photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
  • Figure 1 is a schematic illustration of a touchless interaction enabled apparatus, configured to perform a method according to embodiments of the present invention, during operation.
  • the touchless interaction enabled apparatus includes a projection means 100, which may be a standard projector. Further, the apparatus includes a calibration depth camera 210 and an interaction depth camera 220. The calibration depth camera 210 and the interaction depth camera 220 are fastened to the projection means 100 via a mounting means 300. The projection means 100 emits light and thus provides a projection scene 110 onto a projection target 120.
  • the projection target may be of any suitable material and shape, such as textile, among others.
  • Figure 2 is a further schematic illustration of a touchless interaction enabled apparatus, including a field of view 211 of a calibration depth camera 210, configured to perform a method according to embodiments of the present invention, during operation.
  • the calibration depth camera 210 may capture the projection scene 110 which is within its field of view 211 in order to generate calibration data as described above.
  • Figure 3 is a further schematic illustration of a touchless interaction enabled apparatus, including a field of view 211 of a calibration depth camera 210, a field of view 221 and an input area 222 of the interaction depth camera 220, and an input object 400, configured to perform a method according to embodiments of the present invention, during operation.
  • the input object 400 is a finger or a hand of a user providing user input in order to perform touchless interaction.
  • the field of view 221 and the input area 222 are, in this illustration, the same.
  • the input area 222 may of course also be a distinct part of the field of view 221 of the interaction depth camera 220.
  • the input object 400 is positioned within the input area 222 and thus can be detected in order to recognize a desired user input.
  • Such desired user input is illustrated exemplarily in figure 4, where a pointing direction 410 is illustrated corresponding to the position and orientation of the input object 400.
  • the projection means 100 augments a cursor 130 into the projection scene 110.
  • the holding structure 320 is configured such that the interaction depth camera 220 can be attached frontwards, upwards and backwards, depending on the attachment position 330 which is chosen.
  • the interaction depth camera 220 is attached upwards.
  • the possibility to attach the interaction depth camera 220 to the holding structure 320 in various attachment positions 330 allows it to be oriented and oriented in various ways.
  • a user can optimally adjust the orientation of the interaction depth camera 220 and thus the field of view 221 or the input area 222 of the interaction depth camera 220 to his or her needs or to a specific situation of use.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un procédé d'activation d'interaction sans contact mis en œuvre par ordinateur, le procédé comprenant : la projection, à l'aide d'un moyen de projection, d'une scène de projection sur une cible de projection ; le capture, à l'aide d'une caméra de profondeur d'étalonnage, d'au moins une partie de la scène de projection et la génération de données d'étalonnage, les données d'étalonnage comprenant au moins partiellement un contenu de la scène de projection capturée ; l'étalonnage d'au moins une caméra de profondeur d'interaction sur la base, au moins en partie, des données d'étalonnage, ce qui permet à la ou aux caméras de profondeur d'interaction de détecter un objet d'entrée pour reconnaître une entrée d'utilisateur sans contact souhaitée effectuée par un utilisateur à l'aide de l'objet d'entrée.
EP23755433.2A 2022-08-24 2023-08-21 Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage Pending EP4577892A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP22191830.3A EP4328714A1 (fr) 2022-08-24 2022-08-24 Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage
PCT/EP2023/072953 WO2024042041A1 (fr) 2022-08-24 2023-08-21 Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage

Publications (1)

Publication Number Publication Date
EP4577892A1 true EP4577892A1 (fr) 2025-07-02

Family

ID=83059327

Family Applications (2)

Application Number Title Priority Date Filing Date
EP22191830.3A Withdrawn EP4328714A1 (fr) 2022-08-24 2022-08-24 Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage
EP23755433.2A Pending EP4577892A1 (fr) 2022-08-24 2023-08-21 Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP22191830.3A Withdrawn EP4328714A1 (fr) 2022-08-24 2022-08-24 Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage

Country Status (2)

Country Link
EP (2) EP4328714A1 (fr)
WO (1) WO2024042041A1 (fr)

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9874977B1 (en) * 2012-08-07 2018-01-23 Amazon Technologies, Inc. Gesture based virtual devices
LU92408B1 (en) 2014-03-21 2015-09-22 Olivier Raulot User gesture recognition
WO2018148845A1 (fr) * 2017-02-17 2018-08-23 Nz Technologies Inc. Procédés et systèmes de commande sans contact d'un environnement chirurgical

Also Published As

Publication number Publication date
EP4328714A1 (fr) 2024-02-28
WO2024042041A1 (fr) 2024-02-29

Similar Documents

Publication Publication Date Title
JP6372487B2 (ja) 情報処理装置、制御方法、プログラム、および記憶媒体
CN104641316B (zh) 光标移动设备
US8933882B2 (en) User centric interface for interaction with visual display that recognizes user intentions
CA2862446C (fr) Systeme et procede d'entree interactive
US20130044054A1 (en) Method and apparatus for providing bare-hand interaction
US20160196042A1 (en) Gesture enabled simultaneous selection of range and value
US20110242054A1 (en) Projection system with touch-sensitive projection image
US20130135199A1 (en) System and method for user interaction with projected content
KR20030047328A (ko) 원격 포인팅 방법 및 이에 적합한 장치
US10194124B2 (en) Projection type video display device and projection type video display method
US20140333585A1 (en) Electronic apparatus, information processing method, and storage medium
EP3985484A1 (fr) Procédé d'étalonnage, dispositif d'étalonnage et procédé de commande de geste sans contact
US9817572B2 (en) Overlapped transparent display and control method thereof
EP4328714A1 (fr) Procédé d'activation d'interaction sans contact, appareil et ensemble de rattrapage
US9946333B2 (en) Interactive image projection
US12449912B2 (en) Touchless user-interface control method including fading
US9684415B2 (en) Optical touch-control system utilizing retro-reflective touch-control device
US12422956B2 (en) Touchless user-interface control method including time-controlled fading
EP4400953A1 (fr) Candidats de pointeur pour commande d'interface utilisateur sans contact
US20100155604A1 (en) System and method for distinguishing and detecting multiple infrared signal coordinates

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20250317

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)