Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures

Definitions

• GLAMOS https://www.kickstarter.com/projects/300948436/glamos-bring-your-touchless- screens-to-life), sets out to use lidar technology to turn screens into interactive touch screens.
• Touchjet https://www.touchjet.com/wave-lily/ aims at turning flat screen TVs into a tablet for collaboration and interactive presentations using a digital pen.
• a computer-implemented touchless interaction enablement method may comprise projecting, using a projection means, a projection scene onto a projection target.
• the method may comprise capturing, using a calibration depth camera, at least part of the projection scene and generating calibration data.
• the calibration data may at least partially include content of the captured projection scene.
• the method may comprise calibrating at least one interaction depth camera based at least in part on the calibration data, thereby enabling the at least one interaction depth camera to detect an input object for recognizing a desired touchless user input made by a user using the input object.
• projection means may include any type of means capable of providing a projection scene, including but not limited to projectors or projector arrays.
• the projection means may emit light for this purpose.
• the depth cameras according to the present invention may in particular be infrared stereovision depth cameras.
• the calibration depth camera may be placed at or near the projection means such that it can observe the projection scene.
• the interaction depth camera may be placed at or near the projection means such that it can observe an area in which a user may, or is expected to, provide user-input through an input object.
• the term "calibrating” is understood as a calculation and/or configuration process in which, in particular, the interaction depth camera is enabled to correctly recognize user-input.
• calibration is performed in such a way that the calibration depth camera generates calibration data with which the interaction depth camera is calibrated.
• one depth camera calibrates another depth camera.
• the calibration data may be used to generate a spatial input area that is observed by the interaction depth camera.
• the input area may be mapped to the projection scene such that an input object detected by the interaction depth camera in the input area produces an effect at a correct desired portion of the projection scene.
• the input area may be a frustum of a cone or a frustum of a pyramid.
• Calibration may further include calibration of the projection means.
• the projection scene can be at least partially adjusted to the input area to increase the precision of user-input detection. This may be necessary in particular if the input area of the interaction depth camera cannot be fully adapted to the projection scene due to conflicting positions of the projection means and the interaction depth camera. By at least partially adjusting the projection scene, the conflict can be effectively resolved.
• One aspect of the calibration may include the detection of the edges of the projection scene and the definition of an input area of the interaction depth camera corresponding to the detected edges of the projection scene.
• user input is detected inside the input area. Outside the input area it may be provided that no user input is detected, although the input device in within a field of view of the interaction depth camera.
• each depth camera may be modelled to observe a certain area which may have the form of a frustum of a cone or a frustum of a pyramid.
• the projection means may be modelled as an inverse depth camera, so to say, providing content instead of observing, though light emitted by the projection means may be modelled as a frustum of a cone or a frustum of a pyramid.
• a desired touchless user input may for example be a marking of certain positions on the projection scene, for example, in a similar way as a laser pointer does in PowerPoint presentations or other lectures.
• a hovering cursor can be displayed overlaying the projection scene.
• the cursor can have animations that show the recognition of the gesture, depending on the gesture. For example, a click gesture can cause the cursor to change size momentarily, or a circle can be displayed around the cursor.
• desired touchless user input may be any types of gestures, such as swipe gestures, tap gestures, and other more complex gestures. It may further be provided that a user configures own, i.e. user-defined, gestures and assigns a specific intended user input to these gestures, which is recognized accordingly.
• machine-readable calibration pattern may include all types of suitable patterns that can be used as a calculation basis for a calibration.
• a machine-readable calibration pattern can be designed in such a way that it has uniquely distinctive sections that make it possible to derive the respective positions and any distortions of a projection scene.
• a reference calibration pattern may be stored in a memory against which the captured calibration pattern is compared for calibration purposes, in particular to detect distortions of the projection scene and take them into account for calibration.
• the method enables the provision of a very user-friendly and intuitive operation and/or interaction with a user interface. It is particularly advantageous that even a commercially available projector can be equipped to provide touchless user interaction by using depth cameras as proposed in the method of the invention. In addition to the high user-friendliness and the intuitive operation, the hygiene of the user interaction is further increased, since no physical touches are required for a user interaction.
• detecting the input object includes determining a pointing direction and/or a position of the input object, using the interaction depth camera. Determining a pointing direction may be performed based on the shape of the input object. For example, if an elongated input object, such as a pen or a finger, is used, the main axis of the elongated input object may be determined as the pointing direction towards the projection scene.
• detecting the input object may comprise determining a set of 3D-points corresponding to the input object, using the interaction depth camera.
• the method may comprise recognizing a touchless user input based on a position and/or movement of the input object detected by the interaction depth camera.
• the method may comprise augmenting, using the at least one projection means, a cursor into the projection scene based on the determined touchless user input, wherein the cursor is visually perceivable by the user.
• Movement of the input object may include a change of a pointing direction of the input object and/or successive positions of a moving input object.
• a cursor in particular as a hovering cursor, other hardware, for example a laser pointer in presentations, can be omitted.
• a hovering cursor even has better visibility than a laser pointer, as it is visible even in difficult lighting conditions.
• a user can immediately see how the position of his input object, for example his finger, is interpreted by the system. This allows him to interact in a targeted manner and make precise inputs. User-friendliness is thus further enhanced by augmenting a hovering cursor.
• the method may provide that the projection scene includes a machine-readable calibration pattern, and wherein calibrating at least one interaction depth camera comprises comparing the captured calibration pattern to a predefined reference calibration pattern.
• a machine-readable calibration pattern is a chessboard, among others.
• the machine-readable calibration pattern can be hidden or placed in the projection scene in such a way that it does not interfere with the user experience.
• it can be placed at the edge or in a particularly light color so that it is as unnoticeable as possible.
• the method may provide that calibrating the at least one interaction depth camera comprises defining a spatial input area for recognizing of touchless user input, preferably wherein recognizing of touchless user input is only performed within the spatial input area.
• a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to the first aspect of the present invention.
• All technical implementation details and advantages described with respect to the first and/or second and/or third and/or fourth and/or fifth aspect of the present invention are self- evidently mutatis mutandis applicable for the sixth aspect of the present invention and vice versa.
• Fig. 5 A schematic depiction illustrating the operation of a method according to embodiments of the present invention, including an illustration of a field of view of a calibration depth camera, a field of view and input area of the interaction depth camera, and an input object performing a first gesture.
• Fig. 7c A schematic side view of a first exemplary embodiment of a touchless interaction enabled apparatus, configured to perform a method according to embodiments of the present invention.
• Fig. 8a A schematic perspective view of a second exemplary embodiment of a touchless interaction enabled apparatus, configured to perform a method according to embodiments of the present invention.
• Fig. 10b A schematic front view of a second exemplary embodiment of a retrofitting assembly according to embodiments of the present invention.
• Fig. 10c A schematic side view of a second exemplary embodiment of a retrofitting assembly according to embodiments of the present invention.
• FIG. 11 A schematic illustration of an exemplary input object according to embodiments of the present invention.
• Fig. 12 A first photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
• Fig. 13 A second photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
• Fig. 14 A third photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
• Fig. 15 A fourth photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
• Fig. 16 A fifth photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
• Fig. 17 A sixth photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
• Fig. 18 A seventh photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
• Fig. 19 A eighth photography illustration of a usage situation of a touchless interaction enabled apparatus according to embodiments of the present invention.
• Figure 1 is a schematic illustration of a touchless interaction enabled apparatus, configured to perform a method according to embodiments of the present invention, during operation.
• the touchless interaction enabled apparatus includes a projection means 100, which may be a standard projector. Further, the apparatus includes a calibration depth camera 210 and an interaction depth camera 220. The calibration depth camera 210 and the interaction depth camera 220 are fastened to the projection means 100 via a mounting means 300. The projection means 100 emits light and thus provides a projection scene 110 onto a projection target 120.
• the projection target may be of any suitable material and shape, such as textile, among others.
• Figure 2 is a further schematic illustration of a touchless interaction enabled apparatus, including a field of view 211 of a calibration depth camera 210, configured to perform a method according to embodiments of the present invention, during operation.
• the calibration depth camera 210 may capture the projection scene 110 which is within its field of view 211 in order to generate calibration data as described above.
• Figure 3 is a further schematic illustration of a touchless interaction enabled apparatus, including a field of view 211 of a calibration depth camera 210, a field of view 221 and an input area 222 of the interaction depth camera 220, and an input object 400, configured to perform a method according to embodiments of the present invention, during operation.
• the input object 400 is a finger or a hand of a user providing user input in order to perform touchless interaction.
• the field of view 221 and the input area 222 are, in this illustration, the same.
• the input area 222 may of course also be a distinct part of the field of view 221 of the interaction depth camera 220.
• the input object 400 is positioned within the input area 222 and thus can be detected in order to recognize a desired user input.
• Such desired user input is illustrated exemplarily in figure 4, where a pointing direction 410 is illustrated corresponding to the position and orientation of the input object 400.
• the projection means 100 augments a cursor 130 into the projection scene 110.
• the holding structure 320 is configured such that the interaction depth camera 220 can be attached frontwards, upwards and backwards, depending on the attachment position 330 which is chosen.
• the interaction depth camera 220 is attached upwards.
• the possibility to attach the interaction depth camera 220 to the holding structure 320 in various attachment positions 330 allows it to be oriented and oriented in various ways.
• a user can optimally adjust the orientation of the interaction depth camera 220 and thus the field of view 221 or the input area 222 of the interaction depth camera 220 to his or her needs or to a specific situation of use.

Landscapes

• Engineering & Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• User Interface Of Digital Computer (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=83059327

Family Applications (2)

Family Applications Before (1)

Country Status (2)

Family Cites Families (3)

• 2022
  • 2022-08-24 EP EP22191830.3A patent/EP4328714A1/de not_active Withdrawn
• 2023
  • 2023-08-21 EP EP23755433.2A patent/EP4577892A1/de active Pending
  • 2023-08-21 WO PCT/EP2023/072953 patent/WO2024042041A1/en not_active Ceased

Also Published As

Similar Documents

Legal Events