WO2016189372A2 - Procédés et appareil destinés à une architecture "hyper ui pour dispositifs" centrée sur l'humain qui pourrait servir de point d'intégration avec de multiples cibles/points d'extrémité (dispositifs) ainsi que procédés/système associés permettant une entrée de geste sensible au contexte dynamique vers une plate-forme de contrôleur universel "modulaire" et une virtualisation du dispositif d'entrée - Google Patents

Procédés et appareil destinés à une architecture "hyper ui pour dispositifs" centrée sur l'humain qui pourrait servir de point d'intégration avec de multiples cibles/points d'extrémité (dispositifs) ainsi que procédés/système associés permettant une entrée de geste sensible au contexte dynamique vers une plate-forme de contrôleur universel "modulaire" et une virtualisation du dispositif d'entrée Download PDF

Info

Publication number
WO2016189372A2
WO2016189372A2 PCT/IB2016/000530 IB2016000530W WO2016189372A2 WO 2016189372 A2 WO2016189372 A2 WO 2016189372A2 IB 2016000530 W IB2016000530 W IB 2016000530W WO 2016189372 A2 WO2016189372 A2 WO 2016189372A2
Authority
WO
WIPO (PCT)
Prior art keywords
user
gesture
wearable
fingers
hand
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/IB2016/000530
Other languages
English (en)
Other versions
WO2016189372A3 (fr
Inventor
Xiao QUAN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to CN201680037446.9A priority Critical patent/CN107896508A/zh
Publication of WO2016189372A2 publication Critical patent/WO2016189372A2/fr
Publication of WO2016189372A3 publication Critical patent/WO2016189372A3/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20—Input arrangements for video game devices
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/25—Output arrangements for video game devices
    • A63F13/28—Output arrangements for video game devices responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light
    • A63F13/285—Generating tactile feedback signals via the game input device, e.g. force feedback
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1037—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals being specially adapted for converting control signals received from the game device into a haptic signal, e.g. using force feedback
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1043—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals being characterized by constructional details

Definitions

  • This invention purposes a device physical interaction virtualization technology that allow simulating the physical interaction of user with (multiple) input device(s) with a single (universal) wearable platform, which facilitates input/interaction across different devices, allow fast switching of input/interaction (virtual) device/methods (for cases such as VR/AR), allow high speed (nature) parallel input/interaction.
  • This invention also purpose a unique human centric "Hyper UI” that could serve as an integration point with multiple devices intuitive “high speed/parallel” input/interaction capabilities between human and devices.
  • This invention also includes methods and apparatus (which can be used by other wearable for purpose of gesture related input /output) to integrate with other
  • mobile/wearable(s) such as but not limit to smart watch / smart phone for providing natural/intuitive (such as simulation technology related to realistic key pressing, knob turning, "force” like direct remote interaction/manipulation), context aware gesture input, single hand operation & high speed parallel input/interaction (could be b-directional with feedback such as tactile/pressure pattern as well as with visual/audio feedback)
  • the proposed wearable apparatus might also have modular design making the platform (plug-able) and can be easily cleaned/maintained or extended with new hardware.
  • Embodiments of application/software maybe 3 r party app on either the glove platform or the smart wearable platform-such as smart watch— in company) that is enabled with such technology is also included.
  • a (dynamic) pressure pattern consists of a time sequence of frames of static pressure pattern "presents" to user, (for example can be used to indicate the direction or movement of something on the displaying surface)
  • FMU here refer to inertia measurement unit, this is currently usually integrated circuit MEMS device/sensor that can provide multiple degree of freedom (DOF) inertia and navigational signals such as acceleration (projection on x,y,z axis), rotation (angular speed, around x,y,z axis), 3d-magnetic compass (projection on x,y,z axis) and etc.
  • DOF degree of freedom
  • On-demand display/screen This could be the screen/display providing visual feedback when user interact using the wearable device for gesture/motion input which could also be a wearable display such as the display of a smart watch, the display at the back of hand of a "smart glove” as the wearable device (such as 101 in figure 1(a) ) for gesture input , or it could be a VR/AR display/glass user is wearing, or even a "nearby" display screen (such as that of a laptop or tablet, a desktop computer or smart TV) that is visible to user and connected to the wearable device(s) related to gesture control when user performing such interaction.
  • a wearable display such as the display of a smart watch
  • it could be a VR/AR display/glass user is wearing, or even a "nearby" display screen (such as that of a laptop or tablet, a desktop computer
  • Ergonometric simulated pose/gesture or "device simulation signature gesture” This is a pose/gesture (of limbs and possibly with palm) that user can comfortably take when using the purposed input device/input method, in some cases the pose can be kept in an extended period of time (such as more than several minutes) without getting fatigue, and such pose is similar to the pose user would use when manipulating tools/devices that is being simulated by the purposed input device/input method. For example: the
  • Ergonometric simulated pose for simulating using a mouse, is “palm face down” (and maybe together with forearm considered to be “level” within some tolerances, such as plus or minus degrees, and such tolerance range might be adjustable/configurable), just like the way people would normally use a mouse on a flat surface; the "Ergonometric simulated pose” for simulating using a joystick or jet “steering stick” is palm facing side ways, just like the way people would normally use a joystick/steering stick, The
  • "Ergonometric simulated pose" for simulating using a remote control is palm up, just like the way people would normally use a remote control
  • "Ergonometric simulated pose” for simulating swipe/touch of a smartphone/tablet will be like the way people would normally use a smart phone/tablet, such as palm forward or down or with some angle, which might be configurable, (and might with 1 or 2 fingers stick to the surface)
  • the "Ergonometric simulated pose” for aiming an weapon such as a gun
  • the "Ergonometric simulated pose” for aiming an weapon could be either exactly the way user would hold such weapon (for example for rifle is using two hands, with right palm close-sideways and index finger on “trigger” if user is right handed and vice versa for left handed)
  • a "symbolized" pose that using one pose of holding kind of weapon (such as pistol, need to use just one hand) to represent all weapons and also might allow optimize the pose for the purpose of comfort, such as allow elbow being supported (on the
  • the “device simulation signature gesture” can be used as kind of “hint” or “signature” gesture(such as but not limited to those mentioned in the “device simulation signature gesture” discussion earlier) for entering or existing (switching) context/operation mode of input device, and could be apart of the “context aware gesture” way of input and smart switching technology as in the following discussions.
  • a computer implemented method for "Context aware" gesture input/recognition/input generally including:
  • the current "context" or operation/input mode (which can be switched dynamically in some conditions) can be determined/ is based on user's gesture/hand position/movement (for example, a zone can be defined for a control and when (user's hand/finger) entering such zone the context can be switch to the appropriate mode for such control, and when user's hand/finger moves out of zone, the context can be switched to other mode if the "virtual" control is not designed as following the hand movement), and the current (if any) operation mode/context might be based on/determined by the current control(s) or input devices being simulated that user is currently interacting with, and could also based on the "context” determining (context based or specific) "commands" or
  • events/messages corresponding to a gesture such as but not limit to a hand/finger gesture, could be relative/local or "global" that could indicate the begin or end of a gesture
  • context'Voperation mode for example the context switch can be triggered when user's hand/finger entering a zone(3D space) that is "exclusive” for operating a specific control/input device, for example user's finger on (or “touched” virtually) the surface of a (simulated/virtual) button/key/knob, or a specific input/control selected could determine the only (or the best) operation/input mode, or when there's no other available context/operation mode to switch (such as the current application/device is the only device user is interacting with and there's only one possible way of interacting with it) such current context can be existed or change to another context(operation mode) by : leaving the "zone of operation", using a "hmt"/"symbolic"/"signature” gesture (such as but not limited to those mentioned in the "device simulation signature gesture” discussion above) that is not defined in the current context to signal the exit of such context ' unselect of the current control/device, such as the gesture of "
  • gestures/pose/motion input not defined in current “context” might be checked (for example by the gesture recognize/input system) for "context determining/switching" commands/signals or “short-cut” signals for some "context-less” activities (such as copy- paste).
  • context switch might be triggered or shortcut/global activities might be performed.
  • the context switch might be triggered by gestures from "lower order" body parts (those segments closer to body in connection sequence, such as arm) prioritizing "higher order” body parts gestures (such as those of finger)
  • the context switch might be triggered by determining the pose or "facing direction'V'pitch" of the back of hand, for example when the back of hand is vertical (to the ground) the context might be switched to "joystick” or "pointing" mode (might depends on other finger such as index finger gesture), while when the back of hand is facing up the context might be switched to "mouse" or
  • keyboard mode depends on other finger such as index finger gesture, and when the back of hand is facing down (palm facing up) the context might be switched to "TV remote control” mode, etc.
  • context switch might use "hint” or "natural signature gesture”(such as but not limit to those mentioned in the "device simulation signature gesture” section earlier) which comes from typical action being simulated , such as the (typical) gesture of typing, grabbing a mouse, flipping a page on touch screen, using joy stick, grabbing a remote control, and etc.
  • such context determine or context switching activity might include (but not limited to) using the rotation/orientation (or pitch) of the back of hand to determine which mode/context we are , and then determine "commands" based on gesture (such as those of palm, wrist and fingers).
  • gesture such as those of palm, wrist and fingers.
  • action/command might be "click” from pinky which means bring up the "context” menu, similar to right mouse click.
  • feedback (such as but not limited to tactile/force/feelable, visual ,audio and etc.) is given to user after context changed.
  • the context might also be determined by the object
  • symbolic gestures such as the "OK” sign/Gesture can be detected and used for confirmation (such as for the "OK” button on a UI), and "Kill” gesture can be detected and used for trigger a "cancel” -'"discontinue” action(such as a commands or sate change)
  • the "context-aware” gesture can be used in multiple different scenarios and
  • Mode or context might include changing target/endpoint(such as device or app) for the gesture command, or determining the appropriate context to switch to base on "hint” gesture or some typical "signature” gesture related to the context or input device being simulated (such as plam facing down and fingers arched to "hint” using a mouse, or palm facing sideways and 4 fingers curved to hint a "joystick” context/mode of input.
  • the "automatic switch” when using context based switching based on a body part/limb/palm action/change of "pose” or pitch, the “automatic switch” is conditional, for example the context should not be switched in the middle of a "captured/protected” activity or some sort of continues activity (not finished, in the middle of some gesture). Normally it must be by itself start from relative “static” position and stabilized more than a period of time that is configurable (such as 1 second). It must be a inflection point or "turning point” and not within a continues movement or gesture (that is not finished) .
  • Priority or more "weight” can be assigned to the "gesture/movement" of slower moving/turning part or “bigger” part of all the related parts (such as finger, palm, hand, wrist, arm and etc.) that is considered by the gesture recognition/input system, for example, the orientation of the palm is considered highly relevant in context switching, for example palm face down together with fingers naturally curved (not complete straight) may signify using a mouse while palm facing side ways with fingers curved more (like grabbing a handle) may signify using a joystick, however it is desirable the mode switch is not only rely on such gesture but also based on the "motion status" - for example in order for a "signature pose” of hand to be recognized as a "context switch signal", it must be hold relatively still for a (maybe configurable) period of time, say 1 or 2 seconds, to make sure the context change is intentional and prevent mis-operation from accidentally switching contexts.
  • the wearable device can "detect” the "signature” of the target device, such as but not limit to wavelength, modulation signal/code emitted), or using focused (narrow beam/high directional) infra red/RF/ultra sound
  • the index finger located on the index finger, and maybe also with corresponding locating mechanisms deployed on the device (such as maker/pattern/beacon/emitter for cameras or optical/infra red/ultrasound/RF receivers/transceivers on the finger, or receiver/ transceiver for the emitter from the finger)
  • corresponding locating mechanisms deployed on the device such as maker/pattern/beacon/emitter for cameras or optical/infra red/ultrasound/RF receivers/transceivers on the finger, or receiver/ transceiver for the emitter from the finger
  • Directional antenna RF direction "finder” type of detection
  • RF directional antenna such as but not limit to a loop shaped antenna
  • a wearable device such as inside a brace around the wrist of a smart watch (or of a smart glove) , basically a conduit inside the brace formed a looped shape antenna and connects to the wearable (or chip inside the brace), which is directional to RF signal in certain frequency range/wave length.
  • a second antenna having significant different direction characteristic (such as it is an "omni direction” antenna or it is an directional antenna pointing different direction as the 1st "loop” antenna can be used on the wearable device so that when the 2nd antenna having good reception while the signal from the 1 st "loop” antenna (might be on different frequencies or modulation) becoming null/weakest (comparing to levels when facing other directions), the loop antenna's axis (which parallel to the arm's pointing direction) is pointing to the direction of target device, (and such information can be used to determine/identify which device user is pointing at, and make it the selected device for the wearable device to communicate/interact with.).
  • directional antenna can also help finding the direction in a similar way (when pointing to the direction of the RF source the reception signal level minimized or maximized, so basically at a "extreme value” or “inflection point” which can be used to determine the direction of the RF source)
  • a wearable device having at least one camera and IMU unit (which can tell the pitch of the watch and desirably also with the magnetic direction), by using the camera on the wearable (such as a smart watch on wrist) "looking up” and examine the picture of ceilings, by optical pattern recognition or by using infra red marking/marker/ beacons there (such as projected by a infra red laser “scanning” device), the wearable device is able to determine the
  • the selection gesture needs to be relatively stable (from previous movement) for a period of time (might be configurable) such as 1-2 seconds, and (optionally) a special pointing gesture might be required.
  • Selection interaction by "Point to” -system interprets user "point” gesture as selecting a “remote” device with criteria such as: with at lest index finger, straight and parallel or within 30 deg (or configurable value no more than 40 degrees) of palm surface, the direction of the finger within +-20 degree (or configurable value no more than 30 degrees) of the direction from the that finger tip to the device, and when the criteria satisfied that device is considered “selected” for interaction (or engaged with).
  • criteria such as: with at lest index finger, straight and parallel or within 30 deg (or configurable value no more than 40 degrees) of palm surface, the direction of the finger within +-20 degree (or configurable value no more than 30 degrees) of the direction from the that finger tip to the device, and when the criteria satisfied that device is considered “selected” for interaction (or engaged with).
  • the detection method/elimination method can be various and usually for example using beacon/marker/speaker/microphone/high directional sensor individually or combined (for the purpose of identifying any possible devices at the direction user pointing at.
  • the wearable might allow extension by providing port interfaces (such as but not limited to 12C, SPI, USB, Bluetooth and etc. and hardware port/interface like but not limited to those depicted in Fig. lb) and allow components to be attached/detached, such as in Fig lb.
  • a finger part of a glove can be exchanged, other "finger part” with different function using the same port/interface or attached with other components to enhance/extend the functionality of the wearable, such as in Figl.b directed a "narrow bean” detector/ directional sensor 103 component (such as but not limited to a camera, RF antenna or IR beacon) can be attached to the index finger to perform "point and select" of devices, such sensor provide or enhance the reliability of the device selection/de-selection/communication.
  • a "narrow bean" detector/ directional sensor 103 component such as but not limited to a camera, RF antenna or IR beacon
  • a method of selecting an object on a screen of another device including Detecting user index finger movement or pointing direction and use the movement (position or delta) for 1) determine /identify the device user intended to interact with (such as but not limit to using methods described in "A") and then on the screen, selection of objects or providing navigation of some "visual" pointing device such as cursor, pointer or representation of hand for selection of objects, so that when user's index finger is pointing at a virtual object (within error threshold such as +- 10 degrees), on the device (screen), the object can be identified.
  • the wearable together with a Virtual "dashboard" integrated multiple devices (such as via IoT network) running on a computing system, such as onboard the wearable, or a server computer connected to the wearable, that can generate the
  • GUI synthesized/integrated multi-device information for displaying
  • GUI which possibly displayed on a wearable screen or "On-demand display/screen”.
  • a "force like selection” initial gesture by determining if the following condition(s) are met: 1) The palm is facing away from user or the back of the hand is facing towards user, 2) the hand gesture is a relatively open position like depicted in Fig.3 position 1 (solid line) where the fingers and thumb are not significantly flexed/bent or closely adjacent (closely adjacent means side by side with no space between them);
  • determining the general pointing direction of this gesture can be calculated using algorithms such as but not limit to by “averaging” the vector direction of the five fingers and get a general pointing direction (one way is just to use vector add to add all (normalized) directional vector of available fingers with possible different (configurable) "weight” for different fingers, and determine the direction in space), such averaging algorithm can have variations such as using different "weight” for thumb, and all other fingers, might take into account of the palm facing direction, the "degree of curving" of palm -which should be relatively not curved much,
  • control system may further filter out those situations where the general pointing direction of the this gesture is not at the direction user head is looking at (this could be an option and if related situation not filtered out, will allow user to interact with remote objects they are not facing/looking at);
  • the system will update related directional pointing region and determine if an remote object is in range. And might provide feedback such as highlighting (visually) or tactile/pressure feedback(tactile) to indicate an object/target is in range.
  • the "attracting gesture” can be determined/related message/events can be triggered; or,
  • fingers such as but not limit to from the data of the "bent/flex” sensor of the fingers, when the "degree of curving/bent" are increased for all related fingers to an extent not yet at “fully closing” level (such as “halfway” of open pose and fully closed position, or “half grab” position , and limit/range could be configurable, and desirably all finger moved in a similar fashion, for example but not limit to, all the changed ratios of bent/curving are within 25% (or a configurable value) of each other or the average value, or at least not the values are not moving in a "contradicting way”— some of the fingers are “closing” while some other(s) are “opening”-- which signifies a change of gesture or context) , then the "attracting gesture” can be determined/related message/events can be triggered;
  • Some subsequent processing could be for example (when in a virtual/simulated environment such as in a game or in a 3D GUI), the object might be rendered as
  • a (3D) interface or game setting/environment modeling a 3 dimensional "world/virtual workspace” or “sandbox” is provided for user (maybe via an VR/AR system such as a HMD or glass) to interact with, in which some objects appears to be “remote” or having a distance outside of user's haptic range(outside of immediate hand reach) and can be manipulated using the wearable device in a "Force”-like ( as depicted in the movie “star wars” series) direct remote interaction, such as (but not limit to) move/rotate the object using gesture without actually touching the object, grabbing object remotely by directly moving them into user's "hands” (visually) or moved to appear very close by (within easy reach), and etc.
  • VR/AR system such as a HMD or glass
  • An object can be brought within the haptic range for user to work on, for example to manipulating. If object is big, however user can be self motion towards the object, so that it can be within haptic range, or use a manipulator to remote manipulate it (like a joystick or mouse).
  • the wearable control system (and also could be a hyper UI system) can determine if user is trying to "pick up” some object (using this "force-like” manner, by detecting user's special "force like selection” (gesture arxl possibly other manners sack as bin ⁇ limited to muscle force change which can be detected by EMG or MMG), and high light such candidate objects, if object is too far, user might "reach out” to grab some object or aiming, this can make the system to "Zoom in” the area so that user can easy to pick, and when user really want to use "force” to grab it to hand, he/she will close all the fingers a little, just like “half grab” or “force like attracting gesture” mentioned above which will “attract” the object to user's hand or within “haptic range” (within reach of hands), and user might use the other hand or this hand (when change to other gesture) to manipulate it. And optionally when user finished, user can throw it back and it will automatically return.
  • the context can be determined by the wearable and passed as a message, or it can suggest signal on (context change) and be determined by app, also there could be "high level" messages such as (index finger flex 20 deg) passed on sides of low level data.
  • a wearable information system to facilitate remote (such as those not within immediate reach of user's hands) interaction with other devices (such as IOT devices) over network (such as via TCP/IP) comprised of:
  • EVIU/motion sensor and other kind of input method
  • a display screen (might be touch screen);
  • wireless communication units such as Wifi, Bluetooth and NFC (and maybe also with other methods);
  • processor/OS that use the communication units to connect to at least one outside devices such as IOT devices and present a UI to user on the display;
  • the software use the sensor to capture user gesture and for a particular "recognized” gesture such as turning a knob, flip on/off a switch, point to selection or navigation, press a button, the software use the (dynamic) gesture data to determine commands (of UI control) and/or "messages/events" sent to the IOT device(s) so that user can interact/control the IOT device remotely with gesture.
  • a computer-implemented method for Motion/Position based gesture interaction
  • processors onboard a wearable device including:
  • Detecting user gesture from Motion/Position sensor (such as EVIU) data to determine gesture (specific to the application/control), for example a wrist turning can be detected using the Motion/Position sensors by examining gyro scope data for rotation acceleration at the direction of wrist turning or by examining accelerometer changes to determine gravity direction change and other movement of hand, to determine how much angle have the wrist in which the wearable is on had turned (from initial position), such change can be used to drive/turn a "virtual knob” or a visual control on the screen, flip on/off a switch, point to selection or navigation, press a button (on the UI), and etc; So basically using the dynamic gesture data gathered from the wearable Motion/Position to determine commands (of UI control) and/or "messages/events"
  • a related embodiment further includes establishing a connection/communication with one or more "remote" device(s) (such as but not limit to a IOT device) wirelessly using the wearable, and use the Motion/Position based gesture input method mentioned above for adjusting/making inputs to the control of the "remote” device(s) and optionally display remote device's UI interface (if any) and update any changes.
  • “remote” device(s) such as but not limit to a IOT device
  • said establishing a connection/communication with one or more "remote" device(s) may contain a step of using "Point and Selection” (as described in A above) or "Remote direct interaction” (as described in C above ⁇ possibly together with hardware these methods required, to identify the device to interact with, and use the IMU based gesture input method mentioned above for adjusting/making inputs to the control of the "remote” device(s) and optionally display remote device's UI interface (if any) and update any changes.
  • the wearable device for communicating with other devices wirelessly might used one or more IOT protocols such as (but not limited to) the following (when communicating with IOT devices): AllJoyn, Thread(networking protocol), OIC ( Open Interconnect consortium), IIC (Industrial Internet Consortium) and etc.
  • IOT protocols such as (but not limited to) the following (when communicating with IOT devices): AllJoyn, Thread(networking protocol), OIC ( Open Interconnect consortium), IIC (Industrial Internet Consortium) and etc.
  • a mobile/wearable gesture manipulated UI can be considered as follows:
  • a computer-implemented method for gesture based UI (for example but not limit to: implemented by a control/processing software/application running on either onboard the device with position/motion(such as IMU) sensor or another wearable "processing" device (such as but not limit to a smart phone, smart watch, glove-like wearable device) that connected to this device) performed by one or more processors onboard a wearable device including: receiving data or message(s) from motion/position sensor(s) on the wearable
  • frame/device(s) such as a part of a glove, or an independent ring
  • that measure the movement/position of at least one moving parts for providing corresponding motion data such as but not limit to a ring or a part of glove with EVIU sensors
  • the sensor data indicative of a gesture/pose/movement of the part of user being measured maybe relative to a reference coordinate frame
  • module(s) such as but not limit to a recognition or input module, a 3 rd party app or plug-in module
  • related (desirably configurable) module(s) such as but not limit to a recognition or input module, a 3 rd party app or plug-in module
  • Context- Aware style and/or “smart switching" style gesture input methods(as described in earlier text) is(are) used for more intuitive user input.
  • control/processing software/application also drives the (tactile) feedback to user which could be provided by the wearable(s) involved.
  • motion/position data from a "second source” that measure a different (movable) part of body/limb that could have relative motion from the part body/limb being measured in the previous mentioned embodiments is used to (better) determine user's gesture or providing more kind of gesture input (for the system) as more moving part now being measured.
  • a wearable/mobile device on the back of user's hand or wrist with a display and capable of running software/apps could communicating with a wearable device or frame on user's finger or measuring their motion/position (such as but not limit to a part of a glove, or an independent ring) and combine the position/motion data from the "finger" sensors with the data from its own sensors (such as EVIU sensors) (such as accelerator and gyroscope, magnetic measuring wrist/arm movement) to provide more detailed gesture information for purpose such as gesture recognition/input or related manipulation/control.
  • EVIU gesture like "pose'Vpitch and acceleration/rotation as well as magnetic direction info
  • a wearable device on user's arm/wrist such as but not limited to a smart watch
  • recognition/input for driving/generating UI events such as selection (pointing, click, double click, drag, drop) or navigation/zooming(such as swipe, rubber band will need 2 fingers, grabbing need 2 fingers) , so that such operation normally require the other hand touching the surface of the wearable device (such as watch) can now be performed with a single hand.
  • the gesture input (from the motion/position sensors located on the 1 st wearable pari/device that provides a motion data of a finger or fingers) is communicated to the wearable processing device (such as smart watch or a smart glove), and combined (such as but not limit to: using the acceleration/gyroscope "deferential " values from the corresponding axis of the 2 devices to get a relational motion of the finger to the wearable processing device), to provide (gesture interpreted) input commands to the (UI of) wearable processing device , such as a click , a swipe indicator, direction of navigating/moving of the view ports, turning a "knob” or flop a switch, and etc, so that user can use the single hand for operation (of applications that usually require 2 hands to operate such as one hand hold the device and the other hand touch the screen of the device )
  • the wearable processing device such as smart watch or a smart glove
  • combined such as but not limit to: using the acceleration/gyroscope "deferential " values from
  • a "dialing" gesture input can be detected by using a “ring” or finger part of a glove with motion/position sensor on it such as but not limit to EVIU sensors, flex/bent sensors and etc, and with "second source” motion/position data from other part of hand/glove such as from a smart watch, a more precise/distinctive gesture input can be provided for purpose such as driving related (UI) control, generating related (UI)events/messages, and etc.
  • UI driving related
  • UI generating related
  • visual response to the single hand gesture input such as scroll, highlighting, cursor/window navigation/movement, button clicking, knob turning for the corresponding UI control (being selected/interacting with using the wearable devices) are generated by the UI system of the application/software running on the wearable processing device and displayed on the screen of the wearable processing device.
  • the wearable device for communicating with other devices wirelessly might used one or more IOT protocols such as (but not limited to) the following (when communicating with IOT devices)
  • IIC Industrial Internet Consortium
  • such interaction might be performed to a 3 device with client side/remote UI running on the wearable processing device and communicating with the wearable processing device, So that user can manipulate/operate the 3 rd device in a single hand gesture input way.
  • An embodiment is directed to (computer/software implemented) systems and methods for selecting and exchanging information between different devices (such as between a source device and a target/destination device) , including the possible situation of using a device independent of the source and target device, including:
  • signature/symbolic gesture input or UI control input such as confirm a context menu item
  • the process will not proceed and the state/context/operation mode is not changed.
  • source content can be determined and is an valid one (by related validation system)
  • the state of "copy/paste” or “drag and drop” maybe entered (and control from gesture become “captured” means not switching to other operation mode until dropped or other exit condition are met)
  • the source location information might be transferred and depending on implementation, contents or "short cut'Vlink might be put into clip board or buffer for "drag-drop” operation , and desirably sending Feedback in visual (such as change of cursor, highlighting and moving the icon of the file with hand movement and etc.) as well as tactile (such as pressure or vibration indicating "grab” or “stickiness”);
  • Detecting/Determine drop target (if not the same device) as user move hand/arm (while keeping certain gesture of grabbing/dragging in case of drag and drop) , by using "beam” like "pointing selection” manner to detect/identify possible destination target (in the area of "beam” directed by gesture of user's hand such as pointing or other ways of indicating direction) "drop” or “paste", the "beam” like "pointing selection” method might including such as (but not limit to) IR (Infra red), RF(special antenna, ultra sound) or use “NFC” (in which case user are holding or pointing right at the device within 4cm).
  • Such target might also be indicated to user (such as visually like highlighted or change color/border, zoom in, and etc, and maybe also in tactile such as a different kind of vibration/pressure in different part of hand/arm);
  • the state of operation maybe changed (such as from “dragging" to "drop” or normal “not dragging” ), and communicate with destination/target device, so that the content of the source can be transferred to destination (could be in multiple forms such as but limit to, under the control of a file/content exchange system (such as related to clip board, drag and drop controller) and copy the content directly to destination/target (if the file/content exchange has the content). Or tunneling the 2 devices (source and target), either directly or thought the relay of the wearable device which runs the file/content exchange system for exchanging/streaming of the content to the destination/target.
  • a file/content exchange system such as related to clip board, drag and drop controller
  • IrDa/IR remote or NFC/RFK can be used to negotiate the communication channels(such as that of RF especially bluetooth, paring passwords etc.
  • documents/Messages/contents may be exchanged between a computer and a
  • portable/wearable device equipped with memory storage, network interface and appropriate software.
  • a wearable device with a pointing/gesture means is used to identify/or select a document on a computer screen.
  • the document's location information for retrieval (a binary version or copy of the document) is communicated to the wearable device.
  • the binary version may then be "pasted” onto another computer or viewed on the wearable display (with such information).
  • Additional representative embodiments of the present teachings are directed to computer implemented methods/systems that include capturing/detecting a gesture input(such as pointing/clicking/start dragging on) with the wearable device on a first computer display UI, wherein a document is shown on the on-demand UI or on a first computer, identifying the gesture (such as pointed location) try to find out selected file's path/location information, transmitting a request to a first computer coupled to the computer hosting the document/content using the location information, the request requesting a copy of the document from the first computer, receiving the copy at the 2nd device (such as the wearable device) from the first computer, and rendering the copy onto a on-demand UI or screen of the 2nd device.
  • a gesture input such as pointing/clicking/start dragging on
  • the wearable devices further include a transceiver under control of the processor and configured to transmit and receive data and a document/message exchange application/software module stored in the storage memory.
  • the operating gesture recognition/input application/software is configured to capture/detect gesture input data/message/events through(from) the gesture input device.
  • the wearable devices further include a transceiver under control of the processor and configured to transmit and receive data and a document/message exchange application/software module stored in the storage memory.
  • document/message exchange application/ software module includes an identification module configured to examine the gesture input data from the gesture recognition/input application/software (provided by the gesture input device) for the document/content (selected/to be exchanged) on the On demand UI display (or 1st computer screen), wherein the gesture input data is examined to identify the location information associated with the document/content and its computer "container", and a communication module configured to transmit a request to the computer "container" of the document/content through the transceiver and using the location information, wherein the request requests a copy of the document associated with the location information, wherein the
  • communication module is further configured to receive the copy from the first computer.
  • Still further representative embodiments of the present teachings are directed to specially programmed computers that include a processor coupled to storage memory, a network interface under control of the processor and configured to send and receive data from one or more networks, a computer identifier (ID) stored in the storage memory representing an identity and location of the specially programmed computer, and a document/message exchange application stored in the storage memory.
  • the document exchange application When executed by the processor, the document exchange application includes a communication module configured to receive a request from a wearable/mobile device through the network interface and using location information contained within the request for retrieving a copy of a document/or send out related message associated with the location information, wherein the communication module is further configured to send the copy to the wearable/mobile device.
  • Still further representative embodiments of the present teachings are directed to computer program products having computer readable medium with computer program logic recorded thereon.
  • the computer program products include code for capturing/detecting input data/message(s) provided by the gesture input device wherein a document is shown on the first computer or On demand UI, code for examining associated gesture input data/message(s) to identify location information related to the document(selected/to be exchange), code for transmitting a request to the computer containing the
  • the request requesting a copy of the document from the computer containing the (selected) document/content, code for receiving the copy at the portable device from the first computer, and code for transforming a state of ones of a plurality of pixels of a portable display of the wearable/mobile device to render the copy onto the device display (or On demand UI).
  • Additional representative embodiments of the present teachings are directed to computer implemented methods that include executing instructions on a computing platform so that binary digital electronic signals detect (with sensors on a wearable device) a user gesture pointing/interacting a document/content on a first computer display, wherein a document is shown on the first computer display, executing instructions on the computing platform so that binary digital electronic signals identify from the gesture of user, the location information related to the document, executing instructions on the computing platform so that binary digital electronic signals transmit a request to a first computer coupled to the first computer display using the location information, the request requesting a copy of the document from the first computer, executing instructions on the computing platform so that binary digital electronic signals receive the copy at the portable/wearable device from the first computer and executing instructions on the computing platform so that binary digital electronic signals render the copy onto a portable display of the
  • a human centric "Hyper UI” that could serve as an integration point with multiple devices comprised of:
  • Visual display (system) that can [use said mobile/wearable means as input devices (and might also as output device)] to provide user with a visual/graphic UI for the device(s) he/she is interacting with(could be such as but not limited to: some other devices via the said mobile/wearable means)
  • visual display (system) may include mobile, wearable (such as VR/AR helmets. Glasses, visors etc) or an "on demand" display on external display system (which not mobile with user or worn by user) coordinating with a system that is mobile with/worn by user that is visible to user (with the mobile/wearable system);
  • At least one (3D) gesture/hand pose/motion sensor system/means as (primary) input this could be for example a series of motion/position and related sensors (such as but not limit to EVIU sensors and flex-bent sensors and etc) located on a frame that moves together with user's finger and possibly also hand(palm), wrist, arm, such sensor system(s) might be part of the wearable device in 1) or could be an independent wearable.
  • 3D gesture/hand pose/motion sensor system/means as (primary) input, this could be for example a series of motion/position and related sensors (such as but not limit to EVIU sensors and flex-bent sensors and etc) located on a frame that moves together with user's finger and possibly also hand(palm), wrist, arm, such sensor system(s) might be part of the wearable device in 1) or could be an independent wearable.
  • a "tactile/pressure"(or other "feelable” feedback type) output system is also provided in the UI system (for the purpose such as but not limit to direct feedback, control-location indicating, control surface simulation and etc.) ; so that when a user interacting with multiple "remote" devices (could be at the same time) which can not be very easily reached by user directly (for example out of haptic range of user or when user's hand is occupied for other tasks) and/or with objects displayed on the #2 on demand screen, the gesture/(hand) pose/motion of user/human to machine direction of interaction can be pick up by #3, and interpreted as input to UI system displayed on #2 as well as related modules/devices that expecting user input, and possibly communicating with other device(s) via #1 when necessary (such as when user interacting with other "remote” devices, and desirably also drive the tactile feedback (of device to human direction) can be provided by tactile/pressure "output" means on the wearable (if available)
  • a controller/processing unit(s) of the mobile/wearable device may further coordinating “Task management/scheduling” and “communication” for devices as well as “copy/paste” with multiple endpoints and the related software running on it, which could include 3 rd party app could provide further integration of the devices such as using a "unified GUI" instead of individual UIs.
  • the wearable device for communicating with other devices wirelessly might used one or more IOT protocols such as (but not limited to) the following (when communicating with IOT devices)
  • IIC Industrial Internet Consortium
  • a computer-implemented method for human centric "Hyper UI" performed by one or more processors onboard a wearable device including:
  • gesture/pose/motion input device that can accommodate and move together with at least one (section of) moving part of user's body or limbs such as (but not limit to): finger(s), palm(s), hand(s), wrist(s),arm(s) and etc , the sensor data indicative of a gesture/pose/moverneni of the pari of user being measured (maybe relative to a reference coordinate frame);
  • On-demand visual display of a graphic user interface such as on a display screen on the wearable device(such as a smart glove), on a wearable processing device (such as a smart watch/smart phone) companion with the wearable device for gesture input (such as a gesture input glove) or on a VR/AR display that user is wearing;
  • determining a visual feedback on the On-demand display environment based on the sensor/gesiure data; and desirably also drive the tactile feedback (of device to human direction) can be provided by tactile/pressure "output" means on the wearable (if available) communicate/connecting with other (necessary) devices/systems (user
  • the computer-implemented method/system includes using "context aware and smart switch" method (as mentioned in earlier text) in gesture recognition/input process (or in processing of sensor data for gesture/motion recognition/input) to allow intuitive control for user and fast switching of modes.
  • the computer-implemented method/system includes using "force like” remote direct interaction method enable user to select and manipulate an "remote” device or an virtual object displayed on the "on-demand” display.
  • the computer-implemented method/system further includes using "point and select” like remote interaction method enable user to select and manipulate an "remote” device or an virtual object displayed on the "on-demand” display.
  • the computer-implemented method/system includes using "multi-device copy-paste" way of interaction/inegration method (as described in text earlier) to enable content/documents to be transferred between 2 (or more) devices conveniently (in a way similar to the actions required for user to do this within one device).
  • a controller/processing unit(s) of the mobile/wearable device ⁇ may include (one or more of) further tasks such as (but not limit to): coordinating "Task management/scheduling” ; managing 3 rd party apps installed on the platform; coordinating "copy/paste” with multiple endpoints/devices (as mentioned in the Multiple Device copy paste section above) and the related software running on it
  • the computer-implemented method could further integrate with multiple devices, for example by providing additional services such as (but not limit to) one or more of the following:
  • Hardware extension management (such as plugin, drivers)
  • a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, causes the one or more processors to perform operations, comprising:
  • a wearable device such as a VR, AR system or smart watch, smart glove
  • wearable gesture input device such as glove like frame with sensors on the part covering/move with finger(s) and/or palm;
  • said display environment is three-dimensional (3D) ;
  • remote device further including connecting to another device (“remote device”) wirelessly and allow control/interaction with the "remote device” with the instructions stored or installed/managed (such as a 3 rd party app), for example including using the data from sensor(s) -such as interpreting them as
  • commands/messages maybe based on gesture or context aware gesture- to interact with /control the remote device.
  • a human centric "Hyper UI” that could serve as an integration point with multiple devices having features such as:
  • An architecture includes means for storage, manage and run for apps
  • Integrating the accepted gesture data from sensor with control system, remote device and feedback system for example interpreting gesture data (for example using context aware input method) with OS/Software or application stored or and display the Ul(feedback) and optionally drive "tactile" feedback of the output device;
  • the icon/hand skeleton/cursor can be changed according to the current mode, and the feedback management system provide feedback, the
  • the wearable device for communicating with other devices wirelessly might used one or more IOT protocols such as (but not limited to) the following (when communicating with IOT devices)
  • IIC Industrial Internet Consortium
  • a "direct manipulating" word processor can be created by allowing user to use multiple finger as different tools, and context-aware smart switch of different modes so that when user palm is facing down, the input mode is (automatically, according to context) changed to "keyboard" simulation, so that user can directly type into the word processor editor without the need to look or select from a "visual only virtual keyboard", when user's palm is facing front and one of the fingers pointing at certain text, depends on which finger (which can be assigned to different tools, such as selection/highlight, underline or bold) user is using to interact with the text, and maybe further depends on the "depth” of user's finger interaction with the text (for example a "deep press” by index finger that appears to "penetrate” the depth of the displayed text might indicate "cross out” of the text interact with), there are many other interactions using multiple finger separated or combination, symbolic gestures (such as "Ok” gesture for a confirm button or "kill” gesture for a cancel button on a dialog box).
  • symbolic gestures such as "Ok” gesture
  • VR/AR UI system including combining the IMU data from a
  • IMU data from a wearable on user's arm/wriest such as watch, to form a gesture- manipulated system for the "wearable UI" displayed on Glass/VR helmet of the user (simultaneously), such UI provides feedback to user's gesture input (that is interpreted from the combination of the IMU data)
  • the software/app might run on a wearable device with a display (such as smart watch) that communicate with this glove/helmet
  • An input/output device virtualization system simulating a hardware input device that allow user to operate it with similar gesture/hand movement/pose to comprised of:
  • RF radio frequency
  • At least one (3D) gesture/hand pose/motion sensor system/means as (primary) input could be for example a series of motion/position and related sensors (such as but not limit to IMU sensors and flex-bent sensors and etc) located on a frame that moves together with user's finger and possibly also hand(palm), wrist, arm, such sensor system(s) might be part of the wearable device in 1) or could be an independent wearable.
  • 3D gesture/hand pose/motion sensor system/means as (primary) input, this could be for example a series of motion/position and related sensors (such as but not limit to IMU sensors and flex-bent sensors and etc) located on a frame that moves together with user's finger and possibly also hand(palm), wrist, arm, such sensor system(s) might be part of the wearable device in 1) or could be an independent wearable.
  • a "tactile/pressure"(or other "feelable” feedback type) output system is also provided in the UI system (for the purpose such as but not limit to direct feedback, control-location indicating, control surface simulation and etc.) ; optionally a visual representation of the "physical device/instrument” being simulated (or a part of it such as (vicinity) area that user's hands/fingers and etc. are interacting with) is presented to user (for example for calibration or correction of key stroke);
  • the (spatial) relationship between fingers (especially areas with feelable feedback actuators such as finger tip) to the key(s)/instrument operating area(s) can be determined from the sensor system data (for example those from IMU's acceleration and pose data, magnetic "direction” data (or possibly from the "hand skeleton model” which capture and integrate these data), and maybe with other data from additional bent/flex sensors, for (optionally) optical sensors, (or the combine of such data to get a more reliable location); such data might determine if a control is touched, pressed or other situations for example half pressed, pressed multiple controls/keys at the same time, and etc., and certain events/messages can be generated or corresponding procedures/process is executed; so that to the message/event recipient software/User Interface system or application(s) the output of such interaction of user with virtual device is the same as with a physical (real) device, and from user's perspective the interaction with the simulated input device is similar/close to that of the real device in aspect of functionality, the skills/ges
  • said spatial relationship data could be from relative positional or movement data, or it could be calculated from the location of each moving section or from the position object interacting with.
  • providing "direct'Vmulti-modal feedback including providing ("specific” or “corresponding”)”response” or “feedback” (such as but not limited to: pressure/tactile patterns, vibration patterns/strength/frequency changes etc.) to user's finger tip with tactile means that located on the part of the wearable accommodating finger tip (and other parts) based on the ("specific") spatial relationship between finger and the virtual object(keys), for example , one activated "spot'V'fixel” in the pattern could means (barely) contacting the virtual object (at such location, might be off centered) but not pressed;
  • optical system can be used in addition to EVIU for tracking the hand-based wearable (so that the accuracy can be improved and errors specific to each individual methods can be reduced by referring to the other methods, using ways such as but not limited to Kalman filtering ) such as but not limit to glove-like wearable, comprised of:
  • a least one camera that can be placed on users HMD/Goggle/glasses/AR glasses, or utilizing the camera already on those devices weared by user, or at least one camera weared by user such as around user's neck or in front of user's body (like a body cam) that is able to "see” user's hand in most cases;
  • optical motion/position tracking algorithms on the signals acquired by the camera to determine the direction and distance of the wearable with the beacon to the camera, thus determine the relative position to user's body/head ⁇ this can use for example but not limited to using H.264 motion vector detection (could e native by the camera) for fast detemination of movement of the beacon or other image processing/motion detection and capturing/tracking algorithm for the images;
  • pairs of image can be captured to determine the relative position of the wearable (or the beacons on it);
  • beacons to camera can be established, which can be used for capturing motion or position of the wearable (such as but not limited to accommodating user's hand fingers).
  • a computer-implemented method for human centric "Hyper UI" performed by one or more processors onboard a wearable device including:
  • gesture/pose/motion input device that can accommodate and move together with at least one (section of) moving part of user's body or limbs such as (but not limit to): finger(s), palm(s), hand(s), wrist(s),arm(s) and etc , the sensor data indicative of a gesture/pose/movement of the part of user being measured (maybe relative to a reference coordinate frame);
  • the sensor system data for example those from IMU's acceleration and pose data, magnetic "direction” data (or possibly from the "hand skeleton model” which capture and integrate these data), and maybe with other data from additional bent/flex sensors, for (optionally) optical sensors, (or the combine of such data to get a more reliable location); such data might determine if a control is touched, pressed or other situations for example half pressed, pressed multiple controls/keys at the same time, and etc., and certain events/messages can be generated or corresponding
  • procedures/process is executed; desirably also (optionally) presenting a visual representation of the "physical device/instrument” being simulated (or a part of it such as (vicinity) area that user's hands/fingers and etc. are interacting with) is presented to user (for example for calibration or correction of key stroke); communicate/connecting with other (necessary) devices/systems (user
  • said spatial relationship data could be from relative positional or movement data, or it could be calculated from the location of each moving section or from the position object interacting with.
  • providing "direct'Vmulti-modal feedback including providing ("specific” or “corresponding”)”response” or “feedback” (such as but not limited to: pressure/tactile patterns, vibration patterns/strength/frequency changes etc.) to user's finger tip with tactile means that located on the part of the wearable accommodating finger tip (and other parts) based on the ("specific") spatial relationship between finger and the virual object(keys), for example , one activated "spot'V'fixel” in the pattern could means (barely) contacting the virtual object (at such location, might be off centered) but not pressed,
  • a wearable device such as a smart watch employ such virtulization technology to simulate physical devices that interact with hands such as but not limit to music instruments, keyboard/key pad, remote control, joystick, mouse and etc.
  • a software/application stored and executed by a wearable device use/integrate with virtulization technology to simulate physical devices that interact with hands such as but not limit to music instruments, keyboard/key pad, remote control, joystick, mouse and etc.
  • Identify a key-reaching(-over) gesture of a finger by comparing the gesture(static) or movement of the 10 fingers (find out which one is moving towards which direction), and when detecting a "press” gesture, first give out a "touched” feedback(tactile) and display on the screen the key user is touched but not pressed, then when user press down that key then give more feedback on tactile and the visual.
  • a input/output device virtualization technology that allow multiple different input devices being simulated and capable of instant switching between these simulated devices/methods, and high speed parallel input (that could also serve as an integration point with multiple devices) comprised of:
  • a controller/processing unit(s) on board of the mobile/wearable ⁇ or in smart watch that collaborating such wearable— may further coordinating "Task management/scheduling” and “communication” for devices as well as “copy/paste” with multiple endpoints and the related software running on it, which could include 3 rd party app (running on each) could provide further integration of the devices such as using a "unified GUI" instead of individual UIs.
  • a computer system such as but not limited to wearable device such as a smart watch, smart glove or mobile device and etc.
  • a computer system employing such virtualization technology to communicate with multiple remote devices that and provide interaction for user with hands with corresponding (could be multiple different one) input methods with the devices user is interacting with.
  • mouse simulation for a desktop touch screen interaction with a tablet
  • joystick simulation for a game console joystick simulation for a game console
  • gesture input such as simulating turning a knob
  • IOT device such as a smart thermostat.
  • a wearable device such as a glove or watch have a brace around the wrist (or part of hand if is a small glove) and a conduit inside the brace formed a looped shape antenna and connects to the wearable (or chip inside the brace), which is directional. Inside the wearable device there's another antenna with its plan
  • such wearable could be modular design and could be dissembled, and certain parts (such as index finger part) can be switched.
  • the "finger tip” or “finger” portion of the glove can be removed or replaced.
  • modularity such as but not limit to: 1) A 3 part design, with removable finger (tip) portion (with sensors and possibly actuators) with connectors connecting to the "base'Vpalm part of the glove(with sensors, controllers and power source), a tactile "plate” that is basically the “palm” side of the glove with actuators on it (and could be optional), and a base portion accommodating the palm with controllers, communication, power on the back of the hand, and capable to connect to the other 2 part.
  • an optional design could be 2 part design like depicted in Fig 1 (a) and (b) : the "finger" portion could be removed have sensors (such as motion/position sensors such as but not limit to EVIU sensors, flex-bent sensors) and (optionally) also with actuators (such as tactile actuator array) in them and could have long connectors
  • sensors such as motion/position sensors such as but not limit to EVIU sensors, flex-bent sensors
  • actuators such as tactile actuator array
  • the "interactive" type of the glove can be changed from non-tactile to tactile/pressure type and etc.
  • FIG.1 (c) Yet another 2 part design like depicted in Fig.1 (c) is also possible where the glove can be "split" into an upper and a lower portion, connect with connectors where the upper portion could house most sensors (and the controller on the back of the hand), the lower portion are more devoted to "feedback” and could house mst of the actuators.
  • the "skin" or lining of the glove can be changed.
  • the wearable could adopt a "partial glove” design where motion/position data are only gathered for a few fingers (such as 3 including the index finger) with "ring” like (could be flexible/elastic) "brace” accommodated the finger and connects to a controller on the central "patch" located on the back of the hand (which contain motion/sensors such as IMU sensors for hand position/motion information); and desirably the rings/brace or patch could have button/touchable areas on it to allow additional UI function in interaction.
  • a controller on the central "patch” located on the back of the hand which contain motion/sensors such as IMU sensors for hand position/motion information
  • the rings/brace or patch could have button/touchable areas on it to allow additional UI function in interaction.
  • the connector between the central "patch” and the "brace'Vring on finger could contain additional flex/bent sensors (such as resistors) to provide additional gesture information.
  • a wearable for gesture input further including an earphone/speaker located on the part covering thumb, and a microphone located on the part accommodating either picky or the finger next to it (the "4th" finger from thumb), so that when user make a "telephone gesture” (curved fingers with thumb sticking up and possible picky finger also out. and pointing downwards) close to user's face, the earphone/speaker and the microphone are close enough (to be effective for phone conversation) to user's ear and mouse respectively; the earphone/speaker and microphone is connected to the control system (that may further connect to a
  • such wearable connects to a mobile communication system (such as on a smart watch or a smart phone) desirably wirelessly to provide voice services (pickup and output of audio signal) for voice based communication (such as for placing a phone call, an internet based voice chat)
  • a mobile communication system such as on a smart watch or a smart phone
  • voice services pickup and output of audio signal
  • voice based communication such as for placing a phone call, an internet based voice chat
  • wearable for gesture input such as a smart glove housing abovementioned earphone/speaker and microphone, also equipped with wireless communication system such as but not limit to base band cellular communication system, 3G, 4G mobile communication system (maybe including voice and maybe also together with data services), wifi, and etc.
  • such wearable with micro phone -speaker combination may connect to other devices for voice related services or control.
  • the wearable might also have means for other communication such as but not limit to NFC, Infra-Red, ,Bluetooth, means for supporting IOT protocols .
  • the wearable device could include means/modules wireless communicating with other devices with one or more IOT protocols such as (but not limited to) the following (when communicating with IOT devices) AllJoyn, Thread(networking protocol), OIC ( Open Interconnect consortium) , IIC (Industrial Internet Consortium) and etc.
  • IOT protocols such as (but not limited to) the following (when communicating with IOT devices) AllJoyn, Thread(networking protocol), OIC ( Open Interconnect consortium) , IIC (Industrial Internet Consortium) and etc.
  • Fig 2 depicted a kind of "minimalist” style of design where only motion/position of 3 fingers are gathered by ring like (could be flexible/elastic) "brace” that connects to a controller on the central "patch” located on the back of the hand (which contain montion/sensors such as IMU sensors for hand position/motion information).
  • the connector between the central "patch” and the "brace'Vring on finger could contain additional flex/bent sensors (such as resistors) to provide additional gesture information, and the rings/brace or patch could have button/touchable areas on it to allow additional UI function.
  • Fig 4 depicts several scenarios of the hyper UI with on-demand (wearable) displays:
  • a user's hand interacting with a simulated "knob", which could represent a control of a "remote” device such as a smart (IOT) thermostat 5 feet away, and with it (alternative/wearable) interface displayed on the smart watch user is wearing:
  • User's motion/gesture of turning the "virtual knob” can be captured by the wearable device (the glove) user wearing at the same hand, and it is connected to the smart watch, maybe wirelessly such as via Bluetooth, user's interaction of the control such as turning of the "virtual knob” can change the setting of the remote device and reflected on the smart watch UI; alternatively user could just interact with an app running on this watch and use similar interaction to interact with its "tuning knob” for example, this allows convenient single hand operation and do not need the help from the other hand for related input,
  • (b) shows an alternative situation where the UI is displayed on a AR glasses that serves as

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Position Input By Displaying (AREA)

Abstract

L'invention concerne une technologie de virtualisation d'interaction physique avec un dispositif, qui permet de simuler l'interaction physique d'un utilisateur avec un ou plusieurs dispositifs d'entrée à l'aide d'une seule plate-forme portable (universelle). La présente invention concerne également une "hyper UI" unique centrée sur l'humain qui pourrait servir de point d'intégration avec de multiples dispositifs à capacités d'entrée/interaction intuitives "à grande vitesse/parallèles" entre des humains et des dispositifs. L'invention concerne également des procédés et un appareil (qui peuvent être utilisés par d'autres systèmes portables pour une entrée/sortie associée à des gestes) à intégrer avec d'autres systèmes mobiles/portables, comprenant entre autres une montre intelligente/un téléphone intelligent, afin de fournir une entrée gestuelle sensible au contexte naturelle/intuitive, une entrée/interaction parallèle à grande vitesse et à une seule main (qui peut être bidirectionnelle avec une rétroaction telle qu'un motif de pression/tactile ainsi qu'avec une rétroaction visuelle/audio). Pour faciliter l'utilisation plus polyvalente et une extension de fonction, l'appareil portable proposé (portable en forme de gant) peut également avoir une conception modulaire de la plate-forme (connectable) et peut être facilement nettoyé/conservé ou amélioré avec l'ajout d'un nouveau matériel.
PCT/IB2016/000530 2015-04-25 2016-04-26 Procédés et appareil destinés à une architecture "hyper ui pour dispositifs" centrée sur l'humain qui pourrait servir de point d'intégration avec de multiples cibles/points d'extrémité (dispositifs) ainsi que procédés/système associés permettant une entrée de geste sensible au contexte dynamique vers une plate-forme de contrôleur universel "modulaire" et une virtualisation du dispositif d'entrée Ceased WO2016189372A2 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201680037446.9A CN107896508A (zh) 2015-04-25 2016-04-26 可以作为多个目标/端点（设备）和的整合点的以人为中心的“设备的超ui”体系结构的方法和设备，以及面向“模块化”通用控制器平台以及输入设备虚拟化的具有动态上下文意识的手势输入的相关方法/系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201562152883P 2015-04-25 2015-04-25
US62/152,883 2015-04-25

Publications (2)

Publication Number Publication Date
WO2016189372A2 true WO2016189372A2 (fr) 2016-12-01
WO2016189372A3 WO2016189372A3 (fr) 2017-02-23

Family

ID=57392953

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2016/000530 Ceased WO2016189372A2 (fr) 2015-04-25 2016-04-26 Procédés et appareil destinés à une architecture "hyper ui pour dispositifs" centrée sur l'humain qui pourrait servir de point d'intégration avec de multiples cibles/points d'extrémité (dispositifs) ainsi que procédés/système associés permettant une entrée de geste sensible au contexte dynamique vers une plate-forme de contrôleur universel "modulaire" et une virtualisation du dispositif d'entrée

Country Status (2)

Country Link
CN (1) CN107896508A (fr)
WO (1) WO2016189372A2 (fr)

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108151726A (zh) * 2017-11-24 2018-06-12 北京奇艺世纪科技有限公司 一种运动信息检测系统及方法
CN108687743A (zh) * 2017-04-08 2018-10-23 金子楗 基于振动和气动结合的触觉反馈外骨骼装置
EP3441849A1 (fr) * 2017-08-09 2019-02-13 Nagravision S.A. Dispositif d'entrée de pointage pour ordinateur personnel basé sur les mouvements des doigts
US10265627B2 (en) 2017-06-22 2019-04-23 Centurion VR, LLC Virtual reality simulation of a live-action sequence
CN109782918A (zh) * 2019-01-25 2019-05-21 南阳理工学院 一种计算机辅助教学装置
WO2019156518A1 (fr) 2018-02-09 2019-08-15 Samsung Electronics Co., Ltd. Procédé de suivi d'une posture de main et dispositif électronique associé
CN111158476A (zh) * 2019-12-25 2020-05-15 中国人民解放军军事科学院国防科技创新研究院 一种虚拟键盘的按键识别方法、系统、设备及存储介质
CN111236591A (zh) * 2019-12-26 2020-06-05 陶民 一种用于墙面喷涂、清洗的机器人
US10867448B2 (en) 2019-02-12 2020-12-15 Fuji Xerox Co., Ltd. Low-power, personalized smart grips for VR/AR interaction
CN112351325A (zh) * 2020-11-06 2021-02-09 惠州视维新技术有限公司 基于手势的显示终端控制方法、终端和可读存储介质
CN113223344A (zh) * 2021-05-25 2021-08-06 湖南汽车工程职业学院 一种基于大数据的艺术设计用专业教学展示系统
EP3732672A4 (fr) * 2017-12-27 2021-09-15 Adesanya, Olaoluwa O. Appareil informatique portable pour interactions de réalité augmentée, de réalité virtuelle et d'intelligence artificielle, et procédés associés
CN114281907A (zh) * 2021-11-01 2022-04-05 北京百度网讯科技有限公司 室内环境的行进处理方法、装置、电子设备及可读存储介质
CN115240224A (zh) * 2022-07-22 2022-10-25 福州大学 基于三维手部关键点和图像特征融合的手势特征提取方法
CN115562499A (zh) * 2022-11-16 2023-01-03 深圳市未来感知科技有限公司 基于智能指环的精准交互控制方法、系统及存储介质
CN115645906A (zh) * 2022-10-08 2023-01-31 网易（杭州）网络有限公司 一种虚拟单位的选择方法、装置、设备及存储介质
WO2023061552A1 (fr) * 2021-10-11 2023-04-20 Abb Schweiz Ag Interface d'apprentissage sensible pour la programmation d'un robot industriel
CN116236328A (zh) * 2023-02-22 2023-06-09 中国科学院沈阳自动化研究所 基于视觉的可实现自然抓取的智能假肢系统
CN117697749A (zh) * 2023-12-25 2024-03-15 墨现科技(东莞)有限公司 设备控制方法、装置、抓握设备及存储介质
US20250022353A1 (en) * 2021-11-08 2025-01-16 Bhaptics Inc. Device for providing tactile stimulation
WO2025088497A1 (fr) * 2023-10-24 2025-05-01 Outplay Spolka Z Ograniczona Odpowiedzialnoscia Dispositif, en particulier pour commander un jeu vidéo
US12541256B2 (en) 2018-05-03 2026-02-03 Interdigital Vc Holdings, Inc. Systems and methods for physical proximity and/or gesture-based chaining of VR experiences

Families Citing this family (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110058413A (zh) * 2018-05-23 2019-07-26 王小峰 一种智能穿戴系统
CN108833937B (zh) 2018-05-30 2021-03-23 华为技术有限公司 视频处理方法和装置
CN109032343B (zh) * 2018-07-04 2022-02-11 青岛理工大学 基于视觉和力觉触觉增强现实的工业人机交互系统及方法
CN110728158B (zh) * 2018-07-17 2024-05-03 手持产品公司 用于触发移动扫描设备的装置和系统
CN109144258A (zh) * 2018-08-23 2019-01-04 哈尔滨工业大学 一种位姿测量手套
CN109276881A (zh) * 2018-08-31 2019-01-29 努比亚技术有限公司 一种游戏控制方法、设备
CN109452943B (zh) * 2018-09-17 2021-03-23 周建菊 一种用于重症偏瘫患者的多功能康复衣
WO2021021224A1 (fr) 2019-07-26 2021-02-04 Google Llc Commande sensible au contexte de reconnaissance de geste basée sur un radar
EP4004686B1 (fr) 2019-07-26 2024-11-20 Google LLC Gestion d'authentification par umi et radar
CN118192796A (zh) 2019-08-30 2024-06-14 谷歌有限责任公司 移动设备的输入方法
CN113874812B (zh) * 2019-08-30 2024-09-03 谷歌有限责任公司 用于多输入模式的输入模式通知
US12229341B2 (en) * 2019-09-23 2025-02-18 Apple Inc. Finger-mounted input devices
CN110989467B (zh) * 2019-12-25 2021-05-11 杭州市中医院 一种智能红外手套控制器及其方法
US12124627B2 (en) 2020-01-06 2024-10-22 National University Of Singapore Glove-based human machine interface
US11157086B2 (en) * 2020-01-28 2021-10-26 Pison Technology, Inc. Determining a geographical location based on human gestures
US20230054973A1 (en) * 2020-02-06 2023-02-23 Sony Group Corporation Information processing apparatus, information processing method, and information processing program
CN111708440B (zh) * 2020-07-07 2025-05-30 北京三角座科技有限公司 一种vr全方位运动装置
US11393171B2 (en) 2020-07-21 2022-07-19 International Business Machines Corporation Mobile device based VR content control
CN114764293A (zh) * 2021-01-04 2022-07-19 北京小米移动软件有限公司 一种可穿戴设备的控制方法、装置、可穿戴设备和存储介质
CN117043709A (zh) * 2021-02-08 2023-11-10 视域电脑有限公司 用于生产力的扩展现实
CN115047966B (zh) * 2021-02-26 2025-08-01 华为技术有限公司 交互方法、电子设备与交互系统
CN113138665A (zh) * 2021-04-02 2021-07-20 深圳大学 一种基于智能手表的智能设备指向操控方法及系统
CN114296543B (zh) * 2021-11-15 2024-10-29 北京理工大学 一种指尖力检测及手势识别的智能交互系统及智能指环
CN114489347A (zh) * 2022-04-01 2022-05-13 深圳市赢向量科技有限公司 动态沙盘及其演示方法和系统
CN115253272B (zh) * 2022-07-27 2025-02-14 网易（杭州）网络有限公司 游戏交互方法及装置、存储介质、电子设备
CN117786643B (zh) * 2024-02-22 2024-05-07 清华大学 一种用户身份认证方法及相关设备
CN119024961A (zh) * 2024-07-31 2024-11-26 珠海莫界科技有限公司 智能穿戴设备的交互控制方法、装置、设备及存储介质

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5832296A (en) * 1995-04-26 1998-11-03 Interval Research Corp. Wearable context sensitive user interface for interacting with plurality of electronic devices of interest to the user
US20110210931A1 (en) * 2007-08-19 2011-09-01 Ringbow Ltd. Finger-worn device and interaction methods and communication methods
AU2008290211A1 (en) * 2007-08-19 2009-02-26 Ringbow Ltd. Finger-worn devices and related methods of use
US9377867B2 (en) * 2011-08-11 2016-06-28 Eyesight Mobile Technologies Ltd. Gesture based interface system and method
CN202584010U (zh) * 2012-04-06 2012-12-05 寇传阳 一种腕上手势操控系统
CN103823551B (zh) * 2013-03-17 2017-07-21 浙江大学 一种实现多维度感知虚拟交互的系统和方法
CN103955295A (zh) * 2014-04-17 2014-07-30 北京航空航天大学 一种基于数据手套和物理引擎的虚拟手的实时抓取方法

Cited By (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108687743A (zh) * 2017-04-08 2018-10-23 金子楗 基于振动和气动结合的触觉反馈外骨骼装置
US10792573B2 (en) 2017-06-22 2020-10-06 Centurion Vr, Inc. Accessory for virtual reality simulation
US10265627B2 (en) 2017-06-22 2019-04-23 Centurion VR, LLC Virtual reality simulation of a live-action sequence
US10279269B2 (en) 2017-06-22 2019-05-07 Centurion VR, LLC Accessory for virtual reality simulation
US11052320B2 (en) 2017-06-22 2021-07-06 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
US10456690B2 (en) 2017-06-22 2019-10-29 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
US11872473B2 (en) 2017-06-22 2024-01-16 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
US10792571B2 (en) 2017-06-22 2020-10-06 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
US10792572B2 (en) 2017-06-22 2020-10-06 Centurion Vr, Inc. Virtual reality simulation of a live-action sequence
EP3441849A1 (fr) * 2017-08-09 2019-02-13 Nagravision S.A. Dispositif d'entrée de pointage pour ordinateur personnel basé sur les mouvements des doigts
CN108151726A (zh) * 2017-11-24 2018-06-12 北京奇艺世纪科技有限公司 一种运动信息检测系统及方法
CN108151726B (zh) * 2017-11-24 2021-03-12 北京奇艺世纪科技有限公司 一种运动信息检测系统及方法
US12019801B2 (en) 2017-12-27 2024-06-25 Olaoluwa O. Adesanya Wearable computing apparatus for augmented reality, virtual reality and artificial intelligence interactions, and methods relating thereto
US11635812B2 (en) 2017-12-27 2023-04-25 Olaoluwa O. Adesanya Wearable computing apparatus for augmented reality, virtual reality and artificial intelligence interactions, and methods relating thereto
EP3732672A4 (fr) * 2017-12-27 2021-09-15 Adesanya, Olaoluwa O. Appareil informatique portable pour interactions de réalité augmentée, de réalité virtuelle et d'intelligence artificielle, et procédés associés
US12353624B2 (en) 2017-12-27 2025-07-08 Olaoluwa O. Adesanya Wearable computing apparatus for augmented reality, virtual reality and artificial intelligence interactions, and methods relating thereto
CN112041785A (zh) * 2018-02-09 2020-12-04 三星电子株式会社 跟踪手姿势的方法及其电子设备
US11481029B2 (en) 2018-02-09 2022-10-25 Samsung Electronics Co., Ltd. Method for tracking hand pose and electronic device thereof
WO2019156518A1 (fr) 2018-02-09 2019-08-15 Samsung Electronics Co., Ltd. Procédé de suivi d'une posture de main et dispositif électronique associé
CN112041785B (zh) * 2018-02-09 2024-12-31 三星电子株式会社 跟踪手姿势的方法及其电子设备
US12541256B2 (en) 2018-05-03 2026-02-03 Interdigital Vc Holdings, Inc. Systems and methods for physical proximity and/or gesture-based chaining of VR experiences
CN109782918B (zh) * 2019-01-25 2023-03-31 上海电子信息职业技术学院 一种计算机辅助教学装置
CN109782918A (zh) * 2019-01-25 2019-05-21 南阳理工学院 一种计算机辅助教学装置
US10867448B2 (en) 2019-02-12 2020-12-15 Fuji Xerox Co., Ltd. Low-power, personalized smart grips for VR/AR interaction
CN111158476A (zh) * 2019-12-25 2020-05-15 中国人民解放军军事科学院国防科技创新研究院 一种虚拟键盘的按键识别方法、系统、设备及存储介质
CN111236591A (zh) * 2019-12-26 2020-06-05 陶民 一种用于墙面喷涂、清洗的机器人
CN112351325A (zh) * 2020-11-06 2021-02-09 惠州视维新技术有限公司 基于手势的显示终端控制方法、终端和可读存储介质
CN112351325B (zh) * 2020-11-06 2023-07-25 惠州视维新技术有限公司 基于手势的显示终端控制方法、终端和可读存储介质
CN113223344B (zh) * 2021-05-25 2022-08-23 湖南汽车工程职业学院 一种基于大数据的艺术设计用专业教学展示系统
CN113223344A (zh) * 2021-05-25 2021-08-06 湖南汽车工程职业学院 一种基于大数据的艺术设计用专业教学展示系统
WO2023061552A1 (fr) * 2021-10-11 2023-04-20 Abb Schweiz Ag Interface d'apprentissage sensible pour la programmation d'un robot industriel
CN114281907A (zh) * 2021-11-01 2022-04-05 北京百度网讯科技有限公司 室内环境的行进处理方法、装置、电子设备及可读存储介质
US20250022353A1 (en) * 2021-11-08 2025-01-16 Bhaptics Inc. Device for providing tactile stimulation
US12437618B2 (en) * 2021-11-08 2025-10-07 Bhaptics Inc. Device for providing tactile stimulation
CN115240224A (zh) * 2022-07-22 2022-10-25 福州大学 基于三维手部关键点和图像特征融合的手势特征提取方法
CN115645906A (zh) * 2022-10-08 2023-01-31 网易（杭州）网络有限公司 一种虚拟单位的选择方法、装置、设备及存储介质
CN115562499B (zh) * 2022-11-16 2023-03-17 深圳市未来感知科技有限公司 基于智能指环的精准交互控制方法、系统及存储介质
CN115562499A (zh) * 2022-11-16 2023-01-03 深圳市未来感知科技有限公司 基于智能指环的精准交互控制方法、系统及存储介质
CN116236328A (zh) * 2023-02-22 2023-06-09 中国科学院沈阳自动化研究所 基于视觉的可实现自然抓取的智能假肢系统
CN116236328B (zh) * 2023-02-22 2025-08-12 中国科学院沈阳自动化研究所 基于视觉的可实现自然抓取的智能假肢系统
WO2025088497A1 (fr) * 2023-10-24 2025-05-01 Outplay Spolka Z Ograniczona Odpowiedzialnoscia Dispositif, en particulier pour commander un jeu vidéo
CN117697749A (zh) * 2023-12-25 2024-03-15 墨现科技(东莞)有限公司 设备控制方法、装置、抓握设备及存储介质

Also Published As

Publication number Publication date
CN107896508A (zh) 2018-04-10
WO2016189372A3 (fr) 2017-02-23

Similar Documents

Publication Publication Date Title
WO2016189372A2 (fr) Procédés et appareil destinés à une architecture "hyper ui pour dispositifs" centrée sur l'humain qui pourrait servir de point d'intégration avec de multiples cibles/points d'extrémité (dispositifs) ainsi que procédés/système associés permettant une entrée de geste sensible au contexte dynamique vers une plate-forme de contrôleur universel "modulaire" et une virtualisation du dispositif d'entrée
JP7731362B2 (ja) ウェアラブルシステムのための手のジェスチャ入力
JP7411133B2 (ja) 仮想現実ディスプレイシステム、拡張現実ディスプレイシステム、および複合現実ディスプレイシステムのためのキーボード
US11157080B2 (en) Detection device, detection method, control device, and control method
US12542051B2 (en) Systems and interfaces for location-based device control
US10318011B2 (en) Gesture-controlled augmented reality experience using a mobile communications device
TWI528227B (zh) 一種指環式無線指感控制器、控制方法及控制系統
EP3234742A2 (fr) Procédés et appareil pour une interface homme-ordinateur hautement intuitive
JP2023001328A (ja) ウェアラブルシステムのためのトランスモード入力融合
US20170336882A1 (en) Virtual/augmented reality input device
US20170277282A1 (en) Input device for transmitting user input
US20150253908A1 (en) Electrical device for determining user input by using a magnetometer
KR102021851B1 (ko) 가상현실 환경에서의 사용자와 객체 간 상호 작용 처리 방법
WO2018196552A1 (fr) Procédé et appareil pour affichage de type main destiné à être utilisé dans une scène de réalité virtuelle
US20170087455A1 (en) Filtering controller input mode
WO2022207821A1 (fr) Procédé d'interaction intégrée par le regard avec un environnement virtuel, système de traitement de données, et programme informatique
CN117590931A (zh) 人工现实环境中的手势移动
WO2003003185A1 (fr) Systeme permettant la mise en oeuvre d'une interface utilisateur
JP2024048680A (ja) 制御装置、制御方法、プログラム
KR101962464B1 (ko) 손동작 매크로 기능을 이용하여 다중 메뉴 및 기능 제어를 위한 제스처 인식 장치
KR102322968B1 (ko) 사용자의 손동작에 따른 명령 입력 장치 및 이를 이용한 명령 입력 방법
JP6444345B2 (ja) 仮想空間における入力を支援するための方法および装置ならびに当該方法をコンピュータに実行させるプログラム
Ferreira et al. ARTHE: experiencing projected Augmented Reality with THings of the Everyday
KR20250090310A (ko) 정보 처리 장치 및 정보 처리 방법
CN118466741A (zh) 一种虚拟场景的交互方法和相关装置

Legal Events

Date Code Title Description
NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16799422

Country of ref document: EP

Kind code of ref document: A2