HK1237444A1 - A human-computer interface device and system - Google Patents

Abstract

An interface device for a computerised system, the interface device including an engagement element configured for operable engagement with a region on a touchscreen of the computerised system to enable interfacing with the computerised system via the touchscreen, wherein the engagement element includes a viewing surface configured to allow a representation of content that is displayed at the region on the touchscreen with which the engagement element is operably engaged, to be visible through the viewing surface.

Description

Human-machine interface device and system

Technical Field

The present invention relates to devices and systems for use in interfacing with computerized systems, and in particular, interfacing via a touchscreen interface of a computerized system.

Background

The term "Human Computer Interface" (HCI) is commonly used to refer to the means by which a user may interact with a computerized system. Examples of HCIs include output devices such as monitor screens or printers, as well as input devices such as keyboards, touch screens, input panels, scanners, and/or mice.

One problem associated with touch screen type HCIs is that when a user is interacting with a touch screen by contacting the touch screen with a finger or stylus, the user's view of the content displayed on the touch screen is often blocked by the user's finger or stylus. These existing touch screen HCIs do not have the ability to compensate for what is blocked under the user's finger or stylus.

Disclosure of Invention

The present invention seeks to mitigate at least one of the problems described above.

The present invention may be directed to several broad forms. Embodiments of the invention may comprise one or any combination of the different broad forms described herein.

In a first broad form, the present invention provides an interface device for a computerized system, the interface device comprising an engagement element configured for operable engagement with a region on a touchscreen of the computerized system to enable interfacing with the computerized system via the touchscreen, wherein the engagement element comprises a viewing surface configured to allow a representation of content displayed at the region of the touchscreen to be visible through the viewing surface, the engagement element being operably engaged with the touchscreen.

Preferably, the engagement element may comprise at least one of a mouse and a stylus.

Preferably, the viewing surface may be substantially transparent.

Preferably, the viewing surface may be shaped to represent at least one of a game piece and a character.

Preferably, the viewing surface may comprise a lens.

Preferably, the content displayed at the area on the touch screen may include at least one of a still image and a video stream.

Preferably, the representation of the content displayed at the area of the touch screen may comprise an animation.

Preferably, the representation of the content may comprise an anthropomorphic character.

Preferably, the viewing surface may be configured such that the representation of content displayed on the area of the touchscreen that is visible through the viewing surface is represented in three dimensions.

Preferably, the present invention may be configured to enable the operational characteristics of the engagement elements to be sensed by the computerized system, and whereby the computerized system may be configured to control the output characteristics of the representation of the content at the area of the touch screen visible through the viewing surface in response to the sensed operational characteristics of the engagement elements.

Preferably, the operational characteristic of the engagement element may comprise at least one of a type of the engagement element, a two-dimensional position/orientation of the engagement element relative to the touch screen, a three-dimensional position/orientation of the engagement element relative to the touch screen, and a movement pattern or gesture of the engagement element relative to the touch screen.

Preferably, the present invention may incorporate a touch signaling array, wherein the operational characteristics of the engagement elements may be sensed by the computerized system with reference to the touch signaling array.

In a second broad form, the invention provides a human interface system comprising:

a touch screen configured for operable communication with a computerized system; and

an engagement element configured such that, under control of a person, the engagement element is operably engageable with a region on the touch screen to enable the person to interface with the computerized system via the touch screen;

wherein the engagement element includes a viewing surface configured to allow a representation of content displayed at an area on the touch screen to be visible through the viewing surface, the engagement element being operably engaged with the touch screen.

Preferably, the engagement element may comprise at least one of a mouse and a stylus.

Preferably, the viewing surface may be substantially transparent.

Preferably, the viewing surface may be shaped to represent at least one of a game piece and a character.

Preferably, the viewing surface may comprise a lens.

Preferably, the representation of the content displayed at the area on the touch screen may comprise at least one of a still image and a video stream.

Preferably, the representation of the content may comprise an animation.

Preferably, the representation of the content may comprise an anthropomorphic character.

Preferably, the viewing surface may be configured such that a representation of content displayed on an area of the touchscreen that is visible through the viewing surface may be represented in three-dimensional form.

Preferably, the invention may be configured such that the operating characteristic of the join element is sensed by the touch screen, and whereby the touch screen is configured to control the output characteristic of the representation of the content at the area of the touch screen visible through the viewing surface in response to the sensed operating characteristic of the join element.

Preferably, the operational characteristic of the engagement element may comprise at least one of a type of the engagement element, a two-dimensional position/orientation of the engagement element relative to the touch screen, a three-dimensional position/orientation of the engagement element relative to the touch screen, and a movement pattern or gesture of the engagement element relative to the touch screen.

Preferably, the invention may incorporate a touch signalling array, wherein the operating characteristics of the engagement elements may be sensed by the touch screen with reference to the touch signalling array.

Preferably, the present invention can be configured to be used as at least one of an automated assistant for a person, an advertising medium, a therapeutic tool, an educational medium, a toy, an entertainment medium, and a game medium.

In a third broad form, the present invention provides a robot for use with a touch screen of a computerized system, the robot comprising:

a positioning mechanism configured for positioning the robot at an area of the touch screen;

an engagement element configured to operably engage with the touch screen, an operating characteristic of the robot configured for sensing by a sensor of the computerized system when the engagement element is operably engaged with the touch screen; and

a processing module;

wherein in response to the sensed operational characteristic of the robot, the processing module is configured to perform at least one of:

(a) controlling the robot to perform an action with reference to the sensed operating characteristic of the robot; and

(b) predetermined content is displayed on the touch screen with reference to the sensed operating characteristics of the robot.

Preferably, the robot may contain a character representation of, for example, a pet dog or a chess piece.

Preferably, the operating characteristic of the robot may comprise at least one of a type of the robot and a position/orientation of the robot relative to the touch screen.

Preferably, the processing module may comprise at least one of a smartphone and a touch screen.

Preferably, the bot may include a viewing surface via which representations of content displayed at regions of the touchscreen are visible through the viewing surface, the bot being operably engaged with the touchscreen.

Preferably, the present invention may include a sensor for sensing an operating characteristic of the robot with reference to a capacitive touchpad disposed on the robot, the touchpad configured to be operably engaged with a touch sensor array of a touch screen.

Preferably, the computerized system may be configured to control the actions of the robot and display content on the touchscreen that may simulate interactions between the robot and the content displayed on the touchscreen.

Preferably, the computerized system may be configured to sense a position/orientation of the human user and to control the robot to move towards the human user and/or to perform a greeting in response to the sensed position/orientation of the human user.

In a fourth broad form, the invention provides a human interface system comprising:

(a) at least one separable optic operable to transmit light from a touch screen through the optic,

(b) at least one feature attached to the lens operable to be sensed by a touch screen.

Preferably, the present invention may comprise:

(a) a first set of software instructions operable to receive and process input from at least one feature attached to the lens to estimate at least some state of the lens,

(b) a second set of software instructions operable to determine a response to an output of the first set of software instructions.

Preferably, at least some portion of the response may be operable to control animation of at least one image on the touch screen.

Preferably, at least some portion of the animation may be transmitted through the lens.

Preferably, the invention may comprise software operable to control animation like at least some human non-verbal communication.

Preferably, the present invention may further comprise at least one stylus attached or attachable to the TLM.

Preferably, the present invention may further comprise the form of at least one lens operable to provide a three-dimensional shape to an image transmitted through the lens.

Preferably, the present invention may further comprise a set of software instructions operable to detect at least one pattern in the TLM and thereby identify or estimate at least one of: a particular TLM or TLM type, an orientation of the TLM, a 3D location of the TLM, other particular states of the TLM, and/or a state of a user of the TLM.

Preferably, the present invention may further comprise a lens molded to form the playing piece.

Preferably, the lens can be molded into a character.

Preferably, the present invention may further comprise means operable to detect a specific gesture action of the TLM.

In general, the invention can serve as an interface between at least one person and at least one computer.

In general, the present invention can serve as an interface between at least one person and at least one computer network.

In general, the present invention may act as an automated assistant for a person.

In general, the present invention may serve as an advertising medium.

In general, the present invention may serve as a therapeutic tool.

In general, the present invention may serve as an educational tool.

In general, the present invention may function as a toy.

In general, the present invention may serve as an entertainment medium.

In general, the present invention may serve as a game medium.

In a fifth broad form, the present invention provides a machine for human machine interface comprising:

(a) a detachable mirror having features operable to send input to the touch screen,

(b) an animation operable to respond to the action of the feature.

Preferably, at least some portions of the animation may resemble at least some portions of an anthropomorphic character.

Preferably, the invention may further comprise an element operable to transmit the animation through the lens.

Preferably, the animation may represent information or data.

In general, the present invention may serve as an advertising medium.

In general, the present invention may serve as an educational tool.

In general, the present invention may function as a toy.

In general, the present invention may serve as an entertainment medium.

Preferably, the present invention may further comprise a lens molded to form the playing piece.

Preferably, the present invention may further comprise a lens that molds the character.

Preferably, the present invention may further comprise a lens operable as a display.

In another broad form of the invention, there is provided a human interface system comprising separable lens hardware operable to be touched on and sensed by a touch screen, and a sensor operable to register the position of the lens hardware under the control of a computing system. The system may include a first set of software instructions operable to receive and process input from a sensor in response to a lens determining a position and orientation of the lens on a screen display. The system may further include a second set of software instructions operable to determine a response to the lens position, whereby the computing system will output a signal corresponding to the response such that, in at least some instances, the output signal controls the animation of the image under the lens.

The present invention may assist in providing several important technical advantages. Embodiments of the invention may lack, or have some or all of these technical advantages without departing from the scope of the invention. The present invention may provide a more natural mechanism for people to interact with various electronic devices. By providing the transparent lens as a stylus or mouse, the present invention facilitates the display of images under the mouse that might otherwise be blocked by the mouse. Another important technical advantage is that the present invention can assist in displaying changing information to a person with whom a computer is interacting in an image under a lens to better facilitate computer-to-person interaction.

Drawings

The present invention will become more fully understood from the detailed description of the preferred but non-limiting embodiments of the invention, which is described below in conjunction with the accompanying drawings, wherein:

FIG. 1 illustrates one embodiment of the present invention as a "touch lens mouse" (TLM);

FIG. 2 illustrates an example of the use of a TLM embodiment in conjunction with a touch screen;

FIG. 3 illustrates another embodiment of the present invention in which the TLM is shown as a mouse with cross hairs to show position and click methods;

FIG. 4 is a block diagram illustrating one embodiment of a control system configuration that may be used to achieve control using TLMs;

FIG. 5 is a block diagram illustrating an embodiment of a TLM format used to access, navigate, or modify information presented on a computer network.

Detailed Description

Preferred embodiments of the present invention and its advantages are now described herein with reference to fig. 1-5 of the drawings, like numerals being used for like and corresponding parts of the various drawings.

One aspect of the present invention is an HCI system, referred to as a Touch Lens Mouse (TLM), that includes substantially transparent lenses that feature multi-touch signals when touched or in close proximity to a touch screen in order to produce human-machine interactions that are more enjoyable, easier to understand, and/or more informative than previous HCI systems. One possible means for sensing TLMs by a Touch screen may include a Touch Signaling Array (TSA) comprising a set of elements to signal a Touch detector with a controlled set of Touch points comprised of any suitable material (e.g., rubber and/or metal) embedded within or otherwise attached to a lens or mouse-like article operable to transmit signals to a Touch detection surface and/or area; such TSAs may be arranged in mutually distinguishable signature configurations to identify a given TSA relative to another TSA and to signal the orientation and position of the TSA.

In one embodiment, the TLM includes a device and/or software for the TLM to transmit at least some gestural non-verbal human communication signals, a device and/or software for the TLM to determine a meaningful response to a perceived sensation, and a device and/or software for the TLM to display human input responses using the TLM. To sense and perceive natural human communication signals, TLMs may utilize, without limitation, touch tracking, multi-touch sensing, and pattern recognition. Other hardware and/or software systems may also be used. To decide on a meaningful response to a perceived sensation, the TLM may use any, all, or none of an expert knowledge system, a logical inference system, and/or a statistical inference system. Other related systems may also be used. To display the communication output, the TLM may utilize animations, such as (but not limited to) facial animations, facial expressions, and/or gestures that are transmitted under and up through the TLM lens to affect a 3-dimensional display via the TLM lens.

TLMs may be developed through coordinated integration of other display (or other output) technologies in addition to the animated characters or other displays in the TLM. In some embodiments, this additional output technique may supplement natural communications with conventional computer graphics and text, sound, and the like. Techniques for generating various odors may also be used. Techniques for producing a tactile sensation may also be used. Techniques for generating body gestures and/or motions may also be used.

The following description presents many other options that may be associated with the TLM of the present invention. The following description also presents numerous other structures, methods, compositions of matter, etc., each of which may constitute a separate invention.

Before describing the drawings in more detail, the following description will provide some additional overview information regarding the various embodiments and aspects of the invention disclosed herein. The following description discusses various aspects of TLM and various options for its use. The statements herein are not intended to be exhaustive or to limit the invention in any way not specified in the claims of any issued patent based on this description.

An organism, character, robot, and/or other creature can be simulated in the TLM system by using any suitable animation variants exhibited with the TLM system; herein and hereinafter, "animation" shall mean an animated simulation of the visual behavior of living beings including, but not limited to, skin, fur, flesh, feathers, and/or the like. Any suitable display technology, such as, but not limited to, lcd, tft, plasma, led, oled, hologram, and/or rear projection, may be used anywhere in the displays described in this patent, and anywhere in the TLM system where a display is used. All displays described in use in this patent may be touch screens, but they need not be.

The present invention may also include a mechanical system for achieving conductive multi-touch via a lens or mouse-like object, and will be referred to as a "touch signaling array" hereinafter. The touch signaling array or TSA comprises a set of elements to signal a touch detector with a controlled set of touch points comprised of any suitable material (e.g., rubber and/or metal) embedded within or otherwise attached to a lens or mouse-like object operable to transmit signals to a touch detection surface and/or area.

The invention also relates to several embodiments of a touch signaling array implementing a particular multi-touch gesture. One embodiment of the touch signaling array may simulate the appearance of a dynamic pattern of multiple human fingers, simulating actions similar to those of a human multi-touch input to control information input to a computing system.

Another aspect of the invention may include embodiments of a touch-lens mouse, for example, which may or may not be used in applications including, but not limited to, board games, virtual character display and control, vehicle control systems, environmental control, and/or robotic control.

Another aspect of the invention may include embodiments of a touch lens mouse, for example, which may or may not be used as a generic means for browsing, navigating and/or modifying information on a computer and/or computer network (such as, but not limited to, a web page, a cloud, an intranet, and/or a cellular network), such that interaction with a TLM system may provide for more attractive and higher bandwidth information exchange than some alternative HCIs. In these cases, the TLM may be used for one, some, all, or none of the following operations: (a) displaying information through TLM lenses, (b) allowing manipulation of information represented by icons, symbols and/or characters using visual and gestural aspects of TLMs, (c) obtaining data from human users through gestures, TLM manipulation, tactile sensing and/or accessing user-related data and integrating these data into a model of the user's cognitive, emotional and/or social state, (d) performing diagnosis and assessment of user data through the use of expert system databases, big data analytics, and/or inference systems that may or may not be based on standard knowledge of cognition and psychology, (e) using natural displays expressed through TLMs and synchronized speech to deliver gestures, knowledge, information, entertainment, education, advertising and/or assistance, (f) helping to assess the user's behavior, developmental performance and target progress, (g) performing any other display of information or characters, or any other manipulation of information that is currently performed by other HCIs and/or related computing systems or that may be otherwise serviced by the TLM system.

Embodiments of TLMs may be used as a development training tool, such as an educational plaything, educational toy, or play device and/or a development therapy device, such that interaction with the tool may provide a health development benefit and/or an entertainment benefit to a child. In these cases, the TLM may be used for one, some, all, or none of the following operations: (a) providing cultural, educational, and/or entertaining interaction with children via characters displayed under TLM lenses, (b) maintaining the attention of children to natural displays and verbal expressions through different means that may include games, caring for virtual pets or other characters displayed through TLM, and/or telling stories, to deliver knowledge, encouragement, assignment, and/or behavioral changes to children, (c) obtaining data from children through gestures, visual inspection, tactile sensing, and/or access to children-related data, (d) performing diagnosis and assessment of children's data through the use of expert system databases, big data analytics, and inference systems that may or may not be based on standard knowledge of education and developmental psychology, (e) therapeutically and/or educating and/or grouping children with interactions, speech, and/or gestures, (f) providing a parent, medical personnel, and/or educator with an assessment of the child's behavior, developmental performance, and progress, (g) performing any other display of information or characters, or allowing any other manipulation of information that is currently being performed by other HCIs, toys, and/or related computing systems, or that may be otherwise serviced by the TLM system.

Embodiments of TLMs may be used, for example, as display devices for game toys, game pieces, action figures, dolls, novelty items, simulated animals, and/or video games. In these cases, the TLM may be used for one, some, all, or none of the following operations: (a) maintaining the user's attention through different means and verbal expressions that may include games, interactive conversations, interactions with virtual pets or other characters displayed through TLMs, and/or storytelling, (c) obtaining data from the user through gestures, TLM input, visual inspection, tactile sensing, access to user-related data, or any other means, (d) performing diagnostics and evaluation of user data through the use of expert system databases, big data analytics, smart systems, and/or any other means, (e) influencing the user with interactions, speech, and/or gestures, (f) compiling and presenting an evaluation of user status, behavior, developmental performance, and progress to (g) to perform any other display of information or characters, or any other manipulation of information that is currently being performed by other HCIs, toys, and/or related computing systems or that may be otherwise serviced by the TLM system, (h) connecting to a computer network for multi-user gaming and/or accessing additional knowledge and/or data for enhanced fun functionality.

Embodiments of TLMs may be used as artistic controls including, but not limited to, digital sculpture input, CAD control, music production, movie editing, automated theater control, and/or automated architectural space. In these cases, the TLM may be used for one, some, all, or none of the following operations: (a) providing an aesthetic, cultural, and/or narrative expression for the artist, (b) maintaining the viewer's attention through different means that may include games, language interactions, performance, automatic context changes, and/or storytelling, (c) obtaining data from individuals or people through TLM interactions, (d) evaluating the data in some way, (e) determining responses, (f) using natural displays, verbal expressions, and automatic context changes to deliver sensory impressions to viewers targeting cultural or psychological impact, (f) providing a record and evaluation of people's responses to artwork for artwork supplementation, for afterhistory, and/or for scientific evaluation, (g) executing any other service that can automate content currently being executed by a mouse or stylus or display or character or form of art.

While the above description has provided various options for implementing TLMs and various applications of TLMs, the described TLMs may omit some or all of the described applications and/or mechanical, electronic, electromechanical, software, and other systems, without departing from the scope of the invention. Other systems may be substituted for those disclosed. Other applications may be substituted for those disclosed. While various applications of TLM have been described, TLM may be used for other applications without departing from the scope of the present invention. While a TLM may have the capability to perform all of the applications described herein, it may perform no function, some functions, or other functions without departing from the scope of the present invention. Those skilled in the art will further appreciate that the above description of mechanical systems may include electronic, electromechanical, and biological components. Such systems may also involve computer software.

Fig. 1 illustrates one embodiment of a touch lens mouse, including a conductive element 8, a light transmissive lens 9, and an insulating element 10 that together function as a TSA.

Fig. 2 illustrates one embodiment of a touch-lens mouse, where a human user 11 places a TLM 12 on a touch screen with displayed imagery 13, showing the touch screen imagery through the TLM lens 12, and showing the imagery being manipulated by the TLM to a new location 14 (as shown in fig. 4).

FIG. 3 illustrates one embodiment of a touch lens mouse, wherein the TLM has a handle 15 that allows the TLM to function as a stylus, and a cross-hair 16 marked on the TLM lens that allows the user to more accurately aim at an object in a touch screen display environment.

Figure 4 is a block diagram illustrating one embodiment of a TLM system configuration that may be used to illustrate one embodiment, including a TLM mouse on a touch screen 60 and an electronic control system 88 that governs the operation of various mechanisms in order to achieve at least some inputs to the computer using the TLM TSA to affect data control in the computer and controlled outputs from the control system, e.g., imaging changes on the touch screen. Data may be sent from the TLM sensor to the computer by any suitable communication medium, including but not limited to a cable and/or wireless link. The same or separate communication links may be used for both input and output, and multiple communication links may be used without departing from the scope of the invention. The sensor data may be forwarded to a computing system 88, which in the figure includes a computer and various software, but may reside within a microcontroller, computer network, cellular telephone, tablet, or any other computing hardware and/or software. The functions performed by computing system 88 may also be performed in whole or in part by dedicated hardware. Although the computing system 88 is depicted in fig. 4 as residing outside of the TLM, the computing system 88 may alternatively be partially or fully enclosed within the TLM without departing from the scope of the present invention. Automatic Speech Recognition (ASR)89 may process the relevant audio data to detect Speech and extract words and low-level language meaning. The computer vision 90 may use the video data to perform any of a variety of visual perception tasks, such as detecting human emotions or TLM mouse movements. The decision process 91 may determine appropriate communication responses and other actions based on the received input. Special programs 92 may be used to enhance the functionality of the system. The special program 93 is other software accessed by the TLM. The data storage 94 enables long term storage, knowledge building, and software storage. The speech or other audible response may be generated by any suitable speech synthesis software 95 and made audible using an audio transducer. Any suitable video controller 96 may be used to control animations to implement display actions in the TLM, such as (but not limited to) character animations. Some or all of the components of computing system 88 may be omitted or other components added without departing from the scope of the invention. With respect to the software modules, FIG. 5 shows each of the operations performed in more detail, and describes each's commercially available options. Although the sensors are described as being placed within the TLM system in the described embodiments, the TLM device does not require any of the sensing elements (which may include any, all, none, or others: cameras, microphones, olfactory sensors, touch sensors, speed sensors, accelerometers, and/or positioning sensors) to be integrated into the TLM. Any and/or all of the sensors may be placed in other locations outside of the TLM (or in other locations within the TLM than described herein) without altering the scope of the present invention. In addition, other sensors may or may not be utilized as desired, such as bar code scanners, infrared sensors, ultrasonic sensors, echo location sensors, touch sensors, velocity sensors, accelerometers, location sensors, fingerprint scanners, retina scanners, blood pressure meters, pulse detectors, EEG detectors, megahertz imaging devices, magnetic resonance imaging devices, ground penetrating radars, galvanic skin detectors, and/or any other suitable sensing, sensing or data collection and analysis instrumentation and techniques. The TLM may or may not also access data from external sensors, possibly including sensors contained within other devices (e.g., weather stations, automotive sensors, "smart dust" sensors, robots, and/or other TLM devices), via any suitable data transmission means (e.g., a computing network).

FIG. 5 is a block diagram illustrating one embodiment of a control system configuration that may be used to simulate at least some human-machine interfaces and optional human-human communications in a TLM system. In the depicted embodiment, the input pad 60, video sensor, and audio sensor 67 with a touch screen in the TLM interface 13 transmit touch data and (optionally) related video and audio data to the computing system 88 in response to the human user moving the TSA 12. A set of software and/or embedded hardware processors running in the computational space, referred to as Automatic Speech Recognition (ASR)89, process the audio data to detect speech and extract words and low-level linguistic meanings. Any suitable software may be used to implement ASR, including, for example, the various available systems from Sphinx or Nuance Dragon university in camion. Another set of software and/or embedded hardware handlers referred to as computer vision 90 may use the video data to perform any of a variety of vision-aware tasks. Such visual perception tasks may include detecting and tracking a person's face using any suitable software, which may include, for example, intel's open computer vision library or microsoft's Kinect SDK's various available software. Other visual perception processes may also include tracking facial features or expressions on a person's face, and any suitable software may be used for this task, for example, google's facial perception software. The decision process 91 may be used to determine appropriate communication responses and other behaviors, and any suitable software may be used for this purpose, such as a game engine (e.g., JMonkey or Unreal) or a session engine (e.g., any of the public or commercially available "chat robot" software, such as "Eliza" and "Alice"). Alternatively, one skilled in the art may create software specific to the applications described above to implement the claimed invention. Other processes, referred to as special procedures 92, may be any off-the-shelf or custom software that may be bound to the TLM, including software that enhances the functionality of the TLM for specific applications not included in standard software, such as software that monitors vital functions of the patient, software that detects the presence of a chemical hazard, and/or software that causes a fire or burglar alarm. Other processes, referred to as special programs 93, may be used to enhance the functionality of the system in other ways. These specific programs 93 can be any off-the-shelf software that can be accessed by the TLM, which becomes the layer through which the user interacts with the specific programs; this software may include any, all, or none of the following: spreadsheet software, web browsing software, email programs, word processing software, electronic books, expert systems, artificial intelligence software, math software, geo-location software, and/or gaming software. Data storage 94 may be used to enable long-term storage, build knowledge, and store software. The voice response may be synthesized by any suitable voice synthesis software 95 (e.g., the open source software "Festival") to become audible using the audio transducer 70. Any suitable video controller 96 may be used to implement the non-verbal expressions in TLM 12. The non-verbal response may operate independently of the speech (in one embodiment, by direct emulation) or in synchronization with the content of the synthesized speech.

The following sections discuss data flow through one embodiment of a TLM control system. The touchscreen 60 in the TLM 13 may send signals in response to the gestural actions of the TSA 12 to the computation space 88, which estimates the state of the TSA 12, e.g., position, orientation, trajectory, proximity, and/or pressure on the touchscreen. Status data regarding this TSA may be sent to a decision process 91, which may decide to react to the TSA signal based on variables governing TLM system decisions. If the TLM decides to react to the perceived person, data regarding the expected location of the TLM under the control of the person may be sent to the video controller 96. The video control signal may travel to the touch screen 60 in the TLM interface 13 that controls the animation position and other variables to cause the animation tracking to be perceived as TSA, so that the animation may appear as part of the TLM, and so that the animation may appear as the TLM approaches the touch screen and disappear as the TLM lifts away from the touch screen, thereby creating the illusion that the animation actually originated within the lens. Decision process 91 can also send a signal to video controller 96 to implement a greeting smile; video controller 96 may generate and send appropriate signals to the animation to simulate eating or fighting or other character behavior. The decision process 91 may also send a signal to the speech synthesis 95 to send out a greeting. The speech synthesis 95 may generate a corresponding voice greeting signal and send this signal to the audio transducer 70 in the TLM so that the signal can be heard by a person. After these greeting gestures, the person may return a greeting and then request whether any new email messages have been sent for the person. The microphone in the TLM may detect the speech data, transmit the data to ASR 89, which may decipher the words and may decipher some of the language content. The resulting linguistic analysis will be forwarded to a decision process that can further decipher the linguistic content to understand the person's request. The decision process may invoke a special program 92 for the personal assistant function to translate the email request into the appropriate web browsing function. The special program 92 may then send a signal to the special program 93 which opens the appropriate server and URL on the internet via the network 97 to open the person's email account. The special program 92 then decrypts the hypertext of the message to determine what email message is new and returns this data to the decision process 91 which converts the information into natural language that is forwarded to the speech synthesis 95 which then generates the speech signal that is forwarded to the audio transducer 70 in the TLM. The transmission of this audible signal completes the control loop. During the described transaction, all data can be transferred from the decision process, the special program and the specific program to the data memory 94, where it can be recorded as needed for future reference. The data memory 94 may also be used to store software that is not actively used to call it into executing tasks (active duties) as needed to save processing resources.

Herein, the touch lens mouse can also be used to access, navigate or modify information presented on a computer network 97, where the computer is connected to the computer network 97 in a manner such that the computer can access data presented in other computers; this data is then translated into a simulated natural animation for transmission through the touch screen 60. The TLM may act as an interface to the virtual personal assistant by: this assistant is displayed through the glasses and performs helpful functions in response to natural dialog and TSA gestures from the user. The TLM can act as an alternative display to various computer-generated characters of a smart animated character, a "chat robot," or an avatar (as opposed to "virtual" imagery on a display screen, where the TLM is three-dimensional and emphasized within the TLM lens). In one such embodiment, the TLM can act as an interface to the video game such that the TLM represents the controller and 3D display of the video game character, and in so doing, the character appears to exist outside the game, in a physically embodied world, thus enhancing the video game user's experience and opening many new possibilities in video game design.

In an alternative embodiment of the present invention, a computerized system is provided that includes a touchscreen (e.g., an input board or IPAD) and a robot (e.g., a dual motor robot) that is suitably sized and dimensioned to operatively engage and move across a surface of the touchscreen under remote wireless control. The camera is operatively connected to the touch screen and is capable of determining a position of the human user relative to the touch screen with reference to an image captured by the camera.

The robot includes a positioning mechanism configured to allow the robot to be positioned across the surface of the touch screen. The positioning mechanism may include a wheel assembly, a tractor engagement mechanism, a slip engagement mechanism, and the like.

The robot includes a capacitive touch pad disposed on the underside surface that is configured to be sensed by a sensor array disposed on the touch screen when the robot operably engages the touch screen so as to provide position/orientation information of the robot relative to the touch screen surface at any given time. The capacitive touch pad also provides other information indicative of the operational characteristics of the robot sensed by the touch screen. For example, the capacitive touchpad pattern may be fabricated on a robot in a specific and unique pattern that indicates a particular type or identity of the robot interacting with the touchscreen. After each specific and unique capacitive array pattern is sensed and a specific type or identity of the robot is recognized by using a detection algorithm, the touchscreen is configured to initiate a specific response or sequence of responses that match the identity of the robot. A specific and unique pattern may be used to distinguish robots shaped and made to resemble different types of chess pieces, different types of animals, different types of pets, etc. For example, one robot may have 3 capacitive touch pads arranged in a triangular pattern with sides having lengths of 3mm, and 5mm, but another robot may have 3 capacitive touch pads arranged in a triangular pattern with sides having lengths of 4mm, and 3 mm.

In one example, in response to the sensed positions/orientations of the robot and the human user, the touch screen may be configured to transmit a wireless control signal to the robot to direct the robot to move toward the position of the human user, and transmit a further wireless control signal to the robot to cause it to perform a greeting for the human user. Alternatively, a smartphone or other computing device may be used to wirelessly transmit control signals to the robot to control the movement and operating characteristics of the robot. In yet another example, the robot may represent a pet dog, whereby in response to sensed positions/orientations of the pet dog robot and the human user, the touch screen may be configured to display content on the touch screen and/or wirelessly transmit movement control signals to the robot that cause the robot to interact with the content on the touch screen and/or the human user. For example, after identifying the robot as a "pet dog," the touchscreen may generate content representing a dog disk in a kitchen scene. The touchscreen is configured to then wirelessly transmit a control signal to the pet dog robot, whereby the robot sounds a whine, moves toward the dog disk, scratches and turns at the dog disk to look at the location of the human user to entrust the dog food. The touch screen is further configured to receive and process human user finger gestures via the touch screen, which may cause a representation of a dog food bag to be displayed on the touch screen, which the human user may then drag with his/her fingertips towards an image of the dog plate to hold the dog plate. The touch screen may be further configured such that when the image of the dog food bag touches the image of the dog plate, a video of the dog food pouring from the bag into the dog plate occurs. Thereafter, the touchscreen is configured to transmit control signals to the robot dog that cause it to dance cheerfully before eating dog food, while sounding and turning to gazing at the location of the human user.

In another example, in response to sensed position/orientation information or other sensed operational characteristics of the robot dog, the touch screen may be configured to generate content representative of the toy, and whereby the touch screen is configured to transmit control signals to the pet dog robot to cause the pet dog robot to simulate movement of the toy representation across the touch screen surface toward a human user. The touch screen is further configured to receive and process human user instructions in the form of finger gestures entered through the touch screen, whereby the toy is able to bunch up with the toy from the pet dog robot and make the toy image displayed on the touch screen appear to be "thrown" across the touch screen. In response to the toy image being "thrown" across the touch screen, the touch screen is configured to transmit control signals to the pet dog robot to cause it to move across the touch screen in a manner that instructs the pet dog robot to pick up the toy and return the toy to the human user.

In some embodiments of the invention, the robot further includes a TLM-type lens, such that a representation of content displayed on the touch screen can be projected in three dimensions up through the TLM-type lens.

Embodiments of the present invention may be capable of providing a physically embodied computer animation. That is, content displayed in animated form on the touch screen may be represented by a robot operatively engaged with the touch screen that is capable of physically moving about the touch screen and physically embodying a facial expression representing the animated content displayed on the touch screen. Costs in implementing a physically embodied animation process are reduced by using a robot with dedicated computing and by wirelessly controlling the robot from an external general purpose computing (robot), such as a cell phone, PC or cloud. Advantageously, the use of a curved or flexible screen display and the robot may assist in displaying facial expressions or other imagery.

In some embodiments, the parts of the robot (e.g., body, head, and legs) used in these embodiments may be sold separately and modularly assembled by the user, an advantage of this being to provide a more personalized user experience.

Although the present invention has been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.

It will be understood by those skilled in the art that the invention described herein is susceptible to variations and modifications other than those specifically described without departing from the scope of the invention. All such variations and modifications as would be obvious to one skilled in the art are intended to be included within the spirit and scope of the present invention as broadly described above. It is to be understood that the invention encompasses all such variations and modifications. The invention also includes all of the steps and features referred to or indicated in the specification, individually or collectively, and any and all combinations of any two or more of said steps or features.

The reference to any prior art in this specification is not, and should not be taken as, an acknowledgment or any form of suggestion that prior art forms part of the common general knowledge.

To assist the patent office and any reader of any patent issued in relation to this application in interpreting the claims appended hereto, applicants wish to note that it is not intended that any of the appended claims be incorporated by reference into the U.S. patent law, article 112 (35u.s.c. § 112) paragraph 6, which exists at the time of filing of the present application, unless "means for …" or "step for …" is used in a particular claim.

Claims (33)

1. An interface device for a computerized system, the interface device comprising an engagement element configured for operable engagement with a region on a touchscreen of the computerized system to enable interfacing with the computerized system via the touchscreen, wherein the engagement element comprises a viewing surface configured to allow a representation of content displayed at the region on the touchscreen to be visible through the viewing surface, the engagement element being operably engaged with the touchscreen.

2. The interface device of claim 1, wherein the engagement element includes at least one of a mouse and a stylus.

3. An interface arrangement according to claim 1 or 2, wherein the viewing surface is substantially transparent.

4. The interface device according to any one of the preceding claims, wherein the viewing surface is shaped to represent at least one of a game piece and a character.

5. The interface device of any one of the preceding claims, wherein the viewing surface includes a lens.

6. The interface device according to one of the preceding claims, wherein a representation of the content displayed at the area on the touchscreen includes at least one of a still image and a video stream.

7. The interface device according to one of the preceding claims, wherein a representation of the content displayed at the area of the touchscreen includes an animation.

8. The interface device according to one of the preceding claims, wherein the representation of content includes an anthropomorphic character.

9. The interface device according to one of the preceding claims, wherein the viewing surface is configured such that a representation of content displayed on the area of the touchscreen that is visible through the viewing surface is represented in three dimensions.

10. An interface device according to any one of the preceding claims, configured to enable an operating characteristic of the engagement element to be sensed by the computerized system, and whereby the computerized system is configured to control an output characteristic of the representation of content visible through the viewing surface at the area on the touchscreen responsive to the sensed operating characteristic of the engagement element.

11. The interface device of claim 10, wherein the operational characteristics of the engagement element include at least one of a type of the engagement element, a two-dimensional position/orientation of the engagement element relative to the touch screen, a three-dimensional position/orientation of the engagement element relative to the touch screen, and a movement pattern or gesture of the engagement element relative to the touch screen.

12. The interface device according to claim 10 or 11, including a touch signaling array, wherein the operating characteristics of the engaging elements are capable of being sensed by the computerized system with reference to the touch signaling array.

13. A human interface system, comprising:

a touch screen configured for operable communication with a computerized system; and

an engagement element configured such that under control of a person, the engagement element operably engages with an area on the touch screen to enable interfacing of the person with the computerized system via the touch screen;

wherein the engagement element includes a viewing surface configured to allow a representation of content displayed at the area on the touch screen to be visible through the viewing surface, the engagement element being operably engaged with the touch screen.

14. The human interface system of claim 13, wherein the engagement element includes at least one of a mouse and a stylus.

15. The human interface system of claim 13 or 14, wherein the viewing surface is substantially transparent.

16. The human interface system according to any one of claims 13-15, wherein the viewing surface is shaped to represent at least one of a game piece and a character.

17. The human interface system according to any one of claims 13-16, wherein the viewing surface includes a lens.

18. The human interface system according to one of claims 13-17, wherein the representation of content displayed at the area on the touchscreen includes at least one of a still image and a video stream.

19. The human interface system of any one of claims 13-18, wherein the representation of content includes animation.

20. The human interface system of any one of claims 13-19, wherein the representation of content includes an anthropomorphic character.

21. The human interface system according to one of claims 13-20, wherein the viewing surface is configured such that a representation of the content displayed on the area of the touchscreen that is visible through the viewing surface is represented in three dimensions.

22. The human interface system according to any one of claims 13 to 21, configured to enable an operating characteristic of the engagement element to be sensed by the touch screen, and whereby the touch screen is configured to control an output characteristic of the representation of content at the area of the touch screen visible through the viewing surface in response to the sensed operating characteristic of the engagement element.

23. The human interface system according to claim 22, wherein the operational characteristics of the engagement element include at least one of a type of the engagement element, a two-dimensional position/orientation of the engagement element relative to the touch screen, a three-dimensional position/orientation of the engagement element relative to the touch screen, and a movement pattern or gesture of the engagement element relative to the touch screen.

24. The human interface system according to claim 22 or 23, including a touch signalling array, wherein the operating characteristics of the engagement elements are capable of being sensed by the touchscreen with reference to the touch signalling array.

25. A human-machine interface system, wherein the system is configured to be used as at least one of an automated assistant for a person, an advertising medium, a therapeutic tool, an educational medium, a toy, an entertainment medium, and a gaming medium.

26. A robot for use with a touch screen of a computerized system, the robot comprising:

a positioning mechanism configured for positioning the robot at an area of the touch screen;

an engagement element configured to operably engage with the touch screen, an operating characteristic of the robot configured for sensing by a sensor of the computerized system when the engagement element is operably engaged with the touch screen; and

a processing module;

wherein in response to the sensed operational characteristic of the robot, the processing module is configured to perform at least one of:

(c) controlling the robot to perform an action with reference to the sensed operating characteristic of the robot; and

(d) displaying predetermined content on the touch screen with reference to the sensed operational characteristics of the robot.

27. A system according to claim 26, wherein said robot comprises a character representation, such as a pet dog or a chess piece.

28. The system of claim 26 or 27, wherein the operating characteristics include at least one of a type of the robot and a position/orientation of the robot.

29. The system of any one of claims 26-28, wherein the processing module includes at least one of a smartphone and the touchscreen.

30. The system of any of claims 26-29, wherein the robot includes a viewing surface via which representations of content displayed at regions of the touchscreen are visible through the viewing surface, the robot being operably engaged with the touchscreen.

31. The system of any of claims 26 to 30, including sensors for sensing the operating characteristics of the robot with reference to a capacitive touchpad disposed on the robot, the touchpad configured to operably interface with a touch sensor array of the touchscreen.

32. The system of any of claims 26-31, wherein the computer is configured to control actions of the robot and display content on the touchscreen that simulates interactions between the robot and the content displayed on the touchscreen.

33. The system of any of claims 26-32, wherein the computerized system is configured to sense a position/orientation of a human user and, in response to the sensed position/orientation of the human user, control the robot to move towards the human user and/or execute a greeting.