WO2015004620A2 - Compagnon virtuel - Google Patents

Compagnon virtuel Download PDF

Info

Publication number
WO2015004620A2
WO2015004620A2 PCT/IB2014/062986 IB2014062986W WO2015004620A2 WO 2015004620 A2 WO2015004620 A2 WO 2015004620A2 IB 2014062986 W IB2014062986 W IB 2014062986W WO 2015004620 A2 WO2015004620 A2 WO 2015004620A2
Authority
WO
WIPO (PCT)
Prior art keywords
virtual
virtual companion
companion
user
touch
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/IB2014/062986
Other languages
English (en)
Other versions
WO2015004620A3 (fr
Inventor
Victor Wang
Shuo DENG
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
GeriJoy Inc
Original Assignee
GeriJoy Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US13/939,172 external-priority patent/US20140125678A1/en
Application filed by GeriJoy Inc filed Critical GeriJoy Inc
Priority to CN201480002468.2A priority Critical patent/CN104769645A/zh
Publication of WO2015004620A2 publication Critical patent/WO2015004620A2/fr
Publication of WO2015004620A3 publication Critical patent/WO2015004620A3/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20—Input arrangements for video game devices
    • A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/214—Input arrangements for video game devices characterised by their sensors, purposes or types for locating contacts on a surface, e.g. floor mats or touch pads
    • A63F13/2145—Input arrangements for video game devices characterised by their sensors, purposes or types for locating contacts on a surface, e.g. floor mats or touch pads the surface being also a display device, e.g. touch screens
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/80—Special adaptations for executing a specific game genre or game mode
    • A63F13/825—Fostering virtual characters

Definitions

  • Provisional application 61774591 filed 2013-03-08
  • Provisional application 61670154 filed 2012-07-11
  • the present invention comprises an apparatus for a virtual companion, a method for controlling one or more virtual companions, and a system for a plurality of humans to control a plurality of avatars.
  • Figure 1 Example of a frontend virtual companion user interface in 2D, showing the virtual companion as a pet, the background, and virtual food being fed to the pet, e.g. using touch-drag.
  • Figure 2 Example of the user interface with a drink being fed to the virtual companion.
  • Figure 3 An exemplary embodiment of the virtual companion, with the 3D, realistic appearance of a dog.
  • the dog has raised its paw in response to the user touching the paw.
  • Figure 4 An exemplary embodiment of the virtual companion, with a 3D, semi-realistic, semi- generalized appearance.
  • Figure 5 An exemplary embodiment of the virtual companion, showing embedded presentation of Internet content in a virtual object.
  • Figure 6 Example of the backend user interface login screen.
  • Figure 7 Example of the backend multi-monitor interface, showing the bandwidth-optimized video feeds (eight photographs as shown), audio indicator bars (immediately below each video feed), schedule notices (two-rowed tables immediately below five of the audio indicator bars), alerts (text boxes with the cross icon, replacing three of the schedule notices), session info (text at far top-left), user/pet info (text immediately above each video feed), team notices (text area at far bottom-left), team chat (scrollable text area and textbox at far bottom-right), teammate attention/status indicators (hourglass and hand icons near the right edge with usernames printed below them), and logoff button (far top- right).
  • Figure 8 Example of the backend direct control interface, showing the session info (text at far top-left), video feed (photo of user as shown), look-at indicator (tinted circle between the user's eyes with the word "Look” printed on it faintly), log (two-columned table and textbox immediately to the right of the video feed), user schedule (two-columned table immediately below the log), pet appearance (penguin character), touch indicators (tinted circles on the pet with "Touch” printed on them faintly), pet speech box (text box immediately below the pet), pet actions (the set of 12 buttons to the sides of the pet), pet settings (three checkboxes and six scrollbars on the far left), team notices (text area on the far bottom- left), team chat (scrollable text area and textbox at far bottom-right), a tabbed area to organize miscellaneous information and settings (tab labeled "tab” and blank area below it), and return to multi- view button (button labeled "Monitor AN" at far top-right).
  • FIG. 9 UML (Universal Modeling Language) use case diagram, showing possible users and a limited set of exemplary use cases, divided among the frontend (virtual companion user) and the backend (helper) interfaces.
  • the box marked "Prototype 1" indicates the core functionality that was implemented first to validate the benefits of this invention among the elderly.
  • FIG 10 UML activity diagram, showing the workflow of a single human helper working through the backend interface. It describes procedures the human should follow in the use of the interfaces shown in Figures 4-6.
  • FIG. 11 UML deployment diagram, showing an exemplary way to deploy the frontend/backend system, with a central server system managing communications between multiple tablet devices (frontend for seniors) and multiple computers (backend for helpers).
  • a central server system managing communications between multiple tablet devices (frontend for seniors) and multiple computers (backend for helpers).
  • backend for helpers to reduce audio/video latency, such high fidelity live data streaming is performed via a more direct connection (e.g. TMP protocol) between the frontend and the backend systems.
  • TMP protocol e.g.
  • the frontend of the invention is a virtual companion which forms a personal/emotional connection with its user, and may take the form of a pet. Beyond the intrinsic health benefits of the emotional connection and caretaker relationship with a pet, the personal connection allows an elderly person to have a much more intuitive and enjoyable experience consuming content from the Internet, compared to traditional methods using a desktop, laptop, or even a typical tablet interface. These benefits may be implemented as described below.
  • the visual representation of the companion itself may be either two-dimensional (2D) (as in Figures 1 and 2) or three-dimensional (3D) (as in Figures 3 and 4) to be displayed on a screen such as an LCD, OLED, projection, or plasma display, and may be cartoonish (as in Figures 1 and 2) in appearance, realistic (as in Figure 3), or semi-realistic (as in Figure 4).
  • the form of the virtual companion can be that of a human or humanoid; a real animal, such as a penguin (as in Figures 1 and 2) or dog (as in Figure 3); or it can be an imaginary creature such as a dragon, unicorn, blob, etc.; or even a generalized appearance that blends features of multiple creatures (as in Figure 4, which is mostly dog-like but with the features of a cat and a seal incorporated).
  • the advantages of a generalized appearance are that users have less preconceived notions of what the creature should be capable of and behave like, and thus are less likely to be disappointed, and also users may more easily associate the creature with the ideal pet from their imagination.
  • the representation of the companion may be fixed, randomized, or selectable by the user, either only on initialization of the companion, or at any time. If selectable, the user may be presented with a series of different forms, such as a dog, a cat, and a cute alien, and upon choosing his/her ideal virtual companion, the user may further be presented with the options of customizing its colors, physical size and proportions, or other properties through an interactive interface. Alternatively, the characteristics of the virtual companion may be preconfigured by another user, such as the elderly person's caretaker, family member, or another responsible person. These customizable characteristics may extend into the non-physical behavioral settings for the companion, which will be described later.
  • the companion may be presented initially to the user without any introduction, or it can be hatched from an egg, interactively unwrapped from a gift box or pile of material, or otherwise introduced in an emotionally compelling manner.
  • the companion is a generalized, semi-cartoonish, dog-like pet, without any customization or special introduction.
  • the representation can be implemented as a collection of either 2D or 3D pre-rendered graphics and videos; or it can be (if 2D) a collection of bitmap images corresponding to different body parts, to be animated independently to simulate bodily behaviors; or it can be (if 2D) a collection of vector-based graphics, such as defined by mathematical splines, to be animated through applicable techniques; or it can be (if 3D) one or more items of geometry defined by vertices, edges, and/or faces, with associated material descriptions, possibly including the use of bitmap or vector 2D graphics as textures; or it can be (if 3D) one or more items of vector-based 3D geometry, or even point-cloud based geometry.
  • the virtual companion may also be represented by a physical robot comprising actuators and touch sensors, in which case the physical appearance of the robot may be considered to be the "display" of the virtual companion.
  • the companion is a collection of vertex-based mesh geometries generated using a common 3D modeling and animation package.
  • a bone system is created within the 3D package that associates different vertices on the geometry surface with various bones, similar to how a real animal's skin moves in response to movement of its bones. Additional virtual bones are added within the virtual companion's face to allow emotive control of the facial expression.
  • a variety of basic animations are then created by keyframing the bones.
  • Animations include an idle, or default pose; other fixed poses such as cocking the head left and right, bowing the head, tilting the chin up, raising the left paw, raising the right paw, a sad facial expression, a happy facial expression, and so on; passive, continuous actions such as breathing, blinking, and looking around, and tail wagging; and active actions such as a bark, or the motion of the jaw when talking. All of this information is exported into the common FBX file format.
  • stored sets of actuator positions may serve a similar role as keyframed animations and poses.
  • the textures, mesh and/or skeleton of the virtual companion may be switched to visually reflect growth and/or aging.
  • Other behavior such as the response to multi-touch interaction as described in the following section may also be modified over time.
  • poses may be gradually blended in to the existing skeletal animation, as described in the following section, to reflect growth.
  • a key innovation of this invention is the use of multi-touch input to create dynamically reactive virtual petting behavior.
  • an exemplary embodiment of this invention is software run on a multi-touch tablet such as an iPad or Android tablet
  • the techniques described here may be applied to other input forms. For example, movement and clicking of a computer mouse can be treated as tactile input, with one or more mouse buttons being pressed emulating the equivalent number of fingers touching, or adjacent to, the cursor location.
  • multiple touch sensors on the robot may serve as input.
  • the virtual companion is displayed on a tablet device with an LCD display and multiple simultaneous tactile inputs may be read via an integrated capacitive or resistive touch screen capable of reading multi-touch input.
  • the general principle of this invention is to drive a separate behavior of the virtual companion corresponding to each stimulus, dynamically combining the behaviors to create a fluid reaction to the stimuli.
  • the end result in an exemplary embodiment is a realistic response to petting actions from the user. Further processing of the touch inputs allow the petting behavior to distinguish between gentle strokes or harsh jabbing, for example.
  • the first step involved in the invented technique is touch detection.
  • touches on the screen are polled within the main software loop.
  • Alternative implementation methods may be possible, such as touch detection by triggers or callbacks.
  • the next step is bodily localization.
  • bodily localization the 2D touch coordinates of each touch are allocated to body parts on the virtual companion. If the virtual companion is a simple 2D representation composed of separate, animated 2D body parts, this may be as simple as iterating through each body part and checking whether each touch is within the bounds of the body part, whether a simple bounding box method is used, or other techniques for checking non-square bounds.
  • the 3D model includes skeletal information in the form of bone geometries. Bounding volumes are created relative to the positions of these bones.
  • a 3D capsule (cylinder with hemispherical ends) volume may be defined in software, with its location and rotation set relative to a lower leg bone, with a certain capsule length and diameter such that the entire lower leg is enclosed by the volume.
  • the virtual companion moves its leg (e.g. the bone moves, with the visible "mesh" geometry moving along with it)
  • the bounding volume will move with it, maintaining a good approximation of the desired bounds of the lower leg.
  • Different body parts may use different bounding volume geometries, depending on the underlying mesh geometry.
  • a short and stubby body part/bone may simply have a spherical bounding volume.
  • the bounding volume may even be defined to be the same as the mesh geometry; this is, however, very computationally costly due to the relative complexity of mesh geometry.
  • a bounding volume may be created for all bones, or only those bones that represent distinct body parts that may exhibit different responses to being touched.
  • each touch detected during touch detection is associated with one or more bounding volumes based on the 2D location of the touch on the screen. In an exemplary embodiment, this is performed through raycasting into the 3D scene and allocating each touch to the single bounding volume that it intercepts first. Thus, each touch is allocated to the bodily location on the virtual companion that the user intended to touch, accounting for skeletal deformations and even reorientation of the virtual companion in the 3D environment. By creating bounding volumes for other objects in the 3D environment, interactivity with other objects and occlusions of the virtual companion may be accounted for.
  • Each touch may now be associated with the part of the body that it is affecting, along with its touch status, which may be categorized as “just touched” (if the touch was not present in the previous frame, for example), “just released” (if this is the first frame in which a previous touch no longer exists, for example), or “continuing” (if the touch existed previously, for example). If the touch is continuing, its movement since the last frame is also recorded, whether in 2D screen coordinates (e.g. X & Y or angle & distance) or relative to the 3D body part touched. Once this information has been obtained for each touch, the information is buffered.
  • 2D screen coordinates e.g. X & Y or angle & distance
  • touch buffering the touch information is accumulated over time in a way that allows for more complex discernment of the nature of the touch. This is done by calculating abstracted, "touch buffer" variables representing various higher-level stimuli originating from one or more instances of lower-level touch stimulus. Touch buffers may be stored separately for each part of the body (each defined bounding volume), retaining a measure of the effects of touches on each part of the body that is persistent over time.
  • these abstracted, qualitative variables are constancy, staccato, and movement. Constancy starts at zero and is incremented in the main program loop for each touch occurring at the buffered body part during that loop. It is decremented each program loop such as with no touch inputs, constancy will return naturally to zero.
  • constancy represents how long a touch interaction has been continuously affecting a body part.
  • constancy can be scaled to represent roughly the number of seconds that a user has been continuously touching a body part.
  • Staccato starts at zero and is incremented during every program loop for each "just touched" touch occurring at the buffering part. It is decremented by some fractional amount each program loop.
  • there is some average frequency above which tapping (repeatedly touching and releasing) a body part will cause the staccato value to increase over time. Staccato thus measures the extent to which the user is tapping a body part as opposed to steadily touching it. It should be limited to values between zero and some upper bound.
  • Movement may be calculated separately for each movement coordinate measured for each touch, or as a single magnitude value for each touch. Either way, it is calculated by starting from zero and incrementing during each program loop, for each touch, by the amount that that touch moved since the last loop.
  • the movement values are buffered for both X and Y movement in 2D, screen coordinates, for each body part. Movement can either be decremented during each loop, and/or can be limited by some value derived from the constancy value of the same body part.
  • movement in each of X and Y is limited to +/- a multiple of the current value of constancy in each loop.
  • movement describes how the user is stroking the body part.
  • constancy, staccato, and movement provide an exemplary way of describing the organic nature of any set of touches on the body of a virtual companion.
  • the increment/decrement process may be exponential or of some other higher order in time.
  • the increments may be decreased as the actual current values of constancy, staccato, and movement increase, such that instead of a hard upper limit on their values, they gradually become more and more difficult to increase.
  • the effects of multiple simultaneous touches on a single body part can be ignored, so that, for example, in the event of two fingers being placed on a body part, only the first touch contributes to the touch buffer. Random noise can be introduced either into the rate of increment/decrement or into the actual buffer values themselves.
  • Animation blending refers to a number of established techniques for combining multiple animations of a 3D character into a single set of motions. For example, an animation of a virtual companion's head tilting down may consist of absolute location/orientation coordinates for the neck bones, specified at various points in time. Another animation of the virtual companion's head tilting to the right would consist of different positions of the neck bones over time. Blending these two animations could be accomplished by averaging the position values of the two animations, resulting in a blended animation of the virtual companion tilting its head both down and to the right, but with the magnitude of each right/down component reduced by averaging.
  • the movements of each animation may be specified not as absolute positions, but rather as differential offsets. Then, the animations may be blended by summing the offsets of both animations and applying the resulting offset to a base pose, resulting in an overall movement that is larger in magnitude compared to the former blending technique. Either of these blending techniques can be weighted, such that each animation to be blended is assigned a blend weight which scales the influence of that animation.
  • An innovation of this invention is a method for applying multi-touch input to these animation blending techniques.
  • a number of poses are created for the virtual companion prior to compilation of the software. These poses consist of skeletal animation data with two keyframes each, with the first keyframe being the idle pose and the second keyframe being the nominal pose - the difference between the two frames forms an offset that can be applied in additive animation blending (alternatively, a single frame would suffice if blending with averaged absolute positions will be used).
  • Each of these nominal poses corresponds to the virtual companion's desired steady-state response to a constant touch input.
  • a nominal pose may be created with the virtual companion pet's front-left paw raised up, and in software this pose would be associated with a constant touch of the front-left paw.
  • Another pose might be created with the pet's head tilted to one side, and this could be associated with one of the pet's cheeks (either the pet recoils from the touch or is attracted to it, depending on whether the cheek is opposite to the direction of the tilt motion). These poses may be classified as constancy-based poses.
  • Another set of poses may be created to reflect the pet's response to high levels of staccato in various body parts. For example, a pose may be created with the pet's head reared back, associated with staccato of the pet's nose. Similarly, movement-based poses may be created.
  • Staccato-based poses may also be incorporated to increase apparent emotional realism. For example, a pose in which the pet has withdrawn its paw can be created.
  • the blend weight for this animation could be proportional to the staccato of the paw, thus creating an effect where "violent" tapping of the paw will cause it to withdraw, while normal touch interaction resulting in high constancy and low staccato may trigger the constancy-based pose of raising the paw, as if the user's finger was holding or lifting it.
  • a sad pose can be created by positioning auxiliary facial bones to create a sad expression.
  • the blend weight of this pose can be proportional to the total staccato of the pet, thus creating a realistic effect whereby the pet dislikes being tapped or prodded. Exceptions to this behavior can be created by accounting for staccato at particular locations. For example, the pet may enjoy being patted on the top of the head, in which case staccato at this location could trigger a happier pose and would not be included in total staccato.
  • the pet may enjoy other particular touch techniques such as stroking the area below the pet's jaw.
  • a movement-based happy pose may be implemented, weighted by movement in the desired area.
  • Very realistic responses to petting can be created using these techniques, and the user may enjoy discovering through experimentation the touch styles that their pet likes the most.
  • a pose may be weighted by a combination of constancy, staccato, and/or movement.
  • the response to touch may be randomized to create a less predictable, more natural behavior.
  • the animations associated with various body parts may be switched with different animations at random over the course of time, or multiple animations associated with the same body part can have their relative weights gradually adjusted based on an underlying random process, or perhaps based on the time of day or other programmed emotional state.
  • Procedural components can be added.
  • bone positions can be dynamically adjusted in real time so that the pet's paw follows the position of the user's finger on the screen, or a humanoid virtual companion shakes the user's finger/hand.
  • multi-keyframed animations can be weighted similarly.
  • the head may be animated to oscillate back and forth, and this animation may be associated with constancy of a virtual companion pet's head, as if the pet likes to rub against the user's finger.
  • Special limitations may be coded into the blending process to prevent unrealistic behaviors. For example, a condition for blending the lifting of one paw off the ground may be that the other paw is still touching the ground.
  • Procedural limits to motion may be implemented to prevent the additive animation blending from creating a summed pose in which the mesh deformation becomes unrealistic or otherwise undesirable.
  • Accelerometer data may be incorporated so that the orientation of the physical tablet device can affect blending of a pose that reflects the tilt of gravity.
  • camera data may be incorporated through gestural analysis, for example.
  • audio volume from a microphone could be used to increase staccato of a pet's ears for example, if it is desired that loud sounds have the same behavioral effects as repeated poking of the pet's ears.
  • animations may be blended into the virtual companion's skeleton prior to rendering during each program loop.
  • animations for passive actions such as blinking, breathing, or tail wagging can be created and blended into the overall animation.
  • active actions taken by the virtual companion such as barking, jumping, or talking may be animated and blended into the overall animation.
  • the above techniques including abstraction of touch data and blending animations based on multiple stimuli may be applied to the robot's touch sensor data and actuator positions.
  • this emotional model is based on the Pleasure- Arousal-Dominance (PAD) emotional state model, developed by Albert Mehrabian and James A. ussel to describe and measure emotional states. It uses three numerical dimensions to represent all emotions. Previous work such as that by Becker and Christian et al, have applied the PAD model to virtual emotional characters through facial expressions.
  • PAD Pleasure- Arousal-Dominance
  • the values for long-term PAD and short-term PAD are kept track of in the main program loop.
  • the long-term PAD values are representative of the virtual companion's overall personality, while the short-term PAD values are representative of its current state. They are initialized to values that may be neutral, neutral with some randomness, chosen by the user, or chosen by another responsible party who decides what would be best for the user. Because the short- term values are allowed to deviate from the long-term values, with each passing program loop or fixed timer cycle, the short-term PAD values regress toward the long-term PAD values, whether linearly or as a more complex function of their displacement from the long-term values, such as with a rate proportional to the square of the displacement.
  • the long-term PAD values may also regress toward the short-term values, but to a lesser extent, allowing long-term personality change due to exposure to emotional stimulus. Besides this constant regression, external factors, primarily caused by interaction with the human user, cause the short-term PAD values to fluctuate. Building upon the aforementioned descriptions of multi-touch sensitivity and animated response, examples of possible stimuli that would change the short-term PAD values are as follows:
  • any kind of constancy, staccato, or movement may increase arousal and decrease dominance, to an extent depending on the type and location of the touch.
  • a temporary, time-dependent effect may be superimposed onto long-term PAD (thus causing short-term PAD to regress to the altered values). These effects may reflect a decrease in arousal in the evenings and/or early mornings, for example.
  • the tone of voice may also alter short-term PAD values. For example, if the user speaks harshly or in a commanding tone of voice, pleasure and/or dominance may be decreased. Analysis of the user's breathing speed or other affective cues may be used to adjust the virtual companion's arousal to fit the user's level of arousal.
  • the values of short-term PAD may directly affect the behavior of the virtual companion as follows:
  • Pleasure above or below certain thresholds may affect the weights of facial animation poses during animation blending in the main program loop, such that the level of pleasure or displeasure is revealed directly in the virtual companion's facial expression.
  • arousal and dominance and certain combinations of pleasure, arousal, and dominance within specific ranges may override the aforementioned blending and cue the blending of specific emotions. For example, an angry expression may be blended in when pleasure is quite low, arousal is quite high, and dominance is moderately high.
  • Arousal may affect the speed of a breathing animation and/or related audio, scaling speed up with increased arousal.
  • the magnitude of the breathing animation and/or related audio may also be scaled up with increased arousal. Similar scaling may apply to a tail wag animation or any other animal behavior that the user may expect to increase with arousal.
  • Pleasure, arousal, and/or dominance may increase the blend weight of poses that reflect these respective emotional components.
  • the value of pleasure, arousal, and/or dominance may proportionally scale the blend weight of a pose in which a dog-like virtual companion has its tail erect and pointing upwards or over its body, while the idle pose may have the tail lowered or tucked below the pet's body.
  • Aging of the virtual companion may directly affect the long-term PAD values. For example, arousal may gradually reduce over the course of several years. Long-term PAD values may conversely affect aging. For example, virtual companion with high values of pleasure may age slower or gradually develop more pleasant appearances that aren't normally available to reflect short-term PAD values, such as changes in fur color.
  • the virtual companion may have bodily needs which increase over time, such as hunger (need for food), thirst (need for fluids), need to excrete waste, need for a bath, need for play, etc. Even the need to sleep, blink or take a breath can be included in this model rather than simply occurring over a loop or timer cycle.
  • These needs may be tracked as numerical variables (e.g. floating point) in the main program loop or by a fixed recurring timer that increments the needs as time passes.
  • the rate of increase of these needs may be affected by time of day or the value of short-term arousal, for example.
  • a blend weight for an associated animation pose may directly be visible to the user by proportionally scaling a blend weight for an associated animation pose. For example, need for sleep may scale the blend weight for a pose with droopy eyelids. Alternatively, it may impose an effect on short-term arousal or directly on the blend weights that short-term arousal already affects.
  • Each need may have a variable threshold that depends on factors such as time of day, the value of the current short-term PAD states, or a randomized component that periodically changes.
  • the virtual companion acts on the need. For very simple needs such as blinking, it may simply blink one or more times, reducing the need value with each blink, or for breathing, it may simply take the next breath and reset the need to breathe counter. Sleeping may also be performed autonomously by transitioning into another state a la a state machine architecture implemented in the main program loop; this state would animate the virtual companion into a sleeping state, with the ability to be woken up by sound, touch, or light back into the default state.
  • More complex needs are, in an exemplary embodiment, designed to require user interaction to fulfill, such that the user can form a caretaker type of relationship with the virtual companion, similar to the relationship between gardeners and their plants or pet owners and their pets, which has been shown to have health effects.
  • the virtual companion may indicate this need for user interaction by blending in a pose or movement animation that signifies which need must be satisfied. For example, need for play may be signified by a jumping up and down on the forepaws. Audio cues may be included, such as a stomach growl indicating need for food.
  • the need for food may be indicated by a randomly repeating auditory stomach growl, a blended pose indicating displeasure/hunger, and/or a container that appears, or if always present in the scene, begins to glow or partially open.
  • the user Upon touching the container, the user causes the container to open and a number of food items to slide out from the container. As shown in Figure 1, the user may then drag any of the food items to the pet to feed it, triggering the appropriate feeding animations, possibly in a different state a la a state machine architecture implemented in the program loop.
  • the pet's hunger counter is thus decremented, possibly by an amount dependent on the food item chosen.
  • the food item chosen may also have a small effect on the pet's long-term PAD state; for example, meat items may increase arousal while vegetables decrease arousal.
  • the food items chosen may also contribute to how the pet grows and ages. For example, meat may make for a more muscular body.
  • the need for fluids may be indicated by a randomly repeating auditory raspy breath, a blended pose indicating displeasure/thirst, and/or a container that appears or otherwise attracts attention.
  • the user can choose from a selection of drinks, and feed them to the pet similar to the method described above for food, as shown in Figure 2.
  • Similar effects on PAD states and growth may be applied. For example, unhealthy drinks may cause the pet to become fatter over time and decrease long-term aroused, though sugar-laden drinks may cause a temporary increase in short-term arousal.
  • the need to excrete waste may be relieved by the pet itself by creating a mound of excrement on the ground, if the pet is at a young age.
  • the pet may have lower levels of pleasure and may motion as if there is a bad smell while the excrement is present.
  • the user may remove the excrement and its effects by swiping it off the screen with a finger, or by dragging over it with a sweeper tool that may appear within the 3D environment.
  • a bath can be indicated by repeated scratching as if the pet is very itchy, discolorations or stains textured onto the 3D mesh, animated fumes coming off of the pet, and/or a bathtub that slides into view.
  • a bath may be administered by dragging the pet into the tub.
  • the process may also be gamified somewhat by having the user wipe off stained or discolored areas by touch, with successful completion of the cleaning dependent on actually removing all the dirt.
  • the need for play may be indicated by an increase in arousal and its related effects, auditory cues such as barking, animations such as excited jumping, and/or having the pet pick up a toy or game.
  • Any number of games can be played within the 3D environment, each likely under a new state a la a state machine architecture implemented in the main program loop.
  • Novel games involving the multi-touch interactive behavior detailed in this invention may be included.
  • the goal of one game may be to remove a ball from the pet's mouth, necessitating the use of multi-touch (and likely multi-hand) gestures, during which the pet responds fluidly and realistically.
  • Playing games and the results of the games may greatly increase the pet's short- term pleasure and arousal, for example, and may even directly affect long-term PAD values to an extent greater than that possible through a one-time increment in short-term PAD values due to limits on the maximum value.
  • This invention includes techniques for incorporating conversational intelligence into the virtual companion.
  • all of the conversational intelligence could be generated through artificial means, but in an exemplary embodiment, some or all of the conversational intelligence is provided directly by humans, such that the virtual companion serves as an avatar for the human helpers.
  • the reason for this is that as of present, artificial intelligence technology is not advanced enough to carry on arbitrary verbal conversations in a way that is consistently similar to how an intelligent human would converse.
  • This invention describes methods for integrating human intelligence with the lower-level behavior of the virtual companion.
  • Helpers who may be remotely located, for example in the Philippines for India, contribute their intelligence to the virtual companion through a separate software interface, connected to the tablet on which the virtual companion runs through a local network or Internet connection.
  • helpers log in to the helper software platform through a login screen such as that shown in Figure 6, after which they use an interface such as that depicted in Figure 7 to oversee a multitude of virtual companions, possibly in cooperation with a multitude of co-workers.
  • a helper will either manually or automatically switch into an "detailed view" interface such as that shown in Figure 8 to directly control a specific virtual companion, thus
  • Figure 10 shows through an activity diagram the workflow of one of these human helpers, while Figure 11 shows through a deployment diagram how this system could be implemented.
  • an alert may show (e.g. similar to the alerts in Figure 7) indicating this, or one of the available helpers using the monitor-all interface may be automatically transferred to the direct control interface in Figure 8. It is also possible for a helper in the monitor-all view to manually decide to intervene in a specific virtual companion's conversation through a combination of noticing that the user is talking to the virtual companion through the video feed and through the audio indicator bar, which shows the volume level detected by the microphone in the tablet on which the virtual companion runs.
  • voice recognition software may be used to display the actual conversational history of each virtual companion/user pair in the monitor-all view. Whenever there is a manual intervention, the recent history of the virtual companion and its detected inputs can be used to train a supervised machine learning algorithm as to when to automatically alert a helper that human intelligence is needed.
  • the helper listens to an audio stream from the virtual companion's microphone, thus hearing any speech from the user, and whatever the helper types into the virtual companion speech box is transmitted to the virtual companion to be spoken using text-to- speech technology.
  • the virtual companion may simultaneously move its mouth while the text-to-speech engine is producing speech. This could be as simple as blending in a looping jaw animation while the speech engine runs, which could be played at a randomized speed and/or magnitude each loop to simulate variability in speech patterns.
  • the speech engine may also generate lip-sync cues or the audio generated by the speech engine may be analyzed to generate these cues to allow the virtual companion's mouth the move in synchrony with the speech. Captions may also be printed on the tablet's screen for users who are hard of hearing.
  • the helper may be trained to transmit the first word or phrase of the sentence before typing the rest of the sentence, so that the virtual companion's verbal response may be hastened, or alternatively there may be a built-in functionality of the software to automatically transmit the first word (e.g. upon pressing the space key after a valid typed word) to the virtual companion's text-to- speech engine.
  • the results of speech recognition fed to an artificially intelligent conversational engine may also be automatically entered into the virtual companion speech box, so that if the artificial response is appropriate, the helper may simply submit the response to be spoken.
  • the conversation engine may also present multiple options for responses so that the helper can simply press a key to select or mouse click the most appropriate response.
  • the helper may also be assisted by statistically probable words, phrases, or entire sentences that populate the virtual companion speech box based on the existing typed text, similar to many contemporary "autocomplete" style typing systems.
  • clicking the user's name in the relationship management system could insert it as text without having to type, or aliases may be used, such as typing "/owner" to insert the name of the virtual companion's owner, as recorded by the relationship management system.
  • This data may also be fed directly into any autocomplete or menu-based systems as described previously.
  • the conversational responses may also be generated by an expert system, or an artificial intelligence that embodies the domain knowledge of human experts such as psychiatrists, geriatricians, nurses, or social workers.
  • an expert system may be pre-programmed to know the optimal conversational responses (with respect to friendly conversation, a therapy session for depression, a reminiscence therapy session to treat dementia, etc) to a multitude of specific conversational inputs, possibly with a branching type of response structure that depends on previous conversation inputs.
  • a limitation of such a system may be that the expert system has difficulty using voice recognition to identify what specific class of conversational input is meant by a user speaking to the system.
  • the system may ask “How are you doing?" and know how to best respond based one which one of three classes of responses is provided by the user: "Well”, “Not well”, or “So-so". But the system may have difficulty determining how to respond to "Well, I dunno, I suppose alright or something like that.”
  • a human helper may listen to the audio stream (or speech-recognized text) from the user, and use their human social and linguistic understanding to interpret the response and select which one of the expert system's understood responses most closely fit the actual response of the user (in this example, the helper would probably pick "So-so").
  • the human helper may skip to other points in the expert system's preprogrammed conversational tree, change dynamic parameters of the expert system, and/or completely override the expert system's response with menu-driven, autocomplete-augmented, or completely custom-typed responses to maintain the ability to respond spontaneously to any situation.
  • the expert system takes continuous variables, such as a happiness scale or a pain scale, into account when generating responses, the helper may also select the level of such continuous variables, for example using a slider bar based on the visual interpretation of the user's face via the video feed.
  • the variables could also be the same variables used to represent the virtual companion's emotional scores, such as pleasure, arousal, and dominance, which may affect the conversational responses generated by the expert system.
  • the helper presses the "Alt” and “Enter” keys at the same time to submit the text to the virtual companion, while a simple "Enter” submits any typed text in the team chat area. This is to prevent losing track of which text box the cursor is active in, and accidentally sending text intended for the helper's team/co-workers to the virtual companion/user.
  • the voice of the virtual companion may be designed to be cute-sounding and rather slow to make it easier to understand for the hard of hearing.
  • the speed, pitch, and other qualities of the voice may be adjusted based on PAD states, the physical representation and/or age of the virtual companion, or even manually by the helper.
  • the tone of voice and inflections may be adjusted manually through annotations in the helper's typed text, and/or automatically through the emotional and other behavioral scores of the virtual companion. For example, higher arousal can increase the speed, volume, and/or pitch of the text-to-speech engine, and may cause questions to tend to be inflected upwards.
  • PAD personality settings combined with a log of recent events and a user schedule of upcoming events provides a helper with cues as to how to react conversationally as well as what things are appropriate to discuss.
  • a helper would maintain a relatively pleasant, submissive attitude and may ask about Betty's friend Bob, or what Betty wants for lunch, which is coming up soon.
  • Alternative implementations of the human contribution to the conversation may involve voice recognition of the helper's spoken responses rather than typing, or direct manipulation of the audio from the helper's voice to conform it to the desired voice of the virtual companion, such that different helpers sound approximately alike when speaking through the same virtual companion.
  • a human helper may directly control bodily needs, PAD states, toggle behaviors such as automatic display of PAD states through facial expressions or automatic animations such as blinking, trigger special animations such as dances or expressive faces, or even record custom animations by dragging around the virtual companion's body parts and then play them back.
  • Tactile cues as to the appropriate response may be provided to the helper, as shown in Figure 8, by touch indicators on the virtual companion appearance area, which display the locations of recent touch events as they are transmitted through the network and displayed in real-time on a live rendering of the virtual companion in the state that the user sees it.
  • a further method of expression for the helper through the pet may be the virtual companion's eyes.
  • a "Look” indicator may be placed on the video stream as shown in Figure 8.
  • the helper may click and drag the indicator to any location on the video stream, and a command will be sent to the tablet to turn the virtual companion's eyes to appear to look in the direction of the look indicator.
  • One of the important practical features of this invention is its ability to facilitate increased efficiency in task allocation among the staff of senior care facilities and home care agencies. With a network of intelligent humans monitoring a large number of users through the audio-visual capabilities of the tablets, the local staff can be freed to perform tasks actually requiring physical presence, beyond simple monitoring and conversation.
  • a human helper monitors a set of virtual companions with the cooperation of other helpers, under the supervision of a supervisor.
  • the set of virtual companions may include all the virtual companions deployed in one specific assisted living facility, and the helpers and perhaps the supervisor may be specifically allocated to that facility.
  • the virtual companions may be drawn arbitrarily from a larger set of virtual companions deployed all over the world, based on similarity of the virtual companions, their users, and/or other contextual information. In this case, it may be advantageous to overlap the responsibilities of one or more helpers with virtual companions from other sets, such that no two helpers in the world have the exact same allocation of virtual companion.
  • Figure 7 only shows eight virtual companions, more or less may be displayed at a time and either made to fit within one screen or displayed in a scrollable manner, depending on a helper's ability to reliably monitor all the virtual companions and their corresponding video feeds.
  • the video and other informational feeds may be abstracted into a symbolic status display with simplified visual indications/alerts of touch, motion, and volume, for example, and these status displays may be further augmented by "audio displays" in the form of audio cues or notifications, such as certain sounds that play in response to detection of touch, motion, and volume thresholds by one or more virtual companions.
  • Another factor in determining the number of virtual companions to allocate to a single helper and also the number of helpers who are allocated redundantly to the same virtual companions is the typical frequency and duration of need for focused human intelligence (e.g. use of the direct control interface for a specific virtual companion, as shown in Figure 8).
  • a software system may be implemented which automatically assigns additional helpers to monitor virtual companions which are more frequently directly controlled than average, offloading those helpers from other virtual companions which don't use human intelligence as often. If artificial intelligence is used to automatically call in human intelligence, as described previously, assignment of additional helpers may be based on abnormally long average wait times from the request for human intelligence to the response of an actual human helper, indicating situations in which all helpers assigned to the virtual companion were busy directly controlling other virtual companion. The amount of time spent by all helpers directly controlling a virtual companion and/or the number of helpers assigned to monitor it may be logged and used for customer billing purposes.
  • Time logging may be based on when a dashboard is open, when the audio/video is being streamed, when there is actual keyboard/mouse activity within the dashboard, manual timestamping, or a combination of these techniques.
  • helpers There may be multiple classes of helpers, for example paid helpers, supervisory helpers, volunteer helpers, or even family members acting as helpers.
  • a useful feature for helpers monitoring the same virtual companions may be teammate attention/status indicators, as shown in Figure 7.
  • the indicators would be positioned on a helper's screen in real-time to reflect the screen position of the mouse cursor for each of the helper's co-workers, and the visual representation of an indicator may reflect the status of the corresponding helper; for example, an hourglass may indicate a busy or away status, while a pointing hand may indicate that the helper is actively attentive and working.
  • helper enters the direct control interface for a specific virtual companion, that helper's indicator may disappear to his co-workers, to be replaced by a label underneath the video stream of the controlled virtual companion, indicating that the virtual companion is being controlled by the helper (as shown by "BillHelper9" in Figure 7).
  • this method may allow helpers to avoid wasting visual attention on something that is already being watched, thereby maximizing the attentive resources of the team while still allowing for redundancy in terms of multiple helpers monitoring the same virtual companions.
  • the number of virtual companions assigned to a single helper and the extent to which any virtual companions receive redundant monitoring from multiple helpers can then be adjusted to achieve the overall maximum ratio of number of virtual companions to number of helpers, while maintaining an adequate response time to events requiring direct control.
  • Another useful feature for the supervisor system could be to dynamically match virtual companions with helpers, which guarantees that each virtual companion is monitored by at least one helper at any time when the virtual companion is connected to the server, and monitored by several helpers when the virtual companion is in its 'active period'.
  • This matching procedure may include two phases:
  • helpers with non-overlapping time shifts are assigned to this virtual companion. Each helper will monitor/control the virtual companion during their shift.
  • the server keeps a record of each interaction between the user and the virtual companion. Each record entry includes time, helper ID, and helper's grade on the interaction with the user through the virtual companion (higher grade for happier reactions from the user, possibly self-scored or scored by supervisors and/or users/customers).
  • the server summarizes the active periods (times of the day when the user interacts with the virtual companion), and ranks the helpers based on average grades. This summary may also be done from the very beginning, based on assumed default values, such that there is effectively no learning phase, and learning happens continuously during the matching phase.
  • the system assigns a minimum number (e.g. one) helper to this virtual companion, based on who 1) is currently matched with least number of virtual companions; and/or 2) spent the least amount of time to interacting with any virtual companion during the last hour; and/or 3) got the highest score during the learning phase of this particular virtual companion.
  • a minimum number e.g. one
  • helper may be combined in a weighted average manner and the system assigns the helper with highest combined score.
  • the system first assigns the helper to this virtual companion who 1) has longest accumulative interaction time with; and/or 2) got the highest averaged scores when interacting with this particular virtual companion both in the learning phase and the matching phase.
  • the system also assigns several other helpers to this virtual companion, who 1) is currently matched with the least number of virtual companions; and/or 2) spent the least time interacting with any virtual companion during the last hour; and/or 3) has longest cumulative interaction time with the virtual companion; and/or 4) got the highest averaged scores when interacting with this particular virtual companion both in the learning phase and the matching phase.
  • the helpers also grade or are assigned a grade for their interaction with the user, so that the average scores keep updated.
  • Grading/scoring of the interaction quality may also be performed by automatic voice tone detection of the user, with more aroused, pleasurable tones indicating higher quality of interaction; it could also use other sensors such as skin conductance, visual detection of skin flushing or pupil dilation, etc. It may also depend on the subjective qualities of touches on the screen as the user touches the virtual companion.
  • the virtual companion may indicate to the user when a human helper is viewing a high fidelity/resolution version of the video/audio stream through the virtual companion's onboard camera/microphone. This may be achieved by having the virtual companion indicate in a natural and unobtrusive way that it is being controlled by a helper through the direct control interface, for example, by having a collar on the virtual companion pet's neck light up, changing the color of the virtual companion eyes, or having the virtual companion open its eyes wider than usual.
  • the sleeping or waking status of the virtual companion corresponds to the streaming status of the audio and video. When the audio/video is streaming, the virtual companion is awake, and when the audio/video is not streaming, the virtual companion is asleep.
  • one of the functionalities may be to contact a third party, whether in the event of an emergency or just for some routine physical assistance.
  • the third party may be a nurse working in the senior care facility in which the virtual companion and user reside, or a family member, for example.
  • the contact's information would be stored along with the virtual companion's database containing the schedule, log, and other settings.
  • the contact information for one or more contacts may be listed in the tabbed area.
  • a click of the helper's mouse on a phone number or email address may activate an automatic dialer to a phone number or open an email application, for example.
  • Another useful feature for the supervisory system may be a remote controllable troubleshooting mechanism.
  • One purpose of such a system would be to facilitate operation of the virtual companion for an indefinite period of time.
  • the virtual companion application When connected to a networked system, the virtual companion application periodically may send status summary messages to a server. Helpers who are assigned to this virtual companion are able to receive the messages in real time. Also, the helpers can send a command to the virtual companion through the internet to get more information, such as screenshots. Or the helpers can send commands for the virtual companion software to execute, for instance, "restart the application", "change the volume", and "reboot the tablet”. This command exchange mechanism can be used when the virtual companion is malfunctioning, or daily maintenance is needed.
  • a simplistic, highly reliable "wrapper" program may control the main run-time program which contains more sophisticated and failure-prone software (e.g. the visual representation of the virtual companion, generated by a game engine).
  • the wrapper program may close and restart or perform other troubleshooting tasks on the main run-time program.
  • the wrapper program may be polled periodically by the main run-time program and/or operating system to send/receive information/commands.
  • the virtual companion may be capable of other features that enrich its user's life.
  • a method for delivering news, weather, or other text-based content from the Internet may involve a speech recognition system and artificial intelligence and/or human intelligence recognizing the user's desire for such content, perhaps involving a specific request, such as "news about the election” or "weather in Tokyo.”
  • the virtual companion would then be animated to retrieve or produce a newspaper or other document.
  • it would search for the desired content, for example through SS feeds or web scraping. It would then speak the content using its text-to-speech engine, along with an appropriate animation of reading the content from the document.
  • the virtual companion may be provided with information about the user's family's social media accounts, and may periodically mention, for example, "Hey did you hear your son's post on the Internet?" followed by a text-to-speech rendition of the son's latest Twitter post.
  • a method for delivering image and graphical content from the Internet may be similar to the above, with the virtual companion showing a picture frame or picture book, with images downloaded live according the user's desired search terms (as in Figure 5). Images may also be downloaded from an online repository where the user's family can upload family photos.
  • Similar techniques may be applied to music or other audio (played through a virtual radio or phonograph, for example), or even video, which can be streamed from, for example, the first matching YouTube search result.
  • videoconferencing with family members can be initialized by the user merely by speaking, and integrated seamlessly by the virtual companion as it produces a photo frame, television, or other kind of screen from its 3D environment, on which the family's video stream is displayed.
  • the relevant video conferencing contact information would already be included in the contacts information as described earlier.
  • the virtual companion may synchronize breathing with the user. Then, breathing rate may be gradually slowed to calm the user. This may have applications to aggressive dementia patients and/or autistic, aggressive, or anxious children.
  • Additional objects may be used to interact with the virtual companion through principles akin to augmented reality. For example, we have empirically found that people appreciate having shared experiences with their virtual companion pet, such as taking naps together. We can offer increased engagement and adherence to medication prescriptions by creating a shared experience around the act of taking medication. In one embodiment of this shared experience, a person may hold up their medication, such as a pill, to the camera. Once the pill has been identified by machine vision and/or human assistance, and it is confirmed that the person should be taking that pill at that point in time, a piece of food may appear in the pet's virtual environment. The food may resemble the pill, or may be some other food item, such as a bone.
  • a similar technique can be used to cause the pet to eat the virtual food, and display feelings of happiness.
  • the person may thus be conditioned to associate adherence to a prescribed plan of medication with taking care of the pet, and experience a sense of personal responsibility and also positive emotions as expressed by the pet upon eating.
  • Alternative methods of interacting with the pet and its virtual environment may involve showing the pet a card with a special symbol or picture on it.
  • the tablet's camera would detect the card, and result in an associated object appearing in the virtual environment. Moving the card in the physical word could even move the virtual object in the virtual world, allowing a new way to interact with the pet.
  • Some tablet devices are equipped with near-field or RFID communications systems, in which case special near-field communications tags may be tapped against the tablet to create objects in the virtual environment or otherwise interact with the pet.
  • the tablet may be attached to or propped up against a structure, which we shall call here a "collection stand," that contains a receptacle for such near-field communications tags.
  • the collection stand would be built in such a way that it is easy to drop a tag into it, and tags dropped into the stand would be caused to fall or slide past the near-field communications sensor built into the tablet, causing the tablet to read the tag.
  • an associated virtual item may be made to drop into the virtual world, giving the impression that the tag has actually dropped into the virtual world, as a virtual object.
  • a similar setup may be constructed without the use of near-field communications, to allow dropping visual, symbolic cards into the collection stand; the collection stand would ensure that such cards are detected and recognized by a rear-facing camera in this case.
  • An alternative implementation may involve a web-based demonstration of the virtual companion, for which it is desirable to limit use of valuable staff time for any individual user trying the demo, and for which no previous relationships exist. For example, a user who is not previously registered in the system may click a button in a web browser to wait in a queue for when one of a number of designated helpers becomes available. Upon availability, the virtual companion could wake up and begin to talk with the user through the speaker/microphone on the user's computer, with touch simulated by mouse movement and clicks. A timer could limit the interaction of the user with the system, or the helper could be instructed to limit the interaction. Once the interaction is over, the helper may be freed to wake up the next virtual companion that a user has queued for a demo.
  • Another aspect of the system could be considered the hiring and training process for the human helpers that provide the conversational intelligence.
  • This process may be automated by, for example, having applicants use a version of the Helper Dashboard that is subjected to simulated or pre-recorded audio/video streams and/or touch or other events. Responses, whether keystrokes or mouse actions, may be recorded and judged for effectiveness.
  • Paro http://www.parorobots.com
  • Paro is a physical, therapeutic robot for the elderly.
  • its custom-built hardware results in a large upfront cost, making it too expensive for widespread adoption.
  • its physical body is very limited in range of motion and expressive ability, and it is generally limited in terms of features.
  • Virtual companions and assistants that provide verbal feedback are either limited to repeating the words of its user (e.g. Talking Tom) or handicapped by limited artificial intelligence and voice recognition (e.g. US Patent 6,722,989, US Patent Application 2006/0074831, and Siri: US Patent Application 2012/0016678).
  • This invention may be used to collect usage data to be fed into a machine learning system for predicting or evaluating functional decline, progress in treatment of dementia, etc. For example, depression and social withdrawal may be correlated with decreased use of the virtual companion over time. This may provide for an accurate and non-intrusive aid to clinicians or therapists.
  • This invention may additionally be used by ordinary, young people. It may be employed for entertainment value or via its human intelligence features, as a virtual assistant for managing schedules or performing Internet-based tasks.
  • the multi-touch reactive behavior of the 3D model may be applied instead to other models besides a virtual companion in the form of a human or animal-like pet. For example, it may be used to create interaction with a virtual flower.
  • This invention may be applied to robotic devices that include mechanical components.
  • attachments may be made to the tablet that allow mobility, panning or rotating of the tablet, or manipulation of the physical environment.
  • Another possible class of attachments comprise external structures which give the impression that the virtual companion resides within or in proximity to another physical object rather than just inside a tablet device.
  • a structure resembling a dog house may be made to partially enclose the tablet so as to physically support the tablet in an upright position while also giving the appearance that a 3D dog in the tablet is actually living inside a physical dog house.
  • Attachments may also be added to the tablet that transfer the capacitive sensing capability of the screen to an external object, which may be flexible.
  • This object may be furry, soft, or otherwise be designed to be pleasurable to touch or even to insert a body part into, such as a finger or other member.
  • the 3D model may be made to react to the application of the fluid. For example, depending on the nature of fluid exposure, of the touch screen hardware and of the software interface with the touch screen, fluids on capacitive touch screens often cause rapidly fluctuating or jittery touch events to be registered across the surface of the touch screen. By detecting these fluctuations, the virtual companion may be made to act in a way appropriate to being exposed to fluid.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Multimedia (AREA)
  • Public Health (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Pathology (AREA)
  • Epidemiology (AREA)
  • Theoretical Computer Science (AREA)
  • Primary Health Care (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • User Interface Of Digital Computer (AREA)
  • Medical Treatment And Welfare Office Work (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Rehabilitation Tools (AREA)

Abstract

Le compagnon virtuel d'après la présente invention est susceptible de répondre de façon réaliste à une entrée tactile et, grâce à l'intervention d'une pluralité de personnes, il peut converser avec une véritable intelligence, quelle que soit la personne avec laquelle il interagit. L'application ayant valeur d'exemple vise à tenir compagnie à des personnes âgées et à améliorer la santé mentale grâce à la compagnie d'autrui.
PCT/IB2014/062986 2013-07-10 2014-07-09 Compagnon virtuel Ceased WO2015004620A2 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201480002468.2A CN104769645A (zh) 2013-07-10 2014-07-09 虚拟伴侣

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13/939,172 US20140125678A1 (en) 2012-07-11 2013-07-10 Virtual Companion
US13/939,172 2013-07-10

Publications (2)

Publication Number Publication Date
WO2015004620A2 true WO2015004620A2 (fr) 2015-01-15
WO2015004620A3 WO2015004620A3 (fr) 2015-05-14

Family

ID=52280769

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2014/062986 Ceased WO2015004620A2 (fr) 2013-07-10 2014-07-09 Compagnon virtuel

Country Status (2)

Country Link
CN (1) CN104769645A (fr)
WO (1) WO2015004620A2 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3529707A4 (fr) * 2016-10-20 2020-08-05 RN Chidakashi Technologies Pvt. Ltd. Dispositif compagnon émotionnellement intelligent
US12112285B1 (en) 2021-07-07 2024-10-08 Wells Fargo Bank, N.A. Systems and methods for measuring employee experience

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104959985B (zh) * 2015-07-16 2017-10-17 深圳狗尾草智能科技有限公司 一种机器人的控制系统及其方法
CN105141587B (zh) * 2015-08-04 2019-01-01 广东小天才科技有限公司 一种虚拟玩偶互动方法及装置
US10664741B2 (en) * 2016-01-14 2020-05-26 Samsung Electronics Co., Ltd. Selecting a behavior of a virtual agent
CN105957140A (zh) * 2016-05-31 2016-09-21 成都九十度工业产品设计有限公司 一种基于增强现实技术的宠物狗交互系统及分析方法
CN106375774B (zh) * 2016-08-31 2019-12-27 广州酷狗计算机科技有限公司 一种直播间显示内容控制的方法、装置和系统
CN106850824A (zh) * 2017-02-22 2017-06-13 北京爱惠家网络有限公司 一种智能服务系统以及实现方法
CN107322593B (zh) * 2017-06-15 2020-07-14 重庆柚瓣家科技有限公司 可户外移动陪伴居家养老机器人
CN107168174B (zh) * 2017-06-15 2019-08-09 重庆柚瓣科技有限公司 一种使用机器人做居家养老的方法
CN109313645B (zh) * 2017-08-25 2022-05-24 深圳市大富智慧健康科技有限公司 人工智能终端系统、服务器及其行为控制方法
CN107808191A (zh) * 2017-09-13 2018-03-16 北京光年无限科技有限公司 虚拟人多模态交互的输出方法和系统
US10818288B2 (en) * 2018-03-26 2020-10-27 Apple Inc. Natural assistant interaction
CN108536386A (zh) * 2018-03-30 2018-09-14 联想(北京)有限公司 数据处理方法、设备以及系统
CN108874123A (zh) * 2018-05-07 2018-11-23 北京理工大学 一种通用的模块化的虚拟现实被动力触觉反馈系统
CN109965466A (zh) * 2018-05-29 2019-07-05 北京心有灵犀科技有限公司 Ar虚拟角色智能珠宝
CN110653813A (zh) * 2018-06-29 2020-01-07 深圳市优必选科技有限公司 一种机器人控制方法、机器人及计算机存储介质
CN109521878A (zh) * 2018-11-08 2019-03-26 歌尔科技有限公司 交互方法、装置和计算机可读存储介质
CN113313836B (zh) * 2021-04-26 2022-11-25 广景视睿科技（深圳）有限公司 一种控制虚拟宠物的方法及智能投影设备
CN115793938A (zh) * 2022-11-08 2023-03-14 Oppo广东移动通信有限公司 互动壁纸的交互方法、电子设备及计算机可读存储介质

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6239785B1 (en) * 1992-10-08 2001-05-29 Science & Technology Corporation Tactile computer input device
US6560511B1 (en) * 1999-04-30 2003-05-06 Sony Corporation Electronic pet system, network system, robot, and storage medium
US6545682B1 (en) * 2000-05-24 2003-04-08 There, Inc. Method and apparatus for creating and customizing avatars using genetic paradigm
US8909370B2 (en) * 2007-05-08 2014-12-09 Massachusetts Institute Of Technology Interactive systems employing robotic companions
US20090102805A1 (en) * 2007-10-18 2009-04-23 Microsoft Corporation Three-dimensional object simulation using audio, visual, and tactile feedback
US8795072B2 (en) * 2009-10-13 2014-08-05 Ganz Method and system for providing a virtual presentation including a virtual companion and virtual photography
CN201611889U (zh) * 2010-02-10 2010-10-20 深圳先进技术研究院 即时通讯伴侣机器人
JP5812665B2 (ja) * 2011-04-22 2015-11-17 任天堂株式会社 情報処理システム、情報処理装置、情報処理方法及び情報処理プログラム

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3529707A4 (fr) * 2016-10-20 2020-08-05 RN Chidakashi Technologies Pvt. Ltd. Dispositif compagnon émotionnellement intelligent
US12112285B1 (en) 2021-07-07 2024-10-08 Wells Fargo Bank, N.A. Systems and methods for measuring employee experience

Also Published As

Publication number Publication date
WO2015004620A3 (fr) 2015-05-14
CN104769645A (zh) 2015-07-08

Similar Documents

Publication Publication Date Title
US20140125678A1 (en) Virtual Companion
WO2015004620A2 (fr) Compagnon virtuel
US12450811B2 (en) Emoji recording and sending
US10444876B2 (en) Human-computer interface device and system
US20180335929A1 (en) Emoji recording and sending
JP2016510452A (ja) アクションを決定する際の非言語コミュニケーションの使用
US12315640B2 (en) Atmospheric mirroring and dynamically varying three-dimensional assistant addison interface for interior environments
CN114712862A (zh) 虚拟宠物交互方法、电子设备及计算机可读存储介质
US20210077888A1 (en) Physiological response management using computer-implemented activities
Nijholt Towards social companions in augmented reality: Vision and challenges
Jeong The impact of social robots on young patients' socio-emotional wellbeing in a pediatric inpatient care context
CN118732824A (zh) 界面显示方法、装置、扩展现实设备及存储介质
US12374015B2 (en) Facial capture artificial intelligence for training models
Dow et al. AR-Based Embodied Avatar Assistance for Nonspeaking Autistic People? Design and Feasibility Study
Ensslin et al. Future modes
Huang Development of Human-Computer Interaction for Holographic AIs
JP2024016801A (ja) 対話型高齢者支援システム
Norouzi The Social and Behavioral Influences of Interactions with Virtual Dogs as Embodied Agents in Augmented and Virtual Reality
Zegura Structuring Feeling, Feeling Structure: Affect and the Game Engine
HK1237444A1 (en) A human-computer interface device and system
HK1237444B (zh) 人机接口装置和系统
HK1241089B (en) A human-computer interface device and system
HK1241089A1 (en) A human-computer interface device and system
Herreros Díaz Implementation of Open Source applications “Serious Game” for rehabilitation
Rix Viability of a tangible tabletop for industry storyboarding

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14823493

Country of ref document: EP

Kind code of ref document: A2

122 Ep: pct application non-entry in european phase

Ref document number: 14823493

Country of ref document: EP

Kind code of ref document: A2