WO2020228682A1 - 对象交互方法及装置、系统、计算机可读介质和电子设备 - Google Patents

对象交互方法及装置、系统、计算机可读介质和电子设备 Download PDF

Info

Publication number
WO2020228682A1
WO2020228682A1 PCT/CN2020/089641 CN2020089641W WO2020228682A1 WO 2020228682 A1 WO2020228682 A1 WO 2020228682A1 CN 2020089641 W CN2020089641 W CN 2020089641W WO 2020228682 A1 WO2020228682 A1 WO 2020228682A1
Authority
WO
WIPO (PCT)
Prior art keywords
pose
terminal device
information
real environment
coordinate system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/089641
Other languages
English (en)
French (fr)
Inventor
王宇鹭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Oppo Mobile Telecommunications Corp Ltd
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp Ltd filed Critical Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority to EP20806291.9A priority Critical patent/EP3968131B1/en
Publication of WO2020228682A1 publication Critical patent/WO2020228682A1/zh
Priority to US17/525,772 priority patent/US12067158B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional [3D] objects
    • G06V20/647—Three-dimensional [3D] objects by matching two-dimensional images to three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G06V40/11—Hand-related biometrics; Hand pose recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • the present disclosure relates to the field of augmented reality technology, and in particular to an object interaction method, an object interaction device, an object interaction system, a computer-readable medium, and an electronic device.
  • Augmented Reality is a technology that integrates virtual elements into the real world and can interact with reality. With the improvement of terminal equipment processing capabilities, augmented reality technology can be widely used in various fields such as games, education, medical care, transportation, and information tracking.
  • the process of using augmented reality technology to realize the interaction between real and virtual objects it is completely dependent on the user's touch operation on the terminal screen.
  • the user can realize the translation of the virtual object by touching the control used to control the virtual object. , Rotate, place, delete and other effects.
  • the touch operation based on the terminal screen does not conform to people’s interaction habits in daily activities, resulting in inconvenient operation for users when using it; on the other hand, in some related technologies, virtual reality is realized only based on the positional relationship between objects. Interaction, scenarios that do not conform to the common sense of interaction may appear, and the user's interaction experience is poor.
  • the purpose of the present disclosure is to provide an object interaction method, object interaction device, object interaction system, computer readable medium and electronic equipment, thereby at least to some extent overcome the virtual-real interaction of augmented reality technology due to the limitations and defects of related technologies The problem of poor results.
  • an object interaction method is provided, which is applied to a terminal device.
  • the method includes: determining the pose of the terminal device in a real environment coordinate system; and determining the first object according to the pixel information and depth information of the first object Relative to the pose of the terminal device; where the first object is a real object in the real environment; according to the pose of the terminal device in the real environment coordinate system and the pose of the first object relative to the terminal device, it is determined that the first object is The pose in the real environment coordinate system; determine the pose of the second object in the real environment coordinate system; where the second object is a virtual object configured in the real environment; according to the pose of the first object in the real environment coordinate system And the pose of the second object in the real environment coordinate system to control the interaction between the first object and the second object.
  • an object interaction device which is applied to a terminal device.
  • the device includes a terminal pose determination module, a first object pose determination module, a coordinate conversion module, a second object pose determination module, and an object interaction Module.
  • the terminal pose determination module is used to determine the pose of the terminal device in the real environment coordinate system; the first object pose determination module is used to determine the first object relative to the terminal device according to the pixel information and depth information of the first object The pose; where the first object is a real object in the real environment; the coordinate conversion module is used to determine the first object according to the pose of the terminal device in the real environment coordinate system and the pose of the first object relative to the terminal device The pose in the real environment coordinate system; the second object pose determination module is used to determine the pose of the second object in the real environment coordinate system; where the second object is a virtual object configured in the real environment; the object interaction module It is used to control the interaction between the first object and the second object according to the pose of the first object in the real environment coordinate system and the pose of the second object in the real environment coordinate system.
  • the terminal pose determination module includes a feature point extraction unit, a fixed feature determination unit, and a terminal pose determination unit.
  • the feature point extraction unit is used to extract feature points from the video images collected in the real scene;
  • the fixed feature determination unit is used to track the feature points in continuous video frames to determine the three-dimensional fixation in the real scene
  • the terminal posture determination unit is used to determine the posture of the terminal device in the real environment coordinate system according to the position transformation information of the three-dimensional fixed feature in the video frame and the inertial information of the terminal device.
  • the terminal pose determination module further includes a map construction unit.
  • the map construction unit is used to construct a map of the real scene by using the feature point cloud formed by the three-dimensional fixed feature after determining the three-dimensional fixed feature in the real scene.
  • the first object pose determination module is configured to perform: determine the interactive area of the first object; determine the pixel information and depth information of the interactive area; determine the interactive area according to the pixel information and depth information of the interactive area Position information and posture information relative to the terminal device; wherein the position information and posture information of the interactive area relative to the terminal device are used to characterize the position and posture of the first object relative to the terminal device.
  • the first object is a hand; in this case, the process of determining the interactive area of the first object by the first object pose determination module may include: determining key points of the hand; determining based on the key points of the hand The palm plane out is determined as the interactive area of the first object.
  • the first object pose determination module is configured to execute: determine the position information of the center point of the palm plane according to the pixel information and depth information of the palm plane as the position information of the palm plane relative to the terminal device; and according to the palm plane The pixel information and depth information of, determine the normal orientation information of the palm plane, as the posture information of the palm plane relative to the terminal device.
  • the object interaction module is configured to execute: determine the distance between the first object and the second object according to the position information of the first object and the position information of the second object in the real environment coordinate system; The distance of the object is less than the preset distance threshold and the postures of the first object and the second object match, then the interaction between the first object and the second object is triggered.
  • an object interaction system including a camera, an inertial sensor, a depth sensor, one or more processors, and a storage device.
  • the camera is used to collect image information in the real scene;
  • the inertial sensor is used to collect the inertial information of the terminal device;
  • the depth sensor is used to determine the depth information of the object in the real scene;
  • the storage device is used to store one or more programs; Wherein, when one or more programs are executed by one or more processors, one or more processors are caused to implement the foregoing object interaction method.
  • a computer-readable medium on which a computer program is stored, and the program is executed by a processor to realize the above-mentioned object interaction method.
  • an electronic device including: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by one or more processors, Make one or more processors implement the above object interaction method.
  • the position of the terminal device in the real environment coordinate system and the position of the first object relative to the terminal device are used to determine the position of the first object in the real environment coordinate system. According to the poses of the first object and the second object in the real environment coordinate system, the first object and the second object are controlled to interact.
  • the depth information is combined, so that the object interaction can be extended to the three-dimensional space more conveniently, which improves the robustness of the object interaction process;
  • the object interaction method of the present disclosure considers the pose, including position and posture, relative Compared with some related technologies, the introduction of gestures avoids the possibility of scenes that do not conform to common sense of interaction, makes the interaction effect between the first object and the second object more realistic, and improves the interactive experience of the user using augmented reality technology to achieve virtual-real interaction.
  • Fig. 1 schematically shows a block diagram of an object interaction system according to an exemplary embodiment of the present disclosure
  • FIG. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure
  • FIG. 3 schematically shows a flowchart of an object interaction method according to an exemplary embodiment of the present disclosure
  • FIG. 4 schematically shows a flowchart of a method for determining the pose of a terminal device in a real environment coordinate system according to an exemplary embodiment of the present disclosure
  • FIG. 5 schematically shows a flowchart illustrating the object interaction method of the present disclosure by taking the hand as the first object as an example
  • FIG. 6 shows a schematic diagram of a user interface for explaining the object interaction method of the present disclosure by taking a real human hand grasping a virtual ball as an example
  • Fig. 7 schematically shows a block diagram of an object interaction apparatus according to an exemplary embodiment of the present disclosure
  • Fig. 8 schematically shows a block diagram of a terminal pose determination module according to an exemplary embodiment of the present disclosure
  • Fig. 9 schematically shows a block diagram of a terminal pose determination module according to another exemplary embodiment of the present disclosure.
  • gesture recognition and interaction based on RGB (red, green and blue) images are limited to two-dimensional screens.
  • RGB red, green and blue
  • this two-dimensional interaction is inconvenient in actual operation and does not conform to the real interaction habits of users in daily activities.
  • Fig. 1 schematically shows a block diagram of an object interaction system according to an exemplary embodiment of the present disclosure.
  • the object interaction system of the exemplary embodiment of the present disclosure may be configured in a terminal device.
  • the terminal device may be, for example, a mobile phone, a tablet, a head-mounted display device, etc., and the present disclosure does not impose special restrictions on the type of terminal device.
  • the object interaction system 10 may include a camera 101, an inertial sensor 102, a depth sensor 103, a processor 104, and a storage device 105.
  • the camera 101 is used to collect image information in a real scene.
  • the image collected by the camera 101 is an RGB image;
  • the inertial sensor 102 may also be called an inertial measurement unit (IMU), which is used for Collect the inertial information of the terminal equipment;
  • the depth sensor 103 is used to determine the depth information of each object in the real scene;
  • the processor 104 may be a processor or a processor cluster including multiple processors;
  • the storage device 105 is used for storage One or more programs.
  • the processor 104 when one or more programs are executed by the processor 104, the processor 104 is enabled to implement: using the pixel information collected by the camera 101 in the real scene and the inertial information of the terminal device collected by the inertial sensor 102 to determine the terminal device The pose in the real environment coordinate system; based on the pixel information collected by the camera 101 and the depth information collected by the depth sensor 103, determine the pose of the first object relative to the terminal device; according to the terminal device's pose in the real environment coordinate system The pose and the pose of the first object relative to the terminal device determine the pose of the first object in the real environment coordinate system; determine the pose of the second object in the real environment coordinate system, according to the first object in the real environment coordinate system The next pose and the pose of the second object in the real environment coordinate system control the interaction between the first object and the second object.
  • the first object is an object that actually exists in a real environment, and may be a real object.
  • the second object is a virtual object configured in a real environment using computer technology, and the virtual object does not really exist in the real world.
  • Fig. 2 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present invention.
  • the computer system 200 includes a central processing unit (CPU) 201, which can be based on a program stored in a read only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203 And perform various appropriate actions and processing.
  • CPU 201 central processing unit
  • ROM read only memory
  • RAM random access memory
  • various programs and data required for system operation are also stored.
  • the CPU 201, ROM 202, and RAM 203 are connected to each other through a bus 204.
  • An input/output (I/O) interface 205 is also connected to the bus 204.
  • the following components are connected to the I/O interface 205: an input part 206 including a keyboard, a mouse, etc.; an output part 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part 208 including a hard disk, etc. ; And a communication section 209 including a network interface card such as a LAN card, a modem, etc.
  • the communication section 209 performs communication processing via a network such as the Internet.
  • the drive 210 is also connected to the I/O interface 205 as needed.
  • a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that the computer program read from it is installed into the storage section 208 as needed.
  • an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart.
  • the computer program may be downloaded and installed from the network through the communication section 209, and/or installed from the removable medium 211.
  • CPU central processing unit
  • various functions defined in the system of the present application are executed.
  • the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two.
  • the computer-readable storage medium may be, for example, but not limited to, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.
  • each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the above-mentioned module, program segment, or part of code contains one or more for realizing the specified logical function Executable instructions.
  • the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be It is realized by a combination of dedicated hardware and computer instructions.
  • the units involved in the embodiments described in the present invention can be implemented in software or hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.
  • this application also provides a computer-readable medium.
  • the computer-readable medium may be included in the electronic device described in the above-mentioned embodiments; or it may exist alone without being assembled into the electronic device. in.
  • the above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device realizes the method described in the following embodiments.
  • each step of the object interaction method can be executed by the terminal device.
  • the object interaction apparatus described below can be configured in the terminal device.
  • the object interaction method may include the following steps:
  • a camera may be used to photograph a real scene to determine the pixel information of the image in the real scene.
  • a monocular camera may be used to shoot a real scene to determine the pixel information of each real object in the real scene.
  • the monocular camera is a camera constructed according to the principle of small hole imaging, and is a single color (or grayscale) camera, which has the advantages of simple use, high accuracy, and convenient configuration.
  • the terminal device as a mobile phone
  • the rear camera of the mobile phone can be used as a monocular camera to shoot real scenes.
  • inertial sensors can be used to monitor the inertial information of the terminal equipment in real time, where the inertial information mainly includes translation and rotation.
  • the inertial sensor may include three accelerometers and three gyroscopes.
  • the three accelerometers are used to detect the acceleration signal of the terminal device on each coordinate axis of the three-dimensional coordinate system, and then the amount of translation can be calculated; and the three gyroscopes , Can detect the rotation of the terminal device in the three-dimensional coordinate system.
  • the amount of rotation can be expressed as a rotation matrix.
  • the pixel information of the collected image can be used as the observation constraint
  • the inertial information detected by the inertial sensor can be used as the action constraint.
  • determining the pose of the terminal device in the real environment coordinate system can be described as: the pose of the terminal device determined under the joint constraint of the observation constraint and the action constraint.
  • the pose mentioned in the present disclosure includes position information and pose information.
  • a camera is used to collect a video image, and feature points are extracted from the video image.
  • the feature point may be a point that characterizes the contour of the object.
  • the collected image can be input into a feature point extraction model to determine feature points in the image, where the feature point extraction model can be a trained convolutional neural network.
  • the convolutional neural network it is easy to understand that the types of marked sample images can be enriched to expand the application range of the convolutional neural network, enhance the robustness of the processing process, and improve the accuracy of identifying feature points.
  • the determined feature points can be combined with the inertial information detected by the inertial sensor to establish an initial coordinate system, and establish an initial local map and the initial pose of the terminal device.
  • step S404 the feature points are tracked in consecutive video frames.
  • the feature point tracking may use an algorithm based on the principle of visual odometry, such as optical flow tracking, direct method, and the like, which is not particularly limited in the present disclosure.
  • the KLT tracking algorithm Kanade-Lucas-Tomasi Tracking Method
  • the feature point tracking may be based on ORB (Oriented FAST and Rotated Briefing, fast feature point extraction and Description)
  • ORB Oriented FAST and Rotated Briefing, fast feature point extraction and Description
  • the neighborhood of the feature point can be affine transformed to obtain the projection area of the neighborhood in the current frame, and the color shift between the neighborhood in the previous frame and the projection area in the current frame can be calculated As the tracking deviation of the feature point.
  • multiple feature points with smaller tracking deviation can be selected to realize the tracking process of the feature points.
  • step S406 a three-dimensional fixed feature is determined.
  • the three-dimensional fixed feature in the real environment can be determined based on the tracking of the feature point.
  • These three-dimensional fixed features can be corner points and feature points with significant observation characteristics in the real environment. For example, Sift (Scale-invariant feature transform) feature points, Harris corner points, ORB feature points, etc.
  • the feature point cloud formed by these three-dimensional fixed features can be used to construct a map of the real scene.
  • step S408 the position transformation information of the three-dimensional fixed feature is determined.
  • the terminal device can determine the position transformation information of the three-dimensional fixed feature relative to the camera.
  • the terminal device may use an inertial sensor to monitor inertial information in real time.
  • the pose of the terminal device in the real environment coordinate system may be determined according to the position transformation information of the three-dimensional fixed feature in the video frame and the inertia information of the terminal device.
  • the process of determining the pose of the terminal device in the present disclosure may include: first, using limited RGB images and inertial information to construct an initial three-dimensional feature point cloud and determining the initial pose of the terminal device; Continuously collected video frames, continuous observation constraints on the pose of the map and terminal equipment, and continuous motion constraints on the pose of the map and terminal equipment based on the continuously collected inertial information; then, in the observation constraints and Under the combined action of motion constraints, the pose of the terminal device is determined.
  • the first object refers to a real object that actually exists in real space.
  • the first object is described below by taking a human hand as an example, the first object may also be a human body. Other parts.
  • the first object may also be other objects in reality, such as wooden sticks, rackets, and so on.
  • a depth sensor can be used to determine the depth information of the first object relative to the terminal device.
  • a TOF (Time Of Flight) sensor can be used to determine depth information.
  • the TOF sensor can emit near-infrared light and reflect after encountering the first object.
  • the TOF sensor calculates the distance to the first object by calculating the time difference or phase difference between light emission and reflection to determine the depth information.
  • the depth information corresponds to the pixel information collected in the real scene. In other words, the determination of depth information and the collection of image pixel information often occur simultaneously. For each frame of image, there is corresponding depth information.
  • the depth information can be combined with the image pixel information collected by the camera in step S30 to determine the pose of the first object relative to the terminal device.
  • the first object needs to be identified.
  • the image can be input to an object recognition model.
  • the object recognition module can be a trained machine learning model.
  • the model can perform feature extraction on the first object to determine the category of the first object. Taking human hands as an example, the model can perform semantic segmentation on human hands and extract key points of the hands. When the predicted probability of the human hand is greater than a preset threshold, the first object can be determined to be the human hand.
  • the robustness of determining the pose of the first object relative to the terminal device is improved.
  • processing procedure of step S32 may further include:
  • the interactive area of the first object can be determined.
  • the interactive area may refer to the area where interactive actions are actually performed. For example, taking a hand grasping a virtual ball, a carrying bag, etc., as an example, the first object is the hand, and the interactive area may be the palm plane.
  • key points of the hand may be determined.
  • the key points of the hand may be points corresponding to the roots of the five fingers.
  • the palm plane can be determined as the interactive area of the first object.
  • the depth information corresponding to the pixel information collected in the real scene can be determined, and the position information and posture information of the interactive area relative to the terminal device can be determined according to the depth information and the pixel information.
  • the position and posture information of the interactive area relative to the terminal device can be used to characterize the posture of the first object relative to the terminal device.
  • the position information of the center point of the palm plane can be used as the position information of the palm plane relative to the terminal device.
  • the normal orientation information of the palm plane may be used as the posture information of the palm plane relative to the terminal device, wherein the normal orientation is perpendicular to the direction of the palm plane.
  • the pose is expressed in the form of coordinates, which can be expressed as (R, T), where R represents posture information and T represents position information.
  • the pose of the terminal device in the real environment coordinate system can be expressed as (R1, T1)
  • the pose of the first object relative to the terminal device can be expressed as (R2, T2)
  • the first The pose of the object in the real environment coordinate system is (R2*R1, T2+R2*T1).
  • the second object does not exist in the real environment.
  • the positional relationship and the posture relationship between the first object and the second object may be comprehensively considered to control their interaction.
  • the distance between the first object and the second object can be determined according to the position information of the first object and the position information of the second object in the real environment coordinate system.
  • the preset distance threshold can be set by the developer according to the complexity of the AR scene interaction. For example, the preset distance threshold can be set Set to 2cm.
  • the matching relationship can be pre-configured according to the interaction scene, for example, when grasping a virtual ball,
  • the matching relationship is that the normal of the palm plane is oriented toward the center of the sphere.
  • the interaction between the first object and the second object is triggered.
  • the interactive actions can be set by the developer in advance, including but not limited to changing the color of the second object, changing the shape, and executing preset actions.
  • the present disclosure does not impose any special restrictions on the order of verifying the position relationship and the posture relationship. That is to say, whether the posture relationship is matched can be verified first, and after the verification is successful, it is determined whether the distance between the first object and the second object meets the distance. Claim.
  • the above-mentioned distance requirement is not limited to performing interactive actions when the distance is relatively short.
  • the interactive action may also be triggered.
  • step S502 determine the pose of the terminal device in the real environment coordinate system; in step S504, identify and track the hand in the collected video image, determine the key points of the hand, and determine according to the key points of the hand Out of the palm plane; on the one hand, in step S506, the position information of the center point of the palm plane can be determined; in step S508, according to the position information of the center point of the palm plane, determine the center point of the palm plane and the second object (virtual object) Whether the distance is less than the preset distance. If it is less than the preset distance, skip to step S514; if it is not less than the preset distance, skip to step S506.
  • step S510 determine the normal direction of the palm plane; in step S512, determine whether the normal direction of the palm plane matches the posture of the second object, if it matches, jump to step S514, if it does not match, Then jump to step S510.
  • step S514 in the AR scenario, an interactive operation between the palm and the second object may be triggered. For example, grasping a virtual ball, carrying a virtual basket, holding a virtual door handle, etc.
  • the object interaction method of the present disclosure will be described by taking the first object as a real human hand and the second object as a virtual ball as an example.
  • a virtual ball 62 can be placed on the desktop 60 in a real scene through computer technology. At this time, the user can view a virtual ball on the real desktop through the screen of the terminal device.
  • the user can grab the virtual ball 61 from the desktop 60 with a real human hand 62.
  • the specific process can include: First, the terminal device can use the camera to obtain the pixel information of the human hand 62 and use the depth sensor to obtain the depth information of the human hand 62 to determine the pose of the human hand 62 relative to the terminal device; The pose in the real coordinate system and the pose of the human hand 62 relative to the terminal device determine the pose of the human hand 62 in the real environment coordinate system.
  • the terminal device can determine the distance between the human hand 62 and the virtual ball 61 and the relative relationship between the normal direction of the palm plane of the human hand 62 and the virtual ball 61 in real time.
  • the normal direction of the palm plane is a direction perpendicular to the palm plane and outward from the palm center of the palm.
  • the predetermined interaction rule is: when the distance between the human hand and the virtual ball is less than 3 cm and the normal of the palm plane faces the direction pointing to the center of the sphere, the human hand grabs the virtual ball.
  • the human hand 62 can grab the virtual ball 61. After the human hand 62 grabs the virtual ball 61 from the desktop 60, it can keep the posture and move to change the position or movement of the virtual ball.
  • FIG. 6 shows a schematic diagram of a human hand 62 picking up and moving the virtual ball 61 for a certain distance.
  • the gesture on the premise that the posture of the human hand is determined, the gesture can be recognized, and the corresponding interactive action can be determined according to the result of the gesture recognition.
  • the second object is a virtual character (for example, a virtual cartoon character)
  • the virtual character if the distance between the human hand and the virtual character is greater than 0.5m and less than 2m, and the normal direction of the palm is pointing to the direction of the virtual character, it will be recognized
  • the virtual character is triggered to execute a greeting instruction.
  • the virtual character can also wave their arms to achieve an interactive action of greeting each other.
  • this exemplary embodiment also provides an object interaction device, which is applied to mobile equipment.
  • Fig. 7 schematically shows a block diagram of an object interaction apparatus of an exemplary embodiment of the present disclosure.
  • the object interaction device 7 may include a terminal pose determination module 71, a first object pose determination module 73, a coordinate conversion module 75, a second object pose determination module 77, and an object Interactive module 79.
  • the terminal pose determination module 71 may be used to determine the pose of the terminal device in the real environment coordinate system; the first object pose determination module 73 may be used to determine the first object based on the pixel information and depth information of the first object Relative to the pose of the terminal device; where the first object is a real object in the real environment; the coordinate conversion module 75 can be used to determine the pose of the terminal device in the real environment coordinate system and the position of the first object relative to the terminal device.
  • the second object pose determination module 77 can be used to determine the pose of the second object in the real environment coordinate system; where the second object is configured in the real environment
  • the object interaction module 79 can be used to control the first object to interact with the second object according to the pose of the first object in the real environment coordinate system and the pose of the second object in the real environment coordinate system.
  • the object interaction device of the exemplary embodiment of the present disclosure uses the object interaction device of the exemplary embodiment of the present disclosure, so that the object interaction can be more conveniently extended to the three-dimensional space scale.
  • the robustness of the object interaction process is improved; on the other hand, by aligning the first object and the second object to the same coordinate system, it is easy to analyze and conforms to the real physical interaction experience; on the other hand, the object interaction method of the present disclosure
  • the pose including position and posture.
  • the introduction of posture avoids the possibility of scenes that do not conform to the common sense of interaction, makes the interaction effect between the first object and the second object more realistic, and improves the user Use augmented reality technology to achieve an interactive experience of virtual and real interaction.
  • the terminal pose determination module 71 may include a feature point extraction unit 801, a fixed feature determination unit 803, and a terminal pose determination unit 805.
  • the feature point extraction unit 801 can be used to extract feature points from video images collected in a real scene; the fixed feature determination unit 803 can be used to track feature points in continuous video frames to determine the real scene
  • the terminal pose determination unit 805 can be used to determine the pose of the terminal device in the real environment coordinate system according to the position transformation information of the three-dimensional fixed feature in the video frame and the inertial information of the terminal device.
  • the terminal pose determination module 71 may further include a map construction unit 901.
  • the map construction unit 901 may be used to construct a map of the real scene by using the feature point cloud formed by the three-dimensional fixed feature after determining the three-dimensional fixed feature in the real scene.
  • the first object pose determination module 73 may be configured to perform: determine the interactive area of the first object; determine the pixel information and depth information of the interactive area; according to the pixel information of the interactive area And depth information to determine the position information and posture information of the interactive area relative to the terminal device; wherein the position information and posture information of the interactive area relative to the terminal device are used to characterize the position and posture of the first object relative to the terminal device.
  • the first object is a hand; in this case, the process of the object pose determination module 73 determining the interactive area of the first object may include: determining key points of the hand; The palm plane determined by the key points is determined as the interactive area of the first object.
  • the first object pose determination module 73 may be configured to execute: determine the position information of the center point of the palm plane according to the pixel information and depth information of the palm plane, as the position information of the palm plane relative to the terminal device. Position information; and determining the normal orientation information of the palm plane according to the pixel information and depth information of the palm plane as the posture information of the palm plane relative to the terminal device.
  • the object interaction module 79 may be configured to execute: determine the distance between the first object and the second object according to the position information of the first object and the position information of the second object in the real environment coordinate system; If the distance between the first object and the second object is less than the preset distance threshold and the postures of the first object and the second object match, then the interaction between the first object and the second object is triggered.
  • the exemplary embodiments described herein can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (can be a CD-ROM, U disk, mobile hard disk, etc.) or on the network , Including several instructions to make a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present disclosure.
  • a computing device which may be a personal computer, a server, a terminal device, or a network device, etc.
  • modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory.
  • the features and functions of two or more modules or units described above may be embodied in one module or unit.
  • the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

一种对象交互方法及装置、系统、计算机可读介质和电子设备，涉及增强现实技术领域。该对象交互方法包括：确定终端设备在现实环境坐标系下的位姿（S30）；根据第一对象的像素信息和深度信息确定第一对象相对于终端设备的位姿；其中，第一对象为现实环境下的真实对象（S32）；根据终端设备在现实环境坐标系下的位姿以及第一对象相对于终端设备的位姿，确定第一对象在现实环境坐标系下的位姿（S34）；确定现实环境坐标系下第二对象的位姿；其中，第二对象为虚拟对象（S36）；根据第一对象在现实环境坐标系下的位姿以及现实环境坐标系下第二对象的位姿，控制第一对象与第二对象进行交互（S38）。可以在三维尺度实现虚实对象的交互，使得交互效果更加真实。

Description

对象交互方法及装置、系统、计算机可读介质和电子设备 技术领域
本公开涉及增强现实技术领域，具体而言，涉及一种对象交互方法、对象交互装置、对象交互系统、计算机可读介质和电子设备。
背景技术
增强现实(Augmented Reality，AR)是一种将虚拟元素融入现实世界并可进行虚实交互的技术。随着终端设备处理能力的提升，增强现实技术可以被广泛应用于游戏、教育、医疗、交通、信息跟踪等各个领域。
目前，在采用增强现实技术实现真实物体与虚拟物体之间交互的过程中，完全依赖于用户针对终端屏幕的触摸操作，例如，用户通过触摸用于控制虚拟物体的控件，可以实现虚拟物体的平移、旋转、放置、删除等效果。
一方面，基于终端屏幕的触摸操作不符合人们在日常活动中的交互习惯，导致用户在使用时操作不方便；另一方面，在一些相关技术中，仅基于物体之间的位置关系来实现虚实交互，可能出现不符合交互常识的场景，用户的交互体验差。
需要说明的是，在上述背景技术部分公开的信息仅用于加强对本公开的背景的理解，因此可以包括不构成对本领域普通技术人员已知的现有技术的信息。
发明内容
本公开的目的在于提供一种对象交互方法、对象交互装置、对象交互系统、计算机可读介质和电子设备，进而至少在一定程度上克服由于相关技术的限制和缺陷而导致的增强现实技术虚实交互效果不佳的问题。
根据本公开的一个方面，提供一种对象交互方法，应用于终端设备，该方法包括：确定终端设备在现实环境坐标系下的位姿；根据第一对象的像素信息和深度信息确定第一对象相对于终端设备的位姿；其中，第一对象为现实环境下的真实对象；根据终端设备在现实环境坐标系下的位姿以及第一对象相对于终端设备的位姿，确定第一对象在现实环境坐标系下的位姿；确定现实环境坐标系下第二对象的位姿；其中，第二对象为配置在现实环境下的虚拟对象；根据第一对象在现实环境坐标系下的位姿以及现实环境坐标系下第二对象的位姿，控制第一对象与第二对象进行交互。
根据本公开的一个方面，提供一种对象交互装置，应用于终端设备，该装置包括终端位姿确定模块、第一对象位姿确定模块、坐标转换模块、第二对象位姿确定模块和对象交互模块。
具体的，终端位姿确定模块用于确定终端设备在现实环境坐标系下的位姿；第一对象位姿确定模块用于根据第一对象的像素信息和深度信息确定第一对象相对于终端设备的位姿；其中，第一对象为现实环境下的真实对象；坐标转换模块用于根据终端设备在现实环境坐标系下的位姿以及第一对象相对于终端设备的位姿，确定第一对象在现实环境坐标系下的位姿；第二对象位姿确定模块用于确定现实环境坐标系下第二对象的位姿；其中，第二对象为配置在现实环境下的虚拟对象；对象交互模块用于根据第一对象在现实环境坐标系下的位姿以及现实环境坐标系下第二对象的 位姿，控制第一对象与第二对象进行交互。
可选地，终端位姿确定模块包括特征点提取单元、固定特征确定单元和终端姿态确定单元。
具体的，特征点提取单元用于从在现实场景下采集到的视频图像中提取特征点；固定特征确定单元用于在连续的视频帧中，对特征点进行跟踪，确定现实场景下的三维固定特征；终端姿态确定单元用于根据三维固定特征在视频帧中的位置变换信息以及终端设备的惯性信息，确定终端设备在现实环境坐标系下的位姿。
可选地，终端位姿确定模块还包括地图构建单元。
具体的，地图构建单元用于在确定现实场景下的三维固定特征后，利用三维固定特征形成的特征点云，构建现实场景的地图。
可选地，第一对象位姿确定模块被配置为执行：确定第一对象的可交互区域；确定可交互区域的像素信息和深度信息；根据可交互区域的像素信息和深度信息确定可交互区域相对于终端设备的位置信息和姿态信息；其中，利用可交互区域相对于终端设备的位置信息和姿态信息表征第一对象相对于终端设备的位姿。
可选地，第一对象为手部；在这种情况下，第一对象位姿确定模块确定第一对象的可交互区域的过程可以包括：确定手部关键点；将基于手部关键点确定出的手掌平面确定为第一对象的可交互区域。
可选地，第一对象位姿确定模块被配置为执行：根据手掌平面的像素信息和深度信息确定手掌平面的中心点的位置信息，作为手掌平面相对于终端设备的位置信息；以及根据手掌平面的像素信息和深度信息确定手掌平面的法线朝向信息，作为手掌平面相对于终端设备的姿态信息。
可选地，对象交互模块被配置为执行：根据现实环境坐标系下第一对象的位置信息和第二对象的位置信息，确定第一对象与第二对象的距离；如果第一对象与第二对象的距离小于预设距离阈值且第一对象与第二对象的姿态匹配，则触发执行第一对象与第二对象的交互动作。
根据本公开的一个方面，提供一种对象交互系统，包括相机、惯性传感器、深度传感器、一个或多个处理器、存储装置。
具体的，相机用于在现实场景下采集图像信息；惯性传感器用于采集终端设备的惯性信息；深度传感器用于确定对象在现实场景下的深度信息；存储装置用于存储一个或多个程序；其中，当一个或多个程序被一个或多个处理器执行时，使得一个或多个处理器实现上述对象交互方法。
根据本公开的一个方面，提供一种计算机可读介质，其上存储有计算机程序，程序被处理器执行时实现上述对象交互方法。
根据本公开的一个方面，提供一种电子设备，包括：一个或多个处理器；存储装置，用于存储一个或多个程序，当一个或多个程序被一个或多个处理器执行时，使得一个或多个处理器实现上述对象交互方法。
在本公开的一些实施例所提供的技术方案中，利用终端设备在现实环境坐标系下的位姿以及第一对象相对于终端设备的位姿，确定第一对象在现实环境坐标系下的位姿，并根据第一对象和第二对象在现实环境坐标系下的位姿，控制第一对象与第二对象进行交互。一方面，在确定第一对象相对于终端设备的位姿时，结合了深度信息，由此，可以更方便的使对象交互延伸至三维空间尺度，提高了对象交互过程的鲁棒性；另一方面，通过将第一对象与第二对象对齐到同一坐标系下，便于分析，符合真实的 物理交互体验；再一方面，本公开的对象交互方法考虑的是位姿，包括位置和姿态，相比于一些相关技术，姿态的引入，避免了可能出现不符合交互常识的场景，使得第一对象与第二对象的交互效果更加真实，提高了用户利用增强现实技术实现虚实互动的交互体验。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。在附图中：
图1示意性示出了根据本公开的示例性实施方式的对象交互系统的方框图；
图2示出了适于用来实现本公开实施例的电子设备的计算机系统的结构示意图；
图3示意性示出了根据本公开的示例性实施方式的对象交互方法的流程图；
图4示意性示出了根据本公开的示例性实施方式的确定终端设备在现实环境坐标系下位姿的方法的流程图；
图5示意性示出了以手作为第一对象为例对本公开的对象交互方法进行说明的流程图；
图6示出了以真实人手抓虚拟皮球为例对本公开的对象交互方法进行说明的用户界面示意图；
图7示意性示出了根据本公开的示例性实施方式的对象交互装置的方框图；
图8示意性示出了根据本公开的示例性实施方式的终端位姿确定模块的方框图；
图9示意性示出了根据本公开的另一示例性实施方式的终端位姿确定模块的方框图。
具体实施方式
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本公开将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。在下面的描述中，提供许多具体细节从而给出对本公开的实施方式的充分理解。然而，本领域技术人员将意识到，可以实践本公开的技术方案而省略所述特定细节中的一个或更多，或者可以采用其它的方法、组元、装置、步骤等。在其它情况下，不详细示出或描述公知技术方案以避免喧宾夺主而使得本公开的各方面变得模糊。
此外，附图仅为本公开的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描 述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。
附图中所示的流程图仅是示例性说明，不是必须包括所有的步骤。例如，有的步骤还可以分解，而有的步骤可以合并或部分合并，因此实际执行的顺序有可能根据实际情况改变。
在一些相关技术中，基于RGB(红绿蓝)图像的手势识别与交互，仅限于二维屏幕。针对三维的现实空间，这种二维的交互在实际操作中不方便，不符合用户在日常活动中的真实交互习惯。
另外，在增强现实的交互场景中，往往仅考虑交互双方的位置，而忽略了他们之间的相对姿态，这就导致在交互操作时，可能出现不符合交互常识的场景，造成用户交互体验差的问题。例如，手背触碰到虚拟皮球，也能将虚拟皮球抓起。
鉴于此，需要一种新的对象交互处理方式。
图1示意性示出了根据本公开的示例性实施方式的对象交互系统的方框图。
需要说明的是，本公开示例性实施方式的对象交互系统可以配置在终端设备中。其中，终端设备可以例如为手机、平板、头戴式显示设备等，本公开对终端设备的种类不做特殊限制。
参考图1，对象交互系统10可以包括相机101、惯性传感器102、深度传感器103、处理器104和存储装置105。
相机101用于在现实场景下采集图像信息，在本公开实施例中，相机101采集到的图像为RGB图像；惯性传感器102又可被称为惯性测量单元(Inertial Measurement Unit，IMU)，用于采集终端设备的惯性信息；深度传感器103用于确定各个对象在现实场景下的深度信息；处理器104可以是一个处理器，或者是包括多个处理器的处理器集群；存储装置105用于存储一个或多个程序。
具体的，当一个或多个程序被处理器104执行时，使得处理器104实现：利用相机101在现实场景下采集到的像素信息以及惯性传感器102采集到的终端设备的惯性信息，确定终端设备在现实环境坐标系下的位姿；基于相机101采集到的像素信息以及深度传感器103采集到的深度信息，确定第一对象相对于终端设备的位姿；根据终端设备在现实环境坐标系下的位姿以及第一对象相对于终端设备的位姿，确定第一对象在现实环境坐标系下的位姿；确定现实环境坐标系下第二对象的位姿，根据第一对象在现实环境坐标系下的位姿以及现实环境坐标系下第二对象的位姿，控制第一对象与第二对象进行交互。
另外，在上面的描述中，第一对象是现实环境下真实存在的对象，可以是真实的物体。第二对象是利用计算机技术配置在现实环境下的虚拟对象，虚拟对象并未真实存在于现实世界。
图2示出了适于用来实现本发明实施例的电子设备的计算机系统的结构示意图。
需要说明的是，图2示出的电子设备的计算机系统200仅是一个示例，不应对本发明实施例的功能和使用范围带来任何限制。
如图2所示，计算机系统200包括中央处理单元(CPU)201，其可 以根据存储在只读存储器(ROM)202中的程序或者从存储部分208加载到随机访问存储器(RAM)203中的程序而执行各种适当的动作和处理。在RAM 203中，还存储有系统操作所需的各种程序和数据。CPU 201、ROM 202以及RAM 203通过总线204彼此相连。输入/输出(I/O)接口205也连接至总线204。
以下部件连接至I/O接口205：包括键盘、鼠标等的输入部分206；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分207；包括硬盘等的存储部分208；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分209。通信部分209经由诸如因特网的网络执行通信处理。驱动器210也根据需要连接至I/O接口205。可拆卸介质211，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器210上，以便于从其上读出的计算机程序根据需要被安装入存储部分208。
特别地，根据本发明的实施例，下文参考流程图描述的过程可以被实现为计算机软件程序。例如，本发明的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信部分209从网络上被下载和安装，和/或从可拆卸介质211被安装。在该计算机程序被中央处理单元(CPU)201执行时，执行本申请的系统中限定的各种功能。
需要说明的是，本发明所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本发明中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本发明中，计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。
附图中的流程图和框图，图示了按照本发明各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执 行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本发明实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现，所描述的单元也可以设置在处理器中。其中，这些单元的名称在某种情况下并不构成对该单元本身的限定。
作为另一方面，本申请还提供了一种计算机可读介质，该计算机可读介质可以是上述实施例中描述的电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被一个该电子设备执行时，使得该电子设备实现如下述实施例中所述的方法。
下面将参考图3对本公开示例性实施方式的对象交互方法进行描述。需要说明的是，对象交互方法的各个步骤均可以由终端设备执行，在这种情况下，下面所述的对象交互装置可以配置在终端设备中。
参考图3，所述对象交互方法方法可以包括以下步骤：
S30.确定终端设备在现实环境坐标系下的位姿。
在本公开的示例性实施方式中，一方面，可以采用相机拍摄现实场景，以确定出现实场景下图像的像素信息。在本公开的一个实施例中，可以采用单目摄像头拍摄现实场景，以确定现实场景中各真实物体的像素信息。其中，单目摄像头是依据小孔成像原理构造的摄像头，是单个彩色(或灰度)摄像头，其具有使用简单、准确性高、方便配置等优点。以终端设备是手机为例，可以采用手机的后置摄像头作为单目摄像头拍摄现实场景。
另一方面，可以采用惯性传感器实时监测终端设备的惯性信息，其中，惯性信息主要包括平移量和旋转量。
具体的，惯性传感器可以包含三个加速度计和三个陀螺，三个加速度计用于检测终端设备在三维坐标系的每个坐标轴上的加速度信号，进而可以计算出平移量；而三个陀螺，可以检测终端设备在三维坐标系的旋转量。另外，为了方便计算，可以将旋转量表示为旋转矩阵。
应当理解的是，可以将采集到图像的像素信息作为观测约束，将惯性传感器检测到的惯性信息作为动作约束。在这种情况下，确定终端设备在现实环境坐标系下的位姿可以被描述成：在观测约束与动作约束的联合约束下确定出的终端设备的位姿。其中，本公开所述的位姿包括位置信息和姿态信息。
下面将参考图4对本公开实施方式的确定终端设备在现实环境坐标系下位姿的过程进行说明。
在步骤S402中，利用相机采集视频图像，并从视频图像中提取特征点。在一些实例中，特征点可以是表征物体轮廓的点。具体的，可以将采集到的图像输入一特征点提取模型，以确定出图像中的特征点，其中，该特征点提取模型可以是训练后的卷积神经网络。针对该卷积神经网络，容易理解的是，可以丰富带有标记的样本图像的种类，以扩大该卷积神经网络的应用范围，增强处理过程的鲁棒性并提高识别特征点的准确度。
同时，可以将确定出的特征点与惯性传感器检测到的惯性信息相结合，建立初始坐标系，并建立初始的局部地图以及终端设备的初始位姿。
在步骤S404中，在连续的视频帧中，对特征点进行跟踪。
根据本公开的一些实施例，特征点跟踪可以采用光流追踪、直接法等基于视觉里程计原理的算法，本公开对此不做特殊限制。
在一个实施例中，特征点跟踪可以采用KLT跟踪算法(Kanade-Lucas-Tomasi Tracking Method)；在另一个实施例中，特征点跟踪可以采用基于ORB(Oriented FAST and Rotated BRIEF，快速特征点提取和描述)算法提取的ORB特征描述子进行特征点跟踪。
此外，例如，可以对特征点的邻域进行仿射变换，以得到该邻域在当前帧中的投影区域，计算上一帧中的邻域与当前帧中的投影区域之间的颜色偏移量，作为特征点的跟踪偏差。对于确定的多个特征点，可以选择跟踪偏差较小的多个特征点，以实现特征点的跟踪过程。
在步骤S406中，确定三维固定特征。
在本公开的示例性实施方式中，可以基于对特征点的跟踪，确定出现实环境中的三维固定特征。这些三维固定特征可以是现实环境中具有显著观测特征的角点、特征点。例如，Sift(Scale-invariant feature transform)特征点、哈里斯角点、ORB特征点等。
应当理解的是，如果在连续特征点的跟踪过程中，一些特征点的位置未发生变化，则可以认为这些特征点为本公开所述的三维固定特征。
在确定出三维固定特征后，可以利用这些三维固定特征形成的特征点云，构建现实场景的地图。
在步骤S408中，确定三维固定特征的位置变换信息。
在相机不断移动拍摄的过程中，终端设备可以确定出三维固定特征相对于相机的位置变换信息。
在对相机拍摄的视频图像进行上述分析的同时，在步骤S410中，终端设备可以利用惯性传感器实时监测惯性信息。
在步骤S412中，可以根据三维固定特征在视频帧中的位置变换信息以及终端设备的惯性信息，确定终端设备在现实环境坐标系下的位姿。
综上，可以看出，本公开确定终端设备位姿的过程可以包括：首先，利用有限的RGB图像和惯性信息，构建初始的三维特征点云并确定终端设备初始的位姿；接下来，根据持续采集到的视频帧，对地图和终端设备的位姿进行持续的观测约束，以及根据持续采集到的惯性信息，对地图和终端设备的位姿进行持续的运动约束；然后，在观测约束和运动约束的共同作用下，确定出终端设备的位姿。
S32.根据第一对象的像素信息和深度信息确定第一对象相对于终端设备的位姿；其中，第一对象为现实环境下的真实对象。
在本公开的示例性实施方式中，第一对象是指在现实空间中实际存在的真实对象，虽然下面以人的手部为例对第一对象进行说明，然而，第一对象还可以是人体的其他部位。另外，第一对象还可以是现实中其他物体，例如，木棍、球拍等。
首先，可以采用深度传感器确定第一对象相对于终端设备的深度信息。例如，可以采用TOF(Time Of Flight，飞行时间)传感器来确定深度信息。具体的，TOF传感器可以发出近红外光，遇到第一对象后反射，TOF传感器通过计算光线发射和反射时间差或相位差，来计算距第一对象的距离，以确定深度信息。
需要理解的是，深度信息与现实场景下采集到的像素信息对应。也 就是说，深度信息的确定与图像的像素信息的采集往往同时发生。对于每一帧图像而言，均对应存在相应的深度信息。
接下来，深度信息可以结合步骤S30中利用相机采集的图像像素信息，确定第一对象相对于终端设备的位姿。
在整个过程中，需要对第一对象进行识别。具体的，可以将图像输入一对象识别模型，类似的，该对象识别模块可以是训练后的机器学习模型。该模型可以对第一对象进行特征提取，以确定出第一对象的类别。以人手为例，该模型可以对人手进行语义分割，并提取手部关键点，当预测出人手的概率大于一预设阈值时，则可以确定第一对象为人的手部。
通过联合像素信息和深度信息，提高了确定第一对象相对于终端设备的位姿的鲁棒性。
此外，本公开的一些实施例考虑到在AR交互场景下，第一对象有一些区域不适合进行交互动作，否则将不符合日常生活的交互常识。在这种情况下，步骤S32的处理过程还可以包括：
首先，可以确定第一对象的可交互区域。其中，可交互区域可以表示实际进行交互动作的区域。例如，以手抓虚拟皮球、手提袋子等为例，第一对象为手，可交互区域可以为手掌平面。
针对确定手掌平面的方案，可以确定手部关键点，具体的，手部关键点可以是五根手指的指根对应的点。根据手部关键点可以确定出手掌平面，作为第一对象的可交互区域。
接下来，可以确定与在现实场景下采集到的像素信息对应的深度信息，并根据深度信息以及像素信息确定可交互区域相对于终端设备的位置信息和姿态信息。在这种情况下，可以将利用可交互区域相对于终端设备的位置信息和姿态信息表征第一对象相对于终端设备的位姿。
仍以手掌平面为可交互区域为例，可以将手掌平面的中心点的位置信息作为手掌平面相对于终端设备的位置信息。另外，可以将手掌平面的法线朝向信息作为手掌平面相对于终端设备的姿态信息，其中，法线朝向与手掌平面的方向垂直。
S34.根据终端设备在现实环境坐标系下的位姿以及第一对象相对于终端设备的位姿，确定第一对象在现实环境坐标系下的位姿。
将位姿以坐标形式表示，可以表示为(R，T)，其中，R表征姿态信息，T表征位置信息。
在这种情况下，可以将终端设备在现实环境坐标系下的位姿表示为(R1，T1)，可以将第一对象相对于终端设备的位姿表示为(R2，T2)，则第一对象在现实环境坐标系下的位姿为(R2*R1，T2+R2*T1)。
S36.确定现实环境坐标系下第二对象的位姿；其中，第二对象为配置在现实环境下的虚拟对象。
应当理解的是，第二对象在现实环境中是不存在的，当利用计算机技术在现实环境下放置一个虚拟的第二对象时，需要指定该第二对象的位姿，该位姿对应的坐标可以为(R0，T0)，另外，该位姿坐标亦可被称为锚点坐标。
S38.根据第一对象在现实环境坐标系下的位姿以及现实环境坐标系下第二对象的位姿，控制第一对象与第二对象进行交互。
在本公开的示例性实施方式中，可以综合考虑第一对象与第二对象之间的位置关系以及姿态关系，来控制他们进行交互。
首先，可以根据现实环境坐标系下第一对象的位置信息和第二对象的位置信息，确定第一对象与第二对象的距离。
接下来，判断第一对象与第二对象的距离是否小于预设距离阈值，其中，该预设距离阈值可以根据AR场景交互的复杂程度由开发人员自行设定，例如，可以将预设距离阈值设定为2cm。
如果第一对象与第二对象距离小于预设距离阈值，则判断第一对象的姿态是否与第二对象的姿态匹配，其中，匹配关系可以根据交互场景预先配置，例如，手抓虚拟皮球时，匹配关系为手掌平面的法线朝向指向球心区域的方向。
如果确定出第一对象的姿态与第二对象姿态匹配，则触发执行第一对象与第二对象的交互动作。其中，交互动作可以预先由开发人员进行设置，包括但不限于第二对象改变颜色、改变形状、执行预设动作等。
然而，本公开对验证位置关系与姿态关系的顺序不进行特殊限制，也就是说，可以先对姿态关系是否匹配进行验证，验证成功后，再判断第一对象与第二对象的距离是否满足距离要求。
应当理解的是，上述距离要求不限于距离较近时执行交互动作，当第一对象与第二对象的距离变大时，也可能触发交互动作。
下面参考图5以手部为例对本公开实施方式的对象交互过程进行说明。
在步骤S502中，确定终端设备在现实环境坐标系下的位姿；在步骤S504中，对采集到的视频图像中手部进行识别与跟踪，确定出手部关键点，并根据手部关键点确定出手掌平面；一方面，在步骤S506中，可以确定手掌平面中心点的位置信息；在步骤S508中，根据手掌平面中心点的位置信息，判断手掌平面中心点与第二对象(虚拟对象)的距离是否小于预设距离。如果小于预设距离，则跳转至步骤S514，如果不小于预设距离，则跳转至步骤S506。
另一方面，在步骤S510中，确定手掌平面法线朝向；在步骤S512中，判断手掌平面法线朝向与第二对象的姿态是否匹配，如果匹配，则跳转至步骤S514，如果不匹配，则跳转至步骤S510。
在步骤S514中，在AR场景下，可以触发执行手掌与第二对象的交互操作。例如，手抓虚拟皮球、手提虚拟篮筐、手握虚拟门把手等。
下面以第一对象为真实人手且第二对象为虚拟皮球为例对本公开的对象交互方法进行说明。
参考图6，可以通过计算机技术在现实场景下的桌面60上放置一个虚拟皮球62，此时，用户可以通过终端设备的屏幕，查看到在真实的桌面上有一个虚拟皮球。
在这种情况下，用户可以通过真实的人手62从桌面60上抓起虚拟皮球61。具体过程可以包括：首先，终端设备可以利用摄像头获取人手62的像素信息并利用深度传感器获取人手62的深度信息，以确定人手62相对于终端设备的位姿；接下来，终端设备可以根据自身在现实坐标系下的位姿以及人手62相对于终端设备的位姿，确定出人手62在现实环境坐标系下的位姿。
在实时确定人手62在现实环境坐标系下的位姿的过程中，终端设备可以实时确定人手62与虚拟皮球61之间的距离以及人手62的手掌平面的法线方向与虚拟皮球61的相对关系，具体的，该手掌平面的法线方向 为垂直于手掌平面且自手掌掌心向外的方向。
例如，预定的交互规则为：当人手距虚拟皮球的距离小于3cm且手掌平面的法线朝向指向球心区域的方向时，人手抓出虚拟皮球。在应用这种规则的场景下，如果当前人手62与虚拟皮球61的相对关系满足该预定的交互规则，则人手62可以将虚拟皮球61抓起。在人手62从桌面60上抓起虚拟皮球61后，可以保持姿势移动，以改变虚拟皮球的位置或运动方式。图6示出了人手62将虚拟皮球61抓起并移动一段距离的示意图。
根据本公开的一个实施例，在确定出人手的位姿的前提下，可以对手势进行识别，并根据手势识别结果确定出对应的交互动作。
例如，在第二对象为虚拟人物(例如，虚拟卡通人物)的情况下，如果人手与该虚拟人物的距离大于0.5m且小于2m，并且手掌法线朝向为指向虚拟人物方向，则当识别出手势为左右摆动时，则触发虚拟人物执行打招呼的指令，在这种情况下，虚拟人物也可以挥动手臂，实现相互打招呼的交互动作。
应当注意，尽管在附图中以特定顺序描述了本公开中方法的各个步骤，但是，这并非要求或者暗示必须按照该特定顺序来执行这些步骤，或是必须执行全部所示的步骤才能实现期望的结果。附加的或备选的，可以省略某些步骤，将多个步骤合并为一个步骤执行，以及/或者将一个步骤分解为多个步骤执行等。
进一步的，本示例实施方式中还提供了一种对象交互装置，应用于移动设备。
图7示意性示出了本公开的示例性实施方式的对象交互装置的方框图。参考图7，根据本公开的示例性实施方式的对象交互装置7可以包括终端位姿确定模块71、第一对象位姿确定模块73、坐标转换模块75、第二对象位姿确定模块77和对象交互模块79。
具体的，终端位姿确定模块71可以用于确定终端设备在现实环境坐标系下的位姿；第一对象位姿确定模块73可以用于根据第一对象的像素信息和深度信息确定第一对象相对于终端设备的位姿；其中，第一对象为现实环境下的真实对象；坐标转换模块75可以用于根据终端设备在现实环境坐标系下的位姿以及第一对象相对于终端设备的位姿，确定第一对象在现实环境坐标系下的位姿；第二对象位姿确定模块77可以用于确定现实环境坐标系下第二对象的位姿；其中，第二对象为配置在现实环境下的虚拟对象；对象交互模块79可以用于根据第一对象在现实环境坐标系下的位姿以及现实环境坐标系下第二对象的位姿，控制第一对象与第二对象进行交互。
利用本公开示例性实施方式的对象交互装置，一方面，在确定第一对象相对于终端设备的位姿时，结合了深度信息，由此，可以更方便的使对象交互延伸至三维空间尺度，提高了对象交互过程的鲁棒性；另一方面，通过将第一对象与第二对象对齐到同一坐标系下，便于分析，符合真实的物理交互体验；再一方面，本公开的对象交互方法考虑的是位姿，包括位置和姿态，相比于一些相关技术，姿态的引入，避免了可能出现不符合交互常识的场景，使得第一对象与第二对象的交互效果更加真实，提高了用户利用增强现实技术实现虚实互动的交互体验。
根据本公开的示例性实施例，参考图8，终端位姿确定模块71可以包括特征点提取单元801、固定特征确定单元803和终端姿态确定单元 805。
具体的，特征点提取单元801可以用于从在现实场景下采集到的视频图像中提取特征点；固定特征确定单元803可以用于在连续的视频帧中，对特征点进行跟踪，确定现实场景下的三维固定特征；终端姿态确定单元805可以用于根据三维固定特征在视频帧中的位置变换信息以及终端设备的惯性信息，确定终端设备在现实环境坐标系下的位姿。
根据本公开的示例性实施例，参考图9，终端位姿确定模块71还可以包括地图构建单元901。
具体的，地图构建单元901可以用于在确定现实场景下的三维固定特征后，利用三维固定特征形成的特征点云，构建现实场景的地图。
根据本公开的示例性实施例，第一对象位姿确定模块73可以被配置为执行：确定第一对象的可交互区域；确定可交互区域的像素信息和深度信息；根据可交互区域的像素信息和深度信息确定可交互区域相对于终端设备的位置信息和姿态信息；其中，利用可交互区域相对于终端设备的位置信息和姿态信息表征第一对象相对于终端设备的位姿。
根据本公开的示例性实施例，第一对象为手部；在这种情况下，对象位姿确定模块73确定第一对象的可交互区域的过程可以包括：确定手部关键点；将基于手部关键点确定出的手掌平面确定为第一对象的可交互区域。
根据本公开的示例性实施例，第一对象位姿确定模块73可以被配置为执行：根据手掌平面的像素信息和深度信息确定手掌平面的中心点的位置信息，作为手掌平面相对于终端设备的位置信息；以及根据手掌平面的像素信息和深度信息确定手掌平面的法线朝向信息，作为手掌平面相对于终端设备的姿态信息。
根据本公开的示例性实施例，对象交互模块79可以被配置为执行：根据现实环境坐标系下第一对象的位置信息和第二对象的位置信息，确定第一对象与第二对象的距离；如果第一对象与第二对象的距离小于预设距离阈值且第一对象与第二对象的姿态匹配，则触发执行第一对象与第二对象的交互动作。
由于本发明实施方式的程序运行性能分析装置的各个功能模块与上述方法发明实施方式中相同，因此在此不再赘述。
通过以上的实施方式的描述，本领域的技术人员易于理解，这里描述的示例实施方式可以通过软件实现，也可以通过软件结合必要的硬件的方式来实现。因此，根据本公开实施方式的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中或网络上，包括若干指令以使得一台计算设备(可以是个人计算机、服务器、终端装置、或者网络设备等)执行根据本公开实施方式的方法。
此外，上述附图仅是根据本发明示例性实施例的方法所包括的处理的示意性说明，而不是限制目的。易于理解，上述附图所示的处理并不表明或限制这些处理的时间顺序。另外，也易于理解，这些处理可以是例如在多个模块中同步或异步执行的。
应当注意，尽管在上文详细描述中提及了用于动作执行的设备的若干模块或者单元，但是这种划分并非强制性的。实际上，根据本公开的实施方式，上文描述的两个或更多模块或者单元的特征和功能可以在一个模 块或者单元中具体化。反之，上文描述的一个模块或者单元的特征和功能可以进一步划分为由多个模块或者单元来具体化。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其他实施例。本申请旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由权利要求指出。
应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围仅由所附的权利要求来限。

Claims (11)

  1. 一种对象交互方法，应用于终端设备，其特征在于，包括：
    确定终端设备在现实环境坐标系下的位姿；
    根据第一对象的像素信息和深度信息确定所述第一对象相对于所述终端设备的位姿；其中，所述第一对象为现实环境下的真实对象；
    根据所述终端设备在所述现实环境坐标系下的位姿以及所述第一对象相对于所述终端设备的位姿，确定所述第一对象在所述现实环境坐标系下的位姿；
    确定所述现实环境坐标系下第二对象的位姿；其中，所述第二对象为配置在现实环境下的虚拟对象；
    根据所述第一对象在所述现实环境坐标系下的位姿以及所述现实环境坐标系下第二对象的位姿，控制所述第一对象与所述第二对象进行交互。
  2. 根据权利要求1所述的对象交互方法，其特征在于，确定终端设备在现实环境坐标系下的位姿包括：
    从在现实场景下采集到的视频图像中提取特征点；
    在连续的视频帧中，对所述特征点进行跟踪，确定所述现实场景下的三维固定特征；
    根据所述三维固定特征在所述视频帧中的位置变换信息以及终端设备的惯性信息，确定所述终端设备在现实环境坐标系下的位姿。
  3. 根据权利要求2所述的对象交互方法，其特征在于，在确定所述现实场景下的三维固定特征后，所述对象交互方法还包括：
    利用所述三维固定特征形成的特征点云，构建所述现实场景的地图。
  4. 根据权利要求1所述的对象交互方法，其特征在于，根据第一对象的像素信息和深度信息确定所述第一对象相对于所述终端设备的位姿包括：
    确定第一对象的可交互区域；
    确定所述可交互区域的像素信息和深度信息；
    根据所述可交互区域的像素信息和深度信息确定所述可交互区域相对于所述终端设备的位置信息和姿态信息；
    其中，利用所述可交互区域相对于所述终端设备的位置信息和姿态信息表征所述第一对象相对于所述终端设备的位姿。
  5. 根据权利要求4所述的对象交互方法，其特征在于，所述第一对象为手部；其中，确定第一对象的可交互区域包括：
    确定手部关键点；
    将基于所述手部关键点确定出的手掌平面确定为第一对象的可交互区域。
  6. 根据权利要求5所述的对象交互方法，其特征在于，根据所述可交互区域的像素信息和深度信息确定所述可交互区域相对于所述终端设备的位置信息和姿态信息包括：
    根据所述手掌平面的像素信息和深度信息确定所述手掌平面的中心点的位置信息，作为所述手掌平面相对于所述终端设备的位置信息；以及
    根据所述手掌平面的像素信息和深度信息确定所述手掌平面的法线朝向信息，作为所述手掌平面相对于所述终端设备的姿态信息。
  7. 根据权利要求1或6所述的对象交互方法，其特征在于，根据所 述第一对象在所述现实环境坐标系下的位姿以及所述现实环境坐标系下第二对象的位姿，控制所述第一对象与所述第二对象进行交互包括：
    根据所述现实环境坐标系下所述第一对象的位置信息和所述第二对象的位置信息，确定所述第一对象与所述第二对象的距离；
    如果所述第一对象与所述第二对象的距离小于预设距离阈值且所述第一对象与所述第二对象的姿态匹配，则触发执行所述第一对象与所述第二对象的交互动作。
  8. 一种对象交互装置，其特征在于，包括：
    终端位姿确定模块，用于确定终端设备在现实环境坐标系下的位姿；
    第一对象位姿确定模块，用于根据第一对象的像素信息和深度信息确定所述第一对象相对于所述终端设备的位姿；其中，所述第一对象为现实环境下的真实对象；
    坐标转换模块，用于根据所述终端设备在所述现实环境坐标系下的位姿以及所述第一对象相对于所述终端设备的位姿，确定所述第一对象在所述现实环境坐标系下的位姿；
    第二对象位姿确定模块，用于确定所述现实环境坐标系下第二对象的位姿；其中，所述第二对象为配置在现实环境下的虚拟对象；
    对象交互模块，用于根据所述第一对象在所述现实环境坐标系下的位姿以及所述现实环境坐标系下第二对象的位姿，控制所述第一对象与所述第二对象进行交互。
  9. 一种对象交互系统，其特征在于，包括：
    相机，用于在现实场景下采集图像信息；
    惯性传感器，用于采集终端设备的惯性信息；
    深度传感器，用于确定对象在所述现实场景下的深度信息；
    一个或多个处理器；
    存储装置，用于存储一个或多个程序；
    其中，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述一个或多个处理器实现如权利要求1至7中任一项所述的对象交互方法。
  10. 一种计算机可读介质，其上存储有计算机程序，其特征在于，所述程序被处理器执行时实现如权利要求1至7中任一项所述的对象交互方法。
  11. 一种电子设备，其特征在于，包括：
    一个或多个处理器；
    存储装置，用于存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行时，使得所述一个或多个处理器实现如权利要求1至7中任一项所述的对象交互方法。
PCT/CN2020/089641 2019-05-14 2020-05-11 对象交互方法及装置、系统、计算机可读介质和电子设备 Ceased WO2020228682A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20806291.9A EP3968131B1 (en) 2019-05-14 2020-05-11 Object interaction method, computer-readable medium, and electronic device
US17/525,772 US12067158B2 (en) 2019-05-14 2021-11-12 Object interaction method and system, and computer-readable medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910399099.XA CN111949112A (zh) 2019-05-14 2019-05-14 对象交互方法及装置、系统、计算机可读介质和电子设备
CN201910399099.X 2019-05-14

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/525,772 Continuation US12067158B2 (en) 2019-05-14 2021-11-12 Object interaction method and system, and computer-readable medium

Publications (1)

Publication Number Publication Date
WO2020228682A1 true WO2020228682A1 (zh) 2020-11-19

Family

ID=73288957

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/089641 Ceased WO2020228682A1 (zh) 2019-05-14 2020-05-11 对象交互方法及装置、系统、计算机可读介质和电子设备

Country Status (4)

Country Link
US (1) US12067158B2 (zh)
EP (1) EP3968131B1 (zh)
CN (1) CN111949112A (zh)
WO (1) WO2020228682A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113905220A (zh) * 2021-08-30 2022-01-07 咪咕动漫有限公司 基于赛事场景的沙盘投影方法、装置、设备及存储介质

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113632035B (zh) * 2019-02-25 2024-07-12 德卡产品有限公司 用于表面特征检测和穿越的系统和方法
CN114693890B (zh) * 2020-12-31 2024-09-10 华为技术有限公司 一种增强现实交互方法及电子设备
CN113459098B (zh) * 2021-07-01 2022-06-03 上海宜硕网络科技有限公司 一种机器人关门控制方法、装置和电子设备
CN113386138B (zh) * 2021-07-01 2022-06-03 上海宜硕网络科技有限公司 一种机器人开门控制方法、装置和电子设备
US12639885B2 (en) * 2022-07-27 2026-05-26 Apple Inc. Stereo depth markers
CN117631817A (zh) * 2022-08-18 2024-03-01 北京字跳网络技术有限公司 测量方法、装置、电子设备以及存储介质
CN115620336B (zh) * 2022-09-30 2026-08-11 清华大学 面向手物交互场景的位姿估计方法和装置
CN115409871B (zh) * 2022-10-31 2023-03-24 浙江中测新图地理信息技术有限公司 一种基于位置智能的三维场景虚实交互方法及装置
CN115861581B (zh) * 2023-02-08 2023-05-05 成都艺馨达科技有限公司 一种基于混合现实的移动互联云服务方法及系统
CN116258764A (zh) * 2023-02-24 2023-06-13 联想(北京)有限公司 一种互动状态更新方法及装置
CN117032456A (zh) * 2023-08-10 2023-11-10 北京字跳网络技术有限公司 用于扩展现实环境的位姿校准方法及装置
CN121165927A (zh) * 2024-06-18 2025-12-19 中兴通讯股份有限公司 设备控制方法、设备、存储介质及程序产品
WO2026106540A1 (en) * 2024-11-13 2026-05-21 Dyna.Ai Technology Pte. Ltd. User interaction method and system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105528082A (zh) * 2016-01-08 2016-04-27 北京暴风魔镜科技有限公司 三维空间及手势识别追踪交互方法、装置和系统
US20170154471A1 (en) * 2014-06-26 2017-06-01 Korea Advanced Institute Of Science And Technology Apparatus and method for providing augmented reality interaction service
CN109474816A (zh) * 2018-12-28 2019-03-15 上海北冕信息科技有限公司 增强现实的虚实融合装置及其虚实融合方法、设备、介质
CN109712172A (zh) * 2018-12-28 2019-05-03 哈尔滨工业大学 一种初始位姿测量结合目标追踪的位姿测量方法
CN109725733A (zh) * 2019-01-25 2019-05-07 中国人民解放军国防科技大学 一种基于增强现实的人机交互方法及人机交互设备

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014009561A2 (en) * 2012-07-13 2014-01-16 Softkinetic Software Method and system for human-to-computer gesture based simultaneous interactions using singular points of interest on a hand
US9367136B2 (en) * 2013-04-12 2016-06-14 Microsoft Technology Licensing, Llc Holographic object feedback
US9996797B1 (en) * 2013-10-31 2018-06-12 Leap Motion, Inc. Interactions with virtual objects for machine control
CN104217350B (zh) * 2014-06-17 2017-03-22 北京京东尚科信息技术有限公司 实现虚拟试戴的方法和装置
US9696795B2 (en) * 2015-02-13 2017-07-04 Leap Motion, Inc. Systems and methods of creating a realistic grab experience in virtual reality/augmented reality environments
JP7175281B2 (ja) * 2017-03-28 2022-11-18 マジック リープ， インコーポレイテッド ユーザ走査仮想オブジェクトに関係している空間化されたオーディオを用いる拡張現実システム
CN109683700A (zh) * 2017-10-18 2019-04-26 深圳市掌网科技股份有限公司 基于视觉感知的人机交互实现方法和装置
CN108492316A (zh) 2018-02-13 2018-09-04 视辰信息科技(上海)有限公司 一种终端的定位方法和装置
CN110555883B (zh) 2018-04-27 2022-07-22 腾讯科技（深圳）有限公司 相机姿态追踪过程的重定位方法、装置及存储介质
CN108681402A (zh) * 2018-05-16 2018-10-19 Oppo广东移动通信有限公司 识别交互方法、装置、存储介质及终端设备
US11151798B1 (en) * 2019-11-13 2021-10-19 Apple Inc. Depth rate up-conversion
JP7681609B2 (ja) * 2020-02-14 2025-05-22 マジック リープ， インコーポレイテッド ３ｄオブジェクト注釈
EP4158908A4 (en) * 2020-05-29 2023-11-29 Magic Leap, Inc. SURFACE COLLISIONS

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170154471A1 (en) * 2014-06-26 2017-06-01 Korea Advanced Institute Of Science And Technology Apparatus and method for providing augmented reality interaction service
CN105528082A (zh) * 2016-01-08 2016-04-27 北京暴风魔镜科技有限公司 三维空间及手势识别追踪交互方法、装置和系统
CN109474816A (zh) * 2018-12-28 2019-03-15 上海北冕信息科技有限公司 增强现实的虚实融合装置及其虚实融合方法、设备、介质
CN109712172A (zh) * 2018-12-28 2019-05-03 哈尔滨工业大学 一种初始位姿测量结合目标追踪的位姿测量方法
CN109725733A (zh) * 2019-01-25 2019-05-07 中国人民解放军国防科技大学 一种基于增强现实的人机交互方法及人机交互设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3968131A4 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113905220A (zh) * 2021-08-30 2022-01-07 咪咕动漫有限公司 基于赛事场景的沙盘投影方法、装置、设备及存储介质
CN113905220B (zh) * 2021-08-30 2023-11-21 咪咕动漫有限公司 基于赛事场景的沙盘投影方法、装置、设备及存储介质

Also Published As

Publication number Publication date
US20220066569A1 (en) 2022-03-03
EP3968131B1 (en) 2025-12-24
CN111949112A (zh) 2020-11-17
US12067158B2 (en) 2024-08-20
EP3968131A4 (en) 2022-06-08
EP3968131A1 (en) 2022-03-16

Similar Documents

Publication Publication Date Title
US12067158B2 (en) Object interaction method and system, and computer-readable medium
US11762475B2 (en) AR scenario-based gesture interaction method, storage medium, and communication terminal
CN111556278B (zh) 一种视频处理的方法、视频展示的方法、装置及存储介质
CN113034652B (zh) 虚拟形象驱动方法、装置、设备及存储介质
CN110866977B (zh) 增强现实处理方法及装置、系统、存储介质和电子设备
CN104081317B (zh) 信息处理设备和信息处理方法
US20180088663A1 (en) Method and system for gesture-based interactions
US20140009384A1 (en) Methods and systems for determining location of handheld device within 3d environment
CN110942479B (zh) 虚拟对象控制方法、存储介质及电子设备
CN115994944A (zh) 三维关键点预测方法、训练方法及相关设备
CN114972958B (zh) 关键点检测方法、神经网络的训练方法、装置和设备
US11561651B2 (en) Virtual paintbrush implementing method and apparatus, and computer readable storage medium
CN116686006A (zh) 基于可变形模型的三维扫描配准
CN116848556A (zh) 使用多视图细化来增强三维模型
CN116311519A (zh) 动作识别方法、模型训练方法及装置
US20220375092A1 (en) Target object controlling method, apparatus, electronic device, and storage medium
WO2025001819A9 (zh) 坐标系对齐方法、设备、存储介质及程序产品
CN110069125B (zh) 虚拟对象的控制方法和装置
CN110794959A (zh) 一种基于图像识别的手势交互ar投影方法及装置
CN113158912B (zh) 手势识别方法及装置、存储介质及电子设备
CN115937430A (zh) 用于展示虚拟对象的方法、装置、设备及介质
WO2023025181A1 (zh) 图像识别方法、装置和电子设备
CN109981989A (zh) 渲染图像的方法、装置、电子设备和计算机可读存储介质
CN112991179B (zh) 用于输出信息的方法、装置、设备以及存储介质
CN110941974B (zh) 虚拟对象的控制方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20806291

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020806291

Country of ref document: EP

Effective date: 20211210

WWG Wipo information: grant in national office

Ref document number: 2020806291

Country of ref document: EP