WO2021032097A1 - 一种隔空手势的交互方法及电子设备 - Google Patents

一种隔空手势的交互方法及电子设备 Download PDF

Info

Publication number
WO2021032097A1
WO2021032097A1 PCT/CN2020/109893 CN2020109893W WO2021032097A1 WO 2021032097 A1 WO2021032097 A1 WO 2021032097A1 CN 2020109893 W CN2020109893 W CN 2020109893W WO 2021032097 A1 WO2021032097 A1 WO 2021032097A1
Authority
WO
WIPO (PCT)
Prior art keywords
gesture
electronic device
user
preset
air
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/109893
Other languages
English (en)
French (fr)
Inventor
吴义镇
勾军委
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to US17/636,529 priority Critical patent/US12001612B2/en
Priority to EP20855678.7A priority patent/EP4002066A4/en
Publication of WO2021032097A1 publication Critical patent/WO2021032097A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16—Constructional details or arrangements
    • G06F1/1613—Constructional details or arrangements for portable computers
    • G06F1/1633—Constructional details or arrangements of portable computers not specific to the type of enclosures covered by groups G06F1/1615 - G06F1/1626
    • G06F1/1684—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
    • G06F1/1686—Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675 the I/O peripheral being an integrated camera
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04812—Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04817—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance using icons
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0483—Interaction with page-structured environments, e.g. book metaphor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04847—Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/0485—Scrolling or panning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/0485—Scrolling or panning
    • G06F3/04855—Interaction with scrollbars
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048—Indexing scheme relating to G06F3/048
    • G06F2203/04806—Zoom, i.e. interaction techniques or interactors for controlling the zooming operation

Definitions

  • the present application relates to the field of terminal technology, and in particular to an interactive method and electronic device for space gestures.
  • some mobile phones can recognize gesture actions (ie, air gestures) performed by users in three-dimensional space through devices such as cameras or ultrasonic sensors, so as to realize air-space operations on some functions in the mobile phone.
  • gesture actions ie, air gestures
  • the mobile phone can recognize the air gesture performed by the user through the image collected by the camera. For example, if it is detected that the user performs the first space gesture of waving to the left, the mobile phone can perform the first operation of "flipping to the previous page" in the e-book APP in response to the first space gesture. If it is detected that the user performs the second air gesture of waving to the right, the mobile phone can respond to the second air gesture to perform a second operation of "flipping to the next page" in the e-book APP.
  • the user can perceive that the mobile phone has successfully received the air gesture input by the user.
  • the user cannot perceive whether the mobile phone detects the air gesture input by himself. As a result, the user cannot determine whether the air gesture being input is correct.
  • the use experience is low and the error operation rate is high.
  • the present application provides a space gesture interaction method and an electronic device, which can improve the user experience when performing space interaction with the electronic device and reduce the error operation rate of the space gesture.
  • the present application provides an interactive method for an air gesture, including: an electronic device displays a first display interface of a first application; at the same time, the electronic device can detect a user input of the first air gesture through a camera; A space gesture, the electronic device can move the indicator icon in the first display interface along the movement track of the first space gesture in the preset direction; at the same time, in response to the first space gesture, the electronic device can display the The second display interface.
  • the electronic device can guide the user to complete the space gesture through the movement of the indicator icon in the preset direction; on the other hand, the user can Through the movement of the indicator icon, the execution progress of the current air gesture is sensed in real time, thereby improving the user experience when interacting with the electronic device in the air and reducing the error operation rate of the air gesture.
  • the space gesture for example, the first space gesture mentioned above
  • the above-mentioned preset direction may be the x-axis direction of the rectangular coordinate system, and the x-axis direction is the horizontal direction when the user normally uses the electronic device; at this time, the electronic device is in the first space gesture
  • the movement track in the preset direction to move the indication icon in the first display interface specifically includes: the electronic device determines the first movement distance of the first air-space gesture in the x-axis direction; further, the electronic device can be in the first movement distance according to the first movement distance.
  • the above-mentioned preset direction may be the y-axis direction of the rectangular coordinate system.
  • the electronic device moves the indication in the first display interface along the movement track of the first space gesture in the preset direction.
  • the icon specifically includes: the electronic device determines the second movement distance of the first air gesture in the y-axis direction; further, the electronic device can move the indicator icon along the y-axis direction in the first display interface according to the second movement distance.
  • the electronic device may move the indicator icon along the y-axis direction according to the scale factor k according to the second moving distance, and prompt the user of the execution progress of the air gesture in the y-axis direction by moving the indicator icon.
  • the method before the electronic device detects the first gesture input by the user through the camera through the camera, the method further includes: the electronic device detects that the user inputs a first preset gesture, and the first preset gesture is used to enter the void Gesture operation mode; in response to the first preset gesture, the electronic device displays the above-mentioned indication icon in the preset position of the first display interface, so that the user can learn that the electronic device has entered the air gesture operation mode, thereby reminding the user to start inputting the corresponding Gesture in the air.
  • the electronic device displaying the indicator icon at the preset position of the first display interface includes: the electronic device displays the indicator icon along the top or bottom of the first display interface; or; the electronic device displays the indicator icon along the first display interface.
  • the indicator icon is displayed on the side of the interface.
  • the above-mentioned first preset gesture may be a preset hovering gesture; wherein the first preset gesture input by the user is detected by the electronic device, including: an image collected by the electronic device through the camera The hovering gesture input by the user is detected.
  • the electronic device can also detect the operation intention of the user to enter the operation mode of the air gesture by means of voice or the like.
  • the electronic device detects the user's input of the first space gesture through the camera, including: the electronic device uses the hovering position of the above hovering gesture as the starting position, and detects the user input through the image collected by the camera The first air gesture.
  • the electronic device displaying the second display interface of the first application includes: the electronic device reports the sliding event to the first application according to the movement distance of the first air gesture in the preset direction; the electronic device The first application in displays a second display interface corresponding to the sliding event.
  • the above method further includes: when the first air gesture ends, the electronic device restores the indication icon to the preset position, thereby prompting the user that the first air gesture entered this time has ended , The electronic device has completed the response to the first air gesture.
  • the above method further includes: after the first space gesture ends, the electronic device can detect the user input the second space gesture through the camera; in response to the second space gesture, the electronic device can follow The movement track of the second air gesture in the preset direction moves the indication icon; at the same time, in response to the second air gesture, the electronic device may display the third display interface of the first application.
  • the electronic device detects the user’s input of the second space gesture through the camera, including: when the first space gesture ends, the electronic device detects the first space gesture through the image collected by the camera. Backhaul operation; in order to avoid interference from the backhaul operation of the first gap gesture when the electronic device recognizes the second gap gesture, the electronic device can use the position where the backhaul operation ends as the starting position, and detect user input through the image collected by the camera The second air gesture.
  • the above method further includes: the electronic device detects that the user inputs a second preset gesture, the second preset gesture is used to exit the operation mode of the air gesture; in response to the second preset gesture, the electronic The device hides the displayed indicator icon, thereby prompting the user to exit the air gesture operation mode.
  • the above-mentioned indicator icon may specifically be a progress bar, a slider, a line segment or an arc; wherein, the electronic device moves in the first display interface along the movement track of the first air-space gesture in the preset direction.
  • the indication icon of includes: the electronic device slides, extends, or enlarges the indication icon according to the movement track of the first air gesture in the preset direction.
  • this application provides an electronic device, including: a touch screen, a camera, one or more processors, a memory, and one or more computer programs; wherein the processor is coupled with the touch screen, the camera and the memory, and one or more of the above A plurality of computer programs are stored in the memory, and when the electronic device is running, the processor executes one or more computer programs stored in the memory, so that the electronic device executes any one of the aforementioned space gesture interaction methods.
  • the present application provides a computer storage medium, including computer instructions, which when the computer instructions are executed on an electronic device, cause the electronic device to execute the interactive method of air gestures as described in any one of the first aspect.
  • the present application provides a computer program product that, when the computer program product is run on an electronic device, causes the electronic device to execute the interactive method of space gestures as described in any one of the first aspect.
  • the electronic equipment described in the second aspect, the computer storage medium described in the third aspect, and the computer program product described in the fourth aspect provided above are all used to execute the corresponding methods provided above.
  • the beneficial effects that can be achieved please refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.
  • FIG. 1 is a first structural diagram of an electronic device according to an embodiment of the application
  • FIG. 2 is a schematic diagram of a photographing principle of a camera provided by an embodiment of the application.
  • FIG. 3 is a schematic diagram 1 of an application scenario of an air gesture interaction method provided by an embodiment of the application;
  • FIG. 4 is a schematic structural diagram of an operating system in an electronic device provided by an embodiment of the application.
  • FIG. 5 is a second schematic diagram of an application scenario of an air gesture interaction method provided by an embodiment of the application.
  • FIG. 6 is a schematic flowchart of a method for interacting with an air gesture according to an embodiment of the application
  • FIG. 7 is a third schematic diagram of an application scenario of an air gesture interaction method provided by an embodiment of the application.
  • FIG. 8 is a fourth schematic diagram of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • FIG. 9 is a schematic diagram 5 of an application scenario of an air gesture interaction method provided by an embodiment of the application.
  • FIG. 10 is a sixth schematic diagram of an application scenario of an air gesture interaction method provided by an embodiment of the application.
  • FIG. 11 is a schematic diagram 7 of an application scenario of an air gesture interaction method provided by an embodiment of the application.
  • FIG. 12 is an eighth schematic diagram of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • FIG. 13 is a schematic diagram 9 of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • FIG. 14 is a tenth schematic diagram of an application scenario of an air gesture interaction method provided by an embodiment of the application.
  • 15 is a schematic diagram eleventh of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • 16 is a schematic diagram 12 of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • FIG. 17 is a schematic diagram 13 of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • FIG. 18 is a fourteenth schematic diagram of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • FIG. 19 is a schematic diagram 15 of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • 20 is a sixteenth schematic diagram of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • 21 is a schematic diagram 17 of an application scenario of an air gesture interaction method provided by an embodiment of this application.
  • FIG. 22 is a second structural diagram of an electronic device provided by an embodiment of this application.
  • the interactive method of air gestures provided in the embodiments of this application can be applied to mobile phones, tablet computers, notebook computers, ultra-mobile personal computers (UMPC), handheld computers, netbooks, and personal digital computers.
  • Electronic devices such as personal digital assistants (PDAs), wearable electronic devices, in-vehicle devices, virtual reality devices, etc., are not limited in the embodiments of the present application.
  • FIG. 1 shows a schematic structural diagram of an electronic device 100.
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2.
  • Mobile communication module 150 wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, sensor module 180, camera 193, display screen 194 and so on.
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100.
  • the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU), etc.
  • AP application processor
  • modem processor modem processor
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • NPU neural-network processing unit
  • the different processing units may be independent devices or integrated in one or more processors.
  • a memory may also be provided in the processor 110 to store instructions and data.
  • the memory in the processor 110 is a cache memory.
  • the memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
  • the charging management module 140 is used to receive charging input from the charger.
  • the charger can be a wireless charger or a wired charger.
  • the power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.
  • the power management module 141 can receive input from the battery 142 and/or the charging management module 140, and supply power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160.
  • the power management module 141 can be used to monitor performance parameters such as battery capacity, battery cycle times, battery charging voltage, battery discharging voltage, and battery health status (such as leakage, impedance).
  • the wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.
  • the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals.
  • the mobile communication module 150 can provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the electronic device 100.
  • the mobile communication module 150 may include one or more filters, switches, power amplifiers, low noise amplifiers (LNA), etc.
  • the mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering and amplifying the received electromagnetic waves, and then transmitting them to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation via the antenna 1.
  • at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110.
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
  • the wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), and global navigation satellites.
  • WLAN wireless local area networks
  • BT Bluetooth
  • GNSS global navigation satellite system
  • FM frequency modulation
  • NFC near field communication technology
  • infrared technology infrared, IR
  • the wireless communication module 160 may be one or more devices integrating one or more communication processing modules.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110.
  • the wireless communication module 160 can also receive the signal to be sent from the processor 110, perform frequency modulation, amplify it, and convert it into electromagnetic wave radiation via the antenna 2.
  • the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.
  • the electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor.
  • the GPU is a microprocessor for image processing, connected to the display 194 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations for graphics rendering.
  • the processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos, etc.
  • the display screen 194 includes a display panel.
  • the display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active-matrix organic light-emitting diode active-matrix organic light-emitting diode
  • AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the electronic device 100 may include one or N display screens 194, and N is a positive integer greater than one.
  • the electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
  • the ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transfers the electrical signal to the ISP for processing and is converted into an image visible to the naked eye.
  • ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be provided in the camera 193.
  • the camera 193 is used to capture still images or videos.
  • the mobile phone 100 may include 1 or N cameras, and N is a positive integer greater than 1.
  • the camera 193 may be a front camera or a rear camera.
  • the camera 193 generally includes a lens and a sensor.
  • the photosensitive element can be a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor, complementary metal oxide semiconductor). ) And other arbitrary photosensitive devices.
  • the reflected light of the object being photographed can generate an optical image after passing through the lens.
  • the optical image is projected on the photosensitive element, and the photosensitive element converts the received optical signal into electricity.
  • the camera 193 sends the obtained electrical signal to a DSP (Digital Signal Processing, digital signal processing) module for digital signal processing, and finally each frame of digital image is obtained.
  • DSP Digital Signal Processing, digital signal processing
  • the camera 193 may be used to collect air gestures input by the user.
  • the camera 193 can be set to be always on to collect current images in real time.
  • the camera 193 can send each frame of image collected to the processor 110. If it is recognized from the collected images that the user has performed the preset opening gesture, indicating that the user wants to interact with the electronic device by means of an air gesture, the processor 110 can recognize the user at this time according to the image sent by the camera 193 in real time. The specific air gesture entered.
  • the aforementioned preset opening gesture may be a gesture of hovering the palm for 200 ms. Then, if the image collected by the camera 193 in the last 200 ms contains the pattern of the user's palm, and the user's palm does not move significantly, the processor 110 may determine that the user has performed a preset opening gesture. Furthermore, the camera 193 can continue to send each frame of image collected to the processor 110, and the processor 110 can determine, according to the multi-frame images sent by the camera 193, that the user’s current gesture for airspace input is sliding up, sliding down, and Swipe left or right, etc.
  • the first coordinate system can be preset in the mobile phone.
  • the first coordinate system includes the x-axis and y-axis.
  • the x-axis is the horizontal direction when the mobile phone is normally displayed, that is, when the user normally reads the content displayed on the mobile phone.
  • the horizontal direction correspondingly, the y-axis is the vertical direction perpendicular to the x-axis.
  • the x-axis is the horizontal direction along the short side of the mobile phone screen
  • the y-axis is the vertical direction along the long side of the mobile phone screen.
  • the processor 110 may determine the movement distance of the user's palm on the x-axis and the y-axis according to the position change of a certain point on the user's palm in the continuous multiple frames of images. If the movement distance of the user's palm on the x-axis is greater than the threshold 1, the processor 110 may determine that the user has performed a space gesture of sliding left (or right). If the movement distance of the user's palm on the y-axis is greater than the threshold 2, the processor 110 may determine that the user has performed an upward (or downward) air gesture of sliding.
  • the processor 110 may not only determine the air gesture input by the user in real time, but also instruct the display screen 194 to display the execution progress of the air gesture in real time.
  • the processor 110 may instruct the display screen 194 to display the slider 302 in the interface 301 being displayed.
  • the processor 110 may instruct the display screen 194 to slide the slider 302 down in real time.
  • the electronic device can guide the user to complete the air gesture through the movement of the slider 302, and on the other hand, the user can perceive the current air gesture in real time through the movement of the slider 302.
  • the execution progress of the gestures improves the user experience when interacting with the electronic device in the air, and reduces the misoperation rate of the gestures in the air.
  • the processor 110 when it detects a specific space gesture input by the user, it can respond to the space gesture to perform corresponding operations such as page turning, return, or next step.
  • the embodiment of the present application does not impose any restriction on this, so that Users can control electronic devices to achieve corresponding functions through air gestures.
  • the electronic device may also collect the air gesture input by the user through one or more devices such as an infrared sensor and an ultrasonic sensor. Do any restrictions.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects the frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in a variety of encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.
  • MPEG moving picture experts group
  • the external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save music, video and other files in an external memory card.
  • the internal memory 121 may be used to store one or more computer programs, and the one or more computer programs include instructions.
  • the processor 110 can execute the above-mentioned instructions stored in the internal memory 121 to enable the electronic device 100 to execute the methods provided in some embodiments of the present application, as well as various functional applications and data processing.
  • the internal memory 121 may include a storage program area and a storage data area. Among them, the storage program area can store the operating system; the storage program area can also store one or more application programs (such as a gallery, contacts, etc.) and so on.
  • the data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, universal flash storage (UFS), etc.
  • the processor 110 executes the instructions stored in the internal memory 121 and/or the instructions stored in the memory provided in the processor to cause the electronic device 100 to execute the method provided in the embodiments of the present application. , And various functional applications and data processing.
  • the electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.
  • the audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal.
  • the audio module 170 can also be used to encode and decode audio signals.
  • the audio module 170 may be provided in the processor 110, or part of the functional modules of the audio module 170 may be provided in the processor 110.
  • the speaker 170A also called a “speaker” is used to convert audio electrical signals into sound signals.
  • the electronic device 100 can listen to music through the speaker 170A, or listen to a hands-free call.
  • the receiver 170B also called “earpiece” is used to convert audio electrical signals into sound signals.
  • the electronic device 100 answers a call or voice message, it can receive the voice by bringing the receiver 170B close to the human ear.
  • the microphone 170C also called “microphone”, “microphone”, is used to convert sound signals into electrical signals.
  • the user can approach the microphone 170C through the mouth to make a sound, and input the sound signal to the microphone 170C.
  • the electronic device 100 may be provided with one or more microphones 170C.
  • the electronic device 100 may be provided with two microphones 170C, which can implement noise reduction functions in addition to collecting sound signals.
  • the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions.
  • the earphone interface 170D is used to connect wired earphones.
  • the earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA
  • the sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc., in the embodiments of the present application There are no restrictions on this.
  • the electronic device 100 provided in the embodiment of the present application may further include one or more devices such as the button 190, the motor 191, the indicator 192, and the SIM card interface 195, which is not limited in the embodiment of the present application.
  • the software system of the above electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.
  • the embodiment of the present application takes a layered Android system as an example to illustrate the software structure of the electronic device 100.
  • FIG. 4 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers through software interface.
  • the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.
  • the application layer can include a series of application packages.
  • APPs applications
  • call memo
  • browser contact
  • camera camera
  • gallery calendar
  • map map
  • Bluetooth music, video, short message, etc.
  • the application layer also includes Android core applications such as SystemUI.
  • SystemUI can be used to feed back the system and application status and maintain interaction with users.
  • the application framework layer provides application programming interfaces (application programming interface, API) and programming frameworks for applications in the application layer.
  • the application framework layer includes some predefined functions.
  • the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager.
  • the application framework layer may also include an AI engine.
  • the AI engine can be used to recognize the air gesture input by the user in real time based on the images collected by the camera. After the AI engine recognizes the space gesture input by the user, it can also pass the space gesture to the related application in the application layer, so that the application can perform corresponding operations in response to the space gesture input by the user.
  • the specific name of the aforementioned AI engine is not limited in the embodiments of the present application, and a functional module with any name can be set in the application framework layer to perform the aforementioned functions.
  • the working state of the AI engine may include a normal mode and an air gesture mode.
  • the AI engine can determine whether the user has input the opening gesture of the air gesture mode based on the images collected by the camera in real time, that is, whether the user has the operation intention to enable the air gesture. For example, the AI engine can obtain every frame of images collected by the camera in real time by calling the camera driver at the kernel layer. Furthermore, the AI engine can recognize whether the user has input a preset opening gesture based on these images. Taking the preset opening gesture can be a palm hovering gesture for 200ms, the AI engine can continuously obtain N (N>1) frames of images collected in the last 200ms, and identify whether these N frames of images contain the user's palm pattern . If the N frames of images contain the pattern of the user's palm, and the position movement of the user's palm in the N frames of images is less than the threshold, the AI engine can determine that the user has input the opening gesture at this time.
  • N N>1 frames of images collected in the last 200ms
  • the AI engine After the AI engine determines that the user has input the opening gesture, it can report the first event to the system UI in the application framework layer, and the first event can be used to indicate the opening of the air gesture mode.
  • the system UI After the system UI receives the first event, as shown in (a) in FIG. 5, the system UI can display the indication icon 502 of the air gesture on the preset position of the current display interface 501.
  • the indication icon 502 may prompt the user that the above-mentioned opening gesture has been detected and the gap gesture input by the user has been detected.
  • the indicator icon 502 may specifically be a pattern such as a progress bar, a slider, and the embodiment of the present application does not impose any restrictions on the display effect such as the size, position, shape, and color of the indicator icon 502.
  • the AI engine determines that the user has input the opening gesture
  • the AI engine enters the air gesture mode from the normal mode.
  • the AI engine can determine the specific air gesture input by the user based on the images collected by the camera in real time. For example, the AI engine can obtain every frame of images collected by the camera in real time by calling the camera driver at the kernel layer. Furthermore, the AI engine can recognize the space gesture input by the user based on the movement distance of the user gesture in these images. For example, the AI engine can use the position of the user's palm when the user performs the above-mentioned opening gesture as the initial position A. Furthermore, the AI engine can calculate the x-axis and y-axis directions of the user's palm from the initial position A based on the images collected in real time. The moving distance. In this way, according to the movement distance of the user's palm in the x-axis direction and the y-axis direction, the AI engine can further determine that the user is outputting an air gesture of sliding up, sliding down, sliding left, or sliding right.
  • the AI engine determines that the user is entering the left swipe gesture, it can report the corresponding sliding event to the system UI and the album APP running in the foreground.
  • the album APP receives the sliding event, it can start to display the next picture 504 after the current picture 503 in response to the sliding event.
  • the system UI can slide the indicator 502 to the left in response to the sliding event.
  • the sliding distance of the indicator icon 502 is proportional to the moving distance of the air gesture being input by the user.
  • the user when the user no longer wants to use the air gesture to interact with the mobile phone, he can also input the end gesture of the air gesture mode to the camera.
  • the end gesture may be a gesture of the user holding a fist and hovering for 200ms.
  • the AI engine determines that the user has entered a preset end gesture based on the image collected by the camera, the AI engine can switch from the air gesture mode to the normal mode.
  • the AI engine can report the second event to the system UI, and the second event can be used to indicate the end of the air gesture mode.
  • the system UI receives the second event, it can hide the indication icon 502 in the current display interface, thereby prompting the user to exit the air gesture operation mode.
  • the above-mentioned window manager is used to manage window programs.
  • the window manager can obtain the size of the display, determine whether there is a status bar, lock the screen, take a screenshot, etc.
  • the above-mentioned content provider is used to store and obtain data, and make these data accessible by applications.
  • the data may include video, image, audio, phone calls made and received, browsing history and bookmarks, phone book, etc.
  • the above-mentioned view system includes visual controls, such as controls for displaying text and controls for displaying pictures.
  • the view system can be used to build applications.
  • the display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon may include a view that displays text and a view that displays pictures.
  • the above-mentioned telephone manager is used to provide the communication function of the electronic device 100.
  • the management of the call status (including connecting, hanging up, etc.).
  • the aforementioned resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.
  • the above notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction.
  • the notification manager is used to notify download completion, message reminders, etc.
  • the notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window.
  • a notification for example, prompt text information in the status bar, sound a prompt tone, electronic device vibration, flashing indicator light, etc.
  • the system library can include multiple functional modules.
  • surface manager surface manager
  • media library Media Libraries
  • three-dimensional graphics processing library for example: OpenGL ES
  • 2D graphics engine for example: SGL
  • the surface manager is used to manage the display subsystem and provides a combination of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files.
  • the media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to realize 3D graphics drawing, image rendering, synthesis, and layer processing.
  • the 2D graphics engine is a drawing engine for 2D drawing.
  • Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.
  • the core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.
  • the application layer and the application framework layer run in a virtual machine.
  • the virtual machine executes the java files of the application layer and the application framework layer as binary files.
  • the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, etc., which are not limited in the embodiment of the present application.
  • the method includes:
  • S601 The mobile phone detects a preset opening gesture input by the user through the camera.
  • the camera of the mobile phone can be set to a normally open state.
  • the camera of the mobile phone can always be in working state, and the camera can collect images within the shooting range according to a certain working frequency.
  • the aforementioned camera may also be a 3D camera (also referred to as a depth camera or a depth camera), and the image captured by the mobile phone through the 3D camera can obtain the object in the image or the distance between the user and the mobile phone.
  • the mobile phone can turn off the air gesture function of the mobile phone by default to save the power consumption of the mobile phone.
  • the camera of the mobile phone is still working.
  • the opening gesture of the space gesture can be input within the shooting range of the mobile phone camera (for example, the front camera).
  • the opening gesture may be a gesture of hovering the user's palm for 200ms. Since the camera of the mobile phone is normally open, the camera can collect multi-frame images when the user's palm is hovering, and the mobile phone can recognize whether the user is currently inputting the aforementioned opening gesture according to the multi-frame images collected by the camera.
  • the mobile phone can acquire N frames of images collected by the camera within the last 200ms in real time.
  • a two-dimensional rectangular coordinate system (or a spatial rectangular coordinate system) can be preset in the mobile phone.
  • the x-axis and y-axis are established with the upper right corner of each frame of image as the origin O.
  • the x-axis and the y-axis can be in pixels, and distance units such as mm and cm can also be set.
  • the mobile phone can recognize the pattern 801 of the user's palm in the N frames of images and the reference point C in the pattern 801, and the reference point C can be any point in the pattern 801.
  • the reference point C may be the center point of the pattern 801, or the reference point C may be the end point of the user's index finger or the like.
  • each of the aforementioned N frames of images contains the pattern 801, or the number of images containing the pattern 801 in the aforementioned N frames of images is greater than the preset value 1, it means that the user's palm has been present in the shooting range of the camera during the 200ms.
  • the mobile phone can calculate the movement distance of the user's palm according to the coordinates of the reference points C1, C2,..., C(n) in the above N frames of images. If the movement distance of the user's palm is less than the preset value 2 (for example, the preset value 2 is 50 pixel units), it means that the user's palm has been hovering within the shooting range of the camera during the 200ms. At this time, the mobile phone can determine the user input The preset opening gesture is used.
  • the mobile phone after the mobile phone detects that the user has entered a preset opening gesture, it can also record the position of the user's palm at this time. For example, the mobile phone may use the coordinates of the reference point C(n) of the user's palm in the Nth frame of image in FIG. 8 as the position of the user's palm, and this position may also be used as the starting position A1 of the subsequent user's input of the space gesture.
  • the mobile phone determines that the number of images containing the pattern 801 in the above N frames of images is less than the preset value 1, or the mobile phone determines that the movement distance of the user's palm is greater than the preset value 2, it means that the user has not input the palm hung in the 200ms. Stop the opening gesture, the mobile phone can continue to obtain the N frames of images collected by the camera within the last 200ms, and continue to recognize whether the user has input the preset opening gesture at this time according to the above method.
  • the user can also trigger the mobile phone to enter the air gesture operation mode by other means.
  • the user can use the voice assistant APP in the mobile phone to input a voice signal for opening the air gesture.
  • the mobile phone detects the voice signal, it can turn on the camera to start collecting the air gesture input by the user.
  • a corresponding button such as a virtual button or a physical button
  • an operation such as a pressing operation, a double-clicking operation, etc.
  • the mobile phone can turn on the camera to start collecting the air gesture input by the user.
  • the position of the user's palm collected by the camera for the first time can be used as the starting position A1 for the subsequent user to input the space gesture.
  • the mobile phone can detect whether the user performs a preset hovering gesture after turning on the camera. If it detects that the user has performed a preset hovering gesture, the mobile phone can use the hovering position of the user as the start of the subsequent user input air gesture Location A1.
  • the mobile phone After the mobile phone detects that the user has input the preset opening gesture, it can execute the following steps S602 and S603 respectively in response to the opening gesture. It should be noted that the embodiment of this application does not impose any restriction on the execution order of the following steps S602 and S603.
  • the mobile phone can execute the following steps S602 and S603 at the same time, or it can execute step S602 first and then step S603, or execute step S603 first. S603 executes step S602 again.
  • the mobile phone In response to the aforementioned opening gesture, the mobile phone displays a preset indicator icon on the current first display interface.
  • step S602 after the mobile phone detects that the user has input a preset opening gesture, it indicates that the user wants to use the air gesture to control the running application in the mobile phone.
  • the mobile phone can display a preset indicator icon at a preset position of the interface being displayed (for example, the first display interface). The position, length, or length of the indicator icon in the interface The shape can be changed.
  • the mobile phone running the e-book APP in the foreground as an example, as shown in FIG. 9, when the user inputs the aforementioned opening gesture, the mobile phone is displaying the first display interface 901 of the e-book APP. After detecting the user's input of the aforementioned opening gesture, the mobile phone can display the first indicator icon 902 on the top of the screen in the horizontal direction. After the first indication icon 902 appears in the first display interface 901, the user can learn that the mobile phone has entered the air gesture operation mode, thereby reminding the user to start inputting the corresponding air gesture.
  • the above-mentioned first indicator icon 902 can be an icon such as a slider, a progress bar, etc.; the above-mentioned first indicator icon 902 can be a linear or arc-shaped icon; the specific shape of the first indicator icon 902 is There are no restrictions on the display effects such as, size, position or color.
  • the mobile phone can display the first indicator icon 902 in the middle of the top along the top of the first display interface 901 to prevent the first indicator icon 902 from affecting the display content in the first display interface 901. Produce occlusion.
  • the first indicator icon 902 can move in the horizontal direction (that is, the x-axis direction), and the first indicator icon 902 can be used to indicate the moving distance of the air gesture input by the user in the horizontal direction. That is to say, after the mobile phone enters the operation mode of the air gesture, the mobile phone can decompose the air gesture input by the user into a sliding operation in the x-axis direction, and prompt the user this air gesture by moving the first indicator icon 902 The execution progress in the x-axis direction.
  • the mobile phone can also display indicator information such as text, arrows, or preset icons.
  • indicator information such as text, arrows, or preset icons.
  • the first arrow can be displayed on the left side of the first indicator icon 902, and the user is prompted by text to swipe left to turn pages; and, the second arrow can be displayed on the right side of the first indicator icon 902, The text prompts the user to swipe to the right to turn the page, so that the user can purposefully execute the corresponding air gesture according to the instructions.
  • the mobile phone after detecting that the user has entered the preset opening gesture, can display the preset icon 903 in the status bar to remind the user that the mobile phone has entered the air gesture operation mode and can start to input the corresponding Gesture in the air.
  • the mobile phone after the mobile phone detects that the user has input the above-mentioned opening gesture, it can also remind the user to enter the air gesture operation mode by voice or other means, and prompt the user to input the corresponding air gesture, which is not limited in this embodiment of the application.
  • the mobile phone can display the second in the vertical direction (that is, the y-axis direction).
  • Indication icon 1102. the mobile phone may display the second indicator icon 1102 in the middle of the left side (or right side) along the left side (or right side) of the display screen.
  • the second indicator icon 1102 can move in the vertical direction (that is, the y-axis direction), and the second indicator icon 1102 can be used to indicate the moving distance of the air gesture input by the user in the vertical direction.
  • the mobile phone can decompose the air gesture input by the user into a sliding operation in the y-axis direction, and prompt the user this air gesture by moving the second indicator icon 1102 The execution progress in the y-axis direction.
  • the mobile phone can also determine whether the currently displayed interface (such as interface 1101) supports sliding up and down or sliding left and right according to the type of application running in the foreground or the type of interface being displayed. If the currently displayed interface supports the operation of sliding left and right, the mobile phone can display the aforementioned first indicator icon 902 at the top of the screen; if the currently displayed interface supports the operation of sliding up and down, the mobile phone can display the aforementioned second indicator icon 1102 on the side of the screen. ; If the currently displayed interface supports both up and down sliding operations and left and right sliding operations, the mobile phone can display the above-mentioned first indicator icon 902 and second indicator icon 1102 at the same time, which is not limited in this embodiment of the application.
  • the currently displayed interface such as interface 1101
  • the mobile phone can also display the indicator icon (the first indicator icon 902 shown in FIG. 9 ). Subsequent, if the user enters an air gesture of sliding up and down, the mobile phone can display the indicator icon on the side of the screen (the second indicator icon 1102 shown in Figure 11), and move the second indicator icon in response to the air gesture 1102.
  • the corresponding relationship between each position on the display screen and each position on each frame of images collected by the camera may be established in the mobile phone in advance.
  • the upper right corner of the mobile phone display screen can be used as the origin O1
  • the first coordinate system can be established with O1 as the origin.
  • the upper right corner of each frame of images captured by the camera can be taken as the origin O2
  • the second coordinate system can be established with O2 as the origin.
  • the mobile phone When the mobile phone detects the opening gesture input by the user, the mobile phone can determine the position of the user's palm collected at the end of the opening gesture (ie point A1) as the starting position of the user inputting the air gesture. Furthermore, the mobile phone can determine the B point corresponding to the starting position A1 point in the first coordinate system. In other words, the current position of the user's palm corresponds to point B on the display screen. Then, as shown in FIG. 13, taking the mobile phone displaying the desktop 1301 as an example, the mobile phone may display a third indicator icon 1302 at point B of the desktop 1301, thereby prompting the user that the current palm position corresponds to point P of the desktop 1301. The third indicator icon 1302 can also move in the desktop 1301 following the space gesture input by the user.
  • the mobile phone when the mobile phone enters the operation mode of the air gesture, the mobile phone can display a preset indicator icon on the current display interface to prompt the user to start the air gesture. In the future, the mobile phone can also guide the user to input the execution progress of the air gesture by moving the indicator icon, thereby improving the user experience of using the air gesture to interact with the mobile phone.
  • the mobile phone starts to detect the first air-space gesture input by the user through the camera.
  • step S603 after the mobile phone detects that the user has input the preset opening gesture, it is explained that the user wants to use the air gesture to control the running application in the mobile phone, and the mobile phone enters the operation mode of the air gesture. Then, the mobile phone can continue to use the camera to collect each frame of images in real time, and the mobile phone can recognize the specific space gesture input by the user according to the collected images.
  • the mobile phone can start position A1 at the position of the user's palm collected at the end of the opening gesture, and then determine the user input in real time based on the most recently collected M frames of images The first air gesture.
  • the mobile phone after the mobile phone enters the operation mode of the air gesture, after each frame of image acquired by the mobile phone, the pattern of the user's palm in the image can be recognized. Furthermore, the mobile phone can determine the position of the user's palm in the image. For example, the mobile phone may use the coordinates of the center point A(m) of the user's palm as the position of the user's palm in each frame of image. In this way, starting from the coordinates of the starting position A1, the mobile phone can calculate the movement distance S1 of the user's palm on the x-axis and the movement distance S2 of the user's palm on the y-axis according to the coordinates of the center point A(m) in each frame of image.
  • the mobile phone can also calculate the movement distance S3 of the user's palm on the z-axis.
  • the mobile phone can decompose the first air gesture that the user is inputting into gesture operations in the three directions of the x-axis, y-axis, and z-axis.
  • the movement distance S1 of the user's palm in the positive x-axis direction is greater than the threshold 1, it means that the first space gesture input by the user is moving to the left; when the movement distance S1 of the user's palm in the negative x-axis direction is greater than the threshold At 1 o'clock, it means that the first air gesture input by the user is moving to the right.
  • the movement distance S2 of the user's palm in the positive y-axis direction is greater than the threshold 2
  • the movement distance S2 of the user's palm in the negative y-axis direction is greater than the threshold 2
  • it means The first air gesture entered by the user is moving up.
  • the movement distance S3 of the user's palm in the z-axis direction is greater than the threshold 3, it indicates that the first space gesture input by the user is a pressing operation.
  • the mobile phone can set corresponding step lengths on the x-axis, y-axis, and z-axis. Taking the step size set on the x-axis of the mobile phone as an example of 50 pixel units, the mobile phone can use the starting position A1 of the user's palm as the starting point, and detect the movement distance S1 of the user's palm in the positive x-axis direction in real time based on the most recently acquired M frame image.
  • the mobile phone can move to the application layer
  • the system UI reports a sliding event of sliding to the left, so that the mobile phone can perform the following step S604 in response to the sliding event.
  • the mobile phone can also report a left swipe to the application that is running in the foreground in the application layer (such as the e-book APP mentioned above)
  • the mobile phone can perform the following step S605 in response to the sliding event.
  • the mobile phone may execute the following steps S604-S605 in response to the first air gesture according to the movement distance of the detected first air gesture in the x-axis, y-axis and z-axis directions. It should be noted that the embodiment of this application does not impose any restriction on the execution order of the following steps S604 and S605.
  • the mobile phone can execute the following steps S604 and S605 at the same time, or it can execute step S604 and then step S605, or execute step S605 first. S605 then executes step S604.
  • the mobile phone moves the indication icon according to the motion track of the first air-space gesture.
  • the mobile phone In response to the first airspace gesture, the mobile phone displays a second display interface corresponding to the first airspace gesture.
  • step S604 if it is detected that the moving distance S1 of the first airspace gesture in the positive x-axis direction is greater than the corresponding step size, as shown in FIG. 15, the mobile phone
  • the first indicator icon 902 can also be extended along the positive x-axis direction (ie, to the left). In this way, the user can simultaneously learn the movement track of the first space gesture through the first indicator icon 902 in the display interface while inputting the first space gesture.
  • the mobile phone can calculate the distance L1 by which the mobile phone extends the first indicator icon 902 to the left according to the moving distance S1 of the first air-space gesture in the positive direction of the x-axis and a certain proportional coefficient k (k>0).
  • the scale factor k may be proportional to the distance R between the user and the mobile phone.
  • the mobile phone can detect the distance R between the user and the mobile phone through a 3D camera.
  • the mobile phone can dynamically determine the aforementioned proportional coefficient k according to the distance R between the user and the mobile phone.
  • the corresponding scale factor k can be set to be larger; when the distance R between the user and the mobile phone is smaller, the corresponding scale factor k can be set to be smaller.
  • the mobile phone may also report a sliding event to the running e-book APP, and the sliding event may include Parameters such as sliding direction and moving distance S1.
  • the e-book APP can perform a page turning operation in response to the sliding event.
  • the mobile phone can display the second display interface 1501 corresponding to the moving distance S1.
  • the second display interface 1501 may have part of the same display content as the first display interface 901 displayed before the first air-space gesture is input. In this way, while the user inputs the first space gesture, the mobile phone can respond to the first space gesture to control related applications to implement corresponding functions, and realize the operation experience of space interaction between the mobile phone and the user.
  • step S604 if it is detected that the movement distance S2 of the first space gesture in the positive direction of the y-axis is greater than the corresponding step length, as shown in FIG. 16, the mobile phone The second indicator icon 1102 can be extended downward in a certain proportion. In this way, the user can simultaneously learn the movement track and execution progress of the first space gesture through the second indicator icon 1102 in the display interface while inputting the first space gesture.
  • the mobile phone may also report a sliding event to the running browser APP, and the sliding event may include parameters such as the sliding direction and the moving distance S2.
  • the browser APP can perform a page turning operation in response to the sliding event.
  • the mobile phone can display a display interface 1601 corresponding to the moving distance S2, so that the mobile phone can respond to the first space input by the user.
  • the gesture completes the corresponding function.
  • FIG. 16 is an example of a browser APP to illustrate how the mobile phone responds to the space gesture when the user inputs the space gesture of sliding up and down.
  • the above-mentioned browser APP may also be an application that supports up and down sliding operations, such as a video APP.
  • the mobile phone is playing the short video A of the Douyin APP, if the user inputs a preset operation of hovering the palm for 200ms, the mobile phone can display the preset instructions on the first display interface of the short video A Icon to remind the user that the user has entered the air gesture operation mode.
  • the indicator icon can be displayed at the top of the first display interface.
  • the mobile phone can display the indicator icon on the side of the first display interface according to the movement distance of the space gesture on the y axis, and move along the y axis The indicator icon to remind the user of the execution progress of this air gesture on the y axis.
  • the Douyin APP can display the second display interface of the short video B in response to the air gesture, thereby responding to the air gesture of sliding upwards input by the user.
  • step S604 if it is detected that the moving distance S3 of the first air gesture in the z-axis direction is greater than the threshold 3, it indicates that the user wants to press the screen, as shown in FIG. ,
  • the mobile phone can enlarge the third indicator icon 1302 according to a certain ratio. In this way, the user can learn the movement track and execution progress of the first space gesture through the third indicator icon 1302 in the display interface while inputting the first space gesture.
  • the mobile phone may also report a pressing event to the running desktop APP, and the pressing event may include the position coordinate point B where the third indicator icon 1302 is located.
  • the browser APP can open the WeChat APP corresponding to point B in response to the pressing event.
  • the mobile phone can display the display interface 1701 of the WeChat APP, so that the mobile phone can respond to the first interval input by the user.
  • the empty gesture completes the corresponding function.
  • the display effects of the extended indicator icon (such as the aforementioned first indicator icon 902 and the second indicator icon 1102) and the enlarged indicator icon (such as the aforementioned third indicator icon 1302) are taken as examples for illustration. It is understandable that those skilled in the art can set the display effect of the indicator icon in response to the user's input of an air gesture according to actual experience or actual application scenarios.
  • the mobile phone can slide the first indicator icon 902 and the second indicator icon 1102 to feedback the execution progress of the space gesture to the user on the display interface, which is not limited in the embodiment of the present application.
  • the mobile phone can calculate the movement distance of the first air gesture on the x axis in real time according to the above method. S1, and then move the first indicator icon 902 to the left (or right) according to the moving distance S1 in real time.
  • the first indicator icon 902 moves to a preset end position, such as the leftmost or rightmost side of the screen, it means that the first space gesture input by the user this time is over.
  • the e-book APP has turned to the previous page (or the next page) in response to the first space gesture.
  • the mobile phone in the process of the user inputting the first space gesture, if the mobile phone does not detect the relevant pattern of the user's palm in the recent consecutive multiple frames of images, it means that the first space gesture input by the user has exceeded the shooting range of the camera. When the mobile phone can determine the end of the first gesture input by the user this time.
  • the mobile phone in the process of the user inputting the first air gesture, if the mobile phone detects that the position of the user's palm has not changed significantly in the most recent consecutive frames of images, indicating that the air gesture input by the user has stopped, the mobile phone can also be used at this time. Determine the end of the first air gesture input by the user this time.
  • the first indicator icon 902 is still used as an example.
  • the mobile phone can restore the first indicator icon 902 to the preset value in the first display interface when the first indicator icon 902 appears.
  • the position such as the middle position of the top of the screen, prompts the user that the first air gesture has ended and the mobile phone has completed the response to the first air gesture.
  • S607 The mobile phone continues to detect the second air gesture input by the user through the camera.
  • the mobile phone can also continue to collect the second air gesture input by the user through the camera.
  • the user will generate a return operation of the first air gesture between the processes of performing the first air gesture and the second air gesture.
  • the return operation refers to a gesture return operation performed in a direction opposite to the movement of the first space gesture after the user performs the first space gesture.
  • the first air gesture is an operation for the user to slide the palm up
  • the second air gesture is also an operation for the user to slide the palm up.
  • the user's palm moves along the negative direction of the y-axis in the multi-frame images of the first space gesture collected by the mobile phone.
  • the palm needs to be moved along the positive direction of the y axis. Retract the direction to generate a return operation, and then start a second time to slide the palm up along the negative direction of the y-axis to perform the second space gesture.
  • the mobile phone can detect the backhaul operation performed by the user in real time according to the collected image. For example, when the first gap gesture ends at T1, the user starts to perform the return operation of the first gap gesture; when the mobile phone detects that the movement direction of the user's palm changes from the positive direction along the y axis to the negative direction along the y axis At time (that is, time T2), the return operation is over. In order to prevent the mobile phone from being interfered by this backhaul operation when recognizing the second air gesture, the mobile phone can discard the images collected at T1-T2.
  • the mobile phone can use the position of the user's palm in the image collected at time T2 as the starting position of the second gap gesture (for example, the starting position is B1), and recognize the second gap input by the user in real time according to the collected image. Empty gesture.
  • the mobile phone can use the coordinates of the starting position B1 as the starting point to calculate the movement distance of the user's palm in the x-axis, y-axis and z-axis directions in the second air gesture, thereby identifying the second air gesture input by the user .
  • the specific method for the mobile phone to recognize the second space gesture input by the user is similar to the specific method for the mobile phone to recognize the first space gesture input by the user in step S603, so it will not be repeated here.
  • the mobile phone moves the indication icon according to the movement track of the second air-space gesture.
  • the mobile phone In response to the second air gesture, the mobile phone displays a display interface corresponding to the second air gesture.
  • the mobile phone can report corresponding events (such as sliding events, pressing events, etc.) to the system UI, so that the system UI responds to the event
  • the indicator icon for example, the first indicator icon 902, the second indicator icon 1102, or the third indicator icon 1302
  • the mobile phone can report corresponding events (such as page turning events, return events, etc.) to applications running in the foreground, so that related applications can respond to the events to implement corresponding application functions.
  • the mobile phone can restore the indicator icon in the current display interface to the starting position again. In the future, the mobile phone can continue to recognize and respond to the third air gesture, the fourth air gesture, ... input by the user according to the above method.
  • the mobile phone can continuously recognize and respond to the multiple air gestures performed by the user continuously, and feedback the execution progress of each air gesture input to the user through the indicator icon on the display interface. So as to improve the user experience of interacting with the mobile phone using air gestures.
  • the mobile phone may start a preset timer after detecting the user input to turn on the gesture. Taking timer 1 of 10s as an example, after detecting the user’s input of a palm hovering gesture for 200ms, the mobile phone can start timer 1. Within 10s when timer 1 does not expire, the mobile phone can continuously recognize Respond to multiple air gestures continuously performed by the user, and feedback the execution progress of each air gesture input to the user on the display interface through the indicator icon.
  • the mobile phone can automatically exit the air gesture operation mode, and no longer continue to recognize and respond to the air gesture performed by the user based on the images collected by the camera.
  • the user can input the preset opening gesture to the mobile phone again.
  • the mobile phone can continue to execute the space gesture interaction method provided in the embodiment of the present application from the above step S601.
  • the mobile phone can hide the originally displayed indicator icon in the current display interface, thereby prompting the user that the mobile phone has exited the air gesture operation mode.
  • the end gesture may be a gesture of the user holding a fist and hovering for 200ms.
  • the mobile phone can determine that the user has entered the preset End gesture. Then, in response to the end gesture, as shown in (b) in FIG. 21, the mobile phone can hide the above-mentioned indication icon (for example, the first indication icon 902) in the current display interface 901, thereby prompting the user to exit the air gesture operation mode.
  • the above-mentioned indication icon for example, the first indication icon 902
  • the embodiment of the present application discloses an electronic device including a processor, and a memory, an input device, and an output device connected to the processor.
  • the input device and the output device can be integrated into one device.
  • a touch sensor can be used as an input device
  • a display screen can be used as an output device
  • the touch sensor and display screen can be integrated into a touch screen.
  • the above-mentioned electronic device may include: a touch screen 2201 including a touch sensor 2206 and a display screen 2207; one or more processors 2202; one or more cameras 2208; a memory 2203; one Or multiple application programs (not shown); and one or more computer programs 2204.
  • the above-mentioned devices may be connected through one or more communication buses 2205.
  • the one or more computer programs 2204 are stored in the aforementioned memory 2203 and configured to be executed by the one or more processors 2202.
  • the one or more computer programs 2204 include instructions, and the aforementioned instructions can be used to execute the aforementioned implementations.
  • the steps in the example Among them, all relevant content of the steps involved in the above method embodiments can be cited in the functional description of the corresponding physical device, which will not be repeated here.
  • the above-mentioned processor 2202 may specifically be the processor 110 shown in FIG. 1
  • the above-mentioned memory 2203 may specifically be the internal memory 121 shown in FIG. 1
  • the above-mentioned camera 2208 may specifically be the camera 193 shown in FIG.
  • the display screen 2207 may specifically be the display screen 194 shown in FIG. 1
  • the aforementioned touch sensor 2206 may specifically be a touch sensor in the sensor module 180 shown in FIG. 1, and the embodiment of the present application does not impose any limitation on this.
  • the functional units in the various embodiments of the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • a computer readable storage medium includes a number of instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Hardware Design (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

一种隔空手势的交互方法及电子设备，涉及终端技术领域和AI技术领域，例如AI图像识别领域，可提高用户与电子设备进行隔空交互时的使用体验，降低隔空手势的误操作率。该方法包括：电子设备显示第一应用的第一显示界面；所述电子设备通过摄像头检测到用户输入第一隔空手势；响应于所述第一隔空手势，所述电子设备沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标；响应于所述第一隔空手势，所述电子设备显示所述第一应用的第二显示界面。

Description

一种隔空手势的交互方法及电子设备
本申请要求于2019年8月19日提交国家知识产权局、申请号为201910765254.5、发明名称为“一种隔空手势的交互方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及终端技术领域，尤其涉及一种隔空手势的交互方法及电子设备。
背景技术
随着电子技术的不断发展，用户对手机等电子设备的使用，不再局限于通过接触触摸屏来进行操作。目前，一些手机可以通过摄像头或超声波传感器等器件，识别用户在三维空间内执行的手势动作(即隔空手势)，从而实现对手机中一些功能的隔空操作。
以用户浏览手机中的电子书APP举例，手机可通过摄像头采集到的图像识别用户执行的隔空手势。例如，如果检测到用户执行向左挥手的第一隔空手势，则手机可响应第一隔空手势在电子书APP中执行“翻转到上一页”的第一操作。如果检测到用户执行向右挥手的第二隔空手势，则手机可响应第二隔空手势在电子书APP中执行“翻转到下一页”的第二操作。
一般，当手机响应用户输入的隔空手势执行相应的操作后，用户可感知到手机已经成功接收到自己输入的隔空手势。而在用户向手机输入隔空手势的过程中，用户无法感知手机是否检测到自己输入的隔空手势，导致用户无法确定正在输入的隔空手势是否正确，使用户与手机进行隔空交互时的使用体验较低，误操作率较高。
发明内容
本申请提供一种隔空手势的交互方法及电子设备，可提高用户与电子设备进行隔空交互时的使用体验，降低隔空手势的误操作率。
为达到上述目的，本申请采用如下技术方案：
第一方面，本申请提供一种隔空手势的交互方法，包括：电子设备显示第一应用的第一显示界面；同时，电子设备可通过摄像头检测到用户输入第一隔空手势；响应于第一隔空手势，电子设备可沿第一隔空手势在预设方向上的移动轨迹移动第一显示界面中的指示图标；同时，响应于第一隔空手势，电子设备可显示第一应用的第二显示界面。
这样，用户在输入隔空手势(例如上述第一隔空手势)的过程中，一方面电子设备可通过指示图标在预设方向上的移动指导用户完成本次隔空手势，另一方面用户可通过指示图标的移动实时的感知到当前隔空手势的执行进度，从而提高了用户与电子设备进行隔空交互时的使用体验，降低隔空手势的误操作率。
在一种可能的实现方式中，上述预设方向可以为直角坐标系的x轴方向，该x轴 方向为用户正常使用电子设备时的水平方向；此时，电子设备沿第一隔空手势在预设方向上的移动轨迹移动第一显示界面中的指示图标，具体包括：电子设备确定第一隔空手势在x轴方向的第一移动距离；进而，电子设备可根据第一移动距离在第一显示界面中沿x轴方向移动该指示图标。例如，电子设备可根据第一移动距离，按照比例系数k沿x轴方向移动该指示图标，通过移动指示图标向用户提示本次隔空手势在x轴方向的执行进度。
在一种可能的实现方式中，上述预设方向可以为直角坐标系的y轴方向，此时，电子设备沿第一隔空手势在预设方向上的移动轨迹移动第一显示界面中的指示图标，具体包括：电子设备确定第一隔空手势在y轴方向的第二移动距离；进而，电子设备可根据第二移动距离在第一显示界面中沿y轴方向移动该指示图标。例如，电子设备可根据第二移动距离，按照比例系数k沿y轴方向移动该指示图标，通过移动指示图标向用户提示本次隔空手势在y轴方向的执行进度。
在一种可能的实现方式中，在电子设备通过摄像头检测到用户输入第一隔空手势之前，还包括：电子设备检测到用户输入第一预设手势，第一预设手势用于进入隔空手势的操作模式；响应于第一预设手势，电子设备在第一显示界面的预设位置显示上述指示图标，使得用户可获知电子设备已经进入隔空手势的操作模式，从而提醒用户开始输入相应的隔空手势。
在一种可能的实现方式中，电子设备在第一显示界面的预设位置显示指示图标，包括：电子设备沿第一显示界面的顶部或底部显示该指示图标；或者；电子设备沿第一显示界面的侧边显示该指示图标。
在一种可能的实现方式中，上述第一预设手势可以为预设的悬停手势；其中，电子设备检测到用户输入的第一预设手势，包括：电子设备通过该摄像头采集到的图像检测到用户输入该悬停手势。当然，电子设备也可以通过语音等等方式检测用户进入隔空手势的操作模式的操作意图。
在一种可能的实现方式中，电子设备通过摄像头检测到用户输入第一隔空手势，包括：电子设备以上述悬停手势悬停的位置为起始位置，通过摄像头采集到的图像检测用户输入的第一隔空手势。
在一种可能的实现方式中，电子设备显示第一应用的第二显示界面，包括：电子设备根据第一隔空手势在该预设方向上的移动距离向第一应用上报滑动事件；电子设备中的第一应用显示与该滑动事件对应的第二显示界面。
在一种可能的实现方式中，上述方法还包括：当第一隔空手势结束后，电子设备将上述指示图标恢复显示至预设位置，从而提示用户本次输入的第一隔空手势已经结束，电子设备已经完成对第一隔空手势的响应。
在一种可能的实现方式中，上述方法还包括：当第一隔空手势结束后，电子设备可通过摄像头检测到用户输入第二隔空手势；响应于第二隔空手势，电子设备可沿第二隔空手势在预设方向上的移动轨迹移动该指示图标；同时，响应于第二隔空手势，电子设备可显示第一应用的第三显示界面。
在一种可能的实现方式中，电子设备通过摄像头检测到用户输入第二隔空手势，包括：当第一隔空手势结束后，电子设备通过该摄像头采集到的图像检测第一隔空手 势的回程操作；为了避免电子设备识别第二隔空手势时受到第一隔空手势的回程操作的干扰，电子设备可以以该回程操作结束的位置为起始位置，通过摄像头采集到的图像检测用户输入的第二隔空手势。
在一种可能的实现方式中，上述方法还包括：电子设备检测到用户输入第二预设手势，第二预设手势用于退出隔空手势的操作模式；响应于第二预设手势，电子设备隐藏已显示的指示图标，从而提示用户已经退出隔空手势的操作模式。
在一种可能的实现方式中，上述指示图标具体可以为进度条、滑块、线段或弧线；其中，电子设备沿第一隔空手势在预设方向上的移动轨迹移动第一显示界面中的指示图标，包括：电子设备按照第一隔空手势在预设方向上的移动轨迹滑动、延长或放大该指示图标。
第二方面，本申请提供一种电子设备，包括：触摸屏、摄像头、一个或多个处理器、存储器以及一个或多个计算机程序；其中，处理器与触摸屏、摄像头和存储器均耦合，上述一个或多个计算机程序被存储在存储器中，当电子设备运行时，该处理器执行该存储器存储的一个或多个计算机程序，以使电子设备执行上述任一项所述的隔空手势的交互方法。
第三方面，本申请提供一种计算机存储介质，包括计算机指令，当计算机指令在电子设备上运行时，使得电子设备执行如第一方面中任一项所述的隔空手势的交互方法。
第四方面，本申请提供一种计算机程序产品，当计算机程序产品在电子设备上运行时，使得电子设备执行如第一方面中任一项所述的隔空手势的交互方法。
可以理解地，上述提供的第二方面所述的电子设备、第三方面所述的计算机存储介质，以及第四方面所述的计算机程序产品均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
附图说明
图1为本申请实施例提供的一种电子设备的结构示意图一；
图2为本申请实施例提供的一种摄像头的拍摄原理示意图；
图3为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图一；
图4为本申请实施例提供的一种电子设备内操作系统的架构示意图；
图5为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图二；
图6为本申请实施例提供的一种隔空手势的交互方法的流程示意图；
图7为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图三；
图8为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图四；
图9为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图五；
图10为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图六；
图11为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图七；
图12为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图八；
图13为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图九；
图14为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图十；
图15为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图十一；
图16为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图十二；
图17为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图十三；
图18为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图十四；
图19为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图十五；
图20为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图十六；
图21为本申请实施例提供的一种隔空手势的交互方法的应用场景示意图十七；
图22为本申请实施例提供的一种电子设备的结构示意图二。
具体实施方式
下面将结合附图对本实施例的实施方式进行详细描述。
示例性的，本申请实施例提供的一种隔空手势的交互方法可应用于手机、平板电脑、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、手持计算机、上网本、个人数字助理(personal digital assistant，PDA)、可穿戴电子设备、车载设备、虚拟现实设备等电子设备，本申请实施例对此不做任何限制。
示例性的，图1示出了电子设备100的结构示意图。
电子设备100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，摄像头193以及显示屏194等。
可以理解的是，本发明实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
充电管理模块140用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。
电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141可接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，显示屏194，摄像头193，和无线通信模块160等供电。
电源管理模块141可用于监测电池容量，电池循环次数，电池充电电压，电池放电电压，电池健康状态(例如漏电，阻抗)等性能参数。
电子设备100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。
移动通信模块150可以提供应用在电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括一个或多个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(Bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成一个或多个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
在一些实施例中，电子设备100的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得电子设备100可以通过无线通信技术与网络以及其他设备通信。
电子设备100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，电子设备100可以包括1个或N个显示屏194，N为大于1的正整数。
电子设备100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头 被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。在一些实施例中，手机100可以包括1个或N个摄像头，N为大于1的正整数。摄像头193可以是前置摄像头也可以是后置摄像头。
如图2所示，摄像头193一般包括镜头(lens)和感光元件(sensor)，该感光元件可以为CCD(charge-coupled device，电荷耦合元件)或者CMOS(complementary metal oxide semiconductor，互补金属氧化物半导体)等任意感光器件。
仍如图2所示，在拍摄照片或视频的过程中，被拍摄物体的反射光线经过镜头后可生成光学图像，该光学图像投射到感光元件上，感光元件将接收到的光信号转换为电信号，进而，摄像头193将得到的电信号发送至DSP(Digital Signal Processing，数字信号处理)模块进行数字信号处理，最终得到每一帧数字图像。
在本申请实施例中，摄像头193可用于采集用户输入的隔空手势。例如，可将摄像头193设置为常开状态(always on)实时采集当前的图像。摄像头193可将采集到的每一帧图像发送给处理器110。如果从采集到的图像中识别出用户执行了预设的开启手势，说明用户希望使用隔空手势的方式与电子设备进行交互，则处理器110可根据摄像头193实时发来的图像识别此时用户输入的具体隔空手势。
例如，上述预设的开启手势可以为手掌悬停200ms的手势。那么，如果最近200ms内摄像头193采集到的图像中包含用户手掌的图案，且用户手掌没有明显的位置移动，则处理器110可确定用户执行了预设的开启手势。进而，摄像头193可继续将采集的每一帧图像发送给处理器110，处理器110可根据摄像头193发来的多帧图像确定用户本次输入的隔空手势为向上滑动、向下滑动、向左滑动或向右滑动等。需要说明的是，本申请实施例中所述的“上”、“下”、“左”、“右”均为基于用户正常使用手机时的相对方向，可结合具体实施例附图进行理解，本申请实施例对此不做任何限制。
以手机为电子设备举例，可在手机中预先设置第一坐标系，第一坐标系中包括x轴和y轴，x轴为手机正常显示时的水平方向，即用户正常阅读手机显示内容时的水平方向，相应的，y轴为与x轴垂直的竖直方向。例如，x轴为沿手机屏幕短边的水平方向，y轴为沿手机屏幕长边的竖直方向。处理器110可以根据连续多帧图像中用户手掌上某一点的位置变化，确定用户手掌在x轴和y轴上的移动距离。如果用户手掌在x轴上的移动距离大于阈值1，则处理器110可确定用户执行了向左(或向右)滑动的隔空手势。如果用户手掌在y轴上的移动距离大于阈值2，则处理器110可确定用户执行了向上(或向下)滑动的隔空手势。
在本申请实施例中，处理器110不仅可以实时的确定出用户输入的隔空手势，还可以指示显示屏194实时显示该隔空手势的执行进度。以向下滑动的隔空手势举例，处理器110确定出用户执行了预设的开启手势后，如图3所示，处理器110可指示显示屏194在正在显示的界面301中显示滑块302，以提示用户进入隔空手势的操作模 式。在检测到用户输入向下滑动的隔空手势的过程中，处理器110可指示显示屏194实时的向下滑动滑块302。
这样，用户在输入隔空手势的过程中，一方面电子设备可通过滑块302的移动指导用户完成本次隔空手势，另一方面用户可通过滑块302的移动实时的感知到当前隔空手势的执行进度，从而提高了用户与电子设备进行隔空交互时的使用体验，降低隔空手势的误操作率。
同时，处理器110检测到到用户输入的具体隔空手势时，可响应该隔空手势执行对应的翻页、返回、或下一步等操作，本申请实施例对此不做任何限制，从而使得用户可以通过隔空手势控制电子设备实现相应的功能。
需要说明的是，电子设备除了可以通过摄像头193采集用户输入的隔空手势外，还可以通过红外传感器、超声波传感器等一个或多个器件采集用户输入的隔空手势，本申请实施例对此不做任何限制。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。电子设备100可以支持一种或多种视频编解码器。这样，电子设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器121可以用于存储一个或多个计算机程序，该一个或多个计算机程序包括指令。处理器110可以通过运行存储在内部存储器121的上述指令，从而使得电子设备100执行本申请一些实施例中所提供的方法，以及各种功能应用和数据处理等。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统；该存储程序区还可以存储一个或多个应用程序(比如图库、联系人等)等。存储数据区可存储电子设备100使用过程中所创建的数据(比如照片，联系人等)等。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如一个或多个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。在另一些实施例中，处理器110通过运行存储在内部存储器121的指令，和/或存储在设置于处理器中的存储器的指令，来使得电子设备100执行本申请实施例中所提供的方法，以及各种功能应用和数据处理。
电子设备100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
音频模块170用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块170还可以用于对音频信号编码和解码。在一些实施例中，音频模块170可以设置于处理器110中，或将音频模块170的部分功能模块设置于处理器110中。
扬声器170A，也称“喇叭”，用于将音频电信号转换为声音信号。电子设备100 可以通过扬声器170A收听音乐，或收听免提通话。
受话器170B，也称“听筒”，用于将音频电信号转换成声音信号。当电子设备100接听电话或语音信息时，可以通过将受话器170B靠近人耳接听语音。
麦克风170C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当拨打电话或发送语音信息时，用户可以通过人嘴靠近麦克风170C发声，将声音信号输入到麦克风170C。电子设备100可以设置一个或多个麦克风170C。在另一些实施例中，电子设备100可以设置两个麦克风170C，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，电子设备100还可以设置三个，四个或更多麦克风170C，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。
耳机接口170D用于连接有线耳机。耳机接口170D可以是USB接口130，也可以是3.5mm的开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
传感器模块180可以包括压力传感器，陀螺仪传感器，气压传感器，磁传感器，加速度传感器，距离传感器，接近光传感器，指纹传感器，温度传感器，触摸传感器，环境光传感器，骨传导传感器等，本申请实施例对此不做任何限制。
当然，本申请实施例提供的电子设备100还可以包括按键190、马达191、指示器192以及SIM卡接口195等一项或多项器件，本申请实施例对此不做任何限制。
上述电子设备100的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本申请实施例以分层架构的Android系统为例，示例性说明电子设备100的软件结构。
图4是本申请实施例的电子设备100的软件结构框图。
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，安卓运行时(Android runtime)和系统库，以及内核层。
应用程序层可以包括一系列应用程序包。
如图4所示，应用程序层中可以安装通话，备忘录，浏览器，联系人，相机，图库，日历，地图，蓝牙，音乐，视频，短信息等APP(应用，application)。
另外，应用程序层中还包括SystemUI等Android核心应用，SystemUI可用于反馈系统及应用状态并与用户保持交互。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。
如图4所示，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器等。
在本申请实施例中，仍如图4所示，应用程序框架层中还可以包括AI引擎。AI引擎可用于根据摄像头采集到的图像实时识别用户输入的隔空手势。AI引擎识别出用户输入的隔空手势后，还可向应用程序层中的相关应用传递该隔空手势，使得应用能够响应用户输入的隔空手势执行相应的操作。可以理解的是，本申请实施例中对上述AI引擎的具体名称不做限定，可以在应用程序框架层中设置任意名称的功能模块用于 执行上述功能。
示例性的，AI引擎的工作状态可以包括普通模式和隔空手势模式。
在普通模式下，AI引擎可根据摄像头实时采集到的图像确定用户是否输入了隔空手势模式的开启手势，即用户是否有启用隔空手势的操作意图。例如，AI引擎通过调用内核层的摄像头驱动，可获取到摄像头实时采集到的每一帧图像。进而，AI引擎可根据这些图像识别用户是否输入了预设的开启手势。以预设的开启手势可以为手掌悬停200ms的手势举例，AI引擎可不断的获取最近200ms内采集到的N(N＞1)帧图像，并识别这N帧图像中是否包含用户手掌的图案。如果这N帧图像中包含用户手掌的图案，且用户手掌在这N帧图像中的位置移动小于阈值，则AI引擎可确定此时用户输入了开启手势。
当AI引擎确定用户输入了开启手势后，可向应用程序框架层中的system UI上报第一事件，第一事件可用于指示开启隔空手势模式。system UI接收到该第一事件后，如图5中的(a)所示，system UI可在当前的显示界面501的预设位置上显示隔空手势的指示图标502。指示图标502可提示用户已经检测到上述开启手势并开始检测用户输入的隔空手势。其中，指示图标502具体可以为进度条、滑块等图案，本申请实施例对指示图标502的大小、位置、形状以及颜色等显示效果不做任何限制。
同时，当AI引擎确定用户输入了开启手势后，AI引擎从普通模式进入隔空手势模式。
在隔空手势模式下，AI引擎可根据摄像头实时采集到的图像确定用户输入的具体隔空手势。例如，AI引擎通过调用内核层的摄像头驱动，可获取到摄像头实时采集到的每一帧图像。进而，AI引擎可根据这些图像中用户手势的移动距离识别用户输入的隔空手势。例如，AI引擎可将用户执行上述开启手势时用户手掌的位置作为初始位置A，进而，AI引擎可根据实时采集到的图像，以初始位置A为起点计算用户手掌在x轴方向和y轴方向的移动距离。这样，根据用户手掌在x轴方向和y轴方向的移动距离，AI引擎可进一步确定出用户正在输出向上滑动、向下滑动、向左滑动或向右滑动的隔空手势。
以用户输入向左滑动的隔空手势举例，AI引擎确定出用户正在输入向左滑动的隔空手势后，可向system UI和正在前台运行的相册APP上报对应的滑动事件。此时，如图5中的(b)所示，相册APP接收到该滑动事件后，可响应该滑动事件开始显示位于当前图片503之后的下一张图片504。同时，system UI接收到该滑动事件后，可响应该滑动事件向左滑动指示图标502。例如，指示图标502的滑动距离与用户正在输入的隔空手势的移动距离呈正比。
这样一来，用户通过隔空手势控制相关应用的同时，还可以通过显示界面中的指示图标502实时的了解到当前隔空手势的执行进度，手机也可通过指示图标502实时的指导用户如何完成本次隔空手势，从而提高了用户与手机进行隔空交互时的使用体验，降低隔空手势的误操作率。
另外，当用户不再希望使用隔空手势与手机交互时，还可以向摄像头输入隔空手势模式的结束手势。例如，该结束手势可以为用户握拳悬停200ms的手势。那么，AI引擎根据摄像头采集到的图像确定出用户输入了预设的结束手势时，AI引擎可从隔空 手势模式切换为普通模式。同时，AI引擎可向system UI上报第二事件，第二事件可用于指示结束隔空手势模式。system UI接收到该第二事件后，可在当前的显示界面中隐藏上述指示图标502，从而提示用户已经退出隔空手势的操作模式。
另外，上述窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。上述内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。上述视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。上述电话管理器用于提供电子设备100的通信功能。例如通话状态的管理(包括接通，挂断等)。上述资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。上述通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，电子设备振动，指示灯闪烁等。
如图4所示，系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如:MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。2D图形引擎是2D绘图的绘图引擎。
Android Runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。
内核层是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动等，本申请实施例对此不做任何限制。
以手机为电子设备举例，以下将结合附图详细阐述本申请实施例提供的一种隔空手势的交互方法，如图6所示，该方法包括：
S601、手机通过摄像头检测用户输入预设的开启手势。
示例性的，可将手机的摄像头设置为常开状态。也就是说，手机的摄像头可一直处于工作状态，摄像头可按照一定的工作频率采集拍摄范围内的图像。示例性的，上 述摄像头还可以为3D摄像头(也可称为深度摄像头或景深摄像头)，手机通过3D摄像头拍摄的图像可以获取到图像中物体或用户与手机之间的距离。
一般，手机可默认关闭手机的隔空手势功能以节省手机的功耗，此时，手机的摄像头仍然处于工作状态。那么，当用户希望使用隔空手势与手机交互时，如图7所示，可在手机摄像头(例如前置摄像头)的拍摄范围内输入隔空手势的开启手势。例如，该开启手势可以为用户手掌悬停200ms的手势。由于手机的摄像头为常开状态，因此，摄像头可采集到用户手掌悬停时的多帧图像，手机可根据摄像头采集到的多帧图像识别出用户当前是否输入上述开启手势。
仍以用户手掌悬停200ms的手势为开启手势举例，如图8所示，手机可实时的获取最近200ms内摄像头采集到的N帧图像。并且，手机内可预设二维直角坐标系(或空间直角坐标系)。仍如图8所示，以每一帧图像的右上角为原点O建立x轴和y轴举例，x轴和y轴可以以像素为单位，也可以设置mm、cm等距离单位。手机可识别这N帧图像中用户手掌的图案801以及图案801中的参考点C，参考点C可以是图案801中的任一点。例如，参考点C可以是图案801的中心点，或参考点C可以是用户食指的端点等。
如果上述N帧图像中每帧图像内均包含图案801，或上述N帧图像中包含图案801的图像数目大于预设值1，说明用户手掌在这200ms内一直出现在摄像头的拍摄范围内。同时，手机可根据上述N帧图像中参考点C1、C2、……、C(n)的坐标，计算用户手掌的移动距离。如果用户手掌的移动距离小于预设值2(例如，预设值2为50个像素单位)，说明用户手掌在这200ms内一直悬停在摄像头的拍摄范围内，此时，手机可确定用户输入了预设的开启手势。
另外，手机检测到用户输入了预设的开启手势后，还可以记录此时用户手掌所在的位置。例如，手机可将图8中第N帧图像内用户手掌的参考点C(n)的坐标作为用户手掌所在的位置，该位置也可作为后续用户输入隔空手势的起始位置A1。
当然，如果手机确定出上述N帧图像中包含图案801的图像数目小于预设值1，或者，手机确定出用户手掌的移动距离大于预设值2，则说明用户在这200ms内没有输入手掌悬停的开启手势，则手机可继续获取最近200ms内摄像头采集到的N帧图像，并按照上述方法继续识别用户此时是否输入预设的开启手势。
在另一些实施例中，用户也可以通过其他方式触发手机进入隔空手势的操作模式。例如，用户可以通过手机中的语音助手APP输入开启隔空手势的语音信号。进而，手机检测到该语音信号后，可打开摄像头开始采集用户输入的隔空手势。又例如，手机中可设置对应的按键(例如虚拟按键或实体按键)或操作(例如按压操作、双击操作等)用于开启隔空手势的操作模式。当检测到用户点击上述按键或执行上述操作后，手机可打开摄像头开始采集用户输入的隔空手势。另外，手机打开摄像头后可将摄像头首次采集到的用户手掌所在的位置作为后续用户输入隔空手势的起始位置A1。或者，手机打开摄像头后可检测用户是否执行预设的悬停手势，如果检测到用户执行了预设的悬停手势，则手机可将用户悬停的位置作为后续用户输入隔空手势的起始位置A1。
手机检测到用户输入了预设的开启手势后，可响应该开启手势分别执行下述步骤S602和S603。需要说明的是，本申请实施例对下述步骤S602和S603的执行顺序不 做任何限制，手机可同时执行下述步骤S602和S603，也可以先执行步骤S602再执行步骤S603，或先执行步骤S603再执行步骤S602。
S602、响应于上述开启手势，手机在当前的第一显示界面中显示预设的指示图标。
在步骤S602中，手机检测到用户输入了预设的开启手势后，说明用户希望使用隔空手势控制手机中正在运行的应用。为了指导和提示用户高效、正确的输入隔空手势，手机可在正在显示的界面(例如第一显示界面)的预设位置显示预设的指示图标，该指示图标在界面中的位置、长度或形状是可以改变的。
示例性的，以手机正在前台运行电子书APP举例，如图9所示，用户输入上述开启手势时，手机正在显示电子书APP的第一显示界面901。检测到用户输入上述开启手势后，手机可沿水平方向在屏幕顶部显示第一指示图标902。第一显示界面901中出现第一指示图标902后，用户便可获知手机已经进入隔空手势的操作模式，从而提醒用户开始输入相应的隔空手势。
其中，上述第一指示图标902可以是滑块、进度条等图标；上述第一指示图标902可以是直线型也可以是弧线形的图标；本申请实施例对第一指示图标902的具体形状、大小、位置或颜色等显示效果不做任何限制。
示例性的，仍如图9所示，手机可沿第一显示界面901的顶部，在顶部的中间位置显示第一指示图标902，避免第一指示图标902对第一显示界面901中的显示内容产生遮挡。第一指示图标902可在水平方向(即x轴方向)移动，第一指示图标902可用于指示用户输入的隔空手势在水平方向的移动距离。也就是说，手机进入隔空手势的操作模式后，手机可将用户后续输入的隔空手势分解为在x轴方向的滑动操作，并通过移动第一指示图标902向用户提示本次隔空手势在x轴方向的执行进度。
又或者，手机除了在第一显示界面901中显示上述第一指示图标902外，还可以显示文字、箭头或预设图标等指示信息。如图10所示，可以在第一指示图标902的左侧显示第一箭头，并通过文字提示用户向左滑动进行翻页；并且，可以在第一指示图标902的右侧显示第二箭头，并通过文字提示用户向右滑动进行翻页，使得用户可以根据这些指示信息有目的性的执行相应的隔空手势。或者，仍如图9所示，检测到用户输入了预设的开启手势后，手机可以在状态栏内显示预设图标903，从而提醒用户手机已经进入隔空手势的操作模式，可以开始输入相应的隔空手势。当然，手机检测到用户输入了上述开启手势后，还可以通过语音等方式提醒用户进入隔空手势的操作模式，并提示用户输入相应的隔空手势，本申请实施例对此不做任何限制。
在另一些实施例中，如图11所示，如果手机在显示浏览器APP的界面1101时检测到用户输入预设的开启手势，手机可在竖直方向(即y轴方向)上显示第二指示图标1102。例如，手机可沿着显示屏的左侧边(或右侧边)，在左侧边(或右侧边)的中间位置显示第二指示图标1102。第二指示图标1102可在竖直方向(即y轴方向)移动，第二指示图标1102可用于指示用户输入的隔空手势在竖直方向的移动距离。也就是说，手机进入隔空手势的操作模式后，手机可将用户后续输入的隔空手势分解为在y轴方向的滑动操作，并通过移动第二指示图标1102向用户提示本次隔空手势在y轴方向的执行进度。
示例性的，检测到用户输入上述开启手势后，手机还可以根据正在前台运行的应 用类型或正在显示的界面类型确定当前显示的界面(例如界面1101)是否支持上下滑动或左右滑动的操作。如果当前显示的界面支持左右滑动的操作，则手机可在屏幕顶部显示上述第一指示图标902；如果当前显示的界面支持上下滑动的操作，则手机可在屏幕侧边显示上述第二指示图标1102；如果当前显示的界面既支持上下滑动的操作又支持左右滑动的操作，则手机可同时显示上述第一指示图标902和第二指示图标1102，本申请实施例对此不作任何限制。
或者，检测到用户输入上述开启手势后，手机也可以默认在当前的第一显示界面的预设位置(例如屏幕顶部的中央位置)显示指示图标(如图9中所示的第一指示图标902)。后续如果检测到用户输入上下滑动的隔空手势时，手机可将指示图标显示在屏幕侧边(如图11中所示的第二指示图标1102)，并响应该隔空手势移动第二指示图标1102。
在另一些实施例中，如图12所示，可在手机中预先建立显示屏上各个位置与摄像头采集的每帧图像上各个位置之间的对应关系。例如，可将手机显示屏的右上角作为原点O1，并以O1为原点建立第一坐标系。同样，可将摄像头拍摄的每一帧图像的右上角作为原点O2，并以O2为原点建立第二坐标系。第一坐标系中的每一坐标点与第二坐标系中的每一坐标点之间具有映射关系。
当手机检测到用户输入的开启手势时，手机可将开启手势结束时采集到的用户手掌所在的位置(即A1点)确定为用户输入隔空手势的起始位置。进而，手机可确定在第一坐标系中与起始位置A1点对应的B点。也就是说，用户手掌当前所在的位置对应于显示屏中的B点。那么，如图13所示，以手机正在显示桌面1301举例，手机可在桌面1301的B点显示第三指示图标1302，从而提示用户当前手掌所在的位置对应于桌面1301的P点。第三指示图标1302也可随着用户输入的隔空手势在桌面1301中移动。
可以看出，在本申请实施例中，当手机进入隔空手势的操作模式时，手机可在当前的显示界面中显示预设的指示图标，提示用户开始输入隔空手势。后续，手机还可以通过移动该指示图标指导用户输入隔空手势的执行进度，从而提高用户使用隔空手势与手机交互的使用体验。
S603、响应于上述开启手势，手机通过摄像头开始检测用户输入的第一隔空手势。
在步骤S603中，手机检测到用户输入了预设的开启手势后，说明用户希望使用隔空手势控制手机中正在运行的应用，手机进入隔空手势的操作模式。那么，手机可继续使用摄像头实时采集每一帧图像，并且，手机可根据采集到的图像识别用户输入的具体隔空手势。
示例性的，检测到用户输入了预设的开启手势后，手机可以开启手势结束时采集到的用户手掌所在的位置为起始位置A1，进而根据最近采集到的M帧图像实时的确定用户输入的第一隔空手势。
例如，如图14所示，手机进入隔空手势的操作模式后，手机每获取到一帧图像后，均可识别该图像中用户手掌的图案。进而，手机可确定用户手掌在图像中的位置。例如，手机可将用户手掌的中心点A(m)的坐标作为每帧图像中用户手掌所在的位置。这样，手机从起始位置A1的坐标开始，可根据每帧图像中中心点A(m)的坐标计算 出用户手掌在x轴上的移动距离S1以及用户手掌在y轴上的移动距离S2。
另外，如果手机的摄像头为3D摄像头，则用户手掌的起始位置A还包括用户手掌在z轴(即垂直于手机屏幕方向)上的距离，即A1＝(X1，Y1，Z1)。并且，手机采集到的每一帧图像中用户手掌的位置(即中心点A(m))均包括z轴上的距离，即A(m)＝(X(m)，Y(m)，Z(m))。那么，手机除了可以计算出用户手掌在x轴上的移动距离S1以及用户手掌在y轴上的移动距离S2之外，还可以计算出用户手掌在z轴上的移动距离S3。
也就是说，手机将用户正在输入的第一隔空手势可分解为在x轴、y轴和z轴三个方向上的手势操作。
示例性的，当用户手掌在x轴正方向上的移动距离S1大于阈值1时，说明用户输入的第一隔空手势正在向左移动；当用户手掌在x轴负方向上的移动距离S1大于阈值1时，说明用户输入的第一隔空手势正在向右移动。当用户手掌在y轴正方向上的移动距离S2大于阈值2时，说明用户输入的第一隔空手势正在向下移动；当用户手掌在y轴负方向上的移动距离S2大于阈值2时，说明用户输入的第一隔空手势正在向上移动。当用户手掌在z轴方向上的移动距离S3大于阈值3时，说明用户输入的第一隔空手势为按压操作。
又或者，由于第一隔空手势的输入过程是一个连续的输入过程，那么，手机可在x轴、y轴和z轴上设置对应的步长。以手机在x轴上设置的步长为50个像素单位举例，手机可以用户手掌的起始位置A1为起点，根据最近获取的M帧图像实时检测用户手掌在x轴正方向上的移动距离S1。
每当检测到用户手掌在x轴正方向上的移动距离S1增加50个像素单位时，说明用户正在输入的第一隔空手势向左移动了一定距离，此时，手机可向应用程序层中的system UI上报一次向左滑动的滑动事件，使得手机可响应该滑动事件执行下述步骤S604。
另外，每当检测到用户手掌在x轴正方向上的移动距离S1增加50个像素单位时，手机还可以向应用程序层中的正在前台运行的应用(例如上述电子书APP)上报一次向左滑动的滑动事件，使得手机可响应该滑动事件执行下述步骤S605。
也就是说，手机可根据检测到的第一隔空手势在x轴、y轴和z轴三个方向上的移动距离，响应该第一隔空手势执行下述步骤S604-S605。需要说明的是，本申请实施例对下述步骤S604和S605的执行顺序不做任何限制，手机可同时执行下述步骤S604和S605，也可以先执行步骤S604再执行步骤S605，或先执行步骤S605再执行步骤S604。
S604、响应于第一隔空手势，手机按照第一隔空手势的运动轨迹移动上述指示图标。
S605、响应于第一隔空手势，手机显示与第一隔空手势对应的第二显示界面。
以图9中所示的第一指示图标902举例，在步骤S604中，如果检测到第一隔空手势在x轴正方向的移动距离S1大于对应的步长，如图15所示，则手机可同样沿x轴正方向(即向左)延长第一指示图标902。这样，用户在输入第一隔空手势的同时可同步在显示界面中通过第一指示图标902获知第一隔空手势的移动轨迹。
示例性的，手机可根据第一隔空手势在x轴正方向的移动距离S1，按照一定的比例系数k(k＞0)计算手机向左延长第一指示图标902的距离L1。
例如，可预先设置移动距离S1、距离L1、以及比例系数k之间的关系为：L1＝S1*k。那么，当手机检测到第一隔空手势在x轴上的移动距离S1增大或减小时，可根据上述比例系数k计算出第一指示图标902需要移动的距离L1，从而动态的向左或向右延长第一指示图标902。
另外，上述比例系数k的大小可与用户距离手机的距离R成正比。例如，手机可以通过3D摄像头检测用户与手机之间的距离R。进而，手机可根据用户与手机之间的距离R动态的确定上述比例系数k。当用户与手机之间的距离R越大时，可设置对应的比例系数k越大；当用户与手机之间的距离R越小时，可设置对应的比例系数k越小。
同时，如果检测到第一隔空手势在x轴正方向的移动距离S1大于对应的步长，则在步骤S605中，手机还可以向正在运行的电子书APP上报滑动事件，该滑动事件可包括滑动方向和移动距离S1等参数。那么，仍如图15所示，电子书APP响应该滑动事件可执行翻页操作，此时，手机可显示与移动距离S1对应的第二显示界面1501。第二显示界面1501与输入第一隔空手势前显示的第一显示界面901中可以有部分显示内容相同。这样，用户在输入第一隔空手势的同时手机可响应该第一隔空手势控制相关应用实现对应的功能，实现手机与用户进行隔空交互的操作体验。
以图11中所示的第二指示图标1102举例，在步骤S604中，如果检测到第一隔空手势在y轴正方向的移动距离S2大于对应的步长，如图16所示，则手机可按照一定比例向下延长第二指示图标1102。这样，用户在输入第一隔空手势的同时可同步在显示界面中通过第二指示图标1102获知第一隔空手势的移动轨迹和执行进度。
同时，在步骤S605中，手机还可以向正在运行的浏览器APP上报滑动事件，该滑动事件可包括滑动方向和移动距离S2等参数。那么，仍如图16所示，浏览器APP响应该滑动事件可执行翻页操作，此时，手机可显示与移动距离S2对应的显示界面1601，使得手机能够响应用户正在输入的第一隔空手势完成对应的功能。
另外，图16是以浏览器APP举例说明用户在输入上下滑动的隔空手势时手机对该隔空手势的响应过程。可以理解的是，上述浏览器APP还可以为视频类APP等支持上下滑动操作的应用。以抖音APP举例，手机在播放抖音APP的短视频A时，如果检测到用户输入手掌悬停200ms的预设操作，则手机可在短视频A的第一显示界面中显示预设的指示图标，提示用户已经进入隔空手势的操作模式。例如，可将指示图标显示在第一显示界面的顶部。后续，如果通过摄像头检测到用户输入向上滑动的隔空手势，则手机可根据该隔空手势在y轴上的移动距离，将指示图标显示在第一显示界面的侧边，并沿y轴移动该指示图标，以提示用户本次隔空手势在y轴上的执行进度。同时，抖音APP可响应该隔空手势显示短视频B的第二显示界面，从而响应用户输入的向上滑动的隔空手势。
以图13中所示的第三指示图标1302举例，在步骤S604中，如果检测到第一隔空手势在z轴方向的移动距离S3大于阈值3，说明用户希望按压屏幕，如图17所示，手机可按照一定比例放大第三指示图标1302。这样，用户在输入第一隔空手势的同时 可同步在显示界面中通过第三指示图标1302获知第一隔空手势的移动轨迹和执行进度。
同时，在步骤S605中，手机还可以向正在运行的桌面APP上报按压事件，该按压事件中可包括第三指示图标1302所在的位置坐标B点。那么，仍如图17所示，浏览器APP响应该按压事件可打开与B点对应的微信APP，此时，手机可显示微信APP的显示界面1701，使得手机能够响应用户正在输入的第一隔空手势完成对应的功能。
上述实施例中是以延长指示图标(例如上述第一指示图标902、第二指示图标1102)以及放大指示图标(例如上述第三指示图标1302)为显示效果举例说明的。可以理解的是，本领域技术人员可以根据实际经验或实际应用场景设置指示图标响应用户输入隔空手势时的显示效果。例如，手机可通过滑动上述第一指示图标902和第二指示图标1102向用户在显示界面中反馈隔空手势的执行进度，本申请实施例对此不作任何限制。
S606、当第一隔空手势结束时，手机将上述指示图标恢复至预设位置。
仍以图9和图15中所示的第一指示图标902举例，在用户输入第一隔空手势的过程中，手机可按照上述方法实时的计算第一隔空手势在x轴上的移动距离S1，进而实时的按照移动距离S1向左(或向右)移动第一指示图标902。如图18中的所示，当第一指示图标902移动至预设的结束位置，例如屏幕的最左侧或最右侧时，说明本次用户输入的第一隔空手势结束，此时，如图18中的所示，电子书APP响应该第一隔空手势已经翻页至上一页(或下一页)。或者，当检测到第一隔空手势在x轴上的移动距离S1大于预设的门限值(例如500个像素单位)时，说明本次用户输入的第一隔空手势结束，此时电子书APP已经翻页至上一页(或下一页)。
或者，在用户输入第一隔空手势的过程中，如果手机在最近连续多帧图像内没有检测到用户手掌的相关图案，说明用户输入的第一隔空手势已经超出摄像头的拍摄范围，则此时手机可确定本次用户输入的第一隔空手势结束。
又或者，在用户输入第一隔空手势的过程中，如果手机在最近连续多帧图像内检测到用户手掌的位置没有明显变化，说明用户输入的隔空手势出现停顿，则此时手机也可确定本次用户输入的第一隔空手势结束。
当第一隔空手势结束后，仍以第一指示图标902举例，如图19所示，手机可将第一指示图标902恢复至第一指示图标902出现时在第一显示界面中的预设位置，例如屏幕顶端的中间位置，从而提示用户本次的第一隔空手势已经结束，手机已经完成对第一隔空手势的响应。
S607、手机继续通过摄像头检测用户输入的第二隔空手势。
当第一隔空手势结束后，手机还可以通过摄像头继续采集用户输入的第二隔空手势。一般，用户在执行第一隔空手势和第二隔空手势的过程之间会产生第一隔空手势的回程操作。该回程操作是指用户执行第一隔空手势后，沿第一隔空手势移动的反方向执行的手势返回操作。
例如，第一隔空手势为用户向上滑动手掌的操作，第二隔空手势也为用户向上滑动手掌的操作。如图20所示，手机采集到的第一隔空手势的多帧图像中用户手掌沿y轴的负方向移动，在用户开始执行第二隔空手势前，需要先将手掌沿y轴的正方向收 回产生回程操作，再开始第二次向上滑动手掌沿y轴的负方向执行第二隔空手势。
那么，当第一隔空手势结束后，手机可根据采集到的图像实时检测到用户执行的回程操作。例如，当T1时刻第一隔空手势结束后，说明用户开始执行第一隔空手势的回程操作；当手机检测到用户手掌的移动方向从沿y轴的正方向变化为沿y轴的负方向时(即T2时刻)，说明本次回程操作结束。为了避免手机识别第二隔空手势时受到本次回程操作的干扰，手机可将T1-T2时刻采集到的图像丢弃。
并且，手机可将T2时刻采集到的图像中用户手掌的位置作为第二隔空手势的起始位置(例如起始位置为B1)，并根据采集到的图像实时的识别用户输入的第二隔空手势。例如，手机可以起始位置B1的坐标为起点，计算在第二隔空手势中用户手掌在x轴、y轴和z轴三个方向上的移动距离，从而识别用户输入的第二隔空手势。
其中，手机识别用户输入的第二隔空手势的具体方法与步骤S603中手机识别用户输入的第一隔空手势的具体方法类似，故此处不予赘述。
S608、响应于第二隔空手势，手机按照第二隔空手势的运动轨迹移动上述指示图标。
S609、响应于第二隔空手势，手机显示与第二隔空手势对应的显示界面。
与步骤S604-S605类似的，当手机检测到用户输入的第二隔空手势时，一方面，手机可向system UI上报对应的事件(例如滑动事件、按压事件等)，使得system UI响应该事件在当前的显示界面中控制指示图标(例如上述第一指示图标902、第二指示图标1102或第三指示图标1302)按照第二隔空手势的运动轨迹移动。另一方面，手机可向正在前台运行的应用上报对应的事件(例如翻页事件、返回事件等)，使得相关应用能够响应该事件实现对应的应用功能。
另外，当检测到上述第二隔空手势结束后，与步骤S606类似的，手机可再次将当前显示界面中的指示图标恢复至起始位置。后续，手机还可按照上述方法继续识别并响应用户输入的第三隔空手势、第四隔空手势、……。
这样，手机进入隔空手势的操作模式后，手机可连续识别并响应用户连续执行的多次隔空手势，并通过指示图标向用户在显示界面中反馈每次输入的隔空手势的执行进度，从而提高用户使用隔空手势与手机交互的使用体验。
在一些实施例中，手机可以在检测到用户输入开启手势后启动预设的定时器。以10s的定时器1举例，检测到用户输入手掌悬停200ms的开启手势后，手机可启动定时器1，在定时器1未超时的10s内，手机可以按照上述实施例中的方法连续识别并响应用户连续执行的多次隔空手势，并通过指示图标向用户在显示界面中反馈每次输入的隔空手势的执行进度。
当定时器1超时后，手机可自动退出隔空手势的操作模式，不再根据摄像头采集到的图像继续识别并响应用户执行的隔空手势。此时，如果用户希望继续使用隔空手势与手机交互，则用户可再次向手机输入预设的开启手势。那么，手机可继续从上述步骤S601开始执行本申请实施例提供的隔空手势的交互方法。另外，当定时器1超时后，手机可将原本显示出的指示图标隐藏在当前的显示界面中，从而提示用户手机已经退出隔空手势的操作模式。
又或者，当用户不再希望使用隔空手势与手机交互时，还可以向摄像头输入预先 为退出隔空手势模式设置的结束手势。例如，该结束手势可以为用户握拳悬停200ms的手势。
如图21中的(a)所示，如果手机检测到最近200ms采集到的图像中包含用户握拳的图案，且用户握拳的位置在图像中没有发生明显变化，则手机可确定用户输入了预设的结束手势。那么，响应于结束手势，如图21中的(b)所示，手机可在当前的显示界面901中隐藏上述指示图标(例如第一指示图标902)，从而提示用户已经退出隔空手势的操作模式。
本申请实施例公开了一种电子设备，包括处理器，以及与处理器相连的存储器、输入设备和输出设备。其中，输入设备和输出设备可集成为一个设备，例如，可将触摸传感器作为输入设备，将显示屏作为输出设备，并将触摸传感器和显示屏集成为触摸屏。
此时，如图22所示，上述电子设备可以包括：触摸屏2201，所述触摸屏2201包括触摸传感器2206和显示屏2207；一个或多个处理器2202；一个或多个摄像头2208；存储器2203；一个或多个应用程序(未示出)；以及一个或多个计算机程序2204，上述各器件可以通过一个或多个通信总线2205连接。其中该一个或多个计算机程序2204被存储在上述存储器2203中并被配置为被该一个或多个处理器2202执行，该一个或多个计算机程序2204包括指令，上述指令可以用于执行上述实施例中的各个步骤。其中，上述方法实施例涉及的各步骤的所有相关内容均可以援引到对应实体器件的功能描述，在此不再赘述。
示例性的，上述处理器2202具体可以为图1所示的处理器110，上述存储器2203具体可以为图1所示的内部存储器121，上述摄像头2208具体可以为图1所示的摄像头193，上述显示屏2207具体可以为图1所示的显示屏194，上述触摸传感器2206具体可以为图1所示的传感器模块180中的触摸传感器，本申请实施例对此不做任何限制。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请实施例各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：快闪存储器、移动硬盘、只读存储器、随机存取存储器、磁碟或者光盘等各种可以存储程序代码的 介质。
以上所述，仅为本申请实施例的具体实施方式，但本申请实施例的保护范围并不局限于此，任何在本申请实施例揭露的技术范围内的变化或替换，都应涵盖在本申请实施例的保护范围之内。因此，本申请实施例的保护范围应以所述权利要求的保护范围为准。

Claims (28)

  1. 一种隔空手势的交互方法，其特征在于，包括：
    电子设备显示第一应用的第一显示界面；
    所述电子设备通过摄像头检测到用户输入第一隔空手势；
    响应于所述第一隔空手势，所述电子设备沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标；
    响应于所述第一隔空手势，所述电子设备显示所述第一应用的第二显示界面。
  2. 根据权利要求1所述的方法，其特征在于，所述预设方向为直角坐标系的x轴方向，所述x轴方向为所述电子设备显示时的水平方向；
    其中，所述电子设备沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标，包括：
    所述电子设备确定所述第一隔空手势在x轴方向的第一移动距离；
    所述电子设备根据所述第一移动距离在所述第一显示界面中沿x轴方向移动所述指示图标。
  3. 根据权利要求1所述的方法，其特征在于，所述预设方向为直角坐标系的y轴方向，
    其中，所述电子设备沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标，包括：
    所述电子设备确定所述第一隔空手势在y轴方向的第二移动距离；
    所述电子设备根据所述第二移动距离在所述第一显示界面中沿y轴方向移动所述指示图标。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，在所述电子设备通过摄像头检测到用户输入第一隔空手势之前，还包括：
    所述电子设备检测到用户输入第一预设手势，所述第一预设手势用于进入隔空手势的操作模式；
    响应于所述第一预设手势，所述电子设备在所述第一显示界面的预设位置显示所述指示图标。
  5. 根据权利要求4所述的方法，其特征在于，所述电子设备在所述第一显示界面的预设位置显示所述指示图标，包括：
    所述电子设备沿所述第一显示界面的顶部或底部显示所述指示图标；或者；
    所述电子设备沿所述第一显示界面的侧边显示所述指示图标。
  6. 根据权利要求4所述的方法，其特征在于，所述第一预设手势为预设的悬停手势；
    其中，所述电子设备检测到用户输入的第一预设手势，包括：
    所述电子设备通过所述摄像头采集到的图像检测到用户输入所述悬停手势。
  7. 根据权利要求6所述的方法，其特征在于，所述电子设备通过摄像头检测到用户输入第一隔空手势，包括：
    所述电子设备以所述悬停手势悬停的位置为起始位置，通过所述摄像头采集到的图像检测用户输入的所述第一隔空手势。
  8. 根据权利要求1-7中任一项所述的方法，其特征在于，所述电子设备显示所述第一应用的第二显示界面，包括：
    所述电子设备根据所述第一隔空手势在所述预设方向上的移动距离向所述第一应用上报滑动事件；
    所述电子设备中的所述第一应用显示与所述滑动事件对应的第二显示界面。
  9. 根据权利要求4-8中任一项所述的方法，其特征在于，所述方法还包括：
    当所述第一隔空手势结束后，所述电子设备将所述指示图标恢复显示至所述预设位置。
  10. 根据权利要求1-9中任一项所述的方法，其特征在于，所述方法还包括：
    当所述第一隔空手势结束后，所述电子设备通过所述摄像头检测到用户输入第二隔空手势；
    响应于所述第二隔空手势，所述电子设备沿所述第二隔空手势在所述预设方向上的移动轨迹移动所述指示图标；
    响应于所述第二隔空手势，所述电子设备显示所述第一应用的第三显示界面。
  11. 根据权利要求10所述的方法，其特征在于，所述电子设备通过所述摄像头检测到用户输入第二隔空手势，包括：
    当所述第一隔空手势结束后，所述电子设备通过所述摄像头采集到的图像检测所述第一隔空手势的回程操作；
    所述电子设备以所述回程操作结束的位置为起始位置，通过所述摄像头采集到的图像检测用户输入的所述第二隔空手势。
  12. 根据权利要求1-11中任一项所述的方法，其特征在于，所述方法还包括：
    所述电子设备检测到用户输入第二预设手势，所述第二预设手势用于退出隔空手势的操作模式；
    响应于所述第二预设手势，所述电子设备隐藏已显示的所述指示图标。
  13. 根据权利要求1-12中任一项所述的方法，其特征在于，所述指示图标为进度条、滑块、线段或弧线；
    其中，所述电子设备沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标，包括：
    所述电子设备按照所述第一隔空手势在预设方向上的移动轨迹滑动、延长或放大所述指示图标。
  14. 一种电子设备，其特征在于，包括：
    触摸屏，所述触摸屏包括触摸传感器和显示屏；
    一个或多个摄像头；
    一个或多个处理器；
    存储器；
    其中，所述存储器中存储有一个或多个计算机程序，所述一个或多个计算机程序包括指令，当所述指令被所述电子设备执行时，使得所述电子设备执行以下步骤：
    显示第一应用的第一显示界面；
    通过所述摄像头检测到用户输入第一隔空手势；
    响应于所述第一隔空手势，沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标；
    响应于所述第一隔空手势，显示所述第一应用的第二显示界面。
  15. 根据权利要求14所述的电子设备，其特征在于，所述预设方向为直角坐标系的x轴方向，所述x轴方向为所述电子设备显示时的水平方向；
    其中，所述电子设备沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标，具体包括：
    确定所述第一隔空手势在x轴方向的第一移动距离；
    根据所述第一移动距离在所述第一显示界面中沿x轴方向移动所述指示图标。
  16. 根据权利要求14所述的电子设备，其特征在于，所述预设方向为直角坐标系的y轴方向，
    其中，所述电子设备沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标，具体包括：
    确定所述第一隔空手势在y轴方向的第二移动距离；
    根据所述第二移动距离在所述第一显示界面中沿y轴方向移动所述指示图标。
  17. 根据权利要求14-16中任一项所述的电子设备，其特征在于，在所述电子设备通过摄像头检测到用户输入第一隔空手势之前，所述电子设备还用于执行：
    检测到用户输入第一预设手势，所述第一预设手势用于进入隔空手势的操作模式；
    响应于所述第一预设手势，在所述第一显示界面的预设位置显示所述指示图标。
  18. 根据权利要求17所述的电子设备，其特征在于，所述电子设备在所述第一显示界面的预设位置显示所述指示图标，具体包括：
    沿所述第一显示界面的顶部或底部显示所述指示图标；或者；
    沿所述第一显示界面的侧边显示所述指示图标。
  19. 根据权利要求17所述的电子设备，其特征在于，所述第一预设手势为预设的悬停手势；其中，所述电子设备检测到用户输入的第一预设手势，具体包括：
    通过所述摄像头采集到的图像检测到用户输入所述悬停手势。
  20. 根据权利要求19所述的电子设备，其特征在于，所述电子设备通过摄像头检测到用户输入第一隔空手势，具体包括：
    以所述悬停手势悬停的位置为起始位置，通过所述摄像头采集到的图像检测用户输入的所述第一隔空手势。
  21. 根据权利要求14-20中任一项所述的电子设备，其特征在于，所述电子设备显示所述第一应用的第二显示界面，具体包括：
    根据所述第一隔空手势在所述预设方向上的移动距离向所述第一应用上报滑动事件；
    所述第一应用显示与所述滑动事件对应的第二显示界面。
  22. 根据权利要求17-21中任一项所述的电子设备，其特征在于，所述电子设备还用于执行：
    当所述第一隔空手势结束后，将所述指示图标恢复显示至所述预设位置。
  23. 根据权利要求14-21中任一项所述的电子设备，其特征在于，所述电子设备 还用于执行：
    当所述第一隔空手势结束后，通过所述摄像头检测到用户输入第二隔空手势；
    响应于所述第二隔空手势，沿所述第二隔空手势在所述预设方向上的移动轨迹移动所述指示图标；
    响应于所述第二隔空手势，显示所述第一应用的第三显示界面。
  24. 根据权利要求23所述的电子设备，其特征在于，所述电子设备通过所述摄像头检测到用户输入第二隔空手势，具体包括：
    当所述第一隔空手势结束后，通过所述摄像头采集到的图像检测所述第一隔空手势的回程操作；
    以所述回程操作结束的位置为起始位置，通过所述摄像头采集到的图像检测用户输入的所述第二隔空手势。
  25. 根据权利要求14-24中任一项所述的电子设备，其特征在于，所述电子设备还用于执行：
    检测到用户输入第二预设手势，所述第二预设手势用于退出隔空手势的操作模式；
    响应于所述第二预设手势，隐藏已显示的所述指示图标。
  26. 根据权利要求14-25中任一项所述的电子设备，其特征在于，所述指示图标为进度条、滑块、线段或弧线；其中，所述电子设备沿所述第一隔空手势在预设方向上的移动轨迹移动所述第一显示界面中的指示图标，具体包括：
    按照所述第一隔空手势在预设方向上的移动轨迹滑动、延长或放大所述指示图标。
  27. 一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，其特征在于，当所述指令在电子设备上运行时，使得所述电子设备执行如权利要求1-13中任一项所述的隔空手势的交互方法。
  28. 一种包含指令的计算机程序产品，其特征在于，当所述计算机程序产品在电子设备上运行时，使得所述电子设备执行如权利要求1-13中任一项所述的隔空手势的交互方法。
PCT/CN2020/109893 2019-08-19 2020-08-18 一种隔空手势的交互方法及电子设备 Ceased WO2021032097A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/636,529 US12001612B2 (en) 2019-08-19 2020-08-18 Air gesture-based interaction method and electronic device
EP20855678.7A EP4002066A4 (en) 2019-08-19 2020-08-18 Air gesture interaction method and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910765254.5A CN112394811B (zh) 2019-08-19 2019-08-19 一种隔空手势的交互方法及电子设备
CN201910765254.5 2019-08-19

Publications (1)

Publication Number Publication Date
WO2021032097A1 true WO2021032097A1 (zh) 2021-02-25

Family

ID=74603490

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/109893 Ceased WO2021032097A1 (zh) 2019-08-19 2020-08-18 一种隔空手势的交互方法及电子设备

Country Status (4)

Country Link
US (1) US12001612B2 (zh)
EP (1) EP4002066A4 (zh)
CN (2) CN117784927A (zh)
WO (1) WO2021032097A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113095836A (zh) * 2021-04-22 2021-07-09 北京市商汤科技开发有限公司 自助购物方法及装置、电子设备和存储介质
CN113535061A (zh) * 2021-07-20 2021-10-22 深圳市汇春科技股份有限公司 手势控制方法、装置、终端设备及耳机

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210395070A1 (en) * 2020-06-19 2021-12-23 Bunn-O-Matic Corporation Dispensing control system for accommodating persons with disabilities and method
JP7405720B2 (ja) * 2020-09-18 2023-12-26 シャープ株式会社 画像形成装置、制御プログラムおよび制御方法
CN112947755A (zh) * 2021-02-24 2021-06-11 Oppo广东移动通信有限公司 手势控制方法与装置、电子设备及存储介质
CN113138663B (zh) * 2021-03-29 2024-09-13 北京小米移动软件有限公司 设备调节方法、设备调节装置、电子设备及存储介质
CN115484394B (zh) * 2021-06-16 2023-11-14 荣耀终端有限公司 一种隔空手势的引导使用方法及电子设备
CN115484391B (zh) * 2021-06-16 2023-12-12 荣耀终端有限公司 一种拍摄方法及电子设备
CN115904057A (zh) * 2021-08-17 2023-04-04 华为技术有限公司 一种设备控制方法与电子设备
CN116193243B (zh) * 2021-11-25 2024-03-22 荣耀终端有限公司 拍摄方法和电子设备
TWI835053B (zh) * 2022-01-18 2024-03-11 大陸商廣州印芯半導體技術有限公司 手勢感測系統及其感測方法
CN115562472B (zh) * 2022-02-11 2023-09-22 荣耀终端有限公司 手势交互方法、介质及电子设备
CN114690905A (zh) * 2022-03-30 2022-07-01 歌尔股份有限公司 一种手势控制方法、系统、装置、可穿戴产品及介质
CN114840086B (zh) * 2022-05-10 2024-07-30 Oppo广东移动通信有限公司 一种控制方法、电子设备及计算机存储介质
CN115079822B (zh) * 2022-05-31 2023-07-21 荣耀终端有限公司 隔空手势交互方法、装置、电子芯片及电子设备
US12436620B2 (en) * 2022-08-18 2025-10-07 Meta Platforms Technologies, Llc Multi-stage gestures detected based on neuromuscular-signal sensors of a wearable device to activate user-interface interactions with low-false positive rates, and systems and methods of use thereof
US12360608B2 (en) 2022-08-18 2025-07-15 Meta Platforms Technologies, Llc Navigating a user interface using in-air gestures detected via neuromuscular-signal sensors of a wearable device, and systems and methods of use thereof
CN118276670A (zh) * 2022-12-30 2024-07-02 荣耀终端有限公司 隔空手势处理方法、电子设备及存储介质
CN117111727B (zh) * 2023-02-22 2024-12-13 荣耀终端有限公司 手方向的检测方法、电子设备及可读介质
CN118474521B (zh) * 2023-10-26 2025-05-06 荣耀终端股份有限公司 隔空识别方法、电子设备及存储介质
CN117373135B (zh) * 2023-12-07 2024-03-15 湖北星纪魅族集团有限公司 基于视觉的滑动手势识别方法、系统及相关设备
CN119701344B (zh) * 2025-01-13 2026-02-10 网易（杭州）网络有限公司 游戏编辑处理方法、装置、程序产品与电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130141326A1 (en) * 2011-12-05 2013-06-06 Pin-Hong Liou Gesture detecting method, gesture detecting system and computer readable storage medium
CN103246351A (zh) * 2013-05-23 2013-08-14 刘广松 一种用户交互系统和方法
CN104142730A (zh) * 2014-07-04 2014-11-12 华南理工大学 一种手势跟踪结果映射成鼠标事件的方法
CN104407705A (zh) * 2014-11-27 2015-03-11 上海斐讯数据通信技术有限公司 隔空手势操作图片的方法及电子终端
CN106055098A (zh) * 2016-05-24 2016-10-26 北京小米移动软件有限公司 隔空手势操作方法及装置

Family Cites Families (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7692627B2 (en) * 2004-08-10 2010-04-06 Microsoft Corporation Systems and methods using computer vision and capacitive sensing for cursor control
US7725547B2 (en) * 2006-09-06 2010-05-25 International Business Machines Corporation Informing a user of gestures made by others out of the user's line of sight
US7877707B2 (en) * 2007-01-06 2011-01-25 Apple Inc. Detecting and interpreting real-world and security gestures on touch and hover sensitive devices
WO2008137708A1 (en) * 2007-05-04 2008-11-13 Gesturetek, Inc. Camera-based user input for compact devices
JP4840620B2 (ja) * 2009-04-30 2011-12-21 株式会社デンソー 車載用電子機器操作装置
TWI482077B (zh) 2009-06-02 2015-04-21 Htc Corp 電子裝置、其桌面瀏覽方法與電腦程式產品
CN102713794A (zh) * 2009-11-24 2012-10-03 奈克斯特控股公司 用于手势识别模式控制的方法和装置
WO2011119154A1 (en) * 2010-03-24 2011-09-29 Hewlett-Packard Development Company, L.P. Gesture mapping for display device
KR101708696B1 (ko) * 2010-09-15 2017-02-21 엘지전자 주식회사 휴대 단말기 및 그 동작 제어방법
US9619104B2 (en) * 2010-10-01 2017-04-11 Smart Technologies Ulc Interactive input system having a 3D input space
US9766718B2 (en) * 2011-02-28 2017-09-19 Blackberry Limited Electronic device and method of displaying information in response to input
US20120242793A1 (en) * 2011-03-21 2012-09-27 Soungmin Im Display device and method of controlling the same
AU2013205613B2 (en) * 2012-05-04 2017-12-21 Samsung Electronics Co., Ltd. Terminal and method for controlling the same based on spatial interaction
US9696867B2 (en) * 2013-01-15 2017-07-04 Leap Motion, Inc. Dynamic user interactions for display control and identifying dominant gestures
CN103218042A (zh) * 2013-03-27 2013-07-24 苏州百纳思光学科技有限公司 一种用于便携电子设备的远红外手势识别装置
WO2014207288A1 (en) * 2013-06-24 2014-12-31 Nokia Corporation User interfaces and associated methods for controlling user interface elements
CN103558920B (zh) * 2013-11-15 2018-06-19 努比亚技术有限公司 非接触式姿势的处理方法及装置
CN103648008A (zh) * 2013-12-23 2014-03-19 乐视致新电子科技（天津）有限公司 一种利用手势对智能电视进行操作控制的方法和装置
CN105278771A (zh) * 2014-07-25 2016-01-27 南京瀚宇彩欣科技有限责任公司 无挡触控的手持式电子装置、方法及图形化使用者界面
US10015402B2 (en) * 2014-09-08 2018-07-03 Nintendo Co., Ltd. Electronic apparatus
JP6684042B2 (ja) * 2014-09-08 2020-04-22 任天堂株式会社 電子機器
US10042445B1 (en) * 2014-09-24 2018-08-07 Amazon Technologies, Inc. Adaptive display of user interface elements based on proximity sensing
JP6561400B2 (ja) * 2015-02-10 2019-08-21 任天堂株式会社 情報処理装置、情報処理プログラム、情報処理システム、および、情報処理方法
JP6534011B2 (ja) * 2015-02-10 2019-06-26 任天堂株式会社 情報処理装置、情報処理プログラム、情報処理システム、および、情報処理方法
US9804667B2 (en) * 2015-02-10 2017-10-31 Nintendo Co., Ltd. Electronic apparatus
US10521101B2 (en) * 2016-02-09 2019-12-31 Microsoft Technology Licensing, Llc Scroll mode for touch/pointing control
CN107918481B (zh) * 2016-10-08 2022-11-11 深圳巧牛科技有限公司 基于手势识别的人机交互方法与系统
WO2018176014A1 (en) * 2017-03-24 2018-09-27 Inmentis, Llc Social media system with navigable, artificial-intelligence-based graphical user interface with a carousel view
US20180357479A1 (en) * 2017-06-08 2018-12-13 Microsoft Technology Licensing, Llc Body-worn system providing contextual, audio-based task assistance
US20190073040A1 (en) * 2017-09-05 2019-03-07 Future Mobility Corporation Limited Gesture and motion based control of user interfaces
US20200233568A1 (en) 2017-09-30 2020-07-23 Huawei Technologies Co., Ltd. Home screen editing method, graphical user interface, and electronic device
US11393170B2 (en) * 2018-08-21 2022-07-19 Lenovo (Singapore) Pte. Ltd. Presentation of content based on attention center of user
CN109491584A (zh) * 2018-10-17 2019-03-19 深圳传音制造有限公司 一种基于移动终端的屏幕控制方法及一种移动终端
CN109725727A (zh) * 2018-12-29 2019-05-07 百度在线网络技术（北京）有限公司 有屏设备的手势控制方法和装置
CN109725724B (zh) * 2018-12-29 2022-03-04 百度在线网络技术（北京）有限公司 有屏设备的手势控制方法和装置
CN110045819B (zh) * 2019-03-01 2021-07-09 华为技术有限公司 一种手势处理方法及设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130141326A1 (en) * 2011-12-05 2013-06-06 Pin-Hong Liou Gesture detecting method, gesture detecting system and computer readable storage medium
CN103246351A (zh) * 2013-05-23 2013-08-14 刘广松 一种用户交互系统和方法
CN104142730A (zh) * 2014-07-04 2014-11-12 华南理工大学 一种手势跟踪结果映射成鼠标事件的方法
CN104407705A (zh) * 2014-11-27 2015-03-11 上海斐讯数据通信技术有限公司 隔空手势操作图片的方法及电子终端
CN106055098A (zh) * 2016-05-24 2016-10-26 北京小米移动软件有限公司 隔空手势操作方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4002066A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113095836A (zh) * 2021-04-22 2021-07-09 北京市商汤科技开发有限公司 自助购物方法及装置、电子设备和存储介质
CN113535061A (zh) * 2021-07-20 2021-10-22 深圳市汇春科技股份有限公司 手势控制方法、装置、终端设备及耳机

Also Published As

Publication number Publication date
CN112394811A (zh) 2021-02-23
EP4002066A1 (en) 2022-05-25
US20220269351A1 (en) 2022-08-25
US12001612B2 (en) 2024-06-04
CN112394811B (zh) 2023-12-08
EP4002066A4 (en) 2022-09-07
CN117784927A (zh) 2024-03-29

Similar Documents

Publication Publication Date Title
WO2021032097A1 (zh) 一种隔空手势的交互方法及电子设备
JP7391102B2 (ja) ジェスチャ処理方法およびデバイス
US12073071B2 (en) Cross-device object drag method and device
US20240168624A1 (en) Screen capture method and related device
US11921987B2 (en) System navigation bar display method, system navigation bar control method, graphical user interface, and electronic device
EP4057120B1 (en) Application icon display method and electronic device
WO2021000921A1 (zh) 一种基于垂直同步信号的图像处理方法及电子设备
CN111147660B (zh) 一种控件的操作方法及电子设备
CN113132526B (zh) 一种页面绘制方法及相关装置
WO2021036571A1 (zh) 一种桌面的编辑方法及电子设备
WO2021063098A1 (zh) 一种触摸屏的响应方法及电子设备
CN115185440A (zh) 一种控件显示方法及相关设备
WO2021057699A1 (zh) 具有柔性屏幕的电子设备的控制方法及电子设备
CN114356196B (zh) 一种显示方法及电子设备
CN114816200A (zh) 一种显示方法及电子设备
US20240231565A1 (en) Display method, electronic device, storage medium, and program product
WO2023088093A1 (zh) 显示方法和电子设备
CN115016697A (zh) 投屏方法、计算机设备、可读存储介质和程序产品
WO2024037542A1 (zh) 一种触控输入的方法、系统、电子设备及存储介质
CN114527926A (zh) 一种按键操作方法及电子设备
CN114356186B (zh) 一种拖动阴影动画效果的实现方法及相关设备
CN118138813B (zh) 一种光标控制方法和电子设备
EP4517520A1 (en) Display method and electronic device
CN120029502A (zh) 一种隔空手势的交互方法及电子设备
KR20150026395A (ko) 이미지 촬영 방법 및 그 전자 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20855678

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020855678

Country of ref document: EP

Effective date: 20220221