WO2020097830A1 - Procédé de commande de terminal mobile portable et appareil associé - Google Patents

Procédé de commande de terminal mobile portable et appareil associé Download PDF

Info

Publication number
WO2020097830A1
WO2020097830A1 PCT/CN2018/115452 CN2018115452W WO2020097830A1 WO 2020097830 A1 WO2020097830 A1 WO 2020097830A1 CN 2018115452 W CN2018115452 W CN 2018115452W WO 2020097830 A1 WO2020097830 A1 WO 2020097830A1
Authority
WO
WIPO (PCT)
Prior art keywords
terminal
user
gesture
screen
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/115452
Other languages
English (en)
Chinese (zh)
Inventor
叶波
孙忠
李大伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201880099510.5A priority Critical patent/CN113056718A/zh
Priority to PCT/CN2018/115452 priority patent/WO2020097830A1/fr
Publication of WO2020097830A1 publication Critical patent/WO2020097830A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • This application relates to the field of terminals and artificial intelligence (AI), in particular to a handheld mobile terminal control method and related devices.
  • AI artificial intelligence
  • terminals are equipped with a touch screen, and users can manipulate the terminal through touch gestures on the touch screen to use various functions of the terminal. For example, the user can click the application icon on the touch screen to start the application, and can switch the display interface by sliding the finger left and right on the touch screen.
  • the present application provides a handheld mobile terminal control method and related device.
  • the user can input a user gesture to make the terminal perform a corresponding operation when the terminal is turned off or turned on, which can make the terminal control more intelligent and improve the user experience.
  • the present application provides a method for controlling a handheld mobile terminal.
  • the method is applied to a handheld mobile terminal.
  • the handheld mobile terminal has a display screen and a camera.
  • the method may include: the terminal obtains a first image through a camera, and the camera is on the terminal The display of the screen remains on when the screen is off or on to capture the image; the terminal analyzes whether the first image includes a valid user gesture; when the first image includes a valid user gesture, the terminal performs and the effective The operation corresponding to the user's gesture.
  • the user can input the user's gesture to make the terminal perform the corresponding operation when the terminal is turned off or on, which can make the terminal's manipulation more intelligent and improve the user experience.
  • the display screen off means that the display function of the touch screen is completely or partially turned off.
  • the terminal does not turn off the screen, the terminal lights up.
  • Whether the display is lit and whether the terminal is locked are independent of each other. That is, in some embodiments, when the display screen is turned off, the display screen may also be in a locked or unlocked state at the same time. In some embodiments, when the display screen is bright, the display screen may also be in a locked or unlocked state at the same time.
  • the camera is a camera based on the terminal architecture provided by the present application, which can achieve low power consumption and normally open.
  • the camera is used to collect images, which may include still images or dynamic videos, three-dimensional stereoscopic images or planar images.
  • the effective user gesture refers to a gesture for manipulating the terminal to perform the corresponding operation, and a gesture irrelevant to manipulating the terminal to perform the corresponding operation is an invalid user gesture.
  • the effective user gesture may be set by the terminal by default, or may be set by the user.
  • the effective user gesture may be the default setting when the terminal is shipped from the factory.
  • the user may select the user gesture as an effective user gesture from a setting menu provided by the terminal that includes multiple user gestures.
  • effective user gestures may be static or dynamic.
  • effective user gestures may include: palm of hand, back of hand, sliding gesture, gesture of drawing patterns or letters.
  • the effective user gesture may be a finger-only motion, or a combination of finger motion and duration and distance.
  • the user gesture may be the back of the hand provided by the user, or the duration of the back of the hand provided by the user may exceed the specified duration.
  • operations corresponding to valid user gestures are set by the terminal by default, or set by the user.
  • operations corresponding to effective user gestures include at least one of the following: performing an unlock operation, turning on the screen, turning off the screen, turning up / down the volume, turning up / down the screen brightness, turning on / Off flight mode, perform screenshot operation on the current screen display content, start the application or start the internal function of the application.
  • the terminal before the terminal performs an operation corresponding to a valid user gesture, it may verify whether the current user has permission to use the terminal.
  • the terminal When the current user has the authority to use the terminal, the terminal corresponds to the operation corresponding to the effective user gesture.
  • the terminal can verify whether the current user has the authority to use the terminal in the following ways: the terminal uses the camera's second image; recognizes the user's face in the second image; If the pre-stored face information matches, it is determined that the current user has the authority to use the terminal. Verifying whether the user has the authority to use the terminal through face verification can ensure that the terminal is used by authorized users, thereby ensuring the data security of the terminal.
  • the terminal may prompt the user to perform the second permission verification.
  • the terminal may prompt the user to enter verification information. If the verification information entered by the user matches the pre-stored verification information, it is determined that the current user is authorized to use the terminal; if the verification information entered by the user does not match the pre-stored verification information, It is determined that the current user does not have permission to use the terminal.
  • the verification information includes any one of the following: a password, an iris, a fingerprint, and a voice; wherein the pre-stored verification information corresponding to the voice includes: semantic information of the voice and / or voiceprint information of the voice.
  • face verification fails, the terminal prompts the user to perform second verification, which can enable a user with authority to use the terminal.
  • the terminal may call an AI algorithm to analyze whether the first image includes a valid user gesture.
  • the terminal when the terminal invokes the AI algorithm to analyze whether the first image includes valid user gestures, the terminal may input the first image into the AI algorithm model, and the AI algorithm model invokes the corresponding operator analysis in the AI operator library Whether the effective user gesture is included in the first image.
  • the AI operator library is solidified in the hardware of the terminal.
  • the present application provides a handheld mobile terminal, including: a coprocessor, a main processor, a camera, and a display screen, and the coprocessor is connected to the camera and the main processor. among them:
  • the coprocessor is used to obtain the first image through the camera, and the camera remains on when the display is off or bright to collect the image; analyze whether the first image includes valid user gestures; include the effective user in the first image In the case of a gesture, the instruction information of the valid user gesture is sent to the main processor;
  • the main processor is used to perform the operation corresponding to the effective user gesture according to the instruction information.
  • the effective user gesture may be set by the terminal by default, or may be set by the user.
  • the effective user gesture may be the default setting when the terminal is shipped from the factory.
  • the user may select the user gesture as an effective user gesture from a setting menu provided by the terminal that includes multiple user gestures.
  • effective user gestures may be static or dynamic.
  • effective user gestures may include: palm of hand, back of hand, sliding gesture, gesture of drawing patterns or letters.
  • the effective user gesture may be a finger-only motion, or a combination of finger motion and duration and distance.
  • the user gesture may be the back of the hand provided by the user, or the duration of the back of the hand provided by the user may exceed the specified duration.
  • operations corresponding to valid user gestures are set by the terminal by default, or set by the user.
  • effective user gestures may be static or dynamic.
  • effective user gestures may include: palm of hand, back of hand, sliding gesture, gesture of drawing patterns or letters.
  • the effective user gesture may be a finger-only motion, or a combination of finger motion and duration and distance.
  • the user gesture may be the back of the hand provided by the user, or the duration of the back of the hand provided by the user may exceed the specified duration.
  • operations corresponding to effective user gestures include at least one of the following: performing an unlock operation, turning on the screen, turning off the screen, turning up / down the volume, turning up / down the screen brightness, turning on / Off flight mode, perform screenshot operation on the current screen display content, start the application or start the internal function of the application.
  • the main processor is further configured to verify whether the current user has permission to use the terminal before performing operations corresponding to valid user gestures.
  • the main processor is specifically used to execute an operation corresponding to the effective user gesture when the current user has permission to use the terminal.
  • the main processor can be used to verify whether the current user has the authority to use the terminal in the following ways: identify the user's face in the second image; if the user's face in the second image and the face information pre-stored by the terminal If it matches, it is determined that the current user has the authority to use the terminal; wherein, the second image is acquired by the camera.
  • the main processor is further configured to prompt the user to enter verification information if the user's face in the second image does not match the pre-stored face information of the terminal; if the user enters verification If the information matches the pre-stored verification information, it is determined that the current user has permission to use the terminal; if the verification information entered by the user does not match the pre-stored verification information, it is determined that the current user does not have permission to use the terminal.
  • the verification information includes any one of the following: a password, an iris, a fingerprint, and a voice; wherein the pre-stored verification information corresponding to the voice includes: semantic information of the voice and / or voiceprint information of the voice.
  • the coprocessor is specifically used to call an AI algorithm to analyze whether the first image includes a valid user gesture.
  • the coprocessor may include: an AI engine module, an AI algorithm model, an AI operator library, and an AI application layer module. among them:
  • the AI engine module is used to call the corresponding AI algorithm in the AI algorithm model to perform AI calculation on the first image
  • the AI algorithm model is used to call the corresponding AI operator in the AI operator library to analyze whether the first image includes a valid user gesture, and report the recognition result to the AI application layer module;
  • the AI application layer module is used to report the indication information of the effective user gesture to the main controller if the effective user gesture is included in the first image.
  • the AI operator library is solidified in the hardware of the coprocessor.
  • the present application provides a chip that is applied to a handheld mobile terminal.
  • the handheld mobile terminal includes a camera and a display screen.
  • the chip includes: a coprocessor, a main processor, and the coprocessor is connected to the main processor . among them:
  • the coprocessor is used to obtain the first image through the camera, and the camera remains on when the display is off or bright to collect the image; analyze whether the first image includes valid user gestures; include the effective user in the first image In the case of gestures, send valid user gesture instructions to the main processor;
  • the main processor is used to perform the operation corresponding to the effective user gesture according to the instruction information.
  • a computer-readable storage medium having instructions stored on it, which when executed on a computer, causes the computer to execute the method for controlling a handheld mobile terminal described in the first aspect.
  • a computer program product containing instructions, which when executed on a computer, causes the computer to execute the method for controlling a handheld mobile terminal described in the first aspect.
  • the user can input the user's gesture to make the terminal perform the corresponding operation when the terminal is off or on, which can make the terminal's manipulation more intelligent and improve the user experience.
  • FIGS. 1-10 are schematic diagrams of human-computer interaction provided by this application.
  • FIG. 11 is a schematic flowchart of a method for controlling a handheld mobile terminal provided by this application.
  • FIG. 12 is a schematic structural diagram of a terminal 100 provided by this application.
  • FIG. 13 is a schematic diagram of cooperation of various components when the terminal 100 recognizes a user gesture to brighten a screen provided by the present application.
  • first and second are used for description purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.
  • the features defined as “first” and “second” may explicitly or implicitly include one or more of the features.
  • the meaning of “plurality” is two or more.
  • the terminals may be mobile electronic devices such as mobile phones, tablet computers, personal digital assistants (PDAs), wearable devices, and the like.
  • portable electronic devices include, but are not limited to, portable electronic devices equipped with iOS, android, microsoft, or other operating systems.
  • the above portable electronic device may also be other portable electronic devices, such as a laptop with a touch-sensitive surface (for example, a touch panel) and the like.
  • the terminal may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (such as a touch panel).
  • the terminal is configured with a touch screen, which can be used to display the interface content currently output by the terminal's system.
  • the interface content may include the interface of the running application and the system-level menu, etc., and may specifically be composed of the following interface elements: input interface elements, such as buttons, text input boxes, and scroll bars , Menus, etc .; and output interface elements, such as windows, labels, etc.
  • AI technology is a theory, method, technology, and application system that uses digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machine has the functions of perception, reasoning and decision-making.
  • Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.
  • AI artificial intelligence technology With the development of AI artificial intelligence technology, the application of AI technology on terminals is becoming more and more extensive, making the functions of terminals more and more intelligent. For example, with the popularization and promotion of AI technology on terminals, the functions of terminals in the fields of perception, image processing, audio processing, and language processing are becoming more and more powerful.
  • the AI function is integrated into the software system of the terminal, and the AI function basically requires a certain action of the user or a trigger of other application modules to be enabled.
  • the corresponding application module will call the corresponding AI function.
  • the AI function of the terminal will not continue to be normally open. In many cases, the terminal will not perceive the user's behavior, behavior intention, and environmental changes through the AI function.
  • part of the AI function of the terminal can be continuously turned on with low power consumption, and the AI perception capability can be used autonomously.
  • the terminal can sense the change of the user's intention through AI technology in real time and execute the corresponding function, making the terminal more intelligent and man-machine The experience is more comfortable.
  • the terminal realizes the low-power continuous normally-open AI function, reference may be made to the relevant description of subsequent device embodiments, which will not be repeated here.
  • the terminal control method of the present application utilizes the AI function of the terminal's local terminal low power consumption that is constantly on and can control the terminal without the user touching the terminal's touch screen.
  • the AI function at the local end of the terminal refers to various processing capabilities of the terminal based on AI technology.
  • the AI function of the terminal where the low-power consumption of the terminal continues to be normally on may include an image processing function, an audio processing function, and the like.
  • the terminal can complete image processing tasks by using the low-power continuous and normally-open image processing function.
  • Image processing tasks may include: image gesture recognition, image user handprint (fingerprint or palmprint, etc.) recognition, image face recognition, image facial expression recognition, image person iris recognition, image semantic segmentation, etc.
  • the image here may include a still image or a dynamic video, a three-dimensional stereoscopic image or a planar image.
  • the low power consumption of the image processing function continues to be normally open means that the terminal can complete the image collection and processing tasks with low power consumption.
  • Image acquisition can be achieved through the terminal's low-power camera, and image processing tasks can be achieved through the terminal's low-power processor.
  • the terminal specifically implements the low-power continuous normally-open image processing function, reference may be made to the relevant description of the subsequent device embodiments, and details are not described here.
  • Audio processing tasks may include: voiceprint recognition, audio understanding, audio semantic segmentation, and so on.
  • audio understanding refers to the semantic understanding of audio.
  • the low power consumption of the audio processing function continues to be normally open means that the terminal can complete the audio collection and processing tasks with low power consumption. Audio collection can be achieved through the terminal's low-power microphone, and audio processing tasks can be achieved through the terminal's low-power processor.
  • the terminal specifically implements the low-power continuous normally-open audio processing function, reference may be made to the relevant descriptions of subsequent device embodiments, and details are not described here.
  • the application of the low-power camera or the low-power AI function is in consideration of saving power.
  • a conventional camera may also be used, which is not limited in the present invention.
  • a method for controlling a terminal with a floating gesture in the prior art is introduced.
  • the terminal when the terminal is in a bright screen and unlocked state, and the terminal enters certain specific applications, the user can be supported to control the terminal through gestures.
  • the terminal when the terminal is bright and unlocked, and enters the reading application, the terminal can turn on the camera to capture user gestures and perform operations corresponding to the user gestures, such as turning pages.
  • the terminal must be in a bright screen and unlocked state before the user can manipulate the terminal through user gestures, which cannot meet the user's needs to manipulate the terminal through gestures in other scenarios (such as when the terminal is turned off or locked).
  • Application scenario 1 The terminal's low power consumption continues to normally open the image processing function, and the user controls the terminal through user gestures.
  • the user's gesture may be a gesture that does not touch the touch screen, or a gesture that touches the touch screen.
  • the user gesture may be a static gesture or a dynamic gesture.
  • the user gesture may be a hand gesture within a certain range around the terminal.
  • the user gesture may be a hand gesture in the space above the touch screen.
  • the operation performed by the user gestures to control the terminal can be set by the terminal by default, or can be opened to the user in the terminal's setting module and configured by the user freely. That is to say, in some embodiments, user gestures can only control the terminal to perform some operations.
  • the terminal can be configured with a low-power camera.
  • the low-power camera can always be turned on at a specific frame rate to collect images around the terminal in real time, that is, the camera takes images at a certain frequency. It should be understood that the low-power camera can be used as an infrastructure to continuously collect images around the terminal, and provide a hardware basis for the terminal to implement autonomous operation of image processing functions based on AI technology.
  • the camera remains on, or keeps on means that the camera shoots at a certain frequency to obtain image data.
  • This frequency may be different when the terminal where the camera is located is in different application scenarios.
  • the camera may be kept on when the screen is turned off or on, or may be kept on when the screen is turned off and on, and the frequency and the screen on which the camera collects images when the screen is turned off The frequency of image acquisition is different.
  • the status of the terminal includes: whether the terminal's screen is lit, whether the terminal is locked, etc.
  • the terminal may be in a screen-off state.
  • the terminal being in the screen-off state means that the display function of the touch screen is completely or partially turned off.
  • the display function of the touch screen is all turned off means that the touch screen does not display any interface content and no longer consumes power to emit light.
  • the display function of the touch screen is partially turned off, it means that some areas of the touch screen can display some interface elements (such as time, calendar, and notification messages). This area consumes a small amount of power to emit light, and the remaining areas of the touch screen do not consume power to emit light.
  • the touch screen is an organic light-emitting diode (OLED) display screen. Some pixels on the OLED screen are lit for display time. When the remaining pixels are off, the terminal can still be regarded as being off. Screen status.
  • OLED organic light-emitting diode
  • the terminal may be in a bright screen state.
  • the terminal's bright screen state is the opposite of the bright screen state. When the terminal is not in the bright screen state, the terminal is in the bright screen state.
  • the terminal may be in a lock screen state.
  • the terminal is in the lock screen state means that some functions of the terminal are locked and cannot be used. Only after the user enters a password or releases the lock through a preset unlocking method, the locked function can be used.
  • the terminal is in the screen lock state, if the touch screen is lit, the touch screen can display part of the content.
  • the touch screen can display the status bar, date, time, lock screen, and so on.
  • the status bar under the lock screen may include the name of the operator (for example, China Mobile), Wi-Fi icon, signal strength, current remaining power, Bluetooth icon, alarm clock icon, lock screen icon, etc.
  • the functions that can be used have lower requirements for data security.
  • the functions available in the lock screen state may include: answering the phone, hanging up the phone, playing music, pausing music, adjusting the music volume, starting the camera application, turning on / off the airplane mode, and so on.
  • the functions that are locked and cannot be used have high requirements on data security.
  • the functions that are not available in the lock screen state may include: launching applications (such as instant messaging applications, shopping applications, file management applications, photo galleries, etc.) and internal functions of the applications (such as sending messages through instant messaging applications, purchasing goods, Delete pictures in the gallery, etc.).
  • whether the screen is lit and whether the terminal is locked are independent of each other.
  • the terminal When the terminal is in the screen off state, it may be in the screen lock state at the same time, or may be in the state of unlocking.
  • the terminal When the terminal is in the screen lock state, it may be in the state of turning off the screen at the same time, or it may be in the state of turning on the screen.
  • the low-power camera of the terminal can collect images around the terminal in real time.
  • the terminal can analyze whether a specific event occurs according to the collected images. Further, the terminal can call the corresponding AI algorithm to analyze the collected image to determine whether a specific event occurs.
  • the terminal performs the operation corresponding to the specific event, and the user completes the manipulation of the terminal.
  • the following uses a specific event as a gesture of a specific user as an example to describe the operation method of the terminal of the present application in different states.
  • the terminal When the terminal turns off the screen and locks the screen, the user can light the screen through the user's gesture without releasing the lock.
  • the terminal 100 is in a screen-off and screen-locked state, the user's hand is suspended in the space above the touch screen, and the back of the user's hand faces the touch screen.
  • the low-power camera of the terminal 100 is always on, and images are collected in real time.
  • the terminal 100 can call a gesture recognition algorithm to analyze the image to determine that the user gesture input by the user in the image is: the user's hand is suspended in the space above the touch screen, and the back of the hand is facing the touch screen.
  • the touch screen may be lit without unlocking.
  • the touch screen is lit but the lock is not released.
  • the touch screen can display the status bar 110, the date and time 120, the lock screen 130, and so on.
  • the status bar 110 under the lock screen may include the name of the operator (for example, China Mobile), Wi-Fi icon, signal strength, battery icon and remaining power, lock screen icon, and so on.
  • the status bar 110 may further include an alarm clock icon, a Bluetooth icon, and so on.
  • the touch screen may also display the notification message.
  • the terminal 100 may also determine whether to light up the screen without releasing the lock in conjunction with the duration of the back of the hand provided by the user. For example, if the duration that the user provides the back of the hand exceeds a specified duration (for example, 1 second), the terminal 100 determines to light the screen but does not release the lock.
  • a specified duration for example, 1 second
  • the terminal 100 may also determine whether to light up the screen without releasing the lock based on the change in the distance between the back of the user's hand and the touch screen. For example, if the distance between the back of the user's hand and the touch screen is getting larger and larger, the terminal 100 determines to light up the screen without releasing the lock.
  • the terminal 100 can detect the change in the distance between the back of the user's hand and the touch screen in the following ways: (1) The terminal 100 shoots multiple images through a low-power camera, and analyzes the change in the size of the user's back in the multiple images. If the back of the user's hand gradually becomes smaller in the multiple images, it can be confirmed that the distance between the back of the user's hand and the touch screen is getting larger. (2) The terminal 100 determines that the user provides the back of the hand, and can measure the change in the distance between the back of the user's hand and the touch screen through the infrared sensor.
  • the terminal 100 may also determine whether to light the screen without releasing the lock based on the duration of the back of the hand provided by the user and the distance between the back of the hand and the touch screen. For example, if the vertical distance between the back of the user's hand and the touch screen is less than a preset threshold and the duration exceeds a preset length, the terminal determines to light the screen but does not release the lock. That is to say, the terminal may require the user's hand to stay at a position close to the touch screen for a period of time, which can prevent misoperation.
  • the terminal 100 may detect the duration of the back of the hand provided by the user in the following manner: the terminal 100 records the duration of the low-power camera continuously shooting the back of the user's hand, and determines the duration as the length of the back of the hand provided by the user.
  • the terminal 100 detects the change in the distance between the back of the user's hand and the touch screen.
  • the user gesture for lighting the screen without unlocking is not limited to the back of the hand in the embodiment of FIG. 1, but may also be other user gestures.
  • a user's finger suspended above the touch screen can be drawn with an arc based on the center of the finger root, and the user's five fingers suspended above the touch screen can be gripped until spread out.
  • the user gesture may be a finger-only motion, or a combination of finger motion and duration and distance.
  • the user gesture may be that the user's thumb is suspended above the touch screen for more than a specified duration, or the distance between the user's thumb and the touch screen is within a specified range.
  • the user can view the information under the lock screen, such as date, time, notification message, etc.
  • the terminal is still in the lock screen state, which can ensure the data security of the terminal.
  • the terminal When the terminal is in the bright screen state, the terminal may also be in the lock screen state at the same time, or it may not be locked, which is not limited in this application.
  • the user can turn off the screen through user gestures.
  • the user uses the user gesture to turn off the screen as an example for description.
  • the terminal 100 is in a bright screen and unlocked state, and the touch screen may display one or more of a status bar 210, a desktop 220 for displaying installed icons of the terminal, and a navigation bar 230 .
  • the status bar in the unlocked state may include one or more of the following: the name of the operator (for example, China Mobile), Wi-Fi icon, signal strength, battery icon and remaining power, time alarm icon, and Bluetooth icon.
  • the navigation bar 230 may include: a return key icon (triangle in the figure), a home screen icon (circle in the figure), and a multitasking key icon (square in the figure).
  • the user's hand is suspended in the space above the touch screen, and the user's hand palm (also referred to as the palm) faces the touch screen.
  • the low-power camera of the terminal 100 is always on, and images are collected in real time.
  • the terminal 100 can call a gesture recognition algorithm to analyze the image to determine that the user gesture input by the user in the image is: the user's hand is suspended in the space above the touch screen, and the palm of the hand is facing the touch screen.
  • the touch screen may be turned off.
  • the terminal is on and the screen is unlocked, when the user turns off the screen by the user's gesture, the terminal can lock the screen at the same time, or wait for a specified time (for example, 2 seconds) to automatically lock the screen. There is no restriction here.
  • the user can also turn off the screen through user gestures.
  • the user gesture for turning off the screen when the terminal is bright and locked can be the same as the user gesture for turning off the screen when the terminal is bright and unlocked.
  • the bright screen and the locked state of the terminal may be obtained by the user through the method shown in FIG. 1, or the user may press the power button of the terminal, or the terminal may not automatically receive the user input for a long time and the screen is automatically locked but not Turn off the screen to get.
  • the terminal 100 may also determine whether to turn off the screen in combination with the duration of the palm provided by the user. For example, if the duration that the user provides the palm of the hand exceeds the specified duration (for example, 1 second), the terminal 100 determines to turn off the screen.
  • the specified duration for example, 1 second
  • the terminal 100 may also determine whether to turn off the screen based on the change in the distance between the user's palm and the touch screen. For example, if the distance between the user's palm and the touch screen is getting smaller and smaller, the terminal 100 determines to turn off the screen.
  • the way in which the terminal 100 detects the change in the distance between the palm of the user and the touch screen is the same as the way in which the distance between the back of the user's hand and the touch screen is detected when the terminal 100 is in the above (1) state, and reference may be made to the related description.
  • the terminal 100 may also determine whether to turn off the screen based on the duration of the palm provided by the user and the distance between the palm and the touch screen. For example, if the vertical distance between the palm of the user and the touch screen is less than a preset threshold and the duration exceeds a preset length, the terminal determines to turn off the screen. That is to say, the terminal may require the user's palm to stay at a position close to the touch screen for a period of time, which can prevent misoperation.
  • the manner in which the terminal 100 detects the duration for which the user provides the palm of the hand is the same as the manner in which the duration for the user to provide the back of the hand when the terminal 100 is in the above-mentioned (1) state is described in the related description.
  • the way in which the terminal 100 detects the change in the distance between the palm of the user and the touch screen is the same as the way in which the distance between the back of the user's hand and the touch screen is detected when the terminal 100 is in the above (1) state, and reference may be made to the related description.
  • the user gesture for extinguishing the screen is not limited to the palm of the hand in the embodiment of FIG. 2, but may also be other user gestures.
  • the user can use a dynamic user gesture that changes from five fingers spread to five fingers close to turn off the screen.
  • the terminal When the terminal is in the screen lock state, the terminal may also be in the screen-off state at the same time, or may also be in the screen-on state at the same time.
  • Unlocked functions refer to the functions that the terminal can still use under the lock screen.
  • the functions that the terminal can still use under the lock screen include but are not limited to: answering the phone, hanging up the phone, playing music, pausing music, adjusting the audio volume, and starting the camera application.
  • the terminal 100 is in a locked and bright screen state, and the touch screen may display a status bar 110, a date and time 120, a locked screen 130, and a notification message 140 received by the terminal 100 (such as an instant messaging application News) etc.
  • the user's hand is suspended in the space above the touch screen, and an "O" -shaped pattern is drawn by fingers, or an "O" -shaped pattern is laid out by the hand.
  • the low-power camera of the terminal 100 is always on, and images are collected in real time.
  • the terminal 100 After the terminal 100 acquires an image, it can call a gesture recognition algorithm, analyze the image, and determine that the user gesture entered by the user in the image is: the user's hand is suspended in the space above the touch screen, and the finger draws an "O" pattern, or, the hand There is an "O” pattern on the inside.
  • the camera application may be started.
  • the photos in the gallery belong to highly private data
  • the preview window in the lower left corner of the touch screen may not display the photos previously taken by the user.
  • the user can also manipulate the terminal 100 to perform some functions inside the camera through user gestures.
  • This part of the function is a function that can still be used under the terminal lock screen, such as taking pictures.
  • the user's hand can be suspended in the space above the touch screen, and the gesture of pinching with two or more fingers can be used to control the terminal to take a photo (equivalent to pressing the shutter button).
  • the photos taken by the terminal may be displayed in the preview window in the lower left corner of the touch screen.
  • the terminal 100 is in a locked and bright screen state.
  • the user's hand may be suspended in the space above the touch screen to process the communication message 140 received by the terminal 100 through a specific gesture.
  • the user may input a user gesture of sliding the finger to the left.
  • the low-power camera of the terminal 100 can collect an image in real time. After the image is collected, a gesture recognition algorithm can be called to determine that the user gesture input by the user in the image is: a finger slides to the left.
  • the view control 141 and the cancel control 142 of the communication message 140 may be displayed.
  • the user may also input a user gesture simulating the click cancel control 142, and after the terminal 100 confirms that the user inputs the user gesture simulating the click cancel control 141 through image analysis, the corresponding notification message may be canceled from the currently displayed interface content , Such as shown in 4c.
  • the user can also input a user gesture simulating a click view control 141, the terminal 100 can display an interface prompting the user to unlock (for example, a password unlock interface), and the user can view a specific notification message after unlocking.
  • the terminal 100 may first determine that the touch focus simulated by the user is on the cancel control 141 of the touch screen, and then determine that the user's finger performs the simulated click operation on the touch focus.
  • the touch focus is a point where the user's fingertip is projected onto the touch screen.
  • the terminal 100 can detect the user's fingertip through image analysis or infrared detection, and determine the touch focus of the user's fingertip projected onto the touch screen. After determining the touch focus, the terminal 100 may analyze several images that are continuously collected.
  • the preset time period may be set in advance, for example, may be set to 1 second.
  • other processing of the message notification 140 can also be done through user gestures.
  • the user may click the notification message 140 through the user's gesture, and the terminal 100 may display an interface prompting the user to release the lock (for example, a gesture unlock interface).
  • the terminal 100 displays a plurality of notification messages 140
  • the user may slide the notification message 140 up and down through a user gesture, and the terminal 100 may slide and display a plurality of notification messages 140 along with the user gesture.
  • the terminal 100 is in a locked and bright screen state, the terminal 100 is currently receiving a voice call based on the GSM network, and the touch screen displays the screen shown in the left drawing.
  • the user can input the user gesture of swiping to the right.
  • the low-power camera of the terminal 100 can collect an image in real time. After the image is collected, a gesture recognition algorithm can be called to determine that the user gesture input by the user in the image is: slide the finger to the right.
  • the voice call can be accepted, and the user can talk with the other party.
  • the user can also input a user gesture of swiping left to reject the voice call.
  • FIG. 4 to FIG. 5 describe the case where the user controls the terminal to perform the unlocked function through the user ’s gesture when the terminal is in the locked and bright screen state. Understandably, when the terminal is in the locked and off state, The user can also manipulate the terminal to execute the unlocked function through user gestures.
  • the terminal receives a user gesture for manipulating the terminal to perform an unlocked function while the screen is locked and turned off, the screen may be lit first, and then the unlocked function may be executed.
  • the gesture used to control the terminal to execute the unlocked function when the terminal is locked and turned off may be the same as the gesture used to control the terminal to execute the unlocked function when the terminal is locked and turned on.
  • the user gestures for manipulating the terminal to perform the unlocked function are not limited to the examples in the embodiment of FIG. 4 or FIG. 5, but may also be other user gestures.
  • the user can draw the pattern "eight" by hovering the finger to increase the volume of the terminal's audio playback, or the user can reduce the terminal's audio playback by pinching the user's gesture with two or more fingers. volume.
  • the user can use the palm of his hand to face the touch screen to pause media playback, or the user can draw the letter “M” with his finger to continue media playback, or the user can draw the number “3” to switch the terminal ’s current playback. song.
  • the terminal will be described as an example when the terminal is in a locked and bright screen state, and the user controls the terminal to release the lock through a user gesture.
  • the terminal is in a locked and bright screen state, and the touch screen displays a screen as shown in the figure on the left side of FIG. 6.
  • the user's hand is suspended in the space above the touch screen, and an OK gesture is made through the hand.
  • the low-power camera of the terminal 100 is always on, and images are collected in real time. After the terminal 100 collects an image, it can call a gesture recognition algorithm to analyze the image, and determine that the user gesture input by the user in the image is: an OK gesture.
  • the terminal 100 can call a gesture recognition algorithm to analyze the image, and determine that the user gesture input by the user in the image is: an OK gesture.
  • the lock may be released. As shown in the drawing on the right side of FIG.
  • the touch screen may display a desktop for displaying icons of applications installed on the terminal, and may also display a status bar and a navigation bar.
  • the user gesture for unlocking is not limited to the OK gesture shown in FIG. 6, but may be other user gestures.
  • the user can also operate the terminal to unlock the terminal through user gestures.
  • the terminal when the terminal receives the user gesture for manipulating the terminal to unlock while the screen is locked and turned off, the screen may be turned on at the same time, or may remain off.
  • the gesture for controlling the unlocking of the terminal when the terminal is locked and turned off may be the same as the gesture for controlling the unlocking of the terminal when the terminal is locked and bright.
  • the process of unlocking the terminal is actually equivalent to verifying whether the current user has the right to use the terminal. If the current user has the right to use the terminal, the terminal is unlocked.
  • the person who has the right to use the terminal may be the owner or someone close to the owner. Since user gestures are not unique, people who do not have the right to use the terminal may also give user gestures for unlocking. In order to ensure the safety of data, to prevent people who do not have the right to use the terminal from unlocking the terminal and viewing the data of the terminal, This application also provides some other unlocking methods.
  • the face can be unlocked.
  • the user can provide a face to the low-power camera / normal power camera of the terminal, and the camera collects images in real time.
  • the terminal can call the face recognition algorithm to compare the user face in the collected image with the pre-stored user face (the user's face pre-stored by the owner) to determine the face in the image and the pre-stored face Whether it matches. If the face in the image matches the pre-stored face, the terminal can release the lock.
  • biometrics include: iris, palm print, fingerprint, etc.
  • the user can provide biometrics to the low-power camera / normal power camera, and the camera collects images in real time.
  • the terminal can call the iris / palmprint / fingerprint recognition algorithm to compare the biometrics in the collected image with the pre-stored biometrics.
  • the terminal can also collect biometrics in other ways, for example, the terminal can collect fingerprints through a fingerprint sensor. If the collected biometrics match the pre-stored biometrics, the terminal may release the lock.
  • the user can control the mobile phone through touch screen gestures (that is, gestures that directly touch the touch screen) or user gestures or voice.
  • touch screen gestures that is, gestures that directly touch the touch screen
  • voice user gestures or voice.
  • the user can use the gesture to take a picture, send a message, make a call, open a certain sub-interface of the application (such as Alipay's payment interface), manage files, and so on.
  • the user can manipulate the terminal through user gestures.
  • user gestures include but are not limited to: provide palm / hand back gestures, click gestures, press gestures, swipe gestures, and so on.
  • the locked function refers to a function that cannot be used when the terminal is locked.
  • the functions that cannot be used by the terminal in the lock screen state include: launching applications (such as instant messaging applications, shopping applications, file management applications, and galleries, etc.) and internal functions of the applications (such as sending messages through instant messaging applications, purchasing goods, and deleting galleries) Pictures, etc.) etc.
  • the terminal is in a locked and bright screen state, and the touch screen displays a screen as shown in the figure on the left side of FIG. 7.
  • the user's hand is suspended in the space above the touch screen, and the letter "W" is drawn by the finger.
  • the terminal 100 may analyze the collected image and determine that the user gesture input by the user in the image is: drawing the letter "W".
  • the terminal 100 may release the lock and start an application (such as WeChat). That is to say, when the terminal is in the locked and bright screen state, the user can manipulate the terminal to unlock and start an application by drawing a user gesture of the letter “W”.
  • the terminal 100 may prompt the user to unlock, and start an application (such as WeChat) after the user contacts the locked state of the terminal 100.
  • the manner in which the terminal 100 prompts the user to unlock may include: displaying an unlock screen on the touch screen (such as a password unlock screen, a gesture unlock screen, etc.), a voice prompting the user to unlock, a breathing light flashing to prompt the user to unlock, and so on.
  • the user can also manipulate the terminal to execute the locked function through user gestures.
  • the gesture used to control the terminal to execute the locked function when the terminal is locked and turned off may be the same as the gesture used to control the terminal to execute the locked function when the terminal is locked and turned on.
  • the user can manipulate the terminal to unlock, brighten the screen, and start an application by drawing a user gesture of the letter “W”.
  • the user gesture for manipulating the terminal to execute the locked function is not limited to drawing the letter “W” shown in FIG. 7, but may also be other user gestures.
  • the user can control the terminal to start a payment application (for example, Alipay) by drawing a user gesture of the letter “Z”, and the user can also start the payment interface (for example, the payment interface of Alipay) by drawing the letter “P”.
  • user gestures for controlling the terminal can be set in the following ways:
  • the user gesture for manipulating the terminal may be set by the user.
  • the user may select the user gesture as the user gesture for manipulating the terminal from the setting menu provided by the terminal that contains various user gestures.
  • the user gesture for lighting the screen may be selected by the user from the setting menu.
  • the user gesture for extinguishing the screen may be selected by the user from the setting menu.
  • each user gesture may also be selected by the user from the setting menu.
  • the user can also customize user gestures for manipulating the terminal according to their own habits.
  • the user may input a user gesture to the terminal in advance, and specify the role of the user gesture.
  • the following uses the user gesture of drawing the letter “W” in FIG. 7 to start WeChat as an example for description.
  • the setting interface 10 of the terminal 100 may provide an option “custom user gesture” 801, the user may directly touch the touch screen and click the option 701, and the terminal 100 enters the custom user gesture shown in 8b.
  • Interface 20 can provide custom gesture setting options for multiple operations.
  • the multiple functions may include, but are not limited to: performing unlocking operations, turning on the screen, turning off the screen, turning up / down the volume, turning up / down the screen brightness, and answering calls , Hang up the phone, turn on / off the flight mode, take a screenshot of the current screen display, start the application or start the internal function of the application.
  • the application may be WeChat, Alipay and other applications, and the internal function of the application may be to open the payment interface of Alipay and so on.
  • the user can directly touch the touch screen and click the option 802, and the terminal 100 enters the recording interface 30 of the user gesture as shown in 8c. While displaying the recording interface 30 shown in 8c, the terminal 100 turns on the low power camera / normal power camera.
  • the user may input a user gesture for drawing the letter “W” according to his own habit.
  • the terminal 100 records the user gesture through a low-power camera / normal power camera and stores the user gesture. After the terminal 100 stores the user gesture, the user completes the custom user gesture for starting WeChat.
  • each user gesture can also be user-defined.
  • the user gestures set by the user to control the terminal are more in line with the user's usage habits, and the user experience is even better.
  • the user gesture for manipulating the terminal may be set by the terminal.
  • the user gesture used to manipulate the terminal may be the default setting when the terminal is shipped from the factory.
  • user gestures for manipulating the terminal may also be determined in other ways.
  • the user gesture used to manipulate the terminal may also be randomly set by the terminal, which is not limited in this application.
  • the present application provides an operation prompt for prompting the user how to make the terminal perform the corresponding operation based on the user gesture.
  • the terminal can simulate and display the user's touch focus.
  • the terminal can simulate the point projected by the user's fingertip on the touch screen as the touch focus of the user, and the user can input user gestures such as click, press, or slide under the instruction of the touch focus.
  • the point projected by the fingertip of the user on the touch screen is simulated as the touch focus 901
  • the current touch focus 901 is located on the WeChat icon on the desktop displayed by the terminal.
  • the touch focus 901 may be a circle.
  • the user may input a user gesture under the instruction of the touch focus, for example, a user gesture of clicking may be directly input to start WeChat, or a finger may be moved to shift the touch focus 901.
  • the terminal may project the user's fingertip onto the touch screen, and the user may click the cancel control 142 according to the current touch focus to cancel displaying the first message in the notification message 140.
  • the terminal may prompt the user how to manipulate the terminal through user gestures.
  • FIG. 10 exemplarily shows an operation prompt provided by the present application.
  • the terminal currently opens an application for reading a book, and the touch screen can display prompt information for a user gesture prompting the user to turn pages.
  • the prompt information may be a translucent palm 1001 displayed on the touch screen. After seeing the prompt message, the user can learn that when the hand is changed from the palm of the hand toward the touch screen to the back of the hand toward the touch screen, the terminal can be triggered to turn pages.
  • the time when the translucent palm 1001 disappears may include: 1. When a user gesture for manipulating the terminal to turn pages is detected (such as changing the hand from the palm toward the touch screen to the user operation from the back of the hand to the touch screen ); 2. When it is detected that the user's finger touches the touch screen (such as the user's finger tapping the touch screen); 3. When the time that the translucent palm 1001 is continuously displayed exceeds a certain length of time (such as 2 seconds).
  • the terminal may also give operation prompts in other scenarios.
  • the terminal can give an operation prompt to prompt the user how to perform the corresponding operation using the terminal based on the user gesture.
  • the terminal may perform an operation corresponding to the user gesture.
  • the terminal since the user gesture and the user are not the only corresponding relationship, in order to ensure the data security of the terminal, before performing the operation corresponding to the user gesture, it can be verified whether the current user is entitled to use the terminal. After verifying that the current user has the right to use the terminal, the terminal will perform the corresponding operation according to the recognized user gesture, such as turning on the screen but keeping it locked, turning off the screen, and starting the application.
  • the terminal can verify whether the current user has permission to use the terminal through the face.
  • the terminal may acquire an image through a low-power camera to identify the user's face in the image; if the user's face in the image matches the pre-stored face information of the terminal, it is determined that the current user has permission to use the terminal.
  • the terminal may pre-store face information of a user who is authorized to use the terminal.
  • the terminal may prompt the user to enter the verification information. That is to say, after face verification fails, the terminal can provide users with the opportunity of secondary verification. If the verification information entered by the user matches the pre-stored verification information, it is determined that the current user has permission to use the terminal; if the verification information entered by the user does not match the pre-stored verification information, it is determined that the current user has no permission to use the terminal.
  • the verification information includes any one of the following: password, iris, fingerprint and voice.
  • the pre-stored verification information corresponding to the sound includes: semantic information of the sound and / or voiceprint information of the sound.
  • the terminal may pre-store voiceprint information of a user who is authorized to use the terminal, or the terminal may pre-store voice semantic information used to verify the user's authority.
  • the voiceprint information of a user who is authorized to use the terminal can be entered into the terminal by the user, and the voice semantic information used to verify the user's authority can be set by the terminal by default or can be customized by the user.
  • the terminal can perform an operation corresponding to the recognized user gesture.
  • the terminal first performs face verification on the user, and after verifying that the current user has the authority to use the terminal, performs the operation corresponding to the recognized user gesture.
  • the terminal in the embodiment of FIG. 11, the terminal first unlocks the identity of the owner of the face verification machine, and then controls the terminal through user gestures.
  • the method may include the following steps:
  • the user places the face in the image collection area of the low-power camera, and the terminal collects the image through the low-power camera.
  • the terminal collects an image through a low-power camera, and the image may contain the user's face.
  • the terminal recognizes the face contained in the collected image through a face recognition algorithm and verifies the owner's identity.
  • the terminal may first perform face detection on the collected image through a face detection algorithm. If it is detected that the user's facial contour features in the image are complete and conform to the facial features, it can be determined that a human face appears in the image collected by the low-power camera.
  • the terminal can determine whether the facial features in the collected image match the pre-stored facial image of the owner through a facial recognition algorithm. If they match, the identity of the owner is successfully verified and the subsequent steps are performed. If they do not match, you can perform step 1 again.
  • the user provides user gestures, and the terminal collects images through a low-power camera.
  • the terminal collects an image through a low-power camera, and the image may include user gestures input by the user.
  • the terminal recognizes the gesture contained in the collected image through a gesture recognition algorithm.
  • the terminal may first perform gesture detection on the collected image through a gesture detection algorithm. If it is detected that the contour features in the image match the contour features of the human hand, it can be judged that gestures appear in the image collected by the low-power camera.
  • the terminal can recognize the user gesture in the collected image through a gesture recognition algorithm. For example, the terminal can recognize whether the user input is the palm of the hand, the back of the hand, or the OK gesture through a gesture recognition algorithm.
  • the terminal performs the corresponding operation according to the recognized gesture.
  • the terminal may determine whether the user gesture in the collected image is a valid user gesture. If so, the corresponding operation is performed according to the recognized gesture. If not, the terminal may perform step 3 again.
  • the effective user gesture refers to a gesture used for manipulating the terminal to perform the corresponding operation, and a gesture that is not related to manipulating the terminal to perform the corresponding operation is an invalid user gesture.
  • the effective user gesture can be set by the terminal by default, or can be set by the user independently.
  • the terminal may store one or more valid user gestures in advance, and determine whether the collected image contains valid user gestures based on the stored user gestures.
  • FIGS. 1 to 7 an example to illustrate that the terminal performs corresponding operations according to valid user gestures.
  • the terminal recognizes that the user provides the back of the hand, and can light the screen but keep the locked state.
  • the terminal may turn off the screen.
  • the terminal after the terminal recognizes that the user draws an “O” pattern with a finger, it can start the camera application; after the terminal recognizes the gesture of the user swiping left, it can manipulate the notification message displayed on the touch screen.
  • the terminal after receiving the voice call request, the terminal can receive the voice call if it recognizes the gesture of the user swiping to the right.
  • the terminal may release the screen lock.
  • the terminal may start WeChat.
  • the terminal can collect user images in real time through a low-power camera, and can autonomously run AI perception capabilities.
  • the terminal can autonomously recognize the gesture and perform the corresponding operation according to the user's gesture input by the user.
  • the user may manipulate the terminal without directly touching the touch screen. This application can make the operation of the terminal more intelligent and improve the user experience.
  • Application scenario 2 The terminal's low power consumption continues to open the audio processing function, and the user controls the terminal through voice.
  • the operation that the voice can control the terminal can be set by the terminal by default, or it can be opened to the user in the terminal's setting module and can be freely configured by the user. That is to say, in some embodiments, the voice can only control the terminal to perform some operations.
  • the terminal can be configured with a low-power microphone.
  • the low-power microphone can be turned on all the time to collect audio data around the terminal in real time. It should be understood that the low-power microphone can be used as an infrastructure to continuously collect audio around the terminal, and provide a hardware basis for the terminal to implement the audio processing function based on AI technology autonomously.
  • the low-power microphone of the terminal can collect audio around the terminal in real time regardless of the state of the terminal.
  • the terminal can analyze whether a specific event occurs according to the collected audio. Further, the terminal can call the corresponding AI algorithm to analyze the collected audio to determine whether a specific event occurs.
  • the terminal performs the operation corresponding to the specific event, and the user completes the manipulation of the terminal.
  • the following uses a specific event as a specific voice as an example, and describes the manipulation method in the application scenario 2 in combination with different states of the terminal.
  • the terminal When the terminal turns off the screen and locks the screen, the user can light the screen through the user's gesture without releasing the lock.
  • the terminal when the terminal turns off and locks the screen, the user can input the voice "light up the screen".
  • the low-power microphone of the terminal is always on and collects audio in real time. After collecting audio, the terminal can call a voice recognition algorithm to analyze the audio, and determine that the voice input by the user is "lighting the screen.” After the terminal determines the voice input by the user, it can light up the touch screen according to the voice without releasing the lock.
  • the terminal When the terminal is in the bright screen state, the terminal may also be in the lock screen state at the same time, or it may not be locked, which is not limited in this application.
  • the user can turn off the screen by voice.
  • the terminal when the terminal is on, the user can input the voice "turn off the screen".
  • the low-power microphone of the terminal is always on and collects audio in real time. After collecting audio, the terminal can call a voice recognition algorithm to analyze the audio, and determine that the voice input by the user is "turn off the screen.” After determining the voice input by the user, the terminal may turn off the screen according to the voice point.
  • the terminal When the terminal is in the screen lock state, the terminal may also be in the screen-off state at the same time, or may also be in the screen-on state at the same time.
  • the user may input a voice "increase volume" to control the terminal to increase the volume of playing media.
  • the user may input a voice "unlock” to control the terminal to unlock.
  • the voiceprint can also be used for unlocking.
  • the user can provide voice to the low power consumption microphone / normal power consumption microphone, the microphone collects voice in real time, and the terminal calls the voiceprint recognition algorithm to compare the voiceprint corresponding to the collected voice with the stored voiceprint. If the voiceprint corresponding to the voice matches the pre-stored voiceprint, the terminal may release the lock.
  • the user can control the mobile phone through touch screen gestures (that is, gestures that directly touch the touch screen) or user gestures or voice.
  • touch screen gestures that is, gestures that directly touch the touch screen
  • voice user gestures or voice.
  • the user can use voice to take a photo, send a message, make a phone call, open a certain sub-interface of the application (such as Alipay's payment interface), and manage files.
  • the user may input a voice "start WeChat" to control the terminal to start WeChat.
  • the above voice for controlling the terminal is not limited to the voices in the above examples (1) (2) (3), but may also be other voices, which are not limited here.
  • the voice used to control the terminal in this application may be set in the following ways: (1) The voice used to control the terminal may be set by the user. For example, the user may select a voice as a voice for controlling the terminal from a setting menu provided by the terminal that contains multiple voices, or a user-defined voice according to his own habits. (2) The voice used to control the terminal may be set by the terminal. For example, the voice used to control the terminal may be the default setting when the terminal leaves the factory.
  • the terminal can collect user audio data in real time through a low-power microphone, and can autonomously run AI perception capabilities.
  • the terminal can autonomously perform voice recognition and perform corresponding operations according to the voice input by the user.
  • This application can make the operation of the terminal more intelligent and improve the user experience.
  • the terminal of the present application may include: a main processor, a coprocessor, and a normally-open low-power camera.
  • the coprocessor and low-power camera in the terminal have always been low-power and always on.
  • the main processor in the terminal sleeps normally and enters a low power consumption mode.
  • the coprocessor in the terminal reports the AI event message, the main processor is woken up.
  • the main processor can implement various business functions according to product business requirements, or pass event messages to other related business modules, and other business modules complete the final processing.
  • the terminal may report the AI recognition result to the main controller if it is determined that a specific event occurs.
  • the coprocessor may generate an AI message at the AI application layer, and may report the AI message to the main controller. This will be described in detail below in conjunction with FIG. 12, and will not be repeated here.
  • the coprocessor in the terminal can analyze whether the user can be detected in the image according to the collected image and the corresponding AI algorithm (such as a gesture recognition algorithm) gesture.
  • the coprocessor can also recognize the user's user gesture and report the recognized user's user gesture to the main controller.
  • the main controller may be responsible for running various applications of the terminal, including but not limited to: user interface (user interface), human interaction interface, gesture recognition, face recognition, environment recognition, screen off, etc.
  • FIG. 12 is a schematic block diagram of a hardware architecture of a terminal 100 provided by an embodiment of the present application.
  • the hardware architecture shown in FIG. 12 may include a main processor 210, a coprocessor 220, and a low-power camera 230.
  • Coprocessor 220 Integrated AI capabilities, can continue to run in low-power mode to detect user action intentions and environmental changes.
  • the coprocessor 220 is connected to the main processor 210, and when a corresponding event is detected, the main controller 210 is triggered to wake up by reporting an AI event message to the main processor 210.
  • Main processor 210 When there is no service, the main controller 210 system can perform a normal sleep standby state and enter a low power consumption mode. After receiving the AI event message sent by the coprocessor 220, after the main processor 210 is woken up, it receives the event reported by the coprocessor 220 and triggers the corresponding business scene function.
  • Low-power normally-on camera 230 Connected to the coprocessor 220 through a peripheral chip software interface (driver) provided by the coprocessor 220, and provides a data source for the coprocessor 220 to process AI services.
  • driver peripheral chip software interface
  • the coprocessor 220 module may be a real-time operating system (RTOS).
  • RTOS real-time operating system
  • the results of its processing can control the production process or respond quickly to the processing system within the prescribed time, and schedule all available resources to complete real-time tasks. And control all the real-time tasks coordinated operation of the operating system, fast response and high reliability.
  • the RTOS system of the coprocessor 220 may include: a kernel 221, a framework layer (framework layer) 222, and an APP application layer 223.
  • the kernel 221 includes a peripheral driver module 2211, a hardware acceleration module 2212, and an AI operator library module 2213.
  • Framework layer 222 includes: AI application management module 2221, AI algorithm management module 2222, and AI algorithm model 2223.
  • the APP application layer 223 includes: an AI application layer module 2231, an AI engine module 2232, and an AI model management module 2233.
  • Peripheral driver module 2211 It can provide a software interface for connecting various peripheral chips.
  • a low-power camera 230 may be connected, and the low-power camera 230 may provide a hardware basis for the coprocessor 220 to perceive user behavior intentions or environmental changes.
  • the coprocessor 220 can analyze the characteristics of the user's actions and surrounding environment according to the image collected by the low-power camera 230, and provide a data source for the coprocessor 220 to process AI services.
  • the terminal may acquire the image in real time through the normally-open low-power camera 230 connected to the peripheral drive module 2211.
  • peripheral devices that can be connected to the peripheral drive module 2211 may also include, but are not limited to: sensors (which can be used to identify user actions), low-power microphones (which can be used to analyze user voice and other features ), Location sensors (for example, global positioning system (GPS), wireless local area network (WIFI), modem (modem), which can be used to provide user location information).
  • GPS global positioning system
  • WIFI wireless local area network
  • modem modem
  • the coprocessor in the terminal may analyze whether the user's voice is detected in the audio according to the collected audio and the corresponding AI algorithm (such as a speech recognition algorithm). The coprocessor can also recognize the user's voice (that is, understand the semantics of the voice), and report the recognized voice to the main controller.
  • AI algorithm such as a speech recognition algorithm
  • AI application management module 2221 It can classify the data reported by the peripheral drive module 2211. For example, the received data is divided into image categories, video categories, audio categories, etc., so as to call AI algorithm models 2223 of different categories for analysis and processing.
  • AI engine module 2232 it can be responsible for scheduling and coordinating AI algorithm model 2223 for operation. Since there can be multiple AI algorithm models 2223 running at the same time, the scheduling management control of the AI engine module 2232 can ensure that the software runs in an orderly manner.
  • AI algorithm management module 2222 responsible for algorithm management, according to the different types of data reported by the AI application management module 2221, the corresponding AI algorithm model can be selected from a plurality of running AI algorithm models 2223 for analysis.
  • AI algorithm model 2223 It can be a set of algorithm features that conform to the image and sound of certain services.
  • the AI algorithm model 2223 may be a set that matches a certain gesture feature.
  • the AI algorithm model 2223 may be a set that conforms to the contour features of the face.
  • the AI algorithm model 2223 may be a set of features that match a certain voice.
  • the AI algorithm model 2223 can be trained through large-scale images. After the training is completed, an algorithm model can be generated, and the corresponding AI operator can run the algorithm model to perform gesture recognition or face recognition and other operations.
  • the co-processing 220 in the terminal may receive the image reported by the normally-open low-power camera 230, and the AI application management module 2221 may call the corresponding AI algorithm through the AI engine module 2232 to analyze the user in the collected image gesture.
  • AI algorithm model 2223 may be integrated into the software system by default, or may be updated into the coprocessor 220 through the main controller 210, which is not specifically limited in this embodiment of the present application.
  • the main controller 210 may also optimize the AI algorithm model 2223. For example, positioning information such as GPS / WIFI / modem can be used to comprehensively judge the results of the AI algorithm model 2223 to improve the accuracy of the AI algorithm model 2223.
  • the AI model management module 2233 can modify certain features in the AI algorithm model 2223.
  • AI operator library module 2213 The AI engine module 2232 can run the AI model management module 2233 to perform gesture recognition or face recognition operations by calling the operator in the AI operator library module 2213. Due to the limited resources of the coprocessor 220, the AI operator library module 2213 that designs a large number of mathematical calculations can be solidified in hardware, and most of the AI operators can be implemented by the hardware, which can avoid the high processor load generated by the software implementation operator. The interface of the hardware curing operator can be provided by the kernel 221 to the AI model management module 2233 for use.
  • curing the AI operator library module 2213 in the hardware may be writing the software on the coprocessor chip, and the programming software may be run through the coprocessor chip.
  • Software curing is to make software on silicon chips (so-called firmware) to realize software functions, so that the complexity of the operating system and language processing is shared by both hardware and software.
  • the AI operator library module 2213 is fixed on the hardware of the coprocessor.
  • the operation of the software curing can increase the operation speed of the entire system, improve reliability, reduce costs, and facilitate large-scale production and standardization.
  • Hardware acceleration module 2212 The process of running the AI model management module 2233 can be accelerated by calling the operator in the AI operator library module 2213 to the AI engine module 2232 through the acceleration mode. It can ensure that the AI engine module 2232 can quickly call the operators in the AI operator library module 2213 in real time, and provide capability interfaces for various AI algorithms in the framework model 222AI model management module 2233.
  • AI application layer module 2231 It can be located in the APP application layer 223, and can implement various continuous AI applications in the APP application layer 223 according to the scene requirements of the terminal service design.
  • the AI application layer module 2231 can call various algorithms to obtain AI recognition results of peripheral connected devices, and can report corresponding AI event messages to the main controller 210. If the main controller 210 is in a dormant state, it can perform secondary processing on the AI event message after being awakened.
  • the AI application management module 2221 reports the gesture detection result to the AI application layer module 2231. After obtaining the recognition result, the AI application layer module 2231 will form a recognition event message, and report the recognition event message to the AI event message manager 212 in the main controller 210.
  • the system architecture of the main processor 210 is described in detail below.
  • Main processor 210 responsible for running various applications of the terminal, including UI human-computer interaction interface, and cloud interaction. When there is no business, the main controller system sleeps normally and enters a low power consumption mode.
  • the main processor 210 may include: AI native (AI native) 211, AI event message manager (AI service) 212, application (application, APP) 213, APP 214, APP 215.
  • AI local (AI) 211 The AI event message reported by the coprocessor 220 can be received, and the main controller 210 is woken up.
  • the AI algorithm model 2223 optimized by the main controller 210 may also be sent to the AI engine module 2232 of the coprocessor 220, and the AI engine module 2232 may update the AI algorithm model 2223 through the AI model management module 2233.
  • AI event message manager (AI) service 212 It can receive AI event messages reported by AI native 211, and manage the AI capability interface of the terminal in a unified manner, and provide an AI application program interface (application program interfae, API) for each business module. According to product business needs, realize various business functions. For example, different business functions can be realized according to different applications (APP213 or APP214 or APP215).
  • the main controller 210 is woken up.
  • the main controller 210 may determine the operation corresponding to the user gesture data detected in the image reported by the low-power camera 230. After determining the terminal operation corresponding to the user gesture detected in the image reported by the low-power camera 230, the operation may be performed (for example, turning on the screen, unlocking, starting the application, etc.).
  • the AI service 212 can also transfer the data to the cloud to complete the low-power service processing mode of combining the terminal and the cloud.
  • the main frequency of the coprocessor is low
  • the AI operators involved in a large number of mathematical operations are integrated in a hardware-hardened manner
  • the peripheral devices are low-power devices, which can be operated in a low-power mode Normally open and run AI perception capabilities, so that the terminal can not rely on specific actions, can sense changes in user actions or changes in the environment.
  • FIG. 13 is a schematic flowchart of a terminal recognizing a user gesture bright screen scene provided by an embodiment of the present application.
  • the method shown in FIG. 13 may include the following steps:
  • the low-power camera 230 collects images.
  • the low-power camera 230 connected to the coprocessor 220 continuously collects images around the terminal.
  • the image collected by the low-power camera 230 includes the back of the hand provided by the user.
  • the low-power camera 230 reports the collected image to the coprocessor 220.
  • the low-power camera can report the collected image to the AI application management module 2221.
  • the coprocessor 220 invokes a gesture recognition algorithm model to recognize user gestures.
  • the coprocessor 220 may call the gesture detection algorithm model before step 2 to confirm whether the user gesture appears in the image collected by the low-power camera 230. Specifically, if it is detected that the contour feature in the image matches the contour feature of the human hand, it can be determined that a user gesture appears in the image collected by the low-power camera.
  • the AI application management module 2221 can analyze the user gesture in the image according to the data to be processed, and call the corresponding AI gesture recognition algorithm model 2223 through the AI engine module 2232 Analyze user gestures in the collected images.
  • the AI gesture recognition algorithm model 2223 may call the corresponding AI operator in the AI operator library module 2213, and run the AI gesture recognition algorithm model 2223 to perform gesture recognition.
  • the AI application management module 2221 may perform step 3.
  • the AI application management module 2221 may re-execute step 1.
  • the coprocessor 220 reports the gesture recognition message to the main controller 210.
  • the AI application management module 2221 reports the gesture recognition result to the main controller 210.
  • the AI application management module 2221 in the coprocessor 220 may report the gesture recognition result to the AI application layer module 2231.
  • the AI application layer module 2231 forms a gesture recognition event message, and reports the gesture recognition event message to the AI event message manager 212 in the main controller 210.
  • the main controller is woken up.
  • the AI event message manager 212 in the main controller 210 wakes up after receiving the gesture recognition event message sent by the AI application layer module 2231.
  • the main controller lights the screen but keeps the screen locked.
  • the main controller 210 may perform an operation corresponding to the gesture.
  • the operation corresponding to the gesture provided by the user on the back of the hand is to light up the screen.
  • the main controller 210 lights up the screen.
  • the terminal recognizes the user's gesture in other specific scenarios and performs the corresponding operation according to the user's gesture
  • the cooperation mode of the various components in the hardware architecture shown in FIG. 12 is similar to that in FIG. 13, and details are not repeated here.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be from a website site, computer, server or data center Transmission to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.).
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device including a server, a data center, and the like integrated with one or more available media.
  • the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk).

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Telephone Function (AREA)

Abstract

L'invention concerne un procédé de commande de terminal mobile portable ainsi qu'un appareil associé. Un terminal peut collecter l'image d'un utilisateur en temps réel au moyen d'une caméra et exécuter une capacité de perception d'AI de manière autonome. Lorsqu'il est nécessaire d'utiliser le terminal, un utilisateur doit seulement entrer un geste d'utilisateur, puis le terminal peut reconnaître le geste de manière autonome et exécuter une opération correspondante selon le geste d'utilisateur entré. L'invention peut rendre la commande du terminal plus intelligente, ce qui permet d'améliorer l'expérience de l'utilisateur.
PCT/CN2018/115452 2018-11-14 2018-11-14 Procédé de commande de terminal mobile portable et appareil associé Ceased WO2020097830A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201880099510.5A CN113056718A (zh) 2018-11-14 2018-11-14 手持移动终端操控方法及相关装置
PCT/CN2018/115452 WO2020097830A1 (fr) 2018-11-14 2018-11-14 Procédé de commande de terminal mobile portable et appareil associé

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/115452 WO2020097830A1 (fr) 2018-11-14 2018-11-14 Procédé de commande de terminal mobile portable et appareil associé

Publications (1)

Publication Number Publication Date
WO2020097830A1 true WO2020097830A1 (fr) 2020-05-22

Family

ID=70731019

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/115452 Ceased WO2020097830A1 (fr) 2018-11-14 2018-11-14 Procédé de commande de terminal mobile portable et appareil associé

Country Status (2)

Country Link
CN (1) CN113056718A (fr)
WO (1) WO2020097830A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115495721A (zh) * 2021-06-18 2022-12-20 华为技术有限公司 一种访问控制方法及相关装置
US20230282023A1 (en) * 2016-06-01 2023-09-07 Pixart Imaging Inc. Electronic device unlocked and controlled by gesture recognition
CN120751245A (zh) * 2024-06-27 2025-10-03 荣耀终端股份有限公司 手势感知方法及电子设备

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119484697A (zh) * 2025-01-10 2025-02-18 深圳市泰衡诺科技有限公司 控制方法、智能终端及存储介质
CN119597156B (zh) * 2025-02-10 2025-06-10 湖南芒果数智艺术科技有限责任公司 在iOS系统上的自定义手势交互方法及相关装置

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104914982A (zh) * 2014-03-12 2015-09-16 联想(北京)有限公司 一种电子设备的控制方法和装置
WO2015176358A1 (fr) * 2014-05-19 2015-11-26 深圳市中兴微电子技术有限公司 Procédé d'exploitation d'un programme d'application basé sur un geste, terminal et support de stockage
CN105892671A (zh) * 2016-04-22 2016-08-24 广东小天才科技有限公司 一种根据手掌状态产生操作指令的方法和系统
WO2016155090A1 (fr) * 2015-03-30 2016-10-06 中兴通讯股份有限公司 Procédé, appareil et terminal de déverrouillage basés sur une empreinte palmaire et support de stockage informatique
CN106126995A (zh) * 2016-06-30 2016-11-16 宇龙计算机通信科技(深圳)有限公司 解锁方法、解锁装置和终端
CN106896907A (zh) * 2015-12-21 2017-06-27 东莞酷派软件技术有限公司 一种根据用户手势操作终端的方法及装置
CN107024988A (zh) * 2017-03-20 2017-08-08 宇龙计算机通信科技(深圳)有限公司 一种基于用户动作执行操作的方法及装置

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006093934A (ja) * 2004-09-22 2006-04-06 Nec Corp カメラ付き携帯電話端末及びその認証方法
CN202101760U (zh) * 2010-11-12 2012-01-04 山西省电力公司忻州供电分公司 热红外图像采集器
WO2015194918A1 (fr) * 2014-06-20 2015-12-23 주식회사 퓨처플레이 Procédé et système de fourniture d'interface utilisateur et support d'informations non-transitoire lisible par un ordinateur
CN104331149B (zh) * 2014-09-29 2018-08-10 联想(北京)有限公司 一种控制方法、装置和电子设备
WO2017113407A1 (fr) * 2015-12-31 2017-07-06 华为技术有限公司 Procédé et appareil de reconnaissance gestuelle, et dispositif électronique
CN105759935B (zh) * 2016-01-29 2019-01-18 华为技术有限公司 一种终端控制方法及终端
CN106200913A (zh) * 2016-06-28 2016-12-07 珠海市魅族科技有限公司 一种屏幕状态处理方法以及终端
CN107395889B (zh) * 2017-07-28 2020-01-21 Oppo广东移动通信有限公司 降低移动终端功耗的方法、装置、存储介质及移动终端
CN107563168A (zh) * 2017-09-05 2018-01-09 广东欧珀移动通信有限公司 移动终端的解锁方法、装置及移动终端

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104914982A (zh) * 2014-03-12 2015-09-16 联想(北京)有限公司 一种电子设备的控制方法和装置
WO2015176358A1 (fr) * 2014-05-19 2015-11-26 深圳市中兴微电子技术有限公司 Procédé d'exploitation d'un programme d'application basé sur un geste, terminal et support de stockage
WO2016155090A1 (fr) * 2015-03-30 2016-10-06 中兴通讯股份有限公司 Procédé, appareil et terminal de déverrouillage basés sur une empreinte palmaire et support de stockage informatique
CN106896907A (zh) * 2015-12-21 2017-06-27 东莞酷派软件技术有限公司 一种根据用户手势操作终端的方法及装置
CN105892671A (zh) * 2016-04-22 2016-08-24 广东小天才科技有限公司 一种根据手掌状态产生操作指令的方法和系统
CN106126995A (zh) * 2016-06-30 2016-11-16 宇龙计算机通信科技(深圳)有限公司 解锁方法、解锁装置和终端
CN107024988A (zh) * 2017-03-20 2017-08-08 宇龙计算机通信科技(深圳)有限公司 一种基于用户动作执行操作的方法及装置

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230282023A1 (en) * 2016-06-01 2023-09-07 Pixart Imaging Inc. Electronic device unlocked and controlled by gesture recognition
CN115495721A (zh) * 2021-06-18 2022-12-20 华为技术有限公司 一种访问控制方法及相关装置
CN120751245A (zh) * 2024-06-27 2025-10-03 荣耀终端股份有限公司 手势感知方法及电子设备

Also Published As

Publication number Publication date
CN113056718A (zh) 2021-06-29

Similar Documents

Publication Publication Date Title
US11837038B2 (en) User interfaces for managing locks
JP6921338B2 (ja) 電子デバイスの制限された動作
US12079458B2 (en) Image data for enhanced user interactions
US11981181B2 (en) User interfaces for an electronic key
CN111176516B (zh) 用于操纵用户界面的设备、方法和图形用户界面
US11430276B1 (en) User interfaces for managing locks
WO2020097830A1 (fr) Procédé de commande de terminal mobile portable et appareil associé
JP2021073563A (ja) 拡張されたユーザ対話のための画像データ
US20180205880A1 (en) Initializing camera subsystem for face detection based on sensor inputs
US20240253419A1 (en) User interfaces for an electronic key
CN112860169B (zh) 交互方法及装置、计算机可读介质和电子设备
WO2017063146A1 (fr) Procédé et appareil de fonctionnement utilisant une identification d'empreinte digitale, et terminal mobile
US12430000B2 (en) User input interfaces
CN104899029A (zh) 一种屏幕控制方法及装置
CN108351688A (zh) 组合的抓握和移动性感测
WO2020253495A1 (fr) Procédé de commande de verrouillage d'écran, dispositif, terminal portable et support d'enregistrement
CN111684403A (zh) 用于图形用户界面的媒体捕获锁定示能表示
WO2022213945A1 (fr) Procédé et appareil de rappel d'application, et dispositif électronique
CN107562356B (zh) 指纹识别定位方法、装置、存储介质及电子设备
CN116409333A (zh) 交互方法、设备和存储介质
WO2023274033A1 (fr) Procédé de commande d'accès et appareil associé
CN109409333A (zh) 指纹解锁方法、装置、设备和存储介质
CN110210198A (zh) 一种应用处理方法及移动终端

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18940305

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18940305

Country of ref document: EP

Kind code of ref document: A1