CN113473198B

CN113473198B - Control method of intelligent equipment and intelligent equipment

Info

Publication number: CN113473198B
Application number: CN202010582730.2A
Authority: CN
Inventors: 陈维强; 冯谨强; 孟祥奇; 高雪松
Original assignee: Qingdao Hisense Electronic Industry Holdings Co Ltd
Current assignee: Hisense Group Holding Co Ltd
Priority date: 2020-06-23
Filing date: 2020-06-23
Publication date: 2023-09-05
Anticipated expiration: 2040-06-23
Also published as: CN113473198A

Abstract

The application relates to the technical field of intelligent equipment, and provides a control method of intelligent equipment and the intelligent equipment, wherein the method comprises the following steps: receiving a plurality of first images acquired by an image acquisition device in a set period, each time reading one first image, intercepting a second image from the one first image according to a preset intercepting rule, identifying the second image, and determining at least one gesture identification area and a corresponding gesture identification result; and if the same control gestures exceeding the set number threshold exist in the same gesture recognition area, controlling the intelligent equipment to execute corresponding operations based on the control gestures. The first image is preprocessed according to the preset interception rule, so that the hand detection difficulty is reduced, the time spent for gesture recognition can be shortened, and the user experience can be improved by collecting the static gestures of the user.

Description

Control method of intelligent equipment and intelligent equipment

Technical Field

The application relates to the technical field of intelligent equipment, and provides a control method of intelligent equipment and the intelligent equipment.

Background

Along with the intelligent upgrading of the product equipment, a user performs different continuous actions (namely dynamic gestures) by hands to operate the screen, so that the functions of adjusting volume, changing channels, fast forwarding, fast rewinding and the like are realized, and the operation of controlling the intelligent equipment by the user is simpler and simpler.

However, when the intelligent device is controlled in the above manner, the following problems occur: the farther the user is from the camera, the smaller the hand image ratio of the user in the acquired images is, so that the hand detection difficulty is greatly improved; when a user performs a dynamic gesture, the camera acquires a plurality of continuous images and inputs the plurality of continuous images into the neural network for recognition, and the recognition process of the neural network can take a long time because of more input images, so that the real-time response requirement cannot be met; finally, the user needs to make dynamic gestures to meet the response, and the user experience is poor.

In view of the above, the embodiment of the application provides a novel intelligent device control method and intelligent device.

Disclosure of Invention

The embodiment of the application provides a control method of intelligent equipment and the intelligent equipment, which are used for reducing the difficulty of hand detection, shortening the time spent on gesture recognition and improving the user experience.

In a first aspect, an embodiment of the present application provides an intelligent device, including:

a display configured to display a screen;

an image collector configured to collect a plurality of first images in a set period and transmit the plurality of first images to a controller;

The controller is configured to receive the plurality of first images acquired by the image acquirer in a set period;

respectively executing the following processes for each first image, wherein each first image is read, and one second image is intercepted from the first image according to a preset intercepting rule, wherein the ratio of the hand image of the user to the second image is higher than that of the hand image of the user to the first image; identifying the second image, and determining at least one gesture identification area on the second image and a corresponding gesture identification result;

and if the same control gestures exceeding the set quantity threshold exist in the same gesture recognition area, controlling the intelligent equipment to execute corresponding operations based on the same control gestures.

Optionally, the controller is configured to:

determining size information and angular point coordinate information of the hand image according to the field angle of the image collector, a set field angle threshold value and the size information of the first image;

and cutting out the second image from the first image according to the size information of the hand image and the angular point coordinate information.

Optionally, the controller is further configured to:

and if the angle of view of the image collector in the X direction and the angle of view of the image collector in the Y direction are lower than the angle of view threshold, determining the first image as the second image.

Optionally, the controller is configured to:

if the same gesture recognition area has continuous M wake-up gestures, determining the same gesture recognition area as a gesture control area;

if the gesture control area has continuous N first control gestures after the continuous M wake-up gestures, controlling the intelligent device to execute corresponding operations based on the first control gestures;

wherein M, N is a positive integer.

Optionally, the controller is configured to:

if the gesture control area is behind the continuous N first control gestures, the continuous N second control gestures exist, and the intelligent device is controlled to execute corresponding operations based on the second control gestures; wherein N is a positive integer.

In a second aspect, an embodiment of the present application further provides a method for controlling an intelligent device, including:

receiving a plurality of first images acquired by an image acquisition device in a set period;

Optionally, according to a preset interception rule, intercepting a second image from the first image includes:

Optionally, before determining the size information and the corner coordinate information of the hand image, the method further includes:

Optionally, if the same control gesture exceeding the set number threshold exists in the same gesture recognition area, controlling the intelligent device to execute a corresponding operation based on the same control gesture includes:

wherein M, N is a positive integer.

Optionally, after controlling the smart device to perform a corresponding operation based on the first control gesture, the method further includes:

In a third aspect, an embodiment of the present application further provides a computer readable storage medium, which includes program code for causing a terminal to execute the steps of the control method of any one of the above-mentioned smart devices, when the program product is run on the terminal.

The application has the following beneficial effects:

the application provides a control method of intelligent equipment and the intelligent equipment, wherein the method comprises the following steps: receiving a plurality of first images acquired by an image acquisition device in a set period, each time reading one first image, intercepting a second image from the one first image according to a preset intercepting rule, identifying the second image, and determining at least one gesture identification area and a corresponding gesture identification result; and if the same control gestures exceeding the set number threshold exist in the same gesture recognition area, controlling the intelligent equipment to execute corresponding operations based on the control gestures. The first image is preprocessed according to the preset interception rule, so that the hand detection difficulty is reduced, the time spent for gesture recognition can be shortened, and the user experience can be improved by collecting the static gestures of the user.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

FIG. 1a schematically illustrates an operational scenario between a smart device and a control device;

a block diagram of the configuration of the control device 100 in fig. 1a is exemplarily shown in fig. 1 b;

a block diagram of the configuration of the smart device 200 of fig. 1a is schematically shown in fig. 1 c;

an architectural configuration block diagram of an operating system in the memory of the smart device 200 is exemplarily shown in fig. 1 d;

FIG. 2 illustrates a flow diagram for controlling a smart device;

FIG. 3a illustrates a view angle schematic;

fig. 3b shows an exemplary top view of the field angle in the X direction;

FIG. 3c illustrates a side view of the field angle in the Y direction;

FIG. 4a illustrates a wake gesture schematic;

FIG. 4b illustrates a control gesture to turn up volume;

FIG. 4c illustrates a control gesture to turn down volume;

FIG. 4d illustrates a control gesture schematic for video fallback;

FIG. 4e illustrates a control gesture schematic of video fast forward;

FIG. 4f illustrates a determined control gesture schematic;

FIG. 4g illustrates a control gesture diagram for cancellation;

fig. 4h illustrates a control gesture schematic of mute/end.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the technical solutions of the present application, but not all embodiments. All other embodiments, based on the embodiments described in the present document, which can be obtained by a person skilled in the art without any creative effort, are within the scope of protection of the technical solutions of the present application.

A schematic diagram of an operational scenario between a smart device and a control device is schematically shown in fig. 1 a. As shown in fig. 1a, the control device 100 and the smart device 200 may communicate in a wired or wireless manner.

Wherein, the control device 100 is configured to control the smart device 200, which can receive an operation instruction input by a user, and convert the operation instruction into an instruction recognizable and responsive by the smart device 200, and plays an intermediary role in interaction between the user and the smart device 200. Such as: the user responds to the channel addition and subtraction operation by operating the channel addition and subtraction key on the control apparatus 100.

The control device 100 may be a remote control 100A, including an infrared protocol communication or a bluetooth protocol communication, and other short-range communication modes, and the intelligent device 200 is controlled by a wireless or other wired mode. The user may control the smart device 200 by inputting user instructions through keys on a remote control, voice input, control panel input, etc. Such as: the user can input corresponding control instructions through volume up-down keys, channel control keys, up/down/left/right movement keys, voice input keys, menu keys, on-off keys and the like on the remote controller, so as to realize the function of controlling the intelligent device 200.

The control device 100 may be a mobile terminal 100B, a tablet computer, a notebook computer, or the like. For example, the smart device 200 is controlled using an application running on the mobile terminal 100B. The application program may provide various controls to the user through an intuitive User Interface (UI) on a screen associated with the mobile terminal 100B through configuration.

The user interface in the embodiment of the application is a medium interface for interaction and information exchange between an application program or an operating system and a user, and the user interface realizes conversion between an internal form of information and an acceptable form of the user. A commonly used presentation form of a user interface is a Graphical User Interface (GUI), which refers to a user interface graphically displayed in connection with computer operations. It may be an interface element such as an icon, a window, a control, etc. displayed in the display screen of the smart device 200, where the control may include a visual interface element such as an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc.

For example, the mobile terminal 100B may install a software application with the smart device 200, and implement connection communication through a network communication protocol for the purpose of one-to-one control operation and data communication. Such as: the mobile terminal 100B may be caused to establish a control instruction protocol with the smart device 200 to implement functions such as physical keys arranged by the remote controller 100A by operating various function keys or virtual buttons of a user interface provided on the mobile terminal 100B. The audio and video contents displayed on the mobile terminal 100B may also be transmitted to the display of the smart device 200, so as to implement a synchronous display function.

In other exemplary embodiments, the smart device 200 may also invoke an internally configured image collector, such as a camera, webcam, etc., for collecting external environmental scenes to adaptively change the display parameters of the smart device 200; and the intelligent device is used for collecting a plurality of first images in a set period, wherein the first images possibly contain attributes of a user or interaction gestures of the user so as to realize the interaction function between the intelligent device and the user.

The smart device 200 may provide a broadcast receiving function and a network television function of a computer supporting function. The smart device may be a digital television, a web television, an Internet Protocol Television (IPTV), etc. The display of the smart device 200 may be a liquid crystal display, an organic light emitting display, a projection device. The specific display type, size, resolution, etc. are not limited.

The smart device 200 is also in data communication with the server 300 via a variety of communication means. The smart device 200 may be allowed to make communication connections via a Local Area Network (LAN), a Wireless Local Area Network (WLAN), and other networks. The server 300 may provide various content and interactions to the smart device 200. By way of example, the smart device 200 may send and receive information, such as: receiving Electronic Program Guide (EPG) data, receiving software program updates, or accessing a remotely stored digital media library. The servers 300 may be one group, may be multiple groups, and may be one or more types of servers. Other web service content such as video on demand and advertising services are provided through the server 300.

A block diagram of the configuration of the control device 100 is schematically shown in fig. 1 b. As shown in fig. 1b, the control device 100 includes a controller 110, a memory 120, a communicator 130, a user input interface 140, an output interface 150, and a power supply 160.

The controller 110 includes a Random Access Memory (RAM) 111, a Read Only Memory (ROM) 112, a processor 113, a communication interface, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication collaboration between the internal components, external and internal data processing functions.

For example, when an interaction that a user presses a key arranged on the remote controller 100A or an interaction that touches a touch panel arranged on the remote controller 100A is detected, the controller 110 may control to generate a signal corresponding to the detected interaction and transmit the signal to the smart device 200.

The memory 120 stores various operation programs, data, and applications for driving and controlling the control device 100 under the control of the controller 110. The memory 120 may store various control signal instructions input by a user.

The communicator 130 performs communication of control signals and data signals with the smart device 200 under the control of the controller 110. Such as: the control apparatus 100 transmits a control signal (e.g., a touch signal or a button signal) to the smart device 200 via the communicator 130, and the control apparatus 100 may receive the signal transmitted by the smart device 200 via the communicator 130. Communicator 130 may include an infrared signal interface 131 and a radio frequency signal interface 132. For example: when the infrared signal interface is used, the user input instruction needs to be converted into an infrared control signal according to an infrared control protocol, and the infrared control signal is sent to the intelligent device 200 through the infrared sending module. And the following steps: when the radio frequency signal interface is used, the user input instruction is converted into a digital signal, and then the digital signal is modulated according to a radio frequency control signal modulation protocol and then transmitted to the intelligent device 200 through the radio frequency transmission terminal.

The user input interface 140 may include at least one of a microphone 141, a touch pad 142, a sensor 143, keys 144, etc., so that a user may input user instructions regarding controlling the smart device 200 to the control apparatus 100 through voice, touch, gesture, press, etc.

The output interface 150 outputs a user instruction received by the user input interface 140 to the smart device 200 or outputs an image or voice signal received by the smart device 200. Here, the output interface 150 may include an LED interface 151, a vibration interface 152 generating vibrations, a sound output interface 153 outputting sound, a display 154 outputting an image, and the like. For example, the remote controller 100A may receive an output signal of audio, video, or data from the output interface 150, and display the output signal as an image form on the display 154, as an audio form at the sound output interface 153, or as a vibration form at the vibration interface 152.

A power supply 160 for providing operating power support for the various elements of the control device 100 under the control of the controller 110. May be in the form of a battery and associated control circuitry.

A hardware configuration block diagram of the smart device 200 is exemplarily shown in fig. 1 c. As shown in fig. 1c, a modem 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a memory 260, a user interface 265, a video processor 270, a display 275, an audio processor 280, an audio output interface 285, a power supply 290 may be included in the smart device 200.

The modem 210 receives broadcast television signals through a wired or wireless manner, and may perform modulation and demodulation processes such as amplification, mixing, and resonance, for demodulating an audio/video signal carried in a frequency of a television channel selected by a user and additional information (e.g., EPG data) from among a plurality of wireless or wired broadcast television signals.

The tuning demodulator 210 is responsive to the frequency of the television channel selected by the user and the television signal carried by that frequency, as selected by the user, and as controlled by the controller 250.

The tuning demodulator 210 can receive signals in various ways according to broadcasting systems of television signals, such as: terrestrial broadcasting, cable broadcasting, satellite broadcasting, internet broadcasting, or the like; according to different modulation types, a digital modulation mode or an analog modulation mode can be adopted; and the analog signal and the digital signal can be demodulated according to the kind of the received television signal.

In other exemplary embodiments, the modem 210 may also be in an external device, such as an external set-top box or the like. In this way, the set-top box outputs a television signal after modulation and demodulation, and inputs the television signal to the smart device 200 through the external device interface 240.

The communicator 220 is a component for communicating with an external device or an external server according to various communication protocol types. For example, the smart device 200 may transmit content data to an external device connected via the communicator 220, or browse and download content data from an external device connected via the communicator 220. The communicator 220 may include a network communication protocol module or a near field communication protocol module such as a WIFI module 221, a bluetooth communication protocol module 222, a wired ethernet communication protocol module 223, etc., so that the communicator 220 may receive a control signal of the control device 100 according to the control of the controller 250 and implement the control signal as a WIFI signal, a bluetooth signal, a radio frequency signal, etc.

The detector 230 is a component of the smart device 200 for collecting signals of the external environment or interaction with the outside. The detector 230 may include a sound collector 231, such as a microphone, that may be used to receive a user's sound, such as a voice signal of a control instruction of the user controlling the smart device 200; alternatively, environmental sounds for identifying the type of environmental scene may be collected, and the implementation smart device 200 may adapt to environmental noise.

In other exemplary embodiments, the detector 230 may further include an image collector 232, such as a camera, a video camera, etc., that may be used to collect external environmental scenes to adaptively change the display parameters of the smart device 200; and the intelligent device is used for collecting a plurality of first images in a set period, wherein the first images possibly contain attributes of a user or interaction gestures with the user so as to realize the interaction function between the intelligent device and the user.

In other exemplary embodiments, the detector 230 may further include a light receiver for collecting the ambient light intensity to adapt to changes in display parameters of the smart device 200, etc.

In other exemplary embodiments, the detector 230 may also include a temperature sensor, such as by sensing ambient temperature, the smart device 200 may adaptively adjust the display color temperature of the image. Illustratively, when the temperature is higher, the smart device 200 may be adjusted to display a colder shade of the image color temperature; when the temperature is lower, the smart device 200 can be adjusted to display the color temperature and the warm tone of the image.

The external device interface 240 is a component that provides the controller 250 to control data transmission between the smart device 200 and an external device. The external device interface 240 may be connected to an external device such as a set-top box, a game device, a notebook computer, etc., in a wired/wireless manner, and may receive data such as a video signal (e.g., a moving image), an audio signal (e.g., music), additional information (e.g., an EPG), etc., of the external device.

The external device interface 240 may include: any one or more of a High Definition Multimedia Interface (HDMI) terminal 241, a Composite Video Blanking Sync (CVBS) terminal 242, an analog or digital Component terminal 243, a Universal Serial Bus (USB) terminal 244, a Component terminal (not shown), a Red Green Blue (RGB) terminal (not shown), and the like.

The controller 250 controls the operation of the smart device 200 and responds to the user's operations by running various software control programs (e.g., an operating system and various application programs) stored on the memory 260. The controller 250 may further perform the following processing for each first image, where each first image M is read, and a second image N is intercepted from the first image M according to a preset intercepting rule, where a ratio of a hand image of a user to the second image N is higher than a ratio of the hand image of the user to the first image M; identifying the second image N, and determining at least one gesture identification area on the second image N and a corresponding gesture identification result; and if the same control gesture exceeding the set threshold exists in the same gesture recognition area, controlling the intelligent device to execute corresponding operation based on the same control gesture.

As shown in fig. 1c, the controller 250 includes a Random Access Memory (RAM) 251, a Read Only Memory (ROM) 252, a graphics processor 253, a CPU processor 254, a communication interface 255, and a communication bus 256. The RAM251, the ROM252, the graphics processor 253, and the CPU 254 are connected to each other via a communication bus 256.

A ROM252 for storing various system boot instructions. If the power of the smart device 200 starts to be started when the power-on signal is received, the CPU processor 254 executes a system start instruction in the ROM252, copies the operating system stored in the memory 260 into the RAM251, and starts to run the start operating system. When the operating system is started, the CPU processor 254 copies various applications in the memory 260 to the RAM251, and then starts running the various applications.

The graphic processor 253 generates various graphic objects such as icons, operation menus, and user input instruction display graphics, etc. The graphic processor 253 may include an operator for performing an operation by receiving user input of various interactive instructions, thereby displaying various objects according to display attributes; and a renderer for generating various objects based on the operator, and displaying the result of rendering on the display 275.

CPU processor 254 is operative to execute operating system and application program instructions stored in memory 260. And executing processing of various application programs, data and contents according to the received user input instructions so as to finally display and play various audio and video contents.

In some exemplary embodiments, the CPU processor 254 may comprise a plurality of processors. The plurality of processors may include one main processor and a plurality or one sub-processor. A main processor for performing some initialization operations of the smart device 200 in a smart device preloading mode and/or an operation of displaying a screen in a normal mode. A plurality of or a sub-processor for performing an operation in a state of a standby mode of the smart device or the like.

Communication interface 255 may include a first interface through an nth interface. These interfaces may be network interfaces that are connected to external devices via a network.

The controller 250 may control the overall operation of the smart device 200. For example: in response to receiving a user input command for selecting a GUI object displayed on the display 275, the controller 250 may perform an operation related to the object selected by the user input command.

Wherein the object may be any one of selectable objects, such as a hyperlink or an icon. The operation related to the selected object, for example, an operation of displaying a link to a hyperlink page, a document, an image, or the like, or an operation of executing a program corresponding to the object. The user input command for selecting the GUI object may be a command input through various input means (e.g., mouse, keyboard, touch pad, etc.) connected to the smart device 200 or a voice command corresponding to a voice uttered by the user.

The memory 260 is used to store various types of data, software programs, or applications that drive and control the operation of the smart device 200. Memory 260 may include volatile and/or nonvolatile memory. And the term "memory" includes memory 260, RAM251 and ROM252 of controller 250, or a memory card in smart device 200.

In some embodiments, the memory 260 is specifically configured to store an operating program that drives the controller 250 in the smart device 200; storing various application programs built in the smart device 200 and downloaded from an external device by a user; data for configuring various GUIs provided by the display 275, various objects related to the GUIs, visual effect images of selectors for selecting GUI objects, and the like are stored.

In some embodiments, the memory 260 is specifically configured to store drivers and related data for the modem 210, the communicator 220, the detector 230, the external device interface 240, the video processor 270, the display 275, the audio processor 280, etc., such as external data (e.g., audio-visual data) received from the external device interface or user data (e.g., key information, voice information, touch information, etc.) received from the user interface.

In some embodiments, memory 260 specifically stores software and/or programs for representing an Operating System (OS), which may include, for example: a kernel, middleware, an Application Programming Interface (API), and/or an application program. Illustratively, the kernel may control or manage system resources, as well as functions implemented by other programs (such as the middleware, APIs, or application programs); at the same time, the kernel may provide an interface to allow middleware, APIs, or applications to access the controller to implement control or management of system resources.

An architectural configuration block diagram of the operating system in the memory of the smart device 200 is illustrated in fig. 1 d. The operating system architecture is an application layer, a middleware layer and a kernel layer in sequence from top to bottom.

The application layer, the application program built in the system and the non-system application program belong to the application layer. Is responsible for direct interaction with the user. The application layer may include a plurality of applications, such as a setup application, an electronic post application, a media center application, and the like. These applications may be implemented as Web applications that execute based on WebKit engines, and in particular may be developed and executed based on HTML5, cascading Style Sheets (CSS), and JavaScript.

Here, HTML, which is called a hypertext markup language (HyperText Markup Language) in its entirety, is a standard markup language for creating web pages, which are described by markup tags for describing words, graphics, animations, sounds, tables, links, etc., and a browser reads an HTML document, interprets the contents of tags within the document, and displays them in the form of web pages.

CSS, collectively referred to as cascading style sheets (Cascading Style Sheets), is a computer language used to represent the style of HTML files and may be used to define style structures such as fonts, colors, positions, and the like. The CSS style can be directly stored in an HTML webpage or a separate style file, so that the control of the style in the webpage is realized.

JavaScript, a language applied to Web page programming, can be inserted into HTML pages and interpreted by a browser. The interaction logic of the Web application is realized through JavaScript. The JavaScript can be used for packaging the JavaScript extension interface through the browser to realize communication with the kernel layer.

Middleware layer, some standardized interfaces may be provided to support the operation of various environments and systems. For example, the middleware layer may be implemented as multimedia and hypermedia information coding expert group (MHEG) of middleware related to data broadcasting, as DLNA middleware of middleware related to external device communication, as middleware providing a browser environment for each application program in the smart device, and the like.

A kernel layer providing core system services such as: file management, memory management, process management, network management, system security authority management and other services. The kernel layer may be implemented as a kernel based on various operating systems, such as a kernel based on the Linux operating system.

The kernel layer also provides communication between system software and hardware at the same time, providing device driver services for various hardware, such as: providing a display driver for a display, providing a camera driver for a camera, providing a key driver for a remote control, providing a WIFI driver for a WIFI module, providing an audio driver for an audio output interface, providing a Power Management (PM) module with a power management driver, and the like.

A user interface 265 receives various user interactions. Specifically, an input signal for a user is transmitted to the controller 250, or an output signal from the controller 250 is transmitted to the user. Illustratively, the remote control 100A may send input signals such as a power switch signal, a channel selection signal, a volume adjustment signal, etc., input by a user to the user interface 265, and then forwarded by the user interface 265 to the controller 250; alternatively, the remote controller 100A may receive an output signal such as audio, video, or data, which is processed by the controller 250 to be output from the user interface 265, and display the received output signal or output the received output signal in the form of audio or vibration.

In some embodiments, a user may input a user command through a Graphical User Interface (GUI) displayed on the display 275, and the user interface 265 receives the user input command through the GUI. In particular, the user interface 265 may receive user input commands for controlling the position of a selector in a GUI to select different objects or items.

Alternatively, the user may enter a user command by entering a particular sound or gesture, and the user interface 265 recognizes the sound or gesture through the sensor to receive the user input command. The video processor 270 is configured to receive an external video signal, and perform video data processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image composition according to a standard codec protocol of an input signal, so as to obtain a video signal that is directly displayed or played on the display 275.

By way of example, video processor 270 includes a demultiplexing module, a video decoding module, an image compositing module, a frame rate conversion module, a display formatting module, and the like.

Wherein, the demultiplexing module is used for demultiplexing the input audio/video data stream, such as the input MPEG-2 stream (based on the compression standard of the digital storage media moving image and voice), and then the demultiplexing module demultiplexes the input audio/video data stream into video signals, audio signals and the like.

And the video decoding module is used for processing the demultiplexed video signal, including decoding, scaling and the like.

And an image synthesis module, such as an image synthesizer, for performing superposition mixing processing on the graphic generator and the video image after the scaling processing according to the GUI signal input by the user or generated by the graphic generator, so as to generate an image signal for display.

The frame rate conversion module is configured to convert a frame rate of an input video, for example, convert a frame rate of an input 60Hz video into a frame rate of 120Hz or 240Hz, and a common format is implemented in an inserting frame manner.

And a display formatting module for converting the signal output by the frame rate conversion module into a signal conforming to a display format such as a display, for example, format converting the signal output by the frame rate conversion module to output an RGB data signal.

And a display 275 for receiving image signals from the video processor 270 and displaying video content, images and menu manipulation interfaces. The video content may be displayed from the broadcast signal received by the modem 210, or may be displayed from the video content input by the communicator 220 or the external device interface 240. And a display 275 for simultaneously displaying a user manipulation interface UI generated in the smart device 200 and used for controlling the smart device 200.

And, the display 275 may include a display screen assembly for presenting pictures and a drive assembly for driving the display of images. Alternatively, if the display 275 is a projection display, a projection device and a projection screen may be included.

The audio processor 280 is configured to receive an external audio signal, decompress and decode according to a standard codec of an input signal, and perform audio data processing such as noise reduction, digital-to-analog conversion, and amplification, so as to obtain an audio signal that can be played in the speaker 286.

Illustratively, the audio processor 280 may support various audio formats. Such as MPEG-2, MPEG-4, advanced Audio Coding (AAC), high efficiency AAC (HE-AAC), etc.

An audio output interface 285 for receiving the audio signal output from the audio processor 280 under the control of the controller 250, the audio output interface 285 may include a speaker 286, or an external audio output terminal 287, such as a headphone output terminal, for outputting to a generating device of an external device.

In other exemplary embodiments, video processor 270 may include one or more chip components. Audio processor 280 may also include one or more chip components.

And, in other exemplary embodiments, video processor 270 and audio processor 280 may be separate chips or integrated with controller 250 in one or more chips.

The power supply 290 is configured to provide power supply support for the smart device 200 with power input from an external power source under the control of the controller 250. The power supply 290 may be a built-in power supply circuit installed inside the smart device 200 or may be a power supply installed outside the smart device 200.

In the prior art, a dynamic gesture image of a user needs to be acquired and identified, and the intelligent device is controlled to execute corresponding operation according to a corresponding identification result, but when the intelligent device is controlled in the above manner, the following problems occur: the farther the user is from the camera, the smaller the hand image ratio of the user in the acquired images is, so that the hand detection difficulty is greatly improved; the neural network identification process can take a long time, and continuous multiple images are identified, so that the real-time response requirement cannot be met; and, the user needs to make dynamic gestures to meet the response, so that the user experience is poor. In order to solve the foregoing problems, an embodiment of the present application provides a new method for controlling an intelligent device, as shown in fig. 2, including the following steps:

S201: a plurality of first images acquired by the image acquirer in a set period are received.

When the intelligent equipment is in a starting-up state, the image collector is called to shoot a current scene according to a preset collection frequency in a set period, and a plurality of continuous first images are obtained. Because the image collector performs image collection operation according to the set period and the preset collection frequency, a pure background image without user can be collected, and an image containing one user or a plurality of users can be collected. For example, the camera captures 20 images in 1 second, where the first 5 images are pure background images that do not contain the user, and the 6 th-20 th images are images that contain two users.

S202: a first image M is read.

S203: and intercepting a second image N from the first image M according to a preset intercepting rule, wherein the ratio of the hand image of the user to the second image N is higher than the ratio of the hand image to the first image M.

In the embodiment of the application, the hand image of the user needs to be detected, but the number of pixels occupied by the hand image in the acquired first image is generally smaller, and the smaller the occupation ratio of the hand image in the acquired first image is along with the farther the user is from the camera, the more the hand detection difficulty is improved, so that the first image needs to be preprocessed before the hand image is detected, so that the second image with the high occupation ratio of the hand image is intercepted, and the detection difficulty is reduced. Specifically, the step of capturing the second image N from the first image M is as follows:

A1: and determining the size information and the corner coordinate information of the hand image according to the field angle of the image collector, the set field angle threshold value and the size information of the first image M.

In the image collector, the lens of the image collector is taken as the vertex, and the included angle formed by the two edges of the maximum range of the object image of the measured object passing through the lens is called as the angle of view, and referring to fig. 3a, a schematic view of the angle of view is shown.

Referring to FIG. 3b, when the line passing through the center of the lens (i.e. the optical axis of the lens) is parallel to the ground, the object image of the measured object can pass through the angle formed by the two edges of the maximum range of the lens, which is called the angle of view in the X direction (i.e. V _X ) The method comprises the steps of carrying out a first treatment on the surface of the Referring to FIG. 3c, when the optical axis of the lens is perpendicular to the ground, the object image of the measured object can pass through the angle formed by the two edges of the maximum range of the lens, which is called the angle of view in the X direction (i.e. V _Y )。

If V _X And V _Y When the size of the first image M is smaller than the view angle threshold, which means that the size of the first image M is smaller and the occupancy rate of the hand image in the first image M is higher, the whole first image M is input into the trained hand detection network as the second image N, and at least one gesture recognition area included in the second image N is detected.

If V _X Greater than the angle of view threshold, calculating the width to be truncated, V, of the region of interest (Region Of Interest, ROI) containing the hand image using equation (1) _X Characterizing the angle of view in the X-direction, T _V Characterizing a view angle threshold, L _X The width to be truncated of the ROI is characterized, W represents the width of the first image M.

If V _Y Above the angle of view threshold, calculating the height to be truncated, V, of the region of interest (Region Of Interest, ROI) containing the hand image using equation (2) _Y Characterizing the angle of view in the Y-direction, T _V Characterizing a view angle threshold, L _Y The height to be truncated of the ROI is characterized, H characterizes the height of the first image M.

Further, in the embodiment of the application, the upper left corner of the ROI is determined as the corner point, and the corner point coordinate information of the ROI is calculated by adopting a formula (3). The size information and corner coordinate information of the final ROI can be expressed as (x, y, w, h), (x, y) representing the corner coordinate information of the ROI, w being equal to L _X Characterizing the width to be intercepted of the ROI; h is equal to L _Y The height to be truncated of the ROI is characterized.

A2: and cutting out a second image N from the first image M according to the size information and the corner coordinate information of the hand image.

S204: and identifying the second image N, and determining at least one gesture identification area and a corresponding gesture identification result on the second image N.

B1: inputting the second image N into the trained hand detection network to obtain the number of gesture recognition areas contained in the second image N, coordinate information of each gesture recognition area on the second image N, and probability values (namely confidence degrees of the gesture recognition areas) of the hand images in each gesture recognition area.

In order to ensure the accuracy of hand detection, the embodiment of the application adopts a multi-scale SSD algorithm and designs an end-to-end hand detection network. The architecture design and training process of the hand detection network is as follows:

the hand detection network is composed of an input layer, a convolution layer, a pooling layer and an output layer, wherein the convolution layer is used for extracting useful features from a second image N transmitted by the input layer, such as horizontal, vertical, edge or diagonal features; the pooling layer is used to increase the receptive field of the extracted features, which refers to the size of the area of one pixel corresponding to the first image M, and to reduce the difficulty of optimization.

In order to quickly reduce the size of the second image N, embodiments of the present application provide for a larger sampling step for the convolutional layer and the pooling layer. For example, the sampling step length of the convolution layer 1 is 4, and the sampling step lengths of the convolution layer 2, the pooling layer 1 and the pooling layer 2 are all 2, so that the size of the second image N is reduced by 32 times after passing through two convolution layers and 2 pooling layers. Furthermore, the embodiment of the application adopts an acceptance module comprising various convolution branches, and the acceptance module is composed of pooling layers and convolution layers with different structures, so that the width of a network is increased on one hand, the adaptability of the network to the scale is increased on the other hand, and the diversity of receptive fields is effectively improved.

Labeling each sample image, namely, a rectangular frame containing a hand image on the sample image is called a positive rectangular frame, a rectangular frame containing only a pure background is called a negative rectangular frame, and coordinate information corresponding to the positive rectangular frame and the negative rectangular frame is determined;

in order to solve the problem, in the embodiment of the application, only all positive rectangular frames and a plurality of negative rectangular frames which are difficult to detect are selected, a classification loss function is called, the class error values between the negative rectangular frames and the corresponding predicted rectangular frames are calculated, and the class error values between all positive rectangular frames and the corresponding predicted rectangular frames are calculated; invoking a position loss function, and only calculating a position error value between the predicted rectangular frame coordinate position and the corresponding positive rectangular frame coordinate position; finally, readjusting parameters of the hand detection network based on the category error value and the position error value;

Repeating the training process until the set iteration times are reached, or all sample images are read, or the error value is lower than the set error threshold value, and outputting the trained hand detection network.

B2: and respectively inputting the images corresponding to the gesture recognition areas into the trained gesture classification network to obtain corresponding gesture recognition results.

The gesture classification network consists of a basic network, a full connection layer and a softMax classification network, an image is input into the basic network, and finally the softMax classification network outputs the gesture category of the image and the confidence of the gesture category (namely, the gesture recognition result). The base Network may be any of the following visual geometry group networks (Visual Geometry Group Network, VGGNet), residual networks (Residual networks), alexNet, or other convolutional neural networks (Convolutional Neural Network, CNN), among others.

Gesture categories may be categorized into invalid gestures, wake gestures, and control gestures. The wake-up gesture in the embodiment of the application is shown in fig. 4a, and the wake-up gesture is set to inform the intelligent device that the following action needs to be executed, so that misoperation is avoided, the problem that the intelligent device determines a gesture control area in a multi-person interaction scene is solved, and only one user is allowed to control the intelligent device in the multi-person interaction scene is solved. In the embodiment of the present application, the control gesture shown in fig. 4b represents a volume up, the control gesture shown in fig. 4c represents a volume down, the control gesture shown in fig. 4d represents a video back, the control gesture shown in fig. 4e represents a video fast forward, the control gesture shown in fig. 4f represents a determination, the control gesture shown in fig. 4g represents a cancellation, and the control gesture shown in fig. 4h represents a mute/end. Other gestures that do not belong to the preset gesture are defined as invalid gestures.

S205: judging whether all the first images are read completely, if so, executing step 206; otherwise, return to step 202.

S206: and if the same control gestures exceeding the set number threshold exist in the same gesture recognition area, controlling the intelligent equipment to execute corresponding operations based on the same control gestures.

According to the embodiment of the application, static gesture recognition is adopted, namely gesture recognition operation is carried out on one image, and a gesture recognition result is output, because the image collector collects a plurality of first images in a set period, if the intelligent device controls the intelligent device to execute corresponding operation according to the gesture recognition result of each first image, the intelligent device can execute the same operation for a plurality of times in a short time, and the control of the intelligent device is not facilitated. In order to solve the above problems, the intelligent device according to the embodiment of the present application comprehensively determines the operation to be executed finally according to the gesture recognition result of the multiframe.

Firstly, each time a first image is read, steps 203-204 are performed on the first image to obtain a gesture recognition result of a gesture recognition area included in the image, that is, when all the first images are read, at least one gesture recognition area and a corresponding gesture recognition result set are obtained. Thus, if there are consecutive M wake-up gestures in the same gesture recognition region, the same gesture recognition region is determined as a gesture control region, where M is a positive integer.

And secondly, if the gesture control area is provided with N continuous first operation gestures after the M continuous wake-up gestures, controlling the intelligent equipment to execute corresponding operations based on the first operation gestures, wherein N is a positive integer.

For example, the gesture recognition area 1 includes 20 gesture recognition results, wherein, 3 rd to 10 th are all wake-up gestures, and 11 th to 15 th are all control gestures as shown in fig. 4b, and the intelligent device will turn up the volume of video playing according to the indication of the control gestures.

Further, after the intelligent device is controlled to execute the corresponding operation based on the first control gesture, if the gesture control area is after the continuous N first control gestures, and the continuous N second control gestures exist, the intelligent device is controlled to execute the corresponding operation based on the second control gestures. Wherein N is a positive integer, and the first control gesture and the second control gesture may be the same control gesture or different control gestures.

In this example, after the intelligent device performs the operation of turning up the volume of the video playing, if the 16 th to 20 th are all control gestures as shown in fig. 4e, the intelligent device performs the fast forward operation on the currently playing video according to the indication of the control gesture.

Further, after the intelligent device is controlled to execute the corresponding operation based on the first control gesture, if the gesture control area does not have the continuous N second control gestures after the continuous N first control gestures, when determining that the continuous M wake-up gestures exist in the other gesture recognition areas, transferring the control focus from the current gesture recognition area to the other gesture recognition areas, and determining the other gesture recognition areas as new gesture control areas, so that the intelligent device executes the new operation according to the gesture recognition result set of the new gesture control areas.

Further, after the intelligent device is controlled to execute the corresponding operation based on the first control gesture, if the gesture control area does not have the continuous N second control gestures after the continuous N first control gestures and none of the other gesture recognition areas have the continuous M wake-up gestures, the control focus is not transferred until the gesture control area has the control gesture meeting the requirement, and the corresponding operation is executed according to the control gesture. However, after the smart device is restarted, by default, no valid gesture control area exists, and the gesture control area needs to be redetermined according to the wake-up gesture that meets the requirements.

In some possible embodiments, aspects of the control method of a smart device provided by the present application may also be implemented in the form of a program product, which includes a program code for causing a computer device to perform the steps of the traffic control method according to the various exemplary embodiments of the present application described above when the program product is run on the computer device, for example, the computer device may perform the steps as shown in fig. 2.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The program product for traffic control of embodiments of the present application may employ a portable compact disc read only memory (CD-ROM) and include program code and may run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with a command execution system, apparatus, or device.

The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with a command execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's equipment, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. An intelligent device, comprising:

a display configured to display a screen;

respectively executing the following processes for each first image, wherein each first image is read, and one second image is intercepted from the first image according to a preset intercepting rule, wherein the ratio of the hand image of the user to the second image is higher than that of the hand image of the user to the first image; performing static gesture recognition on the second image, and determining a plurality of gesture recognition areas on the second image and a gesture recognition result corresponding to each gesture recognition area in at least one gesture recognition area; the static gesture recognition is used for carrying out gesture recognition operation on one image and outputting a gesture recognition result; the gesture recognition result corresponding to the gesture recognition area is determined based on the image corresponding to the gesture recognition area, and the gesture recognition area is the area where the hand is located;

Based on at least one gesture recognition area of the second image intercepted by each first image and gesture recognition results corresponding to the gesture recognition areas, if continuous M wake-up gestures exist in the same gesture recognition area, determining the same gesture recognition area as a gesture control area; and if the gesture control area is behind the continuous M wake-up gestures, controlling the intelligent device to execute corresponding operations based on the first control gestures; wherein M, N is a positive integer.

2. The smart device of claim 1, wherein the controller is configured to:

3. The smart device of claim 2, wherein the controller is further configured to:

4. The smart device of claim 1, wherein the controller is configured to:

5. A method of controlling a smart device, comprising:

6. The method of claim 5, wherein capturing a second image from the one first image according to a preset capture rule comprises:

7. The method of claim 6, further comprising, prior to determining the size information and corner coordinate information for the hand image:

8. The method of claim 5, further comprising, after controlling the smart device to perform a corresponding operation based on the first control gesture: