CN110413111B - Target keyboard tracking system and method - Google Patents
Target keyboard tracking system and method Download PDFInfo
- Publication number
- CN110413111B CN110413111B CN201910615492.8A CN201910615492A CN110413111B CN 110413111 B CN110413111 B CN 110413111B CN 201910615492 A CN201910615492 A CN 201910615492A CN 110413111 B CN110413111 B CN 110413111B
- Authority
- CN
- China
- Prior art keywords
- keyboard
- tracking
- mid
- points
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
- Position Input By Displaying (AREA)
Abstract
The invention discloses a target keyboard tracking system and a target keyboard tracking method, which are applied to an interactive scene of moving intelligent glasses and paper keyboard input; the system comprises: the keyboard tracking module and the self-adaptive tracking module; the keyboard tracking module comprises the steps of keyboard extraction, characteristic point pair selection, transformation matrix calculation and keyboard coordinate conversion; the self-adaptive tracking module comprises sensor data acquisition, data segmentation analysis, state judgment and signal transmission for judging whether to continue tracking. The invention adopts the multithreading optimization technology to reduce the time delay of tracking processing; real-time keyboard tracking is realized, and the keyboard interaction input requirement of a user is well met.
Description
Technical Field
The invention belongs to the technical field of video processing and inertial sensing, and particularly relates to a target keyboard tracking system and a target keyboard tracking method, which are applied to an interactive scene of moving intelligent glasses and paper keyboard input.
Background
In recent years, the appearance and volume of mobile devices tend to be portable and small, and people generally have a habit of carrying a smart phone, smart glasses or a smart watch instead of carrying a heavy and large notebook computer. However, interacting on such small smart mobile devices is a great challenge, and the typical pain point is that there is no physical keyboard to assist in typing, resulting in very inefficient input.
The current solutions are mainly wearable keyboards, screen keyboards and voice-based keyboard input. The wearable keyboard introduces additional equipment burden, resulting in an unfriendly user experience; the screen keyboard usually occupies large-area screen resources, and is unrealistic to deploy on the current intelligent glasses and intelligent watches; although the keyboard based on the audio signal removes the additional hardware burden, it requires the user to make a very distinct harsh sound with the fingertips and nails when clicking the keys, and this unnatural input mode also makes the user experience poor. Meanwhile, built-in sensors on mobile devices such as smart phones and smart glasses are more and more abundant, the mobile devices are generally provided with cameras and inertial sensors, and the energy consumption of the sensors is very low, so that a new, more convenient and efficient input interaction mode is possible to generate.
Disclosure of Invention
In view of the above-mentioned deficiencies of the prior art, the present invention provides a system and a method for tracking a target keyboard, so as to solve the problems caused by the conventional keyboard input. According to the invention, the intelligent glasses are connected with the printed common paper keyboard through the camera, so that the input experience similar to a physical keyboard can be provided for a user, the change of the coordinate position of the keyboard caused by the shaking of the head of the user in the input interaction process is tracked, the self-adaptive tracking transformation is completed according to the motion state of the equipment monitored by the low-energy-consumption sensor, the energy consumption of the equipment is reduced in a multi-mode perception data fusion mode, and the scene application of high-efficiency and low-energy-consumption keyboard input interaction is finally realized.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
the invention discloses a target keyboard tracking system, which is applied to intelligent glasses equipment and comprises: a keyboard tracking module and an adaptive tracking module, wherein,
a keyboard tracking module for detecting and extracting coordinates of the keyboard and the keys, selecting four corresponding groups of feature point pairs between two continuous frames based on the inherent geometric features of the keyboard, calculating a transformation matrix C by using the four groups of feature points, and calculating the coordinates [ X ] of the previous framei,Yi,1]Calculating to obtain the coordinate [ U ] of the current frame by a left multiplication transformation matrix Ci,Vi,Wi]Compressing the current frame coordinate in the Z direction to convert the keyboard coordinate;
the self-adaptive tracking module acquires the rotation motion data of the intelligent glasses equipment through the inertial sensor, calculates and analyzes the sensor data based on a sliding window, judges whether to continuously track the target keyboard in real time or not by comparing the mean value and the variance of the data in the sliding window with a given experience threshold value, and sends the generated judgment result data to the keyboard tracking module.
Further, the adaptive tracking module specifically further includes: if the intelligent glasses equipment is in the motion state, a tracking starting signal is sent to enable the keyboard tracking module to continuously track the coordinates of the keyboard; and if the intelligent glasses equipment is in a static state, sending a tracking stopping signal to stop the tracking of the keyboard tracking module.
Further, the selecting of the characteristic point pairs specifically includes:
a. removing the influence of the hand region on the feature point selection: segmenting out the possible hands in the frame picture by using skin color detection, and then filling background white to wipe out the hand area;
b. extracting convex hull points of the keyboard area: after removing the hand area, detecting the outlines of the rest keyboards, and extracting convex hull point sequences of the keyboards according to outline results, and recording the convex hull point sequences as C;
c. firstly, the first two convex points P in the C are taken outbeginAnd PmidIf the distance between the two is less than 10, the two are very close and do not conform to the size characteristics of the keyboard, and then P is updatedmidThe next convex point in C until PbeginAnd PmidThe distance between them is greater than 10; obtaining the appropriate PbeginAnd PmidThen, taking out a convex package point from C and marking as Pend(ii) a As above, if PmidAnd PendIs less than 10, P is updatedendThe next convex point in C until PmidAnd PendThe distance between them is greater than 10;
d. computing vectors
Sum vector
If the included angle is smaller than the threshold value of 8 degrees, the three convex points have the same direction, and P ismidIs updated to PendReturning to step c) to continue taking Pend(ii) a If the included angle is larger than 8 degrees, the three convex points are represented as PmidThere is a significant turning, at which point P will bemidStored as detected feature points and stores PbeginIs updated to Pmid，PmidIs updated to PendReturning to C to continue taking Pend；
e. After the convex hull point set C is analyzed in a traversing manner, the number of the stored convex hull points is four or five; if the number of the feature points is four, directly taking the feature points as detected feature points, and skipping to the step f) for execution; if five, the following treatment is required: the five convex hull points are connected in sequence and are arranged according to the absolute value of the slope of the straight lineSequencing; removing the straight line with the slope value of the median, and recording the rest of the straight lines as [ l ] according to the relative order0,l1,l2,l3]，l0And l2、l3Intersect, l1And l2、l3Intersecting to obtain four feature points, namely detected feature point results;
f. recording the convex hull point set of two continuous frames as C1And C2To C1And C2The operations of the steps b) to e) are carried out, so that two groups of characteristic points which are respectively marked as (X) can be obtainedi,Yi) And (U)i,Vi) I is more than or equal to 0 and less than or equal to 3, the two groups of feature points are sorted according to the horizontal and vertical coordinate values, and a mapping relation is established according to the sequence of upper left, upper right, lower left and lower right to obtain four feature point pairs (X)0,Y0,U0,V0)。
Further, the method for calculating the transformation matrix specifically comprises: according to the characteristic point pair (X)i,Yi,Ui,Vi) And i is more than or equal to 0 and less than or equal to 3, establishing a linear equation set, and calculating a transformation matrix C:
further, the method for converting the coordinates of the keyboard specifically comprises the following steps: let the keyboard coordinate of the previous frame be (X)i,Yi) Converted into coordinates (U) of the framei,Vi) Where C is a transformation matrix:
the invention discloses a target keyboard tracking method, which is applied to an intelligent glasses device and paper keyboard input interaction scene and comprises the following steps:
1) acquiring a video input and interaction by a user on a paper keyboard at a rate of 30 frames per second by using a camera of intelligent glasses equipment, and acquiring gyroscope data at a frequency of 50Hz in the video acquisition process;
2) smoothing and morphologically processing the acquired frame sequence in real time to reduce noise;
3) for the first frame of the frame sequence, detecting and extracting coordinates of the keyboard and each key;
4) for the subsequent frame sequence, selecting and matching angular points of the keyboard between two continuous frames as characteristic point pairs;
5) calculating a corresponding transformation matrix according to the selected characteristic point pairs, and transforming the coordinates of the previous frame into the current frame;
6) the inertial sensor is used for collecting motion data of the intelligent glasses equipment and monitoring the state of the intelligent glasses equipment to judge whether to continuously track the keyboard.
Further, the method for completing adaptive tracking by monitoring the state of the smart eyewear device with the sensor in step 6) includes:
61) monitoring the state of the intelligent glasses equipment in real time by using a gyroscope, and receiving sensor data;
62) calculating the mean value and the variance of the data of the three-axis sensor by using a sliding window with the size of 10, and when the mean value and the variance of the data in the sliding window are both smaller than a threshold value, indicating that the equipment is kept still at the moment, sending a tracking stopping signal, and stopping the coordinate tracking of the keyboard; when the mean value or the variance of the data in the window is larger than the threshold value, the device at the moment is moving, a signal for starting tracking is sent, and the real-time tracking of the coordinates of the keyboard is continued.
The step 1), the step 2 to the step 5) and the step 6) adopt independent threads to process in parallel respectively; capturing a sequence of video frames in parallel using three threads, extracting feature points and converting coordinates, and analyzing sensor data to decide whether to continue tracking; and by means of multi-thread parallel processing, the processing time of tracking conversion is reduced, and real-time keyboard tracking is realized.
The invention has the beneficial effects that:
the invention can reduce the hardware burden in the keyboard input process of the user, and only one printed common paper keyboard is required to be connected with the intelligent glasses equipment through the camera; the application scene is that the user wears the intelligent glasses to complete certain typing tasks (such as sending messages), and the keyboard tracking module can calculate the transformation process of the equipment in a matrix form in real time and convert the coordinates of the keyboard and each key; in addition, the invention also utilizes the low-energy-consumption inertial sensor equipped in the equipment to monitor the motion state of the equipment in real time, realizes the self-adaptive adjustment of whether to track the keyboard, and reduces the high-energy-consumption operation of the real-time tracking of the keyboard as far as possible, thereby reducing the overall energy consumption of the equipment on the basis of not obviously reducing the keyboard input precision of the user, and simultaneously bringing more natural keyboard input experience of the intelligent equipment to the user.
Drawings
FIG. 1 is a block diagram of a system according to the present invention.
Fig. 2 shows a feature point selection process flow diagram.
FIG. 3 is a flow chart of a method.
FIG. 4 is a diagram illustrating multi-thread processing.
FIG. 5a is a schematic diagram of a system initialization interface.
FIG. 5b is a schematic diagram of the system start-up interface.
FIG. 5c is a schematic diagram of the interface of the system operation process.
FIG. 5d is a schematic diagram of a system tracking keyboard interface.
FIG. 5e is a schematic diagram of the resume run interface after the system tracks the keyboard.
FIG. 5f is a schematic diagram of the system end run interface.
Detailed Description
In order to facilitate understanding of those skilled in the art, the present invention will be further described with reference to the following examples and drawings, which are not intended to limit the present invention.
Referring to fig. 1, the target keyboard tracking system of the present invention, applied in an interactive scene of smart glasses and paper keyboard input, includes: a keyboard tracking module and an adaptive tracking module, wherein,
a keyboard tracking module for detecting and extracting coordinates of the keyboard and the keys, selecting four corresponding groups of feature point pairs between two continuous frames based on the inherent geometric features of the keyboard, calculating a transformation matrix C by using the four groups of feature points, and calculating the coordinates [ X ] of the previous framei,Yi,1]Calculating to obtain the coordinate [ U ] of the current frame by a left multiplication transformation matrix Ci,Vi,Wi]Compressing the current frame coordinate in the Z direction to convert the keyboard coordinate;
the self-adaptive tracking module acquires the rotation motion data of the intelligent glasses equipment through the inertial sensor, calculates and analyzes the sensor data based on a sliding window, judges whether to continuously track the target keyboard in real time or not by comparing the mean value and the variance of the data in the sliding window with a given experience threshold value, and sends the generated judgment result data to the keyboard tracking module.
If the intelligent glasses equipment is in the motion state, a tracking starting signal is sent to enable the keyboard tracking module to continuously track the coordinates of the keyboard; and if the intelligent glasses equipment is in a static state, sending a tracking stopping signal to enable the keyboard tracking module to stop tracking the coordinates of the keyboard.
Referring to fig. 2, the method for selecting feature points based on the inherent geometric features of the keyboard between two consecutive frames includes:
a. removing the influence of the hand region on the feature point selection: the user naturally performs keyboard input in the captured video frame sequence, but the keyboard is shielded by hands to influence the selection of the feature points; firstly, segmenting out the possible hands in a frame picture by using skin color detection, and then filling background white to wipe out the hand area;
b. extracting convex hull points of the keyboard area: after removing the hand area, detecting the outlines of the rest keyboards, extracting a convex hull point sequence of the keyboards according to the outline result to be recorded as C, and extracting four corner points of the keyboards from the C to be characteristic points;
c. firstly, the first two convex points P in the C are taken outbeginAnd PmidIf the distance between the two is less than 10, the two are very close and do not conform to the size characteristics of the keyboard, and then P is updatedmidThe next convex point in C until PbeginAnd PmidThe distance between them is greater than 10; obtaining the appropriate PbeginAnd PmidThen, taking out a convex package point from C and marking as Pend(ii) a As above, if PmidAnd PendIs less than 10, P is updatedendThe next convex point in C until PmidAnd PendThe distance between them is greater than 10;
d. computing vectors
Sum vector
If the included angle is smaller than the threshold value of 8 degrees, the three convex points have the same direction, and P ismidIs updated to PendReturning to step c) to continue taking Pend(ii) a If the included angle is larger than 8 degrees, the three convex points are represented as PmidThere is a significant turning, at which point P will bemidStored as detected feature points and stores PbeginIs updated to Pmid，PmidIs updated to PendReturning to C to continue taking Pend；
e. After the convex hull point set C is analyzed in a traversing manner, the number of the stored convex hull points is four or five; if the number of the feature points is four, the feature points are directly used as the last feature points; if five, the following treatment is required: connecting the five convex hull points in sequence, and sequencing according to the absolute value of the slope of the straight line; since the edge of the keyboard is close to horizontal and vertical, the absolute value of the slope is either close to 0 or tends to be large, the lines with the median slope value are removed, and the remaining lines are noted in relative order as l0,l1,l2,l3]，l0And l3、l4Intersect, l1And l3、l4Intersect to obtain fourFeature points, i.e. the final feature point result;
f. recording the convex hull point set of two continuous frames as C1And C2(ii) a To C1And C2The operations of the steps b) to e) are carried out, so that two groups of characteristic points which are respectively marked as (X) can be obtainedi,Yi) And (U)i,Vi) (i is more than or equal to 0 and less than or equal to 3), sorting according to the magnitude of the horizontal and vertical coordinate values, establishing a mapping relation between the two groups of characteristic points according to the sequence of upper left, upper right, lower left and lower right to obtain (X)i,Yi,Ui,Vi)。
The method for calculating the transformation matrix and converting the keyboard coordinate specifically comprises the following steps:
a. let the keyboard coordinate (X) of the previous framei,Yi) Is converted into the frame (U)i,Vi) Wherein C is a transformation matrix,
b. according to the characteristic point pair (X)i,Yi) And (U)i,Vi) (i is more than or equal to 0 and less than or equal to 3) establishing a linear equation system, and calculating a transformation matrix C:
referring to fig. 3, the target keyboard tracking method of the present invention is applied in an intelligent glasses device and paper keyboard input interaction scenario, and includes the following steps:
1) the method comprises the steps that a camera of intelligent glasses equipment is used for collecting a video input by a user on a paper keyboard (a printed common paper keyboard) at a rate of 30 frames per second, and gyroscope data are collected at a frequency of 50Hz in the video collection process;
2) smoothing and morphologically processing the acquired frame sequence in real time to reduce noise;
3) for the first frame of the frame sequence, detecting and extracting coordinates of the keyboard and each key;
4) for the subsequent frame sequence, selecting and matching angular points of the keyboard between two continuous frames as characteristic point pairs;
5) calculating a corresponding transformation matrix according to the selected characteristic point pairs, and transforming the coordinates of the previous frame into the current frame;
6) the inertial sensor is used for collecting motion data of the intelligent glasses equipment and monitoring the state of the intelligent glasses equipment to judge whether to continuously track the keyboard.
Referring to fig. 4, the step 1), the steps 2 to 5), and the step 6) are respectively processed in parallel by using separate threads; in particular, three threads are used to capture a sequence of video frames in parallel, extract feature points and transform coordinates, and analyze sensor data to decide whether to continue tracking; and by means of multi-thread parallel processing, the processing time of tracking conversion is reduced, and real-time keyboard tracking is realized.
Fig. 5 a-5 f are schematic diagrams of keyboard tracking system interaction interfaces. When the user runs the program and enters the program main interface, the preview picture of the camera can be seen, as shown in fig. 5 a. When the user clicks the Capture button in the menu bar, the program starts to work, extracting the keyboard and keys for the first frame captured, and tracking the key coordinates of the keyboard in real time in the frame sequence captured later, as shown in fig. 5 b. While the real-time tracking is being performed, the program will collect the data of the gyro sensor in the background at a frequency of 50Hz, and fig. 5c to 5e are diagrams of the user keyboard interaction scenario, where the entry "Update keyMap" shown in fig. 5d indicates that the tracking of the keyboard is being performed at this time. And after the user finishes the interactive task of the keyboard, clicking the Capture button in the menu bar again to stop the system.
While the invention has been described in terms of its preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.
Claims (6)
1. The utility model provides a target keyboard tracking system, is applied to in the intelligent glasses equipment, its characterized in that includes: a keyboard tracking module and an adaptive tracking module, wherein,
a keyboard tracking module for detecting and extracting coordinates of the keyboard and the keys, selecting four corresponding groups of feature point pairs between two continuous frames based on the inherent geometric features of the keyboard, calculating a transformation matrix C by using the four groups of feature points, and calculating the coordinates [ X ] of the previous framei，Yi，1]Calculating to obtain the coordinate [ U ] of the current frame by a left multiplication transformation matrix Ci，Vi，Wi]Compressing the current frame coordinate in the Z direction to convert the keyboard coordinate;
the self-adaptive tracking module acquires the rotation motion data of the intelligent glasses equipment through the inertial sensor, calculates and analyzes the sensor data based on a sliding window, judges whether to continuously track the target keyboard in real time or not by comparing the mean value and the variance of the data in the sliding window with a given experience threshold value, and sends the generated judgment result data to the keyboard tracking module;
the characteristic point pair selection specifically comprises the following steps:
a. removing the influence of the hand region on the feature point selection: segmenting out the possible hands in the frame picture by using skin color detection, and then filling background white to wipe out the hand area;
b. extracting convex hull points of the keyboard area: after removing the hand area, detecting the outlines of the rest keyboards, and extracting convex hull point sequences of the keyboards according to outline results, and recording the convex hull point sequences as C;
c. firstly, the first two convex points P in the C are taken outbeginAnd PmidIf the distance between the two is less than 10, the two are very close and do not conform to the size characteristics of the keyboard, and then P is updatedmidThe next convex point in C until PbeginAnd PmidThe distance between them is greater than 10; obtaining the appropriate PbeginAnd PmidThen, taking out a convex package point from C and marking as Pend(ii) a As above, if PmidAnd PendIs less than 10, P is updatedendThe next convex point in C until PmidAnd PendThe distance between them is greater than 10;
d. computing vectors
Sum vector
If the included angle is smaller than the threshold value of 8 degrees, the three convex points have the same direction, and P ismidIs updated to PendReturning to step c) to continue taking Pend(ii) a If the included angle is larger than 8 degrees, the three convex points are represented as PmidThere is a significant turning, at which point P will bemidStored as detected feature points and stores PbeginIs updated to Pmid，PmidIs updated to PendReturning to C to continue taking Pend；
e. After the convex hull point set C is analyzed in a traversing manner, the number of the stored convex hull points is four or five; if the number of the feature points is four, directly taking the feature points as detected feature points, and skipping to the step f) for execution; if five, the following treatment is required: connecting the five convex hull points in sequence, and sequencing according to the absolute value of the slope of the straight line; removing the straight line with the slope value of the median, and recording the rest of the straight lines as [ l ] according to the relative order0，l1，l2，l3]，l0And l2、l3Intersect, l1And l2、l3Intersecting to obtain four feature points, namely detected feature point results;
f. recording the convex hull point set of two continuous frames as C1And C2To C1And C2The operations of the steps b) to e) are carried out, so that two groups of characteristic points which are respectively marked as (X) can be obtainedi，Yi) And (U)i，Vi)，0≤i≤3Sorting according to the magnitude of the horizontal and vertical coordinate values, establishing a mapping relation between two groups of characteristic points according to the sequence of upper left, upper right, lower left and lower right, and obtaining four characteristic point pairs (X)i，Yi，Ui，Vi)。
2. The target keyboard tracking system of claim 1, wherein the adaptive tracking module further comprises: if the intelligent glasses equipment is in the motion state, a tracking starting signal is sent to enable the keyboard tracking module to continuously track the coordinates of the keyboard; and if the intelligent glasses equipment is in a static state, sending a tracking stopping signal to enable the keyboard tracking module to stop tracking the coordinates of the keyboard.
3. The system for tracking a target keyboard of claim 1, wherein the method for computing a transformation matrix is specifically: according to the characteristic point pair (X)i，Yi，Ui，Vi) And i is more than or equal to 0 and less than or equal to 3, establishing a linear equation set, and calculating a transformation matrix C:
4. the system for tracking a target keyboard of claim 3, wherein the method for transforming the coordinates of the keyboard comprises: let the keyboard coordinate of the previous frame be (X)i，Yi) Converted into coordinates (U) of the framei，Vi)：
5. A target keyboard tracking method is applied to an intelligent glasses device and paper keyboard input interaction scene, and is characterized by comprising the following steps:
1) acquiring a video input and interaction by a user on a paper keyboard at a rate of 30 frames per second by using a camera of intelligent glasses equipment, and acquiring gyroscope data at a frequency of 50Hz in the video acquisition process;
2) performing real-time smooth filtering and morphological processing on the acquired frame sequence;
3) for the first frame of the frame sequence, detecting and extracting coordinates of the keyboard and each key;
4) for the subsequent frame sequence, selecting and matching angular points of the keyboard between two continuous frames as characteristic point pairs;
5) calculating a corresponding transformation matrix according to the selected characteristic point pairs, and transforming the coordinates of the previous frame into the current frame;
6) acquiring motion data of the intelligent glasses equipment by using the inertial sensor, monitoring the state of the intelligent glasses equipment, and judging whether to continuously track the keyboard;
the characteristic point pair selection specifically comprises the following steps:
a. removing the influence of the hand region on the feature point selection: segmenting out the possible hands in the frame picture by using skin color detection, and then filling background white to wipe out the hand area;
b. extracting convex hull points of the keyboard area: after removing the hand area, detecting the outlines of the rest keyboards, and extracting convex hull point sequences of the keyboards according to outline results, and recording the convex hull point sequences as C;
c. firstly, the first two convex points P in the C are taken outbeginAnd PmidIf the distance between the two is less than 10, the two are very close and do not conform to the size characteristics of the keyboard, and then P is updatedmidThe next convex point in C until PbeginAnd PmidThe distance between them is greater than 10; obtaining the appropriate PbeginAnd PmidThen, taking out a convex package point from C and marking as Pend(ii) a As above, if PmidAnd PendIs less than 10, P is updatedendThe next convex point in C until PmidAnd PendThe distance between them is greater than 10;
d. computing vectors
Sum vector
If the included angle is smaller than the threshold value of 8 degrees, the three convex points have the same direction, and P ismidIs updated to PendReturning to step c) to continue taking Pend(ii) a If the included angle is larger than 8 degrees, the three convex points are represented as PmidThere is a significant turning, at which point P will bemidStored as detected feature points and stores PbeginIs updated to Pmid，PmidIs updated to PendReturning to C to continue taking Pend；
e. After the convex hull point set C is analyzed in a traversing manner, the number of the stored convex hull points is four or five; if the number of the feature points is four, directly taking the feature points as detected feature points, and skipping to the step f) for execution; if five, the following treatment is required: connecting the five convex hull points in sequence, and sequencing according to the absolute value of the slope of the straight line; removing the straight line with the slope value of the median, and recording the rest of the straight lines as [ l ] according to the relative order0，l1，l2，l3]，l0And l2、l3Intersect, l1And l2、l3Intersecting to obtain four feature points, namely detected feature point results;
f. recording the convex hull point set of two continuous frames as C1And C2To C1And C2The operations of the steps b) to e) are carried out, so that two groups of characteristic points which are respectively marked as (X) can be obtainedi，Yi) And (U)i，Vi) I is more than or equal to 0 and less than or equal to 3, sorting the two groups of feature points according to the magnitude of the horizontal and vertical coordinate values, and arranging the two groups of feature points according to the upper leftEstablishing mapping relation in the sequence of upper right, lower left and lower right to obtain four characteristic point pairs (X)i，Yi，Ui，Vi)。
6. The method for tracking the target keyboard according to claim 5, wherein the method for performing adaptive tracking by monitoring the state of the smart glasses device with the sensor in step 6) comprises:
61) monitoring the state of the intelligent glasses equipment in real time by using a gyroscope, and receiving sensor data;
62) calculating the mean value and the variance of the data of the three-axis sensor by using a sliding window with the size of 10, and when the mean value and the variance of the data in the sliding window are both smaller than a threshold value, indicating that the equipment is kept still at the moment, sending a tracking stopping signal, and stopping the coordinate tracking of the keyboard; when the mean value or the variance of the data in the window is larger than the threshold value, the device at the moment is moving, a signal for starting tracking is sent, and the real-time tracking of the coordinates of the keyboard is continued.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910615492.8A CN110413111B (en) | 2019-07-09 | 2019-07-09 | Target keyboard tracking system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910615492.8A CN110413111B (en) | 2019-07-09 | 2019-07-09 | Target keyboard tracking system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110413111A CN110413111A (en) | 2019-11-05 |
| CN110413111B true CN110413111B (en) | 2021-06-01 |
Family
ID=68360748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910615492.8A Active CN110413111B (en) | 2019-07-09 | 2019-07-09 | Target keyboard tracking system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413111B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105229574A (en) * | 2013-01-14 | 2016-01-06 | 纽昂斯通信有限公司 | Reduce the error rate based on the keyboard touched |
| CN106933376A (en) * | 2017-03-23 | 2017-07-07 | 哈尔滨拓博科技有限公司 | A kind of scaling method of smooth projected keyboard |
| CN107066935A (en) * | 2017-01-25 | 2017-08-18 | 网易（杭州）网络有限公司 | Hand gestures method of estimation and device based on deep learning |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101907954B (en) * | 2010-07-02 | 2012-06-13 | 中国科学院深圳先进技术研究院 | Interactive projection system and interactive projection method |
| JP6362391B2 (en) * | 2014-04-10 | 2018-07-25 | キヤノン株式会社 | Information processing terminal, information processing method, and computer program |
| CN206711065U (en) * | 2017-03-23 | 2017-12-05 | 哈尔滨拓博科技有限公司 | Device for nominal light projected keyboard |
| CN107507247B (en) * | 2017-08-28 | 2018-09-11 | 哈尔滨拓博科技有限公司 | A kind of real-time dynamic autoization scaling method of projected keyboard |
| CN108829268A (en) * | 2018-07-11 | 2018-11-16 | 南京邮电大学 | Keyboard and input method based on single RGB camera |
-
2019
- 2019-07-09 CN CN201910615492.8A patent/CN110413111B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105229574A (en) * | 2013-01-14 | 2016-01-06 | 纽昂斯通信有限公司 | Reduce the error rate based on the keyboard touched |
| CN107066935A (en) * | 2017-01-25 | 2017-08-18 | 网易（杭州）网络有限公司 | Hand gestures method of estimation and device based on deep learning |
| CN106933376A (en) * | 2017-03-23 | 2017-07-07 | 哈尔滨拓博科技有限公司 | A kind of scaling method of smooth projected keyboard |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110413111A (en) | 2019-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Hand gesture recognition using depth data | |
| US10095033B2 (en) | Multimodal interaction with near-to-eye display | |
| CN101344816B (en) | Human-computer interaction method and device based on gaze tracking and gesture recognition | |
| Rautaray et al. | Real time multiple hand gesture recognition system for human computer interaction | |
| CN101697199B (en) | Detection method of head-face gesture and disabled assisting system using same to manipulate computer | |
| CN103135754B (en) | Adopt interactive device to realize mutual method | |
| CN104410883B (en) | The mobile wearable contactless interactive system of one kind and method | |
| CN103984928B (en) | Finger gesture recognition methods based on depth image | |
| CN101976330B (en) | Gesture recognition method and system | |
| CN103353935A (en) | 3D dynamic gesture identification method for intelligent home system | |
| CN102520790A (en) | Character input method based on image sensing module, device and terminal | |
| CN101901052A (en) | The method of target control with mutual reference of both hands | |
| CN106200971A (en) | Man-machine interactive system device based on gesture identification and operational approach | |
| CN111209812A (en) | Target face picture extraction method and device and terminal equipment | |
| CN111898407A (en) | A Human-Computer Interaction Operating System Based on Face Action Recognition | |
| CN107357414B (en) | Click action recognition method and device | |
| CN108521594B (en) | Free viewpoint video playing method based on motion sensing camera gesture recognition | |
| CN106814853A (en) | A kind of eye control tracking based on machine learning | |
| CN106909871A (en) | Gesture instruction recognition methods | |
| Zhu et al. | Toward real-time human-computer interaction with continuous dynamic hand gestures | |
| CN110413111B (en) | Target keyboard tracking system and method | |
| CN114610156A (en) | Interaction method and device based on AR/VR glasses and AR/VR glasses | |
| JPH0648458B2 (en) | Information input device | |
| CN113660420B (en) | Video frame processing method and video frame processing device | |
| Choi et al. | RGB-D camera-based hand shape recognition for human-robot interaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |