TWI800249B - How to customize gestures - Google Patents

Abstract

A method for defining custom gestures, comprising: a touch screen, a calculation unit connected to the touch screen, and a gesture database connected to the calculation unit, the method comprising the following steps: recording an input gesture trajectory data on the touch screen; converting the input gesture trajectory data into a two-dimensional trajectory graph of the input gesture and transmitting it to the calculation unit; the calculation unit... The unit sequentially reads one-dimensional and two-dimensional gesture reference images from the gesture database. If a two-dimensional gesture reference image can be successfully read, the calculation unit generates a set of one-dimensional and two-dimensional gesture reference images relative to the read two-dimensional gesture reference image. The calculation unit performs a similarity comparison between the two-dimensional trajectory image of the input gesture and each reference image in the set of two-dimensional gesture reference images.

Description

Methods of Customizing Gestures

A method for adding new gestures, especially a method for customizing gestures.

The detection and recognition of user gestures is another method of controlling electronic devices besides touch and voice control. In recent years, with the popularity of Augmented Reality (AR) and Virtual Reality (VR), it has gradually become a popular human-computer interaction interface, and is used in many fields such as games, tour guides, and healthcare. Doppler radar has advantages in detecting and recognizing user gestures, such as fast response, fewer blind spots, and the ability to be concealed. Therefore, Doppler radar has an irreplaceable advantage in the application of detecting and recognizing user gestures.

In practical applications, a device with gesture detection and recognition capabilities, in addition to its own gesture detection and recognition technology, must also consider the issue of user-defined gestures. This is because when users define gestures, they need to repeat the desired gestures multiple times to record and mark them. At the same time, the device also needs to be trained. This process of recording, marking, and training the user-defined gestures is often lengthy and uncertain. As a result, users of the device are often stumped by the recording, marking, and training process and lose patience. Consequently, the device with gesture detection and recognition capabilities often cannot fully exert its due effectiveness due to the problem of user-defined gestures.

Therefore, there is an urgent need to enable devices with gesture detection and recognition functions to add custom gestures with minimal effort from the user.

To solve the above problems, this invention discloses a method for customizing gestures, comprising the following technical content: Providing: a touch screen, a calculation unit connected to the touch screen, and a gesture database connected to the calculation unit. The method comprises the following steps: Step S1: Recording input gesture trajectory data on the touch screen; Step S2: Converting the input gesture trajectory data into a two-dimensional trajectory graphic of the input gesture and transmitting it to the calculation unit; Step S3: The calculation unit sequentially reads one-dimensional and two-dimensional gesture reference images from the gesture database. If the two-dimensional gesture reference image can be successfully read, the process jumps to step S4; if it cannot be successfully read, the process jumps to step S6. Step S4: The calculation unit generates a set of one-dimensional and two-dimensional gesture reference images corresponding to the read two-dimensional gesture reference image. Step S5: This calculation unit performs a similarity comparison between the 2D trajectory of the input gesture and each reference shape in the 2D gesture reference shape set to confirm whether the 2D trajectory of the input gesture is similar to the 2D gesture reference shape set. If any reference shape successfully matches the 2D trajectory of the input gesture, it is confirmed that the 2D trajectory of the input gesture is similar to the 2D gesture reference shape set, and the process ends; if the 2D trajectory of the input gesture is not similar to the 2D gesture reference shape set, the process jumps back to step S3. Step S6: Write the 2D trajectory of the input gesture into the gesture database to add a new 2D gesture reference graphic and assign a name to it. This input gesture is a newly added gesture.

Preferably, the two-dimensional trajectory of the input gesture and each reference graphic in the two-dimensional gesture reference graphic set are time series composed of multiple coordinate points on a preset one-dimensional coordinate plane.

Preferably, the similarity comparison in step S5 refers to the similarity comparison between two time series using the Dynamic Time Warping (DTW) method, and the input gesture in step S6 is a newly added gesture.

Please refer to Figure 1, which illustrates the terminal device, server, and cloud for implementing the custom gesture method of this invention. A terminal device 10 includes a touchscreen 11, a computing unit 12 connected to the touchscreen 11, a memory 13 connected to the computing unit 12, a communication module 14 connected to the computing unit 12, and a Doppler radar 15 connected to the computing unit 12. The terminal device 10 connects to the server 20 and cloud 21 via the communication module 15. Thus, with appropriate configuration, the terminal device 10 can share the burden of information storage and computation with the server 20 and cloud 21. The terminal device 10 can be a mobile device, such as a mobile phone, tablet computer, or other device with the above-mentioned functions; the touch screen 11 can be a general two-dimensional graphic touch input device; the computing unit 12 can be a system chip, such as a central processing unit (CPU) or micro-controller; the memory 13 can be solid-state memory or disk drive; the communication module 14 has wired or wireless communication functions, such as local area network (LAN), WiFi, 2G/3G/4G/5G/6G, etc.; and the Doppler radar 15 has the function of detecting three-dimensional gesture trajectory.

Please refer to Figure 2A, which shows a new gesture trajectory data input by the user. The user inputs a new gesture trajectory data 30 into the touch screen 11, and then the touch screen 11 transmits the new gesture trajectory data 30 to the calculation unit 12. The new gesture trajectory data 30 is a time sequence composed of the position points of multiple gesture trajectories.

Please refer to Figure 2B, which shows a two-dimensional trajectory of a new gesture. After receiving the new gesture trajectory data 30, the calculation unit 12 converts the new gesture trajectory data 30 into a two-dimensional trajectory of a new gesture 40.

Please refer to Figure 2C, which shows the composition of the two-dimensional trajectory graph 40 of the new gesture in Figure 2B. The two-dimensional trajectory graph 40 of the new gesture is composed of, for example, position points P0-P23 (only the labeled positions in Figure 2C), indicating that the user-input gesture moves sequentially from position point P0 to position point P23. These position points P0-P23 are obtained by a one-to-one conversion of the position points of multiple gesture trajectories in the new gesture trajectory data 30, where each position point P0-P23 corresponds one-to-one to the coordinate points (X0, Y0) ~ (X23, Y0). The coordinates (X0, Y0) of position point P0 are sampled at time t0, and the coordinates (X1, Y1) of position point P1 are sampled at time t1, and so on. Therefore, these position points P0-P23 also constitute a time series, where these coordinate points (X0, Y0)~(X23, Y23) are coordinate points on a preset two-dimensional X-Y coordinate plane. When these position points P0-P23 are sampled at fixed time intervals, the distance between any two adjacent points among these position points P0-P23 is proportional to the movement speed of the new gesture trajectory input by the user. As can be seen from the above, the two-dimensional trajectory diagram 40 of this new gesture is a time sequence composed of multiple coordinate points on a preset two-dimensional X-Y coordinate plane.

Please refer to Figure 3, which shows a two-dimensional gesture reference graph set 50. This two-dimensional gesture reference graph set 50 is generated from the two-dimensional gesture reference graph 500. The two-dimensional gesture reference graph 500 is stored in a gesture database. The two-dimensional gesture reference graph 500 is one of the existing two-dimensional gestures stored in the gesture database, which is arranged in a certain order. The two-dimensional gesture reference graph 500 is also a time sequence composed of multiple coordinate points on a preset two-dimensional X-Y coordinate plane. The gesture database is stored in memory 13. In one embodiment, the gesture database may also be stored on server 20 or cloud 21. The two-dimensional gesture reference image set 50 also has nine two-dimensional gesture extended reference images 501-509. These two-dimensional gesture extended reference images 501-509 are generated by various transformations of the two-dimensional gesture reference image 500, such as automatic transformation methods like scaling, shifting, rotating, and random cropping of the two-dimensional gesture reference image 500. Therefore, the two-dimensional gesture reference graph set 50 is composed of the two-dimensional gesture reference graph 500 and the two-dimensional gesture extended reference graphs 501-509. Each of the two-dimensional gesture extended reference graphs 501-509 is also a time sequence composed of multiple coordinate points on a preset two-dimensional X-Y coordinate plane.

Please refer to Figure 4, which shows the flowchart of the custom gesture method of this invention. The flowchart includes the following steps: Step S1: Record an input gesture trajectory data on the touch screen 11. Step S2: Convert the input gesture trajectory data into a two-dimensional trajectory graphic of the input gesture and transmit it to the calculation unit 12. Step S3: The calculation unit 12 sequentially reads a two-dimensional gesture reference graphic from the gesture database. If the two-dimensional gesture reference graphic can be successfully read, the process jumps to step S4; if the two-dimensional gesture reference graphic cannot be read, the process jumps to step S6. Step S4: This calculation unit 12 generates a set of two-dimensional gesture reference graphics relative to the read two-dimensional gesture reference graphics. Step S5: The calculation unit 12 performs a similarity comparison between the 2D trajectory of the input gesture and each reference shape in the 2D gesture reference shape set to confirm whether the 2D trajectory of the input gesture is similar to the 2D gesture reference shape set. If any reference shape successfully matches the 2D trajectory of the input gesture, it is confirmed that the 2D trajectory of the input gesture is similar to the 2D gesture reference shape set, and the process jumps to step S8; if the 2D trajectory of the input gesture is not similar to the 2D gesture reference shape set, the process jumps back to step S3. Step S6: Write the 2D trajectory of the input gesture to the gesture database to add a new 2D gesture reference graphic and assign it a name. Step S7: The process ends, and the input gesture becomes a newly added gesture. Step S8: The process ends, and the input gesture now exists.

Since the existing 2D gestures stored in the gesture database are arranged in a sequential order, in step S3, the process reads the corresponding 2D gesture reference image from the gesture database one by one until all the 2D gesture reference images in the gesture database have been read and there are no more 2D gesture reference images available for reading. Only then does the process jump to step S6.

In step S5, since the two-dimensional trajectory graph of the input gesture and each reference graph in the two-dimensional gesture reference graph set are time series composed of multiple coordinate points on the preset two-dimensional X-Y coordinate plane, the similarity comparison between the two time series can be performed using the Dynamic Time Warping (DTW) method used in time series analysis.

In step S6, since the input gesture trajectory data recorded in step S1 is not similar to any of the existing 2D gestures stored in the gesture database, the input gesture is indeed a new gesture. Therefore, the new gesture must be given a name and written into the gesture database to complete the creation of the new gesture.

In step S8, since the gesture already exists, the input gesture trajectory data recorded in step S1 obviously cannot be considered as a new gesture.

Please refer to Figure 5, which shows the testing process for the new gesture added in this invention. The process includes the following steps: Step S11: The calculation unit 12 reads the two-dimensional gesture reference image relative to the new gesture from the gesture database and generates a set of one-dimensional gesture reference images for the new gesture. Step S12: The Doppler radar 15 detects and records the three-dimensional new gesture test trajectory data relative to the new gesture and transmits the three-dimensional new gesture test trajectory data to the calculation unit 12. Step S13: The calculation unit 12 converts the three-dimensional new gesture test trajectory data into a two-dimensional test trajectory image of the new gesture. Step S14: Calculation unit 12 compares the 2D test trajectory of the new gesture with the 2D reference shape set for the new gesture. If the 2D test trajectory of the new gesture is not similar to the 2D reference shape set, the process jumps back to step S12. Step S15: The process ends, and the new gesture test is complete.

In order to test the two-dimensional reference image of a newly added gesture stored in the gesture database, in step S11, the calculation unit 12 first reads the two-dimensional reference image of the newly added gesture from the gesture database and generates a set of two-dimensional reference images of the new gesture. Then, the Doppler radar 15 detects and records the three-dimensional test trajectory data of the newly added gesture made by the user, and transmits the three-dimensional test trajectory data of the newly added gesture to the calculation unit 12. Then, the calculation unit 12 converts the three-dimensional new gesture test trajectory data into a new gesture two-dimensional test trajectory graph. That is, it projects the three-dimensional position points of multiple new gesture test trajectories in the three-dimensional new gesture test trajectory data one-to-one onto coordinate points on a preset two-dimensional X-Y coordinate plane to generate the new gesture two-dimensional test trajectory graph. Therefore, the new gesture two-dimensional test trajectory graph is a time series composed of multiple coordinate points on the preset two-dimensional X-Y coordinate plane. Then, the calculation unit 12 compares the two-dimensional test trajectory of the new gesture with the two-dimensional reference image set of the new gesture to see if they are similar. If they are not similar, the user needs to make the new gesture again to determine if the two-dimensional test trajectory of the new gesture is similar to the two-dimensional reference image set of the new gesture. The new gesture test is considered successful and the test process ends when the two are similar.

In one embodiment, when a user makes the new gesture, there is no need for a clear start or end. That is, the test trajectory data of the new gesture collected by the Doppler radar 15 can accumulate and expand over time. During the accumulation of the test trajectory data of the new gesture, as long as the test trajectory of the new gesture is similar to the two-dimensional gesture reference graph of the new gesture at any time, it can be confirmed that the new gesture made by the user has been successfully tested.

Although the processes and embodiments disclosed above in this invention all take the terminal device 10 as an example to perform the main data processing and data storage, servers and cloud terminals connected to the terminal device 10 can also undertake computing and data storage functions after appropriate configuration. Therefore, the execution device for data processing and data storage in the processes and embodiments of this invention is not limited to the terminal device 10.

The method for customizing gestures disclosed in this invention involves converting input gesture trajectory data input from a touchscreen into a two-dimensional trajectory graphic of the input gesture, and then sequentially comparing it with a set of two-dimensional gesture reference graphics corresponding to two-dimensional gesture reference graphics in a gesture database to confirm whether the input gesture trajectory data can become a new gesture. When the input gesture trajectory data can become a new gesture, this invention further discloses a process for testing the new gesture, so that the correctness and effectiveness of the new gesture can be further confirmed after it is completed. Because this invention discloses the use of a two-dimensional gesture reference image set for similarity comparison, when adding a gesture, the user only needs to input the gesture once. Furthermore, because this invention discloses that during the process of accumulating test trajectory data for a new gesture, as long as the test trajectory of the new gesture is similar to the two-dimensional gesture reference image set for any given period of time, the user does not need to explicitly start or end the gesture. Therefore, when testing a new gesture, the user can perform the gesture easily and naturally. This greatly reduces the time and patience required for the user when adding a gesture, thus achieving the purpose of this invention.

10: Terminal Device

11: Touchscreen

12: Calculation Unit

13: Memory

14: Communication Module

15: Dopleradar

20: Server

21: Cloud

30: Beginner's Pose Track Data

40: Two-dimensional trajectory graph of beginner gestures

50: 2D Gesture Reference Image Set

500: Two-dimensional gesture reference graphics

501-509: Reference diagrams for expanding two-dimensional gestures

Figure 1 is a schematic diagram of the terminal device, server, and cloud for implementing the custom gesture method of this invention. Figure 2A is a schematic diagram of the trajectory data of the new gesture of this invention. Figure 2B is a schematic diagram of the two-dimensional trajectory diagram of the new gesture of this invention. Figure 2C shows the composition of the two-dimensional trajectory diagram of the new gesture in Figure 2B. Figure 3 is a schematic diagram of the two-dimensional gesture reference graphic set of this invention. Figure 4 is a flowchart of the custom gesture method of this invention. Figure 5 is a flowchart of the testing method for the new gesture added in this invention.

Claims (6)

A method for defining custom gestures, comprising a touch screen, a calculation unit connected to the touch screen, and a gesture database connected to the calculation unit, the method comprising the following steps: Step S1: recording an input gesture trajectory data on the touch screen; Step S2: converting the input gesture trajectory data into a two-dimensional trajectory graphic of the input gesture and transmitting it to the calculation unit; Step S3: the calculation unit sequentially extracting gestures from the gesture database. The first step is to read a two-dimensional gesture reference image. If the two-dimensional gesture reference image can be successfully read, the process jumps to step S4; if the two-dimensional gesture reference image cannot be successfully read, the process jumps to step S6. Step S4: The calculation unit generates a set of two-dimensional gesture reference images relative to the read two-dimensional gesture reference image. Step S5: The calculation unit compares the two-dimensional trajectory image of the input gesture with each parameter in the set of two-dimensional gesture reference images. A similarity comparison is performed on the reference graphics to confirm whether the 2D trajectory of the input gesture is similar to the set of 2D gesture reference graphics. If any reference graphic successfully matches the 2D trajectory of the input gesture, the 2D trajectory of the input gesture is considered similar to the set of 2D gesture reference graphics, and the process ends. If the 2D trajectory of the input gesture is not similar to the set of 2D gesture reference graphics, the process jumps back to step S. Step 3; Step S6: Write the 2D trajectory of the input gesture into the gesture database to add a new 2D gesture reference shape and assign a name to the 2D gesture reference shape. The input gesture is a newly added gesture. The 2D gesture reference shape set includes the original 2D gesture reference shape and 2D gesture augmentation reference shapes generated through automatic transformation methods such as scaling, shifting, rotating, or random cropping of the original 2D gesture reference shape. The method for defining custom gestures as described in claim 1, wherein the two-dimensional trajectory graph of the input gesture and each reference graph in the two-dimensional gesture reference graph set are time series composed of multiple coordinate points on a preset one-dimensional coordinate plane. The method for the custom gesture as described in claim 1, wherein the similarity comparison in step S5 refers to performing a similarity comparison between two time series using a dynamic time warp method. The method for defining a custom gesture as described in claim 1 further includes a testing process for the newly added gesture, comprising the following steps: Step S11: The calculation unit reads a two-dimensional gesture reference image relative to the newly added gesture from the gesture database and generates a two-dimensional gesture reference image set for the newly added gesture; Step S12: A Doppler radar detects and records a three-dimensional newly added gesture test trajectory data relative to the newly added gesture, and sets the three-dimensional new gesture... The hand gesture test trajectory data is transmitted to the calculation unit; Step S13: The calculation unit converts the three-dimensional new hand gesture test trajectory data into a new hand gesture two-dimensional test trajectory graphic; Step S14: The calculation unit compares the new hand gesture two-dimensional test trajectory graphic with the new hand gesture two-dimensional reference graphic set. If the new hand gesture two-dimensional test trajectory graphic is not similar to the new hand gesture two-dimensional reference graphic set, the process jumps back to step S12. As described in claim 4, in step 13 of the method for defining custom gestures, the conversion of the 3D new gesture test trajectory data into a 2D new gesture test trajectory graphic by the calculation unit means that the calculation unit projects the 3D position points of multiple new gesture test trajectories in the 3D new gesture test trajectory data one-to-one onto coordinate points on a preset 2D coordinate plane to generate the 2D new gesture test trajectory graphic. As described in claim 4, in step 11 of the method for defining custom gestures, the three-dimensional new gesture test trajectory data is generated by the user making the new gesture. When the user makes the new gesture, the new gesture test trajectory data collected by the doppler radar can accumulate and expand over time. During the accumulation of the new gesture test trajectory data, as long as the new gesture test trajectory at any point in time is similar to the two-dimensional gesture reference graph set of the new gesture, it can be confirmed that the new gesture made by the user has been successfully tested.