JPH10208023A - Sign language recognition device - Google Patents

Abstract

(57) [Summary] [Problem] To correctly evaluate basic units of all actions and recognize sign language with high accuracy. In other words, the problem of the recognition range of the basic unit of the static operation or the sequential operation unit composed only of the basic unit of the static operation is affected by the detection time of the basic unit of each operation. The problem can be solved and an appropriate evaluation value of the basic unit of operation can be obtained, thereby improving the sign language recognition accuracy. After detecting a basic unit of a dynamic operation and recognizing a sequential operation unit composed of the basic units, a static operation within a time range of the recognized sequential operation unit is detected. Integrate the evaluation results of the basic units. Recognition is performed for a sequential operation unit composed of only basic units of static operation by obtaining an evaluation value of the entire operation at each time and obtaining a time at which the evaluation value reaches a maximum.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a method for inputting sign language,
The present invention relates to a sign language recognition device that supports communication between a hearing-impaired person and a hearing-impaired person by accurately outputting the result in the form of voice or text.

[0002]

2. Description of the Related Art Conventionally, various devices have been proposed in which a sign language is input, the input result is analyzed, and the sign language is recognized. These conventional recognition methods include a technique of pattern matching using sign language motion data (for example, Japanese Patent Application No. Hei 5-12569).
No. 8 and the drawings), a technique of performing recognition using a neural network technique, and a technique of performing sign language recognition based on a basic unit of motion constituting a sign language motion (for example, Japanese Patent Application No. Hei 6-253457). References and drawings). The former method (Japanese Patent Application No. 5-125698) is a method in which the entire hand movement pattern is compared with a standard hand movement pattern stored in a word dictionary and recognized based on whether or not they match, and the latter ( Japanese Patent Application No. 6-25345
In the method of No. 7), instead of comparing the hand movement patterns themselves, the hand movement patterns are first recognized for each basic unit of movement (partial pattern), and then the results of the basic units of the movement are integrated to form a word. It recognizes. As described above, in the latter technique, first, all basic units of the operation constituting the sign language for recognition are recognized. Recognition is performed by comparing the result with the temporal relationship between the basic units of the motion in the sign language template in which the sign language is previously stored as a combination of the basic units of the motion. In this case, the types and attributes of the basic units of the motion in the sign language template are all described by symbols.

[0003]

In the conventional sign language recognition technology based on the basic unit of motion (the latter technology described above), the basic unit of static motion and the basic unit of dynamic motion are simultaneously recognized. Sign language was recognized by integrating the basic units of the recognized motion. Among these, the basic unit of the static operation is a feature of the operation indicating a state where the parameters are stable in a certain time range such as a shape (a hand shape or the like) or a direction (a finger direction or the like). On the other hand, the basic unit of the dynamic operation is a feature of the operation representing a change (movement or the like) of a parameter in a certain time range such as a linear motion or a circular motion. In order to detect a basic unit of static operation, a threshold is set for a difference between a reference parameter (a parameter extracted from a sample taken therefrom) and an input parameter, and a section within the threshold is set. May be detected by detecting However, since the parameter of the basic unit of the static operation is easily changed, it is often recognized in a range larger than the time range to be recognized or detected in a smaller range. For example, if the input parameters fluctuate in the same manner each time at the boundary of the threshold value, the threshold value may reciprocate, and as a result, may be detected as a short break section. Conversely, a case may be detected in which a large section is detected. Further, depending on the state of parameter fluctuation, one operation section may be recognized as being divided into two or more sections. For this reason, in the related art in which the basic unit of the static operation and the basic unit of the dynamic operation are treated equally, there is a problem that the basic unit of the static operation cannot be correctly evaluated. Further, in the conventional technology, since all sign language templates are described by symbols representing basic units of motion, the basic units of motion are recognized based on a preset reference value, and the evaluation value is also used as a reference. The calculation was based on that. Furthermore, the recognition result of the sign language is also calculated based on the evaluation value of the basic unit of the recognized motion (a value indicating how close to the reference value, the higher the value, the closer). For this reason, there is a problem that a correct evaluation value is often not obtained due to a difference between a parameter range in an actual operation and a parameter range given by a predetermined reference value, and recognition accuracy is low.
That is, the location where the parameters are concentrated is not always close to the reference value, and may be concentrated at an intermediate position between the reference values.
In such a case, the difference between the input parameter and the reference value increases, and the evaluation value decreases. Therefore, an object of the present invention is to solve such a conventional problem and to provide a sign language recognition device that correctly evaluates the basic units of all operations and recognizes sign language with high accuracy.

[0004]

In order to achieve the above object, a sign language recognition apparatus according to the present invention detects a basic unit of a dynamic operation and recognizes a sequential operation unit constituted by the basic units. And integrating the evaluation result of the basic unit of the static operation within the time range of the recognized sequential operation unit. Recognition is performed for a sequential operation unit composed of only basic units of static operation by obtaining an evaluation value of the entire operation at each time and obtaining a time at which the evaluation value reaches a maximum. Also, in order to correctly evaluate the basic unit of motion, the basic unit of motion described in the sign language template is
The attribute value determined using the actual sign language data is described using a continuous amount. The evaluation value of the basic unit of the recognized action is
It is determined using the continuous amount. As described above, since the basic unit of the static operation is evaluated based on the time range determined from the recognition result of the basic unit of the dynamic operation, there is no problem regarding the recognition range of the basic unit of the static operation. In addition, for the sequential operation unit composed only of the basic unit of the static operation, the evaluation value of the entire sequential operation unit is obtained at each time, and the time when the evaluation value becomes the maximum value is detected. By doing so, the recognition is performed, so that the detection time of the basic unit of each operation is not affected. Furthermore, an appropriate evaluation value of the basic unit of the motion is used to represent the attribute value representing the characteristic of the basic unit of the motion by the continuous amount obtained from the actual motion data and to evaluate the basic unit of the recognized motion based on the attribute value. Can be obtained, and the sign language recognition accuracy can be improved.

[0005]

Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 2 is a diagram showing a sign language operation model based on an operation element in the present invention. In order to explain the present invention, a sign language operation model will be described first. The sign language operation is configured by a combination of basic units of the operation.
The basic unit of operation in this sign language will be hereinafter referred to as an operation element. Since there is temporal sequentiality and synchronism between the operation elements constituting the operation of the sign language, in order to recognize the sign language, it is necessary to also describe a temporal relationship between the operation elements. For this purpose, the model shown in FIG. 2 is used. In FIG. 2, reference numeral 201 denotes the entire operation of the sign language morpheme. A sign language morpheme is a unit of meaning in sign language. The sign language morpheme is a sequential element 2 which is a sequential unit.
02, 203, and 204. Since the horizontal axis represents the passage of time based on the left end, the sequential element 20
The operations have occurred in the order of 2, 203, and 204. A sequential element is a unit of operation that is always expressed continuously and is not simultaneously expressed. The sequential element is a simultaneous element 20 which is a plurality of units expressed in the same time range.
5,206,207. The simultaneous elements include operation elements 208 and 209. There are two types of simultaneous elements, a simultaneous element composed of only one operation element and a simultaneous element composed of only two or more operation elements.
A simultaneous element composed of two or more operation elements is considered to exist when the operation elements included therein are sequentially expressed. However, it is assumed that the operation elements included in one simultaneous element are always the same type of operation element. The operation element is an operation element such as the same direction but a different hand shape. The sign language operation is thus constituted by a combination of the sequential structure and the simultaneous structure of the operation elements.

FIG. 1 is a conceptual block diagram of a sign language recognition apparatus showing one embodiment of the present invention. In FIG. 1, a sign language input unit 101 (data glove) converts an operation in sign language into an electric signal, and inputs the electric signal to the dynamic operation element recognition unit 102 and the static operation element recognition unit 103 as time-series data. The dynamic operation element recognition unit 102 recognizes a dynamic operation element from the operation data. The static operation element recognizing unit 103 obtains an evaluation value of a static operation element for the data at each time of the operation data. The dynamic sequential element recognition unit 104 recognizes a sequential element composed of the recognized dynamic operation elements. The static sequential element recognition unit 105 recognizes a sequential element composed of only static operation elements. The static operation element integration unit 106 recognizes the sign language morpheme by integrating the recognition result of the static operation element into the sequential element composed of the dynamic operation elements. The present invention is characterized in that the static operation element integration unit 106 is provided. As described earlier in the description of the model, the recognition range of the static operation element is detected in a range smaller than the time range to be recognized or recognized in a larger range because the parameter of the basic unit is easily changed, Usually, since the static operation element appears together with the dynamic operation element, the recognition result of the sequential element of the dynamic operation element and the recognition result of the static operation element are integrated and evaluated.
That is, since the time of the dynamic operation element is clearly determined by the linear motion or the circular motion, the static operation element may be evaluated within the recognized time range. In the relationship between the model of FIG. 2 and FIG. 1, it may be considered that the dynamic elements and the static elements are included in the operation elements 208 and 209 of FIG. The recognition result of the sign language morpheme is output to the monitor 109 and the speaker 110 by the output unit 108. The sign language morpheme dictionary 111 stores a sign language template described by a combination of motion elements for each sign language morpheme that is a unit of meaning in sign language.

FIG. 3 is a block diagram showing the structure of the dynamic operation element recognition unit, and FIG. 4 is a block diagram showing the structure of the static operation element recognition unit. Dynamic operation element recognition unit 102
As shown in FIG. 3, is constituted by independent recognition units 301, 302, and 303 for each operation element.
Each motion element recognition unit is provided with recognition parameters 304, 305, and 306 necessary for the respective recognition processes.
The dynamic motion element recognition unit shown in FIG. 3 recognizes all conceivable motion elements, but may recognize only the dynamic motion elements in the sign language morphological dictionary. As shown in FIG. 4, the static operation element recognition unit 103 also includes recognition units 401, 402, and 403 for each operation element. The static operation element recognition unit 103 uses data stored in the sign language morphological dictionary for all parameters required for recognition.

FIG. 5 is a diagram showing an example of a configuration of hardware for realizing the sign language recognition device in FIG. In FIG. 5, a sign language input device 501 is a device that converts a hand motion in a sign language into an electric signal, and is a device that is well known as a device that installs a sensor on a glove and converts the shape and movement of a hand into an electric signal ( Data gloves). The sign language input device 501 converts the hand movement of the sign language into multidimensional time-series data including the bending angle of the finger, the position of the hand, and the like. The arithmetic device 502 is a device for recognizing a motion element or a sign language morpheme.
Programs are read from 4, 506, 507, 508, 509, and 511, and recognition processing is performed according to those programs. The output device 503 is a device that outputs a recognition result of a sign language morpheme, and can use a character output or a voice output device using voice synthesis. The memory 504 is a storage device for storing a program for recognizing a dynamic operation element, the memory 505 is a storage device for storing parameters necessary for recognizing a dynamic operation element, and the memory 506 is a dynamic device. A memory for storing a program for recognizing a sequential element composed of various operation elements, a memory 507 is a storage for storing a program required for recognizing a static operation element, and a memory 508 is a memory for storing a program for recognizing a static operation element. Storage device 509 for storing a program for integrating a sequential element composed of static operation elements and a recognition result of static operation elements, and 509 recognizes a sequential element composed only of static operation elements And a memory 510 for storing a sign language morpheme dictionary which is operation data of a sign language morpheme. 1 is a storage device for storing a program for recognizing sign language morphemes. In the hardware configuration of FIG. 5, the execution of all the recognition programs is performed by only one arithmetic device. In addition, the execution of the recognition program is distributed to the respective arithmetic devices by using a plurality of arithmetic devices. Configurations are also possible.

FIG. 6 is a format diagram of operation data input by the sign language input unit of FIG. In FIG.
Reference numeral 601 denotes data relating to the position of the hand. The position of the hand further includes data 602 on the x-axis, data 603 on the y-axis, and data 604 on the z-axis. Reference numeral 605 denotes data on the direction of the hand. The direction of the hand further includes an angle 606 about the x axis, an angle 607 about the y axis, and an angle 608 about the z axis.
It is composed of Reference numeral 609 denotes data relating to the bending of the finger. The bending of the finger further includes a bending angle 610 of the second joint of the thumb, a bending angle 611 of the third joint of the thumb, a bending angle 612 of the first joint of the index finger, and a second angle of the second finger of the index finger. Bending angle of joint 613, bending angle of first joint of middle finger 614, bending angle of second joint of middle finger 615, bending angle of first joint of ring finger 616, bending angle of second joint of ring finger 617, first finger of little finger
Bending angle 618 of the joint, bending angle 61 of the second joint of the little finger
9. 620, 621, 622
Are the hand positions at times t1, t2, and tn, respectively.
Represents data on direction and finger bending. As described above, the operation in the sign language is represented as time-series data including the hand position 601, the hand direction 605, and the finger bending 609. FIG.
FIG. 8 is a format diagram of parameters stored in a memory (505) for storing dynamic operation element recognition parameters in FIG. In FIG. 7, an operation element name 701 indicates the name of an operation element used for recognition processing of the parameter, a parameter number 702 indicates the number of parameters used for recognition of the operation element, and 703 and 704 indicate each parameter. The parameter types 705 and 707 represent names representing the meanings of the parameters, and the parameters 706 and 708 represent parameter values actually used in the recognition processing.

FIG. 8 is a format diagram stored in the sign language morphological dictionary memory of FIG. In FIG. 8, a sign language morpheme name 801 represents a name of a sign language morpheme represented by a combination of operation elements described below. Repeat count 8
02 represents the number of times the operation described below is repeated. The number of sequential elements 803 indicates the number of sequential elements constituting the sign language operation. The degree of overlap between sequential elements 804 indicates an allowable range for an overlap or a gap that occurs when each sequential element is expressed during an actual sign language operation. That is, when the elements are actually recognized, the elements may overlap with each other or there may be a gap between the elements. Therefore, the degree is registered. In this case, the value is + when separated and-when overlapped. The degree of overlap between sequential elements is effective when the number of sequential elements is two or more. Sequential elements 805, 806, and 807 represent descriptions of the respective sequential elements. The number of simultaneous elements 808 is
Represents the number of simultaneous elements that make up a sequential element. The overlapping degree 809 between the simultaneous elements represents an allowable range for the overlapping that occurs when each of the simultaneous elements constituting the sequential elements is expressed in an actual sign language operation. The overlapping degree between simultaneous elements is effective when the number of simultaneous elements is two or more. 81 repetitions
0,815 represents the number of times the sequence of operation elements described below is repeated. Number of operating element states 811,816
Represents the number of operation elements constituting each simultaneous element. The degree of overlap between motion elements 812 and 817 indicates an allowable value for the overlap or gap that occurs when the sequentially expressed motion elements are expressed in an actual sign language operation. The degree of overlap between operation elements is effective when the number of operation element states is two or more. Operating elements 813, 814, 818, 8
Reference numeral 19 denotes an operation element constituting each simultaneous element.

FIG. 9 is a diagram showing a description format of an operation element. FIG. 10 is a view showing the types of operation elements and the types of respective attribute values. FIG. 11 is a format diagram of attribute values of the operation elements. is there. In FIG. 9, reference numeral 901 denotes a type of an operation element; 902, a part of a hand used to represent the operation element; and 903, 904, 905, attribute values necessary to represent the operation element. The type of the operation element is selected from 14 types of operation elements as shown in FIG. Further, as shown in FIG. 10, the type of the attribute value is determined in advance according to the type of the operation element. In the attribute value format of FIG.
Reference numeral 2 1103 denotes an average value of attribute values learned from a plurality of pieces of motion data, and reference numerals 1104, 1105 and 1106 denote variances of attribute values learned from a plurality of pieces of motion data. Here, the average value of the attribute values refers to the average value p1
pn, and the variance of the attribute value is the variation with respect to the average value, and it is calculated from the same data how many times the parameter varies and expressed as s1 to sn It is. The dimension of the attribute value is determined in advance according to the type of the attribute value shown in FIG.

Next, the recognition processing according to the present invention will be described. FIG. 12 is a flowchart illustrating a process of recognizing a static operation element included in a simultaneous element having two or more operation element states. The dynamic motion element recognition unit 102 in FIG. 1 performs dynamic motion elements such as vibrations and straight lines, and static motions such as shapes and methods included in simultaneous elements having two or more motion element states in the sign language morphological dictionary. Two types of recognition of elements (recognition of dynamic operation elements and static operation elements) are performed. An existing technique (for example, see Japanese Patent Application No. 6-253457 and the drawing "Sign Language Recognition Apparatus") can be used as a technique for dynamic motion element recognition. Recognition of a static operation element included in a simultaneous element having two or more operation element states can be performed according to the flowchart shown in FIG. In FIG. 12, at step 1201, first, a static operation element included in a simultaneous element having two or more operation element states is extracted from the sign language morphological dictionary, and a list thereof is created. The format of the operation element in the list may be the same as the format of the operation element shown in FIG. Next, in step 1202, data for one time is read from the sign language input unit. In step 1203, if the operation data is the last, the process ends. If it is not the last, the process proceeds to step 1204. In step 1204, the evaluation value at that time is obtained from the attribute values of the operation elements and the read data for all the operation elements in the static operation element list.

The evaluation value is such that the type of the attribute value of the static operation element is n, the number of dimensions of the i-th attribute value is m (i), the i-th attribute described in the sign language morphological dictionary of the static operation element. The average of the values is (P (i, 1), P (i, 2),..., P (i, m
(I))) and the variances are (S (i, 1), S (i, 2),
, S (i, m (i)), the input time is t, and the i-th attribute value of the input data is (X (t, i, 1), X
(T, i, 2),..., X (t, i, m (i))
The evaluation value E1 (t) at each time is obtained by the following equation (Equation 1).

(Equation 1) Next, in step 1205, a time range in which the time series of the evaluation value including the evaluation value obtained so far and the evaluation value newly obtained is maximized is obtained for each operation element. When the time series of the evaluation values is obtained, for example, as a curve 1301 shown in FIG.
The range of 03 is detected as an operation element. Step 12
At 06, the obtained time range, the corresponding operation element, and the evaluation value are output as a recognition result. Step 120
In step 7, the obtained evaluation value for each operation element is stored in a buffer for use in recognition at the next time.
Return to 202. FIG. 14 is a format diagram of the operation element detected in the flow of FIG. 1401 is the start time of the time range in which the operation element is detected, 1402 is the end time of the time range in which the operation element is detected, 1403 is the evaluation value for the detected operation element, 1404 is the type of the detected operation element, 1405 Is a part of the hand used to represent the action element, and 1406 and 1407 are attribute values attached to each action element. The attribute value is a value obtained from operation data in a range where the operation element is detected.

Next, a dynamic sequential element recognition unit 10 shown in FIG.
4, a method of recognizing sequential elements including dynamic operation elements will be described. FIG. 15 is a flowchart of the dynamic sequential element recognition process. In this processing, roughly divided two-step processing of simultaneous element recognition and sequential element recognition is performed. In step 1501, a simultaneous element constituted by a dynamic operation element and a simultaneous element constituted by a static operation element having two or more operation element states are extracted from the sign language morphological dictionary, and a simultaneous element list is created. FIG. 16 shows the format of the simultaneous element in this case. In FIG. 16, reference numeral 1601 denotes the serial number of the simultaneous element in the sign language morphological dictionary, 1602 denotes the degree of overlap between the operation elements, 1603 denotes the number of operation element states, and 1604 and 1605 denote each operation element included in the simultaneous element. The format of the operation element is the same as the format shown in FIG. 9, and the attribute values in the format of the operation element are the same as the format of FIG.
In step 1502, a simultaneous element constituted by a dynamic operation element and a sequential element including a simultaneous element including a static operation element having two or more operation element states are extracted from the sign language morphological dictionary. And a correspondence list of the simultaneous elements including the static operation elements having the number of operation elements of 2 or more and the sequential elements is created. FIG. 17 shows the format of the sequential elements in the correspondence list of the simultaneous elements and the sequential elements. In FIG. 17, reference numeral 1701 denotes the serial number of a sequential element in the sign language morphological dictionary, 1702 denotes the degree of overlap between the simultaneous elements, and 1703 denotes the number of simultaneous elements including dynamic operation elements included in the sequential element and the number of operation element states of 2 or more. The number of simultaneous elements, 1704 and 1705, are serial numbers in the sign language morphological dictionary of the simultaneous elements including the dynamic operation elements and the simultaneous elements including the static operation elements having two or more operation element states.

In step 1503, one recognition result from the dynamic motion element recognition unit 102 is read. Next, in step 1504, if the recognition result is the last, the process ends. Otherwise, go to step 1505. In step 1505, a simultaneous element constituted by the read operation elements is recognized. This process is performed according to the flowchart shown in FIG. In FIG. 18, in step 1801, a simultaneous element including the operation element read in step 1503 is searched from the simultaneous element list.
In step 1802, the number of found simultaneous elements is substituted for the counter i. In step 1803, if the number of operating element states of the i-th simultaneous element among the searched simultaneous elements is 1, the flow proceeds to step 1804; otherwise, the flow proceeds to step 1805. In step 1804, an evaluation value is obtained from the attribute value of the operation element in the i-th simultaneous element and the attribute value of the read operation element. Evaluation value E in this case
2, the attribute value type of the i-th simultaneous element is n, the dimension number of the j-th attribute value of the i-th simultaneous element is m (j), and the j-th attribute value of the i-th simultaneous element is (P (J, 1), P
(J, 2),..., P (j, m (j))) and the variance to (S
(J, 1), S (j, 2), ..., S (j, m
(J))), and the j-th attribute value of the read operation element is represented by (X (j, 1), X (j, 2),..., X (j, m
(J))) can be obtained by the following equation (Equation 2).

(Equation 2)

Next, in step 1805, a search is made from the read operation elements and the operation elements in the buffer for the same operation element sequence as the operation element sequence constituting the i-th simultaneous element. In step 1806, an evaluation value of the simultaneous element is obtained from the attribute value of the operation element in the searched operation element sequence and the attribute value of the operation element in the i-th simultaneous element. The evaluation value E3 in this case is i
The number of operation elements of the i-th simultaneous element is n, the type of attribute value of the j-th operation element of the i-th simultaneous element is m (j), and the k-th attribute of the j-th operation element of the i-th simultaneous element The dimension number of the value is q (j, k), and the k-th attribute value of the j-th operation element of the i-th simultaneous element is (P (j, k, 1), P (j,
, P (j, k, q (j, k))) and the variance to (S (j, k, 1), S (j, k, 2), ..., S (j,
k, q (j, k))), the read-out operation element corresponding to the j-th operation element of the i-th simultaneous element or the k-th attribute value of the operation element in the buffer is (X (j, k,
1), X (j, k, 2),..., X (j, k, q (j,
k))), the overlap or gap between the j-th and j + 1th motion elements is G (j) (positive for overlap, negative for gap), the overlap or gap between the motion elements is A, and the variance is σ can be obtained by the following equation (Equation 3).

(Equation 3)

In step 1807, the obtained evaluation value and i
The second simultaneous element is stored in the buffer of the recognized simultaneous element as a result of the recognition of the simultaneous element. FIG. 19 shows the format of the simultaneous element stored in the buffer. 19, reference numeral 1901 denotes the start time of the time range of the recognized simultaneous element, 1902 denotes the end time of the time range of the recognized simultaneous element, 1903 denotes the serial number of the simultaneous element in the sign language morphological dictionary, and 1904 denotes the evaluation of the simultaneous element. Value.
The start time 1901 and the end time 1902 are calculated based on the time range of the operation element from which the simultaneous element is determined to be recognized. Returning to the original, in step 1506 of FIG. 15, the sequential elements constituted by the simultaneous elements recognized in step 1505 are recognized. FIG.
0 is a flowchart of the sequential element recognition process shown in FIG. In FIG. 20, first, in step 2001, a sequential element including a newly recognized simultaneous element in the buffer of the recognized simultaneous element is searched from the correspondence list of the simultaneous element and the sequential element. In step 2002, the number of searched sequential elements is substituted for a counter i. In step 2003, the simultaneous element forming the i-th sequential element of the searched sequential element is searched from the buffer of the simultaneous element. If it is determined in step 2004 that all the simultaneous elements constituting the i-th sequential element have been searched, the process proceeds to step 2005. If all necessary simultaneous elements have not been found, the process proceeds to step 2007.

In step 2005, the evaluation value of the i-th sequential element is obtained based on the searched evaluation value of the simultaneous element. The evaluation value is n, the number of simultaneous elements forming the i-th sequential element, the evaluation value of the j-th simultaneous element is E4 (j), the overlap between the simultaneous elements is O, and the average of the overlap between the simultaneous elements is Is A and the variance is σ. In addition, E4 in the equation can be easily obtained from the equation (Equation 1).

(Equation 4) Further, the degree of overlap O between the simultaneous elements in the equation can be obtained by the following equation (5), where the start time of the j-th simultaneous element is s (j) and the end time is e (j).

(Equation 5) In step 2006, the obtained evaluation value and its sequential element are output as a recognition result. FIG. 21 shows the format of the output sequential element. In FIG. 21, 210
1 is the start time of the time range of the sequential element, 2102 is the end time of the time range of the sequential element, 2103 is the serial number of the sequential element in the sign language morphological dictionary, and 2104 is the evaluation value of the sequential element. The time range of the sequential element is obtained based on the time range of the simultaneous element used for recognition of the sequential element. For example, the time range of the overlap of all simultaneous elements can be used. Alternatively, the average of the start time and the end time of the time range of the simultaneous element may be used.

FIG. 22 is a flowchart showing the processing of the static operation element recognition unit shown in FIG. In the recognition processing of the static operation element recognition unit 103 in FIG. 1, an evaluation value of a static operation element is obtained for each time. Step 220 in FIG.
In step 1, a static operation element is extracted from a sign language morphological dictionary, and a list of static operation elements is created. In this case, a static operation element that constitutes a simultaneous element having two or more operation element states is deleted. This is because the process has been completed. FIG. 23 shows the format of the operation element in the operation element list. 23, reference numeral 2301 denotes a serial number of an operation element in the sign language morphological dictionary, 2302 denotes a type of the operation element, 2303 denotes a part of a hand used to express the operation element, and 2304 and 2305 denote the operation element. This is the attribute value. In step 2202 of FIG. 22, data for one time is read from the sign language input unit.
Next, in step 2203, if the data is the last, the process is terminated. Otherwise, step 220
Move to 4. In step 2204, an evaluation value at that time is obtained from the attribute values of the operation elements and the read data for all the operation elements in the static operation element list. The evaluation value can be obtained by the equation (Equation 1). In step 2205, the obtained evaluation value and operation element are output as a recognition result. In addition, the static operation element integration unit 1
To use the past recognition result of the static operation element in 06,
A buffer of the static operation element is provided, and the recognition result is stored therein. FIG. 24 shows the format of the output operation element. In FIG. 24, 2401 is a time, 2402 is a serial number of an operation element in the sign language morphological dictionary, 2403
Is the evaluation value of the motion element.

FIG. 25 is a flowchart of a recognition process in the static sequential element recognition unit shown in FIG. In the recognition processing of the static sequential element recognition unit 105 in FIG. 1, the sequential element composed of only the static operation elements is recognized based on the evaluation values of the static operation elements recognized by the static operation element recognition unit 103. . In step 2501 of FIG. 25, a sequential element composed of only static operation elements is extracted from the sign language morphological dictionary, and a correspondence list of the sequential elements and the static operation elements is created. The format of the sequential elements in the correspondence list is shown in FIG.
6 is shown. In FIG. 26, 2601 is the serial number of the sequential element in the sign language morphological dictionary, 2602 is the number of static operation elements constituting the sequential element, 2603 and 2604 are the serial numbers of the static operation element forming the sequential element in the sign language morphological dictionary It is. In step 2502 of FIG. 25, the static operation element recognition result is read from the static operation element recognition unit 103 for one time. Next, in step 2503, if the recognition result is the last, the process ends. Otherwise, the process moves to step 2504. In step 2504, for each sequential element, a required operation element is selected from the recognition result of the read static operation element, and an evaluation value of the sequential element at that time is obtained. The evaluation value E7 (t) at time t is
The number of the static operation elements constituting the sequential element can be obtained by the following expression (Equation 6), where n is the evaluation value of the i-th static operation element at time t and E6 (t, i).
Note that E6 (t, i) can be easily obtained by the equation (Equation 1).

(Equation 6) In step 2505, for each sequential element, a time range in which the evaluation value is maximized is searched for from the contents of the buffer storing the history of the obtained evaluation value of the sequential element and the history of the evaluation values of the past sequential elements. The range and the evaluation value are output as the sequential element recognition result. The format of the sequential element recognition result is the same as the format shown in FIG.
In step 2506, the obtained evaluation value of the sequential element is stored in the buffer for use in the processing at the next time.

FIG. 27 is a flowchart of the integration process in the static operation element integration section shown in FIG. The static operation element integration unit 106 integrates the evaluation value of the static operation element with the evaluation result of the sequential element composed only of the dynamic operation element, and obtains the evaluation value of the entire sequential element. In FIG. 27, first, in step 2701, a sequential element including a dynamic operation element and a sequential element including a simultaneous element including a static operation element having two or more operation element states are extracted from the sign language morphological dictionary. Create a correspondence list of sequential elements and static operation elements. The format of each sequential element in the correspondence list is the same as the format shown in FIG. In step 2702, one dynamic sequential element recognition result is read from the dynamic sequential element recognition unit 104. Step 270
In 3, if the result of the recognition of the dynamic sequential element is the last, the process ends. Otherwise, it proceeds to step 2704. In step 2704, the recognition result (evaluation value) of the static operation element corresponding to the dynamic sequential element is read from the static operation element recognition unit 103 in the time range of the read dynamic sequential element. In step 2705, the average of the read evaluation values of the static operation elements is obtained as the evaluation value of the static operation element corresponding to the dynamic sequential element. In step 2706,
From the evaluation value of the dynamic sequential element and the evaluation value of the static operation element,
Find the evaluation value of the entire sequential element. The calculation of this evaluation value is
It can be obtained by calculating an average or a geometric mean of two types of evaluation values. In addition, the evaluation value can be stored for each type of the operation element constituting the sequential element, and the average or geometric mean thereof can be calculated. In step 2707, the time range of the dynamic sequential element and the obtained evaluation value are output as an integrated result. The format of the integration result is the same as the format shown in FIG. FIG.
In the flowchart shown in FIG. 7, the evaluation value of the static operation element is obtained for all times, and only the evaluation value in the necessary time range is received from the evaluation values. However, the evaluation value of the static operation element is obtained from the recognition result of the dynamic sequential element recognition unit. Only for a given time range, an instruction is sent to the static operation element recognizing unit 103 so as to obtain a required evaluation value of the static operation element, and the evaluation result is received to perform the integration processing of the static operation element. Can also.

FIG. 28 is a flowchart of a recognition process in the sign language morpheme recognition unit shown in FIG. The sign language morpheme recognition unit 107 recognizes a sign language morpheme from the recognition result of the sequential elements. In FIG. 28, first, in step 2801,
From the sign language morpheme dictionary, a correspondence list of sign language morphemes and their sequential elements is created. FIG. 29 shows the format of each sign language morpheme in the correspondence list. In FIG. 29, 2
Reference numeral 901 denotes a sign language morpheme name, 2902 denotes the degree of overlap between sequential elements constituting the sign language morpheme, 2903 denotes the number of sequential elements constituting the sign language morpheme, and 2904 and 2905 denote serial numbers of the sequential elements constituting the sign language morpheme in the sign language morpheme dictionary. It is. In step 2802, one recognition result of the sequential element is read. In step 2803, if the recognition result of the sequential element is the last, the process ends. Otherwise, it proceeds to step 2804. In step 2804, a sign language morpheme corresponding to the sequential element sequence configured is searched from the correspondence list from the sequential elements in the buffer storing the history of the read sequential elements and past sequential elements.
In step 2805, for the retrieved sign language morpheme, an evaluation value is determined based on the evaluation value of the sequential elements and the overlap or gap between the sequential elements. The evaluation value E9 is
The number of sequential elements constituting the sign language morpheme is n, the evaluation value of the i-th sequential element is E8 (i), and the overlap or gap between the i-th and i + 1-th sequential elements is G (i) (positive, When the gap is negative), the overlap between the sequential elements or the average of the gap is A, and the variance is σ, it can be obtained by the following equation (7). E8
(I) is easily obtained by the above equation (Equation 1).

(Equation 7)

In step 2806, the obtained evaluation value, sign language morpheme name, and time range are output as recognition results. The time range of the sign language morpheme is obtained based on the time range of the sequential element used for recognition. For example, the time range from the start time of the first sequential element to the end time of the last sequential element can be a time range of the sign language morpheme. FIG. 30 shows a format of the recognition result of the sign language morpheme. In FIG. 30, reference numeral 3001 denotes the start time of the time range of the sign language morpheme;
02 is the end time of the time range of the sign language morpheme, 3003 is the sign language morpheme name, and 3004 is the evaluation value of the sign language morpheme. Note that the previous equations (Equation 1), (Equation 2), (Equation 3), (Equation 4),
In (Equation 6) and (Equation 7), each of the evaluation values is calculated using the geometric mean of the evaluation values of the constituent elements. However, a simple average (all of the evaluation values of the constituent elements are added and divided by the number) Calculate)
May be used. Also, the evaluation value of the component and the evaluation value for the gap or overlap may be weighted.

[0024]

As described above, according to the present invention,
Since the static motion element is evaluated based on the time range determined from the recognition result of the dynamic motion element, the recognition accuracy does not decrease due to the recognition range of the static motion element. Also, for sequential elements composed only of static operation elements, the evaluation value of the entire sequential element is obtained at each time, and recognition is performed by detecting the time when the evaluation value has the maximum value. Recognition degradation due to a shift in element detection time is eliminated. In addition, attribute values representing the characteristics of the operation element
Since it is expressed by the continuous amount obtained from the actual motion data and the motion element recognized based on the continuous amount is evaluated, it is possible to obtain an appropriate evaluation value of the motion element, thereby improving sign language recognition accuracy. .

[Brief description of the drawings]

FIG. 1 is a conceptual block diagram of a sign language recognition apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram showing a sign language operation model based on an operation element in the present invention.

FIG. 3 is a block diagram illustrating a structure of a dynamic operation element recognition unit illustrated in FIG. 1;

FIG. 4 is a block diagram illustrating a structure of a static operation element recognition unit illustrated in FIG. 1;

FIG. 5 is a hardware configuration diagram of a sign language recognition device for realizing one embodiment of the present invention.

FIG. 6 is a format diagram of data input from the sign language input device of FIG. 1;

FIG. 7 is a format diagram of parameters for dynamic operation element recognition.

FIG. 8 is a description format of a sign language morphological dictionary.

FIG. 9 is a description format of an operation element.

FIG. 10 is a diagram showing types of operation elements and types of respective attribute values.

FIG. 11 shows a format of an attribute value of an operation element.

FIG. 12 is a flowchart illustrating a process of recognizing a static operation element included in a simultaneous element having two or more operation element states.

FIG. 13 is a diagram illustrating a process of detecting a static operation element.

FIG. 14 is a format diagram of the dynamic operation element recognized in FIG. 13;

FIG. 15 is a flowchart illustrating a recognition process of a dynamic sequential element recognition unit illustrated in FIG. 1;

FIG. 16 is a format diagram of a simultaneous element in a correspondence list of a simultaneous element and an operation element.

FIG. 17 is a format diagram of sequential elements in a correspondence list of sequential elements and simultaneous elements.

FIG. 18 is a flowchart for explaining a simultaneous element recognition process in the dynamic sequential element recognition unit shown in FIG. 1;

FIG. 19 is a format diagram of recognized simultaneous elements.

20 is a flowchart for explaining a sequential element recognition process in the dynamic sequential element recognition unit shown in FIG. 1;

FIG. 21 is a format diagram of a sequential element recognized in FIG. 20;

FIG. 22 is a flowchart illustrating a recognition process of a static operation element recognition unit illustrated in FIG. 1;

FIG. 23 is a format diagram of an operation element in a static operation element list.

FIG. 24 is a format diagram of a static operation element recognized in FIG. 22;

FIG. 25 is a flowchart illustrating a recognition process of a static sequential element recognition unit illustrated in FIG. 1;

FIG. 26 is a format diagram of a sequential element in a static sequential element list.

FIG. 27 is a flowchart illustrating an integration process of a static operation element integration unit illustrated in FIG. 1;

FIG. 28 is a flowchart illustrating a recognition process performed by the sign language morpheme recognition unit illustrated in FIG. 1;

FIG. 29 is a format diagram of a sign language morpheme in a correspondence list of sign language morphemes and sequential elements.

FIG. 30 is a format diagram of the sign language morpheme recognized in FIG. 28;

[Explanation of symbols]

101: Sign language input unit, 102: Dynamic motion element recognition unit, 1
08 output unit 103 static operation element recognition unit 104
Dynamic sequential element recognition unit, 105 ... static sequential element recognition unit, 1
06: dynamic motion element integration unit, 107: sign language morpheme recognition unit, 109: monitor, 110: speaker, 111: sign language morpheme dictionary, 201: sign language morpheme, 208, 209: motion elements, 202, 203, 204: sequential elements , 205,
206, 207 ... simultaneous elements, 301, 302, 303 ...
Dynamic operation element recognition unit, 304 to 306... Parameters, 40
1,402,403 ... Static operation element recognition unit, 504-5
11 ... memory, 601 ... hand position, 605 ... hand direction,
609: bending of finger, 701: name of operation element, 705, 70
7 ... parameter type, 706, 708 ... parameter, 9
01: Kind of operation element, 902: Hand part, 903-9
05: attribute values, 1101 to 1103: average of attribute values,
1104 to 1106: dispersion of attribute values; 1301, time series of evaluation values; 1302: start time of a time range; 1303 ...
End time of the time range.

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Masaru Takeuchi 1-280 Higashi Koigakubo, Kokubunji City, Tokyo Inside the Central Research Laboratory of Hitachi, Ltd. Central Research Laboratory

Claims (11)

[Claims] 1. Converting the shape and movement of a hand into an electric signal,
Sign language input means for inputting as time-series sign language data; dynamic action element recognition means for recognizing a basic unit of dynamic action among the basic units of action input from the sign language input means; input from the sign language input means A static operation element recognizing means for recognizing a basic unit of a static operation among the basic units of operation performed, and one or more basic units of a dynamic operation taken from the dynamic operation element recognizing means. Dynamic sequential element recognizing means for recognizing a group of sequential operations to be performed, and a sequential operation composed of one or more basic units of static operations fetched from the static operation element recognizing means. A static sequential element recognizing means for recognizing a unit, and a sequential operation unit which is fetched from the dynamic sequential element recognizing means and is constituted by a basic unit of a dynamic operation; A static operation element integration means for integrating the basic unit of the static operation taken from the stage, and a recognition result of the sequential element taken from the static operation element integration means and the static sequential element recognition means. A sign language morpheme recognition means for recognizing a motion as a sign language, a sign language template expressed by a combination of basic units of motion are stored, and the sign language morpheme dictionary means and the sign language morpheme recognition means are referred to by the respective means. Means for outputting the sign language in the form of voice or characters. 2. The sign language recognition device according to claim 1, wherein the basic unit of the operation stored in the sign language morphological dictionary means for storing the sign language template is a symbol representing a type of the operation and an operation represented by a continuous amount. A sign language recognition device characterized by being represented by a combination of attribute values. 3. The sign language recognition device according to claim 2, wherein the attribute value of the basic unit of the action comprises its average and variance. 4. The sign language recognition device according to claim 1, wherein the dynamic action element recognizing means for recognizing a basic unit of the dynamic action is a dynamic action element extracted from a sign language morphological dictionary means storing a sign language template. A sign language recognition device for recognizing a basic unit of motion. 5. The sign language recognition device according to claim 1, wherein the static action element recognizing means for recognizing a basic unit of the static action is a static action element extracted from a sign language morphological dictionary means storing a sign language template. A sign language recognition device for recognizing a basic unit of motion. 6. The sign language recognition device according to claim 1, wherein a unit of a sequential operation constituted by the basic unit of the dynamic operation and a basic unit of the static operation are integrated. A sign language recognition device, wherein the integrating means selects only a recognition result of the basic unit of the static operation in a time range determined by a unit of the sequential operation constituted by the basic unit of the dynamic operation. . 7. The sign language recognition device according to claim 6, wherein a unit of a sequential operation constituted by the basic unit of the dynamic operation and a basic unit of the static operation are integrated. The integrating means first sends a time range determined by a sequential operation unit constituted by the basic unit of the dynamic operation to the static operation element recognizing unit for recognizing the basic unit of the static operation. The static operation element recognition means performs recognition processing only for the time range sent, and integrates the result into a unit of sequential operation composed of basic units of dynamic operation and basic units of static operation A sign language recognition device for sending to a static operation element integrating means. 8. The sign language recognition device according to claim 1, wherein the dynamic sequential element recognizing means for recognizing a group of sequential motions constituted by the basic units of the dynamic motions comprises: A sign language recognition device that performs recognition based on the degree of overlap of a time range of a basic unit of a simple operation. 9. The sign language recognition device according to claim 1, wherein the static action element recognizing means for recognizing a basic unit of the static action is configured to convert data at each time of the time-series sign language data into a sign language template. A sign language recognition apparatus characterized in that evaluation is performed based on an attribute value of a basic unit of a static action stored in a sign language morphological dictionary means to be stored. 10. The sign language recognition device according to claim 1, wherein the static sequential element recognizing means for recognizing a sequential operation unit constituted by a basic unit of the static operation comprises a static sequential element recognition unit at each time. By summarizing the evaluation results of the basic units of various operations for each unit of sequential operation, the evaluation value for the unit of sequential operation is obtained, and the time range in which the obtained evaluation value is the maximum value is set for the sequential operation. A sign language recognition apparatus characterized in that the recognition result is a unit. 11. The sign language recognition device according to claim 1, wherein the sign language morpheme recognition means for recognizing the operation as the sign language obtains a sequence of sequential operation units corresponding to the operation of the sign language. A sign language recognition device characterized by performing: