JP2000284682A - Sign language recognition method and apparatus - Google Patents

Abstract

(57) [Summary] [Problem] To detect a boundary of each sign language word from sign language sentence data and generate a correct sign language word string from a result of recognition of the sign language word. A time when the operation speeds at the palms of both hands and the left and right hands and at the tip of each finger are minimized is defined as an operation boundary.
If the change in the movement direction of the left and right palms and the tip of each finger exceeds the threshold, the time is detected as the boundary of the movement. The boundary detected by the change of the operation direction is associated with the temporally closest boundary among the boundaries detected by the minimum value of the operation speed. Recognized sign language words are associated with each section in the sign language sentence data divided by the detected boundaries based on the feature of the operation of the sign language words. In each section, the feature of the motion is extracted from the sign language sentence data, and the priority of the sign language word associated with the section is changed. Finally, a sign language word sequence is generated based on the correspondence between the sign language word and each section and the priority of the sign language word in each section.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sign language recognizing device for detecting individual sign language words from a sign language sentence in which sign language words are continuously expressed, and a medium recording a program for operating a computer as a sign language recognizing device.

[0002]

2. Description of the Related Art Japanese Patent Laid-Open Publication No. Hei 6-3 discloses a technique for detecting a boundary of a motion from a motion of a sign language sentence in which a plurality of sign language words are continuously expressed, and using the result for a sign language word string generation process.
No. 37629 (Sign Language Recognition Apparatus). In this technique, a time at which the hand movement speed becomes a minimum and a section where the hand movement speed becomes equal to or less than a predetermined value are detected as movement boundaries. Further, by judging the presence or absence of a change in the hand shape and the presence or absence of repetition, an extra boundary is deleted, and a section representing a sign language word is determined. Based on this result, the sign language word recognized over the boundary is deleted. Also,
Based on the position of the determined section and the expression position of the sign language word,
Restrict sign language words for recognition processing.

[0003]

In the prior art for detecting a sign language word boundary and generating a sign language word string based on the result, the motion boundary is detected only from the movement speed of the hand, and the hand shape is determined. It was only used to remove extra boundaries detected during the operation of sign language words. In addition, a change in the direction of the hand is not considered at all in detecting the boundary of the motion.
However, among the sign language words, there are sign language words in which only the shape of the hand changes while the position of the hand remains stationary, and sign language words in which only the direction of the hand changes. Also, there is a sign language word in which the shape of the hand and the direction of the hand change as the position of the hand changes. When trying to detect a boundary from sign language sentence data including such a sign language word by a conventional technique, a change in the shape or direction of a hand cannot be used as a condition for removing an extra boundary. The section representing the sign language word cannot be correctly detected.

In the prior art, a sign language word recognized over a detected boundary is deleted as an error. However, a sign language word is not always composed of only one operation, but may be composed of a plurality of operations. In the related art, a boundary is detected for such a sign language word, so that a correct sign language word is deleted.

Further, in the prior art, in each section in the sign language sentence data divided by the boundary, the priority is given to the sign language words recognized based only on the position. However, such a restriction is not sufficient because many sign language words whose positions fluctuate variously are found in the sign language words. SUMMARY OF THE INVENTION An object of the present invention is to provide a sign language recognition device including means for detecting a boundary of an operation representing each sign language word during operation of a sign language sentence expressing various forms of sign language words as described above. It is in.

Another object of the present invention is to provide a sign language recognition apparatus including means for generating a correct sign language word sequence based on a detected boundary and a recognition result of a sign language word. Still another object of the present invention is to provide a sign language recognition device including means for effectively prioritizing sign language words corresponding to each section in a sign language sentence divided by a detected boundary. It is in. Another object of the present invention is to provide a medium in which a program for causing a computer to operate as a sign language recognition device having the above characteristics is recorded.

[0007]

In order to achieve the above object, according to the present invention, at each time in the sign language sentence data, the operating speed at the tip of the palm and each finger is obtained, and the time at which the obtained operating speed becomes a minimum is obtained. Is detected as an operation boundary. Further, in each section in the sign language sentence data divided by the detected boundary, the time at which the operating speed of the palm and each finger is minimized for each of the left and right hands is detected, and the detected time is added to the boundary of the operation. . At that time, it is determined whether or not to add a boundary based on the change pattern of the operating speed of the left and right hands. In addition, the change of the movement direction of the tip of the left and right palms and each finger at each time in the sign language sentence data is calculated,
When the change in the operation direction exceeds a predetermined threshold, the time is detected as a boundary of the operation. The boundary detected by the change in the operation direction is associated with the temporally closest boundary among the boundaries detected by the minimum value of the operation speed.

Next, a recognized sign language word is associated with each section in the sign language sentence data divided by the detected boundary. At this time, the section to be associated is determined based on the feature of the operation of each sign language word. In each section,
The feature of the motion is extracted from the data, and the priority of the sign language word associated with the section is changed. Finally, a sign language word sequence is generated based on the correspondence between the sign language words and each section and the priority of the sign language words in each section.

That is, the sign language recognition device according to the present invention comprises:
Sign language action input means for converting the operation of the sign language sentence into an electric signal and inputting the data as time-series sign language sentence data; and means for recognizing each sign language word expressed in the sign language sentence data input from the sign language action input means. A sign language data dividing unit for detecting a boundary of the motion in the sign language sentence data input from the sign language motion input unit based on a feature amount of the motion; and a sign language sentence data divided by the boundary detected by the sign language data dividing unit. Operation characteristic detecting means for detecting the characteristic of the operation in each section, sign language word information storing means for storing information about the sign language word, and sign language data division based on the information about the sign language word stored in the sign language word information storing means. For associating each section in the sign language sentence data divided by the boundary detected by the means with the sign language word recognized by the sign language word recognition means. A section sign language word correspondence unit, and a sign language word string generation unit that generates a sign language word string based on the correspondence between each section in the sign language minute data and the sign language word associated by the divided section sign language word correspondence unit. It is characterized by.

The feature amount of the operation includes the speed and direction of the operation. The features of the motion include the type of hand used (both hands, right hand, left hand), the position where the motion is performed, and the like. The information about the sign language word includes the features and structure of the operation of the sign language word. The sign language data dividing means detects a boundary of the motion based on at least one characteristic amount among a motion speed of the palm, a change in the motion direction of the palm, a motion speed of the tip of each finger, and a change in the motion direction of the tip of each finger. Can be configured. Preferably, the boundary of the motion is detected by using a plurality of these feature amounts or by using all of these feature amounts. More specifically, the sign language data dividing means includes:
The time when the operating speed of the palm or the tip of each finger is minimized, the time when the operating speed of the palm or the tip of each finger is lower than a predetermined speed, the operating speed of the palm or the tip of each finger The boundary of the motion is defined as the time at which the motion speed of the hand changes from a state lower than the predetermined speed to a state higher than the predetermined speed, and a time at which the change of the motion direction of the tip of the palm or each finger exceeds a predetermined threshold. Can be detected as

The sign language data dividing means detects a boundary of the movement based on the movement speed of both the left and right palms and the movement speed of the tip of each finger, and then registers each of the sign language sentence data divided by the detected boundary. For the section, a motion boundary is detected based on the palm operation speed in the right hand and the operation speed of each finger tip, and the palm operation speed in the left hand and the operation speed of each finger tip, and the detected boundary is operated. May be configured to be added as a boundary. The sign language data dividing means further detects, as a boundary, a time at which a change in the movement direction of the tip of the palm or each finger exceeds a predetermined threshold, and detects the detected boundary as a time out of the detected boundaries based on the operation speed. It can be configured to correspond to the closest boundary.

The motion feature detecting means includes a difference or a ratio between the maximum motion speeds of the right and left hands in the section,
The difference or ratio of the average operating speed of the left and right hands in the section, the difference or ratio of the moving distance of the left and right hands in the section, the difference of the operating speed change of the left and right hands in the section, Based on at least one feature amount of the motion direction difference in, each of the sections is determined as to whether the action in the section uses both hands, only the right hand, or only the left hand. Can be classified.

The information stored in the sign language word information storage means is used for information relating to the characteristics of the operation of the sign language word, the number of sections in which the operation of the sign language word is divided by the boundaries of the operation, and the operation of the sign language word. At least one of the following types of hands. The information on the characteristics of the sign language word operation includes "upper limit" in which the sign language word operation may be divided into a predetermined number of sections, and "upper limit" in which the number of divided sections is unclear. The result may be classified into any of "none" and "presentation operation" including an operation of pressing a mark. The divided section sign language word corresponding means determines a section corresponding to the sign language word based on the feature of the operation of the sign language word and the number of sections divided by the boundaries of the operation. The divided section sign language word correspondence unit may be configured to change the priority of the sign language word associated with each section based on the feature of the operation in each section.

A computer-readable recording medium according to the present invention comprises: sign language operation input means for converting an operation of a sign language sentence into an electric signal and inputting the data as time-series sign language sentence data; and sign language sentence data input from the sign language operation input means. Means for recognizing each sign language word expressed therein, sign language data dividing means for detecting a boundary of the motion in the sign language sentence data input from the sign language motion input means based on a feature amount of the motion, and sign language data dividing Sign language data division based on the sign language word information stored in the sign language word information storage means, and an operation feature detecting means for detecting a feature of the motion in each section in the sign language sentence data divided by the boundary detected by the means. Associates each section in the sign language sentence data divided by the boundary detected by the means with the sign language word recognized by the sign language word recognition means. Sign language comprising divided section sign language word correspondence means, and sign language word string generation means for generating a sign language word string based on the correspondence between each section in the sign language data and the sign language word associated by the divided section sign language word correspondence means. A program for operating a computer as a recognition device is recorded.

As a recording medium for supplying the program,
Arbitrary ones such as a floppy disk, a magnetic disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, and a nonvolatile memory card can be used. The program read from the recording medium causes the computer to operate as the sign language recognition device described above.

[0016]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a conceptual block diagram of a sign language recognition device according to the present invention. In FIG. 1, a sign language operation input unit 101 is a unit for converting an operation representing a sign language sentence into an electric signal and inputting the signal as time-series sign language sentence data.
The well-known glove-type device 102 can be used. As the sign language operation input unit 101, a video camera and an image recognition device that recognizes the position and shape of a hand from a video image can be used. Sign language word recognition unit 10
A unit 3 recognizes each sign language word expressed in the sign language sentence data input by the sign language operation input unit 101. The sign language word recognition unit can use an existing technology, for example, a technology described in Japanese Patent Application Laid-Open No. 10-208023 (sign language recognition device).

The sign language data division unit 104 detects the boundaries of each sign language word and the transition operation (transition operation between sign language words) in the sign language sentence data input by the sign language operation input unit 101, and converts the sign language sentence data into the sign language word. This is a means for dividing into sections representing jumping movements. The motion feature extraction unit 105 is a unit for extracting information on the feature of the motion in the section from each section in the sign language sentence data divided by the sign language data division unit 104. Sign language word information storage 1
Reference numeral 06 denotes a unit for storing information about the sign language word such as the type of hand and the structure of the motion used when expressing the sign language word.

The divided section sign language word correspondence section 107 stores each section in the sign language sentence data divided by the sign language word division section 104 and the sign language word recognized by the sign language word recognition section 103 into the contents of the sign language word information storage section 106. This is means for assigning priorities to the sign language words associated with each section based on the characteristics of the movement of each section extracted by the movement feature extraction unit 105. The sign language word string generation unit 108 includes a divided section sign language word correspondence unit 107.
This is means for generating a sign language word string based on the correspondence between the divided sections in the sign language sentence data obtained and the recognized sign language words, and the priority of the sign language words in each section. As an output of the sign language word string generation unit 108, the generated sign language word string 109 is output.

FIG. 2 shows a format of the sign language sentence data input by the sign language operation input unit 101. In FIG. 2, reference numeral 201 denotes data relating to the position of the hand, and the position of the hand is further represented by x-axis data 202, y-axis data 203,
It is composed of z-axis data 204. Reference numeral 205 denotes data relating to the direction of the hand. The direction of the hand further includes a rotation angle 206 about the x axis, a rotation angle 207 about the y axis, and a rotation angle 208 about the z axis. Reference numeral 209 denotes data related to the bending of the finger. The bending of the finger further includes a bending angle 210 of the second joint of the thumb, a bending angle 211 of the third joint of the thumb, a bending angle 212 of the first joint of the index finger, and a bending angle 212 of the index finger. Bending angle 213 of the joint, bending angle 214 of the first joint of the middle finger, bending angle 215 of the second joint of the middle finger, bending angle 216 of the first joint of the ring finger, bending angle 217 of the second joint of the ring finger, first bending of the little finger Joint bending angle 218, little finger second
It consists of a joint bending angle 219. Also, 22
0, 221,..., 222 are time t1, t2, respectively.
.., Tn represents the data of the hand position, direction, and finger bending. As described above, the operation in the sign language is based on the hand position 20.
1, the hand direction 205 and the finger bending 209 are represented as time-series data.

FIG. 3 shows a format of information on each sign language word stored in the sign language word information storage unit 106.
In FIG. 3, the sign language word name 301 is a character string representing the name of the sign language word. The action type 302 is a symbol indicating the type of action of the sign language word, and any one of “with upper limit”, “without upper limit”, and “presentation action” is described. “With upper limit” is a sign language word that is expected to be divided into one or more sections by the sign language data dividing unit 104, and is a case where the number of divided sections can be predicted in advance. on the other hand,
“No upper limit” is expected to be divided into one or more sections as in the case of “with upper limit”, but the number of sections to be split cannot be predicted in advance. “Presentation action” is a sign language word that performs an action such as pressing a mark in space.
This corresponds to the case where the number of sections is one among the sign language words classified as “with upper limit”. The difference between the case where “there is an upper limit” where the number of sections is 1 and the “presentation action” is that the sign language word action corresponds to the whole or a part of the section divided by the sign language data dividing unit 104 in the former, whereas Means that there is a meaning in the part where the state changes from a state of movement to a state of no movement, so that it corresponds only to the end part of the section. The user's hand 303 is a type of hand used when expressing a sign language word, and describes any one of “right hand”, “left hand”, and “both hands”. In addition, as the information on the sign language word, the position and hand direction of the hand when expressing the sign language word, or the positional relationship of both hands when using both hands can be described. The number of sections 304 represents the number of sections in which the operation representing the sign language word is divided by the sign language data dividing unit 104. In the case of "with upper limit", describe the upper limit number of the section, and in the case of "without upper limit", 0 indicating that there is no limit
Describe. In the case of “presentation operation”, the upper limit value 1 is described.

Next, referring to FIG. 4 to FIG. 8, the sign language data dividing unit 104 detects the time when the operation speed of the hand is extremely low in the method of dividing the sign language sentence data. The operation speed of both hands is obtained and described by the following (Equation 1). In the operation of the sign language sentence, as shown in FIG. 4, the operation speed tends to be minimal at the boundary of the sign language word (time indicated by X in the figure). Therefore, both are used as boundaries of operation.

[0022]

(Equation 1)

In the equation, V0 (t) is the motion speed of both hands at time t, and P (t, i, j) is the hand i (i =
0: right hand, i = 1: left hand, site j (j = 0: palm, j =
1: thumb, j = 2: index finger, j = 3: middle finger, j = 4:
Ring finger, j = 5: little finger). Also,
The denominator of (Equation 1) is 12.

In the sign language sentence data format shown in FIG. 2, the position of the tip of each finger is not included, so that the hand position, hand direction, finger bending angle, and hand position as shown in FIG. The position of the tip of each finger is calculated using the model. As a calculation method, a method well known in the field of three-dimensional graphics or the like can be used. Alternatively, the input can be made directly by attaching a position sensor to the tip of the finger.

Also, the time when the operation speed becomes lower than the predetermined speed and the time when the operation speed changes from a state lower than the predetermined speed to a state higher than the predetermined speed are detected as boundaries. Next, in each section divided by the detected boundary,
Detect the minimum value of the operating speed for each of the left and right hands,
The result is also added to the boundary. The motion speed of each of the right and left hands is obtained by the following (Equation 2).

[0026]

(Equation 2)

In the formula, V1 (i, t) is a hand i (i = 0: right hand,
i = 1: operating speed at time t (left hand). The denominator of (Equation 2) is 6. However, in the case of the minimum value obtained from the operation speed of each hand, there is a possibility that the time at which the operation speed becomes minimum is shifted even when the left and right hands move synchronously. Therefore, a boundary to be added is determined according to the flowchart shown in FIG.

In step 601 of FIG. 6, the numbers of the minimum values of the operation speed detected by the right and left hands are compared.
If the number of the minimum values is the same, the process proceeds to step 602; otherwise, the process proceeds to step 606. In step 602, for each of the right and left hands, a difference in time length is obtained for all combinations of sections divided by the minimum value, and the average thereof is obtained. In step 603, the difference in time length between the right hand and the left hand is compared. If the right hand is smaller, the process proceeds to step 604. If the left hand is smaller, the process proceeds to step 605. In step 604, the time of the minimum value detected from the operation speed of the right hand is added to the boundary. In step 605, the time of the minimum value detected from the operation speed of the left hand is added to the boundary. In step 602, a ratio of time lengths may be obtained instead of obtaining a difference in time lengths. In step 606, the number of minimum values detected from the movement speed of each of the right and left hands is compared. If the right hand has more, the process proceeds to step 607;
Proceed to 08. In step 607, the time of the minimum value detected from the operation speed of the right hand is added to the boundary. Step 60
At 8, the time of the minimum value detected from the operation speed of the left hand is added to the boundary.

The sign language data dividing unit 104 detects a boundary based on a change in the operation direction in addition to the boundary based on the minimum value of the operation speed, and adds the boundary to the boundary. A boundary due to a change in the operation direction is detected when the following condition (Equation 3) is satisfied.

[0030]

(Equation 3)

In the equation, I (t, i, j) is a change in the movement direction at time t, hand i, and site j, n is a data range for obtaining the change in the movement direction, and Θ is a threshold value of the change in the movement direction. The boundary detected by the above (Equation 3) may be temporally shifted from the boundary due to the minimum value of the operation speed. For this reason, the boundary detected by the movement direction change is considered to be detected as overlapping with the temporally closest boundary among the boundaries detected by the operation speed, and the movement direction change is included as a characteristic of the boundary. Is added.

Information on each detected boundary is described in a format as shown in FIG. In FIG. 7, a time 701 indicates a time when the boundary is detected. The boundary type 702 includes “speed only”, “change direction”,
Either “stop transition” or “operation transition” is described.
“Only speed” indicates that the boundary is a candidate detected only by the operation speed. On the other hand, "direction change"
Indicates that the boundary is detected not only by the operation speed but also by the change in the operation direction. Also, “stop transition” is a boundary detected when the operation speed has become smaller than the predetermined threshold, and “operation transition” is because the operation speed has changed from a state smaller than the predetermined threshold to a larger state. Indicates a detected boundary. The user's hand 703 indicates whether the boundary is detected from the operation speed of both hands, the operation speed of only the left hand, or the operation speed of only the right hand.

The sign language sentence data is divided based on the detected boundaries, and divided into sections. If the division is performed only on the detected boundary, there is a possibility that the division may be made into an extra section due to a slight fluctuation of the operation or the like. Therefore, here, the extra boundary is processed by the processing according to the flowchart shown in FIG. delete. As shown in FIG. 14, this processing is performed on the boundary (i−1) and the boundary i for the boundary i whose boundary type is “velocity only”.
Is smaller than the difference D1 between the maximum operating speed in the section between and the operating speed at the boundary i, or the difference D2 between the maximum operating speed in the section between the boundary i and the boundary (i + 1) and the operating speed at the boundary i, This is the process of deleting the boundary i.

In FIG. 8, in step 801, a counter i indicating a detected boundary is set to 1 indicating the first boundary. In step 802, it is checked whether the value of the counter i is larger than the detected number of boundaries. If smaller or equal, the process proceeds to step 803. If it is larger, the process ends. In step 803, it is checked whether the boundary type of the boundary indicated by the counter i is “stop transition” or “operation transition”. If it is “stop transition” or “operation transition”, the process proceeds to step 815.
Otherwise, go to step 804. Step 804
Then, it is checked whether the value of (i-1) is smaller than 1.
If it is smaller than 1, the process proceeds to step 807, and a value larger than a predetermined threshold is set to the variable D1. (I
If the value of -1) is 1 or more, the process proceeds to step 805. In step 805, the maximum value of the operation speed in the section between the boundary indicated by (i-1) and i is obtained. In step 806, the difference D between the obtained maximum operation speed and the operation speed at the boundary indicated by the counter i is calculated.
Find 1 When calculating the difference between the operation speeds, the user's hand 703 in the information regarding the boundary is referred to, and the difference between the operation speeds of both hands is calculated for both hands, and the difference between the operation speeds of the respective hands is calculated for the left or right hand.

In step 808, it is checked whether or not the value of (i + 1) is larger than the detected number of boundaries. If it is larger, the process proceeds to step 811; otherwise, the process proceeds to step 809. In step 811, a value larger than a predetermined threshold is set to the variable D2. In step 809,
The maximum value of the operation speed in a section between the boundary indicated by (i + 1) and i is obtained. In step 810, a difference D2 between the obtained maximum operation speed and the operation speed at the boundary indicated by the counter i is obtained. In step 812, it is checked whether D1 and D2 are smaller than a predetermined threshold. If either D1 or D2 is smaller than the threshold, the process proceeds to step 813. Otherwise, go to step 815. In step 813, it is checked whether the boundary type of the boundary indicated by the counter i is “change in direction”, and if it is “change in direction”, the process proceeds to step 815. Otherwise, proceed to step 814 to delete the boundary indicated by counter i. In step 815, the value of the counter i is set to 1
Increase and return to step 802.

Next, a description will be given of a process of extracting a feature of an operation performed by the operation feature extracting unit 105. The motion feature extraction unit 105 extracts the type of hand used as a motion feature in each section in the sign language sentence data divided by the boundary detected by the sign language data division unit 104. Therefore, in each section, the ratio of the maximum operation speed and the difference in the change in the operation speed are obtained. The ratio of the maximum operation speed is calculated by (Equation 4) after obtaining the maximum operation speed of each of the right and left hands in each section.

[0037]

(Equation 4)

In the equation, R0 is the ratio of the maximum operation speed, M1 is the maximum operation speed in the right hand, and M2 is the maximum operation speed in the left hand. Also, the difference in the change in the operating speed is calculated by (Equation 5).

[0039]

(Equation 5)

In the equation, R1 is the difference in the change in operating speed, s is the start time of the section, and e is the end time of the section. The denominator of (Equation 5) is (e−s + 1). If both the parameters R0 and R1 obtained by (Equation 4) and (Equation 5) are equal to or smaller than the threshold value, it is determined that both hands are used in the section. In other cases, it is determined that only one hand is used. In (Equation 5), normalization is performed so that the maximum operation speed of each of the right and left hands coincides with each other, and the difference in the operation speed change is obtained. As the difference in the operation speed change, the difference between the operation speed changes in the right and left hands may be obtained, and then the normalization may be performed based on the ratio of the maximum operation speed. Also, in addition to the maximum operating speed ratio and the difference in operating speed change, the determination may be made based on the maximum operating speed difference, the average operating speed difference or ratio, the moving distance difference or ratio, the operating direction difference, and the like. it can. After the determination of both hands or one hand, it is further determined whether the left or right hand is mainly used. For this purpose, the ratio of the maximum operation speed is obtained in each section. The ratio of the maximum operation speed obtained here is basically the same as (Equation 4), but is calculated by (Equation 6) in consideration of the sign.

[0041]

(Equation 6)

In the equation, R2 is the ratio of the maximum operating speed for determining the hand in use. When the parameter R2 calculated by (Equation 6) is positive, it is determined that the right hand is mainly used, and when it is negative, it is determined that the left hand is mainly used. This determination can also be made based on a difference between the maximum operation speeds, a difference or ratio between the average operation speeds, a difference or ratio between the moving distances, a difference between the operation directions, and the like, in addition to the ratio between the maximum operation speeds.

After the motion feature extraction unit 105 extracts the motion feature, information on the section divided by the boundary of the motion is described in a format as shown in FIG.
In FIG. 9, reference numeral 901 denotes the start time of the section. 9
02 represents the end time of the section. The section type 903 indicates the feature of the section, and any one of “both hands”, “right hand”, “left hand”, “stop”, and “pause” is described.
“Both hands” indicates that the section is an operation using both hands. “Right hand” and “left hand” indicate a movement using one hand, and mainly describe the type of hand determined to be used. “Stop” and “pause” are added to a section sandwiched by boundaries whose boundary types 702 are “stop transition” and “operation transition”. "Stop" means that if the duration of the section is longer than the predetermined duration,
“Pause” is designated when the time length is shorter than a predetermined time length. The main player 904 is described only when the section type is “both hands”. Even when the type of the section is determined to be “two hands” based on (Equation 4) and (Equation 5), there is a possibility that the operation is actually one-handed. For this reason, when it is determined to be “both hands”, the type of the mainly used hand determined by (Equation 6), that is, “right hand” or “left hand” is described in the main hand 904. As the section type 903, in addition to the type of hand used in the sign language operation, information about the hand position and hand direction, or in the case of both hands, information on the positional relationship between both hands is extracted and described. Can also.

Next, a description will be given of a process of associating each section divided by the boundary detected from the sign language sentence data with the recognition result of the sign language word. FIG. 10 shows a format of a recognition result of the sign language word output from the sign language word recognition unit 103. In FIG. 10, the sign language word name 1001 is the name of the recognized sign language word, the start time 1002 is the start time of the recognized sign language word, the end time 1003 is the end time of the recognized sign language word, and the evaluation value 1004 is recognized. This is an evaluation value indicating the certainty of the sign language word.

The information on the section shown in FIG.
Based on the recognition result of the recognized sign language words shown in FIG. 11, the sections and the sign language words are associated with each other in accordance with the flowchart shown in FIG. In FIG. 11, in step 1101, first, a counter i for identifying a recognized sign language word is set to 1 representing the first sign language word. In step 1102, it is checked whether the value of the counter i is larger than the number of recognized sign language words. If the value of the counter i is larger than the number of sign language words, the process ends. Otherwise step 110
Proceed to 3. In step 1103, a section in which the degree of overlap with the time range of the sign language word indicated by the counter i is equal to or larger than a threshold is searched. The degree of overlap between the time range of the sign language word and the time range of each section is calculated by (Equation 7).

[0046]

(Equation 7)

In the formula, O is the degree of overlap between the time range of the section and the sign language word, Rs is the start time of the section, Re is the end time of the section, Ws is the start time of the sign language word, and We is the end time of the sign language word. . In step 1104, it is checked whether “stop” is included in the searched section. If “stop” is included, the process proceeds to step 1122. If the stop is not included, the process proceeds to step 1105 to check whether the sign language word indicated by the counter i is “presentation operation”. If the sign language word is “presentation operation”, the process proceeds to step 1106.
Otherwise, go to step 1107. Step 11
At 06, the earliest section in time of the searched sections is registered as a section corresponding to the sign language word indicated by the counter i, and the process proceeds to step 1122. In step 1107, it is checked whether or not the sign language word indicated by the counter i is "no upper limit". Otherwise, go to step 1109. In step 1108, all the searched sections are registered as sections corresponding to the sign language word indicated by the counter i, and in step 11
Proceed to 22. In step 1109, it is checked whether or not the number of searched sections is equal to or less than the number of sections 304 in the sign language word information.
Otherwise, go to step 1110.

At step 1110, the degree of overlap between the earliest and the latest sign language words in the searched sections is calculated, and the process proceeds to step 1111. The calculation of the degree of overlap can be performed by (Equation 7). In step 1111, it is determined whether or not the overlap degree of the earliest section is smaller than the overlap degree of the latest section. If the overlap degree is smaller, the process proceeds to step 1112. Otherwise, go to step 1118. In step 1112, the earliest section is deleted, and it is checked in step 1113 whether the number of remaining sections is equal to or less than the number 304 of sections in the sign language word information. If it is equal to or less than the number of sections, the process proceeds to step 1115. Otherwise, go to step 1116. In step 1115, all remaining sections are registered as sections corresponding to the sign language word indicated by the counter i, and the process proceeds to step 1122. In step 1116, the section that is the latest in time is deleted, and in step 1117 it is checked whether the number of remaining sections is equal to or less than the number of sections 304 in the sign language word information. If less than the number of sections, step 11
Proceed to 15. Otherwise, go to step 1122.
In step 1118, the section that is the latest in time is deleted, and in step 1119, it is checked whether the number of remaining sections is equal to or less than the number of sections 304 in the sign language word information.
If it is equal to or less than the number of sections, the process proceeds to step 1115. Otherwise, go to step 1120. In step 1120, the earliest section in time is deleted, and in step 1121
, The number of remaining sections is 30 in the sign language word information.
Check if it is 4 or less. If it is equal to or less than the number of sections, the process proceeds to step 1115. Otherwise, step 112
Proceed to 2. In step 1122, the value of the counter i is increased by 1, and the process returns to step 1102. In step 1115, before registering the remaining section as the section corresponding to the sign language word, the overlap between the time range represented by the remaining section and the time range of the sign language word is checked. For example, the remaining sections may be registered.

After associating the sign language words with the sections,
A list of sign language words corresponding to each section is created. The sign language word list can be easily created by searching for sign language words corresponding to the same section. FIG. 12 shows the format of the sign language word list for each section. In FIG. 12, reference numeral 1201 denotes a section start time, and 1202 denotes a section end time. The start time 1201 and the end time 1202 of the section are the same as the start time 901 and the end time 902 described in the information on the section shown in FIG. Reference numeral 1203 denotes the number of sign language words corresponding to the section. The sign language word names 1204 and 1208 are the names of the sign language words corresponding to the section, and the evaluation values 1205 and 1209 are the evaluation values of the sign language words corresponding to the section. The sign language word name and the evaluation value describe the sign language word name 1001 and the evaluation value 1004 included in the recognition result of the sign language word. Start section 1
Reference numerals 206 and 1210 are numbers representing the first section corresponding to the sign language word, and end sections 1207 and 1211 are numbers representing the last section corresponding to the sign language word. Section number 12
061, 1207, 1210, and 1211 are used to generate a correct candidate in the sign language word string generation described later when a plurality of sections correspond to the sign language word.

After the sign language word list is created, the priority of the sign language words in the sign language word list is determined by comparing the section type 903 and the main sign 904 in the information about the section with the user's hand 303 in the sign language word information. change. this is,
This is performed by raising the priority of the sign language used by the hand 303 used, which is the same as the section type 903. In addition, section type 9
When 03 is “both hands”, the priority order of the sign language word whose hand 303 is the same as the main hand 904 is also increased. However, in this case, the degree of raising the priority is smaller than the case where the users 303 are the same. For example, when a method of increasing the evaluation value by multiplying the evaluation value by a constant greater than 1 is used as a method of changing the priority, the constant used when the section type 903 and the user 303 are the same is α, and the main operator 904 is used.
And β is a constant used when the user 303 is the same as
Set a constant such that α> β. As a method of changing the priority, a method of adding a predetermined constant to the evaluation value may be used. Alternatively, a sign language word having the same section type 903 and the user's sign 303 has unconditional priority over other sign language words, and then the master sign 904 and the user's hand 303 unconditionally prioritize the same sign language word. A method can also be used. In addition, as the sign language word information and the section type, in addition to the type of the hand to be used, information on the position and hand direction of the hand, or the positional relationship of both hands when using both hands may be described. In a similar manner, the priority of the sign language word corresponding to the section can be changed.

After the priority of the sign language word corresponding to each section is changed, a sign language word string is generated based on the correspondence between each section and the sign language word. In the generation of the sign language word sequence, the corresponding sign language words are selected from each section in descending order of priority,
By combining them, a sign language word string candidate is generated. that time,
An evaluation value of the entire sign language word string candidate is calculated, and a candidate having a high evaluation value is selected as a result. At this time, in the case of a sign language word corresponding to a plurality of sections, the sign language words overlapping the section of the sign language word are prevented from being combined. Therefore, the generation of the sign language word string is performed according to the flowchart shown in FIG.

In FIG. 13, in step 1301,
Clear the area for storing sign language word string candidates. In step 1302, the counter i indicating the section is set to 1 indicating the first section. In step 1303, it is checked whether or not the value of the counter i is greater than the number of sections detected from the sign language sentence data. If it is larger, the process proceeds to step 1305. Step 13
At 05, the sign language word string stored in the area for storing the sign language word string is output as a result, and the process is terminated. In step 1304, the counter j indicating the sign language word associated with the section indicated by the counter i is set to 1 representing the first sign language word. At step 1306, the counter j
Is greater than the number of sign language words corresponding to the section indicated by the counter i. Otherwise, go to step 1310.

In step 1307, the sign language word string candidates are rearranged in descending order of evaluation value. The evaluation value of the sign language word string candidate can be obtained by averaging the evaluation values of the sign language words included in the candidate. Alternatively, the geometric mean of the evaluation values of the sign language words may be used. Alternatively, an evaluation value or the like based on the positional relationship, grammatical, or semantic relationship between sign language words can be used. In step 1308, candidates consisting of the same sign language word string in the sign language word string candidates are deleted, a predetermined number of candidates are selected in order from the candidate having the highest evaluation value, and the other candidates are deleted. Step 13
In step 09, the value of the counter i is increased by one, and in step 130
Return to 3.

At step 1310, a sign language word indicated by the counter j in the section indicated by the counter i is selected. In step 1311, a counter k indicating an existing sign language word string candidate is set to 1 representing the first sign language word string candidate. In step 1312, it is checked whether or not the value of the counter k is larger than the number of the existing sign language word string candidates. If smaller or equal, go to step 1314. In step 1313, the counter j
Is incremented by 1, and the process returns to step 1306.

In step 1314, the section corresponding to the last sign language word in the sign language word string candidate indicated by the counter k is
It is checked whether the sign language word indicated by the counter j overlaps with the corresponding section. In step 1315, if there is an overlap, the process proceeds to step 1316; otherwise, the process proceeds to step 1317. In step 1316, the sign language word indicated by the counter j is replaced by the counter k after the sign language word overlapping with the section corresponding to the sign language word indicated by the counter j is removed from the sign language words in the sign language word string candidate indicated by the counter k. It is combined with the indicated sign language word string candidate and stored as a new sign language word string candidate. In step 1317, the sign language word indicated by the counter j is combined with the sign language word string candidate indicated by the counter k, and the sign language word string candidate indicated by the counter k is updated.
At step 1318, the counter k is incremented by 1, and the process returns to step 1312.

In step 1314, instead of checking whether the section corresponding to the last sign language word in the sign language word string candidate indicated by the counter k overlaps the section corresponding to the sign language word indicated by the counter j, the counter k sets The overlap between the time range of the section corresponding to the last sign language word in the candidate sign language word string shown and the time range of the sign language word indicated by the counter j may be calculated. In this case, if the size of the overlap is equal to or larger than the threshold in step 1315,
Proceed to 16; otherwise, proceed to step 1317. In step 1316, the sign language words in the sign language word sequence candidates indicated by the counter k are changed until the overlap between the time range of the sign language words indicated by the counter j and the time range of the sign language word sequence candidates indicated by the counter k becomes less than the threshold value. After deleting,
The sign language word indicated by the counter j is combined with the sign language word string candidate and registered as a new candidate.

[0057]

According to the present invention, the boundaries between sign language word motions are detected in various forms by detecting a motion speed change in both hands, a motion speed change in each of the left and right hands, and a motion direction change. From the sign language sentence data including the sign language word, it is possible to accurately detect the boundary of the operation representing each sign language word.

Further, based on the characteristics of the operation of the sign language word,
By associating the recognized sign language words with each section in the divided sign language sentence data and generating a sign language word string based on the result, it is possible to generate an appropriate sign language word string. Furthermore, by changing the priority of the recognized sign language words based on the feature of the operation in each section, it is possible to improve the accuracy in generating the sign language word string.

[Brief description of the drawings]

FIG. 1 is a conceptual block diagram of a sign language recognition device according to the present invention.

FIG. 2 is a diagram showing a format of sign language sentence data.

FIG. 3 is a diagram showing a format of information stored in a sign language word information storage unit.

FIG. 4 is a view for explaining an example of a relationship between a change in operation speed in sign language sentence data and a sign language word and a transition.

FIG. 5 is a diagram showing an example of a hand model used to calculate the position of the tip of each finger.

FIG. 6 is a flowchart of processing for adding a boundary detected from the movement speed of each of the right and left hands.

FIG. 7 is a diagram showing a format of information regarding a boundary.

FIG. 8 is a flowchart of a process of dividing sign language sentence data into sections according to detected boundaries.

FIG. 9 is a diagram showing a format of information on sections.

FIG. 10 is a diagram showing a format of information on recognized sign language words.

FIG. 11 is a flowchart of a process of associating a recognized sign language word with each section obtained by dividing the sign language sentence data.

FIG. 12 is a diagram showing a format of a sign language word list corresponding to each section.

FIG. 13 is a flowchart of a process of generating a sign language word string.

FIG. 14 is an explanatory diagram of processing for deleting a boundary.

[Explanation of symbols]

101: Sign language operation input unit, 102: Glove type device, 103
... Sign language word recognition unit, 104 ... Sign language data division unit, 105
... operation feature extraction unit, 106 ... sign language word information storage unit, 10
7: Signed section sign language word correspondence unit, 108: Sign language word string generation unit, 109: Sign language word string

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Masaru Takeuchi 1-280 Higashi Koigabo, Kokubunji-shi, Tokyo F-term in Central Research Laboratory, Hitachi, Ltd. 5B057 AA20 DA20 DC20 DC08 DC08 5B091 CB01

Claims (11)

[Claims] 1. Sign language operation input means for converting an operation of a sign language sentence into an electric signal and inputting it as time-series sign language sentence data, and each sign language expressed in the sign language sentence data input from the sign language operation input means. Means for recognizing a word; sign language data dividing means for detecting a boundary of an action in the sign language sentence data input from the sign language action input means based on a feature amount of the action; and a boundary detected by the sign language data dividing means. Feature detecting means for detecting the feature of the action in each section in the sign language sentence data divided by the sign language sentence data; sign language word information storage means for storing information relating to the sign language word; and sign language words stored in the sign language word information storage means On the basis of the information on the sign language word recognition means in each section in the sign language sentence data divided by the boundary detected by the sign language data division means. Thus, a divided section sign language word correspondence unit for associating the recognized sign language words, and a sign language word string is generated based on a correspondence relationship between each section in the sign language data and the sign language word associated by the divided section sign language word correspondence unit. A sign language word string generating means. 2. The sign language recognition device according to claim 1, wherein
The sign language data dividing means determines a boundary of the motion based on at least one characteristic amount among a motion speed of the palm, a change in the motion direction of the palm, a motion speed of the tip of each finger, and a change in the motion direction of the tip of each finger. A sign language recognition device characterized by detecting. 3. The sign language recognition device according to claim 1,
The sign language data dividing means performs a time when the operation speed of the palm or the operation speed of the tip of each finger is minimized, a time when the operation speed of the palm or the operation speed of the tip of each finger is smaller than a predetermined speed, The time when the operating speed or the operating speed of the tip of each finger changes from a state lower than the predetermined speed to a state higher than the predetermined speed, and a change in the operating direction of the palm or the end of each finger is a predetermined threshold. A sign language recognition device that detects a time that exceeds a time as a boundary of an action. 4. The sign language recognition device according to claim 1,
The sign language data dividing means detects a boundary of the motion based on the motion speed of both the left and right palms and the motion speed of the tip of each finger, and then, for each section in the sign language sentence data divided by the detected boundary, the right hand Based on the motion speed of the palm and the motion speed of the tip of each finger, and the motion speed of the palm in the left hand and the motion speed of the tip of each finger, the motion boundaries are detected, and the detected boundaries are added as motion boundaries. Sign language recognition device characterized by performing. 5. The sign language recognition device according to claim 3, wherein said sign language data dividing means further detects, as a boundary, a time at which a change in the movement direction of the tip of the palm or each finger exceeds a predetermined threshold. A sign language recognition device that associates the detected boundary with a temporally closest boundary among the boundaries detected based on the operation speed. 6. The sign language recognition device according to claim 1,
The operation feature detecting means includes: a difference or a ratio of the maximum operation speed of each of the right and left hands in the section; a difference or a ratio of the average operation speed of each of the right and left hands in the section; Based on at least one characteristic amount among the difference or ratio of the moving distance, the difference between the movement speeds of the right and left hands in the section, and the difference in the movement direction in the section, the movement in the section uses both hands. A sign language recognition apparatus characterized in that each section is classified into one of the following states: using only a right hand or using only a left hand. 7. The sign language recognition device according to claim 1,
The information stored in the sign language word information storage means includes information on the characteristics of the sign language word motion, the number of sections in which the sign language word motion is divided by the boundaries of the motion, and the hand used when performing the sign language word motion. A sign language recognition device comprising at least one of the types. 8. The sign language recognition device according to claim 7,
The information on the characteristics of the sign language word operation includes "upper limit" in which the sign language word operation may be divided into a predetermined number of sections, and "upper limit" in which the number of divided sections is unclear. A sign language recognition device characterized by including a result classified into any of "none" and "presentation operation" including an operation of pressing a mark. 9. The sign language recognition device according to claim 1, wherein the divided section sign language word correspondence unit is based on the number of sections divided by the feature of the sign language word operation and the boundary of the operation. A sign language recognition device for determining a section corresponding to a sign language word. 10. The sign language recognition device according to claim 9, wherein the divided section sign language word correspondence unit changes the priority of the sign language word associated with each section based on the feature of the operation in each section. Sign language recognition device. 11. A sign language operation input means for converting an operation of a sign language sentence into an electric signal and inputting the signal as time-series sign language sentence data; and each sign language expressed in the sign language sentence data input from the sign language operation input means. Means for recognizing a word; sign language data dividing means for detecting a boundary of an action in the sign language sentence data input from the sign language action input means based on a feature amount of the action; and a boundary detected by the sign language data dividing means. Based on information on the sign language words stored in the sign language word information storage means, based on the motion feature detection means for detecting the features of the actions in each section in the sign language sentence data divided by the sign language sentence data. A divided section sign language word pair that associates each section in the sign language sentence data divided by the boundary with the sign language word recognized by the sign language word recognition unit. Means, a sign language word string generating means for generating a sign language word string based on the correspondence between each section in the sign language data and the sign language word associated by the divided section sign language word corresponding means, A computer-readable recording medium on which a program for operating a computer is recorded.