JPH07282235A - Motion recognition device - Google Patents

Abstract

(57) [Abstract] [Purpose] To facilitate the recognition of hand motions by extracting skin color areas, coding finger shapes, and coding movements in the space of the left and right hands. [Structure] From the time-series images of hand movements taken by the imaging device 3, the skin color area is extracted, the centers of gravity of the face, right hand, and left hand are extracted, and the spatial position calculation device 7 calculates the spatial positions of the left palm and the right palm. To do. The finger shape code creating device 11 creates a finger shape code string. The group that includes the recognition target hand motion is specified by the use state of the hand, the static movement, and the change of the finger shape. The motion code creation device 16 creates motion codes for the left and right hands. The start position code creating device 17 creates a start position code for manual operation. The hand action word matching device 22 determines a hand action word corresponding to the hand action taken by the image taking device 3 from the hand action words in the identified group based on the motion code, the start position code, and the finger shape code. .

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a motion recognition device,
In particular, it recognizes human gestures and hand gestures without using sensors or colored gloves, interfaces between humans and machines based on the recognition results, and can be used for pointing devices and sign language recognition. Regarding the device.

[0002]

2. Description of the Related Art As a method of understanding human hand gestures and gestures, there is a method of measuring each state by attaching a sensor to the body.

Further, in the method using a camera, contour information is extracted and a state is estimated by comparison with a model prepared in advance, or a method in which a colored glove is used and a movement of a finger or the like is detected by color information. There is.

[0004]

When recognizing human hand movements and gestures, there are commercially available devices for attaching an optical fiber to the body and seeing the change in the light quantity by the change in the finger shape. However, since the code is attached to the body, the movement is restricted, and there are problems with the feeling of wearing. There is also a method of wearing colored gloves and recognizing the shape depending on the difference in color, but this method also requires wearing gloves on the human body, and when the gloves are not present, hand movements and gestures cannot be recognized.

In addition, in order to measure the spatial position of the hand when recognizing the hand motion, a large-scale device using a plurality of photographing devices has been required.

Further, in a device having an input section for using body movements such as body movements and hand movements, it has been difficult to give an appropriate timing for starting the movement recognition while continuously monitoring.

In view of the above problems, an object of the present invention is to recognize a hand motion by extracting a skin color portion and coding a motion without wearing gloves.

Another object of the present invention is to make it possible to recognize a manual operation with only one photographing device.

A further object of the present invention is to provide an accurate timing for starting motion recognition.

[0010]

In order to achieve the above object, the present invention provides a photographing means for photographing one object from left and right cameras from different directions, and an image of the left and right cameras where photographing is performed. Storage means for storing, skin color extraction means for extracting a skin color portion in the image, center of gravity detection means for measuring the center of gravity position of the area of the skin color portion, and center of gravity position coordinates obtained by the center of gravity detection means for the left and right cameras. It is a motion recognition device provided with a spatial position calculation means for obtaining absolute coordinates in the space of the left and right palms by parallax, and a spatial position storage means for storing the value obtained by the spatial position calculation means.

In addition to the above construction, a palm specifying means for specifying the palm to be used, and a hand movement / movement judging means for judging whether the palm being used is fixed or moving in space. , Finger shape code creating means for coding the shape of the palm, shape changing means for determining a change in finger shape, group specifying means for specifying a group according to a motion of a hand motion, group of hand motion based on a motion of a hand motion Group storage means for performing conversion, change point detection means for detecting a change point of motion, motion code creation means for creating a motion code based on that, start position code creation for creating a start position code of the palm currently in use Means, finger shape collating means for collating the stored finger shape and the code created by the finger shape code creating means, manual operation word collating means for collating the manual operation word, Hand gesture word storage means for registering and coding operation for each group, the audio hand operation word specified, character, constituted by the finger movement word conversion means for outputting an animation.

Further, one photographing means for photographing the recognition target, arm shape extracting means for extracting the shape of each part of the arm from the photographed screen, and the shape of each part of the arm and each part of the arm. It is a motion recognition device provided with a spatial position calculating means for calculating a spatial position from the shape of an attached index object.

Further, a start code registering means for holding a feature serving as a recognition start signal as a start code, a feature code creating means for extracting a feature from an image stored in the storage means to create a feature code, and a start code And a recognition start code determining means for comparing the feature code with each other, and a recognition start notifying means for outputting a recognition start signal to the motion recognizing section when the start code and the feature code match.

Further, start image registration means for holding an image serving as a recognition start signal as a start image, image processing means for pre-processing the image stored in the storage means, pre-processed image and start image. When the recognition start image determination means for comparing and the preprocessed image and the start image match,
The motion recognition device includes a recognition start notification unit that outputs a recognition start signal to the motion recognition unit.

[0015]

According to the motion recognition apparatus of the present invention, one object is photographed by the left and right cameras from different directions, the skin color part is extracted from the photographed image, and the absolute coordinates in the space of the left and right palms are obtained. By memorizing the position, the motion of the hand can be recognized.

Further, it is possible to identify the group of manual movements by identifying the palm used, determining whether the palm is fixed or moving in space, and determining the change in the shape of the palm. By creating a motion code, creating a start position code, and specifying the finger shape, it is possible to recognize the manual operation word by collating the manual operation word in the specified group.

Further, the skin color detecting means can extract the skin color area, the center of gravity detecting means can calculate the positions of the center of gravity of the face and the left and right palms in the image, and the arm shape detecting means can know the shape information of the arm. The position of the center of gravity of the left and right palms in the image, the shape information of the arm, and the ring-shaped index attached to the arm are used to calculate the position of the center of gravity of the left and right palms in the three-dimensional space by the spatial position calculation means. Therefore, it is possible to recognize the manual operation with only one photographing device.

Further, the recognition start code determining means determines whether or not the feature code of the input image coded by the feature code creating means matches the start code registered in the start code registering means. The recognition start notifying means can generate a recognition start signal in the case of a determination result that they match, and give an appropriate timing for starting the motion recognition.

Further, the recognition start image determining means determines whether the preprocessed input image by the image processing means matches the image registered in the start image registration means. The recognition start notifying means can generate a recognition start signal in the case of a determination result that they match, and give an appropriate timing for starting the motion recognition.

[0020]

First Embodiment A motion recognition apparatus according to a first embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the arrangement of a motion recognition apparatus according to the first embodiment of the present invention. The TV cameras 1 and 2 are for stereoscopically photographing hands.

A photographing device 3 for photographing the images of the left and right cameras, a storage device 4 for storing the images of the left and right cameras, a skin color extracting device 5 for extracting the skin color part in the image, and a barycentric position of the area of the skin color part are measured. Centroid detecting device 6, centroid detecting device 6
The spatial position calculation device 7 for obtaining the absolute coordinates in the space of the left and right palms by the parallax of the barycenter position coordinates obtained by the spatial position storage device 8 for storing the value obtained by the spatial position calculation device 7, A palm identifying device 9 for use that identifies whether the motion is using only the left palm, only the right palm, or using both hands, whether the palm being used is fixed in space A hand motion / movement determination device 10 that determines whether or not the presence of the palm is obtained based on the absolute coordinates of the fresh palm that is obtained by the spatial position measuring device 8, and a finger shape code creation device 11 that codes the shape of the fresh palm that is obtained by the skin color detection device 5. , A shape change determination device 12 for determining a change in finger shape by comparing time-series images of the finger shape encoding device 11, and a palm specification device , A group specifying device 13 for specifying a group of sign language words based on the results of the three devices of the hand motion / movement determining device 10 and the hand shape change determining device 12, and an element for grouping the sign language words. A group storage device 13, a change point detection device 15 for comparing time series images of the spatial position calculation device 8 to detect a change point of motion, and a motion code creation device 16 for creating a motion code based on the change point detection device 15.
A starting position code creating device 17 for creating a starting position code of the new palm used in the first image stored in the spatial position calculating device 8, and a shape and a finger shape code creating device stored in the finger shape storage device 19. Finger shape matching device 18, motion code creating device 16, start position code creating device 17, finger shape matching device 1 for matching the code created in 11
A hand operation word collation device 20 for collating sign language words using
Hand movement word storage device 22 in which sign language words are coded for each group and registered 22 Hand movement word conversion device 21 for outputting the specified sign language words by voice, characters, or animation
It is composed of

Next, regarding the specific operation of this embodiment,
This will be described with reference to the flowcharts shown in FIGS. 5A to 5D and FIG.

As shown in FIG. 1, the upper half of the body is photographed by the photographing device 3 by the television cameras 1 and 2 at different angles and stored in the storage device 4 (step 101).

Although the storage device 4 is stored at a sampling interval of 1/30 seconds, which is the standard of NTSC, as a standard, by changing the parameters of the storage device 4, the sampling cycle is, for example, 1/10 seconds or 1/5. It is also possible to change to seconds. As shown in FIG. 10, the number of images sampled is time-series numbers, IMG_1, IM
G_2 to IMG_N are added (step 102).

Next, from the image of IMG_1, the skin color detecting device 5
The skin color portions S1, S2, S3, S shown in FIG.
4, S5 is detected (step 103).

Regions which have a skin color in the natural world but do not have an area larger than a certain size such as S4 and S5 are removed as isolated points (step 104).

In the flesh color area, the right palm is S1, the left palm is S2, and the head is S3 (step 1).
05).

Next, the contour of the skin color portion is extracted, and the center of gravity position of the area portion is calculated by the center of gravity detecting device 6. Figure 10
P1, P2, and P3 are the detected barycentric positions (step 106).

FIG. 11 shows the position on the image plane of the right palm detected by the center-of-gravity detection device 6 by the left and right cameras. With the center of each of the left and right image planes as the origin of the image plane, P1_left and P1_right are barycentric positions of the right palm projected on the images of the left and right cameras. At this time, the coordinate value when taken by the right camera is P1_Right
t = (XR, YR), and the coordinate value when the image is taken by the left camera is P1_left = (XL, YL). The respective coordinate values are sent to the spatial position calculation device 7.

[0031]

[Equation 1]

[Mathematical formula-see original document] Based on (Equation 1), the absolute coordinates of the right palm are calculated based on the left and right parallax. In the present embodiment, calibration is performed before photographing, and the position of the person's neck to be photographed is the absolute origin as shown in FIG. 14, the right hand direction is the positive direction of the X axis, and the head direction is the Y axis. The positive direction of the person is defined as the Z-axis positive direction. Similar to the right palm, the left palm and the absolute coordinates of the center of gravity of the face are also calculated (step 107).

The absolute positions of the right palm, the left palm and the face are calculated for the number of sampled images, and the absolute coordinate values are stored in the spatial position storage device 8. FIG. 12 shows an example in which specific values are stored (step 10).
8).

The values in FIG. 12 are put into the palm identifying device 9 (step 109). In this embodiment, both the right palm and the left palm are detected and the position in the absolute coordinate system is specified, so it is determined that the right palm and the left palm are in use (step 110).

Next, the values shown in FIG.
Enter (step 111). By substituting the X, Y, and Z coordinate values of the values shown in FIG. 12 which are adjacent in time series, it can be seen that the right hand is moving, but the left hand is not moving (step 112).

[0036]

[Equation 2]

Next, feed finger shape codes are created in the finger shape code creating apparatus 11 for the number of sampled images of the right and left palms detected by the skin color detecting device 5.
FIG. 13 shows an image of the right palm sent to the finger shape code creating apparatus 11.

Next, the algorithm for encoding the finger shape will be described. As shown in FIG. 14, the position of the skin color portion farthest from the center of gravity P1 obtained by the spatial position calculation device 7 is defined as an end point e1,
A line connecting the center of gravity P1 and the end point s1 is defined as a palm direction vector VEC.

Next, the square SQ having the smallest area capable of enclosing all the skin color portions is aligned with the palm orientation vector VEC as the center line of the square SQ, and one side of the square SQ is aligned with the end point e1 to form the square SQ. To do.

Next, as shown in FIG. 15, the square SQ is divided into four equal parts. Replace each square with SQ_0, SQ_1, SQ
_2 and SQ_3, the ratio of the area of each skin color is calculated. Square area SQ_0, S
The code is set to 1 when more than 80% of Q_1, SQ_2 and SQ_3 is set, and set to 0 when less than 20%. When the area ratio is between 20% and 80%, the code is indefinite and the region is divided into four again. In Figure 15, SQ
Since the skin color area ratios of _0, SQ_1, SQ_2, and SQ_3 are between 20% and 80%, all the codes are undefined.

Next, SQ_0, SQ_1, SQ_2, SQ
Divide _3 into four equal parts. The state at that time is shown in FIG. Here, SQ_13 and SQ_16 are code 1,
SQ_4, SQ_6, SQ_9, SQ_11, SQ_1
2, SQ_14 and SQ_19 are code 0, as shown in FIG.
SQ_5, SQ_8, SQ_7, SQ as shown in _b
Codes of _10, SQ_15, SQ_17, and SQ_18 are indefinite.

The area where the code is undefined again is divided into four equal parts. The state of being divided into four is shown in FIG. SQ at this time
_26, SQ_30, SQ_31, SQ_34, SQ_
36, SQ_40, SQ_41, SQ_44, SQ_4
5, SQ_46 becomes code 1, SQ_20, SQ_
22, SQ_25, SQ_27, SQ_33, SQ_3
5, SQ_37 and SQ_39 are code 0.

As shown in FIG. 17B, a hatched portion is a coded area and a white portion is a code undefined portion. The code undefined area is 4 again
It is divided and the same processing is performed. In the present embodiment, the four divisions are performed up to this point. When the area ratio is between 20% and 80% when the four divisions are performed, the code is specified as 1 or 0 with 50% as the boundary.

By the above processing, the image of the right palm has been entirely coded as shown in FIG. Although the number of layers is four here, the layers may be deeper to be 5, 6 for more detailed coding (step 11).
3).

Next, the time-series coded values of the hand shape are sent to the shape change determination device 12 (step 114).

When the time series images shown in FIG. 13 are similarly coded, it becomes as shown in FIG. When the coded image of IMG_1 and the image of IMG_2 are overlapped with the center of gravity P1 and the finger direction vector VEC, the code shows the same shape.
The hand orientation direction VEC has changed on the image, but the coded shape shows the same shape. That is, in this embodiment, no change in shape is recognized and it is determined that there is no change in shape (step 115).

The results of the palm identifying device 9, the hand movement / moving motion determining device 10, and the shape change determining device 12 used are sent to the group identifying device 13 and the values in the group storage device 14 are collated to identify the group.

In the present embodiment, as shown in FIG. 20, both hands are used (from the palm identifying device 9 in use), left hand is fixed, right hand is moved (from hand movement / motion determination device 10), no change in right hand shape,
With no change in the shape of the left hand (from the shape change determination device 12), it is specified as the group i (step 116).

When the group i is specified, the group specifying device 13 sends the group number i to the manual word collating device 20.
Is sent (step 117).

Next, FIG. 13 obtained from the spatial position storage device 8
From the image IMG_2 of IMG_1, the difference between the respective coordinates of X, Y and Z of IMG_1 is obtained to obtain the direction of movement. Similarly, the difference between IMG_3 and IMG_2 is IMG_4 through I
Difference of MG_3. ． ． From IMG_N to IMG_ (N-
When the difference of 1) is obtained, the movement as shown in FIG. 21 is obtained (step 118).

When the time-series images obtained in FIG. 21 are sent to the change point detecting device 15, it can be seen that the movement is changing downward from the front at (IMG_4-IMG_3). This is the change point CHANGE (step 11
9).

When the time-series code shown in FIG. 21 is passed to the motion code generator 16, a change in motion is recognized at the change point CHANGE, and the code is coded as (forward direction, downward direction). At this time, the time factor is gone. 22_b
The motion code is determined based on such a table, and in this embodiment, the motion code is (9, 4). The movement direction is a relative value indicating in which direction a movement is made from a certain point as shown in FIG. In this embodiment, there are 26 types of motion codes as shown in FIG. 22_b (step 120).

Next, the first value of the spatial position storage device 7, FIG.
The X, Y, and Z coordinates of IMG_1 shown in 3 are passed to the start position code creating device 17. In this embodiment, when the code shown in FIG. 23 is used as the basis, the start position code of the right palm is (1, -1, 1) (step 121).

The shape obtained by the finger shape code creating device 11 is compared with the pattern stored in the finger shape storage device to specify the finger shape number of the finger shape storage device 19 (step 122).

The motion code found in step 120,
The start position code obtained in step 121 and the finger shape number obtained in step 122 are used as the hand operation word collation device 2
0 (step 123).

In the manual operation word collation device 20, step 11 is executed.
Since the sign language words are stored in the format shown in FIG. 24 obtained in step 7, the sign language words corresponding to the group code i are extracted as candidates. The sign language word in the candidates is identified based on the code and number sent in step 123 (step 124).

The sign language word recognized by the manual motion word collating device 20 is sent to the output device, and the voice, the character, and the code to the remote place are transmitted based on the code of the manual motion word, and converted into animation or the like at the receiving end. Then, the output can be performed.

In this embodiment, the person who has recognized the sign language word can recognize the body motion arbitrarily by using the above algorithm even in the body motion.

(Embodiment 2) Next, a motion recognition apparatus according to a second embodiment of the present invention will be described. The device of this embodiment is
This can be realized by replacing the photographing device 3, the storage device 4, the skin color detection device 5, the center of gravity detection device 6, and the spatial position calculation device 7 of the configuration of the first embodiment shown in FIG. 1 with the configuration shown in FIG.

FIG. 2 shows an example of the structure of the motion recognition apparatus of this embodiment until the spatial position is measured. Reference numeral 30 is a photographing device. Reference numeral 31 is an arm shape detecting device. 32 is a spatial position calculation device. The imaging device 30 is composed of one camera.

The overall processing flow up to the spatial position calculation device 32 will be described. The imaging device 30 is installed so that the area above the waist of a person is sure to enter.

A series of time-series images photographed by the photographing device 30 and sampled are time-sequentially numbered to be IMG_1, IMG_2, ..., IMG_N.
Time series images IMG_1, IMG_2 ,. ． ． , IMG_
N is stored in the storage device 4.

Time series images IMG stored in the storage device 4
_1, IMG_2 ,. ． ． , IMG_N is IMG_1
To the final image IMG_N one frame at a time and sent to the skin color detection device 5, the arm shape detection device 31, and the spatial position calculation device 32, and the following procedure is performed for each image IMG_i (i = 1 ,. , N).

From the image IMG_i, the skin color detecting device 5 detects the face skin color area FSP, the right hand skin color area RSP, and the left hand skin color area L.
The SP is detected and passed to the center of gravity detection device 6.

The center of gravity detection device 6 detects the center of gravity FC of the face, the center of gravity LC of the left palm, and the center of gravity RC of the right palm on the image IMG_i, and passes them to the arm shape detection device 31.

The processing in the skin color detecting device 5 and the center of gravity detecting device 6 is as described in the first embodiment.

The arm shape detecting device 31 receives the passed face center of gravity F
Based on C, left palm center of gravity LC, right palm center of gravity RC, and image IMG_i, right shoulder SR, left shoulder SL, right elbow ER, left elbow EL, arm shape information ASI are calculated, and arm shape information ASI is calculated.
And the left center of gravity LC, right right center of gravity RC, right shoulder S
The R, the left shoulder SL, the right elbow ER, and the left elbow EL are passed to the spatial position calculation device 32. The spatial position calculation device 32 uses the left-hand center of gravity space coordinates (Xwl, Ywl, Zwl) in the three-dimensional space,
The right-hand center of gravity space coordinates (Xwr, Ywr, Zwr) are calculated.

Next, the processing procedure for one image IMG_i in the arm shape detecting device 31 will be described in detail.

FIG. 25 shows the target person H in the image IMG_i.
The contour image of M and the arm shape information ASI detected by the arm shape detection device 31 are shown.

The coordinate values in the explanation of the arm shape detecting device 31 are such that the center of the image plane IMP of the photographing device 30 is the origin Og, the X axis positive direction is horizontal rightward, and the Y axis positive direction is vertical upward. It is a coordinate value in the system Og-XY. The coordinate value of the upper right point of the image plane IMP is (WIDETH / 2,
HEIGHT / 2), and the coordinates of the lower left point of the image plane IMP are (-WIDETH / 2, -HEIGHT / 2).
WIDETH is the width of the image plane IMP, and HEIGHT
Is the height of the image plane IMP. RUdx is the upper right arm R
It is the length of the projected image of U in the X-axis direction. RUdy is the length in the Y-axis direction of the projected image of the upper right arm RU. RLdx
Is the length in the X-axis direction of the projected image of the right lower arm RL. R
Ldy is the length in the Y-axis direction of the projected image of the lower right arm RL. LUdx is the length in the X-axis direction of the projected image of the upper left arm LU. LUdy is the length in the Y-axis direction of the projected image of the upper left arm LL. LLdx is the length in the X-axis direction of the projected image of the left lower arm LL. LLdy is the length in the Y-axis direction of the projected image of the left lower arm LL.

FIG. 6 shows a flow chart of the processing procedure of the arm shape detecting device 31. The procedure will be described in detail below with reference to the flow chart.

First, the contour image of the target person HM is detected.
As an example of the method of extracting the contour image of the target person HM,
Before the recognition, the background image BGIMG including only the background not including the target person is photographed and stored. The image IMG_i and the background image BGI passed to the arm shape detection device 31
The difference of MG is taken and only the part where the change is large is imaged IMG
Extract from _i. The edge of the extracted portion is detected, and the detected edge is set as the contour image of the target person HM.
FIG. 26 shows an example of the extracted contour image (step 20).
0).

Next, the left and right ends of the body are detected. Straight line IP
On L (Y = -HEIGHT / 2), X = -WIDTH
From // 2 in the positive X direction, a point that intersects the contour image is detected first, the detected point is the left end BL of the body, and its coordinate value is (Xbl, -HEIGHT / 2). Similarly, on the straight line IPL, a search is made from X = WIDTH / 2 in the negative X direction, a point that intersects the contour image first is detected, the detected point is set as the right end BR of the body, and its coordinate value is (Xbr,- H
EIGHT / 2) (step 201).

Next, the left and right shoulders are detected. The face center of gravity FC (Xf, Yf) detected by the center of gravity detection device 6, the right palm center of gravity RC (Xrc, Yrc), the left palm center of gravity LC (X
lc, Ylc) based on the right shoulder SR (Xsr, Ysr),
The left shoulder SL (Xsl, Ysl) is calculated. Shoulder is face center of gravity F
Since it is below C, the search is performed from the point SC (Xf, Yf-SDFS) below SDSF below the face center of gravity FC.

Here, SDSF is the difference between the face center of gravity FC and the shoulder height of the person who has been registered in advance. SDS
In F, the data of the person to be recognized is registered, or the average value of the data of a plurality of persons is registered. Search in the positive X direction parallel to the X axis from the point SC to detect the end points of the contour image,
Let that point be TSR (Xtsr, Ytsr).

If Xtsr <Xbr, the shoulder is point SC.
And below the line of point TSR, Xtsr = X
The detection of the end points of the contour is repeated while decreasing the value of Y by dy until it becomes br. dy is a parameter, and has a small value such as 1. Xtsr = Xb at the nth time
Assuming r, the coordinate value of the right shoulder SR is as follows.

[0077]

[Equation 3]

When Xtsr> Xbr, points SC and T
Assuming that it is above the SR line, the detection of the end points of the contour is repeated while increasing the value of Y by dy until Xtsr = Xbr. Assuming that Xtsr = Xbr at the n-th time, the coordinate values of the right shoulder point SR are as follows.

[0079]

[Equation 4]

When Xtsr = Xbr, the coordinate values of the right shoulder point SR are as follows.

[0081]

[Equation 5]

As shown in FIG. 26 (b), when the point TSR is in the right hand skin color area RSP, Y is dy from the end point.
The contour is searched while gradually lowering, and a point PR (Xpr, Ypr) whose end point does not become the right hand skin color region RSP is detected.

Next, the contour of the right-hand skin color region RSP is searched while increasing Y by dy from TSR, and the point QR (Xqr, Yqr) which collides with the contour of the body is detected. From the point PR and the point QR, the coordinates of the right shoulder point SR are as follows.

[0084]

[Equation 6]

Next, the left shoulder point SL is obtained. Point SC to X
The end points of the contour image are searched in the positive direction parallel to the axis.
Let the detected point be TSL (Xtsl, Ytsl).
In the case of Xtsl> Xbl, it is assumed that it is below the line of the points SC and TSL, and Y is set until Xtsl = Xbl.
The detection of the end points of the contour is repeated while the value of is decreased by dy. Assuming that Xtsl = Xbl at the nth time, the coordinate values of the left shoulder point SL are as follows.

[0086]

[Equation 7]

When Xtsr <Xbr, points SC and T
Assuming that it is above the SL line, the detection of the end points of the contour is repeated while increasing the value of Y by dy until Xtsr = Xbr. Assuming that Xtsl = Xbl at the nth time, the coordinate values of the left shoulder point SL are as follows.

[0088]

[Equation 8]

When Xtsl = Xbl, the coordinate values of the left shoulder point SL are as follows.

[0090]

[Equation 9]

The point TSL is the same as the point TSR in FIG.
Is within the left-hand skin color area LSP, the contour is searched while lowering Y from the point SL by dy, and a point PL (Xpl, Ypl) whose end point does not become the left-hand skin color area LSP is detected. Next, the contour of the left-hand skin color region LSP is searched from the point SL while increasing Y by dy, and a point QL (Xql, Yql) that collides with the contour of the body is detected. From the points PL and QL, the coordinates of the right shoulder point SL are as follows (step 202).

[0092]

[Equation 10]

Next, candidates for the right elbow ER (Xer, Yer) and the left elbow EL (Xel, Yel) are detected. The contour of the right arm region is searched in the clockwise direction from the right shoulder SR, and a point whose change amount in the tangential direction is larger than the threshold value DE is detected as a right elbow candidate point. The threshold value DE is a parameter and is adjusted so that the influence of noise is reduced. Assuming that n right elbow candidate points are detected, the right elbow candidate points are Cri (i =
1 ,. ． ． , N).

Next, the left arm contour is searched in the counterclockwise direction from the left shoulder SL, and a point whose change amount in the tangential direction is larger than the threshold value DE is detected as a left elbow candidate point. Assuming that m left elbow candidate points are detected, the candidate points are Clj (j =
1 ,. ． ． , M) (step 203).

If there is a right elbow candidate point, the process for detecting the right elbow proceeds to step 205, and if not, to step 212. Similarly, when the left elbow candidate point exists, the left elbow detection processing proceeds to the next step 205, and when it does not exist, proceeds to step 212 (step 204).

If there are right shoulder candidate points and left shoulder candidate points,
Check whether the shoulder area is within the skin color area of the hand or the center of gravity of the hand is within the arm area. If the position of the right shoulder SR is in the right hand skin color region RSP, or if the right palm center of gravity RC is in the arm region, step 20.
Go to 6. Similarly, the position of the left shoulder SL is the skin color area LS of the left hand.
If it exists in P, or if the left center of gravity LC of the palm exists in the arm region, the process proceeds to step 206 (step 205).

The farthest point from the SR of the right elbow candidate points Cri is the right elbow ER. Similarly, the left elbow candidate point farthest from SL among the left elbow candidate points Clj is set as the left elbow EL (step 206).

If the condition of step 205 is not satisfied,
Narrow down the candidate points. In the right elbow detection process, the right elbow candidate point Cri, which is a point on the right hand skin color region RSP, is deleted from the right elbow candidate points. Similarly,
In the left elbow detection processing, the left elbow candidate point Clb, which is a point on the left hand skin color area LSP, is deleted from the left elbow candidate points (step 207).

Next, a straight line connecting all the right elbow candidate points Cri and the right shoulder SR is searched for, and the right elbow candidate point Crc at which there is a point on the straight line that deviates from the arm region is extracted from the right elbow candidate points Cri. delete. Similarly, a search is performed on a straight line connecting all the left elbow candidate points Clj and the left shoulder SL, and the left elbow candidate point Cld at which a point on the straight line deviates from the arm region is deleted from the left elbow candidate points Clj (step 208).

Next, all right elbow candidate points Cri and right hand center of gravity RC
The right elbow candidate point Cre where there is a point deviating from the arm region is deleted from the right elbow candidate point Cri by searching on the straight line connecting the two.
Similarly, a search is performed on a straight line connecting all the left elbow candidate points Clj and the center of gravity LC of the left hand, and the left elbow candidate points Clf having points out of the arm region are deleted from the left elbow candidate points Clj (step 209). .

By the processing from step 207 to step 209, the candidate points in the middle of the left and right upper arm portions and the left and right lower arm portions, and the candidate points in the hand area are deleted.

If it is checked again whether there are any remaining candidate points for the right elbow, and the right elbow candidate point Cri remains, the process for detecting the right elbow proceeds to step 211. If there is no right elbow candidate point Cri, the procedure proceeds to step 212. move on. Similarly, the left elbow candidate point Cl
When j is left and left, the processing for detecting the left elbow is performed in step 211.
If no left elbow candidate point Clj remains, the process proceeds to step 212 (step 210).

The distance between the straight line connecting the right shoulder SR and the right center of gravity RC of the palm and the remaining right elbow candidate point Cri is examined, and the farthest right elbow candidate point Crα is defined as the right elbow ER. Left shoulder S
The distance between the straight line connecting L and the left palm center of gravity LC and the remaining left elbow candidate point Clj is checked, and the left elbow candidate point Clβ having the longest distance is set as the right elbow EL (step 211).

In step 204 and step 210, the process when no candidate point exists is performed. Right elbow candidate point Cr
If i does not exist, the right shoulder SR and the right center of gravity R
The middle point of C is the right elbow ER. Similarly, the left elbow candidate point Cli
If there is no left shoulder SL and left palm center of gravity LC
Let the middle point be the left elbow EL (step 212).

Right shoulder SR, left shoulder SL, right elbow ER, left elbow EL
Is calculated, the arm shape information ASI is calculated. Arm shape information A
The SI becomes as follows (step 213).

[0106]

[Equation 11]

Next, the processing procedure for one image IMP_i of the spatial position calculation device 32 will be described with reference to the flowchart. FIG. 7 shows a flowchart of the processing procedure of the spatial position calculation device 32.

The spatial position calculation device 32 is previously stored in the spatial position calculation device 32.
It is assumed that the upper arm length SUAL and the lower arm length SLAL are registered. Upper arm manager SUAL, lower arm manager SLAL,
Before performing recognition, the length of the person performing the manual operation is registered. It is also possible to input data of a plurality of persons and register the average values as the upper arm length SUAL and the lower arm length SLAL.

The spatial position measuring device 36 uses the spatial coordinate system Ow.
-XYZ is taken, and the right-hand center-of-gravity space coordinates (Xwr, Ywr, Zwr) and the left-hand center-of-gravity space coordinates (Xwl, Ywl, Zwl) in the space coordinate system Ow-XYZ are calculated. here,
The origin of the spatial coordinate system Ow-XYZ is the point where the perpendicular line extended from the origin of the image coordinate system Og-XY intersects the body of the target person, and the X axis and the Y axis are parallel to the X axis and Y axis of the image coordinate system. For the Z axis, the image plane direction is positive from the target person.

The Xrc of the center of gravity RC of the right palm passed from the arm shape detecting device 31 is directly substituted for Xwr of the space coordinate of the center of gravity of the right hand. The Yrc of the right palm center of gravity RC passed from the arm shape detecting device 31 is directly substituted into Ywr of the right hand center of gravity space coordinates. Similarly, Xlc of the left palm center of gravity LC passed from the arm shape detecting device 31 is directly substituted for Xwl of the space coordinate of the left hand center of gravity. Ywl of left-hand center of gravity space coordinates
Center of gravity LC of the left palm passed from the arm shape detection device 31 to the
Ylc is directly substituted (step 220).

Image IMG_i sent from storage device 32
From this, a ring-shaped index object is detected. As a method of detecting the index object, for example, the color of the index object is set to red and the image IMG
Extract the red region from _i. Each of the extracted regions is thinned and processed to form a single curved line.
The region intersecting with the line segment connecting the right hand center of gravity RC and the right elbow ER is defined as the right lower arm index MRL. An area intersecting with a line segment connecting the right elbow ER and the right shoulder SR is defined as an upper right arm index MRU.

Similarly, the area intersecting with the line segment connecting the center of gravity LC of the left hand and the left elbow EL is defined as the left lower arm index MLL. An area intersecting a line segment connecting the left elbow EL and the left shoulder SL is defined as an upper left arm index MLU. FIG. 27 shows an example of the extracted index object (step 221).

As shown in FIG. 27, if a line segment L1 passing through both ends of the right lower arm index MRL is drawn and L1 intersects with the line segment connecting the midpoint of the right lower arm index MRL and the center of gravity RC of the right hand, The right center of gravity RC exists at a position closer to the image plane IMP than the right elbow ER, and the right lower arm flag FRL is set to 1. If the line segment between the midpoint of the right lower arm index MRL and the right hand center of gravity RC does not intersect with L1 and the line segment connecting the midpoint of the right lower arm index MRL and the right elbow ER intersects L1, the right elbow The ER exists at a position closer to the image plane IMP than the right center of gravity RC of the palm, and the right lower arm flag FRL is set to -1 (step 222).

Similarly, if a line segment L2 passing through both ends of the right upper arm index MRU is drawn and L2 intersects the line segment connecting the midpoint of the right upper arm index MRU and the right elbow ER, the right elbow ER is the right shoulder. SR
The right upper arm flag FRU is set to 1 because it exists at a position closer to the image plane IMP. Midpoint of right upper arm index MRU and right elbow ER
If L2 intersects with the line segment connecting the midpoint of the right upper arm index MRU and the right shoulder SR without intersecting the line segment with L2, the position of the right shoulder SR is closer to the image plane IMP than the right elbow ER. And the right upper arm flag FRU is set to -1 (step 22).
3).

Similarly, a line segment L3 passing through both ends of the left lower arm index MLL is drawn to determine the middle point of the left lower arm index MLL and the center of gravity LC of the left hand.
If the line segment connecting and intersects L1, the left palm center of gravity LC is located closer to the image plane IMP than the left elbow EL, and the left lower arm flag FLL is set to 1. Left lower arm index MLL
A line segment connecting the midpoint of the left lower arm index MLL and the left elbow EL and L3 without the line segment of the midpoint of the left hand center of gravity LC intersecting with L3.
, The left elbow EL is located closer to the image plane IMP than the center of gravity LC of the left palm, and the left lower arm flag FLL
Is set to -1 (step 224).

Similarly, if a line segment L4 passing through both ends of the left upper arm index MLU is drawn and L4 intersects the line segment connecting the midpoint of the left upper arm index MLU and the right elbow EL, the left elbow EL is the left shoulder SL.
Is located closer to the image plane than the left upper arm flag FLU
Is set to 1. If the line segment connecting the middle point of the left upper arm index MLU and the left shoulder SL intersects with the line segment between the middle point of the left upper arm index MLU and the left elbow EL, and L4 intersects, the left shoulder SL is the left elbow. It exists at a position closer to the image plane IMP than EL, and the upper left arm flag FLU is set to -1 (step 225).

The values of the right lower arm flag FRL, the upper right arm flag FRU, the left lower arm flag FLL, the upper left arm flag FLU and the upper right arm R obtained in steps 222 to 225.
FIG. 28 shows the relationship between the states of U, the lower right arm RL, the upper left arm LU, and the lower left arm LL.

The right lower arm index FRL, the right upper arm index FRU obtained in steps 222 and 223, the arm shape information ASI passed from the arm shape detecting device 31, and the upper arm length SUA.
L, lower arm length SLAL, Zwr of right hand center of gravity space coordinates
Is calculated by the following formula.

[0119]

[Equation 12]

Similarly, step 224 and step 225
From the left lower arm index FLL and the left upper arm index FLU, the arm shape information ASI passed from the arm shape detecting device 31, the upper arm length SUAL, and the lower arm length SLAL, Zwl of the left hand center of gravity space is calculated by the following formula: (Step 22)
6).

[0121]

[Equation 13]

Calculated right hand centroid space coordinates (Xwr, Y
wr, Zwr), left-hand space coordinates (Xwl, Ywl, Zw)
l) is recorded in the spatial position storage device (step 22).
7).

(Embodiment 3) The construction of the third embodiment of the present invention will be described with reference to FIG.

The motion recognition device of this embodiment registers a photographing device 3 for constantly photographing body motions, a storage device 4 for storing photographed images for a certain period of time, and a feature serving as a recognition start signal as a start code. A start code registration device 44, a feature code creation device 42 that encodes a feature in an image stored in the storage device 4, a recognition start code determination device 43 that determines whether the start code and the feature code match, the code matches The recognition start notification device 45 is configured to notify the photographing device 3 of the recognition start timing in response to the determination result.

Next, the operation of a specific embodiment of this embodiment will be described with reference to FIG. Hereinafter, the movement of the body and the change of the shape will be referred to as the body movement, and the movement of the body and the change of the shape serving as a signal to start the recognition will be referred to as the recognition start body movement.

First, a recognition starter motion for giving an accurate timing of motion recognition start is defined by one element or a combination of a plurality of elements. The elements include spatial position, body shape, and movement. Also, body movements are
It can be set for each part of the body such as eyes, mouth, and whole body.

Here, an operation in which the state where the mouth is wide open continues for 2 seconds (60 frames) or more is defined as a recognition starter operation. The start of image recognition at this time is 4 seconds (120 frames) after the first frame of the starter motion.

This recognition starter motion is registered as a code in the start code registration device 44. The description is (mouth,
It looks like a circle, 60). “Mouth” is a place of interest to judge by the area ratio with other color components by paying attention to the red color component, and “Circle” is calculated from the aspect ratio of the mouth part etc. when the mouth is open. It is a shape, and "60" represents the minimum required number of consecutive frames.

Now, it is assumed that the photographing apparatus 3 continues to photograph meaningless images. The meaningless here means that it is neither a recognition start body movement, a sign language, nor a meaningful body movement (step 300).

The photographed image is recorded in the storage device 4 for a certain period of time. Here, 60 frames or more can be stored at one time (step 301).

The image entered in the storage device 4 is coded by the body motion code creating device 42 element by element. Then, it is assumed that the shape of the attention place "mouth" is "circle" and it is coded that it continues for "60" frames or more (step 30).
2).

This result and the code registered in the start code registration device 44 are input to the recognition start code determination device 43, where it is determined whether or not they are the same. Here, as shown in FIG. 29, the codes are collated with each other, and it is determined that the motion is a recognition starter motion (step 303).

When it is determined that the motion is a recognition start body, the recognition start notifying device 45 gives the photographing device 3 an appropriate timing for starting the motion recognition. Here, in FIG.
As shown in FIG. 9, 1 from the first frame of the recognition starter motion
It is 20 frames later (step 304).

In the case of the judgment result that the codes cannot be collated and it is not the recognition starter motion, the recognition start notification device 45
Notifies the image capturing device 3 to capture an image for detecting the recognition start body motion from the next frame, as shown in FIG. In this way, the detection operation is repeated until the recognition starter motion is detected (step 30).
5).

(Embodiment 4) The construction of a fourth embodiment of the present invention will be described with reference to FIG.

A photographing device 3 for constantly photographing body movements, a storage device 4 for storing the photographed images for a certain time, a start image registration device 48 for registering an image as a recognition start signal as a start image, and a storage device. 4, an image processing device 46 that performs preprocessing on the image stored in 4, a recognition start image determination device 47 that determines whether the preprocessed image matches the start image,
The recognition start notification device 45 notifies the photographing device 3 of the recognition start timing in response to the determination result of whether the images match.

Next, with reference to FIG. 9, the operation of a specific embodiment of this embodiment will be described. Hereinafter, the movement of the body and the change of the shape will be referred to as the body movement, and the movement of the body and the change of the shape serving as a signal to start the recognition will be referred to as the recognition start body movement.

First, a recognition starter motion for giving an accurate timing of motion recognition start is defined by one element or a combination of a plurality of elements. The elements include spatial position, body shape, and movement. In addition, body movements are
It can be set for each part of the body such as eyes, mouth, and whole body.

Here, a motion in which the mouth is wide open continues for 2 seconds (60 frames) or more is defined as a recognition starter motion. The start timing of the motion recognition at this time is 4 seconds (120 frames) after the first frame of the recognition starter motion.

The image of the recognition start body motion is registered in the start image registration device 48. If you open your mouth wide,
It should contain a red circular image. Therefore, the image in which only the red component is extracted is registered.

Now, it is assumed that the photographing device 3 continues to photograph meaningless images. The meaningless here means that it is neither a recognition start body movement, a sign language, nor a meaningful body movement (step 300).

The photographed image is recorded in the storage device 4 for a certain period of time. Here, 60 frames or more can be stored at one time (step 301).

The image processing device 46 performs image processing for extracting only the red component from the input image so that the image stored in the storage device 4 can be collated with the registered image. The image processing device 46 is not limited to the process for extracting the red component, and can perform various image processes (step 306).

This result and the image registered in the start image registration device 48 are input to the recognition start image determination device 47, and as shown in FIG. 32, the images are subjected to pattern matching and collation to see if they are the same. With the determination,
If the determination that they are the same, such as 0, continues for 60 frames or more, it is determined that the recognition starter motion.
The same effect can be obtained by pattern matching using a neural network (step 307).

When it is determined that the input image is a recognition start body motion, the recognition start notification device 45 gives the photographing device 3 an appropriate timing for starting motion recognition. Here, as shown in FIG. 30, 120 frames after the first frame of the recognition starter motion (step 30).
4).

In the case of the judgment result that the images cannot be collated with each other and it is not the recognition starter motion, the recognition start notification device 45
Notifies the image capturing device 3 to capture an image for detecting the recognition start body motion from the next frame, as shown in FIG. Like this, like this,
The operation is repeated until the recognition start body motion is detected (step 305).

[0147]

EFFECTS OF THE INVENTION It is possible to recognize a hand gesture or a gesture without using a sensor such as a data glove or a colored glove.

Further, the recognition speed is improved by grouping the hand movements. Further, even when photographing with a single eye, it is possible to grasp the depth by detecting the shape of the arm, and it is possible to recognize a hand gesture or a gesture like the compound eye type.

Further, it is possible to give an accurate timing of motion recognition by detecting a signal to start recognition during constant monitoring.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a motion recognition device according to a first embodiment of the present invention.

FIG. 2 is a configuration diagram up to a spatial position calculation device in a motion recognition device according to a second embodiment of the present invention.

FIG. 3 is a configuration diagram of a part relating to detection and notification of a start-up body motion in the motion recognition device according to the third embodiment of the present invention.

FIG. 4 is a configuration diagram of a part related to detection and notification of a start-up body motion in the motion recognition device according to the fourth embodiment of the present invention.

FIG. 5 is a flowchart of a processing procedure of the motion recognition apparatus according to the first embodiment.

FIG. 6 is a flowchart of a processing procedure of the arm shape detecting device according to the second embodiment.

FIG. 7 is a flowchart of a processing procedure of the spatial position calculation device according to the second embodiment.

FIG. 8 is a flow chart of a procedure of a part relating to detection and notification of a capturing start body motion in the motion recognition apparatus of the third embodiment.

FIG. 9 is a flow chart of a procedure of a part relating to detection and notification of a movement of a capturing start body in the movement recognition apparatus according to the fourth embodiment.

FIG. 10 is a diagram showing a state in which the storage devices 4 are arranged in time series in the first embodiment.

FIG. 11 is a diagram showing how the right palm is projected on the left and right image planes when the right palm is photographed by the left and right cameras in the first embodiment.

FIG. 12 is a diagram showing the positions of the face, the right palm, and the left palm in the absolute space, which are stored in the spatial position storage device 7 in the first embodiment.

FIG. 13 is a diagram showing the shape of the right palm extracted by the skin color detecting device 5 in the first embodiment.

FIG. 14 is a diagram in which the smallest square surrounding the direction of the right palm and the palm of the first embodiment is created.

FIG. 15 is a diagram obtained by dividing the square SQ into four equal parts in the first embodiment.

16A is a diagram in which the hierarchy of FIG. 15 is made deeper by one and the square SQ is divided into 16 equal parts, and FIG. 16B is a diagram in which the region coded in FIG. 17A is hatched.

17A is a diagram in which the hierarchy of FIG. 16 is deepened by one, and the square SQ is divided into 64 equal parts. FIG. 17B is a diagram in which the region coded in FIG. 17A is hatched.

FIG. 18 is a diagram obtained by encoding the right palm in the first embodiment.

FIG. 19 is a diagram in which the hand shapes arranged in time series in FIG. 13 are encoded.

FIG. 20 is a diagram showing a search tree for specifying a group in the first embodiment.

FIG. 21 is a diagram showing a moving direction in the first embodiment.

22A is a diagram showing relative movement in the first embodiment, and FIG. 22B is a diagram showing motion codes in the first embodiment.

FIG. 23 is a diagram showing a start code in the first embodiment.

FIG. 24 is a motion word storage device 22 according to the first embodiment.
Format diagram of files stored in

FIG. 25 is a diagram illustrating arm shape information detected by the arm shape detecting device according to the second embodiment.

FIG. 26 is a diagram of left and right body sides, left and right shoulders, and left and right elbows detected by the arm shape detecting device according to the second embodiment.

FIG. 27 is a detection diagram of an index object detected by the spatial position calculation device according to the second embodiment.

FIG. 28 is a diagram showing the relationship between the state of the arm and the values of the upper right arm flag, the lower right arm flag, the upper left arm flag, and the lower left arm flag.

FIG. 29 is an explanatory diagram of a part related to detection and notification of a start-up body motion in the motion recognition apparatus according to the third embodiment.

FIG. 30 is an explanatory diagram of a part related to detection and notification of a start-up body motion in the motion recognition apparatus according to the fourth embodiment.

FIG. 31 is an explanatory diagram of a part related to detection and notification of a start-up body motion in the motion recognition devices according to the third and fourth embodiments.

FIG. 32 is an explanatory diagram of a part relating to detection of a start-up body motion in the motion recognition apparatus according to the fourth embodiment.

[Explanation of symbols]

S1 Right palm extraction section S2 Left palm extraction section S3 Face area extraction section S4, S5 Skin noise part noise area P1 Right palm center of gravity position P2 Left palm center of gravity position P3 Face area center of gravity position IMG_1 to IMG_N Storage device 4 Image P1_left Coordinates on the image plane of the center of gravity of the right palm taken by the left camera P1_right Coordinates on the image plane of the center of gravity of the right palm taken by the right camera L Left camera position R Right camera position l Camera Inter-distance f Focal length e1 End point VEC Palm orientation vector SQ Minimum area square covering all palms SQ_0 to SQ_47 Area in which square SQ is divided a to x Hand movement group change Motion change point IMG_i Processing in time series image Image FSP Face skin color area RSP Right hand skin color area LSP Hand skin color area FC Face center of gravity RC Right center of gravity center of gravity LC Left side center of gravity SR Right shoulder SL Left shoulder ER Right elbow EL Left elbow ASI Arm shape information Xwr Right hand center of gravity space coordinate X coordinate Ywr Right hand center of gravity space coordinate Ywr Zwr Right hand center of gravity space Z coordinate of coordinates Xwl X coordinate of left hand center of gravity space coordinates Ywr Y coordinate of left hand center of gravity space coordinates Zwr Z coordinate of left hand center of gravity space coordinates HM Target person performing hand movement Og-XY image coordinate system IMP Image plane of imaging device 30 WIDETH Horizontal width of image plane IMP HEIGHT Vertical height of image plane IMP RU Projected image of right upper arm on image plane IMP RUdx Width of projected image of right upper arm on image plane IMP in X-axis direction RUdy Height in the Y-axis direction of the projection image of the upper right arm on the image plane IMP RL right Projection image of the lower arm on the image plane IMP RLdx right Width of the projection image of the lower arm on the image plane IMP in the X-axis direction RLdy Height of the projection image of the right lower arm on the image plane IMP in the Y-axis direction LU LU Projection image of the upper left arm on the image plane IMP LUdx Width of the projected image of the upper left arm on the image plane IMP in the X-axis direction LUdy Height of the projected image of the upper left arm on the image plane IMP in the Y-axis direction LL Left projected image of the lower arm on the image plane IMP LLdx Lower left Width of the projected image of the arm on the image plane IMP in the X-axis direction LLdy Height of the projected image of the left lower arm on the image plane IMP in the Y-axis direction BGIMG Background image not including the target person HM IPL A straight line parallel to the X-axis passing BR body right end BL body left end Xbr body right end BR image coordinate system Og-XY X coordinate Xbl body left end BL image coordinate system Og-XY X coordinate Xf face center of gravity FC image coordinate system Og -In XY X coordinate Yf Y coordinate in the image coordinate system Og-XY of the face center of gravity FC Xrc Right image of the center of gravity RC of the center of gravity RC X coordinate in the image coordinate system Og-XY Yrc Right of the center of gravity of the center of gravity Y of the RC image coordinate system Og-XY Ylc Left of the center of gravity of the palm X coordinate in the image coordinate system Og-XY of LC Ylc Y coordinate in the image coordinate system Og-XY of the left palm center of gravity LC Xsr X coordinate in the image coordinate system Og-XY of the right shoulder SR Ysr Image coordinate system Og- of the right shoulder SR Y coordinate in XY Xsl X coordinate in the image coordinate system Og-XY of the left shoulder SL Ysl Y coordinate in the image coordinate system Og-XY of the left shoulder SL SDSF The face center of gravity FC registered in the spatial position measuring device 32 and the height of the shoulder Difference of Y coordinate SC Point vertically below the distance SDSF from the center of gravity of the face TSR Candidate point of right shoulder Xtsr X in the image coordinate system Og-XY of the point TSR
Coordinate Ytsr Point TSR image coordinate system Og-XY Y
Coordinates dy Parameter of search width in Y direction when searching for shoulder PR Point where right edge of right-hand skin color region RSP intersects shoulder line Xpr point PR image coordinate system Og-XY X coordinate Ypr point PR image coordinate system Y coordinate in Og-XY QR Point where the left end of the right hand skin color region RSP intersects with the shoulder line Xqr X coordinate in the image coordinate system of point QR Og-XY X coordinate Yqr Y coordinate in point QR image coordinate system Og-XY TSL Left shoulder Candidate point Xtsl X in the image coordinate system Og-XY of the point TSL
Coordinates Ytsl Y in the image coordinate system Og-XY of the point TSL
Coordinates PL A point where the left end of the left hand skin color area LSP intersects the shoulder line X coordinate in the image coordinate system Og-XY of the Xpl point PL Y coordinate in the image coordinate system Og-XY of the Ypl point PL QL The right end of the left hand skin color area LSP Point where the shoulder line intersects Xql X coordinate in the image coordinate system Og-XY of the point QL Yql Y coordinate in the image coordinate system Og-XY of the point QL Xer right elbow X coordinate in the image coordinate system Og-XY of the ER Yer right elbow Y coordinate in the image coordinate system Og-XY of the ER Xel X coordinate in the image coordinate system Og-XY of the left elbow EL Yel Y coordinate in the image coordinate system Og-XY of the left elbow EL DE The threshold value for the amount of change in the tangent line during the search Parameter Cri Right elbow candidate point Clj Left elbow candidate point Cra Right elbow candidate point Cri Point in the right hand skin color area RSP Clb Left elbow Point in the left-hand skin color area LSP of the point Clj Crc Point in the inner right lower arm RL of the right elbow candidate point Cri Cld Point in the middle left inner arm LL of the left elbow candidate point Clj Cre Right elbow A point in the middle of the upper right arm RU of the candidate point Cri Clf A point in the middle of the left upper arm LU of the left elbow candidate point Clj Crα A strongest candidate of the right elbow ER Clβ A strongest candidate of the left elbow EL SUAL spatial position Upper arm length pre-registered in the computing device 32 SLAL Spatial position pre-registered in the computing device 32 Ow-XYZ spatial coordinate system MRL Right lower arm attached to the right lower arm Index MRU Right upper arm index attached to the upper right arm MLL Left lower arm index attached to the left lower arm MLU Left upper arm index attached to the upper left arm L1 Right lower arm index Line segment connecting both ends of the mark MRL L2 Upper right Line segment connecting both ends of the index MRU L3 Line segment connecting both ends of the left lower arm index MLL L4 Line segment connecting both ends of the upper left arm index MLL FRL Right lower arm flag FRU Upper right arm flag FLL Left lower arm flag FLU Upper left arm flag 1, TV camera 3 Imaging device 4 Storage device 5 Skin color detection device 6 Center of gravity detection device 7 Spatial position calculation device 8 Spatial position storage device 9 Palm specification device 10 Hand movement / motion determination device 11 Finger shape code generation device 12 shape change determination device 13 group identification device 14 group storage device 15 change point detection device 16 motion code creation device 17 start position code creation device 18 finger shape verification device 19 finger shape storage device 20 hand operation word verification device 22 hand operation word storage Device 30 Imaging device 31 Arm shape detection device 32 Spatial position calculation device 42 Feature code creation Device 43 recognizes the start code judging device 44 starts code registration device 45 recognizes start notification device 46 the image processing apparatus 47 recognizes the starting image determination apparatus 48 starts image registration device

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical display location G09B 21/00 E 9061-5L G06F 15/70 410

Claims (5)

[Claims] 1. A first camera for photographing at least the upper half of the body of an operator.
And a second photographing unit, a storage unit that stores the image photographed by each of the photographing units, a portion from the image stored in the storage unit to the right wrist (hereinafter referred to as the right palm), A skin-colored area (hereinafter, referred to as the left palm) from the left wrist (hereinafter, referred to as the left palm)
(Referred to as a skin color area), a center of gravity detecting means for obtaining the center of gravity of the area of the skin color area extracted by the skin color area extracting means, and the first photographing means for the skin color area of the same body part. First spatial position calculating means for obtaining the spatial position of the skin color region based on the parallax between the center of gravity obtained from the image photographed by the above and the center of gravity obtained from the image photographed by the second photographing means. And a spatial position storage unit that stores the spatial position obtained by the first spatial position calculation unit, and is represented by a skin color region based on a temporal change of the spatial position stored in the spatial position storage unit. A motion recognition device comprising: a motion recognition unit configured to recognize a motion of a body part. 2. A third method for photographing at least the upper body of the operator.
Image capturing means, storage means for storing the image captured by the third image capturing means, and skin color area extracting means for extracting skin color areas of the right palm, the left palm and the face from the images stored in the storage means. And a center of gravity detecting means for obtaining the center of gravity of the area of the skin color area extracted by the skin color area extracting means, and the center of gravity of the right hand palm, the center of gravity of the left palm and the face center of gravity obtained by the center of gravity detection means on the screen. Arm shape extracting means for extracting shape information of the upper right arm portion, the lower right arm portion, the upper left arm portion and the lower left arm portion from the position; the shape information extracted by the arm shape extracting means; and the upper right arm portion, right A second one for obtaining the spatial positions of the right palm and the left palm based on the shapes on the screen of the ring-shaped indicators of the lower arm portion, the upper left arm portion, and the left lower arm portion, which are attached to the respective predetermined portions. Spatial position calculation means A spatial position storage unit that stores the spatial position obtained by the second spatial position calculation unit; and a body represented by a skin color region based on a temporal change of the spatial position stored in the spatial position storage unit. A motion recognition device including a motion recognition unit configured to recognize a motion of a part. 3. A plurality of hands defined by at least a palm used, a motion of the palm being used in a space of the palm being used, a shape change of the palm being used, or a combination thereof. For recognizing an action word, based on the spatial position stored in the spatial position storage means, a palm determination means for determining the palm used, and a palm color extraction means for the palm used. Finger shape code creating means for converting the shape of the extracted skin color area into a finger shape code, and a hand operation for determining whether the palm used is still or moving in space. Static movement determination means, shape change determination means for determining whether or not there is a temporal change in the finger shape code created by the finger shape code storage means, and the plurality of hands Grouping means for grouping composition words based on a combination of the palm to be used, the movement of the palm to be used in the space of the palm, and the shape change of the palm to be used, the palm being used, and the palm being used. Based on the movement of the palm that is present and the shape change of the palm that is being used, a group identification unit that identifies which of the groups the input hand movement word belongs to, and the palm that is being used, Based on the temporal change of the spatial position stored in the spatial position storage means,
Based on the change point detecting means for detecting the change point of the movement, the spatial position stored in the spatial position storage means, and the change point of the movement detected by the change point detecting means, the palm used. Motion code creating means for creating a motion code corresponding to the combination of the motions, and start position code creating means for creating a start position code corresponding to the start position of the operation from the spatial position stored in the spatial position storing means. Corresponding to the shape of the palm defining the hand action word,
The finger shape storage unit storing a plurality of registered finger shape codes, and the finger shape code created by the finger shape code creating unit corresponds to any of the plurality of registered finger shape codes stored in the finger shape storage unit. Finger shape matching means for specifying whether to do, for each group grouped by the grouping means,
A hand action word storage unit that stores a combination of a start position of an action, a motion code, and a shape that defines the hand action word, a start position code created by the start position code creating unit, and the motion code creating unit. A plurality of hands stored in the hand motion word storage means corresponding to the group obtained by the group identification means, based on the motion code created by the above and the registered finger shape code identified by the finger shape matching means. 3. The motion recognition device according to claim 1, further comprising a hand motion word collating unit that specifies the input hand motion word from the motion word. 4. A start code registering means for holding, as a start code, a preselected feature, which is a cue for the start of motion recognition, among the predetermined features of the body part of interest, and an image stored in the storage means. A feature code creating unit that extracts a feature of the body part of interest and creates a feature code corresponding to the feature is compared with the feature code and the start code to determine whether the two match. A recognition start code determining means; and a recognition start notifying means for generating a recognition start signal when the recognition start code determining means determines "match". The motion recognition device according to any one of claims 1 to 3, wherein the motion recognition device is configured to perform motion recognition using an image after a predetermined time has elapsed after receiving the message. 5. A start image registration means for holding, as a start image, a preselected image serving as a signal to start motion recognition, an image processing means for pre-processing the image stored in the storage means, When the pre-processed image and the start image are compared, and the recognition start image determination unit that determines whether the two match, and the recognition start image determination unit determines that they are “matched” And a recognition start notification means for generating a recognition start signal, wherein the motion recognition unit is configured to perform motion recognition using an image after a predetermined time has elapsed since the recognition start signal was received. The motion recognition device according to any one of 3 to 3.