JP2003196655A - Eye image detection device - Google Patents

Abstract

(57) [Problem] To provide an eye image detecting device capable of detecting the position of a human eye by simple processing. An image pickup means and a first extraction means for directly extracting a first eye image candidate from image data by scanning a template for extracting an eye image on image data obtained from the image pickup means. Second extraction means for extracting a second eye image candidate within a predetermined range from the first eye candidate, and a second eye image candidate being paired with the first eye image candidate in one face. Determining means for determining whether or not the image is a so-called eye image;
The second eye image candidate is searched for within a predetermined range with respect to the first eye image candidate, and whether the first eye image candidate and the second eye image candidate have a paired relationship in a human face By determining whether or not the first eye image candidate and the second eye image candidate correspond to a pair of eyes.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye image detecting device for detecting an image of human eyes from image data obtained by an image pickup means, and further to various devices using the eye image detecting device. It is a thing.

[0002]

2. Description of the Related Art As a method for automatically extracting an image of human eyes from a digital image, there is one disclosed in Japanese Patent Application Laid-Open No. 11-149559. This prior art is premised on a color image. First, after extracting a skin color area from the color image, a face area that is considered to correspond to a face is detected from the shape of the skin area. Then, the eyes are detected from the face area by template matching. At that time, the orientation and size of the template are changed according to the orientation and size of the face extracted as the face area. Then, an eye image is extracted by performing a pairing process on the eye candidates in the face area according to the distance and angle conditions.

[0003]

According to the above-mentioned conventional technique, there is a problem that a color image is required and cannot be applied to a monochrome image.

Further, since the eyes are detected after the skin detection and the face detection, there is a problem that the number of processing steps is large and the calculation time is prolonged.

[0005]

The eye image detecting apparatus of the present invention has been made to solve such a problem.

The eye image detecting apparatus of the present invention comprises an image pickup means,
First extraction means for extracting first eye image candidates directly from the image data by scanning a template for extracting eye images on the image data obtained from the imaging means, and from the first eye candidates Second extraction means for extracting second eye image candidates within a predetermined range, and whether or not the second eye image candidates are eye images forming a pair with the first eye image candidates in one face And a determination means for determining.

According to this eye image detecting device, the second eye image candidates are searched for within the predetermined range with respect to the first eye image candidates directly extracted from the image data, and the first eye image candidates are searched. By determining whether or not the second eye image candidate and the second eye image candidate have a paired relationship in the human face, the first and second eye image candidates are obtained.
It is determined that the eye image candidate of 1 corresponds to a pair of eyes.

Therefore, it is possible to detect a pair of eyes existing on the same face without extracting the skin area and the face area as in the prior art. Therefore, it is not necessary to make a color image essential, and the arithmetic processing can be simplified.

Further, the first extracted directly from the image data
If the second eye image candidate is not an eye image, the probability that the second eye image candidate paired with the first eye image candidate in one face is extremely low. Even if the eye image candidates are extracted, there is substantially no false detection.

In the eye image detecting apparatus of the present invention, it is desirable that a plurality of different templates are prepared according to the inclination of the face to the left and right. By doing so, even if the face of the person facing the image pickup means is tilted, the eyes of the person can be reliably detected.

In the eye image detecting apparatus of the present invention, it is desirable that the first and second extracting means extract eye images at different distances from the image pickup means by enlarging or reducing the image data. The eye images at different distances from the image pickup unit can be extracted without preparing a large number of templates according to the distance from the image pickup unit.

In the eye image detecting apparatus of the present invention, the template has a first luminance evaluation area corresponding to the pupil and its vicinity, and a second luminance evaluation area surrounding the first luminance evaluation area. It is preferable that the extraction means of (3) extract the eye image based on the average luminance and the luminance dispersion of each of the first luminance evaluation area and the second luminance evaluation area.

The average luminance of the first luminance evaluation area corresponding to the central portion of the eye and the average luminance of the second luminance evaluation area corresponding to the outer peripheral portion thereof are not used to determine whether or not the image is an eye image, and each luminance evaluation area Since the luminance dispersion information of is also used together, the eye image can be accurately detected even if the average luminance is different due to the difference in lighting environment.

In the eye image detecting device of the present invention, the template obtains the variance of the eye image sample for each pixel in advance within a predetermined area centered on the pupil for a plurality of eye image samples,
Each pixel is classified into a plurality of groups according to the calculated variance value, and the first extraction means extracts the eye image based on the average brightness and the brightness dispersion of each group. desirable.

Each group is characterized by a variance value of a plurality of eye image samples, and the average luminance and the luminance variance of pixels belonging to each group are compared, for example, between the groups to determine the influence of the average luminance of the entire image data. The eye image can be accurately detected without receiving the eye image.

The eye image detecting apparatus of the present invention further comprises distance calculating means and a plurality of image pickup means,
The extraction means, the second extraction means, and the determination means detect the eye image for each image data by processing the respective image data obtained by the plurality of image pickup means, and the distance calculation means It is preferable that the distance from the image pickup means to the eye is calculated based on the parallax by comparing the positions of the eye images of the respective image data.

By collating the image data obtained by different image pickup means, the distance can be calculated with high accuracy based on the parallax. In addition, collation of image data is also effective in reducing erroneous detection.

The motion control apparatus of the present invention controls the operation of a device for providing a predetermined action to a person, and the eye image detecting apparatus of the present invention and the eye detected by the eye image detecting apparatus. And a control means for controlling the device so that the action matches a person having eyes corresponding to the image.

By detecting the eye image, it is possible to recognize the person whose face is facing the image pickup means, so that the device can be controlled so as to be adapted to the selectively recognized person. For example, devices such as a display device, an audio device, an air conditioner, auto tennis, and a rearview mirror of an automobile can be controlled according to the position and posture (face inclination) of the user.

In the operation control device of the present invention, the control means is
It is desirable to detect the corresponding movement of the eye from the change in the eye image detected by the eye image detection device and control the action performed by the device in consideration of the movement direction of the eye.

According to such a configuration, it is possible to selectively extract a person who approaches the image pickup means and approach the device, so that, for example, a sidewalk that moves as a device, an approach warning device, or an automatic door. You can move things etc. appropriately according to the approaching person.

The motion control device of the present invention further comprises facial expression recognition means for recognizing the facial expression of a person having eyes corresponding to the eye image detected by the eye image detection device, and the control means comprises:
It is desirable to change the operation speed of the device according to the facial expression recognized by the facial expression recognition means.

Assuming that the device is a moving walkway or an automatic door, and when the facial expression recognition means recognizes the facial expression of a person coming in a hurry, the moving walkway speed is increased,
The automatic door opening speed can be increased to match the movement of the device to the user's intention.

The viewing inspection apparatus of the present invention includes a video display section for displaying a video, and an eye image detection apparatus of the present invention arranged so that the imaging means captures an area facing the video display section.
It is characterized by further comprising: a person number counting means for counting the number of persons who have their faces facing the video display section based on the eye image detected by the eye image detecting device.

The audience rating can be automatically investigated by selectively extracting only those who are looking at the video display unit from the people near the video display unit and counting the number of the people. .

The automatic ticket gate control device of the present invention is a control device for controlling the passage direction of a plurality of ticket gate passages whose passage directions can be switched, and the eye image detection of the present invention in which the image pickup means is arranged in the ticket gate passage. A device and a user detection unit that detects the number and distance of persons approaching each ticket passage based on the eye image detected by the eye image detection device, and each ticket passage detected by the user detection unit. It is characterized by comprising a traffic direction determining means for determining the traffic direction of each ticket gate passage based on the number of people approaching and the distance.

In an automatic ticket gate having a plurality of ticket passages, the number and distance of people approaching each ticket passage are compared with each other, so that the passage direction of the ticket passage can be efficiently determined so as not to cause congestion in the ticket passage. I can decide.

[0028]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a diagram showing the configuration of an eye image detecting apparatus which is an embodiment of the present invention. The first camera 1 and the second camera 2, which are image pickup means, have optical axes of optical systems that are parallel to each other and are oriented in the same direction.
They are arranged side by side with a distance of d. The image data obtained by the first camera 1 and the second camera 2 are sent to the image processing unit 3, and the image processing unit 3 detects the eye image on the image data.

The image processing section 3 includes an image reducing means 4, a first extracting means 5, a second extracting means 6, a judging means 7, and a control means 8 for controlling the operation timing and operation conditions of each of these means. I have it.

The image reducing means 4 halves the image of the image data obtained from the first camera 1 or the second camera 2,
The first extracting means 5 is selectively reduced to 1/3 times.
Output to. The instruction of the reduction rate is given from the control means 8 and includes the case of 1 time, that is, the case where the reduction processing is not performed.

The first extracting means 5 extracts an image of a person's eye on the non-reduced image data or the reduced image data from the image reducing means 4, and the extracted eye image is a first image. We will call them eye image candidates. In order to extract the eye image, the first extracting means 5 scans the template on the image data and directly extracts eye image candidates from the image data.

The template is for extracting an eye image on the image data of a person at a distance of N meters from the first camera 1 or the second camera 2 on the premise of a non-reduced image. It is prepared. here,
Seven types of templates having inclinations of 0 degree, ± 10 degrees, ± 20 degrees, and ± 30 degrees are prepared, and scanning of image data is performed using each template.

FIG. 2 shows an example of a template having an inclination of 0 degree. The template 10 includes a horizontally elongated elliptical first luminance evaluation region (region A) corresponding to the pupil and its neighboring region, and a horizontally elongated elliptical second luminance evaluation region corresponding to a region further outside the first luminance evaluation region. (Region B). As illustrated, the horizontal length of the first luminance evaluation area corresponds to 11 pixels, and the vertical length corresponds to 3 pixels. The horizontal length of the second luminance evaluation area corresponds to 15 pixels, and the vertical length corresponds to 9 pixels.

The first extracting means 5 is the template 1
0 fitting the image data, the luminance of each pixel in the first luminance evaluation area to calculate an average mu _A and variance sigma _A, further dispersing the mean mu _B luminance of each pixel in the second luminance evaluation region sigma Calculate _B.

Then, μ _A , σ _A , μ _B , and σ _B are substituted into the right side of the following evaluation function formula (1) to obtain the value of the evaluation function E.

[0036]

[Equation 1] Where μ _EYE is the average luminance of the eye based on statistical data.

If the value of the evaluation function E is larger than the preset reference value, it is determined to be an eye image candidate.

Qualitatively explaining this equation (1), the coefficient
Part of (μ_A-Μ_B)²/ (Σ_A ²+ Σ_B ²) As you can see
The average brightness of the first brightness evaluation area and the average brightness of the second brightness evaluation area
The larger the difference in average luminance, and the first luminance evaluation area
And the respective luminance distributions of the second luminance evaluation area are small.
It becomes a big value. That is, the first brightness evaluation area and the second brightness evaluation area
The brightness evaluation area can be clearly distinguished from the brightness evaluation area.
When the value becomes large, it is determined that the eye image candidate.

Considering the exponential function part, the luminance dispersion of the first luminance evaluation area is small and the average luminance becomes a smaller value as it deviates from the statistical average luminance. Therefore, if the luminance variance is very small and the average luminance is significantly different from the statistical average luminance, even if the first luminance evaluation area can be clearly distinguished from the second luminance evaluation area. , It may be determined that it is not an eye image candidate.

The second extracting means 6 extracts eye image candidates within a predetermined range on the image data from the first eye image candidates obtained by the first extracting means 5 as second eye image candidates. .

FIG. 3 shows the eye image candidate extraction range of the second extraction means 6, which is an angle range of 60 degrees from the center in the left-right direction of the first eye image candidate.
The extraction ranges 11 and 12 are up to a distance of 2 to 3 times the lateral width of the template 10 used when extracting the eye image candidates. In addition, it is desirable to appropriately set the extraction range in consideration of the target detection accuracy and the like.

As the eye image candidate extracting method in the second extracting means 6, the same extracting method as that used in the first extracting means 5 is used. However, since the extraction region is limited, the probability of false detection is considered to be low, and thus the processing speed may be increased by using a method with low detection power. On the contrary, in the first extracting means 5, the processing speed may be increased by using the extracting method with low detecting power, and in the second extracting means 6, the extracting method with high detecting power may be applied.

When the image is reduced by 1/2 in the image reducing means 4, the first extracting means 5 and the second extracting means 6 apply the same template as in the non-reduced image to the reduced image. By doing so, it is possible to extract eye image candidates at a distance of N / 2 meters from the first camera 1 and the second camera 2. Similarly, when the image reducing means 4 reduces the image size to 1/3, it is possible to extract eye image candidates located at a distance near N / 3 meters.

The determining means 7 determines whether or not the second eye image candidate is an image of an eye that makes a pair with the first eye image candidate on the face of one human based on the positional relationship between the two. .
That is, if the predetermined condition is satisfied, it is determined that the first and second eye image candidates are human eyes. The determination of the human eye by the determination means 7 is the detection of the eye by the true eye image detection device.

If the image reduction means 4 has not performed the reduction processing, it means that the eyes of a person located near N meters in front of the cameras 1 and 2 have been detected. When the image is reduced by 1/2 or 1/3 by the image reducing means 4, the eyes of a person located near N / 2 meters or N / 3 meters in front of the cameras 1 and 2 are detected. It has been done.

The judging means 7 also makes a strict distance judgment if necessary. The image obtained by the first camera 1 and the second camera 2
It is possible to calculate the accurate distance to the detected eye by comparing the obtained image with the image obtained in (3) based on the parallax between the two cameras. The distance to the eyes is obtained from the inter-camera distance d and the position difference (parallax) on the screen of each eye image estimated to be the same eye in the image of the first camera and the image of the second camera. In other words, the first as seen from the target eye
The angle θ formed by the camera and the second camera is calculated from the parallax on the image, and the angle D is substituted into D = d / tan θ to obtain the distance D to the eye. Here, d is the distance between the cameras.

In FIG. 1, the image processing unit 3 controls four functional blocks and control means for controlling them, that is, the image reducing means 4, the first extracting means 5, the second extracting means 6, the judging means 7 and the control. Although it is configured by the means 7, one processing means (microcomputer) may execute each function.

FIG. 4 is a flowchart showing the operation when the function of the image processing unit 3 is executed by the microcomputer.

First, in step S1, image data of a still image at approximately the same time is fetched from each of the first camera 1 and the second camera 2.

In step S2, N is applied to the non-reduced image.
Prepare multiple eye templates for meters according to the inclination of the eyes. Specifically, the inclination is 0 degree, ± 10 degrees, ±
Seven types of templates of 20 degrees and ± 30 degrees are prepared.

In step S3, first, the image data (first image data) from the first camera 1 is scanned with the above-mentioned seven types of templates to extract eye image candidates. The eye image candidate extracted in this step is called a first eye image candidate. Scanning is interrupted at the time of extraction and step S4
Move to. At this time, if no eye image candidate is found even if the full screen scanning is performed, the process proceeds to step S9.

The extraction of the first eye image candidate is performed by the above-mentioned first
The same method as the method carried out in the extraction means 5 is used. That is, the center of the template is aligned with the target pixel, and the average and variance of the brightness data of the pixels overlapping the first brightness evaluation area and the second brightness evaluation area of the template are calculated and substituted into the right side of the equation (1). If the value of the evaluation function E obtained as a result is a value equal to or larger than the set value, it is extracted as an eye image candidate.

In step S4, the extraction area 1 shown in FIG.
Eye image candidates are extracted by limiting to 1 and 12. The eye image candidates here are called second eye image candidates. The extraction method is the first
It does not have to be the same as the extraction of eye image candidates, but the same method is used here.

When the second eye image candidate cannot be extracted, the process returns to step S3 to restart the interrupted extraction scan of the first eye image candidate.

When the second eye image candidate has been extracted, the process proceeds to step S5, and it is determined whether the first eye image candidate and the second eye image candidate are eye images forming a pair in one human face. Is determined based on the positional relationship between the two. If it is determined that the eyes are a pair of human eyes, the eyes are detected. The detection of eyes means the detection of faces and thus the detection of people.

Next, in step S6, it is determined whether or not the template scanning is substantially completed for the entire image of the first image data. If it is not completed, the process returns to step S3 and is interrupted. The detection of the first image candidate is restarted. When it is determined in step S6 that the template scanning of the entire image is completed, step S7
Proceed to.

In step S7, the image determined to be the end of template scanning in step S6 is a reduced image,
Moreover, it is determined whether the image has the minimum reduction ratio. If the image does not have the minimum reduction ratio, the process proceeds to step S8. Therefore, in the case of a non-reduced image, the process naturally goes to step S8 and the image reduction process is executed. The reduction ratio of the reduction process performed in step S8 is 1/2 or 1/3, which is 2 times.
The street is prepared. Therefore, in this embodiment, 1
/ 3 times is the minimum reduction ratio in step S7.

In step S8, if the current image is a non-reduced image, it is reduced to 1/2, and if it is a 1/2 reduced image, it is reduced to 1/3 of the non-reduced image. Step S8
The image reduced in step S3 returns to step S3, and the extraction of the first-eye image candidates is performed again using the same template as the non-reduced image. When the template described above is used for a non-reduced image, it extracts an eye image at a distance near N meters. Images of eyes at a distance near / 2 meters can be extracted. Similarly, when the template is used for a 1/3 times reduced image, an eye image at a distance near N / 3 meters can be extracted.

When the extraction of the eye images for the 1/2 times reduced image and the 1/3 times reduced image is completed, the affirmative determination is made in step S7, and the process proceeds to step S9.

In step S9, it is determined whether or not the extraction of the eye image from the second image data (image data obtained by the second camera 2) has been completed. If the eye image is extracted only for the first image data, step S3
Then, the same processing as the extraction of the eye image performed on the first image data is executed.

When the extraction of the eye image is completed for both the first image data and the second image data, the process proceeds from step S9 to step S10, and the distance between the eye corresponding to the extracted eye image and the cameras 1 and 2 is reached. The exact distance is calculated by comparing the two images. The calculation method is as described above, and the angle θ formed by the first camera and the second camera viewed from the target eye is calculated from the parallax between the two images, and the calculated angle θ is substituted into D = d / tan θ. The distance D from the camera to the eyes is calculated. d is the distance between the cameras.

In the above embodiment, the reduction ratio is 1 /
Two types, 2 times and 1/3 times, were prepared, but if a reduction ratio with a larger reduction ratio is used, eyes at positions closer to each other can be detected. For example, assuming that a template for 10 meters is used as a non-reduced image template, 1/1
If you apply a 0x reduction to the template, you get 1
Images of eyes at a distance of meters can be extracted. Further, if the enlargement ratio is applied, although the accuracy is lowered, it is also possible to extract the image of the distant eye.

FIG. 5 shows an image of eye image detection and distance measurement in the eye image detection apparatus of this embodiment, and particularly shows eye image detection and distance measurement of a non-reduced image.

The screen 20 shows an image based on the first image data acquired by the first camera 1 in step S1. The screen 21 shows a state in which the first image candidates are extracted in step S3, and the screen 22 is displayed in step S3.
4 shows a state in which the second image candidate has been extracted.

The screen 23 shows an image based on the second image data acquired by the second camera 2 in step S1 at the same time when the first camera 1 acquires the image of the screen 20. The screen 24 shows a state in which the first eye image candidate is extracted in step S3, and the screen 25 shows a state in which the second eye image candidate is extracted in step S4.

The screen 26 shows a state in which the first image data and the second image data are overlapped and the first image candidates are compared with each other in order to execute the distance calculation in step S9.

On the screens 20 and 23, respectively,
It is assumed that the person in the center is the same person and is located at a place about N meters away from the camera. As a result of scanning these screens with the template for N meters, it is shown that the right eye of the person (left eye when facing) was extracted as the first eye image candidate by the template of -20 degrees. Are screens 21 and 24.
On the screens 21 and 24, the templates of -20 degrees are displayed in an overlapping manner on the right eye of the person.

When the first eye image candidate is extracted, the second eye image candidate is extracted in the peripheral areas 11 and 12 of the first eye image candidate in step S4. Screens 22 and 25 are second
The result of extraction of eye image candidates is shown. In this example, since the second eye image candidate is the left eye of the person, it is determined that the second eye image candidate is paired with the first eye image candidate on any screen, and the first eye image candidate and Second
The eye image candidates are recognized as a pair of eye images in one face.

On the screen 26, the first image data and the second image data are overlapped and the eye-to-eye image distance of the right eye of the person is measured. The distance is the parallax between the first camera 1 and the second camera 2. The method of calculating the distance from the parallax is as described above.

Next, an eye image detecting device using different templates will be described. A different template is used instead of the template shown in FIG. 2 used in the first extraction means 5 and the second extraction means 6 or in the eye image candidate extraction processing of steps S3 and S4, and the template is supported. The eye image candidates are detected using the evaluation function. The other processes do not need to be changed, and the description thereof will be omitted.

The template of FIG. 2 has the first area of the pupil peripheral region.
A region which was divided into a luminance evaluation region and a second luminance evaluation region further outside thereof was used. On the other hand, the second template to be described below obtains the variance of the eye image sample for each pixel in a predetermined region centered on the pupil for a plurality of eye image samples in advance, and determines each pixel according to the obtained variance value. It is classified into a plurality of groups (areas).

First, a method of creating the second template will be described.

FIG. 6 shows a region frame forming the second template, which is a rectangle having a length of 5 pixels in the vertical direction and a length of 23 pixels in the horizontal direction. However,
The vertical and horizontal lengths and shapes are examples, and the present invention is not limited to these.

FIGS. 7 (a) to 7 (c) show three eye image samples appropriately selected from a large number of eye image samples, and the area frames of the second template are respectively added to these eye image samples. The fitted state is shown. In each eye image sample, the area surrounded by the broken line is the area frame corresponding to the second template shown in FIG.

As shown in the drawing, the center of the second template region frame is made to coincide with the center of the pupil of the eye image sample, and the longitudinal direction of the second template frame is aligned with the left-right direction of the eye. Take out the brightness for each. This operation is performed not only on the three eye image samples shown, but also on all of a wide variety of eye image samples prepared in advance.

Next, for each pixel in the second template region frame, the luminance variance of the whole-eye image sample is obtained, and classification is performed by an arbitrarily set threshold value.

FIG. 8 is a graph showing the luminance dispersion for the 23 pixels in the third row in the second template area frame. In this graph, the horizontal axis represents the pixel position and the vertical axis represents the variance value. The graph has a convex shape downward,
Although there are three portions showing values less than "0", these correspond to the so-called white-eye portions at the center of the pupil and the left and right sides thereof, respectively. Similarly, the variance value is obtained for each pixel in the first line, the second line, the fourth line, and the fifth line.

Next, the luminance dispersion value for each pixel in the second template region frame is classified into three groups by two threshold values. "200" and "50" as threshold values
"0" is adopted, and less than "200" is group 0, "20".
All pixels in the second template region frame are divided into three groups, where 0 is more than "500" and less than "500" is group 1 and "500" is more than group 2. FIG. 9 shows the result of assigning group numbers to all (5 × 23) pixels in the second template region frame, and this is the second template.

There are three evaluation functions for extracting eye image candidates, which are represented by the following equations (2) to (4), and all the evaluation functions are equal to or more than the threshold value set corresponding to each of them. In this case, it is determined to be an eye image candidate.

[0080]

[Equation 2]

The evaluation function (0,1) of the equation (2) is for judging whether or not there is a high possibility that it will be an eye image candidate from the relationship between the groups 0 and 1 of the second template.
Bias (0,1) in the equation is a constant set in advance based on the average brightness of image data and the like. After calculating the mean μ ₀ and μ ₁ and the variances σ ₀ and σ ₁ for group 0 and group 1, respectively, substitute μ ₀ , σ ₀ , μ ₁ , and σ ₁ in the right side of the evaluation function (2) for evaluation. The value of the function (0,1) is calculated.

The evaluation function (1, 2) of the equation (3) is for judging whether or not there is a high possibility that it will be an eye image candidate based on the relationship between the groups 1 and 2 of the second template.
Bias (1,2) in the equation is a constant preset based on the average brightness of image data. After calculating the average μ ₂ and the variance σ ₂ for group 2, the values of the evaluation functions (1, 2) are obtained by substituting them into the right side of the evaluation function (3) together with the already calculated μ ₁ , σ ₁ .

The evaluation function (2,0) of the equation (4) is for determining whether or not there is a high possibility that it will be an eye image candidate from the relationship between the group 2 and the group 0 of the second template.
Bias (2,0) in the equation is a constant set in advance based on the average brightness of image data and the like. Since the average and variance of group 2 and group 0 have already been calculated, they are substituted into the right side of evaluation function (4) to obtain the value of evaluation function (2,0).

In this way, the evaluation value is obtained for each of the evaluation function (0,1), the evaluation function (1,2), and the evaluation function (2,0), and each evaluation value is set to the corresponding threshold value. Compare with. If all the evaluation values are larger than the threshold value, it is determined to be an eye image candidate.

Next, various devices of the present invention using the eye image detecting device of the present invention will be described together with their embodiments.

FIG. 10 is a diagram showing an embodiment of a personal image projector equipped with the eye image detecting device of the present invention. The screen 30 is in front of the image projector 31.
Are arranged, and a desired image is displayed on the screen 30. On the back surface of the image projector 31, cameras 32 and 33 corresponding to the first and second cameras of the eye image detecting device are provided.
Are arranged, and a viewer behind the image projector 31 can be captured as an image of the eye image detection device. The image processing unit of the eye image detection device is housed in the housing of the image projector device 31.

Now, it is assumed that the viewer 34 is in the direction of the horizontal angle θ with respect to the position directly behind the image projector 31. In the case of this positional relationship, when viewed from the viewer 34, the screen 3
Since the right side of 0 is closer than the left side, the image on the right looks large and the image on the left looks small. Therefore, in the image projector 31, as shown in FIG. 11, if the right side of the video screen 35 is corrected so that it becomes smaller with respect to the left side according to the distance difference, the viewer 34 will start from a position directly in front of the screen. It feels like you're watching.

The image projector 31 of the present embodiment automatically performs this correction. FIG. 12 is a flowchart showing an operation procedure performed for automatically correcting the image screen shape.

In step S21, the eye image of the viewer 34 is detected. The detection of the eye image here includes the detection of the position (distance and angle) of the viewer 34 with respect to the image projector 31 of the eye.

In step S22, the position of the viewer 34 with respect to the screen 30 is calculated from the detected eye position information. If the positional relationship between the screen 30 and the image projector 31 is known in advance, the position (distance and angle) of the viewer 34 with respect to the screen 30 can be obtained from the eye position information.

In step S23, the image screen size is determined according to the distance between the screen 30 and the viewer 34, and in step S24, the left-right size ratio of the image screen is determined according to the distance and the angle.

Then, in step S25, a video screen is generated according to the size and the correction value determined in steps S23 and S24.

According to this image projector, since the size and shape of the video screen are automatically adjusted according to the position of the viewer, the viewer always displays an image of an appropriate size without distortion regardless of his / her position. You can appreciate it.

FIG. 13 is a diagram showing an embodiment of a television monitor device with a sound field adjusting function, which is equipped with the eye image detecting device of the present invention. The television monitor device 40 includes a main body 51.
And a support base 50 provided with an actuator 49 that supports the structure so as to be swingable. Actuator 49
Can tilt the main body 51 at a desired angle based on a command from a control device incorporated in the main body 51.

The main body 51 is provided with speakers 42, 43 and 44 arranged on the left and right sides and below the display screen 41, respectively, and a stereoscopic sound field can be produced by these speakers. Further, the main body 51 incorporates the above-described eye image detecting device of the present invention, and cameras 45 and 46 are arranged at the upper left and upper right corners of the front of the main body 51 as cameras for the eye image detecting device. Note that cameras 47 and 48 may be added to the lower left and lower right corners as needed.

Next, the adjustment operation of the television monitor device 40 will be described with reference to the flowchart shown in FIG.

First, in step S31, a viewer's eye image is detected. The detection of the eye image mentioned here includes the detection of the position (distance and angle) of the viewer's eyes. The position detection of the viewer's eyes is nothing but the position detection of the viewer's face. There is only one viewer.

In step S32, the inclination of the face of the viewer is detected from the eye image. Examples of the case where the viewer tilts his face to watch the case include a case where the viewer wears a cheek cane and a case where he is lying down.

Next, in step S33, the sound field control value and the tilt angle of the television monitor body are calculated.
The speaker control circuit built in the main body 51 can change the optimum viewing position regarding the sound field. Here, a control value is calculated so that the position of the face of the viewer becomes the optimum viewing position regarding the sound field. Further, how much the main body 51 is tilted is calculated based on the tilt of the viewer's face.

In step S34, the setting of the speaker control circuit is adjusted based on the sound field control value calculated in step S33, and the optimum viewing position regarding the sound field is matched with the position of the face of the viewer.

In step S35, the actuator 49
Is driven to set the inclination of the main body 51 to the angle calculated in step S34.

By constantly repeating these steps S31 to S35, the viewer is always adjusted to be in the center of the sound field, and the volume is adjusted according to the distance. Also,
When the tilt of the viewer's face changes, the TV monitor device body 51 also tilts automatically.

Next, the superimposed display window device of the present invention equipped with the eye image detection device of the present invention will be described.

The superimposition display window device is a device that has a display function in the window itself and that displays desired information in a superimposed manner on a landscape that can be seen through the window.

A method of using this superimposed display window when applied to the front window of an automobile will be briefly described. For example, if there is a well-known restaurant in front of the car, the driver will see the restaurant through the overlay display window like a normal front window, but the overlay display window can be obtained from a navigation device or the like. Information on the restaurant, such as a telephone number, is displayed. If the display information is displayed at the position where the straight line connecting the driver and the restaurant crosses the overlay display window, the driver can overlay the information such as the telephone number on the location of the restaurant viewed through the overlay display window. .

In the conventional superimposed display window, the information display position is calculated on the assumption that the position of the driver with respect to the window is a fixed position. Therefore, if the driver's position is different from the preset position, there is a problem that the target object and the display information do not overlap with each other.

On the other hand, according to the superimposed display window of the present embodiment, the display position is determined after accurately recognizing the position of the eyes of the driver, so that the display information and the target object properly overlap each other.

FIG. 15 shows an embodiment of the superimposed display window device of the present invention, in which the superimposed display window is applied to the front window of an automobile. A first camera 55 and a second camera 56 are arranged on the upper right side of the superimposed display window 57 which is the front window of the automobile. The first camera 55 and the second camera 56 are shown in FIG.
It corresponds to the first camera 1 and the second camera 2 of the eye image detecting device shown in FIG. 1 and is directed to the driver's seat behind the steering wheel 58. The control unit of the superimposed display window device includes the image processing unit 3 of the eye image detection device, and its operation is shown in the flowchart of FIG.

In FIG. 16, first, in step S41, the driver's eye image is detected. The detection of the eye image mentioned here includes the detection of the position (distance and angle) of the driver's eyes. Next, in step S42, the correction value of the display position is calculated from the difference between the detected eye position of the driver and the reference position. Subsequently, in step S43, the display position is corrected and the superimposed display control is performed.

As shown in FIG. 17, if the position of the driver 60 is deviated from the reference position 62, the target 61
And the reference display position 64 on the straight line connecting the reference position 62 with the reference position 62.
Therefore, if the display position 63 is shifted on the straight line connecting the target object 61 and the driver 60, the target object 61 and the display information 63 appear to overlap each other. Therefore, in step S42, a deviation amount of the display position is calculated as a correction value according to the deviation amount of the driver position from the reference position, and the display position is corrected in step S43.

Therefore, according to this superimposed display window device, the target object and its display information are always properly overlapped regardless of the position of the driver.

Next, a moving walkway device according to the present invention equipped with the eye image detecting device of the present invention will be described.

The belt moving speed of the conventional footpath is set to the standard speed. However, the walking speed before the user gets on the footpath varies, and if the walking speed and the belt moving speed are different from each other, the user may be impacted when getting on the board.

On the other hand, according to the walkway device of the present invention, the belt moving speed is adjusted according to the movement of the person who gets on the walkway, so that the impact received when getting on the walkway can be alleviated.

In the walkway device of the present invention, two cameras of the eye image detecting device are installed at the entrance (boarding entrance), and the belt control unit has the image processing unit of the eye image detecting device built therein.

FIG. 18 is a flowchart of the operation executed by the belt control unit of the walkway apparatus according to the first embodiment of the present invention. In step S51, whether or not there is a person near the entrance of the walkway is monitored by detecting an eye image by the eye image detecting device, that is, detecting a face. If a person's face is detected, the process proceeds to step S52 to estimate the person's position, moving speed, and traveling direction. The moving direction and the moving direction can be estimated by performing eye image detection a plurality of times at regular time intervals.

In step S53, the possibility that a person who is determined to be near the entrance of the footpath will enter the footpath is determined. Here, it is determined from the estimation result of step S52 whether or not the vehicle is heading toward the entrance. More specifically, if the vehicle is moving toward the entrance at a speed equal to or higher than the predetermined speed within the predetermined distance range, the procedure proceeds to step S54, and otherwise the procedure proceeds to step S51. Return.

In step S54, the belt moving speed is set based on the moving speed of the intruder estimated in step S52. The relationship between the moving speed of the intruder and the moving speed of the belt is acquired in advance by conducting an experiment and is stored as a belt speed setting database. Then, the belt moving speed is adjusted so as to match the output of the belt speed setting database.

When the adjustment of the belt moving speed is completed, the process proceeds to step S56 and it is monitored whether or not the intruder gets on the belt. That is, it waits for an intruder to board at the set belt movement speed. This monitoring can also be achieved by using an eye image detecting device. If it is possible to confirm the boarding here, the process proceeds to step S57, and the belt moving speed is gradually switched to the standard moving speed.

On the other hand, if the intruder does not get on the belt after monitoring for a predetermined time, the process returns to step S51 to newly monitor the presence or absence of a person near the entrance.

By controlling the belt moving speed in this way, it is possible to mitigate the impact at the time of getting on the walkway belt.

FIG. 19 relates to the second embodiment of the walkway apparatus of the present invention and is a flow chart of the operation executed by the belt control section thereof. While the first embodiment adjusts the belt moving speed in consideration of the movement of the intruder, the second embodiment moves the belt in consideration of the intention of the pedestrian riding on the moving belt. It regulates the speed. The pedestrian's intention here means the urgency, that is, the intention to move faster. This device detects the intention from the expression of a pedestrian and changes the belt moving speed. In the present embodiment, two cameras of the eye image detection device are installed near the exit (exit) toward the entrance. The image processing unit of the eye image detection device is built in the belt control unit.

First, in step S60, it is monitored whether or not there is a person on the walkway belt by detecting an eye image, that is, a face by the eye image detecting device. The determination of whether or not the detected pedestrian is riding on the footpath belt is achieved by position measurement by the eye image detection device.

When the face of the person riding on the walking path belt is detected, the process proceeds to step S61, the walking speed of the person on the walking path is detected, and the facial expression is recognized. The moving speed of the pedestrian can be estimated from the moving amount per unit time by performing eye image detection a plurality of times at regular time intervals. The walking speed on the belt can be calculated by subtracting the belt moving speed at that time from the moving speed. Facial expression recognition uses an existing algorithm.

In step S62, the degree of urgency is determined based on the walking speed and the facial expression recognition result. The relationship between the walking speed, the facial expression recognition result, and the urgency is prepared in advance. If the urgency is lower than the preset threshold value, the process returns to step S60 without adjusting the belt moving speed. If the urgency is higher than the threshold, the process proceeds to step S63.

In step S63, it is determined whether or not there is another pedestrian on the walkway, and if there is another pedestrian, step S6.
Return to 0, and if not, move to step S64. In step S64, it is determined whether or not there is a pedestrian newly riding on the moving belt, and if there is a pedestrian, the process returns to step S60. If there is no pedestrian coming in, the process proceeds to step S65, and the belt moving speed is adjusted according to the urgency. The greater the urgency, the higher the belt moving speed.

As can be seen from the contents of steps S63 and S64, the belt moving speed is adjusted only when the number of pedestrians on the belt is one, and when there are a plurality of people, the belt is moved at the standard speed without adjustment. To maintain.

20 to 22 relate to a guide robot using the eye image detecting device of the present invention. 20 is an external view of the hat-type guide robot, and FIG. 21 is an external view of the guide dog-type guide robot. FIG. 22 is a flow chart showing the operation of the guide robot of the hat type and the guide dog type.

In the hat-type guide robot of FIG. 20, the cameras 70 and 71 corresponding to the first camera and the second camera of the eye image detecting device are arranged on the left and right of the hat 72 so as to face forward. The hat 72 is provided with an earpiece 73 having an earphone, and voice instructions can be given to the user via the earphone 73.

The guide dog type guide robot shown in FIG. 21 is provided with a body portion 77 which can freely move on the ground, and a head portion 76 which is arranged in front of the body portion 77. The head 76 is provided with a camera 74 and a camera 75 corresponding to the first camera and the second camera of the eye image detecting device. One end of a guide rope 79 is tied to the neck 80 of the guide dog type guide robot, and the user 7 holding the other end of the guide rope 79.
Induce 8.

Next, the operation of these guide robots will be described with reference to the flowchart of FIG.

First, in step S71, an eye image is detected by the eye image detecting device. The detection of an eye image here is nothing but the detection of the face of a person having an eye corresponding to the eye image and the acquisition of the distance.

The process of step S71 is the same as step S72.
By the determination process of step 1, the process is repeatedly executed until a person within a predetermined distance is found. When a person is found within the predetermined distance, the process proceeds from step S72 to step S73.

At step S73, the owner (user) is notified that a person has approached within a predetermined distance. In the case of a hat-type guide robot, this is announced by voice using an earphone. In the case of a guide dog type guide robot, a speaker built in the head 76 informs that fact.

Then, in step S75, it is determined whether or not the person approaching can be avoided without stopping. When it is determined that it can be avoided, the process proceeds to step S76 to guide the person to avoid it, and when it is determined that it cannot be avoided, the process proceeds to step S74 and the user is stopped to avoid the end as much as possible. Give instructions. The instruction in step S74 continues until it is determined that the instruction can be avoided. When the guidance in step S76 is executed after it is determined in step S75 that it can be avoided, step S76 is executed again.
Returning to 71, the face finding process is executed.

According to this guide robot, unlike a mere obstacle sensor, the oncoming person is recognized and distinguished from other obstacles, and then the person is guided to avoid it.
It is possible to provide the user with guidance equivalent to that of a guide dog.
Moreover, unlike guide dogs, they are not living things, so their handling is simple.

Next, an embodiment of the mirror angle control device of the present invention using the eye image detection device of the present invention will be described with reference to FIGS. 23 and 24. This mirror angle control device is mounted on an automobile and automatically adjusts the angles of the rearview mirror and the side mirror according to the position of the driver. FIG. 23 is a top view showing a driver's seat of an automobile equipped with this mirror angle control device and its surroundings.
4 is a flowchart showing the operation of the mirror angle control device.

The interior of the automobile 81
A room mirror 83 is arranged in the center of the front part of the, and side mirrors 84, 85 are arranged on the left and right of the outdoor front part.
A small motor is attached to each of the room mirror 83 and the side mirrors 84 and 85, and the mirror angle can be adjusted by driving the motor. The adjustment of the mirror angle is executed by an instruction from the control unit (not shown).

A camera section 82 is provided on the upper part of the windshield on the driver's seat side, and the camera section 82 houses the first camera and the second camera of the eye image detecting device.

First, when the power is turned on in step S81, the process moves to step S82, and the eye image detection device detects the eye image. Here, the power is turned on by the key switch of the automobile, but instead of this, a dedicated button, a touch panel, a remote controller, a voice input device, etc. are used so that the power can be turned on at the timing desired by the driver. May be.

In step S82, the three-dimensional position of the eyes of the passenger facing forward is detected based on the eye image detection by the eye image detecting device. In step S83, it is determined whether the detected eyes are a plurality of pairs or a pair, that is, whether the number of persons is large or one. If there are a large number of people, the process proceeds to step S84 to select the eyes of the person closest to the camera unit 82. Since the camera unit 82 is arranged in front of the driver, the person closest to the camera unit 82 can be recognized as the driver. That is, step S8
Reference numeral 4 is a process for selecting a driver. If only one eye is detected in step S83, it is determined that the person is the driver, and the driver selection process of step S84 is skipped and the process proceeds to step S85.

In step S85, the mirror angles of the room mirror 83 and the side mirrors 84 and 85 are adjusted according to the position of the eyes of the person who is certified as the driver. Room mirror 8
3. Assuming that the central directions of the scenes to be captured by the side mirrors 84 and 85 are D83, D84, and D85, respectively, the centers of the respective mirrors are oriented in the intermediate direction between the driver's eye position and D83, D84, D85. Then, regardless of the position of the driver, the driver always sees the scenery in the directions of D83, D84, and D85 in the room mirror 83, the side mirrors 84, 85.
Can be seen through. The relationship between the driver's position and the mirror angle is saved in advance as a data map, and if the mirror angle is extracted from the driver's position from the data map, the mirror angle can be set easily and the tilt of the face is taken into consideration. The mirror angle can also be set. Further, it is desirable that the user can adjust the relationship between the driver's position and the mirror angle.

When the angle adjustment of the mirror is completed, step S
Proceeding to 86, it is confirmed whether or not the power is turned off by operating the key switch. If there is no key switch for turning off the power, the process returns to step S82 to execute the three-dimensional position detection of the eye again. When the operation of the key switch for turning off the power is confirmed, the process proceeds to step S87 and ends. In addition, the mirror angle at the time of termination is maintained in the immediately preceding state.

According to the mirror angle control device of the present invention, the mirror angles of the room mirror and the side mirror are automatically adjusted according to the position of the driver.

25 and 26 relate to an automatic door opening / closing control device of the present invention utilizing the eye image detection device of the present invention. FIG. 25 is a top view of the device, and FIG. 26 is a flow chart showing its operation. is there.

In the case of an automatic door facing a passage, if a simple person sensor is used for opening and closing, it is not possible to distinguish between a mere passerby and a person who wants to enter by opening the door. There was a problem of opening.

On the other hand, the automatic door opening / closing control device of the present invention automatically opens the door only when there is a person directly facing the door in the vicinity of the door, so that the door is opened for a mere passerby. Never.

In FIG. 25, automatic doors 91 and 92 that open to the left and right are arranged on one side of the passage 90. The opening / closing operation of the automatic doors 91 and 92 is performed based on an instruction from a control unit (not shown). Automatic doors 91 and 9
Cameras 93 and 94 are provided on the inner side of 2, that is, on the upper side opposite to the passage 90. The cameras 93 and 94 are
It corresponds to the first camera and the second camera of the eye image detection device mounted as one element of the automatic door opening / closing control device. In this embodiment, the automatic doors 91 and 92 are made of transparent glass, and the cameras 93 and 94 are capable of photographing the situation on the aisle side even when the automatic doors 91 and 92 are closed as shown. There is.

This automatic door opening / closing control device operates so as to open the door when the eyes image detecting device incorporated therein detects the eyes of a person facing the doors 91 and 92. Figure 2
6 is a flowchart showing the operation.

First, in step S91, an eye image of a person near the door is detected. The detection of the eye image mentioned here includes the detection of the position (distance and angle) of the human eye. Next, in step S92, it is determined whether or not there is an eye of a person who is directly facing the door within a predetermined distance from the cameras 93 and 94. When the determination is affirmative, the automatic doors 91 and 92 are opened. If somebody wants it,
In step S93, the door is opened for a certain period of time. If it is determined in step S92 that there is no eye of the person directly facing the door, the process returns to step S91 to perform eye image detection.

As described above, according to the automatic door opening / closing control device of the present invention, the eyes of a person are detected to open / close the door. Therefore, unlike a conventional automatic door opening / closing device using a human sensor, a mere passerby is used. Against accidentally opening the door.

Next, the ball discharging device of the present invention using the eye image detecting device of the present invention will be described. The conventional ball ejecting device in the automatic tennis device or the like does not hit the ball according to the position of the player. On the other hand, the ball emitting device of the present invention acquires the position of the player by detecting the eyes of the player whose face is facing the ball emitting portion by using the eye image detecting device, and outputs the ball according to the position. You can
The first camera and the second camera of the eye image detecting device are arranged in the vicinity of the ball emitting unit, and the image processing unit of the eye image detecting device is incorporated as one element of the control unit.

FIG. 27 is a flow chart showing the control operation of the ball ejection device according to one embodiment of the present invention.

First, in step S101, eye image detection is performed by the eye image detection device. Then, in step S102, the three-dimensional position of the face is acquired from the detected eye image. As described above, the eye image detection device can accurately acquire the distance by using the parallax of the eye images acquired by the two cameras. Therefore, the three-dimensional position of the pair of eyes corresponding to the eye image, that is, the three-dimensional position of the player's face can be obtained from the distance and the angle.

Next, in step S103, the landing point (ball position) of the ball to be launched, the launch angle (depression angle) with respect to the ground, and the initial velocity are determined. These are the skill level of the player set in advance and step S
The position is determined in consideration of the position of the player acquired in step 102. For example, if the skill level of the player is set to advanced, the ball initial velocity is determined to be a relatively large value, and the ball delivery position and depression angle are determined according to the initial velocity.

When the ball delivery position, depression angle and initial velocity are determined, the process proceeds to step S104 and the ball is launched based on the determined values. And after hitting the ball,
Based on the depression angle and initial velocity of the ball, the time (response time) from the time when the ball is launched to the time when the player reacts and returns is calculated, and all the processes are stopped until the response time elapses (step S105). When the response time has elapsed, the process returns to step S101, and the ball output control is executed in the same manner.

As described above, according to the ball discharging device of the present invention, the ball can be discharged according to the position of the user.
You can practice more realistically. Although the case where the ball discharging device is used for the automatic tennis device has been described as an example, the ball discharging device may be used for other ball game practice devices. For example,
It can be used as a ball throwing device for defensive practice of baseball and a throwing device for batting practice.

Next, the fan of the present invention equipped with the eye image detecting device of the present invention will be described. FIG. 28 is a front view of a fan that is an embodiment of the present invention, and FIG. 29 is a flow chart showing its control operation.

In FIG. 28, a three-bladed fan 101
Is attached to the support base 102 via a neck 105, and the wind direction and the air volume during rotation can be adjusted by an instruction from a control unit built inside the support base 102. The camera 1 is provided on the upper surfaces of both sides of the support base 102.
03 and 104 are fixed, and these cameras function as a first camera and a second camera of the eye image detection device. The image processing unit of the eye image detection device is built in the support base 102 as a part of the control unit.

FIG. 29 is a flow chart showing the control operation of this fan. In step S111, the function of the eye image detection device is used to detect the eye image, that is, detect the face facing the fan and calculate the three-dimensional position thereof.

In step S112, step S111
In step S113, it is determined whether or not the eye image has been detected. If not detected, the process returns to step S111 via step S113.
In step S113, if the fan drive switch is on, the switch is turned off.

When it is determined in step S112 that the eye image can be detected, the process proceeds to step S114, and if the switch is off, it is switched on, and then step S114.
Proceeding to 115, the direction and air volume of the fan 101 are determined from the position of the face corresponding to the eye image. In the present embodiment, the direction of the fan 101 is the center direction of the person's body estimated from the position of the face. By increasing the air volume as the face is farther from the fan, the user can always receive a stable wind. Here, a table for obtaining the relationship between the distance and the air volume is stored in advance, and the air volume is derived from the distance.

Next, in step S116, the fan 101 is directed in the direction determined in step S115, and the fan 10 is rotated at the number of revolutions corresponding to the air volume determined in step S115.
Rotate 1. Then, the process returns to step S111 again,
The eye image is detected.

According to this electric fan, the user can operate and stop the electric fan simply by turning his / her face to the electric fan, and the wind direction and the wind strength at the time of operation can be changed according to the position of the user. Adjusted optimally.

Next, the viewing inspection apparatus of the present invention using the eye image detection apparatus of the present invention will be described. FIG. 30 is a perspective view showing the external appearance of a video display device equipped with a viewing survey device according to an embodiment of the present invention. Video display device 111
The left and right upper surfaces of the camera 1 correspond to the first camera and the second camera of the eye image detection device that is a part of the viewing inspection device.
12 and the camera 113 are mounted. The viewing inspection device including the image processing unit of the eye image detection device is arranged behind the video display device. Six persons 114, including viewers, are standing in front of the display unit of the video display device 111.

FIG. 31 is a flow chart showing the operation of the viewing survey apparatus, which is performed in steps S121 to S12.
Reference numeral 5 denotes an operation as an eye image detection device. Step S1
At 21, the images captured by the cameras 112 and 113 are input. Next, the input image is searched to detect the eye image. This search is performed until the eye image when the face is caught from the front is found, and if found, it is determined that the face of the person looking at the video display device 111 is found, and the process proceeds from step S124 to step S125. .

In step S125, information regarding the distance and direction of the corresponding eye is acquired from the eye image. Distance is 2
It can be accurately calculated based on the parallax of the images captured by the two cameras 112 and 113.

In the next step S126, the face is classified based on the distance and direction of the found face and weighted. For example, it can be inferred that the user is looking at the display screen of the video display device 111 with high interest at a short distance from the front. That is, the degree of interest can be weighted based on the distance and the direction. Information about the distance and the direction of the discovered face and weighting based on these is stored.

When this processing is completed, step S1 is executed again.
Returning to step 21, the search for face detection is continued, and step S12
In step 3, when it is determined that the search for the same image is completed, the process proceeds to step S127.

In step S127, the total number of viewers captured by the image is counted. FIG. 32 is an example of an image captured by the camera 112 or 113, and FIG. 33 is a top view showing the situation at that time. Five persons 121 to 125 are standing in front of the video display device 111, and the camera 11
5 or 1 person in the image captured by 2 or 113
21 to 125 are projected. Video display device 111
People 121 to 123 near the front are facing the front and are clearly viewers. On the other hand, the persons 124 and 125
Is a passerby who does not look at the image display device 111.

Since this apparatus detects only the face of the person facing the camera, in this image example, the persons 121 to 12
Three persons of 3 are detected as faces, and their distance, direction, weighting, etc. are stored. Therefore, step S127
Then, 3 people are counted as the total number of viewers.

Then, the process shifts to step S128 to obtain time information at that time, and further shifts to step S129 to output the number of found faces and time information, and returns to the start. The output data is saved as a history in the storage device.

According to the viewing inspection apparatus of the present embodiment, it is possible to automatically and accurately investigate how many people watch the contents displayed by the video display apparatus for each time zone. If it is applied to an advertising poster or a show window instead of the video display device, it is possible to investigate the person who viewed these by time zone. Further, by using weighting, it is possible to separately count the person who looks seriously and the person who tries to wash it.

Next, the automatic ticket gate control apparatus of the present invention equipped with the eye image detection apparatus of the present invention will be described. The automatic ticket gate control device of the present invention is for the most rational control of the passage direction of each ticket gate passage in a ticket gate having a plurality of ticket gate passages.

FIG. 34 is a schematic top view showing an example of an automatic ticket gate. The ticket gate 131 in this example is N ticket gate passages 131
(1) to 131 (N). Each ticket gate 131
All of (1) to 131 (N) are one-way traffic, and the traffic direction D is controlled by the automatic ticket gate control device of the present invention. The direction of traffic is notified to passers-by by opening and closing doors and a display device (not shown).

Zone A on the left side of the ticket gate 131 is divided into each of the ticket gate passages 131 (1) to 131 (N), and the zone (Zo
ne) A ₁ to zone (Zone) A _n , and the right side zone B is similarly divided into zones (Zone) B ₁ to Zone (Bone) B _n .

If zone A is outside the field and zone B is inside the field, the person heading from zone A to zone B is the entrance person, and the person heading from zone B to zone A is the exit person. That is,
In each of the ticket gates 131 (1) to 131 (N), it becomes an entrance passage when the traffic direction D _A is adopted, and becomes an exit passage when the traffic direction D _B is adopted.

In each of the ticket gate passages 131 (1) to 131 (N), cameras are installed on both sides of the passage on both the zone A side and the zone B side. These cameras are used as a first camera and a second camera of the eye image detecting device for each entrance and exit of each passage. However, since the cameras between the adjacent passages are used in common with each other, as shown in the figure, zone A
The (N + 1) cameras A0 to An are provided on the side, and the (N + 1) cameras B0 to Bn are provided on the zone B side.

[0179] For example, the ticket gate passage 131 (1), separate zones A and has camera A0 and A1 which faces _one is used as the first camera and the second camera eye image detecting device to capture divided zone A _1, ticket gate passage In 131 (2),
Separate zones A _2, opposite to that camera A1 and A2 are used as the first camera and the second camera eye image detecting device to capture divided zone A _2.

FIG. 35 is a flow chart showing the operation of the automatic ticket gate control apparatus according to the embodiment of the present invention. This automatic ticket gate control device obtains the number of persons and the distance to each person's ticket gate for all divided areas of the divided zones A ₁ to A _n and the divided zones B ₁ to B _n ,
As a whole, the direction of traffic is decided so that congestion can be avoided most.

[0181] First, after resetting the timer to zero at step S131, the calculating an evaluation function X ^B k of the evaluation function X ^A k and separate zones Bk divided zones Ak for all the divided zones in step S132. The evaluation functions X ^A k and X ^B k are the number of people per unit distance and are given by the following equations (5) and (6).

[0182]

[Equation 3] Here, N ^A k and N ^B k are the numbers of people approaching the ticket gate passage 131 (k) in the zone Ak and the zone Bk, respectively, and L ^A k, j and L ^B k, j are the zone respectively. Ticket passage 13 in Ak and Zone Bk
It is the distance of the j-th person approaching 1 (k).

Number N ^A k (k = 1, ..., N), number N ^B k (k
= 1, ..., N), the distance L ^A k, j (k = 1, ..., N, j)
= 1, ..., N ^A k), distance distance L ^B k, j (k = 1, ...,
n, j = 1, ..., N ^B k) can be obtained by detecting the eye image by the eye image detecting device for each ticket passage.

The larger the values of the evaluation functions X ^A k and X ^B k, the more people are approaching the entrance and exit of each ticket gate passage.

[0185] Next, the total Y ^B Total Y ^A and the evaluation function value X ^B k zone B side of the evaluation function value X ^A k zone A side in Step S133, Y ^A for Y ^A and Y ^B Calculate the ratio R of

In step S134, the passing direction of the k-th ticket gate passage is provisionally determined by comparing the evaluation functions X ^A k and X ^B k. Regarding the kth ticket gate, if X ^A k ≧ X ^B k, the traffic direction D ^A k is adopted (step S135), and if X ^A k <X ^B k, the traffic direction D ^B k is adopted ( Step S136).

[0187] In this way, when the traffic direction is provisionally determined for all N ticket gate passages, step S
In 137, the total number N _A of the traffic directions D ^A k, the traffic direction D
Find the total number N _{B of} ^B k. The ratio R of Y ^A for a total of Y ^B Total Y ^A and the evaluation function value X ^B k of the evaluation function value X ^A k
Determining the difference Diff ideal number of traffic direction D ^A k and (N × R) and the total number N _A tentative traffic direction D ^A k-based.

After step S138, the passage direction of the ticket gate passage is adjusted so that Diff becomes zero. In step S138, when Diff> 0, that is,
Traffic direction D ^A k of the ideal number (N × R) is a step S1 when greater than the total number N _A tentative traffic direction D ^A k
Proceed to 39, the passage direction of the wicket passage nearest traffic direction D ^B k to the ticket checker passage evaluation function value X ^A k is the maximum D ^A k
Switch to. As a result, the total number N _A of the traffic directions D ^A k is reduced by 1, so that in step S140, Diff =
Execute Diff-1. In step S141,
If the result is not Diff = 0, step S139 is executed again, and if Diff = 0, step S142.
Then, the traffic direction D ^A k and the traffic direction D ^B k are determined.

In step S138, when Diff <0
Sometimes, the process proceeds to step S143, and the evaluation function value X^Bk
Passing direction D that is closest to the ticket gate passage^Awicket of k
D is the direction of passage^BSwitch to k. This will
Row direction D^Atotal number of k N_AIs increased by 1, so
In step S144, Diff = Diff + 1 is executed. Ste
In S145, the result must be Diff = 0.
If so, step S143 is executed again, and Diff = 0.
If so, the process proceeds to step S142 and the traffic direction D ^Ak
And traffic direction D^BConfirm k.

At the time when the traffic direction D ^A k and the traffic direction D ^B k are confirmed, if the timer cleared to zero in step S131 is 10 seconds or more, the process proceeds to step S147 and the traffic route determined in step S142. The opening / closing state of the ticket gate passage is changed according to the direction, and the process returns to step S131. If the timer is less than 10 seconds, step S132
Returning to, the provisional traffic acquisition and correction based on the ideal traffic ratio is performed again. This allows
The door is opened and closed every 10 seconds in the optimal direction.

As described above, according to the automatic ticket gate control apparatus of the present invention, in the ticket gate having a plurality of ticket-passages of the passage-direction switching type, the passage direction of each ticket gate passage is set so as to minimize the congestion of passers-by. Controlled.

[0192]

As described above, according to the eye image detecting device of the present invention, the eye image can be detected by a simple arithmetic process, and thereby the three-dimensional position of the human eye corresponding to the eye image can be detected. Can be obtained. Further, according to the operation control device, the viewing inspection device, and the automatic ticket gate control device equipped with the eye image detection device of the present invention, it is possible to execute an appropriate operation according to the position of the user.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of an eye image detecting device according to the present invention.

FIG. 2 is a diagram showing an example of a template with a tilt of 0 degrees used in the eye image detection device.

FIG. 3 is a diagram showing an eye image candidate extraction range in the second extraction means 6.

FIG. 4 is a flowchart showing an operation when the function of the image processing unit 3 is executed by a microcomputer.

FIG. 5 is a diagram showing a procedure of eye image detection and distance measurement in the eye image detection device of the present embodiment.

FIG. 6 is a diagram showing a region frame forming a second template in the eye image detecting device according to the present embodiment.

7A to 7C are diagrams showing three eye image samples appropriately selected from a large number of eye image samples.

FIG. 8 is a graph showing the luminance dispersion for 23 pixels on the third row in the second template region frame.

FIG. 9 is a diagram showing a second template.

FIG. 10 is a layout diagram showing an embodiment of a personal image projector of the present invention.

FIG. 11 is a diagram showing a corrected video screen by the image projector 31.

FIG. 12 is a flowchart showing an operation procedure performed by the image projector 31 for automatic correction of a video screen shape.

FIG. 13 is a diagram showing an embodiment of a television monitor device with a sound field adjusting function of the present invention.

FIG. 14 is a flowchart showing an adjusting operation of the television monitor device 40.

FIG. 15 is a conceptual diagram showing an embodiment of a superimposed display window device of the present invention.

FIG. 16 is a flowchart showing the operation of the control unit of the superimposed display window device.

FIG. 17 is a diagram showing a positional relationship between a target object and its display information in the superimposed display window device.

FIG. 18 is a flowchart of the operation executed by the belt control unit of the first embodiment of the walkway device of the present invention.

FIG. 19 is a flowchart of the operation executed by the belt control unit of the second embodiment of the walkway device of the present invention.

FIG. 20 is an external view of a hat-type guide robot which is an embodiment of the guide robot of the present invention.

FIG. 21 is an external view of a guide dog type guide robot which is an embodiment of the guide robot of the present invention.

FIG. 22 is a flowchart showing the operation of a guide robot of a hat type and a guide dog type.

FIG. 23 is a diagram showing an embodiment of a mirror angle control device of the invention.

FIG. 24 is a flowchart showing the control operation.

FIG. 25 is a view showing an embodiment of an automatic door opening / closing control device of the present invention.

FIG. 26 is a flowchart showing the control operation.

FIG. 27 is a flowchart showing a control operation of an embodiment of the ball discharging device of the present invention.

FIG. 28 is a front view showing an embodiment of the fan of the present invention.

FIG. 29 is a flowchart showing the control operation.

FIG. 30 is a perspective view showing the outer appearance of a video display device equipped with the viewing inspection device of the present invention.

FIG. 31 is a flowchart showing the operation of an embodiment of the viewing survey device of the present invention.

FIG. 32 is a diagram showing an example of an image captured by the camera 112 or 113.

FIG. 33 is a top view showing a situation corresponding to the image shown in FIG. 32.

FIG. 34 is a schematic top view showing an example of an automatic ticket gate equipped with the automatic ticket gate control device of the present invention.

FIG. 35 is a flowchart showing the control operation.

[Explanation of symbols]

1 ... 1st camera, 2 ... 2nd camera, 3 ... Image processing part, 4
... image reduction means, 5 ... first extraction means, 6 ... second extraction means, 7 ... determination means, 8 ... control means, 10 ... first template, 31 ... image projector, 51 ... television monitor device body, 57 ... Superimposition window, 72 ... Hat type guide robot, 77 ... Guide dog type guide robot body part, 91,
92 ... Automatic door, 101 ... Fan, 111 ... Image display device, 131 ... Automatic ticket gate.

Continued front page    (72) Inventor Yoshiyuki Hatakeyama             1 Toyota Town, Toyota City, Aichi Prefecture Toyota Auto             Car Co., Ltd. (72) Inventor Kenichi Kitahama             1 Toyota Town, Toyota City, Aichi Prefecture Toyota Auto             Car Co., Ltd. (72) Inventor Ekio Fukamachi             1 Toyota Town, Toyota City, Aichi Prefecture Toyota Auto             Car Co., Ltd. (72) Inventor Hiroshi Nakamura             1 Toyota Town, Toyota City, Aichi Prefecture Toyota Auto             Car Co., Ltd. (72) Inventor Takeshi Sawamura             1 Toyota Town, Toyota City, Aichi Prefecture Toyota Auto             Car Co., Ltd. (72) Inventor Tomoaki Osawa             1 Toyota Town, Toyota City, Aichi Prefecture Toyota Auto             Car Co., Ltd. F-term (reference) 5B057 BA02 CA12 CA16 DA08 DB02                       DC03 DC33 DC36                 5L096 CA02 FA66 HA08 JA03 JA09                       JA11

Claims (12)

[Claims] 1. A first eye image candidate is directly extracted from image data by scanning an image capturing means and a template for extracting an eye image on the image data obtained from the image capturing means. Extraction means, second extraction means for extracting second eye image candidates within a predetermined range from the first eye candidates, and the second eye image candidates are one of the first eye image candidates and the first eye image candidates. An eye image detection device comprising: a determination unit that determines whether or not the face images are paired eye images. 2. The eye image detecting apparatus according to claim 1, wherein a plurality of different templates are prepared according to the inclination of the face to the left and right. 3. The first and second extraction means extract eye images at different distances from the imaging means by enlarging or reducing the image data. The eye image detection device described. 4. The template has a first brightness evaluation area corresponding to the pupil and its vicinity, and a second brightness evaluation area surrounding the first brightness evaluation area, and the first extracting means is configured to: The eye image according to any one of claims 1 to 3, wherein the eye image is extracted based on an average luminance and a luminance dispersion of each of the first luminance evaluation area and the second luminance evaluation area. Detection device. 5. The template preliminarily obtains a variance of the eye image sample for each pixel within a predetermined area centered on a pupil for a plurality of eye image samples, and groups each pixel into a plurality of groups according to the obtained variance value. 4. The first extraction unit extracts the eye image based on the average luminance and the luminance variance of each of the groups, according to claim 1. The eye image detection device according to one item. 6. A distance calculating means is further provided, and a plurality of said image pickup means are provided, and said first extracting means, said second extracting means and said determining means are obtained by said plurality of image pickup means. The eye image is detected for each image data by performing processing on each of the image data, and the distance calculation unit compares the position of the eye image of each image data to determine the distance from the imaging unit to the eye. The eye image detection device according to claim 1, wherein the eye image detection device is calculated based on parallax. 7. An operation control device for controlling the operation of a device that provides a predetermined action to a person, wherein the eye image detection device according to any one of claims 1 to 6, and the eye image detection device. An operation control device for controlling the device so that the action matches a person having an eye corresponding to the eye image detected by the operation control device. 8. The operation control device according to claim 7, wherein the control by the control means is executed in consideration of the inclination of the eye image detected by the eye image detection device. 9. The control means detects a corresponding eye movement from a change in an eye image detected by the eye image detecting device, and controls an action executed by the device in consideration of the eye moving direction. The operation control device according to claim 7, wherein: 10. The facial expression recognition means for recognizing a facial expression of a person having an eye corresponding to the eye image detected by the eye image detection device, wherein the control means sets the facial expression recognized by the facial expression recognition means. The operation control device according to claim 9, wherein the operation speed of the device is changed in accordance with the operation speed. 11. The eye image detecting device according to claim 1, wherein an image display unit for displaying an image, and an image pickup unit are arranged so as to pick up an image of an area facing the image display unit. And a number-of-people counting device for counting the number of people whose faces are facing the video display unit based on the eye image detected by the eye image detection device. 12. The automatic ticket gate control device for controlling the traffic direction of a plurality of ticket gate passages capable of switching the traffic direction, wherein the imaging means is arranged in the ticket gate passage. The eye image detection device, based on the eye image detected by the eye image detection device, user detection means for detecting the number and distance of people approaching each of the ticket gates, by the user detection means An automatic ticket gate control device comprising: a traffic direction determining means for determining a traffic direction of each of the ticket gates based on the number and distances of persons approaching each of the ticket gates detected.