JP2017102748A - Pupil image learning device, pupil position detection device, and program therefor - Google Patents

Abstract

Kind Code: A1 An object of the present invention is to detect a pupil position with a simple configuration even when a high-resolution pupil-enlarged image cannot be captured. Kind Code: A1 In a pupil detection device, a pupil image learning device generates a pupil image discrimination model through machine learning of an image feature quantity calculation unit for calculating an image feature quantity relating to an edge and an image feature quantity. The pupil position detection device 4 includes a pupil image discrimination model generation unit 26, and a pupil candidate region setting unit 40 for setting a pupil candidate region in a face image obtained by cutting out a person's face region, and a window set in the pupil candidate region. An image feature amount calculator 44 that calculates an image feature amount related to an edge for a region, and a pupil image determination that inputs the image feature amount of the window region into a pupil image determination model and determines whether or not the window region is a pupil image. and a pupil position calculator 48 for calculating the pupil position included in the window region determined to be the pupil image. [Selection drawing] Fig. 1

Description

  The present invention relates to a pupil image learning device that generates a pupil image discrimination model that discriminates a pupil image, a pupil position detection device that detects a pupil position of a person included in a captured image using the pupil image discrimination model, and a program thereof. .

  Conventionally, a method for detecting a pupil position from a video (video) has been used in various fields such as specifying a driver's gaze direction, estimating a concentration of program viewers and conference participants, and adjusting illumination light intensity. (Patent Documents 1 to 7).

  In the inventions described in Patent Literatures 1 and 2, a magnified pupil image is captured by a photographing camera (head mounted device) mounted on the subject's head, and the pupil position is detected. In addition, the inventions described in Patent Documents 3 and 4 capture a pupil enlarged image of a subject (driver) with a photographing camera attached in a passenger compartment, and specify a line-of-sight direction. In addition, the inventions described in Patent Documents 5 to 7 photograph a subject's face with a plurality of photographing cameras, and specify the line-of-sight direction of the subject from the face images photographed by each photographing camera.

JP2012-050837A JP 2011-224213 A JP 2009-157736 A JP 2011-086125 A JP 2014-094186 A JP2012-022646A JP 2012-217524 A

However, in the inventions described in Patent Documents 1 to 4, the camera is mounted on the subject's face by mounting the camera on the subject's head or by placing the camera near the subject's face. It is necessary to take a high-resolution pupil magnified image in a close-up state. In other words, the inventions described in Patent Documents 1 to 4 cannot detect the pupil position when a high-resolution pupil enlarged image cannot be captured.
In addition, since the inventions described in Patent Documents 5 to 7 require a plurality of photographing cameras, the apparatus is increased in size and complexity.

  Accordingly, an object of the present invention is to provide a pupil image learning device, a pupil position detection device, and a program thereof that can detect a pupil position with a simple configuration even when a high-resolution pupil magnified image cannot be captured.

  In view of the foregoing problems, the pupil image learning device according to the present invention is a learning image set that is a set of a pupil image obtained by cutting out a human pupil region and a non-pupil image obtained by cutting out a face region other than the human pupil. Therefore, the pupil image learning apparatus for generating a pupil image discrimination model for discriminating a pupil image is configured to include a first image feature quantity calculation unit and a pupil image discrimination model generation unit.

According to such a configuration, in the pupil image learning device, the first image feature amount calculation unit calculates the image feature amount related to the edge for each of the pupil image and the non-pupil image included in the learning image set.
The pupil image learning device generates a pupil image discrimination model by machine learning of the image feature amount of the pupil image and the image feature amount of the non-pupil image by the pupil image discrimination model generation unit. Since this pupil image discrimination model learns the image feature quantity related to the edge, it is possible to discriminate the pupil image with high accuracy from a shot image obtained by shooting a person with one shooting camera.

  Further, in view of the above-described problems, the pupil position detection device according to the present invention detects the pupil position of a person included in the captured image using the pupil image discrimination model generated by the pupil image learning device according to the present invention. The pupil position detection device includes a pupil candidate area setting unit, a second image feature amount calculation unit, a pupil image determination unit, and a pupil position calculation unit.

According to such a configuration, the pupil position detection device is smaller than the face area in the face image obtained by cutting out the face area of the person included in the photographed image photographed by one photographing camera by the pupil candidate area setting unit. Set the size pupil candidate area.
In the pupil position detection device, the second image feature amount calculation unit calculates an image feature amount related to the edge for the pupil candidate region.

Further, the pupil position detection device determines whether or not the pupil candidate region is a pupil image by inputting the image feature amount of the pupil candidate region to the pupil image determination model by the pupil image determination unit. Since this pupil image discrimination model learns the image feature quantity related to the edge, it is possible to discriminate the pupil image with high accuracy from a shot image obtained by shooting a person with one shooting camera.
Then, the pupil position detection device calculates the pupil position included in the candidate pupil region determined as the pupil image by the pupil position calculation unit.

According to the present invention, the following excellent effects can be obtained.
The pupil image learning device according to the present invention generates a pupil image discrimination model in which the image feature quantity related to the edge is learned, and the pupil position detection device according to the present invention is based on a photographed image obtained by photographing a person with one photographing camera. The pupil position is detected using this pupil image discrimination model. Thus, according to the present invention, even when a high-resolution pupil magnified image cannot be captured, the pupil position can be detected with a simple configuration without requiring a plurality of imaging cameras.

It is a block diagram which shows the structure of the pupil detection apparatus which concerns on embodiment of this invention. (A) is explanatory drawing explaining selection of a pupil area | region, (b) is explanatory drawing explaining the production | generation of a pupil image. (A) is explanatory drawing explaining selection of face areas other than a pupil, (b) is explanatory drawing explaining the production | generation of a non-pupil image. (A) And (b) is explanatory drawing explaining calculation of a multiscale edge direction histogram. (A) And (b) is explanatory drawing explaining calculation of a multiscale edge connection angle histogram. It is explanatory drawing explaining the setting of a left pupil candidate area | region and a right pupil candidate area | region. It is explanatory drawing explaining the slide of a window area | region. (A) And (b) is explanatory drawing explaining calculation of the pupil position in a normalized face area image. (A) And (b) is explanatory drawing explaining conversion of the pupil position on a picked-up image. It is a flowchart of a pupil image learning process. It is a flowchart of a pupil position detection process.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In addition, the same code | symbol was attached | subjected to the means which has the same function, and description was abbreviate | omitted.

[Configuration of pupil detection device]
With reference to FIG. 1, the structure of the pupil detection apparatus 1 which concerns on embodiment of this invention is demonstrated.
The pupil detection device 1 generates a pupil image discrimination model used for discrimination of a pupil image, and detects a pupil position included in the captured image using the pupil image discrimination model. As shown in FIG. 1, the pupil detection device 1 includes a pupil image learning device 2, a pupil image discrimination model storage unit 3, and a pupil position detection device 4.

<Pupil image learning device>
The pupil image learning device 2 generates a pupil image discrimination model for discriminating a pupil image from a learning image set described later. As shown in FIG. 1, the pupil image learning device 2 includes a learning image set generation unit 20 and a pupil image learning unit 22.

The learning image set generation unit 20 generates a learning image set from a face image set input from the outside.
The face image set is a set of face images obtained by photographing a person's face area. In order to improve the accuracy of the pupil image discrimination model, it is preferable that this face image set includes face images of various persons and face images having different face directions even for the same person.
The learning image set is a set of a pupil image obtained by cutting out a person's pupil area and a non-pupil image obtained by cutting out a face area other than the person's pupil.

  For example, the learning image set generation unit 20 converts each face image included in the input face image set to a preset size (width W1 × height H1). Next, the learning image set generation unit 20 displays the converted face image on a display (not shown), and is a rectangle having a preset size (width R × height R) as shown in FIG. The user of the pupil image learning device 2 is made to select the shaped pupil region 90. Then, the learning image set generation unit 20 cuts out the selected pupil region 90 as a pupil image, as shown in FIG.

  Further, as shown in FIG. 3A, the learning image set generation unit 20 uses a face region (non-pupil region) 91 other than the pupil having the same size as the pupil region 90 from the face image displayed on the display as a pupil image. The user of the learning device 2 is selected. Then, the learning image set generation unit 20 cuts out the selected non-pupil region 91 as a non-pupil image, as shown in FIG.

Thereafter, the learning image set generation unit 20 outputs the set of cut out pupil images and non-pupil images to the pupil image learning unit 22 as a learning image set.
The learning image set generation unit 20 adds identification information indicating correct data to a pupil image and performs identification information indicating incorrect data to a non-pupil image in order to perform supervised machine learning. May be added.
Further, the learning image set generation unit 20 may be outside the pupil image learning device 2, and the pupil image learning device 2 may capture the learning image set from the outside.

  The pupil image learning unit 22 learns the learning image set input from the learning image set generation unit 20, and includes an image feature amount calculation unit (first image feature amount calculation unit) 24, a pupil image discrimination model, and the like. And a generation unit 26.

  The image feature amount calculation unit 24 calculates an image feature amount related to an edge (for example, a known edge direction histogram) for each of the pupil image and the non-pupil image included in the learning image set.

Here, in order to accurately detect the pupil position, it is preferable to use an image feature amount that places importance on the shape such as a circle surrounding the pupil and an eye contour. Therefore, the image feature amount calculation unit 24 calculates a multiscale edge histogram as the image feature amount related to the edge.
This multi-scale edge histogram is a histogram of edges extended at different scales, and includes a multi-scale edge direction histogram and a multi-scale edge connection angle histogram.

As shown in FIG. 4A, the multi-scale edge direction histogram shows an edge 93 of scale n when not only the local edge 93 of the pixel 92 separated by K pixels but also the pixel 92 separated by Kn pixels are continuous. Treat as. As shown in FIG. 4B, this multi-scale edge direction histogram shows the frequency of appearance of the edge 93 (number in the image) for each edge direction θ from 0 to 179 degrees from scale 0 to scale N-1. Represents.
N is the number of multiplexed scales and satisfies 1 ≦ n ≦ N.

  As shown in FIG. 5A, the multi-scale edge connection angle histogram is an edge in which a pixel 92B serving as an end point exists within an edge 93A and a circle having a radius R centering on the pixel 92A serving as the end point of the edge 93A. This is based on the connection angle φ with 93B. As shown in FIG. 5B, this multi-scale edge connection angle histogram shows the frequency of appearance of the connection angle φ between the edges 93 for each angle in the direction of 1 degree to 180 degrees from the scale 0 to the scale N-1. The number in the middle.

  Here, the image feature amount calculation unit 24 may calculate only one of the multiscale edge direction histogram or the multiscale edge connection angle histogram as the image feature amount related to the edge. In addition, the image feature amount calculation unit 24 may calculate both the multiscale edge direction histogram or the multiscale edge connection angle histogram, and handle them as multidimensional image feature amounts having these as elements.

Note that the multiscale edge direction histogram and the multiscale edge connection angle histogram are described in detail in the following references, and thus further description thereof is omitted.
References: Application of similar image search technology to the earthquake video archives search system, Takahiro Mochizuki et al., ITE Annual Convention 2014

  The pupil image discrimination model generation unit 26 generates a pupil image discrimination model by machine learning of the image feature amount of the pupil image and the image feature amount of the non-pupil image calculated by the image feature amount calculation unit 24. The pupil image discrimination model generation unit 26 can use known machine learning, and preferably performs supervised machine learning such as a support vector machine or a neural network. Then, the pupil image discrimination model generation unit 26 writes the generated pupil image discrimination model in the pupil image discrimination model storage unit 3.

<Pupil image discrimination model storage unit>
The pupil image discrimination model storage unit 3 is a storage device such as a memory or a hard disk (HDD: Hard Disk Drive) that stores the pupil image discrimination model generated by the pupil image discrimination model generation unit 26.
The pupil image discrimination model storage unit 3 may be provided in the pupil image learning device 2 or in the pupil position detection device 4.

<Pupil position detector>
The pupil position detection device 4 detects the pupil position of a person included in the captured image using the pupil image discrimination model stored in the pupil image discrimination model storage unit 3. As shown in FIG. 1, the pupil position detection device 4 includes a pupil candidate area setting unit 40 and a pupil position detection unit 42.

The pupil candidate area setting unit 40 sets a pupil candidate area having a size smaller than the face area in a face image obtained by cutting out a face area of a person included in a photographed image photographed by one photographing camera C. is there. This photographed image is a frame image constituting a photographed video (moving image).
That is, the pupil position detection device 4 can detect and track the pupil position of the captured video when the processing described below is performed on each frame image.

  First, the pupil candidate area setting unit 40 applies a known face area detection process (for example, OpenCV) to a photographed image input from a general photographing camera C, thereby obtaining a rectangular face area included in the photographed image. The detected face area is cut out to generate a face area image. The upper left coordinate of the face area included in the captured image is (x_f, y_f), and the size of the face area included in the captured image is (width w_f × height h_f).

  Next, the pupil candidate area setting unit 40 generates a normalized face area image 94 by converting the generated face area image into a preset size (width W_f × height H_f) as shown in FIG. To do. Then, the pupil candidate area setting unit 40 sets the left and right pupil candidate areas 95 (the left pupil candidate area 95L and the right pupil candidate area 95R) in the normalized face area image 94. Thereafter, the pupil candidate area setting unit 40 outputs the set left and right pupil candidate areas 95 to the pupil position detection unit 42.

  Here, as shown in FIG. 6, the upper left coordinates, width, and height of the left pupil candidate region 95L and the right pupil candidate region 95R are defined by the following equations. That is, the left pupil candidate region 95L and the right pupil candidate region 95R have the reduced size of the normalized face region image 94 while maintaining the aspect ratio as it is. Note that the coefficients R1, R2, R3, and R4 are preset with values of 0 or more and less than 1.

Upper left coordinates of right pupil candidate area (x_r, y_r) = (R1 x W_f, R3 x H_f)
Width of right pupil candidate area W_r ＝ R2 × W_f
Height of right pupil candidate area H_r ＝ R4 × H_f
Upper left coordinates of candidate left pupil region (x_l, y_l) = ((1-R1-R2) x W_f, R3 x H_f)
Left pupil candidate area width W_l ＝ R2 × W_f
Height of left pupil candidate area H_l = R4 × H_f

  The pupil position detection unit 42 detects a pupil position from the left and right pupil candidate regions 95 input from the pupil candidate region setting unit 40, and includes an image feature amount calculation unit (second image feature amount calculation unit) 44 and A pupil image determination unit 46 and a pupil position calculation unit 48.

  The image feature amount calculation unit 44 calculates an image feature amount related to the edge for the left and right pupil candidate regions. In the present embodiment, the image feature amount calculation unit 44 sets a window region having the same size (width R × height R) as the learning image set as the left and right pupil candidate regions 95. Then, the image feature quantity calculation unit 44 calculates the same image feature quantity as the image feature quantity calculation unit 24 for each window region set at a different position.

  Specifically, the image feature amount calculation unit 44 sets a window region 96 in the right pupil candidate region 95R and calculates the image feature amount of the window region 96 as shown in FIG. Next, the image feature amount calculation unit 44 slides the window region 96 (illustrated by a broken line), and then calculates the image feature amount of the window region 96 at that position. As described above, the image feature amount calculation unit 44 repeats the sliding of the window region 96 and the calculation of the image feature amount until the image feature amount is calculated from the entire range of the right pupil candidate region 95R.

Note that the image feature amount calculation unit 44 can arbitrarily set the slide amount of the window region 96.
In addition, the image feature amount calculation unit 44 can arbitrarily set the sliding method of the window region 96. For example, the image feature amount calculation unit 44 may slide the window area 96 from the upper left to the lower right of the right pupil candidate area 95R like a raster scan.
The image feature amount calculation unit 44 calculates the image feature amount of the window region 96 for the left pupil candidate region 95L as well as the right pupil candidate region 95R.

  The pupil image determination unit 46 inputs the image feature amount of each window region 96 calculated by the image feature amount calculation unit 44 for the left and right pupil candidate regions 95 to the pupil image determination model, so that each window region 96 is a pupil image. It is discriminate | determined whether it is.

  Here, when the image feature amount of the window area 96 is input to the pupil image discrimination model, the probability that the window area 96 is a pupil image is output from the pupil image discrimination model. Accordingly, the pupil image determination unit 46 determines whether each window region 96 is a pupil image by threshold processing of the probability output by the pupil image determination model. For example, if the probability from the pupil image discrimination model is equal to or greater than a preset threshold value, the pupil image discrimination unit 46 discriminates that the window region 96 is a pupil image. On the other hand, if the probability from the pupil image discrimination model is less than the threshold value, the pupil image discrimination unit 46 discriminates that the window region 96 is not a pupil image.

The pupil position calculation unit 48 calculates the pupil position of the window region 96 that has been determined to be a pupil image by the pupil image determination unit 46 for the left and right pupil candidate regions 95.
In the present embodiment, as shown in FIG. 8A, the pupil position calculation unit 48 obtains the center coordinates of all window regions 96 that are determined to be pupil images for the left and right pupil candidate regions 95. Then, as shown in FIG. 8B, the pupil position calculation unit 48 calculates the average value of the center coordinates obtained from the right pupil candidate area 95R as the right pupil position 97R in the normalized face area image 94. Further, the pupil position calculation unit 48 calculates the average value of the center coordinates obtained from the left pupil candidate area 95L as the left pupil position 97L in the normalized face area image 94.

  Next, as shown in FIG. 9, the pupil position calculation unit 48 converts the right pupil position 97R in the normalized face region image 94 into the coordinates of the captured image 98, and outputs the converted coordinates as the right pupil position 99R. In addition, the pupil position calculation unit 48 converts the left pupil position 97L in the normalized face region image 94 into the coordinates of the captured image 98, and outputs the converted coordinates as the left pupil position 99L.

  Here, as shown in FIG. 9, the upper left coordinates, width, and height of the left pupil candidate region 95L and the right pupil candidate region 95R are defined by the following equations. The coordinates of the right pupil position 97R are (X_pr, Y_pr), and the coordinates of the left pupil position 97L are (X_pl, Y_pl). Further, the coordinates of the right pupil position 99R are (x_pr, y_pr), and the coordinates of the left pupil position 99L are (x_pl, y_pl).

x_pr = x_f + (w_f / W_f) x X_pr
y_pr = y_f + (h_f / H_f) x Y_pr
x_pl = x_f + (w_f / W_f) x X_pl
y_pl = y_f + (h_f / H_f) x Y_pl

[Pupil image learning processing]
The pupil image learning process by the pupil image learning device 2 will be described with reference to FIG. 10 (see FIG. 1 as appropriate).

As shown in FIG. 10, in the pupil image learning apparatus 2, a face image set is input from the outside to the learning image set generation unit 20 (step S1).
In the pupil image learning device 2, the learning image set generation unit 20 generates a pupil image obtained by cutting out the pupil region of the person (step S2).
In the pupil image learning apparatus 2, the learning image set generation unit 20 generates a non-pupil image obtained by cutting out a non-pupil region of a person (step S3).

In the pupil image learning device 2, the image feature amount calculation unit 24 calculates the image feature amount related to the edge for each of the pupil image and the non-pupil image (step S4).
The pupil image learning device 2 generates a pupil image discrimination model by machine learning of the image feature amount of the pupil image and the image feature amount of the non-pupil image by the pupil image discrimination model generation unit 26 (step S5).

[Pupil position detection processing]
With reference to FIG. 11, the pupil position detection process by the pupil position detection apparatus 4 will be described (see FIG. 1 as appropriate).

As shown in FIG. 11, the pupil position detection device 4 receives a photographed image from one photographing camera C to the pupil candidate area setting unit 40 (step S10).
In the pupil position detection device 4, the pupil candidate area setting unit 40 detects a rectangular face area included in the captured image, cuts out the detected face area, and generates a face area image (step S11).
In the pupil position detection device 4, the pupil candidate area setting unit 40 generates a normalized face area image having a preset size from the face area image, and sets the left and right pupil candidate areas in the normalized face area image ( Step S12).

In the pupil position detection device 4, the image feature amount calculation unit 44 sets window regions in the left and right pupil candidate regions, and calculates image feature amounts of the window regions (step S13).
In the pupil position detection device 4, the pupil image discrimination unit 46 inputs the image feature amount of each window area to the pupil image discrimination model, and calculates the probability that each window area is a pupil image (step S14).

The pupil position detection device 4 determines whether each window region is a pupil image by the threshold value processing of the probability obtained by the pupil image determination model by the pupil image determination unit 46 (step S15).
The pupil position detection device 4 calculates, as the pupil position, the average value of the center coordinates of all the window regions that are determined to be pupil images by the pupil image determination unit 46 (step S16).

[Action / Effect]
In the pupil detection device 1 according to the embodiment of the present invention, the pupil image learning device 2 generates a pupil image discrimination model in which the image feature amount related to the edge is learned, and the pupil position detection device 4 uses this pupil image discrimination model. Thus, the pupil image is discriminated from the photographed image in which the person is photographed by one photographing camera C. Therefore, the pupil detection device 1 can detect the pupil position with a simple configuration without requiring a plurality of photographing cameras C even when a high-resolution pupil magnified image cannot be captured.

  Further, in the pupil detection device 1, the pupil image learning device 2 and the pupil position detection device 4 connect long edges and edges as image feature quantities related to edges, such as a multiscale edge direction histogram and a multiscale edge connection angle histogram. Since the weighted image feature is used in the method, the pupil position can be detected more accurately.

  Further, the pupil detection device 1 calculates the center coordinates of all window regions that are determined to be pupil images, and the pupil position detection device 4 calculates the average value of the determined center coordinates as the pupil position. Can be detected more accurately.

  In the above-described embodiment, the pupil position detection device 4 has been described as independent hardware, but the present invention is not limited to this. For example, hardware resources such as a CPU, a memory, and a hard disk included in the computer can also be realized by a pupil position detection program that performs cooperative operation as the pupil position detection device 4. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

1 pupil detection device 2 pupil image learning device 3 pupil image discrimination model storage unit 4 pupil position detection device 20 learning image set generation unit 22 pupil image learning unit 24 image feature amount calculation unit (first image feature amount calculation unit)
26 pupil image discrimination model generation unit 40 pupil candidate region setting unit 42 pupil position detection unit 44 image feature amount calculation unit (second image feature amount calculation unit)
46 Pupil image discrimination unit 48 Pupil position calculation unit

Claims (6)

A pupil that generates a pupil image discrimination model that discriminates the pupil image from a learning image set that is a set of a pupil image obtained by cutting out a human pupil region and a non-pupil image obtained by cutting out a face region other than the human pupil. An image learning device,
A first image feature amount calculating unit that calculates an image feature amount related to an edge for each of a pupil image and a non-pupil image included in the learning image set;
A pupil image discrimination model generation unit that generates the pupil image discrimination model by machine learning of the image feature amount of the pupil image and the image feature amount of the non-pupil image;
A pupil image learning apparatus comprising:   The pupil image learning according to claim 1, wherein the first image feature amount calculation unit calculates a multi-scale edge histogram that is a histogram of edges extended with different scales as the image feature amount related to the edge. apparatus. A pupil position detection device that detects a pupil position of a person included in a captured image using the pupil image discrimination model generated by the pupil image learning device according to claim 1,
A pupil candidate area setting unit that sets a pupil candidate area having a size smaller than the face area in a face image obtained by cutting out the face area of the person included in the captured image;
A window region having the same size as the learning image set is set as the pupil candidate region, and a second image feature amount calculation unit that calculates an image feature amount related to an edge for the window region;
A pupil image determination unit that determines whether or not the window region is the pupil image by inputting an image feature amount of the window region to the pupil image determination model;
A pupil position calculation unit that calculates a pupil position included in the window region determined to be the pupil image;
A pupil position detection device comprising: The second image feature amount calculation unit sets the window region at a different position of the pupil candidate region, calculates an image feature amount related to the edge for each window region,
The pupil image determination unit determines whether each window region is the pupil image,
The said pupil position calculation part calculates | requires the center coordinate of all the window areas discriminate | determined as the said pupil image, and calculates the average value of the calculated | required center coordinate as said pupil position. The pupil position detection apparatus described.   The said 2nd image feature-value calculation part calculates the multi-scale edge histogram which is a histogram of the edge extended by the different scale as an image feature-value regarding the said edge, The Claim 3 or Claim 4 characterized by the above-mentioned. Pupil position detection device.   The pupil position detection program for functioning a computer as a pupil position detection apparatus as described in any one of Claims 3-5.

Images