CN106909220A - A kind of sight line exchange method suitable for touch-control - Google Patents

Abstract

The invention discloses a line-of-sight interaction method suitable for touch, and belongs to the technical field of video and multimedia signal processing. This method inputs the image of the human eye region into the CNN, and performs feature extraction and dimensionality reduction on the image through convolution and downsampling operations, which improves the limitations of the traditional line of sight estimation method using high-definition cameras and external infrared equipment to construct eyeball models. In the image acquisition stage, through the image acquisition of different individuals at different locations and at different times, the human eye database has the diversity of illumination, skin color and shape, so that the system can meet the requirements of various conditions of different individuals. The classification accuracy is greatly improved.

Description

A visual interaction method suitable for touch

technical field

The invention relates to a line-of-sight interaction method suitable for touch control, and belongs to the technical field of video and multimedia signal processing.

technical background

Human-computer interaction technology is one of the fastest-growing fields in user interface research. At present, the more common human-computer interaction methods include traditional interaction methods such as keyboards, mice, and cameras. However, with the continuous development of artificial intelligence technology, many non-contact interaction methods have emerged, such as gesture recognition. Compared with the traditional contact interaction method, the non-contact interaction method has the advantages of safety and sanitation, and is more and more used in all aspects of life.

The line-of-sight interaction method involved in the present invention is a kind of non-contact human-computer interaction. The so-called line-of-sight interaction, as the name implies, is a technology that realizes human-computer interaction by estimating the gaze direction of human eyes, and its core technology is the line-of-sight estimation method. Eye-sight interaction is widely used in virtual reality, developmental psychology, medical diagnosis, perception analysis, commercial advertisement evaluation and other fields. At the same time, eye-sight interaction is also a good aid for patients with severe paralysis but good vision. They can express their wishes and needs through eye movements, and control the corresponding system to meet their needs.

The traditional model-based gaze estimation technology obtains the specific positions of the pupil center and corneal reflection point through high-resolution cameras and infrared light sources to establish a mathematical model of the eyeball, and obtains the position of the gaze point of the human eye through a certain mapping relationship. This method has high requirements on hardware equipment, is easily affected by the lighting environment, and requires a calibration process, which sets a high threshold for line-of-sight interaction. The line of sight estimation based on human eye images adopts the method of machine learning. Firstly, the human eye image database is established, and the neural network is used to reduce the dimensionality and feature extraction of the human eye image features, and establish a corresponding relationship between the human eye images and the gaze direction, so as to realize Estimated from the gaze direction of the human eye image. However, such methods aim to obtain accurate gaze direction, and in the training process of machine learning, regression methods are often used; this makes the training process of this type of method complex, the adjustment of network parameters is difficult, and the accuracy of gaze direction obtained Relatively poor. At the same time, most of the human eye image databases used in this type of method are not collected under the scene of gazing at the screen, so the implementation in the screen-based touch application is not ideal.

Contents of the invention

Touch-based human-computer interaction is the most common human-computer interaction method in consumer electronics products. It activates corresponding functions by triggering touch buttons with a certain area on a screen with a limited area. The present invention refers to the characteristics of the touch technology, and aims at the shortcomings of the traditional eyeball model-based and human eye image-based line-of-sight estimation methods that are difficult to apply to touch-control applications. The present invention provides a line-of-sight interaction method suitable for touch. This method divides the gazing screen into blocks, uses the blocks in the screen to simulate the buttons in the touch interface, and innovatively models the relationship between the human eye images gazing at different blocks of the screen and their corresponding area positions. The establishment of the database and the application of the convolutional neural network can estimate the block on the screen that the human eye is looking at according to the human eye image, and realize the touch based on the line of sight interaction. In this process, since it is not necessary to estimate the exact direction of gaze, only the block of gaze needs to be estimated, so the pattern classification method rather than the regression method needs to be used in machine learning. Experiments show that this method can achieve high recognition under different lighting conditions and different individual conditions.

The realization of this method needs a common network camera, and a personal computer. The technical scheme adopted in the present invention is as follows.

A line-of-sight interaction method suitable for touch, which is characterized in that several human eye images of different individuals looking at the same screen position block under different external conditions are classified into one category, and the convolutional neural network (CNN) is used to realize pattern classification, thereby passing The human eye image recognizes its corresponding screen gaze position block, and the method comprises the following steps:

(1) The establishment of the human eye database: under different lighting, different time, and different position conditions, data collection is performed on the process of multiple individuals gazing at randomly appearing fixation points on the screen, and face detection and human eye detection are performed on the collected images , obtain the human eye area image, establish the human eye database and screen the human eye image;

(2) Divide the human eye image into a training set and a verification set, as the input of the convolutional neural network, divide the computer screen into blocks according to the actual application requirements, and regard the human eye images of the same block on the screen as one class, and Classify human eye images and train convolutional neural networks;

(3) When performing line of sight estimation, input the human eye image to be classified into the trained convolutional neural network model to determine its category, and obtain the corresponding screen gaze position block, thereby estimating the line of sight direction.

More specifically, the specific implementation steps of the above step (1) are:

a, human eye image acquisition, the specific steps are as follows:

(a) Considering the practical application, at first the computer screen is divided into several large blocks, and the inside of each large block is further divided into several small blocks, and a gaze point is set in the middle of each small block;

(b) The experimenter sits within the range of the camera in front of the screen, and his eyes move with the gaze point. At the same time, the camera collects face images. In order to prevent eye fatigue, a time interval is set between two adjacent gaze points. At this time, the camera does not collect images;

(c) Screening the collected images to extract available sample data;

b. Perform face detection and human eye detection on the face image, and normalize the human eye rectangle to a uniform size for training the convolutional neural network.

More specifically, the specific implementation steps of the above step (2) are:

a. In terms of data selection, the available samples are divided into a training set and a verification set according to a certain proportion, which are used for convolutional neural network training and verification of classification accuracy;

b. Set the convolutional neural network structure model according to the actual data size as:

(a) The convolutional layer of the convolutional neural network: the network extracts the depth features of the image through the convolutional layer, selects the corresponding convolution kernel according to the size of the feature map and expands the edge of the image, the jth feature of the ith layer of the convolutional layer The value of the graph at position (x,y) is:

Among them, relu( ) is a modified linear unit, and its formula is: g(x)=max(0,x), b _ij is the bias of the jth feature map of the i-th layer, and n refers to the current feature map. The set of feature maps of the previous layer, p _i , q _i respectively refer to the length and width of the i-th layer convolution kernel, is the value of the convolution kernel connected to the upper feature map at (p,q);

(b) The sampling layer of the convolutional neural network: the network performs dimensionality reduction processing on the image through the sampling layer, and the jth feature map of the i-th layer of the sampling layer is expressed as

f _ij ＝f(β _ij down(f _(i-1)j )+b _ij )

Among them, β _ij and b _ij are the multiplier deviation and additional deviation of the jth feature map of the i-th layer respectively, down(·) is the downsampling function, adopts the maximum pooling, connects the LRN layer after the downsampling layer, and the LRN layer By imitating the lateral inhibition mechanism of the biological nervous system, a competition mechanism is created for the activities of local neurons, so that the response is relatively large, and the generalization ability of the model is improved.

(c) The output layer of the convolutional neural network: the network realizes full connection through the inner product layer, and finally outputs the category number.

The above-mentioned classification mechanism based on the convolutional neural network is to input the known category of human eye images into the configured CNN network to calculate the network loss until the loss continues to decrease and tends to be stable. After the network training is completed, the verification set is input into the network to calculate the probability value of each category, and the one with the highest probability value is the final category.

The line of sight estimation technology based on the convolutional neural network directly takes the human eye image as input, and extracts features through learning for classification, avoiding the artificial feature extraction and eyeball modeling stages, and converting the high-dimensional human eye image into a low-dimensional feature map. The classification performance is improved and the complexity of the experiment is reduced. The diversity of the training set is increased by training on different human eye images, so that the classification results are suitable for different individuals. The higher accuracy in multi-classification tasks verifies that the present invention has certain practical value, and provides a new way for line-of-sight estimation technology.

Description of drawings

Fig. 1 is a system hardware diagram;

Fig. 2 is and data acquisition process diagram;

Fig. 3 is a block diagram of the line of sight estimation system based on CNN;

Fig. 4 is a schematic diagram of extraction of face area and human eye area;

Fig. 5 is a convolutional neural network structure diagram;

Fig. 6 is a graph of 6 classification problem training set loss and verification set accuracy;

Fig. 7 is a curve diagram of 54 classification problem training set loss and verification set accuracy;

Figure 8 is a feature map of each layer of the network.

detailed description

Below in conjunction with accompanying drawing, invention is further described:

The present invention provides a new method of line of sight estimation based on convolutional neural network. According to the method proposed by the present invention, the collection of face data is first carried out. The hardware facilities required for the experiment include an ordinary personal computer and a webcam. During the experiment, the camera was placed in the middle of the top of the computer screen. Figure 1 shows the hardware diagram of the system, including a personal computer and a webcam. In the experiment, the screen is first divided into 6 large blocks, and each large block is then divided into 9 small blocks, and a fixation point is set in each small block. As shown in Figure 2, during the experiment, the tester sat 50-60cm in front of the screen, kept his head still, and his eyes moved with the movement of the gaze point on the screen. At the same time, the camera collects face images. Figure 3 shows the flow chart of the entire line of sight estimation system. As shown in the flowchart, we take an individual in the database as an example to introduce in detail the implementation of the invented line-of-sight estimation method based on convolutional neural network. The present invention comprises following specific steps:

1. Extract the face area and human eye area

(1) Figure 4 shows the face region extraction process. Figure 4(a) detects the face region on the original color image. The experiment uses Haar features for face detection. The Haar feature classifier of the face is an XML file, which will describe the Haar feature value of the face, and determine the different areas of the face through the feature values of different areas of the face.

(2) In the extracted face area, further use Haar human eye detection to extract the left and right eyes. Figure 4(b) is the detection of human eyes in the detected face area, and the detected human eye image is shown in Figure 4(c).

(3) The extracted human eye image is subjected to size normalization processing, and the human eye rectangular area is normalized to 40*72 pixels according to the experimental needs.

2. Training set and validation set processing

The experiment selects 181,440 left-eye images in the above step (3), and selects the training set and the verification set according to a certain ratio. Convert images and corresponding initial labels to LMDB format for convolutional neural network training.

3. Build a CNN network

CNN network training uses the Caffe deep learning framework. Caffe is a clear and efficient deep learning framework. Caffe is a pure C++/CUDA architecture that supports command line, Python and MATLAB interfaces; it can seamlessly switch directly between CPU and GPU. The experiment uses GPU training, which greatly saves training time. Figure 5 shows the structure diagram of convolutional neural network for classification. It includes several convolutional layers, maximum pooling layers, fully connected layers to achieve full connections, and several other activation functions. The number of layers in each layer is also given in Figure 5 as an example. The number of layers of each layer can be determined by experiments in actual use.

(1) Convolutional layer of CNN: The network extracts the deep features of the image through the convolutional layer. Select the corresponding convolution kernel according to the size of the feature map and perform edge expansion on the image. The value of the j-th feature map in the i-th layer of the convolutional layer at the (x, y) position is:

Wherein, relu(·) is a rectified linear unit (Rectified linear unit), and its formula is: g(x)=max(0,x). b _ij is the bias of the jth feature map of the i-th layer, n refers to the set of feature maps of the previous layer connected to the current feature map, p _i , q _i refer to the length and width of the convolution kernel of the i-th layer, respectively, is the value of the convolution kernel connected to the upper feature map at (p,q).

(2) Sampling layer of CNN: The network performs dimensionality reduction processing on the image through the sampling layer, and the jth feature map of the i-th layer of the sampling layer is expressed as:

f _ij ＝f(β _ij down(f _(i-1)j )+b _ij )

Among them, β _ij and b _ij are the multiplier bias and additional bias of the jth feature map in the i-th layer, respectively, and down(·) is the downsampling function, using max pooling. Connect the LRN layer after the downsampling layer. The LRN layer imitates the lateral inhibition mechanism of the biological nervous system, and creates a competition mechanism for the activities of local neurons, so that the response value is relatively large, and the generalization ability of the model is improved.

(3) Output layer of CNN: The feature dimensions of the fully connected layer are respectively selected to be 256 and 256, and finally the number of categories is set to realize the classification tasks of 6 major categories and 54 subcategories.

4. Configure the network model and other required parameters, and train the 6-category problem and 54-category classification network respectively.

(1) Analyze the output loss and accuracy of the training process and verification process to determine the network convergence and classification accuracy. Figure 6 and Figure 7 respectively show the loss curves of the training set and the verification set corresponding to the 6-class classification problem and the 54-class classification problem, and the classification accuracy graph of the verification set. In Figure 6 and Figure 7, the dotted line represents the value of the loss function during training, the triangle line represents the value of the loss function during testing, and the star line represents the estimated accuracy y obtained during testing. As the number of iterations increases, the loss continues to decrease until it converges, and at this time, the classification accuracy of the validation set reaches the highest until it is stable. By adjusting the network configuration parameters to determine the optimal network is the classification network we need.

(2) Visualize each layer of the convolutional neural network. Taking a certain human eye image of the 6-classification problem as an example, the feature maps of the CNN network are shown in Figure 8. Figure 8(a) shows the feature maps and corresponding convolution weights output by each convolutional layer and downsampling layer, and Figure 8(b) shows the output features and their histograms after the fully connected layer processing and the final category of . It can be seen from the final probability map that the category with the highest probability is the category of the input image.

(3) The human eye image to be classified only needs to be input into the obtained multi-classification network model to judge its category. By associating the category corresponding to the human eye image with the screen position, the general line of sight direction corresponding to the human eye image can be determined. Experimental results show that the accuracy of 6-category classification of images can reach 93%, and the accuracy of 54-category classification can reach 83%.

Claims (3)

1. a kind of sight line exchange method suitable for touch-control, it is characterised in that：Different Individual is watched attentively under different external conditions Some eye images of same screen position block are classified as a class, are classified with convolutional neural networks CNN implementation patterns, so as to pass through people Eye image recognition its corresponding screen watches block of locations attentively, and the method comprises the following steps：

(1) foundation of human eye database：Under the conditions of different illumination, different time, diverse location attentively screen is watched to multiple individualities The process of the upper random blinkpunkt for occurring carries out data acquisition, and the image to gathering carries out Face datection and human eye detection, obtains Human eye area image, sets up human eye database and eye image is screened；

(2) eye image is divided into training set and checking collects, as the input of convolutional neural networks, according to practical application request pair Computer screen carries out piecemeal, and the eye image for watching same piecemeal on screen attentively is considered as a class, and eye image is classified, training Convolutional neural networks；

(3) when carrying out sight line estimation, eye image to be sorted is input in the convolutional neural networks model for training, you can sentence Disconnected its classification, and correspondingly obtaining corresponding screen watches block of locations attentively, so as to estimate direction of visual lines.

2. the sight line exchange method suitable for touch-control according to claim 1, it is characterised in that：The tool of above-mentioned steps (1) Body realizes that step is：

A, eye image collection, it is comprised the following steps that：

A () considers practical application, computer screen is divided into some bulks first, continues to be divided into several inside each bulk Fritter, each fritter middle setting blinkpunkt；

B () experimenter is sitting in the range of the camera of screen front, eyes follow blinkpunkt to move, simultaneous camera collection face figure Picture, in order to prevent people's kopiopia, two neighboring blinkpunkt sets interval, and now camera does not gather image；

C () screens to the image for gathering, extract usable samples data；

B, Face datection and human eye detection are carried out to facial image, and by human eye rectangle normalize to unified size for train volume Product neutral net.

3. the sight line exchange method suitable for touch-control according to claim 1, it is characterised in that：The tool of above-mentioned steps (2) Body realizes that step is：

A, data selection aspect, usable samples are divided into training set by a certain percentage and checking collection is respectively used to convolutional neural networks Training and the checking of classification accuracy；

B, according to real data size set convolutional neural networks structural model be：

The convolutional layer of (a) convolutional neural networks：Network extracts the depth characteristic of image by convolutional layer, is selected according to characteristic pattern size Select corresponding convolution kernel and edge expansion is carried out to image, value of i-th layer of j-th characteristic pattern of convolutional layer at (x, y) position is：

$f_{ij}^{xy}=relu(b_{ij}+\underset{n}{\Σ}\underset{p=0}{\overset{p_i-1}{\Σ}}\underset{q=0}{\overset{q_i-1}{\Σ}}{w}_{ijn}^{pq}{f}_{(i-1)n}^{(x+p)(y+q)})$

Wherein, relu () is amendment linear unit, and its formula is：G (x)=max (0, x), b_ijIt is i-th layer of j-th characteristic pattern Biasing, n refers to the set of the last layer characteristic pattern being connected with current signature figure, p_i,q_iRefer to i-th layer of length and width of convolution kernel respectively,It is the value of the convolution kernel that is connected with upper strata characteristic pattern at (p, q) place；

The sample level of (b) convolutional neural networks：Network carries out dimension-reduction treatment to image by sample level, j-th of i-th layer of sample level Characteristic pattern is expressed as

f_ij=f (β_ijdown(f_(i-1)j)+b_ij)

Wherein, β_ijAnd b_ijIt is respectively i-th layer of multiplier deviation and additional deviation of j-th characteristic pattern, down () is down-sampled letter Number, using maximum pond, LRN layers of the connection behind down-sampling layer, the LRN layers of lateral inhibition mechanism of mimic biology nervous system, Activity to local neuron creates competition mechanism so that the larger value of response ratio is relatively bigger, improves the generalization ability of model.

The output layer of (c) convolutional neural networks：Network realizes full connection by interior lamination, finally exports classification number.