CN114882995A - Tongue picture constitution distinguishing method based on deep learning - Google Patents

Abstract

The tongue picture constitution distinguishing method based on deep learning is deployed on a mobile terminal, and is convenient for a user to recognize tongue picture images at any time and any place. The method adopts a convolution neural network algorithm to carry out data enhancement on the tongue picture data set in a mode of combining the similar enhancement method and the mixed enhancement method on the tongue picture. The model can adapt to tongue picture data in various environments, the extracted features can directly output classification results, the calculated amount is greatly reduced, and the generalization capability is stronger.

Description

A method for identifying tongue image constitution based on deep learning

technical field

The invention relates to the field of images, in particular to a method for identifying tongue image constitution based on deep learning.

Background technique

At this stage, the use of artificial intelligence technologies such as machine learning and deep learning for image recognition has become mature. Some scholars build a tongue image database by collecting a large number of tongue images with physical labels, and design an algorithm to match the tongue images with the established database, so as to achieve the purpose of identification. Considering the complexity of traditional manual extraction of tongue features and the accuracy of tongue region segmentation, deep learning has been applied to physical fitness recognition due to its advantages in automatic feature extraction. Some scholars have used various models of convolutional neural networks to perform tongue Physical identification. Although these studies have promoted the development of intelligent physical identification of tongue diagnosis, there are also some common problems. First, deep learning requires a large number of data sets as support, and it is difficult to collect sample data of tongue images. Second: simply using the model for classification does not improve and optimize the model, resulting in a low accuracy of physique identification using the above methods.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned problems, the application proposes a method for identifying tongue image constitution based on deep learning, specifically a method for identifying tongue image images based on an improved convolutional neural network GoogleNet model, in the collected tongue image data set. On the basis of using this method for constitution identification, the main contributions of this application are: constructing a tongue image dataset with constitution labels, adding batch normalization operations, improving and optimizing the Inception module, adding an attention mechanism, and combining different Inception structures Using methods such as TCR-GoogleNet, a GoogleNet network model (TCR-GoogleNet) suitable for tongue-like constitution recognition is proposed. An ablation experiment is designed to verify the contribution of each improved module to the prediction accuracy, and a comparison experiment is designed with five mainstream networks for image classification, which proves that the model proposed in this application has a great improvement in accuracy and stability.

The embodiment of the present invention provides the following technical solutions:

A method for distinguishing tongue image constitution based on deep learning, the method is deployed on a mobile terminal, and the method includes:

Obtain the tongue image to be recognized, use the trained improved convolutional neural network GoogleNet model to recognize the tongue image to be recognized, and obtain the tongue recognition result;

Display the recognition result of the tongue image on the mobile terminal;

Among them, the training of the improved convolutional neural network GoogleNet model is as follows:

Obtaining a tongue image data set, and performing enhancement processing on the tongue image data set to obtain a tongue image training data set, wherein the enhancement processing includes two methods of homogeneous enhancement and mixed-class enhancement;

Use the tongue image training data set to train the improved convolutional neural network GoogleNet model, and obtain the trained improved convolutional neural network GoogleNet model;

Among them, the improved part of the improved convolutional neural network GoogleNet model includes:

Each intermediate layer of the model is standardized, which effectively avoids gradient disappearance and gradient explosion, speeds up network convergence, and optimizes network structure;

The residual is integrated into the Inception structure, the Inception structure is improved and optimized, and the input features and the features output by the Inception structure are feature-fused, named Inception B;

The 3ⅹ3 convolution branch in the Inception module is decomposed into 1ⅹ3 and 3ⅹ1 convolution, the 5ⅹ5 convolution is decomposed into 3ⅹ3 convolution and 1ⅹ3, 3ⅹ1 convolution, and Batch Normalization is added after each branch, which effectively reduces the parameters of the model quantity, named Inception C;

On the basis of the original GoogleNet model, the unimproved Inception module is retained, named InceptionA, and the InceptionA, InceptionB and Inception C structures are integrated and applied to the improved GoogleNet network model, and the InceptionA module is added to the shallow layer. The InceptionB module is added to the middle layer, and the Inception C module is added to the deep layer. The A and B modules are used twice respectively. The C module has stronger ability to extract and perceive features, so it is used five times.

Among them, the same type of enhancement refers to the use of data normalization, horizontal or vertical flip, proportional enlargement or reduction of the image, random rotation, random stagger transformation, brightness change and pixel filling methods for the images of the same label to expand the dataset, and the color is also It is an important feature in the identification of tongue image constitution, so the color channel transformation of the tongue image data set is not performed.

Among them, the mixed-class enhancement method is to solve the problem of imbalance between sample data sets. The mixed-class enhancement method mixes two types of sample data with different labels to construct a virtual training sample to enrich the tongue image data set. The mathematical expression is as follows: (1) shows:

where (x _i , y _i )(x _j , y _j ) are two randomly selected samples and corresponding labels, λ is a random number obeying beta distribution, and λ∈[0,1].

Compared with the prior art, the above technical solution has the following advantages:

The deep learning-based tongue image constitution identification method described in this application is deployed on a mobile terminal, which is convenient for users to identify tongue image images anytime and anywhere. The method of the invention adopts the convolutional neural network algorithm, and performs data enhancement on the tongue image data set by combining two methods of homogeneous enhancement and mixed-class enhancement on the tongue image. The model can adapt to the tongue image data in various environments, the extracted features can directly output the classification results, the calculation amount is greatly reduced, and the generalization ability is stronger.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are For some embodiments of the present invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

Fig. 1 is a schematic flowchart of the method for distinguishing tongue image constitution based on deep learning described in the application;

Figure 2 shows the proposed method architecture;

Figure 3 is a sample image;

Fig. 4 is the image after data enhancement;

Figure 5 is the Mixup visualization;

Figure 6 shows the Inception B structure;

Figure 7 shows the Inception C structure;

Figure 8 is the SE module;

Figure 9 is the InceptionA structure;

Figure 10 is a structural diagram of a tongue-like constitution identification model;

Figure 11 shows the accuracy curve (1) and the loss value curve (2) in the training process;

Figure 12 shows the accuracy training curves of the four model test sets.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined in this application may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

As shown in FIG. 1 , the present application provides a method for identifying tongue image constitution based on deep learning, which is as follows:

Obtain the tongue image to be recognized, use the trained improved convolutional neural network GoogleNet model to recognize the tongue image to be recognized, and obtain the tongue recognition result;

Display the recognition result of the tongue image on the mobile terminal;

Among them, the training of the improved convolutional neural network GoogleNet model is as follows:

Obtaining a tongue image data set, and performing enhancement processing on the tongue image data set to obtain a tongue image training data set, wherein the enhancement processing includes two methods of homogeneous enhancement and mixed-class enhancement;

Use the tongue image training data set to train the improved convolutional neural network GoogleNet model, and obtain the trained improved convolutional neural network GoogleNet model;

Among them, the improved part of the improved convolutional neural network GoogleNet model includes:

Each intermediate layer of the model is standardized, which effectively avoids gradient disappearance and gradient explosion, speeds up network convergence, and optimizes network structure;

The residual is integrated into the Inception structure, the Inception structure is improved and optimized, and the input features and the features output by the Inception structure are feature-fused, named Inception B;

The 3ⅹ3 convolution branch in the Inception module is decomposed into 1ⅹ3 and 3ⅹ1 convolution, the 5ⅹ5 convolution is decomposed into 3ⅹ3 convolution and 1ⅹ3, 3ⅹ1 convolution, and Batch Normalization is added after each branch, which effectively reduces the parameters of the model quantity, named Inception C;

On the basis of the original GoogleNet model, the unimproved Inception module is retained, named InceptionA, and the InceptionA, InceptionB and Inception C structures are integrated and applied to the improved GoogleNet network model, and the InceptionA module is added to the shallow layer. The InceptionB module is added to the middle layer, and the Inception C module is added to the deep layer. The A and B modules are used twice respectively. The C module has stronger ability to extract and perceive features, so it is used five times.

Among them, the same type of enhancement refers to the use of data normalization, horizontal or vertical flip, proportional enlargement or reduction of the image, random rotation, random stagger transformation, brightness change and pixel filling methods for the images of the same label to expand the dataset, and the color is also It is an important feature in the identification of tongue image constitution, so the color channel transformation of the tongue image data set is not performed.

Among them, the mixed-class enhancement method is to solve the problem of imbalance between sample data sets. The mixed-class enhancement method mixes two types of sample data with different labels to construct a virtual training sample to enrich the tongue image data set. The mathematical expression is as follows: (1) shows:

where (x _i , y _i )(x _j , y _j ) are two randomly selected samples and corresponding labels, λ is a random number obeying beta distribution, and λ∈[0,1].

The technical effects achieved by this application are as follows, specifically as follows:

1. Add various improvement modules to the original GoogleNet model, such as adding a combination of similar enhancement and mixup enhancement to enhance the tongue data set, adding batch normalization operations, and adding attention mechanism modules.

2. Improve and optimize the Inception structure of GoogleNet's basic module, such as adding a residual structure to the original Inception structure for feature fusion, adding convolution replacement and decomposition operations to the Inception structure, and combining the improved two Inception structures with the original Inception structures are combined to build the model together.

3. The method of the present invention adopts the convolutional neural network algorithm to perform data enhancement on the tongue image data set by combining two methods of homogeneous enhancement and mixed-class enhancement. The model can adapt to the tongue image data in various environments, the extracted features can directly output the classification results, the calculation amount is greatly reduced, and the generalization ability is stronger.

4. The method of the present invention is improved and optimized based on the current mainstream GoogleNet model, and the classifier adopts the softmax classifier, which is a supervised learning method suitable for multi-classification problems such as identification of physical fitness, and is better in calculation method and recognition effect. Effect.

5. Compared with the traditional Chinese medicine constitution identification identification time, the method of the present invention reduces the diagnosis time and improves the diagnosis efficiency, and improves the identification accuracy compared with the current deep learning method using machine learning.

6. The method of the present invention is based on a large number of tongue image data sets, and the deep learning technology is applied to the traditional Chinese medicine constitution identification field. Not only can the constitution be judged by the PC terminal, but also the constitution can be judged by transplanting it to the mobile terminal, which is very convenient and accurate. high and save time.

7. The method of the present invention combines the deep learning method with traditional Chinese medicine constitution identification, and performs identification on the basis of big data, which effectively solves the difficult problem of traditional constitution identification. The method has certain market value and promotion value.

The following describes the specific method of tongue-like constitution identification based on deep learning.

1 Introduction

Traditional Chinese Medicine (TCM) is a treasure of the Chinese nation with a history of preventing and treating diseases for thousands of years. Accepted and frequently used by more and more people. Different treatment for the same disease and the same treatment for different diseases are the basic principles of Chinese medicine to understand and treat diseases. The premise and basis of the treatment method based on syndrome differentiation depends on the individual's constitution. In the theory of traditional Chinese medicine, it is believed that constitution is a relatively stable inherent characteristic of the human body on the basis of congenital conditions and acquired conditions, including human morphological structure and psychological temperament. TCM constitution is closely related to human health. Constitution is also the foundation and core content of TCM constitution research, which affects the diagnosis and treatment of the entire clinical discipline of TCM. Combining the definition of constitution proposed in "The Yellow Emperor's Classic of Internal Medicine" and the principles of clinical application of modern constitution, Professor Wang Qi proposed the nine-point method of constitution, which summarizes the types of constitution into peaceful constitution, qi deficiency constitution, yang deficiency constitution, yin deficiency constitution, and phlegm damp constitution. , Damp-heat, blood stasis, qi stagnation, and peculiar, of which the peaceful constitution is the healthy constitution, and the other eight constitutions are biased constitutions.

Tongue diagnosis is one of the most commonly used methods in TCM diagnosis. TCM believes that tongue images are related to the human body's veins and viscera, and reflect the deficiency and excess of cold and heat in the internal environment of the body, and the rise and fall of yin and yang. Doctors judge the patient's health status and susceptibility to diseases by observing the changes in the color, texture, texture and shape of the tongue coating, and give reasonable suggestions. Clinical practice has also proved that tongue diagnosis is a simple and effective diagnosis method without side effects, and the tongue image contains information directly related to human constitution, so it is feasible to use tongue diagnosis for constitution identification. In 2009, "Classification and Judgment of Constitution of Traditional Chinese Medicine" published by China Traditional Chinese Medicine elaborated the characteristics of the tongue corresponding to each constitution. , lubricating tongue coating" is the tongue characteristics of people with yang deficiency constitution, in addition to introducing other physical characteristics of constitution, these physical characteristics can be used to design questionnaires to identify constitution. However, the questionnaire is very subjective, it is difficult for individuals to make objective choices, and there are many questions on the scale, which takes a lot of time and affects the accuracy of judgment. With the popularization of artificial intelligence methods such as machine learning and deep learning, if an individual's tongue image can be taken and stored in the form of a digital image, combined with artificial intelligence technology, intelligent TCM constitution identification based on tongue diagnosis can be realized. More objective and accurate results.

At this stage, the use of artificial intelligence technologies such as machine learning and deep learning for image recognition has become mature. Some scholars build a tongue image database by collecting a large number of tongue images with physical labels, and design an algorithm to match the tongue images with the established database, so as to achieve the purpose of identification. Other scholars have designed quantitative methods to extract objective tongue features such as the chroma and saturation of tongue quality and fur, texture features, smoothness, and tooth marks of tongue fur. TCM constitution identification model based on machine learning methods such as K-means and artificial neural network. Considering the complexity of traditional manual extraction of tongue features and the accuracy of tongue region segmentation, deep learning has been applied to physical fitness recognition due to its advantages in automatic feature extraction. Some scholars have used various models of convolutional neural networks to perform tongue Physical identification. Although these studies have promoted the development of intelligent physical identification of tongue diagnosis, there are also some common problems. First, deep learning requires a large number of data sets as support, and it is difficult to collect sample data of tongue images. Second: simply using the model for classification does not improve and optimize the model, resulting in a low accuracy of physique identification using the above methods. In order to deal with these problems, this application proposes a new method based on an improved convolutional neural network GoogleNet model, and uses this method to identify the body constitution on the basis of the collected tongue image data set. The main contributions of this application are: constructing a Tongue image data set with physical labels, by adding batch normalization operations, improving and optimizing the Inception module, adding attention mechanism, and combining different Inception structures, a GoogleNet network model suitable for tongue image physical recognition is proposed ( TCR-GoogleNet). An ablation experiment is designed to verify the contribution of each improved module to the prediction accuracy, and a comparison experiment is designed with five mainstream networks for image classification, which proves that the model proposed in this application has a great improvement in accuracy and stability.

2. Method

This chapter introduces the specific process of physique identification through tongue images. The first part briefly describes the specific steps and architecture of the method. The second part briefly describes the construction of the tongue image set, including the basic process and requirements of image acquisition and the method of image preprocessing. Section III introduces the preprocessing of tongue images and data augmentation operations on the tongue dataset. The fourth part introduces the improvement method of the module and the construction of the whole model.

2.1 Method Architecture

In order to better identify the physique of the tongue image, this application proposes the TCR-Googlenet model, which is expected to obtain better physique recognition results by training a large amount of data in the constructed tongue image data set. The infrastructure of the method is shown in Figure 2, with a total of five steps. The first step is the acquisition of tongue images. The second and third steps are image preprocessing, including segmenting and cropping from the face image to extract the tongue region. After segmentation and cropping, the quality of the tongue image dataset can be greatly improved. The fourth step is to input the extracted tongue image into the deep neural network model that has been constructed for training and physical identification. The final classification identifies different physique types.

2.2 Dataset Construction

(1) Characteristic description of nine common types of constitutions

In TCM tongue diagnosis, different constitutions have different tongue characteristics, and doctors often use these different tongue characteristics to judge the patient's constitution and pathology. According to the theory proposed by Professor Wang Qi [6], this application divides the tongues of nine constitutions, namely, calming, qi-deficiency, yang-deficiency, yin-deficiency, phlegm-dampness, damp-heat, blood stasis, qi-stagnation, and idiosyncratic qualities. The image feature description is used as the research object, and its characteristics are shown in Table 1. These feature descriptions can provide a visual basis for the subjective judgment of the accuracy of the deep learning network model for the identification of TCM constitution.

Table 1. Characteristic description of tongue images of nine physical constitutions

(2) Construction of the tongue constitution dataset

The deep learning network model imitates the human brain from low-level to high-level abstraction to extract image features based on large-scale image data sets. In order to recognize tongue image constitution based on deep learning, it is necessary to construct a tongue image data set. , a total of 1752 pieces of tongue image data, and while collecting tongue images, the patients were asked to fill in the TCM constitution questionnaire to determine their physical type, and label the tongue images. Each image was manually processed to accurately extract the tongue region image, thereby reducing The influence of other parts of the original acquired images on the recognition of tongue constitution was investigated. Tongue images were taken in bright natural conditions. The sample distribution of the tongue image dataset is shown in Table 2, where the numerical value represents the number of tongue images per physique. The samples corresponding to each physique are shown in Figure 3.

Table 2 Data set sample distribution

2.3 Data Augmentation

Since the images captured by the mobile phone camera will be affected by external factors such as light and angle, and the number of tongue images collected is relatively small, in order to prevent over-fitting, the model has low robustness and other phenomena, it is necessary to analyze the collected tongue images. Image data augmentation. In this application, the data set is enhanced mainly by the combination of two methods of homogeneous enhancement and mixed-class enhancement.

(1) Similar enhancement

Similar enhancement mainly refers to the use of data normalization, horizontal or vertical flip, proportional enlargement or reduction of images, random rotation, random staggered transformation, brightness change and pixel filling for images with the same label. It is an important feature in the identification of tongue image constitution, so the color channel transformation of the tongue image data set is not performed. Figure 4 is a set of pictures after similar enhancement of data.

(2) Mixed class enhancement

The method of mixed-class enhancement is to solve the problem of imbalance between classes of sample data sets. For example, special aptitude is an uncommon type of constitution, so the sample data is relatively small. The sample data is mixed to construct a virtual training sample to enrich the tongue image dataset. Its mathematical expression is shown in (1):

where (x _i , y _i )(x _j , y _j ) are two randomly selected samples and corresponding labels, λ is a random number obeying beta distribution, and λ∈[0,1]. Figure 5 is the rendering of Mixup randomly generated twice.

2.4 Construction of the tongue image constitution recognition model

This application improves the GoogleNet model for tongue image recognition. GoogleNet is the winner of the 2014 ILSVRC competition and belongs to the deep convolutional neural network model. Since the development of the VGG model, the convolutional neural network has developed towards a deeper architecture, including more and more network layers and parameters. However, blindly increasing the network will inevitably impose a burden on computing resources. The Inception module proposed in GoogleNet effectively alleviates the gradient disappearance and gradient explosion problems caused by deepening the network, and improves the computational efficiency of the model. This application proposes an improved deep neural network model based on the GoogleNet model, which is more suitable for this application. data set and higher accuracy. This chapter mainly introduces the construction of the tongue image recognition model and the improvement strategy.

(1) Batch normalization operation (BatchNormalization, BN)

With the continuous update of parameters in the neural network, the input data distribution of each layer in the middle is often quite different from that before the parameter update, which causes the network to constantly adapt to the new data distribution, which will increase the difficulty of model training. The BN operation proposed in 2015 effectively solves this problem. In the present application, standardization is performed on each intermediate layer (before the activation function) of the model, which effectively avoids problems such as gradient disappearance and gradient explosion, accelerates the network convergence speed, and optimizes the network structure. The calculation process of BN is as follows:

If a batch of input variables are _{x 1} , x ₂ .

Then a batch of input x values are normalized to have a mean of 0 and a variance of 1. The mathematical expression is shown in (4):

Equation (4) corrects x _i from the edge where the gradient disappears to the area with a large gradient, which increases the training speed, where ε is a very small value, in order to avoid the variance denominator being 0 and not affecting the calculation.

Finally, the normalized value is reconstructed, and its mathematical expression is shown in (5):

Among them, α and β are two variables that can be learned. The purpose is to compensate the nonlinear expression ability of the model and restore the distribution to be learned by this layer of network.

(2) Inception structural residual design

He proposed the residual structure in 2015 and demonstrated the advantages of feature additive merging through theory and practice. In image recognition, the residual structure can effectively avoid the problem of network degradation caused by depth deepening, and it also solves the gradient problem, which improves the performance of the network. This application draws on this idea to incorporate the residual into the Inception structure, improves and optimizes the Inception structure, and fuses the input features with the features output from the Inception structure. Figure 6 is an Inception module with residuals added to the application, named as Inception B.

(3) Inception structure convolution decomposition and replacement

The decomposition and replacement operation of convolution is proposed in Inceptionv3: the large convolution kernel is decomposed into symmetric small convolution kernels; it is decomposed into asymmetric convolution kernels. The first method is to replace one 5x5 convolution kernel with two 3x3 convolution kernels. The second method is to replace the n*n convolution kernels with 1*n and n*1 convolution kernels and then stack them. Both methods can reduce the amount of parameters and add more nonlinear capabilities under the condition of the same receptive field. Therefore, in this application, these two methods are used to improve the Inception module. The 3ⅹ3 convolution branch in the Inception module is decomposed into 1ⅹ3 and 3ⅹ1 convolutions, and the 5ⅹ5 convolution is decomposed into 3ⅹ3 convolution and 1ⅹ3, 3ⅹ1 convolution, and in the Batch Normalization is added after each branch, which effectively reduces the number of parameters of the model. Figure 7 shows the Inception structure improved by this method, named InceptionC.

(4) SE attention module

The SE module is composed of two operations, Squeeze and Excitation. Squeeze is a low-dimensional embedding of global information, which is a compression process, and Excitation is a mapping transformation process. The SE module mainly explores the relationship between channels and assigns network channels to the network. Weights to enhance the learning ability of the network, also known as channel attention. Figure 8 is a schematic diagram of an SE module.

(5) Inception structure fusion and model construction

Based on the original model of GoogleNet, this application retains the unimproved Inception module, which we name as Inception A as shown in Figure 9. We fuse Inception A with the Inception B and Inception C structures proposed above, and apply them to the improved GoogleNet network model. The InceptionA module is added to the shallow layer, the Inception B module is added to the middle layer, and the Inception C module is added to the deep layer. The B module was used twice, and the C module was more capable of feature extraction and perception, so it was used five times.

Combining the above methods and improvement methods, the overall structure of the improved tongue image constitution recognition model is proposed as shown in Figure 10.

3. Experiment design and result analysis

This experiment uses win10 operating system, CPU is Intel(R) Core(TM) i9-7900X3.30GHz, GPU is GeForce RTX 3090, Python version is 3.8.4, Cuda and Cudnn versions are 11.2 and 8.1.0 respectively, using tensorflow2 The .5.0 framework is an experimental environment.

3.1 Parameter selection and optimization

During the training process of the deep learning model, the selection of parameters often affects the accuracy of the model. In the process of parameter selection and optimization, the application uses multiple ablation experiments for fine-tuning, and when using this model for training, the batch training method is used. The batch size is 32, and a total of 100 rounds of Epochs are performed. The optimization function adopts the Adam algorithm (Adaptive Moment Estimation of Adaptive Moment Estimation), the momentum is 0.9, and the weight decay value is 1×10 ^-5 . The initial The learning rate is 0.0001. In order to further prevent overfitting, the Dropout value is set to 0.3, that is, 30% of the nodes are randomly discarded, and SoftMax is used for the classifier.

3.2 Evaluation indicators

In order to evaluate the pros and cons of the model, the number of rounds (E) at which the model reaches convergence, that is, the convergence speed, confusion matrix, loss value (L), and Top-1 accuracy rate (Acc), is selected as the evaluation index in model training. The number of rounds means the evaluation of the convergence speed of the model under the same conditions. Physical fitness identification is a multi-classification problem. The loss function adopts the multi-class cross entropy loss function (cross entropy loss), and its loss value expression is:

where _yi is the true label corresponding to the _ith sample; pi is the predicted value of the image trained by the model; N is the total number of physical categories; K is the total number of tongue samples.

The accuracy of Top-1 reflects the proportion of correctly recognized images in all recognized images in the recognition results, and reflects the training effect of the model on the data set. Its mathematical expression is:

TP (True positive) is a positive sample predicted by the model; TN (True negative) is a negative sample predicted by the model; FN (False negative) is a negative sample predicted by the model; FP (False positive) is The model predicts a negative positive sample.

3.3 Ablation experiment

As a common method for model evaluation, ablation experiments refer to deleting a certain part of the network model so that we can better understand the impact and contribution of the improved network module on the accuracy of the network model, and further illustrate the effectiveness of the improved algorithm. This application The effectiveness of each improved strategy is verified by ablation experiments, and the influence of each improved network module on the model accuracy is analyzed.

This experiment combines the design of experiments from adding batch normalization operations, using Inception A, InceptionB, and Inception C structures separately and in combination, and adding attention mechanism. A total of 7 groups of experiments are designed, numbered 1-7, and the experimental design is as follows shown in Table 3.

Table 3 Design scheme of ablation experiments

Table 4 Comparison of the results of ablation experiments

The experimental results are compared as shown in Table 4. The following conclusions can be drawn from the experimental results of each group:

(1) Batch Normalization Operation (BN)

Experiment 1 only used the Inception A module and did not make any changes to the model. It is the original model of GoogleNet. The BN operation was added in Experiment 2. Compared with the results, the accuracy of Experiment 2 was improved by 1.6% compared to the Top-1 of Experiment 1. It shows that the BN operation optimizes the network structure and improves the accuracy of the model to a certain extent.

(2)Mixup

In experiment 3, the Mixup mixed-class enhancement method was used to expand the data set. Compared with experiment 2, the accuracy and loss values were optimized to different degrees. The combination with similar data enhancement made up for the lack of tongue image data to a certain extent. , which makes the model fit the dataset better with almost no additional computational overhead.

(3) Inception structure

In experiment 4, all Inception A modules in the model were replaced by Inception B modules with residual structure added, but the accuracy of the model dropped by about 2.5%, which proved that the blindly stacked residual structure to widen the network cannot be well adapted. dataset for this application.

Experiment 5 all used the Inception C module, and the accuracy of Top-1 improved significantly, indicating that through the convolution decomposition and replacement of the deepened network structure, the extraction of tongue feature information has a better effect, reducing network model parameters and improving recognition. precision.

Experiment 6 combines the operations of Experiment 2, Experiment 4 and Experiment 5, using three methods of Inception structure fusion to improve the model, and at the same time widening and deepening the network model, the extraction of tongue image features is more comprehensive, and its accuracy and The loss value works better than using the Inception structure alone.

(4) SE module

Experiment 7 is the final model after the improvement of this research. It can be seen from the table that compared with the previous experiments, adding the SE module Top-1 accuracy can improve by 1.3%, which shows that the introduction of attention mechanism can increase the network depth, The width and insignificant increase in network parameters improve the top-1 accuracy of the model.

3.4 Experimental results and analysis

Based on the construction of the above model, we take 85% of the data set as the training set and 15% as the test set. When dividing the data set, we ensure that the data is randomly reproducible. The experimental effects of the model of this application are analyzed by comparing three aspects.

(1) Confusion matrix

The model is applied to the test set of the tongue image dataset, and the resulting confusion matrix is shown in Table 5. It includes the number of correctly predicted and incorrectly predicted samples in the test set of 9 physical images. In the confusion matrix, the values on the main diagonal are the number of correctly predicted samples, and the values elsewhere are the number of incorrectly predicted samples.

Table 5 Confusion matrix for test set

It can be seen from the table that qi-deficiency constitution and yang-deficiency constitution are easy to cause confusion in the classification process. The reason is that these two constitutions may have relatively similar tongue characteristics, and the differences between classes are small, so the model is easy to classify these two constitutions. confused. The tongue characteristics of blood stasis constitution and qi stagnation constitution are obviously different from other constitutions, and the classification effect is better. Among them, the recognition rate of special physique is the lowest, because there are very few samples of special physique, there are too few people with special physique in reality, it is difficult to collect a large number of training samples and there are many tongue characteristics of special physique Change. The recognition accuracy of phlegm-dampness and dampness-heat quality is the highest, thanks to the sufficient training samples, the model can fully extract tongue features.

(2) Recognition results of training set and test set

Figure 11 shows the accuracy curve and loss value curve during the training process of the training set and the test set. The training accuracy curve (1) can reflect the change trend of the accuracy of the model prediction as the number of Epoch rounds increases. The training loss value curve (2) represents the deviation and fluctuation between the model predicted value and the actual value as the number of Epoch rounds increases. The accuracy rate of the training set of this application model reaches 92.2%, the loss value is 0.375, and the accuracy rate of the test set reaches 87.4%, and the loss value is 0.501. Compared with other similar studies at this stage, this model has higher recognition accuracy and stability.

(3) Model comparison results

In order to verify the validity of the model of the application and the effect after improvement, the application has selected five mainstream convolutional neural network models for image recognition to carry out comparative experiments. -line chart comparison of the accuracy rate of the five models of -v3-large, mobilenet-v3-small, and the original GoogleNet model trained on the test set to show the difference with other models. Table 3 visually lists the Top-1 accuracy, loss value, and the number of Epoch rounds (convergence speed) for the six model test sets. We take the accuracy of the test set as the standard for model comparison, because the test set can better characterize the model's ability to fit the data set, and can more accurately evaluate the training accuracy of the model. In model evaluation, model prediction accuracy takes precedence over convergence speed.

In this experiment and the comparison experiment, the neural network model is trained from scratch, so the initial accuracy rate is low, but this model can achieve the highest Top-1 accuracy rate at a faster speed, which is 87.4%. Although the model in this application has a slower convergence speed compared to the three models of GoogleNet, mobilenet-v3-large, and mobilenet-v3-small, the Top-1 accuracy of this model is 11.5% higher than that of the GoogleNet model and higher than that of the mobilenet-v3-large model. 17.1% higher and 19.5% higher than the mobilenet-v3-small model. The advantage of the mobilenet-v3 model is that it adopts a depthwise separable convolution, which greatly reduces the amount of computation and parameters. Therefore, the mobilenet-v3 series is very advantageous in terms of convergence speed, but after convergence on this dataset, the oscillation is relatively high. If it is large, the stability of the model is not good and the accuracy rate is low. The Resnet-50 network model with residual added has good performance in Top-1 accuracy and loss value, indicating that the fusion residual structure has a good ability to represent tongue features. Therefore, we also added a residual structure in the process of building this model, and combined with other improved modules, the recognition effect was improved by 5.6% compared with the Top-1 accuracy of Resnet-50, the loss value decreased by 0.112, and the convergence speed was also slightly improved. . The VGG-16 model improves the training accuracy of the model by deepening the number of network layers and reducing the size of the convolution kernel. However, in the comparative experiment, the effect has not reached the expectation. The accuracy rate is only 72.8%, which is quite different from the experimental results of the model in this application. , which proves that simply deepening the network structure cannot extract tongue features very well. In the network model of this application, we not only improved the model from the depth and width of the network, but also proved through ablation experiments that the multiple improved modules added on this basis have improved the prediction accuracy of the model to a certain extent.

Table 6 Comparison of model results

On the whole, the improved model of this application has a greater improvement in recognition accuracy than other models, and its convergence speed is moderate, but the priority of model prediction accuracy is greater than the convergence speed, so the improved model proposed in this application is on the tongue image data set. The recognition effect is better than the other five models.

This application proposes the TCR-GoogleNet model based on GoogleNet, and builds a mature tongue image dataset. At the same time, we should continue to collect images and expand the tongue image dataset to provide stronger data support for subsequent research work. With the advantages of the width and depth of the network and the blessing of various improvements, this model shows strong feature extraction ability and better recognition accuracy for the tongue data set. Aiming at the common tongue image, it can effectively identify the nine TCM constitutions, namely, peaceful, qi-deficiency, yang-deficiency, yin-deficiency, phlegm-dampness, damp-heat, blood stasis, qi-stagnation, and peculiarity, and relieves to a certain extent. In order to overcome the difficulty that physical classification requires rich expert experience, the method of comparing questionnaires greatly shortens the time for physical identification, and provides reference and ideas for the objective process of tongue diagnosis.

There is still more room for development in the future for physical constitution recognition based on tongue diagnosis. The tongue image data set can be expanded more abundantly, and the recognition accuracy can be further improved on this basis. Due to the diversity of each person's constitution, a person may contain multiple constitutions, so follow-up research can be carried out around multi-label learning of tongue image constitution recognition. In addition, the study of tongue diagnosis can not only be used to identify physical constitution, there are already studies combining tongue diagnosis and western medicine to prevent and treat diseases, but the research in this area still needs further development.

Each part in this specification is described in a progressive manner, and each part focuses on the differences from other parts, and it is sufficient to refer to each other for the same and similar parts among the various parts.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined in this application may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not to be limited to the embodiments shown in this application, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (3)

1. A tongue constitution identification method based on deep learning is characterized in that the method is deployed on a mobile terminal and comprises the following steps:

acquiring a tongue picture image to be recognized, and recognizing the tongue picture image to be recognized by using a trained improved convolutional neural network GoogleNet model to obtain a recognition result of the tongue picture;

displaying the recognition result of the tongue picture image on the mobile terminal;

the improved convolutional neural network GoogleNet model is trained as follows:

acquiring a tongue image data set, and performing enhancement processing on the tongue image data set to obtain a tongue training data set, wherein the enhancement processing comprises a similar enhancement mode and a mixed enhancement mode;

training the improved convolutional neural network GoogleNet model by using a tongue picture training data set to obtain a trained improved convolutional neural network GoogleNet model;

the improved GoogleNet model improvement part of the improved convolutional neural network comprises the following steps:

each intermediate layer of the model is subjected to standardization treatment, gradient disappearance and gradient explosion are effectively avoided, the network convergence speed is increased, and the network structure is optimized;

residual errors are fused into the inclusion structure, the inclusion structure is improved and optimized, input features and features output by the inclusion structure are fused, and the Incep structure is named as inclusion B;

the 3 x3 convolution branch in the inclusion module is decomposed into 1 x3 and 3 x 1 convolutions, the 5 x 5 convolution is decomposed into 3 x3 convolution and 1 x3, 3 x 1 convolutions and the Batch normalation is added after each branch, effectively reducing the number of parameters of the model, named inclusion C;

on the basis of a GoogleNet original model, an unmodified inclusion module is reserved and named as an inclusion A, the structures of the inclusion A, the inclusion B and the inclusion C are fused and applied to the improved GoogleNet network model, the inclusion A module is added into a shallow layer, the inclusion B module is added into a middle layer, and the inclusion C module is added into a deep layer, wherein the A and B modules are respectively used twice, and the C module has stronger characteristic extraction and sensing capabilities, so the five times of use are realized.

2. The method of claim 1, wherein homogeneous enhancement is data set expansion using data normalization, horizontal or vertical flipping, scaling up or down images, random rotation, random miscut transform, brightness variation, and pixel filling methods on images of the same label, and color is also an important feature in tongue quality identification, so that no color channel transformation is done on tongue data set.

3. The method of claim 2, wherein the mixed enhancement mode is to solve the problem of imbalance between sample data sets, and the mixed enhancement mode mixes two types of sample data with different labels to construct a virtual training sample rich tongue picture data set, and the mathematical expression of the virtual training sample rich tongue picture data set is shown in (1):

wherein (x) _i ，y _i )(x _j ，y _j ) Is two samples and corresponding labels selected randomly, lambda is a random number obeying beta distribution, and lambda belongs to [0, 1 ]]。

Images