JPH0784977A - Construction method of multilayered neural network - Google Patents

Abstract

(57) [Summary] [Objective] The object of the present invention is to store a reliable classification rule,
An object of the present invention is to provide a method for constructing a multilayered neural network that can refine uncertain classification rules by case learning, improve classification accuracy, and improve learning efficiency. [Configuration] A partial configuration of the neural network is initialized based on the rough classification rule (step 1), a certainty factor is assigned to each connection based on the rough classification rule (step 2), and the suppression rate is calculated according to the certainty value. Is determined (step 3), the neural network is learned (step 4), and it is checked whether the neural network has a classification function for the target data (step 5).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for constructing a multi-layered neural network, and more particularly to a method for constructing a multi-layered neural network for classifying target data represented by multidimensional feature vectors.

[0002]

2. Description of the Related Art Hitherto, a multilayer structure type neural network has been used for the purpose of classifying target data.

FIG. 4 shows the structure of a multilayer structure type neural network. In the figure, the input layer 11 inputs the feature quantities (x ₁ , x ₂ , x ₃ , x ₄ ) of the classification target data, and the output layer 12 outputs the classification results (o _n1 , o _n2 ),
Between the input layer 11 and the output layer 12, there are one or more intermediate layers 13. The inputs of the units in each layer are coupled to the outputs of each unit in the layers before and after it. The output of each layer unit is determined by the following equation.

[Equation 1]

Here, o _j ^(k) is k layers (k ≧ 1, k =
1 is the output value of the j-th unit in the input layer), and w _ij
^(k) is the coupling load from the i-th unit of the k-1 layer to the j-th unit of the k layer, and N ^(k-1) is the total number of units in the k-1 layer. However, w _0j ^(k) in the equation (2) is a coupling weight for giving a bias to the j-th unit in the k layer, and the output value o ₀ ^(k-1) in this case is always 1.
Further, each unit in the input layer (k = 1) outputs the input feature amount as it is.

In such a neural network, the data
In order to classify the data into the units of the input layer 11,
When the characteristics of the target data are given, the
Only the units of the output layer 12 corresponding to the classification category
It outputs a high value and the other units in the output layer 12 output a low value.
As output, the coupling load w between each unit described above_ij
^(k)And the bias value needs to be set.

As this setting method, first, the initial weight of the neural network is set to a random value, and the weight is adjusted so that the error between the actual output value and the desired output value when the vector element of the sample of the target data is input is reduced. w _ij
There is an error backpropagation learning method that repeatedly adjusts ^(k) in small increments. However, since this method adjusts the load in small increments from random initial load values, it takes a long time for the learning to converge, and in many cases the learning does not converge. There is. In addition, since the learning is performed by referring to only the sample data, generalization ability that general classification performance for the target data cannot be obtained when a large amount of sample data cannot be obtained or the sample data is biased. I have a problem with.

Therefore, when the general classification rule which is considered to be almost correct for the conventional target data is known, the structure and weight of the neural network are initialized so as to have an initial classification function equivalent to this general classification rule ( Japanese Patent Laid-Open No. 4-76660), and thereafter, there is a method for performing case learning by a weight variation suppression learning method (Japanese Patent Laid-Open No. 4-242937) for performing case learning while keeping the initial setting value as much as possible.

FIG. 5 shows a flow chart for explaining the conventional method. Hereinafter, each process will be described with reference to FIG. Vector elements x ₁ , x ₂ , x ₃ , of the target data,
Categories of target data y ₁ , y ₂ from x ₄ , x ₅ , x ₆
Classify which one of them belongs to. Given the following general classification rules, which are considered to be generally correct:

IF x ₁ or x ₂ THEN y '(3) IF (y'and x ₃ ) or x ₄ THEN y ₁ (4) IF x ₃ or x ₅ or x ₆ THEN y ₂ (5) Rule (3) ) Is "if the feature x ₁ or x ₂ exists,
The target data belongs to the subcategory y ', and the rule (4) is that the target data is the subcategory y'.
If x ₃ exists or x ₄ exists, the target data belongs to category y ₁ ”, and the rule (5) is“ x ₃ , x _5, or x ₆ Is present, the target data belongs to the category y ₂ ”.

These general classification rules are converted into the following multi-step logical expressions by the classification rule / multi-step logical expression conversion processing (step 21).

Y ′ = x ₁ + x ₂ (6) y ₁ = (y ′ + x ₃ ) + x ₄ (7) y ₂ = x ₄ + x ₅ + x ₆ (8) Next, a multistage logical expression / load conversion process ( In step 22), the connection configuration of the neural network for performing the classification process is as shown in FIG.
It is installed as shown in. FIG. 6 shows an example of a neural network initially set by the multistage logical expression / weight conversion processing.

Here, as shown in the figure, the connection structure of the neural network 31 is such that one input layer unit is allocated to each variable appearing on the right side of the multi-stage logical expression and one second intermediate layer unit is allocated to each multiplication term. , The output layer 12 is implemented so as to realize the addition of the multiplicative term. The load w _ij is determined according to the method described in Japanese Patent Laid-Open No. 4-76660.

Next, a parameter setting process (step 2)
By 3), the learning parameter is set. The contents of the learning parameter will be described in detail in the part for explaining the learning process (step 24). In the parameter setting process, the learning parameter is manually determined by trial and error.

Next, case learning is performed by a case learning process (step 24). The weight variation suppression learning method used in the case learning process will be described below.

In the error back-propagation learning method, only the error of the sample data is reduced. Therefore, when the sample data is small or the sample data is biased, the initial structure changes greatly, and the general classification ability for the target data is improved. There is a problem that Therefore, in the weight variation suppression learning method, the value of the load is changed so as to decrease the evaluation function E represented by the following equation.

[0016]

[Equation 2]

Here, K is the layer number of the output layer 12, that is, the total number of layers of the neural network. Also, y
_j is a desired output value for the learning sample of the j-th unit in the output layer, that is, a correct classification result. W
_ij ^(k) is an initial setting value of the load w _ij ^(k) . f (x,
y) is a function that increases with an increase in the difference between x and y, and is, for example, | x−y |, (x−y) ^2, or the like. η _ij
^(k) is called a suppression rate, and is a learning parameter that determines the magnitude of the contribution of the difference of each load to the evaluation function E. The suppression rate η _ij ^(k) is usually a constant value for all loads or a constant value for each layer. The first term on the right side represents the squared error between the output value of the neural network and the desired output value when the feature amount of the learning sample data is input.
That is, it is equivalent to the evaluation function used in the error backpropagation learning method. The second term is a term that represents the deviation of the load from the initial setting value.

As described above, in the weight variation suppression learning method, the weight is adjusted so that the sum of the squared error of the output value of the neural network and the deviation of the load value from the initial setting value decreases. Therefore, if the suppression rate is set to a large value, the fluctuation of the load value can be prevented, so the rough classification rule given at the beginning is strongly saved, and conversely, if it is set to a small value, the rough rule is not saved much. Become.

The change amount Δw _ij ^(k) of the load w _ij ^(k) for reducing the above evaluation function E is given by the following equation.

[0020]

[Equation 3]

Here, the change amount Δw _ij ^(k) of the load is the adjustment amount of the load or the bias value. Further, the first term on the right side is equivalent to the adjustment amount of the load in the ordinary inverse error propagation learning. ε is a learning parameter called a learning rate, which is a learning parameter that determines the magnitude of the adjustment amount in one iteration. Also, d _j
^(k) is calculated by the following equation when the k layer is the output layer (k = K).

[0022]

[Equation 4]

When the k layer is the intermediate layer, d _j ^(k) is given by the following equation.

[0024]

[Equation 5]

Using the above equations, case learning is repeatedly performed with the same learning parameters, and the processing is terminated when the error falls below a certain value. In addition, the processing is also ended when the error does not decrease even after performing the case learning a certain number of times or when the error does not converge due to the repeated decrease and increase of the vibration.

Next, the classification processing function inspection processing (step 2)
In 5), the test sample data is actually input to the neural network, and it is checked whether or not the correct classification result is output. If the classification function for the test sample data is obtained, all processing is terminated. When the sufficient classification function is not obtained or when the learning falls into the vibration state, the process returns to the parameter setting process (step 23).

The conventional method is excellent in convergence speed and convergence because the structure and weight of the neural network are initialized so as to have an initial classification function equivalent to the rough classification rule. In addition, if the suppression rate can be set to an appropriate value, case learning can be performed while preserving the initial structure even when sample data is not sufficiently obtained or the sample data is biased. Therefore, it is possible to prevent the generalization ability from decreasing and refine the rules given in the early stage.

[0028]

In the above-mentioned conventional method, the reset value of the learning parameter is determined by trial and error.
Therefore, originally, the suppression rate at which a different value can be set for each coupling of each unit requires a great deal of labor for the setting work, and therefore a constant value for all couplings or a coupling within each layer is required. It is usually reset to a constant value. However, there are rules with various certainty in the rough classification rules, such as one that needs to be strongly preserved because it is almost certain, and one that does not need to be strongly preserved because it is uncertain. When learning is performed at the same suppression rate for such rough classification rules, if the suppression rate is large, refining of uncertain rules will be hindered, and if the suppression rate is small, the certain rules will be affected by noise. Therefore, there is a problem that it changes unnecessarily and the generalization ability decreases.

Further, if the overall rough classification rule is fairly reliable, the change of the connection by case learning can be made in a very small amount, so the learning rate for determining the amount of weight to be adjusted at one time is set to a small value. The convergence is better. On the contrary, if the rough classification rule is uncertain, it is more efficient to set the learning rate to a large value in the early stage of learning. But,
In the conventional method, since the certainty of the rule is not taken into consideration, there is a problem that the learning rate cannot be set to an appropriate value as described above and learning cannot be performed efficiently.

The present invention has been made in view of the above points, and when solving the above-mentioned conventional problems and constructing a multilayer structure type neural network for performing a classification process, a reliable classification rule is saved and An object of the present invention is to provide a method for constructing a multilayer structure type neural network which can refine a reliable classification rule by case learning, improve classification accuracy, and improve learning efficiency.

[0031]

FIG. 1 is a diagram for explaining the principle of the present invention. The present invention relates to a multi-layered structure type that performs a classification process of target data when the rough classification rule with certainty factor indicating the certainty of the rule is known for the target data represented by the feature vector composed of a plurality of vector elements. In a method of constructing a neural network, a partial configuration of a neural network is initialized based on a rough classification rule (step 1), and a confidence factor is calculated based on the rough classification rule for each connection of the initialized partial configurations of the neural network. Is assigned (step 2), and the suppression rate, which is a learning parameter that determines the size of the contribution of the difference of each weight to the evaluation function of the neural network, is determined according to the value of the certainty factor assigned to each combination of partial configurations (step 2). 3), learning the neural network using the sample data (step 4), It is checked whether or not the Ural network has a classification function for the target data, and if it does not, the processing for determining the learning parameter and subsequent steps are repeated, and if it does, the processing ends ( Step 5).

In addition, according to the present invention, when the learning parameter is determined (step 3), the learning rate, which is the learning parameter for determining the magnitude of the adjustment amount of the load or the bias in one iteration, is set as a rule. It is determined according to the representative value of the certainty factor.

[0033]

According to the present invention, each rough classification rule is provided with a measure called certainty, which is a measure of the certainty of the rule, and the suppression rate, which is a learning parameter, is set according to this measure. This increases the suppression rate according to the certainty factor assigned to each connection of the neural network so that the rough classification rule is preserved when learning has a small amount of sample data or bias, and the error rate is also increased. If the value does not decrease, the accuracy of the classification process can be improved by adjusting the reduction rate so that the reliable classification rules can be saved and the uncertain classification rules can be refined by case learning. improves.

Further, the learning rate can be set according to the representative value of the certainty factor of each rough classification rule.

[0035]

Embodiments of the present invention will be described below with reference to the drawings.

First, it is assumed that there is the following written general classification rule with certainty factor. IF x ₁ or x ₂ THEN y'with C1 (14) IF (y'and x ₃ ) or x ₄ THEN y ₁ with C2 (15) IF x ₃ or x ₅ or x ₆ THEN y ₂ with C3 (16) Here, C1, C2, C3 are the certainty factors, and the rule (1
4) is that if the feature x ₁ or x ₂ exists, the target data belongs to the subcategory y '.
It shows that it is reliable about 1.

Similarly, the rules (15) and (16) also indicate that the reliability is about C2 and C3, respectively. The confidence factor CF is an analog value of 0 <CF <1, and CF =
It is assumed that 1 is the highest certainty, and conversely, CF = 0 is the lowest certainty.

FIG. 2 is a flow chart showing the outline of one embodiment of the present invention. In the figure, steps 21, 2
2, 24, and 25 are similar to the conventional processing shown in FIG.

First, similarly to the conventional method, the classification rule / multistage logical formula conversion process (step 21) and the multistage logical formula / weight conversion process (step 22) are used to construct a neural network having a connection structure as shown in FIG. Constitute.

FIG. 3 is a diagram for explaining a neural network for setting learning parameters according to an embodiment of the present invention. In the figure, the partial networks 511, 521, and 531 surrounded by the dotted line are rule (14),
It corresponds to (15) and (16).

The certainty factor allocation process (step 41)
The certainty factor of the classification rule assigned to each of the connections within the dotted line shown in FIG. 3 is assigned. For the bias value, the certainty factor of the input combination is assigned. However, as in the unit 522 shown in FIG. 3, the bias value of a unit whose input is a combination to which different certainty factors C1 and C2 are assigned is the minimum value among the certainty factors assigned to each combination. select. The overall confidence level of the rough classification rules is the average value of the confidence levels of all the rules. Hereinafter, the certainty factor assigned to each combination will be referred to as C _ij ^(k) , and the average value of the certainty _factors will be referred to as C _mid .

Next, in the learning parameter calculation process (step 42) based on the certainty factor, the learning parameter setting process is performed according to the following rules. In addition, the setting process according to the following rules and the case learning process (step 24)
This is repeated until the neural network has the classification processing function.

<< Rule 1 >> The initial setting values are as follows. Ho _ij ^(k) = 0 (17) A ₀ = −0.2 × C _mid +0.2 (18) R = 1 (19) where H t _ij ^(k) is the suppression rate at the t-th resetting. The value A _t is the learning rate. Further, R is a counter that stores the number of times that the suppression rate is repeatedly increased and decreased.

<Rule 2> Case study processing (step 24)
In, if the error with respect to the learning sample data does not decrease, reset as follows. First, when the rule 3 is applied in the (t-1) th process, the counter R is incremented. R = R + 1 (20) Next, the suppression rate is set according to the following formula. H (t) _ij ^(k) = H (t-1) _ij ^(k) + _Cij ^(k) / R (21) << Rule 3 >> Classification processing function In the inspection processing (step 25), a sufficient classification processing function is obtained. If not, reset as follows. When the rule 2 is applied in the (t-1) th process, the counter R is incremented. R = R + 1 (22) Next, the suppression rate is set according to the following formula. H (t) _ij ^(k) = H (t-1) _ij ^(k) _-Cij ^(k) / R (23) When the above rules are applied, learning proceeds as follows.

First, rule 1 is applied, and the suppression rate is initialized so that Ho _ij ^(k) = 0, that is, the learning is the same as the conventional error backpropagation learning method. Further, the learning rate is C _min = 1, that is, when the rough classification rule is 100% credible, A ₀ = 0
And C _min = 0, that is, when the rule cannot be trusted at all,
It is set according to the equation (18) so that A ₀ = 0.2.
Here, 0.2 is a learning rate generally used in the error backpropagation learning method.

When the case learning processing is performed with the above settings, if the learning sample data is small or biased, the generalization ability cannot be obtained even if the error related to the sample data is reduced, and the test data is not obtained. The classification function for does not improve. In such a case, rule 2 is applied and the inhibition rate is increased according to the confidence assigned to each join so that the rough classification rule is preserved. If the confidence factor is repeatedly increased, the error may not decrease even if the case learning process is performed. In such cases, Rule 3 is applied to reduce the inhibition rate.

By the above processing, a neural network having a classification processing function is constructed.

The present invention is not limited to the method used in the above embodiment, and various modifications can be made without departing from the scope of the claims. For example, in the present embodiment, as the confidence factor of the bias value of the unit that receives the combination to which different confidence factors are assigned, an average value or the like can be used in addition to the minimum value. Further, the representative value of the certainty factor of the overall rough classification rule can be substituted by a basic method other than the average value such as median.

It is also possible to apply a rule for setting the suppression rate according to the certainty factor from the initial setting.

[0050]

As described above, according to the present invention, in constructing a multilayer structure type neural network for performing classification processing,
Based on the general classification rule initially set in the neural network, the suppression rate can be set for each connection according to the certainty value assigned to each connection, so a reliable classification rule can be saved and uncertain classification can be performed. The rules can be refined by case study, and the classification accuracy can be improved. In addition, the learning rate that determines the amount to be adjusted at one time can be set to a small value when the overall rough classification rule is certain, and can be set to a large value when it is uncertain, so that the learning efficiency is high. Can be done on a regular basis. In addition, it is possible to reduce the work for setting parameters, which has conventionally been performed manually by trial and error.

[Brief description of drawings]

FIG. 1 is a principle configuration diagram of the present invention.

FIG. 2 is a flowchart showing an outline of an embodiment of the present invention.

FIG. 3 is a diagram for explaining a neural network that sets a learning parameter according to an embodiment of the present invention.

FIG. 4 is a diagram showing a configuration example of a multilayer structure type neural network.

FIG. 5 is a flowchart illustrating a conventional method.

FIG. 6 is a diagram showing an example of a neural network initially set by a multi-stage logical expression / weight conversion process.

[Explanation of symbols]

11 Input layer for inputting feature amount of classification target 12 Output layer for outputting classification result 13 Intermediate layer 51, 52, 53 Example of neural network configured according to multistage logical expression 511 Neural network 521 to which rule (14) is assigned 521 Rule Neural network to which (15) is assigned 522 Unit which receives as input a combination to which different certainty factors are assigned 531 Partial neural network to which rule (16) is assigned

Claims (2)

[Claims] 1. A multi-layer for performing a classification process on target data represented by a feature vector composed of a plurality of vector elements when a rough classification rule with a certainty factor indicating the certainty of the rule is known. In a method of constructing a structural neural network, a partial configuration of the neural network is initialized based on the general classification rule, and each combination of the initialized partial configurations of the neural network is based on the general classification rule. And assigning a certainty factor according to the value of the certainty factor assigned to each combination of the partial configurations, a suppression rate that is a learning parameter that determines the magnitude of the contribution of the difference of each weight to the evaluation function of the neural network is determined. , Learning of the neural network using sample data, the neural network Checks whether a classification function for data, if they do not have the
A method for constructing a multilayer structure type neural network, characterized in that the processing after the processing for determining the learning parameter is repeated, and if the learning parameter is included, the processing is terminated. 2. When determining the learning parameter,
The configuration of a multilayer structure type neural network according to claim 1, wherein the learning rate, which is the learning parameter for determining the magnitude of the adjustment amount of the load or the bias in one iteration, is determined according to the representative value of the certainty factor of the general rule. Method.