JPH0749847A - Learning system for series connection type recurrent neural network - Google Patents

Abstract

PURPOSE:To rationalize learning by a BP method and to sufficiently display the original capability by performing the BP method on the basis of the added term type evaluation function generated by adding an additional term for approximating connection loads to zero for a series connection type RNN having self-loops to a specific evaluation function. CONSTITUTION:For the series connection type RNN (recurrent neural network) which has the self-loops W12, W23, W34, W11, W22, W32, and W43, the additional term for approximating connection loads W11, W22, W33, and W44 to zero is added to the specific function. Then, the BP method (error inverse propagation method) is carried out on the basis of the added term type evaluation function obtained as mentioned above. Consequently, the learning by the BP method can rationally be performed and the original capability can sufficiently be displayed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a learning method for a series-coupled recurrent neural network (hereinafter referred to as a series-coupled RNN) in which a plurality of neurons are coupled in series, and more specifically, a series using the error back propagation method. The present invention relates to a learning method of combined RNN.

[0002]

2. Description of the Related Art Conventionally, units constituting a network of a recurrent neural network (RNN) are randomly connected. This is because each unit is composed of a dynamic non-linear unit, and its operation forms a predetermined time series pattern represented by the differential equation (state equation / equation 1, expression 2) shown below. .

[0003]

[Equation 1]

[0004]

[Equation 2]

Here, x _i , y _i , and X _i respectively represent the internal state, output, and external input of the i-th unit at time t, and w _ij is from the j-th unit to the i-th unit. Represents the combined load of. Further, τ _i is the time constant of the i-th unit, and N is the total number of units.
Further, the characteristic function f (x) is usually a nonlinear sigmoid function f (x) = tanh (x) or a linear function f (x) = x.
Either of them is selected.

[0006] Such an RNN includes a series connection type in which 2N units are connected in series in a rod shape. The series-coupled RNNs further include those with a self-loop as shown in FIG. 1 and those without a self-loop as shown in FIG. In any series-coupled RNN, only the information from the units adjacent to each other is reflected in the elapsed time, and the unit at the final stage is used as the output unit. Also, which series coupling type RN
N also has the ability to approximate a multiple damped sine wave (Equation 3) having N frequency components.

[0007]

[Equation 3]

Series-Coupled RN Generally Having Self-Loop
When comparing N and a series-coupled RNN without a self-loop in learning ability, a series-coupled RNN with a self-loop
It is known that the NN is superior to the series-coupled RNN without the self-loop.

[0009]

By the way, there is also a fully coupled type (not shown) in which each neuron is coupled to each other by adjoining each other. This fully connected RN
The learning rule of N is the error backpropagation method (e
rror Back Propagation /
(Abbreviated as BP method) is used. The basic equation of this BP method is represented by the following equation (Equation 4).

[0010]

[Equation 4]

Here, E is an evaluation function of learning from time T1 to time T2, y (t) is an output value of the output unit at time t, and T (t) is a teacher signal at time t. Here, in the BP method, the coupling coefficient w _ij is updated so as to minimize the evaluation function E according to the following equation (Equation 5).

[0012]

[Equation 5]

In this equation, the initial value of the coupling coefficient is given by a small random number. In the fully-coupled RNN, all other neurons are coupled to the output neuron, so B
In the P method, the error is propagated back to all the neurons. Therefore, the fully-coupled RNN can fully exert its ability by using the learning method called the BP method.

On the other hand, in the serially coupled RNN, not all the neurons are coupled to the output neuron.
The error does not directly propagate back. That is, the more the neuron is connected at a position farther from the output neuron, the more easily the error information is lost, and the learning of the coupling coefficient becomes difficult. this is,
This is because the activation value of the neuron far from the output neuron becomes small. As a result, series-coupled RN
In the case of N, there is a drawback that the original ability cannot be exhibited when the BP method is introduced. In particular, in the case of a series-coupled RNN without a self-loop, if the time series waveform is learned by the BP method, the point of t = 1 is not learned, and there is a problem that the approximation capability is hindered.

The present invention has been made to solve the above problems, and the technical problem thereof is that a series-coupling type RNN capable of reasonably performing learning by the BP method and sufficiently exhibiting its original ability. To provide a learning method.

[0016]

According to the present invention, a plurality of neurons including an output neuron are connected in series, and a series combination using an error backpropagation method based on a predetermined evaluation function in a learning rule. In the learning method of the type RNN, the error back-propagation method is based on an additional term type evaluation function in which an additional term for making the coupling weight close to zero is added to a predetermined evaluation function for the serially coupled RNN having a self-loop. A series-coupling RNN learning scheme is obtained.

Further, according to the present invention, the above-mentioned series-coupled RN is used.
By performing N learning methods, a series-coupled RNN that forms a series-coupled RNN without a self-loop is obtained.

[0018]

EXAMPLES Examples will be given below to illustrate the series-coupled R of the present invention.
The NN learning method will be described in detail. First, the outline of the learning method of the series-coupled RNN of the present invention will be briefly described. This learning method adds an additional term for making the coupling load close to zero to the series-coupling type RNN having a self-loop as shown in FIG. 1 and adds the additional term form obtained by this. BP based on the evaluation function E '
It is the act of law. Here, the predetermined evaluation function E is represented by the following equation (Equation 6), and the additional term is represented by (Equation 7).

[0019]

[Equation 6]

[0020]

[Equation 7]

Here, ωii represents the self-loop coupling load in the i-th unit. Therefore, the additional term type evaluation function E ′ in the BP method used in the present invention is expressed by the following equation (Equation 8).

[0022]

[Equation 8]

If learning is performed by the BP method based on the equation (8), the value of the evaluation function E changes so as to decrease. Therefore, the self-loop coupling weight ω _ii in the i-th unit.
The value of decreases to zero.

Further, if the additional term evaluation function E'is usually smaller than a predetermined value and the number of times of learning reaches a prescribed number of times, the learning is ended. At the end of this learning, the series-coupled RNN having a self-loop is used. A series-coupled RNN without a self-loop as shown in FIG. 2 is obtained. That is, if the learning by the BP method is performed, the learning in the series-coupled RNN having a self-loop substitutes the learning in the series-coupled non-self-loop RNN without the self-loop (its approximation ability, parameter estimation ability, etc.). Done in.

[0025]

As described above, according to the present invention, the additional term obtained by adding the additional term for making the coupling load close to zero to the series coupling type RNN having the self-loop is obtained by adding to the conventional evaluation function. Since the BP method is performed based on the shape evaluation function, the learning of the series-coupled self-loop type RNN having the self-loop can be performed by the series-coupled non-self-loop type RNN without the self-loop
You will be able to learn in the form of acting on your behalf. As a result, the learning by the BP method is performed rationally, and the learning method of the series-coupling type RNN capable of sufficiently exerting the original ability is provided. Therefore, the memory for the computer to be used, which is the advantage of the series-coupling type RNN, is provided. In order to reduce the number, the improvement of learning ability is guaranteed as never before, and it is expected to make a great contribution to the computer industry.

[Brief description of drawings]

FIG. 1 is an explanatory diagram of a basic configuration of a series-coupled self-loop type RNN according to a learning method of a series-coupled RNN of the present invention.

FIG. 2 is an explanatory diagram of a basic configuration of a series-coupling non-self-loop RNN according to a learning system of the series-coupling RNN of the present invention.

Claims (2)

[Claims] 1. A learning method for a serially coupled recurrent neural network, which is formed by connecting a plurality of neurons including an output neuron in series and uses an error backpropagation method based on a predetermined evaluation function in a learning rule. The propagation method is
A series-coupling type recurrent neural network having a self-loop, which is performed based on an additional-term type evaluation function obtained by adding an additional term for bringing a connection weight close to zero to the predetermined evaluation function. Recurrent neural network learning method. 2. A learning method for a series-coupling recurrent neural network, wherein a series-coupling recurrent neural network without a self-loop is formed by performing the learning method for a series-coupling recurrent neural network according to claim 1.