JPH03210684A - Neural network device - Google Patents

Abstract

PURPOSE:To speed up learning speed by executing calculation by using a term of first order and the term of second order as the derivative of the energy of a learning equation. CONSTITUTION:Input data and output data are set beforehand in the neural element 11a of an input layer and the neural element 11c of an output layer respectively as learning data, and coupled weight is updated sequentially on the basis of the learning equation, and the neural element 11b of an intermediate layer obtains calculated output at the neural element 11c of the output layer while it total-calculates a signal through a coupled element 12 coupling with the neural element 11a of the preceding layer, and minimizes locally the energy defined by the square sum of difference between the output data and the calculated output. At that time, the calculation is executed by using the term of first order and the term of second order as the derivative of the energy of the learning equation. Thus, the learning speed can be speeded up.

Description

[Detailed Description of the Invention] [Industrial Application Field] This invention simulates biological nerve cells and the connections between them to improve memory, reasoning, judgment, prediction, planning, and control.

The present invention relates to a neural network device that performs pattern recognition, RA optimization, etc.

[Prior art] Fig. 2 shows, for example, the "neural computer" Aihara
In Figure 0, which is an explanatory diagram showing the configuration of a general feedforward type neural network device published in Sachi, Tokyo'& Iso University Press, 1988, pages 10 to 114, (1 It is an element that simulates biological nerve cells (hereinafter referred to as
Called neural elements5), (I Ia) is a neural element that constitutes an input layer, for example, N input layer neural elements, (llb)
is a neural element constituting an intermediate layer, for example, N for one layer
(H-2) intermediate layer neural element, (llc)
are neural elements constituting the output layer, for example, N output layer neural elements.

(+2) is a neural element (an element that simulates a synapse between H
Hereinafter, the bond f is called a child], and the strength of the bond is the bond Φ
It is called Mi (W). In the figure, the arrow indicates the moving direction of the tilt sign, indicating the forward direction.

In this neural network device, the neural elements (11) are connected in layers, and the input layer (l
The human power signal input from ral is propagated to the output layer (l lc) via the intermediate layer (eye b) in the direction shown by the arrow 6
In terms of stability, it becomes more like a stripe. A pair of human data and output data is prepared in advance as learning data. d,' is the first output data of the second learning data in the output layer, u''4 is the J11\th neural element (11
), yh, is the J-long neural element (1
Let the output IIIt, W7j1 of 1) be the connection weight between the first neural element Tl l of the h Itl and the j4th neural element (eye) in the h+1th layer.The human data of the learning data is , input #(is the same as the output value VIJ at llal. In the embodiment shown in FIG. 2, i=l~N, j
=l~N.

The relationship between each variable is as shown in equation (1) and equation (2).

Here, the 2-numerical number g may be a differentiable and non-decreasing function, and equation (3) is used as an example.

11'1. :Σw′′−Zt vh−1, * + +
(11vhJ=g(u″,)...(2
)g (x) =1/ (1+exp (-xli ・
131 Furthermore, the connection weights (W) are determined sequentially with the learning equation. That is, it is determined using the steepest descent method regarding the squared error defined by the output data of the learning data in the output layer (I lc) and the calculation output actually obtained by the neural network. Letting the number of layers of the neural network be H,
The squared error (sometimes referred to as energy below) is expressed by the formula (
4) It will look like this.

E=(1/21 ΣΣ (d,'-V",)"
--+41 Furthermore, when the method of moments is used, the sequential change of the connection weights can be carried out using the learning equation of equation (5) by setting α and β as appropriate parameters.

d ” W ”Jl / dt” + (l-〇) d
W"JI /dt=-(JaE/δW'4+
...(5) For example, if you try to execute the learning equation of equation (5) on the current Neumann type calculation system, the computer will
Since it is f-digital, it is necessary to differentiate equation (5) and execute it sequentially, but an example of the flowchart for this process is shown in FIG. In step (32), training data, which is a pair of human input data and output data, is collected in order to train the neural network.In step (33), a learning equation using the first-order term of the energy differential value is calculated using the method of moments. (
5) and update the connection weights of the neural network. For this reason, first, IJrmΔWh at the weight index of the second layer +1.

B+t) is calculated. α and β in this equation are predetermined parameters. Also, for each layer i =
INN, j=lNN are all set, but h is omitted in FIG.

A new weight w'', + (s+t) is calculated using Equation (6) using Δ''Jl (t+1) for the 0th order, which is written as j, i.

W l′Jl (t + 1) = W ”Jl (
t) to W'Jt(t + 1). . (6) In step (34), forward propagation transforms equation (1).

is calculated, and the output vHJ as a collection of neural networks in the output layer H is determined. This output VW, and learning data d
,'' to calculate the squared error E by calculating equation (4).

When this E is sufficiently small, for example, the allowable error A
Step (351): When E is less than or equal to A, learning is finished; when E is greater than A, L=t+i (step (36)). Repeat the process from step (33).

[Problem to be solved by the invention] The conventional neural network device is configured as described above.
Since the right side of the learning equation (5) that determines the connection weights is only the first-order term of the energy differential value, there is a problem that the learning speed is slow.

This invention was made to solve the above-mentioned problems, and aims to provide a neural network device that can increase learning speed.

[Means for Solving the Problems] This invention performs differentiation a1 that simulates biological nerve cells, and calculates 1M neural elements constituting an input layer, an intermediate layer, and an output layer, and each of the outputs of the neural elements. It is equipped with a coupling element that multiplies the coupling by F and outputs it to the neural element of the next layer, and sets the input data to the neural element of the input layer and the output data to the neural element of the output layer as 7 learning data in advance, and sets the coupling weight. The neural elements in the intermediate layer are updated sequentially based on the learning equation, and each neural element in the intermediate layer calculates the summation of the signals through the connecting elements that connect with the neural elements in the previous layer, while obtaining the calculation output in the neural element in the output layer. In a system that locally minimizes the energy defined by the sum of squares of the difference between output data and calculation output, it is calculated using the first-order term and the second-order term as the differential value of the energy of the learning equation. .

[Effect]

The neural network device according to the present invention uses linear terms and quadratic terms when calculating a learning equation that determines connection weights, so that the learning speed can be improved.

[Example] In order to speed up learning by taking the energy surface into consideration, an example of the learning equation (formula (5)) that takes into account up to the second-order differential value of energy is transformed into formula (7). show.

d ” W”jt /dt” + (1-α) d W
"J l / dL; = -β (a E / a W"
JI)/((7” E/a W”,,”)...
(7) An example of how to prevent the learning equation from becoming unstable when the second-order differential term becomes zero and the right side of equation (7) becomes infinite is the learning equation shown in equation (8). .

In the formula, ρ is a sufficiently small positive constant.

d ” W ”Jt / dt” + (1-α) d
Wl',, /dt=-β(aE/δW"JI)/(1+pa"E/aW"Jl")...
・(8) Also, in equation (8), if we perform Tiller expansion of the second-order differential term and use up to the first-order term of the expansion result,
- As an example, the following learning equation (9) is obtained.

d ” WllJt /dt,” + (1-a
) d W”1/dt knee β (d E/ a W”J
I) *(1-p a ” E / a W”, i2)
· · · (!]) this'? An example of a flowchart when the learning equation 9) is executed sequentially on a Neumann type computer, for example, is shown in FIG.

In step (42), in order to train the neural network,
Collect training data that is a pair of input data and output data. In step (43), a learning equation (9) using both the first-order and second-order terms of the energy differential value is calculated by the method of moments, and the connection weights of the neural network are updated. For this reason, first hrf! tel with weight of 1
Calculate the updated balance ΔW"JI (L + 1) in this equation. α and β in this equation are predetermined parameters. Also, for each layer, i = l − N,
It is set for all j, = 1-N,
In Figure 1, for simplicity, the 6th order is written as h, j, i,
New weight W”J using ΔW”J, (t, +1)
, (jt1) are determined by equation (6). Step (44
), the internal state u h of each layer and the output value v″□ are calculated using equation (2) by forward propagation, and the output V as a collection of neural networks in the output layer H is calculated.
”, is calculated using this output vHJ and learning data d+' to calculate the squared error E. When this E is sufficiently small, for example, by giving a tolerance formula in advance, E and A Step (451)
), when E is less than or equal to A, learning ends at T, and when E is greater than A, t=L+1 (step (46)) and the process from step (43) is repeated.

In this way, in the embodiment described above, the learning speed is improved by considering the first-order term and the second-order term as the differential value of the energy on the right side of the learning equation that determines the connection weight.

Note that in the above embodiment, the second-order term of the fine molecule 11 was calculated as Equation (9) by Tiller expansion, but the present invention is not limited to this.

[Effects of the Invention] As described above, according to the present invention, the input layer has a differentiable and monotonically increasing function that simulates biological nerve cells as an input/output characteristic, and constitutes the input layer, intermediate layer, and output layer. a plurality of neural elements, and a coupling element that multiplies each output of the neural elements by a coupling weight and outputs it to the neural element of the next layer, and inputs human data to the neural element of the input layer and output data to the neural element of output 7JH. is set in advance as learning data, and the connection set is updated sequentially based on the learning equation, and each neural element in the intermediate layer is While performing calculations, a calculation output is obtained in the neural element of the output layer, and the energy defined as the sum of squares of the difference between the output data and the calculation output is locally minimized, and the first order is the differential value of the energy of the 7 Xi equations. By configuring the calculation to use the term and the quadratic term,
This has the effect of providing a neural network device that can increase learning speed.

[Brief explanation of drawings]

Fig. 1 is a flowchart for acquiring learning related to a neural network device according to an embodiment of the present invention, Fig. 2 is an explanatory diagram for explaining the configuration of a general neural network device, and Fig. 3 is a conventional one. 3 is a flowchart for executing learning related to the device of FIG. (11)・・Neural element, (12)・・Coupling element. Agent Masuo Oiwa Figure 11: Neural element 2: Connecting element 3 Figure procedural amendment (voluntary) 521 2008 Name of invention Neural network device 3 Person making the amendment Representative le+% Kimoriya 4, Agent 5゜6゜7゜Drawings to be amended Details of the amendment (1) Figure 2 of the drawing is corrected as shown in the attached document Self-sphere drawing (Figure 2). Figure 2 Il: God I Flea "Element 12: Sum wo Minechi"

Claims (1)

[Claims] It has a differentiable and monotonically increasing function that simulates biological nerve cells as an input/output characteristic, and is connected to each of the plurality of neural elements constituting the input layer, intermediate layer, and output layer, as well as the outputs of the aforementioned neural elements. It is equipped with a coupling element that multiplies the weight and outputs it to the neural element of the next layer, and sets input data to the neural element of the input layer and output data to the neural element of the output layer as learning data in advance, and learns the coupling weight. Updates are performed sequentially based on the equation, and each neural element in the intermediate layer calculates the summation of signals via a coupling element that connects with the neural element in the previous layer, while obtaining a calculation output in the neural element in the output layer. , where the energy defined by the sum of squares of the difference between the above output data and the above calculation output is locally minimized, and the calculation is performed using the first-order term and the second-order term as the differential value of the energy of the above learning equation. A neural network device characterized by: