JPH03210686A - Neural network device - Google Patents

Abstract

(57) [Summary] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

[Detailed Description of the Invention] "Industrial Application Fields" This invention simulates biological nerve cells and the connections between them to improve memory, reasoning, judgment, prediction, J-picture, and control.

It relates to neural network devices that perform pattern recognition, optimization, etc.

[Prior art] Fig. 2 shows, for example, the "neural computer" Aihara
In Figure 0, which is an explanatory diagram showing the configuration of a general feedforward type neural network device published in Sachi, Tokyo Denki University Press, 1988, pages 110 to 114, (11) is Elements that simulate biological nerve cells (hereinafter referred to as neural elements), (l la) are neural elements that constitute the human power layer, for example, N input layer neural elements, (Ilb)
are neural elements that constitute the intermediate layer, for example, N intermediate layer neural elements; (lie) are neural elements that constitute the output layer, for example, N output layer neural elements. (12) is an element (hereinafter referred to as a coupling element) that simulates a synapse between neural elements (!l), and the strength of the connection is called the coupling weight (W)1
In the figure, the arrow indicates the moving direction of the signal, which is the forward direction.

In this neural network device, the neural elements (Ill) are connected in a layered manner, and the dynamics are determined by the human power layer (Ill).
The human input signal input from 0 (la) is propagated to the output layer (I Icl) via the intermediate 0 (Ilbl) in the direction shown by the arrow. *dt' is the i-th output data of the 9th 7th learning data in the output layer, and uhJ is the j-th neural element (Ill
Let vI′IJ be the internal state of the j-th neural element (
! l), Whl is the first neural element (II) in the layer, and h+ is the output value of the jl-th neural element (+) in the layer.
1). The human input data of the learning data is the same as the output fllIV't at the input [(Ila). In the embodiment shown in FIG. 2, i=l~N, j=l~
It is N.

The relationship between each variable is as shown in equation (1) and equation (2).

Here, the function g only needs to be a differentiable and non-decreasing function, and Equation (3) is used as an example.

uh, = Σwh-', v''-', -1
+JV hJ= g (u ”J)
・ ・ ・ (2) g (x) = 1/ (1
+exp (-x)1・(3) Furthermore, connection weight<W
) are determined sequentially by the learning equation. That is, the output layer (
It is determined using the steepest descent method regarding the squared error defined by the science and data in (llc) and the calculation output obtained at the moment by the neural network. The number of layers of the neural network is H
Then, the squared error (hereinafter also referred to as energy) becomes as shown in equation (4).

E=(1/21ΣΣ(V'1-dl')"...(4
) Also, to sequentially change the connection weights, α, β are appropriate parameters, δ 1111 is the error in the hrgth neural element (Equation (5) 1 Equation (6)), and when using the method of moments, is the formula 0 (
5) In Equation (6), h = H indicates the value in the output layer. Nlh' is the number of neural elements in the second layer.

51''= (d, '-V',)V', (1-V',)
Here, i=1~N1111...(5)5
+ ”' = V ”t (IV ”t)Σδ lb+
1lWh, i where i = 1-N ''', h = H~
l...(6)d ” W'Jl /dt” +
(1-a) d W”, t /dt.

=-βa E/ +3 W"-i...
(7) For example, if you try to execute the learning equation of Equation (7) on the current Neumann-type computer, since the computer is digital, it is necessary to differentiate Equation (7) and execute it sequentially. An example of the flowchart is shown in FIG. 3. In step (30), the connection weight W at 1=1,
” and the initial value of the connection weight update amount ΔW h, , are set by uniform random numbers or normally distributed random numbers. At this time,
For each layer, all of i=l to N and j=1=N are set, but in FIG. 3, for simplicity, they are written as h, j, and i. In step (31), initial setting is made as t=1. In step (32), the error δ, IH in the output layer (l lc) is set as the initial value of the error signal.
Using the zero-order backpropagation method to find I using equation (5),
- Calculate the error signal of the layer F from the error chair number on the layer based on equation (6) (step (33)), step (34)
), the connection weights are updated using the method of moments. For this reason, first, the update amount Δ”J at the ha-th weight index +1
l (t + 1) is obtained using equation (8).

ΔW"4t (t + 1) = β5 j""I
V"10αΔW"JI (L) ... (8) α and β in this formula are predetermined parameters. Next, ΔVV"JI (t + 1) is used to create a new weight W+"J1 (1 -+1) is determined by equation (9).

W"4t (t, + 1) = W"JI (L) +
ΔW″1(shi+1)...(9) In step (35), the internal state u h of each layer is calculated by forward propagation using equations (1) and (2), output (1f
iV"J is calculated, and the output VH, as a collection of neural networks in the output layer H is calculated. Using this output V'4. and the learning data dl", equation (4) is calculated, and the squared error E is calculated.
seek. When this E is sufficiently small, for example, give a tolerance A in advance and judge the magnitude relationship between E and A (step (36)). When E is less than A, learning is finished and E is greater than A. When it is large, L=t, +1 (step (371) and the process from step (32) is repeated.

[Problem to be solved by the invention] The conventional neural network device is configured as described above.
α and β are the learning equation parameters that determine the connection weights.
Because it always uses predetermined constants, there was a problem that the learning speed was slow.

This invention was made to solve the above-mentioned problems, and aims to provide a neural network device that can increase learning speed.

[Means for Solving the Problems] The present invention has a differentiable and monotonically increasing function that simulates biological nerve cells as an input/output characteristic, and a plurality of functions that constitute a human layer, an intermediate layer, and an output layer. It is equipped with a neural element and a connecting element that multiplies each output of the neural element by a connection weight and outputs it to the neural element of the next layer, and sends input data to the neural element of the input layer and output data to the neural element of the output layer as learning data. The connection weights are sequentially updated based on the learning equation, and each neural element in the intermediate layer calculates the summation of the signals through the connecting elements that connect with the neural elements in the previous layer. In a neural element that obtains a calculation output and locally minimizes the energy defined by the sum of squares of the difference between the output data and the calculation output, the combination is determined by determining the coefficients of the learning equation using the instantaneous energy. It is configured to dynamically change the weight.

[Operation] The neural network device according to the present invention dynamically changes β, which is a parameter of the 7 Xi equation that determines connection weights, during learning based on the energy at that time.
Improve learning speed.

[Example] In the learning equation (formula (7)), the time T required for learning
is obtained by 2〇〇).

T=S(1/(2β(Eo −E)) ””) d
s...(10) However, here, Eo is the energy at the initial value, and E is the energy in the middle of convergence. Therefore, if you want to increase the time ′■゛, you just need to make the denominator in the integral close to a constant even in the middle of convergence.
Therefore, β=β. / (E, -E) ... (11)
Alternatively, in order to prevent β from diverging, β = β −in * (Eo + γ
)/(E. −E + γ) However, γ=β□, *Eo/(β, □−β□,)...(12)
Figure 4 is a graph showing the energy E (Figure 4 (a)) and β (Figure 4 (b)) that change according to equation (12) against L, and the energy E is When varying with respect to L from Eo to O, β becomes β11. From β
It shows how it changes up to 1.1.

This algorithm can be executed in parallel by hardware such as silicon or optical elements, or by an analog computer, but for example, an example of a flowchart when executed sequentially with a Neumann needle r1 iso is shown in step (40). ) connection weights W, ,' at deshi = 1
and the connection weight changer ΔWh.

Nao's initial HL! is set by a uniform fortune telling number or a normally distributed AL number. At this time, each layer 1=lNH, j=1~I
Settings are made for all I, but for simplicity, they are written as h, j, and i in FIG. L in step (4I)
Initialize as =1. In step (42), the error δ,′ in the output M(IIG) is set as the initial value of the error signal.
-
The error signal δ lI+1 of the lower layer is calculated from the error signal of the upper layer based on equation (6) (step (43)). Step (44) uses equations (1) and (2) to calculate the internal state of each layer u h by forward propagation.
,, calculate the output 1i [V ''a and obtain the output VHj as a collection of neural networks at the output NH. Using this output V'' and the learning data d+', calculate equation (4) and square it. Calculate the error E (step +451). Next, calculate β from this energy E using, for example, equation (12) (step (46)). In step (47),
The connection weights are updated by the method of moments using the calculated U. For this reason, first, the update amount Δ"J+(L+1) at 1+1 of the weight of the first layer is calculated using equation (8). In this equation, α is a predetermined parameter, and β is the one calculated in the previous step. Then, a new weight WhJl (L+1) is calculated using equation (9) using ΔWl'Jl (+1) for the zeroth order that is dynamically changed one after another.When the squared error E is sufficiently small, for example, Give the allowable error and judge the magnitude relationship between E and 8 (step (48)). If E is less than or equal to A, the chronological concavity is finished, and E
is larger than A, t=l;+1 (step (49
As 1), the process from step (42) is repeated.

In this way, in the embodiment described above, the learning speed is improved by dynamically changing β, which is a parameter of the learning equation that determines the connection weight, during learning based on the energy at that time.

In the above embodiment, the learning speed is improved by dynamically changing β during learning using the equation (12) based on the energy at that time, but the invention is not limited to this.

[Effects of the Invention] As described above, according to the present invention, a function that simulates biological nerve cells and has a differential = J function and is monotonically increasing is used as an input/output characteristic, and a human layer, an intermediate layer, and an output layer. i that constitutes
Multiply the number of neural elements and the output of each neural element by a connection weight to supplement the connecting element that outputs to the neural element in the next layer, and send human data to the neural element in the human layer and output data to the neural element in the output layer. It is set in advance as learning data, and the connection weights are updated sequentially based on the learning equation, and each neural element in the intermediate layer calculates the sum of signals through the connecting elements that connect with the neural elements in the previous layer and outputs them. By obtaining the calculation output in the neural elements of the layer and locally minimizing the energy defined by the sum of squares of the difference between the output data and the calculation output, the coefficients of the learning equation are determined using the instantaneous energy. By configuring the system to dynamically change the connection weights, it is possible to use a neural network device that can increase the learning speed.

[Brief explanation of drawings]

FIG. 1 is a flowchart for executing the seven learnings related to the neural network ': A position according to an embodiment of the present invention, FIG. 2 is an explanatory diagram for explaining the configuration of a general neural network device, and FIG.
The figure is a flowchart for executing learning related to a conventional device.
FIG. 4(a) shows the energy E according to an embodiment of the present invention.
A graph showing the characteristics of β with respect to time t, Fig. 4(b) is
It is a graph showing the characteristics with respect to time t. (11)...neural element, (12)...coupling element.

Claims (1)

[Claims] It has a differentiable and monotonically increasing function that simulates biological nerve cells as an input/output characteristic, and is connected to each of the plurality of neural elements constituting the input layer, intermediate layer, and output layer, as well as the outputs of the aforementioned neural elements. It is equipped with a coupling element that multiplies the weight and outputs it to the neural element of the next layer, and sets input data to the neural element of the input layer and output data to the neural element of the output layer as learning data in advance, and learns the coupling weight. Updates are performed sequentially based on the equation, and each neural element in the intermediate layer calculates the summation of signals via a coupling element that connects with the neural element in the previous layer, while obtaining a calculation output in the neural element in the output layer. , where the energy defined by the sum of squares of the difference between the output data and the calculation output is locally minimized, the connection weights are determined by determining the coefficients of the learning equation using the instantaneous energy. A neural network device characterized by dynamic changes.