JPH03210685A - Neural network device - Google Patents

Abstract

(57) [Summary] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

[Detailed Description of the Invention] [Industrial Application Field] This invention simulates biological nerve cells and the connections between them to improve memory, reasoning, judgment, prediction, planning, and control.

This invention relates to a neural network device that performs pattern recognition, Wi-Fi communication, etc.

[Prior art] Figure 4 shows, for example, the "neural computer" Aihara-
In Figure 0, which is an explanatory diagram showing the configuration of a conventional feedforward type neural network device published in Sachi, Tokyo Denki Oaza Publishing Bureau, 1988, pages 110 to 114, An element (hereinafter referred to as a neural element) that simulates the neurons of Two intermediate layer neural elements (Ilc) are neural elements that constitute the output layer, for example 1
There are two output layer neural elements. (12) is the neural element (1
11 (hereinafter referred to as a coupling element), and the strength of the coupling is called a coupling weight.

In this circuit network, the neural elements (!Otori) are connected in layers, and in terms of Guinamitics, the input signal that enters from the input layer (lla) is output via the intermediate layer (tub).
l(Propagated to IIG+. Quantitatively, it is as follows. ed+', which gives a pair of input data and output data in advance as learning data, is the i-th output of the second learning data in the output layer. The data, uh, is the internal state of the fifth neural element (11) in the second layer, ■1. is the output value of the jl)th neural element + 111 in the second layer, and WhJ is the i-th neural element in the second layer. Let it be the connection weight between the neural element 1ll) and the first neural element (ll) in the h+1th layer. The input data of the learning data is the same as the output (i V' j in the input layer (!Ial).

In the example shown in FIG. 4, each subscript has a structure as shown in Table 1. In addition, the relationship between each variable is expressed by Equation (1) and Equation 3 (
2), where the function g only needs to be a differentiable and non-decreasing function, using equation (3) as an example.

uIIJ=Σw"−'Jtv"−', ---(
+)V”J=g (u”i) ・・・(
2) g (x) = 1/ (1+exp (-xl)
...(31 Table 1) Furthermore, the connection weights (W) are sequentially determined by the learning rule shown in equation (4). That is, the learning data in the output layer (l lc) and the neural network are used to It is determined using the steepest descent method regarding the squared error defined by the calculation output obtained in ).

E= (1/2)X:I (V', -di')"
--+41 Also, to sequentially change the connection weights, α and β are appropriate parameters, δ lhl is the error in the th layer and the i-th neural element (Equation (5), 9 Equation (6)), and the method of moments is used. In this case, 0 can be implemented using the learning equation of equation (7).
In equations (5) and (6), when h=H, the value in the output layer is shown. Note that in this example, 1■=3. N'1 is the number of neural elements in h@th.

δ,”'= (ct,'-V',) V', (1-
V”,) where i = l, N tl′ll
...(5) δ1"""V"1 (1-V"+)X5
4'h"'W"1 where i=l~NLh1. h=1■
~l...(6)d ” whJt /dt,” + (
1-α) a W ”J l /at=-βaE/a
Wl″1...(7) For example, if you try to execute the learning equation in equation (7) on a current Neumann computer, since the computer is digital, it is necessary to differentiate the equation and execute it sequentially. An example of the flowchart is shown in FIG. 5. In step (30), initial values of the connection weight W''1 and the connection weight update amount ΔWhJ at L=1 are set by uniform random numbers or normally distributed random numbers. At this time, when h=1 to 2 and h=t, i=1 to 3. j=
1-2, when h=2, i=1-2. Settings are made for all of J=l, but for the sake of simplicity in Figure 5, h, j,
It is written as i. In step (3), initialize as shi = 1. In step (32), the error δ II+ in the output layer (11c) is determined by equation (5) as the initial value of the error signal using the zero-order backpropagation method, and the error signal on the − layer is converted to the error signal on the layer below. is obtained based on equation (6) (step (33)), and this back propagation is shown in FIG. 6. In step (34), the connection weights are updated by the method of moments. For this reason, first, the update IΔW''JI (71) at the weight number +1 is calculated using equation (8).

ΔW”JI (t + 1) = (J 6 J”
"'V"+06w"Jt (t) (8) α and β in this equation are predetermined parameters. Next, a new weight W"JI ( t, + 1 ) is determined by equation (9).

W”JI (t-z)=W”JI (t)+Δw”
, t (t+1) (9) In step (35), the internal state u h J and output value VhJ of each layer are calculated using equations (1) and (2) by forward propagation, and the output value VhJ in the output layer H is calculated. The output VH as a collection of neural networks is determined. Using this output vHJ and the 7 learning data d+', the formula (
4) to find the squared error E. When this E is small enough, for example, by giving a tolerance A in advance, E
Step +36)) to determine the magnitude relationship between and A.
When E is less than or equal to A, learning is terminated, and when E is greater than A, L=t+1 (step (37)) and the process from step (32) is repeated.

[Problem to be solved by the invention] The conventional neural network device is configured as described above.
It was slow for the error E as a collection to fall below the error tolerance value A, and the speed of learning was slow. Furthermore, there is a problem in that the error E does not become less than the allowable value A and converges to a locally optimal solution.

This invention was made to solve the above problems, and it is possible to increase the learning speed and improve the probability of converging to the global optimal solution instead of converging to the local optimal solution. The purpose is to obtain a network device.

[Means for Solving the Problems] The present invention provides a neural element group in which a plurality of neural elements constituting an input layer, an intermediate layer, and an output layer are divided into groups of a predetermined number, and a neural element group that is connected to each of the outputs of the neural elements. It is equipped with a coupling element that multiplies the result by a coefficient and outputs it to the neural element of the next layer.Input data is set in advance to the neural element of the input layer, and output data is set to the neural element of the output layer as learning data, and the neural element of the input layer is The operating states of the neural elements in the output layer are assigned substantially the same within the same group, and the neural elements in the intermediate layer perform summation calculations on the signals via the coupling elements that connect with the neural elements in the respective layers, while performing calculations in the neural elements in the output layer. The configuration is such that the energy defined by the output data and the square of the calculation output is locally minimized.

[Operation] The neural network device according to the present invention makes at least the neural elements of the input layer redundant, and makes the connection weights redundant and connects them, thereby equivalently achieving a sigmoid function which is a neuron transfer function. The gradient is made steeper to speed up learning, and the redundant connection weights become equivalently normally distributed according to the law of large numbers, improving the probability of convergence to a globally optimal solution.

[Embodiment] FIG. 1 is an explanatory diagram showing the configuration of a neural network device according to an embodiment of the present invention. In FIG. ), (lla) is the neural element that constitutes the input layer, for example, six input layer neural elements, (llb) is the neural element that constitutes the intermediate layer, for example, four intermediate layer neural elements, and (llc) is the output layer. The constituent neural elements are, for example, one output layer neural element. (12) is an element (hereinafter referred to as a coupling element) that simulates a synapse between neural elements +II), and the strength of the coupling is called a coupling weight.

In this circuit network, the input layer neural element (l la) has double redundancy, and the neural element V' (■) and the neural elements Vl and iml constitute the same neural element group.

Neural element Vl+I+ and neural element y11!1 neural element v
'(1) and neural element Vllll' similarly constitute the same neural element group. The outputs of these redundant neurons are all the same, that is, the operating states of the neural elements (lla) in the input layer are assigned substantially the same within the same group. In this embodiment, the 29th (llb) neuron constituting the intermediate layer also has double redundancy like the first layer. However, in the second layer (1lb), the outputs of neurons made redundant like the input layer (lla) are not necessarily the same, and redundancy is not necessary in the output layer (I Ic). In addition, in Figure 0, where the connection weights are also redundant along with the redundant input layer (l la), the signal flow from the input layer (11a) to the output layer (l lc) is controlled by the neural element ( I l
) and connection weight (12) are now redundant, but are the same as before. That is, if m is the number of redundancies, ufi, 1ml
is the internal state of the fifth neural element in the second layer, Vll, 1
111 is the output value of the fifth neural element of the th layer, Wl+,
, 1ml is the connection weight between the first neural element in the th layer and the jth neural element in the h+1th layer. The input data of the learning data is the output t of the input layer (lla)
Same as tiv'11.

In the example shown in FIG. 1, the configuration is as shown in Table 2. Moreover, the relationship between each variable is as shown in Equation (11) and Equation (12), where the function g may be a non-decreasing function with differentiability, and the equation (Is) is used as an example.

Table 2 uh, l@l==ΣW11-1. , lal V h-
t, 161...(ll) Vll lal =
:g (uhJl”l) −・、・
++2)g (x) =1/ (1+qxp (
−x)l −−(+31 Furthermore, the connection weight (W) is sequentially determined by the learning rule shown in equation (14). That is,
It is determined using the steepest descent method regarding the squared error defined by the learning data in the output layer (l Ic) and the calculation output actually obtained by the neural network. When the number of layers of the neural network is H, the squared error (hereinafter also referred to as energy) is expressed as in equation (14).

E=(1/21ΣΣΣ(VH, 161-d, Ill@
l) I (14) Also, to sequentially change the connection weights, a and β are appropriate parameters, δ lhl lal1 is the second layer,
Error in the i-th neural element (Equation (15), Equation (1)
61), and when using the method of moments, the equation (17
) can be executed using the learning equation (Is), equation (16)
In, when h=H, the value in the output layer is shown. Note that in this example, I (= 3).

5, 1111 lal, (d, 111+++1-v
ll, 1m1)yH, (1-yH1'ml) where,'
i = l, N 1111 lal...(
15) δ ll+l lal = V l+, ls
l (1-V h, 1m1) Book I X: δ, lh
lll lal whJ, l+ml where i=I~
N lhl lal, h= )(~l・(Ic)d
"W", 11"'/dt" ÷ (l - α)
d W ``Jr ''' / dt=-βclE/
aWI″Jt”′ --(17) For example, the formula (
An example of a flowchart for calculating the learning equation in step (40) is shown in FIG.
The initial values of J, l ml are set using uniform random numbers or normally distributed random numbers. At this time, when h=1 to 2 and b=1, i=1 to 3. When j=1-2, h=2, i=1-2. j
= 1, but for simplicity, h is shown in Figure 2.

j, i are written as M, and for m, m = 2 for both the input layer (lla) and the intermediate layer (llb), and m = 1 for the output layer (llcl). Initialize as t = 1 in step (41). In step (42), the output H (Ilc) is set as the initial value of the error signal.
k Equation (15) Gl: J: Using the linear backpropagation method, the error signal of the lower layer is obtained from the error signal on the - layer based on Equation (16) (step +43)), 3rd
The figure shows backpropagation. In step (44), the connection weights are updated by the method of moments. For this reason, first, the m-th update amount △W'' is multivalued at 1+1 of the weight of the
,,1'''')(t+B is determined by equation (18).

ΔW hJl ”'(shi+') =βδ +h*++
lal V h, 1ljl+αΔW"j, 1"
(t) ... (18) α and β in this equation are predetermined parameters of the first order, ΔW”, (”l
(t + t) to create new weights w",,'"'(
+i) is calculated using equation (19).

W"J, l"' (t, +t) = w"jI"' (t)
+ΔW","' (h+t)...(19) In step (45), equation (II) is obtained by forward propagation.

Using equation (12), the internal state of each layer u h l @ 1
．． The output value Vll, 1161 is calculated 5, and the output V14J'1 as a collection of neural networks in the output layer H is determined. Equation (14) is calculated using this output v8, 1 ml and the learning data d+"" to obtain the squared error E. When this E is sufficiently small, step 1, for example, give a tolerance A in advance and judge the magnitude relationship between E and A.
6]) When rE is less than or equal to A, learning ends, and E becomes A.
If it is larger than E=t+1 (step+47))
The process from step (42) is repeated.

In this way, in the above embodiment, at least the neural elements (lla) of the input layer are made redundant, and the connection weights are made redundant and connected, thereby effectively reducing the gradient of the sigmoid function, which is equivalently Nieron's transfer function. The connection weights, which are made steeper to speed up learning and made redundant, have an equivalent normal distribution according to the law of large numbers, improving the probability of converging to a global optimal solution.

We ran what is known as the XoRO problem. This is the input data of the learning data and TeV'l=
O, V 'x = Ot7), the output data is V3,
;0, 'v', = o, v', = o, output data value V' t = Olv +, = 0, V'
-= 1 (7) When the output data and t, tev', = i,
V'l=1. V', 20 (at 711, when output data V'l = 1.V', = 1.V'□ = 1, when learning is given v', = o as output data, the conventional device What used to take about IO minutes could be learned in about 2 minutes with the device according to this example.If the degree of redundancy is D, then the speed of the conventional device is l, //, depending on the nature of the data.
I was able to learn at around D+ and 'x. Furthermore, when providing redundancy to the neural elements in the middle layer, the operating states within the same group are the same, but if they are not the same and are determined completely using random numbers, the processing speed will be faster.

[Effects of the Invention] This invention provides neural element groups in which a plurality of neural elements constituting an input layer, an intermediate layer, and an output layer are divided into groups of a predetermined number, and each of the outputs of the neural elements is multiplied by a coupling coefficient. It is equipped with a coupling element that outputs to the neural element of the next layer, and input data to the neural element of the input layer and output data to the neural element of the output layer are set in advance as learning data, and the operating state of the neural element of the input layer is Substantially the same allocation is made within the same group, and each neural element in the intermediate layer performs a summation operation on signals via a coupling element that connects with the neural element in the previous layer, while obtaining a calculation output in the neural element in the output layer; ” A neural circuit that can increase the learning speed and improve the probability of converging to the global optimal solution by configuring it to locally minimize the energy defined by the output data and the square of the calculation output. There is an effect that a network device can be obtained.

[Brief explanation of drawings]

The It diagram is an explanatory diagram showing the configuration of a neural network device according to an embodiment of the present invention, FIG. 2 is a flowchart for calculating a learning equation according to this embodiment, and FIG. An explanatory diagram showing backpropagation, Fig. 4 is an explanatory diagram showing the configuration of a conventional neural network device, Fig. 5 is a flowchart for calculating the learning equation related to the conventional device, and Fig. 6 is an error related to the conventional device. It is an explanatory diagram showing back propagation of. (Ill...neural element, (12)...coupling element. In the figures, the same reference numerals indicate the same or equivalent parts.

Claims (1)

[Claims] A neural element group in which a plurality of neural elements constituting the input layer, intermediate layer, and output layer are divided into groups of a predetermined number, and the output of each of the neural elements is multiplied by a coupling coefficient and output to the neural element of the next layer. The input data to the neural elements in the input layer and the output data to the neural elements in the output layer are set in advance as learning data, and the operating states of the neural elements in the input layer are substantially the same within the same group. The neural elements in the intermediate layer each perform summation calculations on the signals via the coupling elements that connect with the neural elements in the previous layer, and obtain the calculation output in the neural elements in the output layer, and a neural network device configured to locally minimize energy defined by the square of the calculation output.