JPH0243665A - Neural network device - Google Patents

Abstract

PURPOSE:To improve a learning speed and a processing speed by composing the variable synapse coupling of a neural network of a weight memory circuit constituting a rewritable memory circuit and of a multiplying circuit to multiply the output of a neuron on a front stage by the weight of the variable synapse coupling. CONSTITUTION:Weighted connection lines 6 to 11 are respectively composed of a weight memory circuit 14 constituting the rewritable memory circuit and a multiplying circuit 15 to multiply the output of the neuron on the front stage by the weight of the weighted connection lines 6 to 11. Further, the weight of the weighted connection lines 6 to 11 is held by the weight memory circuit 14, the output of a neuron 1 is multiplyed by the weight held in the weight memory circuit 14 by the multiplying circuit 15, and the obtained value is inputted to a neuron 2 on a rear stage. Consequently, the variable synapse coupling having the weight >= a ternary which dynamically changes in a learning process can be realized by special hardware. Thus, the learning speed and processing speed can be improved.

Description

[Detailed description of the invention] [Industrial application field] The present invention relates to a neural network device.

In particular, the present invention relates to a neural network device that uses dedicated hardware to realize weighted connection lines (hereinafter referred to as variable synaptic connections) of a neural network that have three or more weights that dynamically change during the learning process.

[Prior art]

Recently, with the development of semiconductor technology and computer technology, research on neural networks has become active again mainly in the United States, and it has also been proposed to implement neural networks in hardware using LSIs and optical elements.

For example, in a three-layer neural network device,
A neural network consists of an input layer, a middle layer, and an output layer, and the units forming each layer are connected in each direction by variable synaptic connections. Therefore, when a certain unit receives a plurality of inputs, the total weight of the output of the previous unit and the weight of the variable synaptic connection becomes the input value. Furthermore, when input data is given to each unit in the input layer, this data is converted in each unit, transmitted to the intermediate layer, and finally output from the output layer. At this time, the actual output and the desired output (expected value) are compared, and the weight of each variable synaptic connection is changed so as to reduce the difference.

Furthermore, a typical learning algorithm in such a neural network is back propagation.

In this backpropagation, the weights of variable synaptic connections are changed from the output layer to the intermediate layer, and from the intermediate layer to the input layer, so as to reduce the difference between the actual output value and the expected value given input data. Repeat this step. As a result, the weights of the variable synaptic connections are converged so that desired output data can be obtained for the input data.

Furthermore, when implementing variable synaptic connections using hardware, the conventional method was to electrically connect or disconnect the outputs of neurons realized by amplifiers or the inputs to neurons.

In this case, the values of the variable synaptic connection weights are -1 (inhibitory connection present), O (no connection), +i (excitatory connection present)
It is possible to statically program three values.

In addition, related to this type of device.

For example, Japanese Patent Application No. 113238/1983 can be mentioned.

Regarding conventional neural network devices, for example, ``neurocomputer research, which is accelerating mainly in the United States,
Development 2 Nikkei Electronics (January 26, 1987 issue Lpρ, 159-1702 Nikkei McGraw-Hill) is discussed.

[Problem to be solved by the invention]

The problem with the above conventional technology is that it cannot be applied to the realization of variable synaptic connections in a neural network that dynamically changes during the learning process and has variable synaptic connections with weights of three or more values, such as a back propagation type network. The purpose of the present invention is to improve such problems and to replace the variable synaptic connections of a neural network by using a weight memory circuit composed of a rewritable memory circuit and a variable synaptic connection for the output of the previous stage neuron. By configuring it with a multiplier circuit that multiplies weights, it is possible to realize variable synaptic connections in hardware for neural networks with three or more weights that dynamically change during the learning process, improving learning speed and processing speed. An object of the present invention is to provide a neural network device.

[Means to solve the problem]

In order to achieve the above object, the neural network device of the present invention has variable synaptic connections with weights of three or more values that dynamically change during the learning process. It consists of a weight storage circuit made up of circuits, and a multiplication circuit that multiplies the output of the previous stage neuron by the weight of the weighted connection line.The weight storage circuit holds the weight of the weighted connection line, and The output of the neuron and the weight held in its weight storage circuit are multiplied by its multiplication circuit,
It is distinctive in that it inputs to later-stage neurons.

[Effect]

In the present invention, the variable synaptic connections of the multilayer neural network device are configured with a rewritable weight storage circuit and a multiplication circuit that multiplies the output of the previous stage neuron by the weight of the variable synaptic connections, thereby facilitating the learning process. Variable synaptic connections with three or more weights that dynamically change can be realized using dedicated hardware.

For example, when a register is used as a weight storage circuit, in order to store the weight of a variable synaptic connection connecting a certain unit in the input layer and a certain unit in the intermediate layer during the learning process,
The intermediate layer unit writes binary data indicating the sign and absolute value of the weight into a register, and the register holds this.

Furthermore, if the output value of the unit in the input layer is a binary value of 0.1 and the multiplication circuit is configured with an AND gate, if the output value is 0, the input to one of the AND gates of the multiplication circuit is 0. Therefore, the input value to the middle layer unit is O.

Further, if the output value of the input layer unit is 1, the input to one of the AND gates of the multiplication circuit becomes 1, and the output of the AND gate matches the output value of the register. This results in
The output value of the previous unit is multiplied by the weight of the variable synaptic connection.

By configuring variable synaptic connections using these circuits, the learning speed and processing speed of the neural network device can be improved.

〔Example〕

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 2 is a configuration diagram of a multilayer neural network to which the present invention is applied.

In FIG. 2, 1 is an input layer unit (A.

B), 2 is an intermediate layer unit (C, D), and 3 is an output layer unit (E). Also, line 6 with an arrow added
11 is a connection line with weight (variable synaptic connection), 4,
5 is an input line to the input layer unit, and 12 is an output line from the output layer unit. Note that each variable synaptic connection between each one is given a weight determined by a suitable algorithm (eg, back propagation algorithm).

For example, the weight of the variable synaptic connection connecting unit 1 in the input layer and unit 2 in the hidden layer is +0.8.

When the output of unit 1 is 1, the input value from unit 1 to unit 2 is +0.8.

FIG. 1 is a configuration diagram of a variable synaptic connection connecting layers of a multilayer neural network device according to an embodiment of the present invention, and FIG. 3 is a configuration diagram of a multiplication circuit of variable synaptic connections according to an embodiment of the present invention. It is.

In FIG. 1, 13 is a variable synaptic connection connecting input layer unit 1 and intermediate layer unit 2, 14 is a rewritable weight memory circuit, 15 is an (output) x (weight) multiplication circuit, and 16 is a multiplication circuit. An input line to the circuit 15, 17 an output line from the unit 2, 18 a signal line to the weight storage circuit 14, 19 an input line to the multiplication circuit 15, and 2o an input line to the unit 2.

Also, the weights of the connection lines are -0, 1, -0, 8, -0°6,
-○, 4, -0.2. O1+0.2,+'0,4,
+0.6, +0.8. It is assumed that 11 values of +1.0 are given, and the weight values are dynamically changed by an appropriate learning algorithm. Further, the output value of the neuron is assumed to be binary, 0 or 1.

First, the operation of the weight storage circuit 14 will be described.

Note that in this embodiment, a register is used as the weight storage circuit.

Now, the weight of the variable synaptic connection 13 connecting unit 1 and unit 2 at a certain time in the learning process is +0.8.
Suppose it was.

In this case, the unit 2 writes the data "1101" in binary to the register 14, which is a weight storage circuit, via the input line 18. Note that the first bit of this data indicates the sign of the weight. positive, O or negative when 0),
The remaining 3 bits give the absolute value of the weight.

As a result, since the first bit is 1, the sign of the weight is positive, and the remaining 3 bits are "101" in binary, so they represent 5 in decimal, and the absolute value of the weight is 5.

Therefore, it becomes the fifth weight counting from 0, that is, +0°8.

In this way, in this embodiment, the weight of the variable synaptic connection 13 is stored by the register 14.

Next, the operation of the multiplication circuit 15 will be described. In this embodiment, the multiplication circuit 15 is configured with an AND gate 21 as shown in FIG.
Multiply by a weight greater than or equal to the value.

For example, if the output of unit 1 in the input layer is 0, the AND gate 21 in the (output) x (weight) multiplication circuit 15
Since one input to is 0, its output is register 1
It becomes 0 regardless of the output value of unit 4, and the direct input to unit 2 also becomes 0. Also, if the output of unit A is 1, (
Since one input to the AND gate 21 in the (output)×(weight) multiplication circuit 15 becomes 1, its output matches the output value of the register 14.

In this way, the (output)×(weight) multiplication circuit 15 multiplies the output (0 or 1) of the unit 1 of the input layer by the weight of the variable synaptic connection.

In this embodiment, the variable synaptic connection 13 connecting the input layer unit 1 and the intermediate layer unit 2 has been described, but other synaptic connections can be similarly realized using dedicated hardware.

Further, although the case has been described in which the number of weight values is 11, it is clear that this number can be easily expanded.

Furthermore, although the case where the output value of the neuron is set to two values of 0 and 1 has been described, by applying this embodiment, variable synaptic connections can be easily created using dedicated hardware even for cases where the output value of the neuron is set to two values of 0 and 1. Can be configured with ware.

〔Effect of the invention〕

According to the present invention, the variable synaptic connections of a neural network can be configured with a rewritable memory circuit and a multiplication circuit, so that the variable synaptic connections of a neural network with weights of three or more values that dynamically change during the learning process can be configured with a rewritable memory circuit and a multiplication circuit. It is possible to realize this with dedicated hardware.

Furthermore, compared to a method of realizing variable synaptic connections using software on general-purpose hardware, learning speed and processing speed can be improved.

[Brief explanation of the drawing]

FIG. 1 is a configuration diagram of variable synaptic connections connecting layers of a multilayer neural network device according to an embodiment of the present invention, FIG. 2 is a configuration diagram of a multilayer neural network to which the present invention is applied, and FIG. 3 1 is a configuration diagram of a multiplier circuit with variable synaptic connections in an embodiment of the present invention. FIG. 1: Input layer units (A, B), 2: Intermediate layer units (C, D), 3: Output layer units (E). 4.5: Input line to input layer unit, 6 to 11: Connection line with weight (variable synaptic connection), 12: Output line from output layer unit, 13: Variable synaptic connection, 14
:Writable weight storage circuit (register), 15:(
Output) x (weight) multiplication circuit. 16: Input line to squaring circuit 15, 17: Output line from unit 2, 18: Signal line to weight storage circuit 14, 19:
Input line to multiplication circuit 15, 2o: input line to unit 2. Patent applicant: Nippon Telegraph and Telephone Corporation Agent: Masatoshi Isomura

Claims (1)

[Claims] (1) In a neural network device having a weighted connection line with weights of three or more values that dynamically change during the learning process, the weighted connection line is connected to a weight storage circuit configured with a rewritable storage circuit and a previous stage. and a multiplier circuit that multiplies the output of the neuron in the weighted connection line by the weight of the weighted connection line, and the weight of the weighted connection line is held in the weight storage circuit, and the output of the neuron in the previous stage is stored in the weight storage circuit. A neural network device characterized in that the multiplication circuit multiplies the held weights and inputs the result to a subsequent neuron.