JPH0443472A - Neural circuit network type calculation device - Google Patents

Abstract

PURPOSE:To speed up calculation by providing a sigmoid function generator giving a sigmoid function to the output of element chips and a broadcast chip which selects the output of the sigmoid function generator and data from an input and data from an input layer and supplies them to plural element chips in parallel. CONSTITUTION:Plural element chips 15 - 11 function as intermediate layers and output layers and they are provided in parallel. Shift registers 25 - 21 are respectively provided for respective element chips 15 - 11. They receive and hold outputs of respective element chips 15 - 11 and they are sequentially shifted. The sigmoid function generator 30 is provided at the end part of the shift registers 25 - 21 and gives the sigmoid function to the outputs of the element chips 15 - 11, which are sequentially transmitted from the last shift register 21. Furthermore, the broadcast chip 40 selects the output of the sigmoid function generator 30 and data from the input layer and supplies them to plural element chips 15 - 11 in parallel. Thus, calculation can be speeded up.

Description

[Detailed Description of the Invention] [Table of Contents] Overview Industrial Application Fields Prior Art (Figures 6 to 8) Problems to be Solved by the Invention Examples of Means and Actions for Solving the Problems Overall of the Invention Description of the configuration (Fig. 1) One embodiment of the present invention (Figs. 2 to 3) Other embodiments of the invention (Figs. 4 to 3)
Figure 5) Effects of the invention [Summary] Regarding a neural network type computing device that can be used as both a multilayer structure type network and a feedback type network, it is possible to achieve high-speed calculation in the neural network type computing device, and to achieve each A neural network computing device that functions as a multilayer network consisting of an input layer, at least one intermediate layer, and an output layer, with the aim of reducing the amount of hardware in the element chips. a plurality of element chips, which function as the intermediate layer and the output layer; a shift register provided for each element chip, which receives and holds the output of each element chip, and sequentially shifts the output; Provided at the end of the register,
a sigmoid function generator that applies a sigmoid function to the outputs of the element chips that are sequentially sent from the terminal shift register; and a sigmoid function generator that selects the output of the sigmoid function generator and data from the input layer to generate the plural and a broadcast chip that supplies the chips in parallel.

[Industrial application field]

The present invention relates to a neural network hardware simulator, and particularly to a neural network computing device that can be used as both a multilayer structure type and a feedback type network.

In recent years, with the demand for the application of neurotechnology, there has been a demand for a neural network type computing device that is capable of high-speed neuronal operations and has a small amount of hardware.

[Conventional technology]

Models for neural network computing devices (neural network computing devices) can be broadly divided into two types: multilayer networks, such as the perceptron, and single-layer feedback structures, such as the Hopfield model. be able to.

FIG. 6 is a diagram showing a multi-layered neural network computing device, and shows a neural network composed of three layers: an input layer, an intermediate layer, and an output layer.

As shown in FIG. 6, the multilayer neural network is composed of elements U called units that simulate neurons and connections that connect them. That is, the neural network has units II, -I
It is composed of three layers: an input layer with units U, an intermediate layer with units MU1""'Mus, and an output layer with units 001 to OU. Each unit is not connected to units in the same layer, but is connected to all units in the layer immediately below (above). For example, between units in the hidden layer, all units in the input layer l-IUI~T
For example, the unit OuI of the output layer is connected to U6 by the connection L+,1, and the unit OuI of the output layer is connected to all the units M of the intermediate layer.
It is connected to U, to MU by a connection LMO. Here, the input layer is a layer that receives input from the outside, and the output layer is a layer that outputs the response of the neural network to the outside. Furthermore, the intermediate layer is a layer that processes signals from the input layer and passes them to the output layer. Depending on the neural network, the intermediate layer may consist of multiple layers.

FIG. 7 is a diagram showing each unit (U) used in the multilayered neural network computing device shown in FIG.

As shown in FIG. 7, each connection between units has a weight. The function of each unit (excluding input layer units) is to output (V+
~V, , ) and its unit U, the connection weight (
1, ~ W, n) (Shiura 8. ~ Shi,
W+ , ,) is summed (Σ) over all units.
, and then a nonlinear function called a sigmoid function (Cf) is applied to the amount obtained by adding a threshold value. Here, the function of the i-th unit Ui(k) in the first layer (excluding the input layer) is expressed as follows.

■%kl=((ΣwiJ(k-11■, (k-11+
θ%kl)f (X) ='A (1+tanh (
X/Xo)) (2) However, ■, (k-11 is the output of the j-th unit of the (k-1)th layer, w
, (k-11 is the i-th unit of the layer and the (k-11)
k-1) is the weight of the connection with the j-th unit of the layer. Also, θ, (kl is the threshold i of this unit. Furthermore, equation (2) represents a sigmoid function, and x
o is a constant that determines the shape of the sigmoid function.

FIG. 8 is a diagram showing a feedback structure type neural network computing device, and specifically shows the Hopfield model.

As shown in FIG. 8, in the pump field type network, all neurons (units) are connected to each other. Each unit receives signals from all other units and its internal state U.

is changed according to the differential equation of equation (3).

Here, I8 is an input from the outside and corresponds to the threshold of the upper hierarchical network. The output of this unit is calculated as Vi +=f (ui) (4
) can be written. When the network is given a certain initial state, it settles into a steady state by repeating the interaction shown in equation (3).

[Problem to be solved by the invention]

As mentioned above, in both multilayer and feedback networks, neural network computation requires
Many product-sum operations and function calculations as described above must be performed for all units. This type of calculation was done by creating a program and having the computer perform it.
Calculations were performed sequentially for all units, which took a long time. Furthermore, in the above-mentioned neural network computing device, it is necessary to provide a sigmoid function generator for each unit, and in a multilayer neural network computing device, it is necessary to provide a unit for each layer. was there.

An object of the present invention is to achieve direct calculation in a neural network computing device and to reduce the amount of hardware of each element chip.

[Means to solve the problem]

FIG. 1 is a diagram showing the overall configuration of a neural network computing device according to the present invention.

According to a first aspect of the present invention, there is provided a neural network type computing device that functions as a multilayer network composed of an input layer, at least one intermediate layer, and an output layer, which are provided in parallel, A plurality of element chips 15 to 11 functioning as the intermediate layer and the output layer, each provided for each element chip 15 to 11, receiving and holding the output of each element chip 15 to 11, and sequentially shifting the output of each element chip 15 to 11. shift registers 21 to 25 provided at the ends of the shift registers 25 to 21;
a sigmoid function generator 30 that applies a sigmoid function to the outputs of the element chips 15 to 11 sent sequentially from 1; element chip 1
A neural network type computing device is provided, characterized in that it is equipped with a broadcast chip 40 that supplies signals 5 to 11 in parallel.

Further, according to a second aspect of the present invention, there is provided a neural network type computing device that functions as a network with a feedback structure, which includes a plurality of element chips that are provided in parallel and each of which is given an initial state in advance. 15 to 11, shift registers 25 to 21 that receive the outputs of the respective element chips 15 to 11, hold the outputs of the respective element chips 15 to 11, and sequentially shift them; and the shift registers 25 to 2.
1, and performs sigmoid function processing on the outputs of the element chips 15 to 11 sent by the shift registers 25 to 21;
and a broadcast chip 40 that selects the output and input data of the sigmoid function generator 30 and supplies it to the plurality of element chips 15 to 11 in parallel. Equipment is provided.

[For production]

According to the neural network computing device of the present invention, the plurality of element chips 15 to 11 function as an intermediate layer and an output layer, and are provided in parallel. shift register 25
21 are respectively provided for each element chip 15-11, and are designed to receive and hold the output of each element chip 15-11, and to sequentially shift the output. Further, the sigmoid function generator 30 includes shift registers 25 to
It is provided at the end of the shift register 21 and applies a sigmoid function to the outputs of the element chips 15 to 11 that are sequentially sent from the shift register 21 at the end. Further, the broadcast chip 40 includes a sigmoid function generator 30
The output and data from the input layer are selected and supplied to the plurality of element chips 15 to 11 in parallel. This neural network computing device not only functions as a multilayer network composed of an input layer, at least one intermediate layer, and an output layer, but also functions as a feedback network.

According to the neural network computing device of the present invention, an integrated circuit chip P having a function of multiply-accumulate operation and a local memory is provided.
E, (units 5 to 11) are arranged in one dimension, and input data Vj is serially supplied to each unit 15 to 11 through a broadcast chip 40. Each unit 15 to 11 calculates the sum of the products of the input data Vj sent from the broadcast chip 40 and the weight Wij corresponding to each input data 2 stored in the local memory.

When the sum of all inputs ΣW i j V j is taken,
A threshold value is added to it and the result is shifted to registers 25 to 21.
Transfer to. The data sent to the shift registers 25 to 21 are sequentially supplied to a sigmoid function generator 30, and serially calculated one data at a time. The data output from the sigmoid function generator 30 is sent again to the broadcast chip 40. As described above, the neural network type calculation device of the present invention is configured as a hardware simulator that performs neural network calculations in parallel, so calculation speed can be increased. Since it is not necessary to provide it on the chip, the amount of hardware in the element chip can be reduced.

〔Example〕

Embodiments of the neural network computing device according to the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing the overall configuration of a neural network computing device according to the present invention.

First, a case of a multi-layered network will be described, but for simplicity, there is only one intermediate layer. As shown in Figure 6, the number of units in each layer is 6 for the input layer and 5 for the middle layer.
There are four output layers. Here, as for the number of element chips (units) in the neural network computing device of this embodiment, one chip is allocated to one unit, so the number of chips is equal to the number of units in the intermediate layer and the output layer. It is only necessary to prepare the number of units (5 units) for the larger layer. In addition, the input layer only outputs data to the intermediate layer unit and does not require a sum-of-products operation, so it does not need to be particularly considered.

FIG. 2 is a diagram for explaining the operation of an embodiment of the neural network computing device according to the present invention, and FIG. FIG. 3B shows the processing operation from the intermediate layer to the output layer in a multilayer network.

As shown in FIG. 2(a), the units of the intermediate layer (intermediate layer-1, intermediate layer-5) include a sigmoid function orderer 30
It is assigned to element chips 11 to 15 that are closest to .

Further, as shown in FIG. 2(b), the output layer units (output layer -1 to specific output -4) are assigned to element chips 11 to 14 near the sigmoid function generator 30. At this time, units 11 to 15 provided in parallel
When using as an output layer, unit 15 will not be used. Also, broadcast chip 4
0 includes a selector 41 and a register 42.

FIG. 3 is a diagram showing the configuration of each element chip in the neural network type computing device of FIG. 2. As shown in the figure,
Each element chip 11 to 15 has a local memory 101. Multiplexer 1051 Multiplier 102. It includes an adder 103 and a control circuit 104.

The local memory 101 stores weight values and threshold values corresponding to data manually input to each element chip 11 to 15. The multiplexer 105 selects either the data input to each element chip or the predetermined level 1° and supplies it to the multiplier 102. Multiplier 1
02 multiplies the output of the multiplexer 105 and the output of the local memo January 01, and the adder 103 sequentially adds the outputs of the multiplier 102. The control circuit 104 controls each element chip 11 to 1.
This is for controlling various circuits provided in 5 to perform predetermined calculations.

The operation of this embodiment will be explained with reference to FIGS. 2 and 3. Here, the element chips 11 to 11 corresponding to each unit
It is assumed that necessary weight data is loaded in advance into the local memo January 01 of No. 15.

First, as shown in FIG. 2(a), the intermediate layer calculation switches the selector 41 of the broadcast chip 40 to the host side. As a result, the output V of the first unit of the input layer is loaded from the host side to the register 42 of the broadcast chip 40, and the first input V is loaded from the broadcast chip 40 to all element chips 11 to 15. The output of the layer unit, V, is broadcast. In each element chip 11 to 15, the output v1 of the first input layer unit and the weight Wi between the intermediate layer unit and the first input layer unit that each element is responsible for.

The multiplier 102 calculates the product Wi,V.

The broadcast chip 40 is then loaded with the output v2 of the second input layer unit from the host and broadcasts it to all element chips 11-15. Each element chip 11 to 15 uses a multiplier to multiply the product wi, vz of the output v2 of the second input layer unit and the weight Wit between the intermediate layer unit and the second input layer unit that each element is in charge of. 102, which W is next?
ilV, is added by an adder 103.

Similarly, when the adder 103 adds up to the output of the sixth unit of the input layer, the local memory 10
The threshold values (θ, to θ,) are read from 1 and added together by the adder 103. The calculated sum (ΣWIj
V, +θ1 to ΣWsjVj+θ,) are sent to shift registers 21-25. Here, addition of the threshold value (external input) is performed by reading θ from the local memo January 01, multiplying it by a predetermined level °1° supplied via the multiplexer 105, and performing the calculation in the multiplier 102. The results are added together in an adder 103.

Next, in the calculation of the output layer, as shown in FIG. The sum of the inputs ΣWIJVj+θ to the first intermediate layer unit is sent, subjected to a sigmoid function f, and the output vl is sent to the broadcast chip 40.

Thereafter, similarly to the intermediate layer calculation described above, the sigmoid function generator 30 applies the sigmoid function f to the data sent from the end 21 of the shift registers 21 to 25, and transfers the data to the broadcast chip 40.

The broadcast chip 40 switches the selector 41 and broadcasts the data sent from the sigmoid function generator 30 to all the element chips 11 to 15. Element chips 11 to 15 take the weighted sum of the supplied data (data corresponding to the output of the intermediate layer), and finally add the threshold values to shift registers 21 to 25.
send to The calculated sum is calculated by the sigmoid function generator 30
It is then subjected to a sigmoid function and sent to the host side via the interface unit as a network output.

As described above, according to the multi-layered neural network computing device which is an embodiment of the present invention, significant speed-up of computation can be achieved by implementing hardware and parallelization of the simulator. Further, since only one external part is required for calculating the sigmoid function, the amount of hardware for each element chip can be reduced. Furthermore, since only a portion of the element chips forming the intermediate layer and the output layer are required, the amount of hardware can be reduced from this point of view as well.

Next, the case of a feedback structure type network (Hopfield model) will be explained.

FIG. 4 is a diagram for explaining the operation of another embodiment of the neural network type computing device according to the present invention, and FIG. 4 (a) and (b) show the processing operation in the feedback structure type network. It is something.

As shown in FIG. 4(a), the unit (unit 1
~units 5) are assigned to element chips 11 to 15 near the sigmoid function generator 30. The broadcast chip 40 also includes a selector 41 and a register 42.

FIG. 5 is a diagram showing the configuration of each element chip in the neural network type computing device of FIG. 4. As shown in the figure,
Each element chip 11~]5 is local memo January 01. Time step size holding register 1o6. first multiplexer 1
07. Second multiplexer 105. Second multiplexer 1051 multiplier 102 . Previous internal state holding register 108. Third multiplexer 109. It is equipped with an adder 103 and a control circuit 04.

The local memory 101 stores weight values and threshold values corresponding to data input to each element chip 11 to 15, respectively. The time step size holding register 106 is configured to hold the time step width Δt of data input to each element chip.

The multiplexer 107 is connected to the time step size holding register 10
6 and the output of local memory 101 are selected and supplied to multiplier 102.

The multiplexer 105 selects one of the data input to each element chip 11 to 15, a predetermined level "1°," and the fed-back internal state, and supplies it to the multiplier 102. It multiplies the outputs of the multiplexers 107 and 105, and the output is supplied to the adder 103.
The previous internal state ui is held, and its output is supplied to the multiplexer 109. The multiplexer 109 selects either the previous internal state outputted from the previous internal state holding register 108 or the current internal state fed back, and supplies the selected one to the adder 103 . Adder 103 adds the output of multiplexer 109 and the output of multiplier 102. The control circuit 104 is for controlling various circuits provided in each of the element chips 11 to 15 to perform predetermined calculations.

The operation of this embodiment will be described with reference to FIGS. 4 and 5. Here, the element chips 11 to 11 corresponding to each unit
It is assumed that necessary initial state data is loaded in advance into the local memory 101 of No. 15.

First, as shown in Figure 4(a), in the case of a feedback structure type network (Hopfield model), the simulator solves equation (3) that describes the operation of the unit by differentiating it, so in reality it is as follows. It becomes like this.

Here, the number of units is five. Also at this time, it is assumed that necessary data has been loaded into the local memory 101 in advance.

First, in each element chip 11 to 15, the initial value u of the internal state of each unit is stored in the local memory 101.
1 (0) is read and it is transferred to shift register 21~
Sent to 25th. The data sent to the shift registers 21 to 25 are sequentially supplied to a sigmoid function generator 30, subjected to a sigmoid function, and then sent to a broadcast chip 4.
Sent to 0. The broadcast chip 40 broadcasts the data sent from the sigmoid function generator 30 to all the element chips 11 to 15.

Next, as shown in FIG. 4(b), the element chip 11
15 calculates the sum ΣW i j vj by multiplying the output value VJ of the unit sent from the broadcast chip 40 by the weight W i j. Here, these weights W! J is stored in the local memory 101 in each element chip 11-15. And finally, threshold 1. At the same time, the total is multiplied by the time step size ΔL, the previous internal state value ui is added thereto, and the result is sent to the shift register.

The data sent to the shift register is again supplied to the sigmoid function generator 30, where it is subjected to a sigmoid function, and the output value of the unit is calculated. The calculated output value of the unit is sent to all element chips through the broadcast chip 40. Below, similar calculations are performed until the network converges (the output value of each unit no longer changes)
repeated until. Whether the network has converged is determined by monitoring the output of the sigmoid function generator on the host side.

As described above, according to the feedbank structure type neural network computing device as another embodiment of the present invention, significant speed-up of computation can be achieved by implementing hardware and parallelization of the simulator. Further, since only one external part is required for calculating the sigmoid function, the amount of hardware for each element chip can be reduced.

[Effects of the Invention] As described above in detail, according to the neural network computing device of the present invention, high-speed computation can be achieved by implementing hardware and parallelization of the simulator, and
Place the part that calculates the sigmoid function outside the unit.
Since only one chip needs to be provided, the amount of hardware for each element chip can be reduced.

[Brief explanation of drawings]

FIG. 1 is a diagram showing the overall configuration of a neural network computing device according to the present invention; FIG. 2 is a diagram illustrating the operation of an embodiment of the neural network computing device according to the present invention; 3 is a diagram showing the configuration of each element chip in the neural network type computing device of FIG. 2, and FIG. 4 is a diagram for explaining the operation of another embodiment of the neural network type computing device according to the present invention. 5 is a diagram showing the configuration of each element chip in the neural network type computing device shown in FIG. 4, FIG. 6 is a diagram showing a multilayered neural network type computing device, and FIG. FIG. 8 is a diagram showing each unit used in the multi-layered neural network computing device shown in the figure, and FIG. 8 is a diagram showing the feedback structure neural network computing device. (Explanation of symbols) 11 to 15... Element chips (units), 21 to 25
Shift register, 30 Sigmoid function generator, 40 Broadcast chip, 41 Selector, 42 Register, 101 Local memory, 102 Multiplier, 103. ...Adder, 104...Control circuit, 105.107.109...Multiplexer, 106
...Time step width holding register, 108...Previous internal state holding register. Figure 1 shows the overall configuration. Figure 1 is a multi-layered network (input layer - middle layer) (Q). Figure 2 is a diagram explaining the operation of an embodiment of the neural network computing device according to the present invention. Structural network (middle layer - output layer) (b) Feedback structured network (a) Fig. Feedback structured network (b) Configuration of each element chip in the neural network computing device shown in Fig. 4 Diagram n showing each unit used in the multi-layered neural network type computing device shown in Figure 6. Figure 7. Figure 6: External input (threshold value) Figure 8: Diagram showing a neural network computing device with feedback structure

Claims (6)

[Claims] 1. A neural network type computing device that functions as a multilayer network configured with an input layer, at least one intermediate layer, and an output layer, wherein a plurality of neural network computing devices are provided in parallel and function as the intermediate layer and the output layer. element chips (15 to 11), shift registers (25 to 21) provided for each element chip, which receive and hold the output of each element chip, and sequentially shift the output, and an end of the shift register. a sigmoid function generator (30) that is provided in the section and applies a sigmoid function to the outputs of the element chips that are sequentially sent from the terminal shift register; and the output of the sigmoid function generator and data from the input layer. a broadcast chip (40) that selects and supplies the selected element chips in parallel to the plurality of element chips. 2. Each of the element chips (15 to 11) has a local memory (101) that outputs stored weight values and threshold values according to data input to each element chip, and a local memory (101) that outputs stored weight values and threshold values according to data input to each element chip. Predetermined level 'l'
a multiplexer (105) that selects one of the above; a multiplier (102) that multiplies the output of the multiplexer and the output of the local memory; an adder (103) that sequentially adds the outputs of the multiplier; 2. The neural network computing device according to claim 1, further comprising a control circuit (104) for controlling the element chips. 3. The neural network computing device according to claim 1, wherein the broadcast chip (40) has a selector (41) and a register (42). 4. A neural network computing device that functions as a network with a feedback structure, comprising a plurality of element chips (15 to 11) arranged in parallel, each of which is given an initial state in advance, and the output of each element chip. Shift registers (25 to 21) that hold and sequentially shift the output of each element chip.
a sigmoid function generator (30) provided at an end of the shift register and performing sigmoid function processing on the output of the element chip sent by the shift register; A neural network computing device comprising a broadcast chip (40) that selects input data and supplies it to the plurality of element chips in parallel. 5. Each element chip (15 to 11) has a local memory (101) that outputs stored weight values and threshold values according to data input to each element chip, and a local memory (101) for outputting stored weight values and threshold values according to data input to each element chip. Time step size (Δ
t); a first multiplexer (107) for selecting one of the output of the time step size holding register and the output of the local memory;
), the data input to each element chip, a predetermined level 'l'
and a second multiplexer (105) that selects one of the fed-back internal states; a multiplier (102) that multiplies the output of the second multiplexer by the output of the first multiplexer; A previous internal state holding register (108) that holds the previous internal state of the element chip, and a third one that selects either the previous internal state output from the previous internal state holding register or the fed-back internal state of the current time. a multiplexer (109); an adder (103) for adding the output of the third multiplexer and the output of the multiplier; and a control circuit (104) for controlling each element chip.
) The neural network type computing device according to claim 4, comprising: 6. 5. The neural network computing device according to claim 4, wherein said broadcast chip (40) has a selector (41) and a register (42).