JPH04270445A - Threshold value logic circuit with reverse propagation learning function - Google Patents

Abstract

PURPOSE:To obtain the threshold value logic circuit with a highspeed learning function by introducing a new method to execute current learning while referring to the learnt result in the past into the learning method. CONSTITUTION:For the calculation of a differential product 9C3AE/9C3AUi, multipliers 50kj and 60kj calculates the product of differential values 9C3AE/9C3 AUk, weight Wkj and threshold value differential coefficient f(Ui) by the weight adding sum of an error function to be reversely propagated, and a holding part 30j of the differential value 9C3AE/9C3AUi cumulatively adds the value. A weight Wji is updated by a weight update part 70ji. The weight update part 70ji is composed of a differential holding part 71ji, change amount holding part 72ji, multiplier 73ji, subtracter 74ji and dividers 75ji and 76ji.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a threshold logic circuit with a learning function, and particularly to a threshold logic circuit network with a learning function that performs learning by a backpropagation method.

0002

BACKGROUND OF THE INVENTION Conventional computers are constructed with logic circuit systems combining AND or OR circuits. It is a well-known fact that these computers operate at extremely high speeds, exhibit performance that far exceeds human computing ability, and contribute to society. However, it has become increasingly clear that conventional computers are unsuitable for the recognition and judgment operations that humans perform on a daily basis. For this reason, in order to construct a computer suitable for recognition and judgment, we have created a neuron circuit that has a learning function on a threshold logic circuit modeled after human brain cells (neurons), and a computer system using this circuit. For example, the technology is Shunichi Amari's ``Mathematics of Neural Networks'' industrial book,
In 1978, L. D. Jackel, R. E. Ho
ward, H. P. Graf, B. Straugh
n, and J. D. Denker, “Artif
cial neural networks for
“computing”, Journal of V
acum Society Technology
B4(1), Jan/Feb. 1986, pp.
61-63.

The operation of the threshold logic circuit will be explained below to clarify the positioning of the present invention. FIG. 2 is a diagram showing the operation of the threshold logic circuit. The threshold logic circuit is a circuit having a threshold section 10, a weight section 20, a plurality of input terminals 101, an input line 102, and at least one output line 103. In the threshold logic circuit, a signal Xi is applied to a plurality of input terminals, and a weighted sum of the signals Xi ΣWiXi
If exceeds the threshold θ, the output becomes “1”, otherwise “
0''.Here, Wi represents the weight.The feature of the threshold logic circuit is its learning function.In other words, by learning, the weight Wi is changed and the circuit finally adapts to the purpose. Build a system. Therefore, in order to configure a threshold logic circuit, it is necessary to have not only a function to perform weighted addition of input signals and switch elements, but also a learning function to change the weight Wi and threshold value θ. Normally, this weight is controlled by a weight control signal input from the control terminal 104.

[0004] Normally, to realize a complex logical function, the method shown in FIG.
The threshold logic circuits shown in the figure are combined in multiple stages in a complex manner. For learning to update the weights, a learning method is used in which the output signal of the logical function is compared with the output signal (teacher signal) expected at that time, and the weights are changed based on the error. FIG. 3 shows an example of the structure of a logic function in which the threshold logic circuit of FIG. 2 is connected, and is composed of three threshold sections 10 and two weight sections 20. Here, let Uj be the difference between the input weighted sum ΣWijyi of the j-th threshold section and the threshold value θj, let f(Uj) be the threshold function, and let the output signal yj be the difference. The focus of this patent is on a learning method that changes weights. Among several learning methods, backpropagation is recognized as the most effective method. This backpropagation method is described in detail in, for example, Kaoru Nakano, "Basics of Neurocomputer", Corona Publishing (April 1990), and others. In the backpropagation method, (
1) The sum of squares of the error between the output signal of the logic function and the teacher signal for various input patterns, as shown in the formula (all formulas below are displayed together at the end of the Detailed Description of the Invention column) (Error function) Change the weight to minimize E.

[0005] Here, yz is the output signal of the logic function, and ψz is the teacher signal at that time. In the conventional backpropagation method, weights and threshold values are changed using the steepest descent method as expressed by equation (2). Here, ε is a parameter that determines the degree of convergence. In the conventional method, ∂ in equation (2) for a complexly connected circuit network is
E/∂Wji, ∂E/∂θj are, for example, (3) (4)
It can be determined by the method shown in equation (5).

In particular, as can be seen from equation (5), ∂E/∂U
j indicates that calculation can be performed in the reverse direction from the output side to the input side of the neuron circuit system. For this reason, (3)
The method of learning using the method shown in equation (5) from equation (5) is called backpropagation. This conventional learning method using equation (2) performs convergence using the steepest descent method, which has poor learning efficiency, and has the disadvantage that the convergence speed becomes slower as the target convergence point is approached. Furthermore, since the parameter ε that determines the degree of convergence is not a value that can be determined theoretically or analytically, it is extremely ambiguous to determine its value. Moreover, the degree of convergence is determined by the value of ε, and in the worst case, it may not converge and diverge, so conventional methods have many drawbacks.

[0007]

[Object of the invention] The object of the present invention is to improve the convergence of learning by using a weight updating method using the backpropagation method.
By eliminating parameter ambiguity, providing a method with high learning efficiency, and providing a learning method for threshold logic circuits that can quickly change weights, we have achieved high-speed and multifunctional learning using threshold logic circuits. The objective is to realize a computer with excellent recognition and judgment functions.

[0008]

[Summary of the Invention] For this purpose, the present invention uses a threshold logic circuit network having a learning function using the backpropagation method, and uses a method of executing current learning while referring to past learning results as a learning method. Newly introduced. The learning method handled by the present invention results in an optimization problem that minimizes the error function shown in equation (1). Therefore, conventional optimization methods can basically be applied to the problems raised here. The present invention focuses on the differential value of the error function, and employs as a learning method a method similar to the Newton-Raphson method for solving equations, in which the value quickly becomes zero.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be explained below using examples. First, a method of changing weights and thresholds according to the present invention will be explained below. Differential of error function ∂E/∂Wji, ∂E/∂
θj is developed focusing on the weighted sum Uj of the input signals, and is expressed by equations (3) and (4). Therefore, the error of the differentials ∂E/∂Wji and ∂E/∂θj of the error function is calculated by the backpropagation method according to equation (5). On the other hand, the weights and thresholds are updated many times through iteration. Here, the present invention proposes a method of updating the next value in the process of each iteration using the previous calculation result. Learning is ∂E/∂Wij, ∂E/∂
The result is to update Wij and θj so that θj becomes zero. This corresponds to the method of finding the roots of equations. There are various methods for finding the roots of an equation, but for example, the Newton-Raphson method is known as an excellent method with a fast convergence rate. In this method, the value of x is updated according to equation (6) in order to find a solution to the function f(x)=0. Here f' (x) is a derivative.

In the learning method to which the present invention relates, the functional form of the error function is generally not made clear, and therefore the differential function in the denominator of equation (6) cannot be determined. For this reason,
In the present invention, the differential function is replaced by an approximate function. The weight updating method when a linear function is adopted as the approximation function is (7
) and (8). Here, ∂E− /∂W
ij, ΔWij−, ∂E− /∂θj, Δθj−
represents the previous calculation result. These values are stored in the storage elements during the previous iteration. Equation (7), (
8) Equation is an equation that updates Wij, θj so that ∂E/∂Wij, ∂E/∂θj, becomes zero, and the denominator of each equation is ∂E/∂Wij,, ∂E/∂θj, This is a first-order approximation formula for the differential coefficient of . FIG. 1 shows a cell configuration diagram of a neuron circuit that executes the learning method shown by equations (7) and (8). FIG. 1 shows a configuration in which two neuron cells j and k are connected. In the embodiment of FIG. 1, each neuron circuit includes a weight section 20ji, a threshold section 25j, a threshold section 10j, a holding section 30j for the differential value ∂E/∂Uj based on the weighted sum of the error function, and a threshold differential coefficient unit 40j, multiplier 50ji
, a multiplier 60kj, a weight updating section 70ji, and a threshold updating section 80j. Calculation of the differential value ∂E/∂Uj is performed using the differential value ∂E/∂Uk, the weight Wkj, and the threshold differential coefficient f'(Uj ), and the holding unit 30j of the differential value ∂E/∂Uj cumulatively adds the value to the differential value ∂E/∂Uj. The weight Wji is updated in the weight update section 70ji. Weight update unit 70
ji are the differential value holding unit 71ji and the variation holding unit 72ji.
, multiplier 73ji, subtracter 74ji, divider 75ji
, a divider 76ji. The differential value holding section 7
1ji, and the change amount holding unit 72ji stores the differential value ∂E/∂Wji and the weight change amount ΔW calculated in the previous iteration.
This is a memory element that stores ij. This weight updating unit 70j
At i, the subtracter 74ji calculates the difference between the differential value ∂E/∂Wji calculated by the multiplier 73ji according to equation (3) and the previous differential value ∂E/∂Wji− stored in the updated value holding unit 72ji. The difference is divided by the value of the change amount holding section 72ji. By dividing the differential value ∂E/∂Wji by this value, the weight change amount ΔWji according to equation (7) is calculated by the divider 76ji. The weight Wij is updated using this weight change amount ΔWij. The differential value ∂E/∂Wji and the weight change amount ΔWij are stored in the differential value holding unit 71ji at the end of the calculation.
, are stored in the change amount holding unit 71ji. The threshold value updating section 80j has the same configuration as the weight updating section 70ji, and includes a differential value holding section 81j, a change amount holding section 82j, a subtracter 84j,
It is composed of a divider 85j and a divider 86j. The threshold updating unit 82j uses a subtracter 84 to calculate the difference between the differential value ∂E/∂θj calculated according to equation (4) and the previous differential value ∂E/∂θj − stored in the updated value holding unit 82j.
j, and the difference is divided by the value of the change amount holding unit 82j. If we divide the differential value ∂E/∂θj by this value, we get (
8) The amount of threshold change Δθj according to the formula is calculated by the divider 86.
Calculated by j.

Although equations (7) and (8) employ a linear function as an approximation function, it is also possible to use a higher-order polynomial function. In this case, Lagrange interpolation can be used to obtain the polynomial. For example, in a series of iterations, the differential value of the previous time and the time before the previous time is expressed as ∂E− /∂Wj
i, ∂E−−/∂Wji, and the value of Wji at that time is W
ji-, Wji--, it can be approximated by equation (9) as a functional expression of ∂E/∂Wji.

Therefore, the differential coefficient of ∂E/∂Wji is (1
0) is approximated by Eq. By adopting equation (10) as the denominator of equation (6), it can be approximated by a second-order polynomial. As explained above, a threshold logic circuit with a learning function can be defined as one unit, and a high-speed learning method for logic functions is realized by arranging a large number of threshold logic circuits in series and parallel. can. According to the present invention, a threshold logic circuit having a learning function can be realized in hardware, and high-speed and efficient learning can be realized.

In the above description, the threshold logic circuit and its learning method using hardware have been described, but it is clear that this method can be implemented using software.

[0014]

[Effects of the Invention] As explained above, by using the present invention, a threshold logic circuit having a high-speed learning function can be constructed. Therefore, according to the present invention, it is possible to realize a high-speed computer suitable for executing recognition judgment using a threshold logic circuit. Therefore, the present invention is indispensable for realizing a high-speed computer that performs this sophisticated recognition judgment.

[0015]

[Math 1]

[0016]

[Math 2]

[0017]

[Math 3]

[0018]

[Math 4]

[0019]

[Math 5]

[0020]

[Math 6]

[0021]

[Math 7]

[0022]

[Math. 8]

[0023]

[Math. 9]

[0024]

[Math. 10]

[Brief explanation of the drawing]

FIG. 1 shows an embodiment of a threshold logic circuit with a learning function according to the present invention.

FIG. 2 is an explanatory diagram of the operation of a threshold logic circuit.

FIG. 3 is a diagram showing a connection example of a threshold logic circuit.

[Explanation of symbols]

10 Threshold part 20 Weight part 25 Threshold section 30 Error function differential value holding unit 40 Threshold function differential coefficient part 50 Multiplier 60 Multiplier 70 Weight update section 71 Differential value holding section 72 Change amount holding section 73 Multiplier 74 Subtractor 75, 76 Divider 80 Threshold update section 81 Differential value holding section 82 Change amount holding section 84 Subtractor 85, 86 Divider 101 Input terminal 102 Input line 103 Output line 104 Control terminal

Claims (7)

[Claims] 1. A threshold logic circuit that compares a weighted sum of input signals and the weighted sum with a threshold value and outputs an output signal according to a predetermined threshold function, wherein the weight value and a means for updating the threshold, the means connecting a plurality of threshold logic circuits each having a function of creating the latest update content using past update results. Threshold logic network with learning function. 2. A threshold logic network with a back-propagation learning function according to claim 1, wherein the threshold logic network has a function of holding a differential value of an error signal based on the weighted sum. It has a function of outputting the differential value of the threshold function, and outputs the product of the differential value of the weighted sum of the subsequent error signal, the weight value, and the differential value of the threshold function of the self as the error signal of the self. 1. A threshold logic circuit network with a back propagation learning function, characterized in that a plurality of threshold logic circuits each having a function of cumulatively adding a differential value based on a weighted sum of are connected. 3. A threshold logic network with a backpropagation learning function according to claim 2, wherein the threshold logic network is configured to calculate a differential value of an error function with respect to the weight value and a threshold of the error function. It has a function to record past calculation results of differential values regarding values, as well as weight values and threshold values at that time, and can create updated values of the weight values and threshold values using the past calculation results. Features a threshold logic network with backpropagation learning function. 4. The threshold logic circuit network with a backpropagation learning function according to claim 3, wherein the method for updating the weight value and the threshold value is based on the differential value and error of an error function with respect to the weight value. A threshold logic circuit network with a backpropagation learning function is characterized in that the differential value of a function with respect to a threshold value is made zero. 5. A threshold logic circuit network with a back-propagation learning function according to claim 4, wherein the weight value and the threshold value are updated by updating a differential value of an error function with respect to the weight value and an error. Approximately calculate the differential value of the differential value with respect to the threshold value of the function using the differential value of the past error function with respect to the weight value and the differential value of the error function with respect to the threshold value, and calculate the differential value of the error function with respect to the weight value. and a threshold logic circuit network with a backpropagation learning function, wherein the threshold logic circuit network is updated with a value proportional to the quotient of the threshold value of the error function and the approximate value. 6. A threshold logic circuit network with a back-propagation learning function according to claim 5, wherein the approximate calculated value includes a differential value with respect to the weight value of the error function at the time of focus and the previous time; A threshold logic circuit network with a back-propagation learning function, characterized in that it uses a difference between differential values of an error function with respect to a threshold value, a weight value at that time, and a quotient of the difference between the threshold values. 7. The threshold logic circuit network with a back-propagation learning function according to claim 5, wherein the back-propagation learning is characterized in that in the approximate calculation, a plurality of past calculation results are approximated by a polynomial. Functional threshold logic network.