JPH03260759A - Signal processing method and device - Google Patents

Abstract

(57) [Summary] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

DETAILED DESCRIPTION OF THE INVENTION The present invention relates to a signal processing method and an apparatus thereof, more specifically,
Regarding neural computers that imitate neural networks,
For example, it is suitable for application to character and figure recognition, motor control of robots, associative memory, etc.

Parallel processing of information is possible by imitating the functions of neurons (neurons), which are the basic units of information processing in organisms, and by configuring these neuron mimicking elements (neuron units) into a network. What we are aiming for is a so-called neural network. There are many things, such as character recognition, associative memory, and motor control, that are easily accomplished in living organisms, but are difficult to achieve with conventional Neumann-type computers. Attempts to solve these problems by imitating the nervous system of living organisms, especially functions unique to living organisms, such as parallel processing and self-learning, are being actively conducted mainly through computer simulations.

FIG. 5 is a diagram for explaining a conventional neural network model. In the figure, A represents one neuron unit, and FIG. 6 shows it configured into a network, where A 1 HA 21A and 21A each represent a neuron unit. One neuron unit is connected to many other neuron units, and processes and outputs signals received from them. In the case of Fig. 6, the network is hierarchical, and neuron unit A2 receives a signal from neuron unit A in the previous layer, and neuron unit A in the next layer (on the right in the figure) .

Output to.

First, to explain the neuron unit A shown in Figure 5, the coupling coefficient (T) represents the degree of coupling between one neuron unit and another neuron unit. Generally, the coupling coefficients between the i-th neuron unit and the j-th neuron unit are expressed as T and J. There are two types of connections: excitatory connections, where the larger the signal from the other neuron, the larger the neuron's own output, and inhibitory connections, where the larger the signal from the other neuron, the smaller the own output. TtJ>O is the excitatory connection, T 1a < O is an inhibitory bond. The input to the jth neuronal unit is i
If the output of the th neuron unit is yl, it is obtained as T14yt, which is multiplied by TIJ.

As mentioned above, one neuron unit is connected to many neuron units, so the sum of TLjyI for those units, that is, ΣTL
JyI becomes the input to one neuronal unit within the network. This is called the internal potential and is expressed in uJ.

u, 1=ΣTIJy1 (1)
Next, this input is subjected to nonlinear processing to become the output of that neuron unit. The nonlinear function used here is called a neuron response function, and a sigmoid function f(x) as shown below is used.

f (x)=1/ (1+e-”)
(2) FIG. 7 is a diagram showing this sigmoid function.

The above neuron units are configured into a network as shown in Fig. 6, and each coupling coefficient is given to form the equations (1) and (2).
By calculating , one after the other, parallel processing of information becomes possible and the final output is obtained.

Figure 8 is a diagram showing an example of the above network realized by an electric circuit (Japanese Patent Application Laid-Open No. 62-295188).
This represents the signal strength of the input and output to the network as a voltage, and the value of the coupling coefficient Ttj between the neuron units is realized as a resistance value. That is, in FIG. 8, the plurality of amplifiers 23 have an inverting output 23a and a non-inverting output 23b, and the input of each amplifier 23 has means 22 for supplying an input current. an interconnection matrix 21 connecting the output of each of the amplifiers to the input with a conductance (TtJ) of a preselected first value or a preselected second value. The TtJ represents the transconductance between the output of the ith amplifier and the input of the We are trying to minimize the energy function that has a local minimum value. In the case of the coupling coefficient TtJ, a negative resistance value cannot be achieved, so this is achieved by inverting the output using the amplifier 23.

Further, an amplifier 23 is used as a function corresponding to the sigmoid function shown in FIG.

Next, the learning function of the network will be explained. The following learning law, called pack propagation, is used in numerical calculations.

First, the coupling coefficient between each neuron unit is set to a random value. If input is given to the network in this state, the output result will not necessarily be desirable.

Therefore, a correct answer (teacher signal) is given to this network, and each coupling coefficient is changed so that when the same input is received again, the output result is the correct answer. For example, in a hierarchical network as shown in Figure 6, the output of the j-th neuron unit in the final layer (layer A3 on the right in the figure) is y'', and the teacher signal for that neuron unit is dJ. Then, E=Σ(dj ya)2 (3
), △TIJ= a E / a TtJ (4) T using hair
4. change. Specifically, first, the error signal δ is determined as follows.

δb= (dJ Y,+)Xf'(+,z)
(5) (Output layer A3) δ, = Σδ1TtjX f' (ut) (Intermediate MA2 of the layer before A3) (6) However, f' is the first-order differential of f. Using this, ΔTtJ=η(δ, y
□)+αΔT.

By setting Tlj=Tlj'+T□, (7), TlJ is changed. However, ΔTLJ″, T
l' is the value at the time of the previous learning. Further, η is called a learning constant, and α is called a stabilization constant, and since they cannot be determined theoretically, they are determined empirically. The algorithm for determining the amount by which the coupling coefficient is changed using this method is called pack propagation.

By repeating this kind of learning, Tt is eventually determined so that a desired result can be obtained for a given input.

Figures 9 to 11 are diagrams showing examples of realizing such a neural network with digital circuits, and Figure 9 is
It is a diagram showing an example of the circuit configuration of a single neuron, where 40 shows a synaptic circuit, 41 shows a dendritic circuit, and 42 shows a cell body circuit.
FIG. 10 is a diagram showing an example of the configuration of the synapse circuit 40 shown in FIG. 9, and FIG. 11 is a diagram showing an example of the configuration of the cell body circuit 42 shown in FIG. , W is a weighting value, and a is a magnification (1 or 2) by which the feedback signal is multiplied. This expresses the input and output of a neuron unit as a pulse train, and the pulse density represents the signal value. The coupling coefficient is handled as a binary number and stored in memory. By inputting the input signal to the rate multiplier clock and inputting the coupling coefficient to the rate value, the pulse density of the input signal is reduced in accordance with the rate value. this is.

This corresponds to the 'r1JyL part of the pack propagation model equation.Next, the Σ part of ΣT□, +yi is realized by an OR circuit shown by the dendrite circuit 41.

Since the connections have excitatory and inhibitory properties, they are divided into groups in advance and OR'd for each group.

In FIG. 9, Fl indicates excitatory output and F2 indicates inhibitory output. These two outputs are input to the counter 4 shown in Figure 11.
By inputting and counting the up and down sides of 2, the output can be obtained. Since this output is a binary number, it is converted into a pulse density using a rate multiplier.

A neural network can be realized by forming a network of these neuron units.

The learning function inputs the final output of the network to an external computer and performs numerical calculations inside the computer.
This is achieved by writing the results into the coupling coefficient memory.

However, many attempts at neural networks have been made using computer simulations, and in order to achieve their original functionality, parallel processing is required and the network needs to be implemented in hardware.

Most attempts at hardware implementation have been made using analog circuits, and these circuits have the following problems.

■Since the signal strength is expressed as an analog value such as voltage or current, and internal calculations are also performed in an analog manner, the value changes due to temperature characteristics, drift immediately after power-on, etc.

■Many elements are required to construct a network, but it is difficult to match the characteristics of each element.

■If the accuracy or stability of one element becomes a problem, new problems may occur when it is used as a network, and the behavior of the entire network cannot be predicted.

■The value of the coupling coefficient is fixed, so it is necessary to use values learned by other methods such as preliminary simulation, and self-learning is not possible.

On the other hand, if an attempt is made to implement a learning method using pack propagation into hardware by any means, it is difficult to implement because the learning requires a large number of arithmetic operations. Also,
The learning method itself is also unsuitable for hardware implementation.

Even in networks using conventional digital circuits, learning is performed by an external computer, so there is no self-learning function at all. Furthermore, the circuit configuration is complicated because the pulse density signal is first converted into a numerical value using a counter and then converted back into pulse density.

To summarize the above, the conventional technology has the following drawbacks.

■Analog circuits do not operate reliably.

■Learning methods based on numerical calculations also require complicated calculations and are not suitable for hardware implementation.

■Digital type circuits have complex circuit configurations.

■Self-learning is not possible on hardware.

- In view of the above-mentioned circumstances, the applicant has first adopted a digital circuit with reliable operation, provided a simple signal processing and learning method that is easy to implement in hardware, and We proposed a signal processing circuit that can actually realize self-learning on hardware. The present invention has been made for the purpose of further improving the calculation accuracy of the network by making the data length (pulse train length) of the coupling coefficient variable in the signal processing circuit.

In order to achieve the above object, the present invention provides (1) a coupling coefficient variable means, and a coupling coefficient generating means for generating a variable coupling coefficient value of the coupling coefficient variable means based on an error signal with respect to a teacher signal. in a signal processing method in which a self-learning means is formed and the self-learning means is coupled to a neuron imitation unit using digital logic, wherein the data length of the coupling coefficient value is made variable, or the above signal processing method (2) A self-learning means using a coupling coefficient variable means and a coupling coefficient generating means that generates a variable coupling coefficient value of the coupling coefficient variable means based on an error signal with respect to a teacher signal. and the self-learning means is coupled to a neuron imitation unit using a digital logic circuit to constitute a signal processing circuit network, further comprising means for making the data length of the coupling coefficient value variable. It is characterized by Hereinafter, the present invention will be explained based on examples.

First, to explain the basic idea of the present invention,
The basic idea of the present invention is as follows: (1) Input/output, intermediate signals, coupling coefficients, teacher signals, etc. are all represented by pulse trains expressed as Oll binary values.

■The amount of signal is expressed by pulse density (within a certain period of time)
1”).

■Calculations within a neuron unit are expressed by logical operations between pulse trains.

■Place the coupling coefficient pulse train in memory.

■Learning is achieved by rewriting this pulse train.

■Calculate the error based on the given teacher signal pulse train,
Based on this, the coupling coefficient pulse train is changed. At this time, all error calculations and calculations of changes in coupling coefficients are
This is done using Oll's pulse train logic and calculations.

■Improve the calculation accuracy of the network by making the data length (pulse train length) representing the coupling coefficient variable.

This will be described in detail below based on examples.

[Signal calculation in forward process] Figure 1 shows
This is a diagram showing a portion corresponding to one neuron unit, and the network configuration uses the same hierarchical structure as the conventional one as shown in FIG. All input and output are binarized to 0.1,
Use a more synchronized one. The intensity of the input signal j is expressed by the pulse density, and is expressed by the number of 1 states within a certain period of time, as in the following pulse train, for example.

Synchronization signal """"-LL This shows a signal representing 4/6, and represents that the input signal has four 1's and two 0's in six synchronizing pulses. At this time, it is desirable that the 1's and 0's are arranged randomly, as will be described later.

On the other hand, the coupling coefficient T1. is similarly expressed in terms of pulse density, and O
and 1 as a pulse train in the summary memory.

Coupling coefficient -=376 (9) Synchronization signal -LLLI-Up" - This represents 3/6, and in this case as well, it is preferable that the arrangement of O and 1 is random. The specific method of determination will be described later.

Then, this pulse train is sequentially read out from the memory in accordance with the synchronization clock, and ANDed with the input pulse train by the AND circuit shown in FIG. 1 (ytnTtj). This is input to neuron unit j. To explain using the above example, when the signal rlo1101J is input, the coupling coefficient pulse train is read from the memory in synchronization with this, and by sequentially ANDing it, the input signal - nee m = 476 coupling coefficient - = 3 /6 (10) A pulse string (bit string) rloloooJ is obtained, which indicates that the input '/l is converted by TLJ and the pulse density becomes 2/6.

The pulse density of the output of the AND circuit is approximately the product of the pulse density of the input signal and the pulse density of the coupling coefficient, and has the same function as the product of signals in the analog system. The longer the signal train (pulse train) is, the more
The more random the arrangement of and O, the closer the function becomes to the product of numbers. Non-random means that 1's (or O's) are clustered (close together). Therefore, by making the data length (pulse sequence arrangement) of the coupling coefficient variable and setting it according to the state of implementation, it is possible to increase the randomness of the arrangement of 1s (or O's).

In particular, when the pulse train of the coupling coefficient is shorter than the input pulse train and there is no more data to read, the pulse train returns to the beginning of the pulse train of the coupling coefficient and repeats the reading. An example of this case is shown below.

Input signal pulse sequence = 12 Coupling coefficient pulse sequence = 68901. .

Coupling coefficient pulse sequence = 8. ．． ．． ．． ．． ．． ．． .

Pulse sequence position of coupling coefficient = 12 In the above example, in order to calculate an input signal with 12 pulse sequence positions, the coupling coefficient with a pulse sequence position of 6 is calculated once, and the coupling coefficient with a pulse sequence position of 8 is calculated 0.5 times more. The whole thing is shifting. As a result, even with the same coupling coefficient of 6/12, the randomness is higher than when the pulse train arrangement is 12, and therefore, the calculation accuracy of the entire network can be improved.

Because one neuron unit has multiple inputs.

As explained earlier, there are many ANDJs between the input signal and the coupling coefficient, and then they are ORed.The inputs are synchronized, so if the first data is 101000J and the second data is rolooooJ. , the OR of both is rl
It becomes llooOJ.

If this is calculated simultaneously with multiple inputs and the output is output, the result will be linear.

A, n To, You-y, nT,, -U--m- U (y+ n T +j) end m-- (11) This part involves the calculation of the sum of signals in the analog system and the nonlinear function ( It corresponds to the part of the sigmoid function).

When the sum of the pulse densities is low, the pulse density obtained by ORing the pulse densities approximately matches the sum of the respective pulse densities. As the pulse density increases, the OR output gradually becomes saturated, so the result does not match the sum of the pulse densities, and nonlinearity appears. In the case of OR, the pulse density never becomes larger than 1 or smaller than 0, and since it is a monotonically increasing function, it is approximately equivalent to a sigmoid function.

Now, coupling has excitatory and inhibitory properties, and in the case of numerical calculations, it is expressed by the sign of the coupling coefficient, and in the case of analog circuits, as mentioned above, when TiJ is negative (inhibitory coupling), The direction of output is reversed using an amplifier, and it is coupled to other neuron units with a resistance value corresponding to TIJ. on the other hand,
In the present invention, each connection is divided into two groups, excitatory connections and inhibitory connections, depending on the sign of TiJ, and then the Group output" is 1 and "
Outputs 1 only when the "output of the inhibitory group" is O.

In order to realize this function, it is sufficient to AND the "N0TJ of the output of the inhibitory group and the output of the excitatory group". In other words, the output of the excitatory group is the output of the inhibitory group.
3) b=U(ytnTtj) (T=inhibitory)
(14) yt=an ding (1
5).

The network of neuron units is of a hierarchical type (FIG. 6) similar to pack propagation.

If the entire network is synchronized, each layer can perform calculations in parallel using the functions described above.

[Signal calculation in learning (pack propagation)] Obtain an error signal using the following method (■) or (2), and then change the value of the coupling coefficient using the method described in (■).

(2) Error signal in the final layer The method for obtaining the error signal in each neuron unit in the final layer will be explained. In the present invention, the error signal is defined as follows. In other words, when an error is expressed numerically, it can generally take both positive and negative values, but since such expression is not possible with pulse density, the error is expressed using two signals: one representing the ten commandments and the other representing the negative component. Express the signal.

That is, among the different portions of the teacher signal pulse and the output signal pulse, the error signal 1 pulse is a pulse that exists on the teacher signal side, and the error signal - pulse is a pulse that exists on the output signal side. In other words, when one error signal pulse is added to the output signal pulse and the error signal pulse is removed, the teacher signal pulse is obtained.

■Error signal in the middle layer The error signal in each neuron unit in the middle layer is determined as follows. That is, error signals in each neuron unit of the next layer (final layer A3 in FIG. 6) are collected and used as error signals. This can be done in the same manner as calculation formulas (8) to (15) within the neuron unit. However, unlike formula 1 (8) to (]5),
While y is one signal, δ has two signals to represent positive and negative, and both signals must be considered. Therefore, it is necessary to divide into four cases: positive and negative of T (coupling coefficient) and positive and negative of δ.

First, the case of excitatory connections will be explained. For a neuron unit with an intermediate layer, the error signal in the next neuron unit of JPJ (the final layer in Figure 6) (δ++nT+, +) is obtained by ANDing the coupling coefficients (δ++nT+, +) for each neuron unit in the next layer, and then ORing these together (LJ
(δ' r 11 T IJ)).

The result is assumed to be the error signal of this layer. That is, it is expressed as follows.

δ”, nT,, .

δ+jn T t J δ・ (2o
) Similarly, by using the error signal in the neuron unit in the next layer, the error signal in this layer can be determined.

δ-, n'r,, δ-r n T t j 5- (21) Next,
The case of inhibitory binding will be explained. AND the error signal from the neuron unit in the next layer and the coupling coefficient between that neuron unit and itself, and then OR these two together.
The result obtained is the error signal of this layer. In other words, δ−□n T t − δ−J n T t J δ+ (22
) Similarly, by using the error signal in the neuron unit in the next layer, the error signal in this layer can be determined.

δ", IIT, 1 δ to nT, j δ- (23) Since the connection from one neuron unit to another neuron unit can be excitatory or inhibitory,
O of δ1 calculated by equation (20) and δ1 calculated by equation (22)
Take R and let it be δ1 of this neuron unit. Similarly, the OR of δ- calculated by equation (21) and δ- calculated by equation (23) is taken to determine δ- of this neuron unit.

To summarize the above, 8"=CUCδ"an'r,))U(U(δ-jn'
r, +)) iE excitatory IC inhibitory δ-=(U(δ-, n'1J)) U(U(δ'', nT
IJ)) iE excitatory iE inhibitory (24).

Furthermore, a function corresponding to the learning rate may be provided,
When the learning rate is 1 or less, the learning ability further increases.

This can be achieved by thinning out the pulse train in the pulse train calculation. In this embodiment, a counter concept is adopted and the following is adopted. For example, learning rate (η)
= (0,5), the pulse train of the original signal is thinned out every other pulse, but even if the pulses of the original signal are not equally spaced, the original pulse train can be thinned out. The following (Example 1) and (Example 2) are diagrams showing examples of thinning,
For both, when η = 0.5, every other pulse is thinned out, when η = 0.33, every second pulse is left, and when η = 0.67, every other pulse is thinned out. It is drawn once every third.

(Example 1) Original signal - When η = 0.5 When η = 0.33 When η = 0.67 (Example 2) When original signal η = 0.5 When η = 0.33 When η=0.67 (25) In this way, by thinning out the error signal, the learning rate function is realized.

■Change each coupling coefficient from the error signal Next, using the error signal obtained by the above ■ or ■,
A method for changing each coupling coefficient will be explained.

AND the signal flowing along the line to which the coupling coefficient you want to change (see Figure 6) and the error signal are taken (δs n y
L). However, in this embodiment, since there are two error signals, 0 and -, each is calculated and determined.

δ+J n yt →ΔT
+,.

(26) δ−jny+ ΔT−L
j (27) The two signals obtained in this way are each ΔT+1
4. Let ΔT-, J be.

A new coupling coefficient TLJ is determined based on these, but since the value of TL in this embodiment is an absolute value, cases are differentiated depending on whether the original TiJ is excitatory or inhibitory.

For the TlJ of Fengjin Yuan, increase the component of ΔT + Lj, ΔT
-1, reduce the component.

Original TLJ ΔT Reduce the component of ΔT+□ with respect to the original Tlj, ΔT
-Increase the rs component.

The entire network is calculated based on the learning side of ΔT-1 or more.

(Circuit example 1 Figures 2 to 4 show circuits constructed based on the above algorithm, but the overall network configuration is shown in Figure 6.
It is similar to the figure. FIG. 2 shows a circuit corresponding to the line in FIG. 6, and FIG. 3 shows a circuit corresponding to the circle (neuron unit A) in FIG. Further, FIG. 4 shows a circuit for calculating the error signal in the final layer from the output of the final layer and the teacher signal. By forming these three diagrams into a network as shown in FIG. 6, a digital neural network capable of self-learning can be realized.

First, FIG. 2 will be explained. 1 is an input signal to the neuron unit and corresponds to equation (8). The coupling coefficient of equation (9) is stored in the shift register 8.

8A is a data output port, and 8B is a data input port. By controlling the control signal 8C (for example, a clear signal) of this shift register from an external control circuit (not shown), the bit length of the data to be shifted can be changed. As long as it has the same function as the shift register, the same operation can be performed by appropriately controlling the address count value even when using other devices such as a RAM+address controller. As a result, the data length of the coupling coefficient (pulse sequence position)
can be made variable. 9 is a circuit corresponding to equation (10), which performs an AND operation between the input signal and the coupling coefficient.

These outputs must be divided into groups depending on whether the connections are excitatory or inhibitory, but it is more versatile to prepare outputs 4.5 for each group in advance and switch which one to output. For this reason, a bit indicating whether the connection is excitatory or inhibitory is stored in the memory 14 and switched by the circuit 13. The formula for processing each human power (11
) is the circuit 16 in FIG. 3. Furthermore, the circuit shown in equation (12) that outputs 1 only when the excitatory group is 1 and the inhibitory group is O is the circuit 1 in FIG.
It is 7.

Next, the error signal will be explained. FIG. 4 shows a circuit for generating an error signal in the final layer. This is expressed by equations (16) to (19)
). An error signal 6.7 is generated from the output 1 from the final layer and the teacher signal 20. In addition, the circuit that realizes equations (2o) to (23) for determining the error signal in the intermediate layer is
This is shown in circuit 10 of FIG. Since the error signal used differs depending on whether the connection is excitatory or inhibitory, the circuit 12 in FIG. 2 makes the distinction between cases. This is set in memory 14 in advance.
Switch depending on the bit set to .

Further, the calculation for collecting the error signal (Equation (24)) is performed in the circuit 18 of FIG. Also, the formula corresponding to the learning rate (
25) is performed by the frequency dividing circuit 19 in FIG.

Next, a description will be given of the part in which a new coupling coefficient is calculated by H1 from the error signal. This is expressed by equations (26) to (29) and is performed by the circuit 11 in FIG.
Since cases must be classified according to suppressiveness, the second
The circuit shown in the figure] 2 realizes this.

Although an example in which the present invention is implemented as a hardware is explained in FIG. It is easy to understand that it is also possible to construct a whole by combining separately constructed components each having a single function.

Study As is clear from the above explanation, according to the present invention, since the data length (pulse sequence position) representing the value of the coupling coefficient is made variable, the calculation accuracy of the network can be improved in accordance with each purpose. I can do it.

[Brief explanation of drawings]

Fig. 1 is a diagram showing one of the neural circuit units, Figs. 2 to 4 are diagrams showing examples of the circuit configuration of each part, and Figs. 5 to 7 are diagrams explaining the operating principle of the neural circuit unit. 8 to 11 are diagrams showing examples of conventional circuit configurations. 1. Input signal, 2.3.6.7...Error signal, 4.
Excitatory signal, 5... Inhibitory signal, 8... Shift register, 20... Teacher signal. Agent: Akira Takano Chika (one idiot), Figure 1--7! 6th Ward 2 ~ 5Q 8th Ward Figure Output No. 0 Figure 1 Ward 0

Claims (1)

[Claims] 1. Forming a self-learning means by a coupling coefficient variable means and a coupling coefficient generating means for generating a variable coupling coefficient value of the coupling coefficient variable means based on an error signal with respect to a teacher signal,
A signal processing method in which the self-learning means is coupled to a neuron imitation unit using digital logic, characterized in that the data length of the coupling coefficient value is made variable. 2. Constructing a self-learning means by a coupling coefficient variable means and a coupling coefficient generating means for generating a variable coupling coefficient value of the coupling coefficient variable means based on an error signal with respect to a teacher signal,
A signal processing device in which the self-learning means is coupled to a neuron imitation unit using a digital logic circuit to form a signal processing circuit network, characterized by comprising means for making the data length of the coupling coefficient value variable. Signal processing device.