JPH045750A - Learning method for data processor - Google Patents

Abstract

PURPOSE:To give efficient learning to the intermediate layer of a neural network by increasing the synapse weight of the intermediate layer and an output layer when neurons more than a fixed rate are ignited. CONSTITUTION:The respective neurons of an intermediate layer 32 correspond to the plural neurons of an input layer 31 and an output layer 33 and the respective neurons of the input layer 31 and the output layer 33 correspond to the plural neurons of the intermediate layer 32 respectively. Accordingly, the adjacent neurons of the input layer 31 and the output layer 33 participate in the judging of the inclination of the ignition of the same neuron of the intermediate layer 32. Besides, the number of the neurons in the respective layers 31, 32 and 33 is provided for the number corresponding to the number of the kinds of characters to be recognized.

Description

DETAILED DESCRIPTION OF THE INVENTION [Field of Industrial Application] The present invention relates to a data processing device based on the concept of neural networks.

[Prior Art] A neural network in this type of data processing device is constructed in a layered manner by providing neuron models (hereinafter referred to as neurons) shown in FIG. 3 in parallel. Neurons in each layer are synaptically connected to all neurons in adjacent layers to input and output data. That is, in neuron 1, externally input data I, ,I
2, I3...In have weights W, , W, , respectively.
W.

. . . are multiplied by Wn, and data 0 is output according to the comparison result between the sum total and the threshold value θ.

Various methods are possible for this comparison, but for example, if the normalization function 1 [f] is adopted, the output data 0 will be 0=1 [ΣWn-In-θ) −・−・−(1)
It is expressed as That is, when ΣWn-In is greater than or equal to the threshold 0, the neuron fires and the output data ○ becomes "1", and when ΣWn-In is less than the threshold θ, the output data O
becomes "0".

A conventional neural network is constructed by arranging such neurons in parallel to form a neural layer and connecting these neural layers in series.

The neural layer is, for example, Rosenblatt (Ros
3 like the Berceptron proposed by
It consists of layers: an input layer, a hidden layer, and an output layer, with neurons in each layer synapsing to all neurons in other adjacent layers.

[Problem to be solved by the invention]

In such a data processing device, the process of optimizing the synaptic weights of each neuron is called "learning," and ensuring its realization and increasing its efficiency are important issues. For example, the backpropagation method, which has been attracting attention in recent years, has problems with escape from the local minimum and convergence time. This tendency is particularly strong in neural networks that include many hidden layers.

The present invention was devised to solve these conventional problems, and an object of the present invention is to provide a learning method that can perform efficient learning on the intermediate layer.

[Means to solve the problem]

The learning method according to the present invention artificially determines the firing pattern of the input layer and the firing pattern of the output layer for a plurality of inputs and outputs, and for each input and output, each neuron of the intermediate layer The weights of synapses in the intermediate layer and the output layer are determined so that when a predetermined proportion or more of the corresponding neurons are firing, they tend to fire. When a large number of intermediate layers are provided, when the learning of one intermediate layer is completed to a predetermined stage, a new intermediate layer is added, the same learning is performed on this, and the intermediate layer is sequentially This increases the Here, the number of neurons in the input/output layer is determined by the data capacity at one time, and the number of data that can be processed is determined by the number of neurons in the entire neural network.

Therefore, in order to process a large amount of data, it is essential to increase the number of intermediate layers.

〔Example〕

The present invention will be explained below with reference to illustrated embodiments.

FIG. 1 shows a character recognition system having a data processing device according to an embodiment of the present invention. This character recognition system includes a video camera 10, a preprocessing device 20, a data processing device 30, a postprocessing device 40, and a display 50. A video camera 10 is provided for inputting characters,
It is connected to the preprocessing device 20. The preprocessing device 20 is, for example, a conventionally known image processing device, and extracts feature data (for example, the number of end points, the number of branch points, etc.) of the input character, and outputs this feature data to the data processing device 30.

The data processing device 30 configures a neural network as described later, recognizes characters based on character feature data input from the preprocessing device 20, and sends data corresponding to the recognition results to the postprocessing device 40. Output. The recognition signal is, for example, a character code, and the post-processing device 40 stores this output data as, for example, word processing data.
At the same time, it is output to the display 5o. display 50
is composed of, for example, a CRT, and displays characters recognized by the data processing device 30 on the west side.

The neural network that constitutes the data processing device 30 is configured as part of computer hardware. This data processing device 30 is schematically represented in FIG. 1, and as can be understood from this figure, an input layer 31,
It has an intermediate layer 32 and an output layer 33, and the intermediate layer 32 is disposed between the input layer 31 and the output layer 33. Each layer 31.
32 and 33 each have a large number of neurons N in this embodiment, and the neurons N of the input layer 31 are connected to the intermediate layer 32.
The neurons N of the intermediate layer 32 are connected to all the neurons N of the output layer 33.

Each neuron N is defined as (
1) Output "1" or rQJ data based on the normalization function of the equation. The neurons N are composed of, for example, operational amplifiers, and the weight Wn by which data input to each neuron N is multiplied is obtained, for example, by a variable resistor connected to the input terminal of the operational amplifier. Further, the threshold function is realized by a switching element or the like. That is, by changing the magnitude of the variable resistance according to the output data of each neuron, the weight Wn is changed, the output data is corrected, and learning is performed.

Figure 2 (a), (b), and (c) show the input layer 31 and the intermediate layer 3.
2 and an output layer 33 schematically. The number of neurons decreases in the order of input layer 31, intermediate layer 32, and output layer 33. Here, for ease of explanation, the number of neurons in the human layer 31 is 64, the number of neurons in the intermediate layer 32 is 49, and the number of neurons in the output layer 33 is 36.

Furthermore, in the input layer 31, neurons in each layer are arranged in eight directions in the horizontal and vertical directions, in the intermediate layer 32, seven neurons are arranged in the horizontal and vertical directions, and in the output layer 33, neurons are arranged in the horizontal and vertical directions. There are 6 arrays. Here, the position of the neuron in the lower left corner of the figure is set as the origin, and the neuron located i-th from the left and j-th from the bottom is N i j.

Each neuron in the input layer 31 fires in response to character feature data obtained via the video camera 10. For example, the number of endpoints is expressed by the combination of firings N11, N18, Ni1s Nz□, and Nl1, N14, N23, N24
Suppose that the number of branching points is expressed by the combination of firings. That is, the firing pattern of neurons in the input layer 31 is artificially determined according to the input characters.

On the other hand, in the output layer 33, for example, there are neurons N□ on a diagonal line connecting the neurons in the lower left corner and the upper right corner.

Characters are represented by (neurons surrounded by dashed lines). That is, the neurons on this diagonal line represent the character code of the character, and the firing pattern of the neurons in the output layer 33 is artificially determined. In this embodiment, there are 64 types of firing patterns of neurons N i i on the diagonal, and therefore, in this embodiment, 64 types of characters can be recognized, for example, alphabets can be recognized.

Now, before character recognition learning is performed, even if character data is input to the data processing device 30, neurons in the output layer 33 do not fire. This neuron becomes capable of firing through learning, and the learning ends when it exhibits a predetermined firing pattern according to the input character data. Note that the input/output cover patterns for learning are representative data of the data to be processed by the neural network, and the data to be actually processed covers a wide range. Learning is then executed until association with this representative input/output data is properly performed. When learning is completed, the input layer 31 and the output layer 33 exhibit firing patterns that are artificially determined as described above in accordance with the input character data. Then, an input layer 31, an intermediate layer 32 and an output layer 3
The firing pattern in each layer of 3 is these layers 31.32
．． It is estimated that it will change smoothly over 33 years.

Therefore, in this embodiment, the weights of the intermediate layer 32 and the output layer 33 are changed during the learning process so that the firing patterns in these layers change smoothly.

First, it is determined whether or not the neuron N i j in the intermediate layer 32 should fire, taking into account the firing of the corresponding plurality of neurons in the input layer 31 and output layer 33 .

That is, if a predetermined proportion or more of neurons among the corresponding neurons in the input 131 and output layers 33 fire or should fire, the intermediate layer 3
It is determined that the neuron No. 2 has a tendency to fire, and the weights of certain neurons in the intermediate layer 32 and the output layer 33 are increased so as to obtain such firing.

A method for determining whether or not neurons in the intermediate layer should fire will be specifically explained with reference to FIGS. 2(a) to 2(c). Note that FIG. 2(a) shows the firing pattern of the input layer 31, FIG. 2(b) shows the firing pattern of the intermediate layer 32, and FIG. 2(C) shows the firing pattern of the output layer 33. indicates a neuron that is firing, and a white circle indicates a neuron that is not firing. In this example, the input layer 31 exhibits a firing pattern as shown in the figure based on the characteristic data of the input character, and in order to recognize this character, the output layer 33 must exhibit a firing pattern as shown in the figure. . In other words, the firing pattern of the illustrated input layer 31 is uniquely determined by the input characters, and the firing pattern of the output layer 33 is the one when learning is completed. On the other hand, the tendency of the firing pattern of the intermediate layer 32 is determined according to the firing patterns of the input layer 31 and the output layer 33, as will be described later.

The correspondence between neurons in each layer will be explained.

In principle, each neuron in the intermediate layer 32 has an input layer 3.
1 and output layer 33 correspond to four neurons each.

That is, the neurons N i j of the intermediate layer 32 have the neurons N ij,N (i +11js
N1(j.11,N,8°011,.1. (in the case of neuron N2. of the intermediate layer, the area surrounded by the symbol E2) corresponds to the neuron N fi-11f of the output layer 33.
j-11, Ni<j-1+, N<=-+, i, N3, (
In the case of the neuron N26 in the middle layer, the area surrounded by the symbol F2) corresponds to the neuron N26. However, each neuron NI1%Nl, N,1, N77 located at the corner in the intermediate layer 32 corresponds to each of the input layer 3104 neurons as described above. Neurons N I I, N 61% N13, Nhb
corresponds. In addition, each neuron N, ..., Ni7, N15, N7J located at the peripheral part of the intermediate layer 32 has the following:
Similarly to the above, the four neurons of the input layer 31 correspond to each other, but regarding the output layer 33, the neuron N□1 of the intermediate layer 32 corresponds to the neuron N.

1.1, N i l, neuron N8 of the hidden layer 32. For neuron N, □-17, N i 6, for neuron NIJ of intermediate layer 32, neuron Nl+j
−11, N1, neuron N of intermediate IW32? Regarding J, N6(j-11 and N6J correspond to each other.

In this embodiment, neurons in the intermediate layer 32 are determined to have a tendency to fire when, for example, 40% or more of the corresponding neurons in the input layer 31 and the output layer 33 are firing. In the example of FIG. 2(b), neurons N! of the hidden layer 32! Neurons in the input layer 31 and output layer 33 corresponding to 6 (total 8 neurons in areas E2 and F2)
out of 3 neurons), 3 neurons are firing.

Therefore, neurons N2. of the intermediate layer 32. is judged to have a tendency to ignite. Also, neuron N7 of the intermediate layer 32. Of the neurons in the input layer 31 and output layer 33 corresponding to (a total of 6 neurons in areas G and H),
Two neurons are firing. Therefore, middle class 3
2's new o 7 N + s has no tendency to fire. Therefore, the neurons in the hidden layer 32 have the second [J(b)
It is judged that there is a tendency to exhibit the firing pattern shown in .

In character recognition learning in this embodiment, the weight of each neuron in the intermediate layer 32 and output layer 33 is increased by a predetermined value so that the firing pattern thus obtained is obtained.

To specifically explain this increase in weight using Figure 3,
If the output data 0 of neuron 1 in the middle layer is the value at the time of firing (for example, "l"), the data input from the firing neuron among the neurons in the input layer connected to neuron l ( For example, the synaptic weights (in this case, "W2" and "Wz") for "I! J and 'L") are increased by, for example, 5%. The synaptic weights of neurons in the output layer are also processed in the same way, and as described above, the synaptic weights for neurons in the middle layer that are determined to be fired are increased by, for example, 5%.

Therefore, the weights of the intermediate layer 32 and the output layer 33 are increased so that the firing pattern changes most smoothly among the input layer 31, intermediate layer 32, and output layer 33. Here, if each neuron in the input layer 31 and output layer 33 is set to fire at the same frequency as possible for all input characters, each neuron in the intermediate layer 32 can also fire equally. This makes it possible to prevent a drop to the local minimum, and each neuron in the intermediate layer 31 fires substantially equally. That is, it is possible to avoid the generation of neurons that do not fire, and the neurons in the intermediate layer 32 can be made to work efficiently.

The increase in synaptic weight during one learning session changes as shown in Figure 4 with respect to the number of learning sessions, and the entire system is gradually learned through multiple learning sessions, and at the end of the learning process, fine adjustments are made due to minute changes. will be held. Also, the rapid increase in initial weights increases the learning speed.

In the above embodiment, the intermediate layer 1132 was connected to the human power layer 31 and the output layer 33, and the number of layers was three. However, the present invention is not limited to a three-layer neural network, but is also applicable to a neural network having four or more neural layers. In this case, first
A first intermediate layer is selected such that each neuron in this intermediate layer has a tendency to fire when a predetermined proportion or more of the corresponding neurons in the input layer and the output layer fire. Increase the synaptic weights of the hidden and output layers. When the learning for all input characters is completed to a predetermined stage, a new second intermediate layer is added, and in the same manner, more than a predetermined percentage of the corresponding neurons fire. The synaptic weights in the second intermediate layer and the layer coupled to the output side of this intermediate layer (output layer or first intermediate layer) are increased in consideration of whether or not the output layer is to be used. Thus, the weight distribution for the case of four layers is obtained. The same applies to the case of five or more layers.

In FIGS. 2(a), (b), and (c), the intermediate layer 320
In principle, one neuron corresponded to four neurons each in the human power layer 31 and the output layer 33. However, the number of corresponding neurons is not limited to this (any number can be selected as needed; for example, one neuron in the intermediate layer 32 has one neuron in the input layer 31 and one in the output layer). Each of the 33 neurons may correspond to 9 neurons.

Here, neurons N26, N, 6, N4 of the intermediate layer 32
6, N, 6, N 611% Focusing on N1k, regions E2 to E correspond to the input layer 31, respectively. In the input layer 31, neurons in each region are
It is also included in the areas adjacent to the left and right, and similarly,
It is also included in the vertically adjacent areas. Therefore, the neurons of the input layer 31, except those at the periphery, are
They belong to four regions, that is, correspond to four neurons in the intermediate layer 32.

On the other hand, looking at the relationship between the intermediate layer 32 and the output layer 33, neurons N81, N6 .
The other neurons N i j correspond to the four neurons of the output layer 33. That is, for example, neurons N21, N36, N46, N, 6, N66, N7 of the intermediate layer 32
6 corresponds to the output layer 330 regions F2 to F, respectively. In the output layer 33, the neurons of each region are overlapped in the left and right adjacent regions, and similarly are overlapped in the vertically adjacent regions. Therefore, the neurons in the output layer 33 belong to four regions, excluding those in the peripheral region, and correspond to the four neurons in the intermediate layer 32.

In this way, each neuron in the intermediate layer 32 is connected to the input layer 31
and corresponds to a plurality of neurons in the output layer 33. Further, each neuron in the input layer 31 and the output layer 33 corresponds to a plurality of neurons in the intermediate layer 32, respectively. That is, adjacent neurons in the input layer 31 and output layer 33 also correspond to the same neuron in the intermediate layer 32. Therefore input layer 31 and output layer 3
The three adjacent neurons will be involved in determining the firing tendency of the same neuron in the intermediate layer 32.

Note that the input/output data to the neuron is not limited to binary data in the case of digital data, but may be multivalued or analog data.

Further, the output layer need not be configured to represent a character code by diagonal neurons, but may be configured to define a recognized character by all neurons.

Further, the number of neurons in each layer 31, 32, and 33 is set according to the number of types of characters to be recognized.

Furthermore, the present invention can be applied not only to character recognition but also to graphic recognition or speech recognition.

〔Effect of the invention〕

As described above, according to the present invention, it is possible to effectively perform learning on the middle layer of a neural network.

[Brief explanation of drawings]

Fig. 1 is a schematic configuration diagram showing a character recognition system using an embodiment of the present invention, Fig. 2 is a conceptual diagram showing neurons in the input layer, intermediate layer, and output layer, and Fig. 3 shows an example of neurons. The conceptual diagram, FIG. 4, is a graph showing the relationship between the number of learning times and the change in weight. IN...Neuron 31...Input layer 32...Middle layer 33...Output layer

Claims (2)

[Claims] (1) an input layer having a plurality of neurons that fire and output predetermined data based on the result of performing predetermined processing on input data; an output layer having fewer neurons than this input layer; data comprising an intermediate layer disposed between an input layer and an output layer, having an intermediate number of neurons in the input layer and the output layer, each neuron being coupled to each neuron in the input layer and the output layer; In a learning method for a processing device, a firing pattern of an input layer and a firing pattern of an output layer are artificially determined for multiple inputs and outputs,
For each input/output, the intermediate layer and A learning method for a data processing device characterized by increasing the weight of synapses in an output layer and performing similar processing on all inputs and outputs. (2) For the data processing device trained by the learning method according to claim 1, a new intermediate layer is added, each neuron of this new intermediate layer is connected to each neuron of an adjacent layer, and a new intermediate layer is created. The new intermediate layer and the adjacent output side are set so that each neuron in the new intermediate layer and the adjacent output side tends to fire when a predetermined proportion or more of the corresponding neurons in each layer on the input side and the output side are firing. Increase the weight of the synapses of the layer, apply similar learning to all inputs and outputs, and when the learning is completed to a predetermined stage, repeat the same learning to add a new hidden layer to increase the number of hidden layers. A learning method for a data processing device characterized by: