WO2018121472A1 - 一种运算方法 - Google Patents

一种运算方法 Download PDF

Info

Publication number
WO2018121472A1
WO2018121472A1 PCT/CN2017/118264 CN2017118264W WO2018121472A1 WO 2018121472 A1 WO2018121472 A1 WO 2018121472A1 CN 2017118264 W CN2017118264 W CN 2017118264W WO 2018121472 A1 WO2018121472 A1 WO 2018121472A1
Authority
WO
WIPO (PCT)
Prior art keywords
input
weight
layer
neuron
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/118264
Other languages
English (en)
French (fr)
Inventor
陈云霁
宋新开
刘少礼
陈天石
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Cambricon Information Technology Co Ltd
Original Assignee
Shanghai Cambricon Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Cambricon Information Technology Co Ltd filed Critical Shanghai Cambricon Information Technology Co Ltd
Priority to EP17886335.3A priority Critical patent/EP3564866A4/en
Publication of WO2018121472A1 publication Critical patent/WO2018121472A1/zh
Priority to US16/455,347 priority patent/US11308398B2/en
Anticipated expiration legal-status Critical
Priority to US16/542,033 priority patent/US11574195B2/en
Priority to US16/542,051 priority patent/US11568258B2/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
    • G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Definitions

  • the present disclosure relates to the field of data processing techniques, and more particularly to an arithmetic method.
  • Neural Networks are simply referred to as Neural Networks (NNs), which is an algorithmic mathematical model that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This kind of network relies on the complexity of the system, and adjusts the interconnection relationship between a large number of internal nodes to achieve the purpose of processing information.
  • the algorithm used by neural networks is vector multiplication, and symbolic functions and their various approximations are widely used.
  • the neural network consists of a number of interconnected nodes, as shown in Figure 1. Each circle represents a neuron, and each arrow represents the connection between two neurons, and the connection is called Is the weight.
  • x i represents all input neurons connected to the output neurons
  • w i represents the corresponding weight between x i and the output neurons.
  • f(x) is a nonlinear function, usually called an activation function. Common functions are: Wait.
  • Neural networks are widely used in a variety of application scenarios: computational vision, speech recognition, and natural language processing.
  • the scale of neural networks has been growing.
  • Lecun's neural network for handwritten character recognition was less than 1M in weight; in 2012, krizhevsky used to participate in the ImageNet competition with a scale of 60M weights.
  • an arithmetic method for training a multi-layer artificial neural network using an artificial neural network computing device, the artificial neural network computing device comprising: a mapping unit, a storage unit, and an arithmetic unit, the operation
  • the method includes: performing a forward operation sequentially on each layer in the multi-layer artificial neural network, and for each layer of the multi-layer artificial neural network, the forward operation includes: the mapping unit processes the input data to Obtaining the processed input data; the storage unit stores the processed input data; the operation unit reads the processed input data, performs an artificial neural network operation, and obtains an output data of the artificial neural network; Performing an inverse operation on each layer in the multi-layer artificial neural network in sequence in the reverse order of the forward operation; performing weight update on each layer in the multi-layer artificial neural network; and repeatedly performing the above steps Second, complete the training of multi-layer artificial neural network.
  • the artificial neural network computing device further includes: a first input buffer and a second input buffer; and performing weight update on each layer in the multi-layer artificial neural network includes: Each layer of the artificial neural network, the first input buffer and the second input buffer respectively store weights and weight gradients of the layer; the arithmetic unit receives the weight gradient from the second input buffer And then using the weight gradient to update the weight to obtain an updated weight.
  • the input data includes: an input neuron and a weight; and the mapping unit processes the input data to obtain the processed input data, including: the mapping unit processes the weight And obtaining a connection relationship between the processed weight and the weight; the mapping unit processes the input neuron based on the connection relationship data of the weight to obtain the processed input neuron.
  • the input data includes an input neuron and a weight
  • the mapping unit processes the input data to obtain the processed input data, including: the mapping unit processes the input neuron And obtaining a connection relationship between the processed input neuron and the input neuron; the mapping unit processes the weight based on the connection relationship data of the input neuron, and obtains the processed weight.
  • the mapping unit processing the weights includes: the mapping unit deletes the connectionless weights, and retains the connected weights as the processed weights.
  • the mapping unit processing the weights includes: deleting, by the mapping unit, the weights of the weights or the absolute values that are less than or equal to the first threshold, and obtaining the processed weights value.
  • the mapping unit processing the input neurons comprises: deleting, by the mapping unit, the input neurons in the input neurons that have a value or an absolute value less than or equal to a second threshold, and processing After the input neurons.
  • the input data includes processed weights, input neurons, and connection relationship data of the input neurons; the mapping unit processes input data to obtain processed input
  • the data includes: the mapping unit processes the input neuron according to the connection relationship data of the input neuron to obtain the processed input neuron.
  • the input data includes processed input neurons, weights, and connection relationship data of the weights; the mapping unit processes the input data to obtain processed input data. Including: the mapping unit processes the weight according to the connection relationship data of the weight to obtain the processed weight.
  • the mapping unit processing the input data to obtain the processed input data includes: the mapping unit processing the at least one input neuron to obtain connection relationship data of the input neuron; Processing the at least one weight to obtain the connection relationship data of the weight; the mapping unit obtains the third connection relationship data based on the connection relationship data of the input neuron and the connection relationship data of the weight, and according to the third connection relationship data
  • the input neurons and weights are processed to obtain the processed input neurons and the processed weights.
  • connection relationship data of the input neuron and the connection relationship data of the weight are expressed in the form of a direct index or a step index.
  • connection relationship data of the input neuron when the connection relationship data of the input neuron is expressed in the form of a direct index, the connection relationship data is a character string composed of 0 and 1, and 0 represents the input neuron. The value is zero, 1 indicates that the value of the input neuron is non-zero; when the connection relationship data of the input neuron is expressed in the form of a step index, the connection relationship data is an input neuron with a value other than zero A string of distance values between input neurons that are non-zero.
  • connection relationship data of the weight when the connection relationship data of the weight is expressed in the form of a direct index, the connection relationship data is a character string composed of 0 and 1, and 0 indicates that the weight is no connection.
  • the weight value, 1 indicates that the weight is a connected weight;
  • the connection relationship data when the weight relationship data is expressed in the form of a step index, the connection relationship data is an input nerve connected to the output neuron A string consisting of the distance between the element and the input neuron connected to the output neuron.
  • the inverse operation includes: a weight gradient calculation step and an input neuron gradient calculation step;
  • the weight gradient calculation step includes: The operation unit obtains a weight gradient of the layer from the output neuron gradient of each layer of the artificial neural network and the input neuron;
  • the input neuron gradient calculation step includes: the mapping unit outputs the neuron gradient and the processed The weight is stored in the storage unit; the operation unit reads the output neuron gradient and the processed weight, and performs an artificial neural network operation to obtain an input neuron gradient.
  • the output data of the layer on the artificial neural network is used as the input data of the next layer; or, the operation data of the layer on the artificial neural network is layered, and the operation result is taken as the next Input data of the layer; wherein, for any two adjacent layers of the artificial neural network: the first layer and the second layer, in the forward operation, the upper layer refers to the first layer, and the next layer refers to The second layer; in the reverse operation, the upper layer refers to the second layer, and the next layer refers to the first layer.
  • the operation method of the present disclosure has the following beneficial effects: storing input neurons and weights, output neurons and input neurons, output neurons and weights in a one-to-one correspondence format, performing forward and reverse operations It can directly operate on the one-to-one correspondence, without finding the corresponding input and output neurons and weights through the index array, reducing the number of accesses to the memory, simplifying the operation, and improving the operation speed and performance.
  • the connection relation data of the sparse artificial neural network can be updated, and the precision of the sparse artificial neural network operation is improved.
  • the dedicated instruction and arithmetic unit for the sparse multi-layer artificial neural network operation are used to solve the problem of insufficient performance of the CPU and the GPU. The problem of large code overhead effectively improves the support for multi-layer artificial neural network algorithms.
  • Figure 1 is a schematic diagram of an artificial neural network
  • FIG. 2 is a schematic structural diagram of an artificial neural network computing device according to an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of a mapping relationship of the embodiment shown in FIG. 2;
  • FIG. 4 is a schematic diagram of an artificial neural network according to an embodiment of the present disclosure.
  • FIG. 5 is a schematic diagram showing the connection relationship of the artificial neural network computing device shown in FIG. 4;
  • FIG. 6 is a schematic structural diagram of an artificial neural network computing device according to another embodiment of the present disclosure.
  • FIG. 7 is a schematic diagram of a mapping relationship of the embodiment shown in FIG. 6;
  • FIG. 8 is a schematic structural diagram of an artificial neural network computing device according to still another embodiment of the present disclosure.
  • FIG. 9 is a schematic diagram of a mapping relationship of the embodiment shown in FIG. 8.
  • FIG. 10 is a schematic structural diagram of an artificial neural network computing device according to still another embodiment of the present disclosure.
  • FIG. 11 is a schematic diagram of a mapping relationship of the embodiment shown in FIG. 10;
  • FIG. 12 is a flow chart of a forward operation of an operation method according to an embodiment of the present disclosure.
  • FIG. 13 is a flowchart of weight update of an operation method according to an embodiment of the present disclosure.
  • 15 is a flow chart of an arithmetic method of an embodiment of the present disclosure.
  • 16 is a flow chart of an operation method according to still another embodiment of the present disclosure.
  • 17 is a diagram showing an operation process of two adjacent layers of an operation method according to still another embodiment of the present disclosure.
  • 1-mapping unit 2-memory unit; 3-DMA; 4-instruction buffer; 5-control unit; 6-first input buffer; 7-synchronization unit; 8-second input buffer;
  • An embodiment of the present disclosure provides an artificial neural network computing device.
  • the method includes: a storage unit 2, a mapping unit 1, an operation unit 7, a DMA3, an instruction cache 4, a first input buffer 6, and a second input buffer 8. , output buffer 9 and control unit 5.
  • the mapping unit 1 is configured to: after receiving the input data, process the input data to obtain processed input data, where the input data includes at least one input neuron and at least one weight, the processed input The data includes the processed input neurons and the processed weights.
  • mapping unit 1 uses each processed input neuron in the processed input neuron and its corresponding processed weight as an input neuron-weight pair, and all input neurons-weights
  • the value pairs constitute a data set, which is stored in the above storage unit 2.
  • the mapping unit 1 deletes the weight of the connectionless connection, retains the weight of the connection as the processed weight, and the mapping unit 1 also obtains the connection relationship data of the weight, and inputs the data according to the connection value of the weight.
  • the neurons are processed to obtain the processed input neurons.
  • the mapping unit 1 determines the connection relationship data of the weight according to the weight, and The connectionless weight is deleted, and the weight of the connection is retained as the processed weight.
  • the mapping unit 1 deletes the weight value whose value or absolute value is less than or equal to the first threshold value, and obtains the processed weight value, and the mapping unit 1 further obtains the connection relationship data of the weight, and the connection according to the weight value
  • the relational data processes the input neurons to obtain the processed input neurons.
  • the mapping unit 1 determines the connection relationship data of the weight according to the weight, and The weight of the weight or weight is less than or equal to the first threshold, and the processed weight is obtained.
  • the mapping unit 1 may further delete the input neurons whose values or absolute values are less than or equal to the second threshold to obtain the input neurons after the de-zeroing, as the processed input neurons, and the mapping unit 1 also obtains the input neurons.
  • the relation data is connected, and the weight is processed according to the connection relationship data of the input neuron, and the processed weight is obtained.
  • the first threshold may be 0.1, 0.08, 0.05, 0.02, 0.01, 0 or other values.
  • the second threshold may be 0.1, 0.08, 0.06, 0.05, 0.02, 0.01, 0 or other values.
  • first threshold and the second threshold may be the same or may not be consistent.
  • connection relationship data may be expressed in the form of a step index or a direct index.
  • connection relationship data represented by the direct index form is a character string composed of 0 and 1.
  • 0 indicates that the weight value or the absolute value of the weight value is less than or equal to the first threshold value, that is, There is no connection between the input neuron corresponding to the weight and the output neuron
  • 1 indicates that the weight or the absolute value of the weight is greater than the first threshold, that is, between the input neuron and the output neuron corresponding to the weight.
  • connection relation data expressed in direct index form has two representation order: a connection string of 0 and 1 is formed by a connection state of each output neuron and all input neurons to represent a connection relationship of weights; or each input nerve The connection state of the element with all the output neurons constitutes a string of 0 and 1 to represent the connection relationship of the weights.
  • connection relationship data of the input neuron 0 indicates that the absolute value of the input neuron or the input neuron is less than or equal to the second threshold, and 1 indicates that the absolute value of the input neuron or the input neuron is greater than the second threshold.
  • connection relation data represented by the step index index is composed of a distance value between the input neuron connected to the output neuron and the input neuron connected to the previous output neuron.
  • String the connection relation data represented by the step index is represented by a string consisting of the distance value between the current non-zero input neuron and the previous non-zero input neuron.
  • the first threshold is 0.01
  • the input neurons are i1, i2, i3, and i4, and for the output neuron o1, the weights are w 11 , w 21 , w 31 , and w 41 ; for the output neuron o2,
  • the weights are w 12 , w 22 , w 32 and w 42 , wherein the weights w 21 , w 12 and w 42 have a value of 0, and the absolute values thereof are all smaller than the first threshold 0.01, and the mapping unit 1 determines the input neurons.
  • I2 is not connected to the output neuron o1, the input neurons i1 and i4 are not connected to the output neuron o2, and the input neurons i1, i3 and i4 are connected to the output neuron o1, and the input neurons i2 and i3 are
  • the output neuron o2 has a connection.
  • connection relationship data is represented by the connection state of each output neuron and all the input neurons, and the connection relationship data of the output neuron o1 is "1011", and the connection relationship data of the output neuron o2 is "0110” (ie, the above The connection relationship data is "10110110”); with the connection relationship between each input neuron and all output neurons, the connection relationship data of the input neuron i1 is "10", and the connection relationship data of the input neuron i2 is "01".
  • the connection relationship data of the input neuron i3 is "11", and the connection relationship data of the input neuron i4 is "10” (that is, the above-described connection relationship data is "10011110").
  • the connection relationship data can be represented by a high-dimensional dynamic array, which can be represented by a linked list or the like.
  • the input neuron-weight pair is not a true data storage structure, but merely represents the correspondence between the input neurons and the weights.
  • the input neurons are stored in vector A
  • the weights are stored in vector B
  • the lengths of vectors A and B are the same
  • the components of the same position of vectors A and B are combined to be considered an input neuron-weight pair. .
  • the mapping unit 101 processes at least one input neuron to obtain connection relationship data of the input neuron, processes at least one weight, obtains connection relation data of the weight, and is based on the input neuron.
  • the connection relationship data of the connection relationship data and the weight obtains the third connection relationship data, and the input neurons and the weights are processed according to the third connection relationship data, and the processed input neurons and the processed weights are obtained.
  • connection relationship data of the input neuron is a character string for indicating the position of the input neuron whose value in the at least one input neuron is non-zero; the connection relation data of the weight is used to represent the input neuron and the output neuron Whether there is a string of connections between.
  • connection relationship data of the input neurons and the connection relationship data of the weights may be represented by a step index or a direct index.
  • step index or a direct index.
  • the mapping unit 1 performs the connection relationship data of the input neurons and the connection relationship data of the weights. operating.
  • the mapping unit 1 When the connection relationship data of the input neurons and the connection relationship data of the weights are expressed in the form of a step index, the mapping unit 1 inputs each element in the connection relation data string of the input neurons from left to right. The elements that are located before it are accumulated, and the obtained new elements constitute the connection relationship data of the input neurons; for the same reason, the mapping unit 1 performs the same processing on the character string of the connection relationship data of the above weights. Then, the mapping unit 1 selects the same element from the string of the connection relationship data of the input neuron and the connection relation data of the weight, according to the order of the connection relationship data of the input neurons or the connection relationship of the weights. The order of the data, composing a new string. The mapping unit 1 above subtracts each element from the previous element adjacent to the new character string to obtain a new element. According to the method, each element in the new string is subjected to a corresponding operation to obtain the third connection relationship data.
  • connection relationship data of the input neurons and the connection relationship data of the weights are expressed in the form of a step index, and the character string of the connection relationship data of the weights is “01111”, and the connection relationship data of the input neurons described above is The string is "022”, and each element in the string of the connection relation data of the above weight is added to the adjacent previous element to obtain the fourth connection relationship data "01234"; similarly, the above input
  • the fifth connection relation data obtained by performing the same processing on the character string of the connection data of the neuron is "024".
  • the same element is selected from the fourth connection relation data "01234" and the fifth connection relationship data "024" to obtain a new character string "024".
  • Each element in the new string is subtracted from its neighboring previous element, that is, 0, (2-0), (4-2) to obtain the above-described third connection data "022".
  • the mapping unit 1 uses each processed input neuron in the processed input neuron and its corresponding processed weight as an input neuron-weight pair, and all input neuron-weight pairs constitute A data set that stores the data set in the storage unit 2 described above.
  • the mapping unit processes the input neurons and weights, rejects the input neurons and weights with a value of 0, reduces the number of input neurons and weights, and reduces With additional overhead, the arithmetic unit performs artificial neural network operations based on the processed input neurons and weights, improving the efficiency of the operation.
  • the mapping unit 1 reads and processes the input neurons and weights stored in the storage unit 2, and acquires the connection relationship data of the artificial neural network.
  • connection relationship data can be expressed in the form of the direct index described above.
  • o1 is connected to i1, i3, and i4, and o1 and i2 are not connected, and the connection state of the output neuron o1 and the input neurons i1, i2, i3, and i4 is: 1011 .
  • connection state of the output neuron o2 and the input neurons i1, i2, i3, i4 is: 0110, and the output neurons o1, o2 and The connection states of all the input neurons are sequentially arranged, and the connection relationship data of the artificial neural network stored in the first direct index form is 10110110.
  • connection relationship data can also be stored in the form of a second direct index.
  • i1 is connected to o1, i1 and o2 are not connected, i2 is not connected to o1, i2 is connected to o2, i3 is connected to o1 and o2, and i4 is connected to o1.
  • connection state of the input neuron i1 and the output neurons o1, o2, o3, and o4 is: 10, similarly, the connection state of the input neuron i2 and the output neurons o1, o2, o3, and o4
  • the input state of the input neuron i3 and the output neurons o1, o2, o3, and o4 is 11, and the connection state of the input neuron i4 and the output neurons o1, o2, o3, and o4 is 10, and the input neurons are input.
  • the connection states of i1, i2, i3, and i4 and all the output neurons are sequentially arranged, and the connection relationship data of the artificial neural network stored in the second direct index form is 10011110.
  • connection relationship data can also be stored in the form of a step index.
  • the output neuron o1 is connected to the input neurons i1, i3, and i4, and the first character of the connection state of the output neuron o1 is 0, that is, the first connection with o1.
  • the input neuron i1, the first input neuron of the artificial neural network is also the distance of i1, the second input neuron i3 connected to o1 and the first input neuron connected to o1
  • the distance of i1 is 2, and the distance between the third input neuron i4 connected to o1 and the second input neuron i3 connected to o1 is 1, and therefore, the connection state of the output neuron o1 is 021.
  • the output neuron o2 is connected to the input neurons i2 and i3.
  • connection state of the output neuron o2 is 11, and the connection states of the output neurons o1 and o2 and all the input neurons are sequentially arranged, and the step index form is arranged.
  • the connection relationship data of the stored artificial neural network is 02111.
  • the mapping unit 1 stores the processed weight and the data set composed of the input neurons to the storage unit 2 based on the connection relationship data, and the processed weight refers to the weight of the connection, or is the index value or the absolute value is greater than The weight of the first threshold.
  • the connection relationship data of the artificial neural network stored in the first direct index form is 10110110
  • the input neurons corresponding to the processed weights w11, w31, w41, w22, and w32 are i1, i3, i4, i2, and i3, respectively.
  • the input neurons and the processed weights are combined into a data set: i1, w11, i3, w31, i4, w41, i2, w22, i3, w32, and stored in the storage unit 2.
  • connection relationship data of the artificial neural network stored in the second direct index form is 10011110, and the input neurons and the processed weights are combined into a data set: i1, w11, i2, w22, i3, w31, i3, w32, i4, w41 And stored to the storage unit 2.
  • the connection relationship data of the artificial neural network stored in the step index form is 02111, and the input neurons and the processed weights constitute a data set: i1, w11, i3, w31, i4, w41, i2, w22, i3, w32, And stored to the storage unit 2.
  • the mapping unit 1 reads and processes the input neurons, weights, and connection relationship data of the storage unit 2, and the mapping unit 1 combines the processed weights with the input neurons.
  • the data set format is stored to the first input buffer 6 and the second input buffer 8, respectively.
  • the control unit 5 controls the instruction cache 4 to read the instructions in the storage unit 2, and controls the arithmetic unit 7 to operate based on the instructions.
  • the arithmetic unit 7 stores the input neurons and weights of the data set stored in the storage unit 2 into the first input buffer 6 and the second input buffer 8 respectively under the control of the control unit 5, and the arithmetic unit 7 respectively receives the first input
  • the input neurons and weights are read in the cache 6 and the second input buffer 8, and the artificial neural network forward operation is performed to obtain the output neurons and stored in the output buffer 9, and the output neurons in the output buffer 9 are stored in the DMA3.
  • the operation unit 7 multiplies the input neuron and the weight data one by one to obtain a weighted output neuron, and adds the weighted output neurons step by step through the addition tree to obtain a summed output neuron, or passes the weighted output neuron through
  • the bias addition adds a biased output neuron, and the summed output neuron or the biased output neuron is subjected to an activation function operation to obtain an output neuron of the artificial neural network, and the forward operation of the artificial neural network is completed.
  • the activation function can be a sigmoid function, a tanh function, or a ReLU function.
  • the operation unit 7, the instruction cache 4, the first input buffer 6, the second input buffer 8, the output buffer 9, and the control unit 5 are implemented by a dedicated processor, the first input buffer 6, the second input buffer 8, The output buffer 9 is an on-chip cache.
  • the mapping unit 1 stores its corresponding connected output neuron gradient and input neurons to the storage unit 2 based on the processed weights.
  • the arithmetic unit 7 stores the output neuron gradient and the input neurons stored in the storage unit 2 in the first input buffer 6 and the second input buffer 8 respectively under the control of the control unit 5, and the arithmetic unit 7 respectively caches from the first input. 6 and the second input buffer 8 reads the output neuron gradient and the input neurons, multiplies the output neuron gradient by the input neurons one by one, and multiplies the multiplication result by the learning rate to obtain the processed weights.
  • the updated amount; the processed weight is subtracted from the corresponding update amount, the updated processed weight is obtained and stored in the output buffer 9, and the updated processed weight in the output buffer 9 is stored in the storage unit via the DMA3.
  • connection relationship data of the artificial neural network stored in the first direct index form is 10110110
  • the input neurons corresponding to the processed weights w11, w31, w41, w22, and w32 are i1, i3, i4, i2, and i3, respectively.
  • the output neuron gradient corresponding to the processed weight is combined with the input neurons to form a data set: To1, i1, To1, i3, To1, i4, To2, i2, To2, i3, and stored in the storage unit 2, To1 and To2 represents the gradient of the output neurons o1 and o2, respectively.
  • To1 and i1, To1 and i3, To1 and i4, To2 and i2, To2 and i3 are multiplied by each other, and the multiplied result is multiplied by the learning rate to obtain the updated amount of processed weights ⁇ w11, ⁇ w31, ⁇ w41, ⁇ w22, ⁇ w32.
  • the processed weights w11, w31, w41, w22, and w32 are respectively subtracted from the update amount, and the updated processed weights w11', w31', w41', w22', and w32' are obtained.
  • connection relationship data of the artificial neural network stored in the second direct index form is 10011110
  • the input weights corresponding to the processed w11, w22, w31, w32, w41 are i1, i2, i3, i3, respectively.
  • I4 the output neuron gradient and the input neurons are combined into a data set: To1, i1, To2, i2, To1, i3, To2, i3, To1, i4, and stored in the storage unit 2.
  • To1 and i1, To2 and i2, To1 and i3, To2 and i3, To1 and i4 are multiplied by each other, and the multiplication result is multiplied by the learning rate to obtain the updated amount of processed weights ⁇ w11, ⁇ w22, ⁇ w31, ⁇ w32, ⁇ w41
  • the processed weights w11, w22, w31, w32, w41 are respectively subtracted from the update amount, and the updated processed weights w11', w22', w31', w32', w41' are obtained.
  • connection relationship data of the artificial neural network stored in the step index form is 02111, and the input neurons corresponding to the processed weights w11, w31, w41, w22, w32 are i1, i3, i4, i2, i3, respectively, and the output will be output.
  • the neuron gradient and the input neurons constitute a data set: To1, i1, To1, i3, To1, i4, To2, i2, To2, i3, and are stored to the storage unit 2.
  • To1 and i1, To1 and i3, To1 and i4, To2 and i2, To2 and i3 are multiplied by each other, and the multiplied result is multiplied by the learning rate to obtain the updated amount of processed weights ⁇ w11, ⁇ w31, ⁇ w41, ⁇ w22, ⁇ w32.
  • the processed weights w11, w31, w41, w22, and w32 are respectively subtracted from the update amount, and the updated processed weights w11', w31', w41', w22', and w32' are obtained.
  • the mapping unit 1 stores the output neuron gradient and the processed weight to the storage unit 2.
  • the mapping unit 1 selects the output neuron corresponding to the processed weight, and stores the output neuron gradient and the processed weight to the storage unit 2, wherein the processed weight refers to the weight of the connection before the update. Or, the value or absolute value is greater than the weight of the first threshold.
  • connection relationship data of the artificial neural network stored in the first direct index form is 10110110
  • the output neuron gradient and the processed weight are combined into a data set: To1, w11, To1, w31, To1, w41, To2, w22, To2, w32, and stored to the storage unit 2.
  • the connection relation data of the artificial neural network stored in the second direct index form is 10011110, and the output neuron gradient and the processed weight are combined into a data set: To1, w11, To2, w22, To1, w31, To2, w32, To1. W41 and stored to the storage unit 2.
  • connection relation data of the artificial neural network stored in the step index form is 02111, and the output neuron gradient and the processed weight are combined into a data set: To1, w11, To1, w31, To1, w41, To2, w22, To2, w32. And stored to the storage unit 2.
  • the operation unit 7 reads the output neuron gradient and the processed weight from the storage unit 2, and stores them in the first input buffer 6 and the second input buffer 8, respectively, and the operation unit 7 reads the first input buffer 6 and the second
  • the output neuron gradient in the input buffer 8 and the processed weight are multiplied by the output neuron gradient and the processed weight one by one, and the multiplied results are added step by step through the addition tree, and the addition result is added
  • the inverse function of the activation function is performed, and the input neuron gradient is obtained and stored in the output buffer 9.
  • the input neuron gradient in the output buffer 9 is stored in the storage unit 2 via the DMA3 to complete the inverse operation of the artificial neural network.
  • the artificial neural network computing device of the present disclosure stores the processed weights and input neurons, output neurons, input neurons, output neurons, and processed weights, and performs forward and reverse operations. You can directly operate on the two without searching for the corresponding input and output neurons and weights through the index array, reducing the number of accesses to the memory, simplifying the operation, improving the speed and performance of the operation, and adopting multi-layer artificial nerves.
  • the dedicated instruction and operation unit 7 of the network operation solves the problem that the CPU and the GPU have insufficient performance and the front-end decoding overhead is large, and the support for the multi-layer artificial neural network operation algorithm is effectively improved; by using the multi-layer artificial neural network operation
  • the dedicated on-chip cache fully exploits the reusability of input neurons and weight data, avoids repeatedly reading these data into memory, reduces memory access bandwidth, and avoids memory bandwidth becoming a multi-layer artificial neural network operation and its training algorithm. Performance bottlenecks.
  • the above-mentioned arithmetic unit 7 includes but is not limited to three parts, respectively a multiplier, one or more adders (optionally, a plurality of adders constitute an addition tree) and an activation function unit.
  • mapping unit 1 stores the output neuron gradient and all input neurons to storage unit 2.
  • connection relation data when the connection relation data is stored in the form of the first direct index form and the step index, the data sets of the output neuron gradient and the input neurons are: To1, i1, To1, i2, To1, i3, To1, i4, To2 , i1, To2, i2, To2, i3, To2, i4, and stored in the storage unit 2, when the second direct index form is used to store the connection relationship data, the data set of the output neuron gradient and the input neuron is: To1. I1, To2, i1, To1, i2, To2, i2, To1, i3, To2, i3, To1, i4, To2, i4.
  • the operation unit 7 reads the output neuron gradient and the input neurons from the storage unit 2, and stores them in the first input buffer 6 and the second input buffer 8, respectively, and the operation unit 7 reads the first input buffer 6 and the second input buffer. 8 output neuron gradient and input neurons, multiply the output neuron gradient and the input neurons one by one, and multiply the multiplication result by the learning rate to obtain the update amount of the ownership value; subtract the corresponding value from the ownership value Update volume to get the updated ownership value.
  • the operation unit 7 determines whether the updated weight is still a connected weight, and if the updated weight is still a connected weight, the weight is set to a connected weight; otherwise, the weight is The weight is set to a connectionless weight, that is, there is no connection between the input neuron corresponding to the weight and the output neuron, and the updated processed weight is obtained.
  • the operation unit 7 compares the updated weight with the first threshold, and if the updated weight or the absolute value of the weight is greater than the first threshold, the weight is retained, that is, the input nerve corresponding to the weight There is a connection between the element and the output neuron; otherwise, the weight is deleted, that is, there is no connection between the input neuron corresponding to the weight and the output neuron, and the updated processed weight is obtained.
  • the mapping unit 1 further selects its corresponding output neuron based on the updated processed weight, and stores the processed weight and the output neuron gradient to the storage unit 2, wherein the processed weight refers to the processed The value of the weight before the update.
  • the operation unit 7 reads the output neuron gradient and the processed weight from the storage unit 2, and stores them in the first input buffer 6 and the second input buffer 8, respectively, and the operation unit 7 reads the first input buffer 6 and the second
  • the output neuron gradient of the buffer 8 and the processed weight are input, and an artificial neural network inverse operation is performed to obtain an input neuron gradient.
  • the present disclosure can also update the connection relationship data of the artificial neural network, and improve the precision of the artificial neural network operation.
  • mapping unit 1 is also located in a dedicated processor slice.
  • the mapping unit 1 reads the input neurons and weights stored in the storage unit 2 and processes them to obtain an input neuron-weight pair, and all the input neuron-weight pairs form a data set, and the data sets are respectively stored to the first The input buffer 6 and the second input buffer 8 are input.
  • the mapping unit 1 reads the input neurons, weights, and connection relationship data of the storage unit 2 and processes them to obtain an input neuron-weight pair, all input neurons.
  • the weight pair constitutes a data set, and the data set is stored to the first input buffer 6 and the second input buffer 8, respectively.
  • the mapping unit 1 stores its corresponding connected output neuron gradient and input neurons to the first input buffer 6 and the second input buffer 8, respectively, based on the processed weights.
  • the mapping unit 1 stores the output neuron gradient and the processed weights to the first input buffer 6 and the second input buffer 8, respectively.
  • mapping unit 1 is also located in the dedicated processor chip and is connected to the first input buffer 6, and is not connected to the second input buffer 8.
  • the mapping unit 1 reads and processes the connection relationship data of the input neuron and the input neuron stored by the storage unit 2, and stores the processed input neuron to the first input buffer 6; the storage unit 2 will process the weight Stored to the second input buffer 8, as shown in FIG.
  • the mapping unit 1 stores its corresponding output neuron gradient to the first input buffer 6 based on the processed weights, and the storage unit 2 directly stores the input neurons connected to the output neuron gradient to the second input buffer 8.
  • the mapping unit 1 stores the output neuron gradient to the first input buffer 6, and the storage unit 2 directly stores the processed weights to the second input buffer 8.
  • mapping unit 1 is also located in the dedicated processor chip and is connected to the second input buffer 8, and is not connected to the first input buffer 6.
  • the mapping unit 1 reads the connection relationship data of the weights and weights stored in the storage unit 2 and processes the processed values, and stores the processed weights in the second input buffer 8; the storage unit 2 stores the processed input neurons to the The first input buffer 6.
  • the mapping unit 1 reads the connection relationship data of the weight and the weight stored in the storage unit 2, processes the connection relationship data of the weight and the weight, and stores the processed weight to the second input buffer 8; 2
  • the processed input neurons are stored in the first input buffer 6, as shown in FIG.
  • the mapping unit 1 stores its corresponding input neurons to the first input buffer 6 based on the processed weights, and the storage unit 2 directly stores the output neuron gradients connected to the input neurons to the second input buffer 8.
  • the mapping unit 1 stores the processed weights to the first input buffer 6, and the storage unit 2 directly stores the output neuron gradients to the second input buffer 8.
  • the artificial neural network computing device can perform not only sparse neural network operations but also dense neural network operations.
  • the artificial neural network computing device described above is particularly suitable for the operation of a sparse neural network because the value of 0-value data or the absolute value of the sparse neural network is very large. These data can be proposed by the mapping unit, and the efficiency of the operation can be improved while ensuring the accuracy of the operation.
  • the sparse neural network is defined as: in the neural network, the number of weights with a value of 0 is large, and the distribution of weights with non-zero values is irregular, and the neural network is said to be a sparse neural network.
  • the ratio of the number of elements defining the weight 0 to the number of elements of the neural network ownership value is the sparsity of the neural network.
  • Still another embodiment of the present disclosure further provides an operation method for performing artificial neural network calculation using the artificial neural network computing device, and the operation method includes a forward operation of the artificial neural network, as shown in FIG. 12, including:
  • the mapping unit 1 processes the input neurons and the weights to obtain the connection relationship data of the artificial neural network
  • the mapping unit 1 obtains the input neuron-weight pair based on the connection relationship data, all the input neuron-weight pairs form a data set, the data set is stored in the storage unit 2, and the processed weight refers to the connected weight Or, is the weight of the index value or the absolute value greater than the first threshold.
  • the arithmetic unit 7 reads the input neurons and the processed weights, performs an artificial neural network forward operation, and obtains output neurons.
  • the arithmetic method includes the weight update of the artificial neural network, as shown in FIG. 13, including:
  • the mapping unit 1 stores its corresponding connected output neuron gradient and input neurons to the storage unit 2 based on the processed weights.
  • the arithmetic unit 7 reads the output neuron gradient and the input neurons, and updates the processed weights.
  • the output neuron gradient and the input neuron are stored to the storage unit; the output neuron gradient and the input neuron stored in the storage unit are respectively stored in the first input buffer and the second input buffer.
  • the updated amount of the weight; the processed weight is subtracted from the corresponding update amount to obtain the updated processed weight.
  • the arithmetic method includes the inverse operation of the artificial neural network, as shown in Figure 14, including:
  • the mapping unit 1 stores the output neuron gradient and the processed weight to the storage unit 2.
  • the output neurons corresponding to the processed weights are selected, and the output neuron gradients and the processed weights are stored in the storage unit 2; the output neuron gradients are read from the storage unit 2 and processed.
  • the weights are stored in the first input buffer 6 and the second input buffer 8, respectively.
  • the arithmetic unit 7 reads the output neuron gradient and the processed weight, and performs an inverse operation of the artificial neural network to obtain an input neuron gradient.
  • the operation unit 7 reads the output neuron gradient and the processed weight in the first input buffer 6 and the second input buffer 8; multiply the output neuron gradient and the connected weights one by one,
  • the multiplication results are added step by step through the addition tree; the addition result is inversely operated on the addition result to obtain an input neuron gradient.
  • Activation functions include sigmoid functions, tanh functions, or ReLU functions.
  • the mapping unit 1 stores the output neuron gradient and the input neurons to the storage unit 2.
  • the arithmetic unit 7 reads the output neuron gradient and the input neurons, and updates the processed weights.
  • the update amount of the weight includes the update amount of the ownership value, and the ownership value is subtracted from the corresponding update amount to obtain the updated ownership value.
  • the updated ownership value is judged. If the updated weight is a connected weight, the weight is set to a connected weight; otherwise, the weight is set to a connectionless weight. Alternatively, the updated ownership value is compared with a threshold value, and if the absolute value of the updated weight or weight is greater than the first threshold, the weight is retained; otherwise, the weight is deleted.
  • the mapping unit 1 further selects its corresponding output neuron based on the updated processed weight, and stores the processed weight and the output neuron gradient to the storage unit 2, wherein the processed weight refers to the processed The value of the weight before the update.
  • the arithmetic unit 7 reads the output neuron gradient and the processed weight, and performs an inverse operation of the artificial neural network to obtain an input neuron gradient.
  • the arithmetic method may include a forward operation, a weight update, and an inverse operation of the artificial neural network, as shown in FIG.
  • a further embodiment of the present disclosure further provides an operation method, in particular, a method for performing artificial neural network training by using the above artificial neural network computing device.
  • the operation method includes:
  • Forward operation steps First, a forward operation is sequentially performed on each layer in the multi-layer artificial neural network to obtain output neurons of each layer.
  • the forward operation of the layer can be performed in accordance with the forward operation steps of the previous embodiment.
  • the arithmetic unit will use the output data of the upper artificial neural network as the next artificial neural network under the control of the operation instruction.
  • the input data of the network is the input neuron of the artificial neural network as the input neuron of the next artificial neural network, or the inter-layer operation of the output neurons of the artificial neural network of the previous layer, and the operation result is taken as The input neurons of the next layer of artificial neural networks.
  • the next artificial neural network performs the forward operation, and at the same time, the weight is also replaced by the weight of the next artificial neural network.
  • the inter-layer operation can be a vector operation or a nonlinear operation.
  • Reverse operation steps Then, in the reverse order of the forward operation, the layers in the multi-layer artificial neural network are sequentially subjected to an inverse operation to obtain a weight gradient of each layer and an input neuron gradient.
  • This step includes a first arithmetic part and a second arithmetic part.
  • the first arithmetic part is used to calculate the weight gradient.
  • the gradient of the weight of the layer is obtained by matrix multiplication or convolution of the output neuron gradient of the layer and the input neurons.
  • the second computational portion is used to calculate the input neuron gradient.
  • the reverse operation of the layer can be performed in accordance with the reverse operation steps of the previous embodiment. Referring to FIG. 17, in this step, after the reverse operation of the upper artificial neural network is completed, the arithmetic unit will use the output data of the upper artificial neural network as the next artificial neural network under the control of the operation instruction. Input data of the network, but in the reverse operation, the output data of the upper artificial neural network is the input neuron gradient, and the input data of the next artificial neural network is the output neuron gradient, that is, the upper artificial neural network.
  • the next artificial neural network performs the inverse operation, and at the same time, the weight is also replaced by the weight of the next artificial neural network.
  • the “upper layer” and “next layer” are opposite to the forward operation, that is, for the adjacent two layers of the artificial neural network: the first layer and the second layer, If in the forward operation, the first layer is “upper layer” and the second layer is “lower layer”, then in the reverse operation, the first layer is "next layer” and the second layer is "upper layer”. layer”.
  • Weight update step Next, weight updates are performed on each layer in the multi-layer artificial neural network to obtain an updated weight.
  • the weight gradient of each layer is calculated.
  • the weight and weight gradient of the layer are respectively stored by using the first input buffer and the second input buffer, and then the weight is updated in the operation unit by using the weight gradient. , get the updated weight.
  • the forward operation step, the reverse operation step, and the weight update step are repeatedly performed multiple times to complete the training of the multi-layer artificial neural network.
  • the entire training method requires repeated execution of the above process until the parameters of the artificial neural network meet the requirements, and the training process is completed.
  • the neural network includes three layers: an input layer, an intermediate layer, and an output layer, wherein: the input layer neurons are responsible for receiving input data from the outside world and transmitting them to the middle layer neurons; the middle layer is responsible for information transformation, According to the requirement of information change capability, the middle layer can be designed as a single hidden layer or multiple hidden layer structure, and the last hidden layer is transmitted to the information of each neuron in the output layer, and after further processing, a forward propagation process of learning is completed; The output layer outputs information processing results to the outside world.
  • the back propagation phase of the error the error is corrected by the output layer, and the weight of each layer is corrected according to the error gradient, and the intermediate layer and the input layer are back-transferred layer by layer.
  • the input neurons and output neurons mentioned in the present invention do not refer to neurons in the input layer of the entire neural network and neurons in the output layer, but to any adjacent two layers in the network, which are under the network feedforward operation.
  • the neurons in the middle are the input neurons
  • the neurons in the upper layer of the network feedforward operation are the output neurons.
  • the operations mentioned above are all operations of a layer in a neural network.
  • the implementation process is that, in the forward operation, when the previous artificial neural network is executed, the next layer of operations is performed.
  • the instruction will operate the output neuron calculated in the operation unit as the input neuron of the next layer (or perform some operations on the output neuron as the input neuron of the next layer), and at the same time, the weight Also replaced with the weight of the next layer; in the reverse operation, when the reverse operation of the previous artificial neural network is completed, the next layer of operation instructions will take the input neuron gradient calculated in the operation unit as The output neuron gradient of one layer is operated (either by performing some operation on the input neuron gradient and then as the output neuron gradient of the next layer), and replacing the weight with the weight of the next layer.
  • arrows in broken lines in the figure indicate reverse operations
  • arrows in solid lines indicate forward operations.
  • the "upper layer” and “next layer” described in the reverse operation are exactly opposite to the forward operation, that is, the adjacent two layers of the artificial neural network: the first layer and the first The second layer, if in the forward operation, the first layer is “upper layer” and the second layer is “lower layer”, then in the reverse operation, the first layer is "next layer”, the second layer It is "upper level.”
  • Each functional unit/module may be hardware, such as the hardware may be a circuit, including digital circuits, analog circuits, and the like.
  • Physical implementations of hardware structures include, but are not limited to, physical devices including, but not limited to, transistors, memristors, and the like.
  • the computing modules in the computing device can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, ASIC, and the like.
  • the storage unit may be any suitable magnetic storage medium or magneto-optical storage medium such as RRAM, DRAM, SRAM, EDRAM, HBM, HMC, and the like.
  • the embodiment of the present disclosure further provides a computer storage medium, wherein the computer storage medium can store a program, and the program includes some or all of the steps of any one of the neural network processing methods described in the foregoing method embodiments.
  • the disclosed apparatus may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be electrical or otherwise.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in various embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present disclosure, or all or part of the technical solution, may be embodied in the form of a software product stored in a memory. A number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present disclosure.
  • the foregoing memory includes: a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and the like, which can store program codes.
  • ROM Read-Only Memory
  • RAM Random Access Memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Feedback Control In General (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Complex Calculations (AREA)
  • Machine Translation (AREA)

Abstract

一种运算方法，使用人工神经网络计算装置对多层人工神经网络进行运算。

Description

一种运算方法 技术领域
本公开涉及数据处理技术领域，更具体地涉及一种运算方法。
背景技术
人工神经网络(Artificial Neural Networks，ANNs)简称为神经网络(NNs)，它是一种模仿动物神经网络行为特征，进行分布式并行信息处理的算法数学模型。这种网络依靠系统的复杂程度，通过调整内部大量节点之间的相互连接关系，从而达到处理信息的目的。神经网络用到的算法就是向量乘法，并且广泛采用符号函数及其各种逼近。
就像大脑里的神经网络一样，神经网络由一些互相连接的节点组成，如图1所示，每个圆圈表示一个神经元，每个箭头表示两个神经元之间的连接，连接又被称为权值。
神经元的计算公式可以简单的描述成：
Figure PCTCN2017118264-appb-000001
其中，x i表示所有和输出神经元相连接的输入神经元，w i表示x i和输出神经元之间对应的权值。f(x)是一个非线性函数，通常称作激活函数，常用的函数如：
Figure PCTCN2017118264-appb-000002
Figure PCTCN2017118264-appb-000003
等。
神经网络被广泛应用于各种应用场景：计算视觉、语音识别和自然语言处理等。在近几年的时间里，神经网络的规模一直在增长。在1998年，Lecun用于手写字符识别的神经网络的规模小于1M个权值；在2012年，krizhevsky用于参加ImageNet竞赛的规模是60M个权值。
公开内容
根据本公开的一个方面，提供了一种运算方法，使用人工神经网络计算装置对多层人工神经网络进行训练，所述人工神经网络计算装置包括：映射单元、存储单元和运算单元，所述运算方法包括：对多层人工神经网络中的各层依次执行正向运算，对于所述多层人工神经网络的每一层，所 述正向运算包括：所述映射单元对输入数据进行处理，以得到处理后的输入数据；所述存储单元存储所述处理后的输入数据；所述运算单元读取所述处理后的输入数据，进行人工神经网络运算，得到人工神经网络的输出数据；按照与所述正向运算相反的顺序，对所述多层人工神经网络中的各层依次执行反向运算；对所述多层人工神经网络中的各层进行权值更新；重复执行上述各个步骤多次，完成多层人工神经网络的训练。
在本公开的一些实施例中，所述人工神经网络计算装置还包括：第一输入缓存和第二输入缓存；所述对多层人工神经网络中的各层进行权值更新包括：对于所述人工神经网络的每一层，所述第一输入缓存和所述第二输入缓存分别存储该层的权值和权值梯度；所述运算单元从所述第二输入缓存接收所述权值梯度，然后使用所述权值梯度对所述权值进行更新，得到更新后的权值。
在本公开的一些实施例中，所述输入数据包括：输入神经元和权值；所述映射单元对输入数据进行处理，以得到处理后的输入数据包括：所述映射单元对权值进行处理，得到处理后的权值和权值的连接关系；所述映射单元基于所述权值的连接关系数据对输入神经元进行处理，得到处理后的输入神经元。
在本公开的一些实施例中，所述输入数据包括输入神经元和权值；所述映射单元对输入数据进行处理，以得到处理后的输入数据包括：所述映射单元对输入神经元进行处理，得到处理后的输入神经元和输入神经元的连接关系；所述映射单元基于所述输入神经元的连接关系数据对权值进行处理，得到处理后的权值。
在本公开的一些实施例中，所述映射单元对权值进行处理包括：所述映射单元将无连接的权值删除，保留有连接的权值作为处理后的权值。
在本公开的一些实施例中，所述映射单元对权值进行处理包括：所述映射单元将所述权值中数值或绝对值小于或者等于第一阈值的权值删除，得到处理后的权值。
在本公开的一些实施例中，所述映射单元对输入神经元进行处理包括：所述映射单元将所述输入神经元中数值或绝对值小于或者等于第二阈值的输入神经元删除，得到处理后的输入神经元。
在本公开的一些实施例中，所述输入数据包括处理后的权值、输入神经元和所述输入神经元的连接关系数据；所述映射单元对输入数据进行处理，以得到处理后的输入数据包括；所述映射单元根据所述输入神经元的连接关系数据对所述输入神经元进行处理，以得到处理后的输入神经元。
在本公开的一些实施例中，所述输入数据包括处理后的输入神经元、权值和所述权值的连接关系数据；所述映射单元对输入数据进行处理，以得到处理后的输入数据包括；所述映射单元根据所述权值的连接关系数据对所述权值进行处理，以得到处理后的权值。
在本公开的一些实施例中，所述映射单元对输入数据进行处理，以得到处理后的输入数据包括：所述映射单元对至少一个输入神经元进行处理，得到输入神经元的连接关系数据；对至少一个权值进行处理，得到权值的连接关系数据；所述映射单元基于输入神经元的连接关系数据和权值的的连接关系数据得到第三连接关系数据，并根据第三连接关系数据对输入神经元和权值进行处理，得到处理后的输入神经元和处理后的权值。
在本公开的一些实施例中，所述输入神经元的连接关系数据和所述权值的连接关系数据以直接索引或者步长索引的形式表示。
在本公开的一些实施例中，当所述输入神经元的连接关系数据以直接索引的形式表示时，所述连接关系数据为由0和1组成的字符串，0表示所述输入神经元的值为零，1表示所述输入神经元的值非零；当所述输入神经元的连接关系数据以步长索引形式表示时，所述连接关系数据为值为非零的输入神经元与上一个值为非零的输入神经元之间的距离值组成的字符串。
在本公开的一些实施例中，当所述权值的连接关系数据以直接索引的形式表示时，所述连接关系数据为由0和1组成的字符串，0表示所述权值为无连接的权值，1表示所述权值为有连接的权值；当所述权值的连接关系数据以步长索引的形式表示时，所述连接关系数据为与输出神经元有连接的输入神经元的与上一个与所述输出神经元有连接的输入神经元之间的距离值组成的字符串。
在本公开的一些实施例中，对于多层人工神经网络的每一层，所述反向运算包括：权值梯度计算步骤和输入神经元梯度计算步骤；所述权值梯 度计算步骤包括：所述运算单元由人工神经网络每一层的输出神经元梯度和输入神经元得到该层的权值梯度；所述输入神经元梯度计算步骤包括：所述映射单元将输出神经元梯度和处理后的权值存储至存储单元；所述运算单元读取输出神经元梯度和处理后的权值，进行人工神经网络运算，得到输入神经元梯度。
在本公开的一些实施例中，将人工神经网络上一层的输出数据作为下一层的输入数据；或者，对人工神经网络上一层的输出数据进行层间操作，将操作结果作为下一层的输入数据；其中，对于人工神经网络的任意相邻两层：第一层与第二层，在正向运算中，所述上一层指所述第一层，所述下一层指所述第二层；在反向运算中，所述上一层指所述第二层，所述下一层指所述第一层。
本公开的运算方法具有以下有益效果：将输入神经元与权值、输出神经元与输入神经元、输出神经元与权值以一一对应的格式进行存储，进行前向运算和反向运算时，可以直接对一一对应的二者进行运算，无需通过索引数组寻找对应的输入、输出神经元和权值，减少了访问存储器的次数，简化了运算，提高了运算速度和性能。可以对稀疏人工神经网络的连接关系数据进行更新，提高了稀疏人工神经网络运算的精度；采用针对稀疏多层人工神经网络运算的专用指令和运算单元，解决了CPU和GPU运算性能不足，前端译码开销大的问题，有效提高了对多层人工神经网络运算算法的支持。
附图说明
附图是用来提供对本公开的进一步理解，并且构成说明书的一部分，与下面的具体实施方式一起用于解释本公开，但并不构成对本公开的限制。在附图中：
图1是一种人工神经网络的示意图；
图2是本公开一实施例的人工神经网络计算装置的结构示意图；
图3是图2所示实施例的映射关系示意图；
图4是本公开一实施例的一种人工神经网络的示意图；
图5是图4所示的人工神经网络计算装置连接关系示意图；
图6是本公开另一实施例的人工神经网络计算装置的结构示意图；
图7是图6所示实施例的映射关系示意图；
图8是本公开又一实施例的人工神经网络计算装置的结构示意图；
图9是图8所示实施例的映射关系示意图；
图10是本公开再一实施例的人工神经网络计算装置的结构示意图；
图11是图10所示实施例的映射关系示意图；
图12是本公开实施例的一种运算方法的正向运算流程图；
图13是本公开实施例的一种运算方法的权值更新流程图；
图14是本公开实施例的一种运算方法的反向运算流程图；
图15是本公开实施例的一种运算方法的流程图。
图16是本公开又一实施例的一种运算方法的流程图。
图17是本公开又一实施例运算方法相邻两层的运算过程图。
符号说明
1-映射单元；2-存储单元；3-DMA；4-指令缓存；5-控制单元；6-第一输入缓存；7-运算单元；8-第二输入缓存；9-输出缓存。
具体实施方式
为使本公开的目的、技术方案和优点更加清楚明白，以下结合具体实施例，并参照附图，对本公开进一步详细说明。
本公开一个实施例提出了一种人工神经网络计算装置，参见图2a，包括：存储单元2、映射单元1、运算单元7、DMA3、指令缓存4、第一输入缓存6、第二输入缓存8、输出缓存9和控制单元5。
上述映射单元1，用于接收输入数据之后，对所述输入数据进行处理，以得到处理后的输入数据，所述输入数据包括至少一个输入神经元和至少一个权值，所述处理后的输入数据包括处理后的输入神经元和处理后的权值。
进一步地，上述映射单元1将上述处理后的输入神经元中的每个处理后的输入神经元与其对应的处理后的权值作为一个输入神经元-权值对，所有的输入神经元-权值对组成一个数据集，将该数据集存储到上述存储单元2中。
具体来说，上述映射单元1将无连接的权值删除，保留有连接的权值 作为处理后的权值，映射单元1还得到权值的连接关系数据，根据权值的连接关系数据对输入神经元进行处理，得到处理后的输入神经元。
例如，当权值的形式为w ij，该w ij表示第i个输入神经元与第j个输出神经元之间的权值；映射单元1根据权值确定上述权值的连接关系数据，将无连接的权值删除，保留有连接的权值作为处理后的权值。
或者，具体来说，上述映射单元1将数值或绝对值小于或等于第一阈值的权值删除，得到处理后的权值，映射单元1还得到权值的连接关系数据，根据权值的连接关系数据对输入神经元进行处理，得到处理后的输入神经元。
例如，当权值的形式为w ij，该w ij表示第i个输入神经元与第j个输出神经元之间的权值；映射单元1根据权值确定上述权值的连接关系数据，并将权值或权值的绝对值小于或者等于第一阈值的权值删除，得到处理后的权值。
上述映射单元1还可以将数值或绝对值小于或等于第二阈值的输入神经元删除，以得到去零后的输入神经元，作为处理后的输入神经元，映射单元1还得到输入神经元的连接关系数据，根据输入神经元的连接关系数据对权值进行处理，得到处理后的权值。
可选地，上述第一阈值可为0.1、0.08、0.05、0.02、0.01、0或者其他值。
可选地，上述第二阈值可为0.1、0.08、0.06、0.05、0.02、0.01、0或者其他值。
需要指出的是，上述第一阈值和上述第二阈值可以一致，也可以不一致。
其中，上述连接关系数据可以步长索引或者直接索引的形式表示。
具体地，以直接索引形式表示的连接关系数据为由0和1组成的字符串，对于权值连接关系数据，0表示该权值或该权值的绝对值小于或者等于上述第一阈值，即该权值对应的输入神经元与输出神经元之间没有连接，1表示该权值或该权值的绝对值大于上述第一阈值，即该权值对应的输入神经元与输出神经元之间有连接。以直接索引形式表示的连接关系数据有两种表示顺序：以每个输出神经元与所有输入神经元的连接状态组成一个 0和1的字符串来表示权值的连接关系；或者每个输入神经元与所有输出神经元的连接状态组成一个0和1的字符串来表示权值的连接关系。
对于输入神经元的连接关系数据，0表示该输入神经元或该输入神经元的绝对值小于或者等于上述第二阈值，1表示该输入神经元或该输入神经元的绝对值大于上述第二阈值。
对于权值的连接关系数据，以步长索引形式表示的连接关系数据为与输出神经元有连接的输入神经元与上一个与该输出神经元有连接的输入神经元之间的距离值组成的字符串。对于输入神经元，以步长索引表示的连接关系数据以当前非零的输入神经元与上一个非0的输入神经元之间的距离值组成的字符串表示。
举例说明，假设上述第一阈值为0.01，输入神经元为i1、i2、i3和i4，对于输出神经元o1，权值为w 11，w 21，w 31和w 41；对于输出神经元o2，权值为w 12，w 22，w 32和w 42，其中权值w 21，w 12和w 42的值为0，其绝对值均小于上述第一阈值0.01，映射单元1确定上述输入神经元i2和输出神经元o1没有连接，上述输入神经元i1和i4与输出神经元o2没有连接，上述输入神经元i1、i3和i4与上述输出神经元o1有连接，上述输入神经元i2和i3与输出神经元o2有连接。以每个输出神经元与所有输入神经元的连接状态表示上述连接关系数据，则上述输出神经元o1的连接关系数据为“1011”，输出神经元o2的连接关系数据为“0110”(即上述连接关系数据为“10110110”)；以每个输入神经元与所有输出神经元的连接关系，则输入神经元i1的连接关系数据为“10”，输入神经元i2的连接关系数据为“01”，输入神经元i3的连接关系数据为“11”，输入神经元i4的连接关系数据为“10”(即上述连接关系数据为“10011110”)。在本实施例中，连接关系数据可以用高维动态数组，可以用链表等等表示。
对于上述输出神经元o1，上述映射单元101将上述i1与w 11，i3与w 31和i4与w 41分别作为一个输入神经元-权值对，所有的输入神经元-权值对组成一个数据集，并将该数据集存储到上述存储单元2中；对于输出神经元o2，上述映射单元101将上述i2与w 22和i3与w 32分别作为一个输入神经元-权值对，所有的输入神经元-权值对组成一个数据集，并将该数据集存储到上述存储单元2中。
所述输入神经元-权值对不是一种真正的数据存储结构，仅仅是表示输入神经元和权值的对应关系。例如，输入神经元存储于向量A中，权值存储于向量B中，向量A和B的长度相同，向量A和B的同一位置的分量组合在一起被认为是一个输入神经元-权值对。
在另一个示例中，上述映射单元101对至少一个输入神经元进行处理，得到输入神经元的连接关系数据，对至少一个权值进行处理，得到权值的的连接关系数据，并基于输入神经元的连接关系数据和权值的的连接关系数据得到第三连接关系数据，根据第三连接关系数据对输入神经元和权值进行处理，得到处理后的输入神经元和处理后的权值。
输入神经元的连接关系数据为用于表示该至少一个输入神经元中值为非0的输入神经元的位置的字符串；权值的连接关系数据为用于表示输入神经元与输出神经元之间是否有连接的字符串。
需要说明的是，上述输入神经元的连接关系数据和权值的连接关系数据均可以步长索引或者直接索引的形式表示，具体可参见上一示例中相关的描述。
具体地，当上述输入神经元的连接关系数据和权值的连接关系数据均以直接索引的形式表示时，上述映射单元1对上述输入神经元的连接关系数据和权值的连接关系数据进行与操作。
当上述输入神经元的连接关系数据和权值的连接关系数据均以步长索引的形式表示时，上述映射单元1从左到右将上述输入神经元的连接关系数据字符串中的每一个元素与位于其之前的元素进行累加，得到的新的元素组成输入神经元的连接关系数据；同理，上述映射单元1对上述权值的连接关系数据的字符串进行同样的处理。然后上述映射单元1从上述输入神经元的连接关系数据和权值的连接关系数据的字符串中，选取相同的元素，按照其在上述输入神经元的连接关系数据的顺序或权值的连接关系数据的顺序，组成一个新的字符串。上述映射单元1将上述新的字符串中将每一个元素与其相邻的前一个元素进行相减，以得到一个新的元素。按照该方法，对上述新的字串中的每个元素进行相应的操作，以得到上述第三连接关系数据。
举例说明，假设以步长索引的形式表示上述输入神经元的连接关系数 据和权值的连接关系数据，上述权值的连接关系数据的字符串为“01111”，上述输入神经元的连接关系数据的字符串为“022”，将上述权值的连接关系数据的字符串中的每个元素与其相邻的前一个元素相加，得到第四连接关系数据“01234”；同理，对上述输入神经元的连接关系数据的字符串进行相同的处理后得到的第五连接关系数据为“024”。从上述第四连接关系数据“01234”和上述第五连接关系数据“024”选组相同的元素，以得到新的字符串“024”。将该新的字符串中的每个元素与其相邻的前一个元素进行相减，即0，(2-0)，(4-2)，以得到上述第三连接数据“022”。
上述映射单元1将上述处理后的输入神经元中的每个处理后的输入神经元与其对应的处理后的权值作为一个输入神经元-权值对，所有的输入神经元-权值对组成一个数据集，将该数据集存储出上述存储单元2中。
可以看出，在本公开实施例的方案中，上述映射单元对输入神经元和权值进行处理，剔除值为0的输入神经元和权值，减少了输入神经元和权值的数量，减少了额外的开销，运算单元根据处理后的输入神经元和权值进行人工神经网络运算，提高了运算的效率。
在本实施例中，当人工神经网络计算装置用于正向运算时，
映射单元1读取存储单元2存储的输入神经元和权值并进行处理，获取人工神经网络的连接关系数据。
该连接关系数据可以采用上述直接索引的形式表示。例如对于如图4所示的人工神经网络，有4个输入神经元：i1、i2、i3、i4，有2个输出神经元：o1、o2。对于该人工神经网络，如图5所示，o1与i1、i3、i4有连接，o1与i2无连接，则输出神经元o1与输入神经元i1、i2、i3、i4的连接状态为：1011。类似地，o2与i2、i3有连接，o2与i1、i4无连接，则输出神经元o2与输入神经元i1、i2、i3、i4的连接状态为：0110，将输出神经元o1、o2与所有输入神经元的连接状态依次排列，第一直接索引形式存储的人工神经网络的连接关系数据为10110110。
该连接关系数据也可以采用第二直接索引形式存储。
例如对于如图5所示的人工神经网络，i1与o1有连接、i1与o2无连接，i2与o1无连接、i2与o2有连接，i3与o1、o2有连接，i4与o1有连接、i4与o2无连接，则输入神经元i1与输出神经元o1、o2、o3、o4的连 接状态为：10，类似地，输入神经元i2与输出神经元o1、o2、o3、o4的连接状态为：01，输入神经元i3与输出神经元o1、o2、o3、o4的连接状态为11，输入神经元i4与输出神经元o1、o2、o3、o4的连接状态为10，将输入神经元i1、i2、i3、i4与所有输出神经元的连接状态依次排列，第二直接索引形式存储的人工神经网络的连接关系数据为10011110。
该连接关系数据还可以采用步长索引形式存储。
例如对于如图4所示的人工神经网络，输出神经元o1与输入神经元i1、i3、i4有连接，输出神经元o1的连接状态第一个字符为0，即和o1有连接的第一个输入神经元i1，与人工神经网络的第一个输入神经元，也是i1，的距离为0，和o1有连接的第二个输入神经元i3与和o1有连接的第一个输入神经元i1的距离为2，和o1有连接的第三个输入神经元i4与和o1有连接的第二个输入神经元i3的距离为1，因此，输出神经元o1的连接状态为021。输出神经元o2与输入神经元i2、i3有连接，类似地，输出神经元o2的连接状态为11，则将输出神经元o1、o2与所有输入神经元的连接状态依次排列，步长索引形式存储的人工神经网络的连接关系数据为02111。
得到人工神经网络的连接关系数据后，可以据此得到处理后的输入神经元和权值。映射单元1基于连接关系数据，将处理后的权值与输入神经元组成的数据集存储至存储单元2，处理后的权值是指有连接的权值，或者，是指数值或绝对值大于第一阈值的权值。例如，第一直接索引形式存储的人工神经网络的连接关系数据为10110110，处理后的权值w11、w31、w41、w22、w32对应的输入神经元分别为i1、i3、i4、i2、i3，则将输入神经元与处理后的权值组成数据集：i1、w11，i3、w31，i4、w41，i2、w22，i3、w32，并存储至存储单元2。
第二直接索引形式存储的人工神经网络的连接关系数据为10011110，将输入神经元与处理后的权值组成数据集：i1、w11，i2、w22，i3、w31，i3、w32，i4、w41，并存储至存储单元2。步长索引形式存储的人工神经网络的连接关系数据为02111，将输入神经元与处理后的权值组成数据集：i1、w11，i3、w31，i4、w41，i2、w22，i3、w32，并存储至存储单元2。
在其他示例中，也可以如图3所示，映射单元1读取存储单元2的输 入神经元、权值和连接关系数据并进行处理，映射单元1将处理后的权值与输入神经元组成的数据集格式分别存储至第一输入缓存6和第二输入缓存8。
控制单元5控制指令缓存4读取存储单元2中的指令，并基于指令控制运算单元7运算。运算单元7在控制单元5的控制下，将存储单元2中存储的数据集的输入神经元和权值分别存入第一输入缓存6和第二输入缓存8，运算单元7分别从第一输入缓存6和第二输入缓存8中读取输入神经元和权值，并进行人工神经网络前向运算，得到输出神经元并存入输出缓存9，输出缓存9中的输出神经元经DMA3存入存储单元2。
运算单元7将输入神经元和权值数据一一对应相乘得到加权输出神经元，将加权输出神经元通过加法树逐级相加得到求和输出神经元，或者，将加权输出神经元通过和偏置相加得到加偏置输出神经元，将求和输出神经元或加偏置输出神经元进行激活函数运算，得到人工神经网络的输出神经元，完成人工神经网络的前向运算。激活函数可以是sigmoid函数、tanh函数或ReLU函数。
本实施例中，运算单元7、指令缓存4、第一输入缓存6、第二输入缓存8、输出缓存9和控制单元5由专用处理器实现，第一输入缓存6、第二输入缓存8、输出缓存9均为片上缓存。
当人工神经网络计算装置用于权值更新时，
映射单元1基于处理后的权值将其对应的有连接的输出神经元梯度与输入神经元存储至存储单元2。
运算单元7在控制单元5的控制下，将存储单元2中存储的输出神经元梯度和输入神经元分别存入第一输入缓存6和第二输入缓存8，运算单元7分别从第一输入缓存6和第二输入缓存8中读取输出神经元梯度和输入神经元，将输出神经元梯度和输入神经元一一对应相乘，并将相乘结果乘以学习率，得到处理后的权值的更新量；将处理后的权值减去对应的更新量，得到更新的处理后的权值并存入输出缓存9，输出缓存9中的更新的处理后的权值经DMA3存入存储单元2。
例如，第一直接索引形式存储的人工神经网络的连接关系数据为10110110，处理后的权值w11、w31、w41、w22、w32对应的输入神经元 分别为i1、i3、i4、i2、i3，则将处理后的权值对应的输出神经元梯度与输入神经元组成数据集：To1、i1，To1、i3，To1、i4，To2、i2，To2、i3，并存储至存储单元2，To1和To2分别表示输出神经元o1和o2的梯度。To1与i1，To1与i3，To1与i4，To2与i2，To2与i3对应相乘，相乘结果再乘以学习率，得到处理后的权值的更新量Δw11、Δw31、Δw41、Δw22、Δw32，处理后的权值w11、w31、w41、w22、w32分别减去更新量，得到更新的处理后的权值w11’、w31’、w41’、w22’、w32’。
类似地，第二直接索引形式存储的人工神经网络的连接关系数据为10011110，处理后的权值为w11，w22，w31，w32，w41对应的输入神经元分别为i1、i2、i3、i3、i4，则将输出神经元梯度与输入神经元组成数据集：To1、i1，To2、i2，To1、i3，To2、i3，To1、i4，并存储至存储单元2。To1与i1，To2与i2，To1与i3，To2与i3，To1与i4对应相乘，相乘结果再乘以学习率，得到处理后的权值的更新量Δw11，Δw22，Δw31，Δw32，Δw41，处理后的权值w11，w22，w31，w32，w41分别减去更新量，得到更新的处理后的权值w11’，w22’，w31’，w32’，w41’。
步长索引形式存储的人工神经网络的连接关系数据为02111，处理后的权值w11、w31、w41、w22、w32对应的输入神经元分别为i1、i3、i4、i2、i3，则将输出神经元梯度与输入神经元组成数据集：To1、i1，To1、i3，To1、i4，To2、i2，To2、i3，并存储至存储单元2。To1与i1，To1与i3，To1与i4，To2与i2，To2与i3对应相乘，相乘结果再乘以学习率，得到处理后的权值的更新量Δw11、Δw31、Δw41、Δw22、Δw32，处理后的权值w11、w31、w41、w22、w32分别减去更新量，得到更新的处理后的权值w11’、w31’、w41’、w22’、w32’。
当人工神经网络计算装置用于反向运算时，
映射单元1将输出神经元梯度和处理后的权值存储至存储单元2。
映射单元1选取处理后的权值所对应的输出神经元，并将输出神经元梯度与处理后的权值存储至存储单元2，其中处理后的权值是指在更新前有连接的权值，或者，数值或绝对值大于第一阈值的权值。
例如，第一直接索引形式存储的人工神经网络的连接关系数据为10110110，将输出神经元梯度与处理后的权值组成数据集：To1，w11，To1， w31，To1，w41，To2，w22，To2，w32，并存储至存储单元2。第二直接索引形式存储的人工神经网络的连接关系数据为10011110，将输出神经元梯度与处理后的权值组成数据集：To1、w11，To2、w22，To1、w31，To2、w32，To1、w41，并存储至存储单元2。步长索引形式存储的人工神经网络的连接关系数据为02111，将输出神经元梯度与处理后的权值组成数据集：To1，w11，To1，w31，To1，w41，To2，w22，To2，w32，并存储至存储单元2。
运算单元7从存储单元2中读取输出神经元梯度和处理后的权值，并分别存入第一输入缓存6和第二输入缓存8，运算单元7读取第一输入缓存6和第二输入缓存8中的输出神经元梯度和处理后的权值，将输出神经元梯度和处理后的权值一一对应相乘，并将相乘结果通过加法树逐级相加，对相加结果进行激活函数反运算，得到输入神经元梯度并存入输出缓存9，输出缓存9中的输入神经元梯度经DMA3存入存储单元2，完成人工神经网络的反向运算。
本公开的人工神经网络计算装置，将处理后的权值与输入神经元、输出神经元与输入神经元、输出神经元与处理后的权值进行存储，进行前向运算和反向运算时，可以直接对二者进行运算，无需通过索引数组寻找对应的输入、输出神经元和权值，减少了访问存储器的次数，简化了运算，提高了运算速度和性能；并且通过采用针对多层人工神经网络运算的专用指令和运算单元7，解决了CPU和GPU运算性能不足，前端译码开销大的问题，有效提高了对多层人工神经网络运算算法的支持；通过采用针对多层人工神经网络运算的专用片上缓存，充分挖掘了输入神经元和权值数据的重用性，避免了反复向内存读取这些数据，降低了内存访问带宽，避免了内存带宽成为多层人工神经网络运算及其训练算法性能瓶颈的问题。
进一步地，上述运算单元7包括但不限定于三个部分，分别为乘法器、一个或多个加法器(可选的，多个加法器组成加法树)和激活函数单元。上述乘法器将第一输入数据(in1)和第二输入数据(in2)相乘得到第一输出数据(out1)，过程为：out1＝in1*in2；上述加法树将第三输入数据(in3)通过加法树逐级相加得到第二输出数据(out2)，其中in3是一个长度为N的向量，N大于1，过称为：out2＝in3[1]+in3[2]+...+in3[N]，和/或将第三 输入数据(in3)通过加法树累加之后得到的结果和第四输入数据(in4)相加得到第二输出数据(out2)，过程为：out2＝in3[1]+in3[2]+...+in3[N]+in4，或者将第三输入数据(in3)和第四输入数据(in4)相加得到第二输出数据(out2)，过称为：out2＝in3+in4；上述激活函数单元将第五输入数据(in5)通过激活函数(active)运算得到第三输出数据(out3)，过程为：out3＝active(in5)，激活函数active可以是sigmoid、tanh、relu、softmax等函数，除了做激活操作，激活函数单元可以实现其他的非线性函数运算，可将输入数据(in)通过函数(f)运算得到输出数据(out)，过程为：out＝f(in)。
本公开另一个实施例的人工神经网络计算装置，与前述实施例相同的特征不再重复描述。
在反向运算中，映射单元1将输出神经元梯度与全部输入神经元存储至存储单元2。
例如，当采用第一直接索引形式和步长索引形式存储连接关系数据时，输出神经元梯度与输入神经元的数据集为：To1、i1，To1、i2，To1、i3，To1、i4，To2、i1，To2、i2，To2、i3，To2、i4，并存储至存储单元2，当采用第二直接索引形式存储连接关系数据时，输出神经元梯度与输入神经元的数据集为：To1、i1，To2、i1，To1、i2，To2、i2，To1、i3，To2、i3，To1、i4，To2、i4。
运算单元7从存储单元2中读取输出神经元梯度和输入神经元，并分别存入第一输入缓存6和第二输入缓存8，运算单元7读取第一输入缓存6和第二输入缓存8的输出神经元梯度和输入神经元，将输出神经元梯度和输入神经元一一对应相乘，并将相乘结果乘以学习率，得到所有权值的更新量；将所有权值减去对应的更新量，得到更新后的所有权值。
运算单元7判断更新后的权值是否仍是有连接的权值，若更新后的权值仍是有连接的权值，则将该权值设置为有连接的权值，否则，则将该权值设置为无连接的权值，即该权值对应的输入神经元与输出神经元之间无连接，得到更新的处理后的权值。
或者，运算单元7将更新后的权值与第一阈值进行比较，若更新后的权值或权值的绝对值大于第一阈值，则将该权值保留，即该权值对应的输入神经元与输出神经元之间有连接；否则，则将该权值删除，即该权值对 应的输入神经元与输出神经元之间无连接，得到更新的处理后的权值。
映射单元1还基于更新的处理后的权值选取其对应的输出神经元，并将处理后的权值与输出神经元梯度存储至存储单元2，其中处理后的权值是指该处理后的权值在更新之前的值。
运算单元7从存储单元2中读取输出神经元梯度和处理后的权值，并分别存入第一输入缓存6和第二输入缓存8，运算单元7读取第一输入缓存6和第二输入缓存8的输出神经元梯度和处理后的权值，并进行人工神经网络反向运算，得到输入神经元梯度。
由此可见，本公开还可以对人工神经网络的连接关系数据进行更新，提高了人工神经网络运算的精度。
本公开又一个实施例的人工神经网络计算装置，与前述实施例相同的特征不再重复描述。如图6所示，本实施例中，映射单元1也位于专用处理器片内。
当人工神经网络计算装置用于正向运算时，
映射单元1读取存储单元2存储的输入神经元和权值并进行处理，得到输入神经元-权值对，所有的输入神经元-权值对组成一个数据集，数据集分别存储至第一输入缓存6和第二输入缓存8。在其他示例中，也可以如图7所示，映射单元1读取存储单元2的输入神经元、权值和连接关系数据并进行处理，得到输入神经元-权值对，所有的输入神经元-权值对组成一个数据集，数据集分别存储至第一输入缓存6和第二输入缓存8。
当人工神经网络计算装置用于权值更新时，
映射单元1基于处理后的权值将其对应的有连接的输出神经元梯度与输入神经元分别存储至第一输入缓存6和第二输入缓存8。
当人工神经网络计算装置用于反向运算时，
映射单元1将输出神经元梯度和处理后的权值分别存储至第一输入缓存6和第二输入缓存8。
本公开再一个实施例的人工神经网络计算装置，与前述实施例相同的特征不再重复描述。如图8所示，本实施例中，映射单元1也位于专用处理器片内且连接第一输入缓存6，不连接第二输入缓存8。
当人工神经网络计算装置用于正向运算时，
映射单元1读取存储单元2存储的输入神经元和输入神经元的连接关系数据并进行处理，并将处理后的输入神经元存储至第一输入缓存6；存储单元2将处理后的权值存储至第二输入缓存8，如图9所示。
当人工神经网络计算装置用于权值更新时，
映射单元1基于处理后的权值将其对应的输出神经元梯度存储至第一输入缓存6，存储单元2直接将与输出神经元梯度有连接的输入神经元存储至第二输入缓存8。
当人工神经网络计算装置用于反向运算时，
映射单元1将输出神经元梯度存储至第一输入缓存6，存储单元2直接将处理后的权值存储至第二输入缓存8。
本公开再一个实施例的人工神经网络计算装置，与前述实施例相同的特征不再重复描述。如图10所示，本实施例中，映射单元1也位于专用处理器片内且连接第二输入缓存8，不连接第一输入缓存6。
当人工神经网络计算装置用于正向运算时，
映射单元1读取存储单元2存储的权值和权值的连接关系数据并进行处理，并将处理后的权值存储至第二输入缓存8；存储单元2将处理后的输入神经元存储至第一输入缓存6。映射单元1读取存储单元2存储的权值和权值的连接关系数据，对权值和权值的连接关系数据进行处理，并将处理后的权值存储至第二输入缓存8；存储单元2将处理后的输入神经元存储至第一输入缓存6，如图11所示。
当人工神经网络计算装置用于权值更新时，
映射单元1基于处理后的权值将其对应的输入神经元存储至第一输入缓存6，存储单元2直接将与输入神经元有连接的输出神经元梯度存储至第二输入缓存8。
当人工神经网络计算装置用于反向运算时，
映射单元1将处理后的权值存储至第一输入缓存6，存储单元2直接将输出神经元梯度存储至第二输入缓存8。
需要说明的是，上述人工神经网络计算装置不仅可以进行稀疏神经网络运算，还可以进行稠密神经网络运算。上述人工神经网络计算装置特别适用于稀疏神经网络的运算，是因为稀疏神经网络里0值数据或者绝对值 很小的数据非常多。通过映射单元可以提出这些数据，在保证运算精度的情况下，可提高运算的效率。
稀疏神经网络的定义为：在神经网络中，值为0的权值的数目很多，并且值为非0的权值分布没有规律，则称该神经网络为稀疏神经网络。定义权值为0的元素数量与神经网络所有权值元素数量的比值为神经网络的稀疏度。
本公开再一个实施例还提出了一种运算方法，其利用上述人工神经网络计算装置进行人工神经网络计算，运算方法包括人工神经网络的正向运算，如图12，包括：
映射单元1对输入神经元和权值进行处理，获取人工神经网络的连接关系数据；
映射单元1基于连接关系数据得到输入神经元-权值对，所有的输入神经元-权值对组成一个数据集，数据集存储至存储单元2，处理后的权值是指有连接的权值，或者，是指数值或绝对值大于第一阈值的权值。
运算单元7读取输入神经元和处理后的权值，进行人工神经网络前向运算，得到输出神经元。
运算方法包括人工神经网络的权值更新，如图13，包括：
映射单元1基于处理后的权值将其对应的有连接的输出神经元梯度与输入神经元存储至存储单元2。
运算单元7读取输出神经元梯度和输入神经元，对处理后的权值进行更新。
具体来说，将输出神经元梯度与输入神经元存储至存储单元；将存储单元中存储的输出神经元梯度和输入神经元分别存入第一输入缓存和第二输入缓存。
从第一输入缓存和第二输入缓存中读取输出神经元梯度和输入神经元；将输出神经元梯度和输入神经元一一对应相乘，并将相乘结果乘以学习率，得到处理后的权值的更新量；将处理后的权值减去对应的更新量，得到更新的处理后的权值。
运算方法包括人工神经网络的反向运算，如图14，包括：
映射单元1将输出神经元梯度和处理后的权值存储至存储单元2。
具体来说，选取处理后的权值所对应的输出神经元，并将输出神经元梯度与处理后的权值存储至存储单元2；从存储单元2中读取输出神经元梯度和处理后的权值，并分别存入第一输入缓存6和第二输入缓存8。
运算单元7读取输出神经元梯度和处理后的权值，进行人工神经网络反向运算，得到输入神经元梯度。
具体来说，运算单元7读取第一输入缓存6和第二输入缓存8中的输出神经元梯度和处理后的权值；将输出神经元梯度和有连接的权值一一对应相乘，并将相乘结果通过加法树逐级相加；对相加结果进行激活函数反运算，得到输入神经元梯度。激活函数包括sigmoid函数、tanh函数或ReLU函数。
其中，可以仅存储处理后的权值，也可以将所有权值均进行存储，当存储所有权值时，
映射单元1将输出神经元梯度与输入神经元存储至存储单元2。
运算单元7读取输出神经元梯度和输入神经元，对处理后的权值进行更新。
权值的更新量包括所有权值的更新量，将所有权值减去对应的更新量，得到更新后的所有权值。对更新后的所有权值进行判断，若更新后的权值为有连接的权值，则将该权值设置为有连接的权值，否则，将该权值设置为无连接的权值。或者，将更新后的所有权值与门限值进行比较，若更新后的权值或权值的绝对值大于第一阈值，则将保留该权值，否则，将该权值删除。
映射单元1还基于更新的处理后的权值选取其对应的输出神经元，并将处理后的权值与输出神经元梯度存储至存储单元2，其中处理后的权值是指该处理后的权值在更新之前的值。
运算单元7读取输出神经元梯度和处理后的权值，进行人工神经网络反向运算，得到输入神经元梯度。
运算方法可以包括人工神经网络的正向运算、权值更新和反向运算，如图15所示。
本公开又一个实施例还提出了一种运算方法，具体来说是一种利用上述人工神经网络计算装置进行人工神经网络训练的方法，参见图16，运算 方法包括：
正向运算步骤：首先，对多层人工神经网络中的各层依次执行正向运算，得到各层的输出神经元。
对于人工神经网络的每一层，可以按照上一实施例的正向运算步骤进行该层的正向运算。参见图17，在该步骤中，当上一层人工神经网络的正向运算执行完毕后，在运算指令的控制下，运算单元会将上一层人工神经网络的输出数据作为下一层人工神经网络的输入数据，即将上一层人工神经网络的输出神经元作为下一层人工神经网络的输入神经元，或者是对上一层人工神经网络的输出神经元进行层间操作，将操作结果作为下一层人工神经网络的输入神经元。得到输入神经元后，下一层人工神经网络进行正向运算，同时，将权值也替换为下一层人工神经网络的权值。其中该层间操作可以是向量运算、非线性运算。
反向运算步骤：然后，按照与正向运算相反的顺序，对多层人工神经网络中的各层依次执行反向运算，得到各层的权值梯度和输入神经元梯度。
该步骤包括第一运算部分和第二运算部分。第一运算部分用于计算权值梯度。对于人工神经网络的每一层，由该层的输出神经元梯度和输入神经元通过矩阵乘法或卷积得到该层的权值梯度。
第二运算部分用于计算输入神经元梯度。对于人工神经网络的每一层，可以按照上一实施例的反向运算步骤进行该层的反向运算。参见图17，在该步骤中，当上一层人工神经网络的反向运算执行完毕后，在运算指令的控制下，运算单元会将上一层人工神经网络的输出数据作为下一层人工神经网络的输入数据，但在反向运算中，上一层人工神经网络的输出数据为输入神经元梯度，下一层人工神经网络的输入数据为输出神经元梯度，即上一层人工神经网络的输入神经元梯度作为下一层人工神经网络的输出神经元梯度，或者是对上一层人工神经网络的输入神经元梯度进行层间操作，将操作结果作为下一层人工神经网络的输出神经元梯度。得到输出神经元梯度后，下一层人工神经网络进行反向运算，同时，将权值也替换为下一层人工神经网络的权值。
需要说明的是，在反向运算中，所述的“上一层”和“下一层”与正向运算相反，即对于人工神经网络的相邻两层：第一层与第二层，如果在 正向运算中，第一层是“上一层”，第二层是“下一层”，则在反向运算中，第一层是“下一层”，第二层是“上一层”。
权值更新步骤：接着，对多层人工神经网络中的各层进行权值更新，得到更新后的权值。
在执行完反向运算之后，就计算出了各层的权值梯度。在这个步骤中，对于人工神经网络的每一层，利用第一输入缓存和第二输入缓存分别存储该层的权值和权值梯度，然后在运算单元中利用权值梯度对权值进行更新，得到更新后的权值。
重复执行正向运算步骤、反向运算步骤和权值更新步骤多次，完成多层人工神经网络的训练。
整个训练方法需要多次重复执行上述过程，直至人工神经网络的参数达到要求，训练过程完毕。
举例来说，假设神经网络包括三层：输入层、中间层和输出层，其中：输入层各神经元负责接收来自外界的输入数据，并传递给中间层各神经元；中间层负责信息变换，根据信息变化能力的需求，中间层可以设计为单隐层或者多隐层结构，最后一个隐层传递到输出层各神经元的信息，经进一步处理后，完成一次学习的正向传播处理过程；输出层向外界输出信息处理结果。误差的反向传播阶段：误差通过输出层，按误差梯度下降的方式修正各层权值，向中间层、输入层逐层反传。
本发明中提到的输入神经元和输出神经元并非是指整个神经网络的输入层中神经元和输出层中神经元，而是对于网络中任意相邻的两层，处于网络前馈运算下层中的神经元即为输入神经元，处于网络前馈运算上层中的神经元即为输出神经元。以卷积神经网络为例，设一个卷积神经网络有L层，K＝1，2，...，L-1，对于第K层和第K+1层来说，我们将第K层称为输入层，其中的神经元为所述输入神经元，第K+1层称为输出层，其中的神经元为所述输出神经元。即除最顶层外，每一层都可以作为输入层，其下一层为对应的输出层。
上文中提到的运算都是神经网络中的一层的运算，对于多层神经网络，其实现过程是，在正向运算中，当上一层人工神经网络执行完成之后，下一层的运算指令会将运算单元中计算出的输出神经元作为下一层的输入 神经元进行运算(或者是对该输出神经元进行某些操作再作为下一层的输入神经元)，同时，将权值也替换为下一层的权值；在反向运算中，当上一层人工神经网络的反向运算执行完成后，下一层运算指令会将运算单元中计算出的输入神经元梯度作为下一层的输出神经元梯度进行运算(或者是对该输入神经元梯度进行某些操作再作为下一层的输出神经元梯度)，同时将权值替换为下一层的权值。如图17所示，图中虚线的箭头表示反向运算，实线的箭头表示正向运算。
需要说明的是，在训练过程中，反向运算所述的“上一层”和“下一层”与正向运算正好相反，即对于人工神经网络的相邻两层：第一层与第二层，如果在正向运算中，第一层是“上一层”，第二层是“下一层”，则在反向运算中，第一层是“下一层”，第二层是“上一层”。
需要说明的是，在附图或说明书正文中，未绘示或描述的实现方式，均为所属技术领域中普通技术人员所知的形式，并未进行详细说明。此外，上述对各元件的定义并不仅限于实施例中提到的各种具体结构、形状，本领域普通技术人员可对其进行简单地更改或替换；本文可提供包含特定值的参数的示范，但这些参数无需确切等于相应的值，而是可在可接受的误差容限或设计约束内近似于相应值；实施例中提到的方向用语，例如“上”、“下”、“前”、“后”、“左”、“右”等，仅是参考附图的方向，并非用来限制本公开的保护范围；上述实施例可基于设计及可靠度的考虑，彼此混合搭配使用或与其他实施例混合搭配使用，即不同实施例中的技术特征可以自由组合形成更多的实施例。
各功能单元/模块都可以是硬件，比如该硬件可以是电路，包括数字电路，模拟电路等等。硬件结构的物理实现包括但不局限于物理器件，物理器件包括但不局限于晶体管，忆阻器等等。所述计算装置中的计算模块可以是任何适当的硬件处理器，比如CPU、GPU、FPGA、DSP和ASIC等等。所述存储单元可以是任何适当的磁存储介质或者磁光存储介质，比如RRAM、DRAM、SRAM、EDRAM、HBM、HMC等等。
本公开实施例还提供一种计算机存储介质，其中，该计算机存储介质可存储有程序，该程序执行时包括上述方法实施例中记载的任何一种神经网络处理方法的部分或全部步骤。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本公开并不受所描述的动作顺序的限制，因为依据本公开，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本公开所必须的。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置，可通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储器中。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储器中，包括若干指令用以使得一台计算机设备(可为个人计算机、服务器或者网络设备等)执行本公开各个实施例所述方法的全部或部分步骤。而前述的存储器包括：U盘、只读存储器(ROM，Read-Only  Memory)、随机存取存储器(RAM，Random Access Memory)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，该程序可以存储于一计算机可读存储器中，存储器可以包括：闪存盘、只读存储器(英文：Read-Only Memory，简称：ROM)、随机存取器(英文：Random Access Memory，简称：RAM)、磁盘或光盘等。
以上对本公开实施例进行了详细介绍，本文中应用了具体个例对本公开的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本公开的方法及其核心思想；同时，对于本领域的一般技术人员，依据本公开的思想，在具体实施方式及应用范围上均会有改变之处，综上上述，本说明书内容不应理解为对本公开的限制。

Claims (15)

  1. 一种运算方法，其特征在于，使用人工神经网络计算装置对多层人工神经网络进行训练，所述人工神经网络计算装置包括：映射单元、存储单元和运算单元，所述运算方法包括：
    对多层人工神经网络中的各层依次执行正向运算，对于所述多层人工神经网络的每一层，所述正向运算包括：
    所述映射单元对输入数据进行处理，以得到处理后的输入数据；
    所述存储单元存储所述处理后的输入数据；
    所述运算单元读取所述处理后的输入数据，进行人工神经网络运算，得到人工神经网络的输出数据；
    按照与所述正向运算相反的顺序，对所述多层人工神经网络中的各层依次执行反向运算；
    对所述多层人工神经网络中的各层进行权值更新；
    重复执行上述各个步骤多次，完成多层人工神经网络的训练。
  2. 如权利要求1所述的运算方法，其特征在于，所述人工神经网络计算装置还包括：第一输入缓存和第二输入缓存；所述对多层人工神经网络中的各层进行权值更新包括：
    对于所述人工神经网络的每一层，所述第一输入缓存和所述第二输入缓存分别存储该层的权值和权值梯度；
    所述运算单元从所述第二输入缓存接收所述权值梯度，然后使用所述权值梯度对所述权值进行更新，得到更新后的权值。
  3. 根据权利要求1或2所述的运算方法，其特征在于，
    所述输入数据包括：输入神经元和权值；
    所述映射单元对输入数据进行处理，以得到处理后的输入数据包括：
    所述映射单元对权值进行处理，得到处理后的权值和权值的连接关系；
    所述映射单元基于所述权值的连接关系数据对输入神经元进行处理，得到处理后的输入神经元。
  4. 根据权利要求1或2所述的运算方法，其特征在于，所述输入数据包括输入神经元和权值；
    所述映射单元对输入数据进行处理，以得到处理后的输入数据包括：
    所述映射单元对输入神经元进行处理，得到处理后的输入神经元和输入神经元的连接关系；
    所述映射单元基于所述输入神经元的连接关系数据对权值进行处理，得到处理后的权值。
  5. 根据权利要求3所述的运算方法，其特征在于，所述映射单元对权值进行处理包括：
    所述映射单元将无连接的权值删除，保留有连接的权值作为处理后的权值。
  6. 根据权利要求3所述的运算方法，其特征在于，所述映射单元对权值进行处理包括：
    所述映射单元将所述权值中数值或绝对值小于或者等于第一阈值的权值删除，得到处理后的权值。
  7. 根据权利要求4所述的运算方法，其特征在于，所述映射单元对输入神经元进行处理包括：
    所述映射单元将所述输入神经元中数值或绝对值小于或者等于第二阈值的输入神经元删除，得到处理后的输入神经元。
  8. 根据权利要求1或2所述的运算方法，其特征在于，所述输入数据包括处理后的权值、输入神经元和所述输入神经元的连接关系数据；
    所述映射单元对输入数据进行处理，以得到处理后的输入数据包括；
    所述映射单元根据所述输入神经元的连接关系数据对所述输入神经元进行处理，以得到处理后的输入神经元。
  9. 根据权利要求1或2所述的运算方法，其特征在于，所述输入数据包括处理后的输入神经元、权值和所述权值的连接关系数据；
    所述映射单元对输入数据进行处理，以得到处理后的输入数据包括；
    所述映射单元根据所述权值的连接关系数据对所述权值进行处理，以得到处理后的权值。
  10. 根据权利要求1或2所述的运算方法，其特征在于，所述映射单元对输入数据进行处理，以得到处理后的输入数据包括：
    所述映射单元对至少一个输入神经元进行处理，得到输入神经元的连 接关系数据；对至少一个权值进行处理，得到权值的连接关系数据；
    所述映射单元基于输入神经元的连接关系数据和权值的的连接关系数据得到第三连接关系数据，并根据第三连接关系数据对输入神经元和权值进行处理，得到处理后的输入神经元和处理后的权值。
  11. 根据权利要求3-10任一项所述的运算方法，其特征在于，所述输入神经元的连接关系数据和所述权值的连接关系数据以直接索引或者步长索引的形式表示。
  12. 根据权利要求11所述的运算方法，其特征在于，
    当所述输入神经元的连接关系数据以直接索引的形式表示时，所述连接关系数据为由0和1组成的字符串，0表示所述输入神经元的值为零，1表示所述输入神经元的值非零；
    当所述输入神经元的连接关系数据以步长索引形式表示时，所述连接关系数据为值为非零的输入神经元与上一个值为非零的输入神经元之间的距离值组成的字符串。
  13. 根据权利要求11所述的运算方法，其特征在于，
    当所述权值的连接关系数据以直接索引的形式表示时，所述连接关系数据为由0和1组成的字符串，0表示所述权值为无连接的权值，1表示所述权值为有连接的权值；
    当所述权值的连接关系数据以步长索引的形式表示时，所述连接关系数据为与输出神经元有连接的输入神经元的与上一个与所述输出神经元有连接的输入神经元之间的距离值组成的字符串。
  14. 如权利要求1至13任一项所述的运算方法，其特征在于，对于多层人工神经网络的每一层，所述反向运算包括：权值梯度计算步骤和输入神经元梯度计算步骤；
    所述权值梯度计算步骤包括：
    所述运算单元由人工神经网络每一层的输出神经元梯度和输入神经元得到该层的权值梯度；
    所述输入神经元梯度计算步骤包括：
    所述映射单元将输出神经元梯度和处理后的权值存储至存储单元；
    所述运算单元读取输出神经元梯度和处理后的权值，进行人工神经网 络运算，得到输入神经元梯度。
  15. 如权利要求1至14任一项所述的运算方法，其特征在于，将人工神经网络上一层的输出数据作为下一层的输入数据；或者，对人工神经网络上一层的输出数据进行层间操作，将操作结果作为下一层的输入数据；其中，
    对于人工神经网络的任意相邻两层：第一层与第二层，在正向运算中，所述上一层指所述第一层，所述下一层指所述第二层；在反向运算中，所述上一层指所述第二层，所述下一层指所述第一层。
PCT/CN2017/118264 2016-12-28 2017-12-25 一种运算方法 Ceased WO2018121472A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP17886335.3A EP3564866A4 (en) 2016-12-28 2017-12-25 CALCULATION PROCEDURE
US16/455,347 US11308398B2 (en) 2016-12-28 2019-06-27 Computation method
US16/542,033 US11574195B2 (en) 2016-12-28 2019-08-15 Operation method
US16/542,051 US11568258B2 (en) 2016-12-28 2019-08-15 Operation method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201611234438 2016-12-28
CN201611234438.1 2016-12-28

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/455,347 Continuation-In-Part US11308398B2 (en) 2016-12-28 2019-06-27 Computation method

Publications (1)

Publication Number Publication Date
WO2018121472A1 true WO2018121472A1 (zh) 2018-07-05

Family

ID=62462713

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/118264 Ceased WO2018121472A1 (zh) 2016-12-28 2017-12-25 一种运算方法

Country Status (4)

Country Link
US (3) US11308398B2 (zh)
EP (1) EP3564866A4 (zh)
CN (3) CN108154228B (zh)
WO (1) WO2018121472A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109635939A (zh) * 2019-01-07 2019-04-16 北京邮电大学 一种基于裁剪的卷积神经网络的确定方法及装置

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108388446A (zh) 2018-02-05 2018-08-10 上海寒武纪信息科技有限公司 运算模块以及方法
CN108962230B (zh) * 2018-07-27 2019-04-23 重庆因普乐科技有限公司 基于忆阻器的语音识别方法
EP3605401A1 (en) * 2018-07-31 2020-02-05 GrAl Matter Labs S.A.S. Data processing module, data processing system and data processing method
CN108960418A (zh) * 2018-08-08 2018-12-07 上海寒武纪信息科技有限公司 神经网络处理装置及其执行矩阵向量乘指令的方法
US11367498B2 (en) * 2018-09-07 2022-06-21 Black Sesame Technologies Inc. Multi-level memory hierarchy
JP6986503B2 (ja) * 2018-09-10 2021-12-22 日立Astemo株式会社 電子制御装置、ニューラルネットワーク更新システム
CN109491956B (zh) * 2018-11-09 2021-04-23 北京灵汐科技有限公司 一种异构协同计算系统
CN111291884B (zh) * 2018-12-10 2024-08-20 中科寒武纪科技股份有限公司 神经网络剪枝方法、装置、电子设备及计算机可读介质
JP7189000B2 (ja) * 2018-12-12 2022-12-13 日立Astemo株式会社 情報処理装置、車載制御装置、車両制御システム
CN109800859B (zh) * 2018-12-25 2021-01-12 深圳云天励飞技术有限公司 一种神经网络批归一化的优化方法及装置
CN109697500B (zh) * 2018-12-29 2020-06-09 中科寒武纪科技股份有限公司 数据处理方法、装置、电子设备及存储介质
CN111382832A (zh) 2018-12-29 2020-07-07 佳能株式会社 多层神经网络模型的训练和应用方法、装置及存储介质
US12493774B2 (en) * 2019-03-26 2025-12-09 Shanghai Cambricon Information Technology Co., Ltd. Neural network operation module and method
US11257483B2 (en) * 2019-03-29 2022-02-22 Intel Corporation On-device neural network adaptation with binary mask learning for language understanding systems
CN110110850A (zh) * 2019-04-29 2019-08-09 山东浪潮人工智能研究院有限公司 基于fpga前向反向可复用的处理单元实现方法
CN110197260B (zh) * 2019-06-06 2020-10-02 百度在线网络技术（北京）有限公司 一种数据处理方法及装置
CN110364232B (zh) * 2019-07-08 2021-06-11 河海大学 一种基于忆阻器-梯度下降法神经网络的高性能混凝土强度预测方法
CN112580790B (zh) * 2019-09-29 2023-06-30 华为技术有限公司 神经网络计算电路、芯片及系统
US11537323B2 (en) 2020-01-07 2022-12-27 SK Hynix Inc. Processing-in-memory (PIM) device
US11422803B2 (en) * 2020-01-07 2022-08-23 SK Hynix Inc. Processing-in-memory (PIM) device
US12033070B2 (en) * 2020-06-12 2024-07-09 Huawei Technologies Co., Ltd. Low resource computational block for a trained neural network
JP7659373B2 (ja) * 2020-07-09 2025-04-09 キヤノン株式会社 畳み込みニューラルネットワーク処理装置
CN111831354B (zh) * 2020-07-09 2023-05-16 北京灵汐科技有限公司 数据精度配置方法、装置、芯片、芯片阵列、设备及介质
CN113704804B (zh) * 2021-10-27 2022-02-22 浙江数秦科技有限公司 一种基于异构神经网络模型的隐私计算方法
US11853558B2 (en) 2021-12-30 2023-12-26 Micron Technology, Inc. Power down workload estimation

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103455843A (zh) * 2013-08-16 2013-12-18 华中科技大学 一种反馈型人工神经网络训练方法及计算系统
CN104572504A (zh) * 2015-02-02 2015-04-29 浪潮(北京)电子信息产业有限公司 一种实现数据预读的方法及装置
CN104809426A (zh) * 2014-01-27 2015-07-29 日本电气株式会社 卷积神经网络的训练方法、目标识别方法及装置
CN104978601A (zh) * 2015-06-26 2015-10-14 深圳市腾讯计算机系统有限公司 神经网络模型训练系统和方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103499921B (zh) * 2013-09-11 2015-12-02 西安交通大学 变结构模糊系统传感器故障诊断方法
US10068170B2 (en) * 2013-09-23 2018-09-04 Oracle International Corporation Minimizing global error in an artificial neural network
CN104463324A (zh) * 2014-11-21 2015-03-25 长沙马沙电子科技有限公司 一种基于大规模高性能集群的卷积神经网络并行处理方法
CN105279554B (zh) * 2015-09-29 2017-12-15 东方网力科技股份有限公司 基于哈希编码层的深度神经网络的训练方法及装置
CN107578099B (zh) * 2016-01-20 2021-06-11 中科寒武纪科技股份有限公司 计算装置和方法
US20170364799A1 (en) * 2016-06-15 2017-12-21 Kneron Inc. Simplifying apparatus and simplifying method for neural network
CN106202946A (zh) * 2016-07-18 2016-12-07 燕山大学 基于深度信念网络模型的水泥熟料游离钙含量预测方法
US11887001B2 (en) * 2016-09-26 2024-01-30 Intel Corporation Method and apparatus for reducing the parameter density of a deep neural network (DNN)

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103455843A (zh) * 2013-08-16 2013-12-18 华中科技大学 一种反馈型人工神经网络训练方法及计算系统
CN104809426A (zh) * 2014-01-27 2015-07-29 日本电气株式会社 卷积神经网络的训练方法、目标识别方法及装置
CN104572504A (zh) * 2015-02-02 2015-04-29 浪潮(北京)电子信息产业有限公司 一种实现数据预读的方法及装置
CN104978601A (zh) * 2015-06-26 2015-10-14 深圳市腾讯计算机系统有限公司 神经网络模型训练系统和方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3564866A4 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109635939A (zh) * 2019-01-07 2019-04-16 北京邮电大学 一种基于裁剪的卷积神经网络的确定方法及装置
CN109635939B (zh) * 2019-01-07 2020-12-18 北京邮电大学 一种基于裁剪的卷积神经网络的确定方法及装置

Also Published As

Publication number Publication date
US20190318246A1 (en) 2019-10-17
CN108154228A (zh) 2018-06-12
US11568258B2 (en) 2023-01-31
CN108171323B (zh) 2021-03-26
US20190370664A1 (en) 2019-12-05
US11574195B2 (en) 2023-02-07
CN108154228B (zh) 2022-04-12
US20190370663A1 (en) 2019-12-05
EP3564866A4 (en) 2020-03-25
EP3564866A1 (en) 2019-11-06
US11308398B2 (en) 2022-04-19
CN108171323A (zh) 2018-06-15
CN111291878A (zh) 2020-06-16

Similar Documents

Publication Publication Date Title
CN108171323B (zh) 一种人工神经网络计算装置和方法
CN115511109B (zh) 一种高泛化性的个性化联邦学习实现方法
CN113393474B (zh) 一种基于特征融合的三维点云的分类和分割方法
US11645529B2 (en) Sparsifying neural network models
US20180260709A1 (en) Calculating device and method for a sparsely connected artificial neural network
CN112163601B (zh) 图像分类方法、系统、计算机设备及存储介质
CN113962358A (zh) 一种基于时序超图注意力神经网络的信息扩散预测方法
CN113449612B (zh) 一种基于子流型稀疏卷积的三维目标点云识别的方法
CN114821058A (zh) 一种图像语义分割方法、装置、电子设备及存储介质
CN111242268A (zh) 一种搜索卷积神经网络的方法
US20190311266A1 (en) Device and method for artificial neural network operation
CN115470889A (zh) 基于强化学习的片上网络自主最优映射探索系统及方法
CN116992151B (zh) 一种基于双塔图卷积神经网络的在线课程推荐方法
CN112633481A (zh) 一种多跳图卷积神经网络模型及其训练方法
US20220343162A1 (en) Method for structure learning and model compression for deep neural network
CN111353534B (zh) 一种基于自适应分数阶梯度的图数据类别预测方法
CN110119805A (zh) 基于回声状态网络分类的卷积神经网络算法
CN117150036A (zh) 一种多信息感知的知识图谱实体对齐方法
CN111325354A (zh) 机器学习模型压缩方法、装置、计算机设备和存储介质
CN117576737A (zh) 基于多连接生成对抗网络的掌纹图像扩充方法与装置
CN111382840B (zh) 一种面向自然语言处理的基于循环学习单元的htm设计方法
CN109711543B (zh) 一种可重构的深度置信网络实现系统
CN116186170A (zh) 三元组的实体确认方法、装置、计算机设备和存储介质
CN115564953A (zh) 图像分割方法、装置、设备及存储介质
Chen Attention augmented differentiable forest for tabular data

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17886335

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017886335

Country of ref document: EP

Effective date: 20190729