WO2019141902A1 - Appareil, procédé et programme informatique pour faire fonctionner un réseau neuronal - Google Patents

Appareil, procédé et programme informatique pour faire fonctionner un réseau neuronal Download PDF

Info

Publication number
WO2019141902A1
WO2019141902A1 PCT/FI2019/050028 FI2019050028W WO2019141902A1 WO 2019141902 A1 WO2019141902 A1 WO 2019141902A1 FI 2019050028 W FI2019050028 W FI 2019050028W WO 2019141902 A1 WO2019141902 A1 WO 2019141902A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural net
layer
weights
computational elements
analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/FI2019/050028
Other languages
English (en)
Inventor
Caglar AYTEKIN
Lixin Fan
Francesco Cricri
Emre Aksu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nokia Technologies Oy
Original Assignee
Nokia Technologies Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Technologies Oy filed Critical Nokia Technologies Oy
Publication of WO2019141902A1 publication Critical patent/WO2019141902A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5022—Mechanisms to release resources
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/70—Services for machine-to-machine communication [M2M] or machine type communication [MTC]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M1/00—Substation equipment, e.g. for use by subscribers
    • H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
    • H04M1/724—User interfaces specially adapted for cordless or mobile telephones
    • H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W88/00—Devices specially adapted for wireless communication networks, e.g. terminals, base stations or access point devices
    • H04W88/02—Terminal devices

Definitions

  • the present invention relates to an apparatus, a method and a computer program for running a neural network.
  • Neural networks are used more and more in various types of devices, from smartphones to self-driving cars. Many small devices are typically very constrained in terms of memory, bandwidth and computation capacity. On the other hand, many small devices are configured to run applications, which could benefit from running the application or part of it as NN-based algorithms.
  • pruning or compressing a neural network may often reduce the accuracy of the resulting model. Furthermore, in some applications, it may be important to have comparable results when the same neural network is run by devices with different memory, bandwidth and computation capabilities.
  • a method comprises obtaining, in a first apparatus, data to be analyzed by a neural net; receiving, from a second apparatus, a set of computational elements of the neural net needed for analysis; storing the set of computational elements of the neural net in a memory of the first apparatus; retrieving, upon carrying out calculations relating to said analysis, the set of computational elements of the neural net from the memory; and deleting, after carrying out calculations relating to said analysis requiring said set of computational elements of the neural net, the set of computational elements of the neural net from the memory of the first apparatus.
  • the set of computational elements of the neural net comprise topology information of at least one layer of the neural net and/or a set of weights for the neural net.
  • the method further comprises carrying out said calculations based on the data to be analyzed, the topology information of the at least one layer of the neural net and the set of weights for the neural net.
  • the data to be analyzed is input data of the neural net or an activation resulting from a previous computation in the neural net.
  • the method further comprises obtaining the weights in the same sequential order as the weights are used in said calculations relating to said analysis.
  • the method further comprises receiving a first stream comprising weights of a first layer of the neural net needed for said analysis; carrying out the calculations using the weights of said first layer; storing activations from said first layer to be used as an input for a subsequent layer; deleting said weights of said first layer from the memory; and repeating the above steps for any subsequent layer of the neural net needed for said analysis.
  • the method further comprises receiving a first stream comprising only a first convolutional filter of a first layer of the neural net needed for said analysis; carrying out the calculations using said first convolutional filter of said first layer; storing activations resulting from using said first convolutional filter to be used as an input for a subsequent calculation; deleting said using said first convolutional filter of said first layer from the memory; and repeating the above steps for any subsequent convolutional filter of said first layer of the neural net needed for said analysis.
  • An apparatus comprises means for obtaining data to be analyzed by a neural net; means for receiving a set of computational elements of the neural net needed for analysis; means for storing the set of computational elements of the neural net in a memory; means for retrieving, upon carrying out calculations relating to said analysis, the set of computational elements of the neural net from the memory; and means for deleting, after carrying out calculations relating to said analysis requiring said set of computational elements of the neural net, the set of computational elements of the neural net from the memory.
  • a third aspect relates to a method comprising: receiving, by a second apparatus, a request from a first apparatus to provide a set of computational elements of a neural net needed for an analysis; providing a first subset of the computational elements to the first apparatus; receiving, from the first apparatus, a request for at least a second subset of computational elements; and providing at least said second subset of the computational elements to the first apparatus.
  • An apparatus comprises means for receiving a request from a remote apparatus to provide a set of computational elements of a neural net needed for an analysis; means for providing a first subset of the computational elements to the remote apparatus; means for receiving, from the remote apparatus, a request for at least a second subset of computational elements; and means for providing at least said second subset of the computational elements to the remote apparatus.
  • Figure 1 shows schematically an electronic device employing embodiments of the invention
  • Figure 2 shows schematically a user equipment suitable for employing
  • Figure 3 further shows schematically electronic devices employing embodiments of the invention connected using wireless and wired network connections;
  • Figure 4 shows a flow chart of a method for running a neural network according to an embodiment of the invention
  • Figure 5 shows a block diagram for a layer- wise streaming of layer weights according to an embodiment of the invention.
  • Figure 6 shows a block diagram for a filter- wise streaming of layer weights according to an embodiment of the invention.
  • Figure 1 shows an example block diagram of an apparatus 50.
  • the apparatus may be an Internet of Things (IoT) apparatus configured to perform various functions, such as for example, gathering information by one or more sensors, receiving or transmitting information, analyzing information gathered or received by the apparatus, or the like.
  • the apparatus may comprise a video coding system, which may incorporate a codec.
  • Figure 2 shows a layout of an apparatus according to an example embodiment. The elements of Figs. 1 and 2 will be explained next.
  • the electronic device 50 may for example be a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or other lower power device.
  • a sensor device for example, a sensor device, a tag, or other lower power device.
  • embodiments of the invention may be implemented within any electronic device or apparatus which may process data by neural networks.
  • the apparatus 50 may comprise a housing 30 for incorporating and protecting the device.
  • the apparatus 50 further may comprise a display 32 in the form of a liquid crystal display.
  • the display may be any suitable display technology suitable to display an image or video.
  • the apparatus 50 may further comprise a keypad 34.
  • any suitable data or user interface mechanism may be employed.
  • the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.
  • the apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input.
  • the apparatus 50 may further comprise an audio output device which in embodiments of the invention may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection.
  • the apparatus 50 may also comprise a battery (or in other embodiments of the invention the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator).
  • the apparatus may further comprise a camera capable of recording or capturing images and/or video.
  • the apparatus 50 may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB/firewire wired connection.
  • the apparatus 50 may comprise a controller 56, processor or processor circuitry for controlling the apparatus 50.
  • the controller 56 may be connected to memory 58 which in embodiments of the invention may store both data in the form of image and audio data and/or may also store instructions for implementation on the controller 56.
  • the controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and decoding of audio and/or video data or assisting in coding and decoding carried out by the controller.
  • the apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.
  • a card reader 48 and a smart card 46 for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.
  • the apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network.
  • the apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and for receiving radio frequency signals from other apparatus(es).
  • the apparatus 50 may comprise a camera capable of recording or detecting individual frames which are then passed to the codec 54 or the controller for processing.
  • the apparatus may receive the video image data for processing from another device prior to transmission and/or storage.
  • the apparatus 50 may also receive either wirelessly or by a wired connection the image for coding/decoding.
  • the structural elements of apparatus 50 described above represent examples of means for performing a corresponding function.
  • the system 10 comprises multiple communication devices which can communicate through one or more networks.
  • the system 10 may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, CDMA, 4G, 5G network etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.
  • a wireless cellular telephone network such as a GSM, UMTS, CDMA, 4G, 5G network etc.
  • WLAN wireless local area network
  • the system 10 may include both wired and wireless communication devices and/or apparatus 50 suitable for implementing embodiments of the invention.
  • the system shown in Figure 3 shows a mobile telephone network 11 and a representation of the internet 28.
  • Connectivity to the internet 28 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.
  • the example communication devices shown in the system 10 may include, but are not limited to, an electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22.
  • PDA personal digital assistant
  • IMD integrated messaging device
  • the apparatus 50 may be stationary or mobile when carried by an individual who is moving.
  • the apparatus 50 may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle or any similar suitable mode of transport.
  • the embodiments may also be implemented in a set-top box; i.e. a digital TV receiver, which may/may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and/or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and/or embedded systems offering hardware/software based coding.
  • a set-top box i.e. a digital TV receiver, which may/may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and/or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and/or embedded systems offering hardware/software based coding.
  • Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24.
  • the base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 11 and the internet 28.
  • the system may include additional communication devices and communication devices of various types.
  • the communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11 and any similar wireless communication technology.
  • CDMA code division multiple access
  • GSM global systems for mobile communications
  • UMTS universal mobile telecommunications system
  • TDMA time divisional multiple access
  • FDMA frequency division multiple access
  • TCP-IP transmission control protocol-internet protocol
  • SMS short messaging service
  • MMS multimedia messaging service
  • email instant messaging service
  • IMS instant messaging service
  • Bluetooth IEEE 802.11 and any similar wireless communication technology
  • communications device involved in implementing various embodiments of the present invention may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.
  • a channel may refer either to a physical channel or to a logical channel.
  • a physical channel may refer to a physical transmission medium such as a wire
  • a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels.
  • a channel may be used for conveying an information signal, for example a bitstream, from one or several senders (or transmitters) to one or several receivers.
  • the embodiments may also be implemented in so-called IoT devices.
  • the Internet of Things may be defined, for example, as an interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure.
  • the convergence of various technologies has and will enable many fields of embedded systems, such as wireless sensor networks, control systems, home/building automation, etc. to be included the Internet of Things (IoT).
  • IoT devices are provided with an IP address as a unique identifier.
  • IoT devices may be provided with a radio transmitter, such as WLAN or Bluetooth transmitter or a RFID tag.
  • IoT devices may have access to an IP- based network via a wired network, such as an Ethernet-based network or a power-line connection (PLC).
  • PLC power-line connection
  • NN artificial neural network
  • Artificial neural networks are parametric computation graphs consisting of units and connections.
  • the units may be arranged in successive layers, and in some neural network architectures only units in adjacent layers are connected.
  • Each connection has an associated parameter or weight, which defines the strength of the connection. The weight gets multiplied by the incoming signal in that connection.
  • each unit in a layer is connected to each unit in the following layer. So, the signal which is output by a certain unit gets multiplied by the connections connecting that unit to another unit in the following layer. The latter unit then may perform a simple operation such as a sum of the weighted signals.
  • the input layer receives the input data, such as images, and the output layer is task- specific and outputs an estimate of the desired data, for example a vector whose values represent a class distribution in the case of image classification.
  • The“quality” of the neural network’s output is evaluated by comparing it to ground-truth output data. This comparison would then provide a“loss” or“cost” function.
  • the weights of the connections represent the biggest part of the leamable parameters of a neural network.
  • Other leamable parameters may be for example the parameters of the batch-normalization layer.
  • the parameters are learned by means of a training algorithm, where the goal is to minimize the loss function on a training dataset.
  • the training dataset is regarded as a representative sample of the whole data.
  • One popular learning approach is based on iterative local methods, where the loss is minimized by following the negative gradient direction.
  • the gradient is understood to be the gradient of the loss with respect to the weights of the neural network.
  • the loss is represented by the reconstructed prediction error.
  • Computing the gradient on the whole dataset may be computationally too heavy, thus learning is performed in sub-steps, where at each step a mini-batch of data is sampled and gradients are computed from the mini-batch. This is regarded to as stochastic gradient descent.
  • the gradients are usually computed by back-propagation algorithm, where errors are propagated from the output layer to the input layer, by using the chain rule for differentiation. If the loss function or some components of the neural network are not differentiable, it is still possible to estimate the gradient of the loss by using policy gradient methods, such as those used in reinforcement learning.
  • the computed gradients are then used by one of the available optimization routines (such as stochastic gradient descent, Adam, RMSprop, etc.), to compute a weight update, which is then applied to update the weights of the network. After a full pass over the training dataset, the process is repeated several times until a convergence criterion is met, usually a generalization criterion.
  • the gradients of the loss i.e., the gradients of the reconstructed prediction error with respect to the weights of the neural network, may be referred to as the training signal.
  • Online training consists of learning the parameters of the neural network
  • neural networks In the neural networks described above, a large memory storage is typically needed for the weights of the neural network. Moreover, the number of operations to be performed is also very high. [0051] Some neural networks, for example feedforward neural networks, which represent the vast majority of neural networks, can operate in a sequential manner, where each layer takes input from a single previous layer and gives output to a single subsequent layer. In the case of a convolution layer consisting of several convolution filters, for the calculations within a layer, operations are conducted separately for each filter in the layer, i.e.,
  • Neural networks are used more and more in various types of devices, from smartphones to self-driving cars.
  • One very important category of devices is represented by very small devices such as IoT devices.
  • IoT devices Many IoT devices, such as the ones mentioned above, are typically very constrained in terms of memory, bandwidth and computation capacity.
  • many IoT devices are configured to run applications, which could benefit from running the application or part of it as NN-based algorithms.
  • An example of such an application is object recognition in IoT devices which are capable of media acquisition, such as IoT devices provided with a camera.
  • the above problem may be alleviated via compressing or quantizing or pruning the neural network weights. Performing these operations on the pre-trained original network results into a new network that requires less memory and less number of computational operations. Therefore, it may be easier to send and run the neural network in bandwidth, memory and computation limited devices.
  • pruning or compressing a neural network may often reduce the accuracy of the resulting model.
  • the method comprises obtaining (400), in a first apparatus implementing a neural net, data to be analyzed by the neural net; receiving (402), from a second apparatus, a set of computational elements of the neural net needed for analysis; storing (404) the set of computational elements of the neural net in a memory of the first apparatus; retrieving (406), upon carrying out calculations relating to said analysis, the set of computational elements of the neural net from the memory; and deleting (408), after carrying out calculations relating to said analysis requiring said set of
  • computational elements of the neural net the set of computational elements of the neural net from the memory of the first apparatus.
  • the method may further comprise storing the results of the calculations relating to said analysis requiring said set of computations elements of the neural net.
  • the set of computational elements may comprise a subset of computational elements of the neural network, for example, computational elements corresponding to one layer of the neural network.
  • a computational element may comprise a numerical value and/or a mathematical operation associated with a node or neuron of the neural network.
  • the method is based on a simultaneous transmission/reception and running of only subsets of neural networks, whereupon the benefits are most prominent in computation, bandwidth and memory limited devices. Accordingly, only a necessary set of computational elements of a neural network is sent to the device running a neural net-based algorithm.
  • the device may not permanently store the set of computational elements, but may only store them temporarily, for example in a cache, and as soon as related computations due to the set of computational elements are completed, the device releases (e.g. deletes) the set of computational elements.
  • the approach enables efficient implementation of neural networks without compressing them in computation, bandwidth and memory limited devices, thus being able to retain the original accuracy of the model.
  • the sequential transfer of the computational elements may be combined with compressing the computational elements before the transfer.
  • the first apparatus running the neural net-based algorithm may also be referred to as“a layer user device”.
  • the second apparatus sending the necessary sets of computational elements to the first device may also be referred to as“a layer provider device”.
  • the first apparatus comprises a communication module allowing it to communicate with one or more other devices, at least with said second apparatus (“the layer provider device”).
  • the first apparatus is configured to perform an analysis of data using a neural network.
  • the data to be analyzed is input data (e.g., gathered from a data source) of the neural net or a set of neural network activations resulting from a previous computation in a subset of the neural net.
  • the first apparatus may be capable of capturing the input data, or input data may be streamed to the first apparatus from a separate data source.
  • the data to be analyzed may be a result of a previous calculation of a subset of the neural network, such as a set of activations from a previous layer of the neural network.
  • the set of computational elements of the neural net comprise topology information of at least one layer of the neural net and a set of weights related to that one layer for the neural net.
  • the first apparatus is configured to store a layer, which consists of both topology information and weights information.
  • the first apparatus is also configured to store either the input data or the activations computed at a previous computation step.
  • the computation step refers to the step during which one set of received computational elements (for example one layer) is run on data (either input data or previous activations).
  • said calculations are carried out based on the data to be analyzed, the topology information of the at least one layer of the neural net and the set of weights for the neural net. Consequently, the first apparatus is configured to perform computation of a neural network layer, given the data or previous activations, and the layer’s topology and weights.
  • the first apparatus obtains the weights in the same sequential order as the weights are used in said calculations relating to said analysis.
  • the second apparatus sends, for example streams, the weights necessary at a particular stage of the neural net-based algorithm to the first apparatus.
  • the weights may be streamed in the same sequential order that they are used in the network.
  • the sequential order is determined layer-wise or filter-wise.
  • the method upon streaming the weights layer-wise, further comprises receiving a first stream comprising weights of a first layer of the neural net needed for said analysis; carrying out the calculations using the weights of said first layer; storing activations from said first layer to be used as an input for a subsequent layer; deleting said weights of said first layer from the memory; and repeating the above steps for any subsequent layer of the neural net needed for said analysis.
  • a layer-wise streaming of neural networks may be applied such that in each stream, one or more (network) layers comprising topology and weights are sent to the first apparatus and the first apparatus stores the topology and the weights only temporarily, for example in a cache, to maintain the topology and the weights during the computation and releases the topology and the weights as soon as the related computations are completed.
  • the first apparatus then stores the activations and, if the computation is not completed, waits for the next stream to provide the next layer’s weights to be stored in the cache. The process is repeated until the entire computation of the neural network is completed.
  • topology information may be predetermined or signalled beforehand to the layer user apparatus.
  • the stream may include the weight information without topology information.
  • the layer user device may store activations of the layers that are going to be used later due to connections from earlier layer.
  • the network may use activation output of layer K (K ⁇ N).
  • topology information can be sent beforehand, or the server can only send another signal when sending the layer K in order to let the layer user device know that this layer activation will be used in layer N, so that the layer user device may avoid deleting those activations after implementation of layer K+l.
  • FIG. 5 illustrates a simplified block chart for implementing caching when performing the layer-wise streaming.
  • the second apparatus (“layer provider device”) 500 sends the topology of the current layer and its weights to the first apparatus (“layer user device”) 510, wherein the layer’s topology and weights are stored in a first cache 520.
  • the topology information of the layer and the weights may be transmitted either together or separately.
  • the topology information of the layer and the weights may be sent as pushed by the second apparatus 500 or after an initial first
  • the first apparatus comprises a layer assembler 530, which assembles the layer needed for the computation by using the topology and weights.
  • the topology may refer, for example, to the type of the layer, the number of neurons, the number of filters (in case of a convolution layer), or the type of non-linearity.
  • the assembler may create the computation graph defined by the topology and weights, and which is then run on either the input data or the previous activations.
  • the layer computation block 540 carries out the actual execution of the layer on given the input controlled by a switch 550.
  • the input may be either the data from a data source or activations output at a previous computation step.
  • the previous activations may be temporarily stored in a second cache 560.
  • the method further comprises receiving a first stream comprising only a first
  • convolutional filter of a first layer of the neural net needed for said analysis carrying out the calculations using said first convolutional filter of said first layer; storing activations resulting from using said first convolutional filter to be used as an input for a subsequent calculation; deleting said using said first convolutional filter of said first layer from the memory; and repeating the above steps for any subsequent convolutional filter of said first layer of the neural net needed for said analysis.
  • neural networks for example feed- forward neural networks
  • feed- forward neural networks during the computation of a layer, there often exists several computations that are independent of each other.
  • One example is multiple convolutions to be performed on the input separately. If parallel processing units, such as GPUs, are not available, these computations are performed at different times. Hence, it is not necessary to store filters in a layer other than the one that is used for the current computation.
  • only one convolutional filter is sent to the first apparatus and the first apparatus stores the first filter only temporarily, for example in a cache, to maintain the first filter during the computation and releases the filter.
  • the first apparatus then obtains a subsequent filter from the next stream, wherein the first and the subsequent filters are related to the same layer.
  • the second apparatus may include a variable into the stream to indicate the layer of the convolutional filter so that the first apparatus knows the input of the convolutional filter. The process is repeated until all filters are sent and the computation is completed.
  • FIG. 6 illustrates a simplified block chart for implementing caching when performing the filter-wise streaming.
  • the second apparatus (herein referred to as“filter provider device”) 600 sends the layer’s topology and the weights of the current feature to the first apparatus (“layer user device”) 610, wherein the layer’s topology and weights of the current feature are stored in a first cache 620.
  • the topology may refer, for example, to the type of the layer, the number of neurons, the number of filters (in case of a convolution layer), or the type of non-linearity.
  • computation block 630 carries out the actual execution of the feature on given the input controlled by a switch 640.
  • the input may be either the data from a data source or activations output at a previous computation step.
  • the previous activations may be temporarily stored in a second cache 650.
  • an output assembler 660 assembles the filter outputs (cached in the second cache 660) from said layer according to the layer topology (cached in the first cache 620).
  • both the layer-wise and the filter-wise streaming of weights and the related caching schemes provide the benefit of allowing dynamic configuration of neural network architectures, which is an advantageous feature for deep learning optimization.
  • the second apparatus provides the first apparatus (“layer user device”) with the topology information of network layer.
  • the topology information comprises a topology layer type and a description for the entire network.
  • the second apparatus may send the whole topology information in advance.
  • the signalling may be implemented, for example, as follows:
  • Topology Layer Type Signal (C-C-C-C-D), wherein four consecutive Convolutional layers (C) are followed by a dense layer (D).
  • the signaling may be (0,0, 0,0,1)
  • the topology information comprises a topology layer type and a description for one layer of the network.
  • the second apparatus may send the topology layer type and the description for each layer sequentially.
  • the first signalling may include, for example, one or more of the following:
  • the first apparatus may request for either layer- wise or feature- wise streaming with a request signal to the second apparatus.
  • the request signal may be, for example, 1 for layer wise 0 for filter wise.
  • the request signal may indicate a type of neural network streaming. It is possible that the first apparatus first evaluates its capability regarding the network topology informed in the first signaling, and only then decides whether to request for a layer- wise or a feature-wise streaming or neither.
  • the second apparatus sends the first layer, where the layer signal includes the weights of the first layer.
  • the first apparatus Based on the initially sent topology, the first apparatus knows what the numbers in the stream mean. For example, for 16 filters, 3x3 filter size example, the first 9 (3x3) numbers in the stream corresponds to the first filter.
  • the first apparatus After the computations related to the layer is completed, the first apparatus sends a next request signal to obtain weights for a subsequent layer. This process continues until the complete execution of the network.
  • the first apparatus may conclude that there are no more layers or filters to execute by one of the following alternatives:
  • the first apparatus has sent a request signal and in return it receives a stop signal comprising an indication that no more layers or filters will be provided.
  • the stream may also include a signaling filed indicating that it is the last layer to execute.
  • the first apparatus provides the final network output to its client, which may be a communication module, a storage device, a UI output device, etc.
  • the second apparatus may send the first filter, where the filter signal includes the weights of the first filter.
  • the first apparatus Based on the initially sent topology, the first apparatus knows how to convert the 1D stream to 2D filters. After the computations related to the feature is completed, the first apparatus sends a request signal to obtain a subsequent filter. This process continues until the complete execution of the network.
  • the computational elements may be packetized or grouped in the stream such that the layer user device is able to distinguish different computational elements from each other. For example a layer identifier and/or a filter identifier may be associated with a group of computational elements.
  • the topology information may indicate the layer identifiers or filter identifier belonging to the neural network. Based on this information a layer user device may determine if necessary computational elements have been received and/or whether they have been received in the correct order.
  • the operation of the second apparatus may be defined by a method comprising: receiving, by a second apparatus, a request from a first apparatus to provide a subset of computational elements of a neural net; providing a first subset of the computational elements to the first apparatus; receiving, from the first apparatus a request for at least a second subset of computational elements; and providing at least said second subset of the computational elements to the first apparatus.
  • the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof.
  • some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto.
  • firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto.
  • While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
  • the embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware.
  • any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions.
  • the software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.
  • the memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.
  • the data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multi-core processor architecture, as non-limiting examples.
  • Embodiments of the inventions may be practiced in various components such as integrated circuit modules.
  • the design of integrated circuits is by and large a highly automated process.
  • Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
  • Programs such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules.
  • the resultant design in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Neurology (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un procédé consistant à : obtenir, dans un premier appareil mettant en oeuvre un réseau neuronal, des données à analyser par le réseau neuronal ; recevoir, en provenance d'un deuxième appareil, un ensemble d'éléments de calcul du réseau neuronal nécessaire à l'analyse ; stocker l'ensemble d'éléments de calcul du réseau neuronal dans une mémoire du premier appareil ; récupérer, lors de la réalisation de calculs relatifs à ladite analyse, l'ensemble d'éléments de calcul du réseau neuronal dans la mémoire ; et supprimer, après la réalisation des calculs relatifs à ladite analyse nécessitant ledit ensemble d'éléments de calcul du réseau neuronal, l'ensemble d'éléments de calcul du réseau neuronal de la mémoire du premier appareil.
PCT/FI2019/050028 2018-01-17 2019-01-15 Appareil, procédé et programme informatique pour faire fonctionner un réseau neuronal Ceased WO2019141902A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FI20185041 2018-01-17
FI20185041 2018-01-17

Publications (1)

Publication Number Publication Date
WO2019141902A1 true WO2019141902A1 (fr) 2019-07-25

Family

ID=67302029

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2019/050028 Ceased WO2019141902A1 (fr) 2018-01-17 2019-01-15 Appareil, procédé et programme informatique pour faire fonctionner un réseau neuronal

Country Status (1)

Country Link
WO (1) WO2019141902A1 (fr)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110490313A (zh) * 2019-08-14 2019-11-22 北京中科寒武纪科技有限公司 一种内存复用方法及其相关产品
US20210049451A1 (en) * 2019-08-14 2021-02-18 Google Llc Communicating a Neural Network Formation Configuration
CN112446463A (zh) * 2019-08-31 2021-03-05 安徽寒武纪信息科技有限公司 一种神经网络全连接层运算方法、装置以及相关产品
US20210342687A1 (en) * 2019-08-14 2021-11-04 Google Llc Base Station-User Equipment Messaging Regarding Deep Neural Networks
CN114692824A (zh) * 2020-12-31 2022-07-01 安徽寒武纪信息科技有限公司 一种神经网络模型的量化训练方法、装置和设备
US11397893B2 (en) 2019-09-04 2022-07-26 Google Llc Neural network formation configuration feedback for wireless communications
US11663472B2 (en) 2020-06-29 2023-05-30 Google Llc Deep neural network processing for a user equipment-coordination set
US11689940B2 (en) 2019-12-13 2023-06-27 Google Llc Machine-learning architectures for simultaneous connection to multiple carriers
US11886991B2 (en) 2019-11-27 2024-01-30 Google Llc Machine-learning architectures for broadcast and multicast communications
US11934337B2 (en) 2019-08-31 2024-03-19 Anhui Cambricon Information Technology Co., Ltd. Chip and multi-chip system as well as electronic device and data transmission method
US12075346B2 (en) 2019-10-31 2024-08-27 Google Llc Determining a machine-learning architecture for network slicing

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120011087A1 (en) * 2010-07-07 2012-01-12 Qualcomm Incorporated Methods and systems for replaceable synaptic weight storage in neuro-processors
US20160379111A1 (en) * 2015-06-25 2016-12-29 Microsoft Technology Licensing, Llc Memory bandwidth management for deep learning applications
WO2017124644A1 (fr) * 2016-01-20 2017-07-27 北京中科寒武纪科技有限公司 Dispositif et procédé de codage par compression de réseau neuronal artificiel
US20170221176A1 (en) * 2016-01-29 2017-08-03 Fotonation Limited Convolutional neural network

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120011087A1 (en) * 2010-07-07 2012-01-12 Qualcomm Incorporated Methods and systems for replaceable synaptic weight storage in neuro-processors
US20160379111A1 (en) * 2015-06-25 2016-12-29 Microsoft Technology Licensing, Llc Memory bandwidth management for deep learning applications
WO2017124644A1 (fr) * 2016-01-20 2017-07-27 北京中科寒武纪科技有限公司 Dispositif et procédé de codage par compression de réseau neuronal artificiel
US20170221176A1 (en) * 2016-01-29 2017-08-03 Fotonation Limited Convolutional neural network

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
AYTEKIN, C. ET AL.: "Memory-Efficient Deep Salient Object Segmentation Networks on Gridized Superpixels", ARXIV.ORG, 27 December 2017 (2017-12-27), Ithaca, US, pages 1 - 6, XP080848558 *
ZOU, S. ET AL.: "Distributed Training Large-Scale Deep Architectures", INTERNATIONAL CONFERENCE ON COMPUTER ANALYSIS OF IMAGES AND PATTERNS. CAIP 2017: COMPUTER ANALYSIS OF IMAGES AND PATTERNS, vol. 10604 Chap.2, no. 558, 10 August 2017 (2017-08-10) - 14 October 2017 (2017-10-14), Ithaca, US, pages 18 - 32, XP047452333 *

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11928587B2 (en) 2019-08-14 2024-03-12 Google Llc Base station-user equipment messaging regarding deep neural networks
US20210049451A1 (en) * 2019-08-14 2021-02-18 Google Llc Communicating a Neural Network Formation Configuration
US20210342687A1 (en) * 2019-08-14 2021-11-04 Google Llc Base Station-User Equipment Messaging Regarding Deep Neural Networks
US12001943B2 (en) 2019-08-14 2024-06-04 Google Llc Communicating a neural network formation configuration
CN110490313A (zh) * 2019-08-14 2019-11-22 北京中科寒武纪科技有限公司 一种内存复用方法及其相关产品
CN112446463A (zh) * 2019-08-31 2021-03-05 安徽寒武纪信息科技有限公司 一种神经网络全连接层运算方法、装置以及相关产品
CN112446463B (zh) * 2019-08-31 2023-06-02 安徽寒武纪信息科技有限公司 一种神经网络全连接层运算方法、装置以及相关产品
US11934337B2 (en) 2019-08-31 2024-03-19 Anhui Cambricon Information Technology Co., Ltd. Chip and multi-chip system as well as electronic device and data transmission method
US11397893B2 (en) 2019-09-04 2022-07-26 Google Llc Neural network formation configuration feedback for wireless communications
US12075346B2 (en) 2019-10-31 2024-08-27 Google Llc Determining a machine-learning architecture for network slicing
US11886991B2 (en) 2019-11-27 2024-01-30 Google Llc Machine-learning architectures for broadcast and multicast communications
US12236347B2 (en) 2019-11-27 2025-02-25 Google Llc Machine-learning architectures for broadcast and multicast communications
US11689940B2 (en) 2019-12-13 2023-06-27 Google Llc Machine-learning architectures for simultaneous connection to multiple carriers
US11663472B2 (en) 2020-06-29 2023-05-30 Google Llc Deep neural network processing for a user equipment-coordination set
CN114692824A (zh) * 2020-12-31 2022-07-01 安徽寒武纪信息科技有限公司 一种神经网络模型的量化训练方法、装置和设备

Similar Documents

Publication Publication Date Title
WO2019141902A1 (fr) Appareil, procédé et programme informatique pour faire fonctionner un réseau neuronal
Zhou et al. Edge intelligence: Paving the last mile of artificial intelligence with edge computing
US11915144B2 (en) Apparatus, a method and a computer program for running a neural network
US12536990B2 (en) Model training method, speech recognition method, device, medium, and apparatus
WO2020183059A1 (fr) Appareil, procédé et programme d'ordinateur pour l'apprentissage d'un réseau neuronal
CN114418121B (zh) 模型训练方法、对象处理方法及装置、电子设备、介质
CN119150862B (zh) 模型微调方法、文本处理方法、介质、设备及程序产品
CN111656365A (zh) 网络结构搜索的方法及装置、计算机存储介质和计算机程序产品
CN113610228B (zh) 神经网络模型的构建方法及装置
CN113327599A (zh) 语音识别方法、装置、介质及电子设备
CN116662411B (zh) 场景模板库的构建方法、对象预测方法及装置、电子设备
CN116310582A (zh) 分类模型训练方法、图像分类方法、装置、介质及设备
EP3683733B1 (fr) Procédé, appareil et produit-programme d'ordinateur pour réseaux neuronaux
CN116644783B (zh) 模型训练方法、对象处理方法及装置、电子设备、介质
CN115527526B (zh) 端到端远场语音识别系统训练方法、装置、计算机设备
CN116090543B (zh) 模型压缩方法及装置、计算机可读介质和电子设备
CN114746870A (zh) 用于神经网络压缩中优先级信令的高级语法
CN111582456B (zh) 用于生成网络模型信息的方法、装置、设备和介质
WO2019141905A1 (fr) Appareil, procédé et programme informatique pour faire fonctionner un réseau neuronal
WO2020008108A1 (fr) Appareil, procédé et programme informatique pour l'entraînement d'un réseau neuronal
CN118823686B (zh) 仓储物流自动配送系统及方法
CN118228255A (zh) 一种应用程序的风险检测方法、装置及设备
CN114330239B (zh) 文本处理方法及装置、存储介质及电子设备
CN114648712B (zh) 视频分类方法、装置、电子设备及计算机可读存储介质
CN115309935B (zh) 模型训练方法、音频分类方法、装置、介质及程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19741189

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19741189

Country of ref document: EP

Kind code of ref document: A1