WO2020008108A1 - Appareil, procédé et programme informatique pour l'entraînement d'un réseau neuronal - Google Patents

Appareil, procédé et programme informatique pour l'entraînement d'un réseau neuronal Download PDF

Info

Publication number
WO2020008108A1
WO2020008108A1 PCT/FI2019/050495 FI2019050495W WO2020008108A1 WO 2020008108 A1 WO2020008108 A1 WO 2020008108A1 FI 2019050495 W FI2019050495 W FI 2019050495W WO 2020008108 A1 WO2020008108 A1 WO 2020008108A1
Authority
WO
WIPO (PCT)
Prior art keywords
dimensional
vector
activation
neural network
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/FI2019/050495
Other languages
English (en)
Inventor
Caglar AYTEKIN
Francesco Cricri
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nokia Technologies Oy
Original Assignee
Nokia Technologies Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Technologies Oy filed Critical Nokia Technologies Oy
Publication of WO2020008108A1 publication Critical patent/WO2020008108A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Definitions

  • the present invention relates to an apparatus, a method and a computer program for running a neural network.
  • Binarization in neural networks has many applications such as weight binarization of neural networks, where the aim is to have neural networks of very small size. Another application is binarization of activations, i.e., binarization of the output of one or more layers of a neural network. This is particularly important for applications like image compression where the codes are aimed to be quantized/binarized for high compression rates.
  • binarization as a noise, where the binarization is considered as a noise process on top of the real- valued activation
  • binarization as a random variable where the binarization is considered as a random variable having a Bernoulli distribution. Both of these can be used at the training stage of a neural network.
  • a method comprises training or inferring a neural network comprising a plurality of layers to produce binarized activations and/or weights, said training or inferring comprising: mapping an N-dimensional vector to a sub-region of an N-dimensional space in a first layer; and binarizing an output vector of the first layer in a second layer.
  • the sub-region of the N-dimensional space comprises a surface at the N-dimensional space.
  • the surface comprises a hypersphere having radius of 0.5 and centre point located at 0.5 x 1, wherein 1 is an N-dimensional vector of ones .
  • the method further comprises calculating the mapping by
  • /(x) 0.5 + 0.5— - , where x is N-dimensional activation vector.
  • the method further comprises dividing the N- dimensional input vector into pairs of 2-dimensional activations; and mapping each 2- dimensional activation to a circle that intersects comers of a square having side length of one.
  • the method further comprises calculating the mapping of the 2-dimensional activation z by
  • each pair of the N-dimensional activation vectors is created from neighbouring/adjacent activations x £ and x i+1 within the activation vector.
  • the binarization of the output vector of the first layer comprises adding noise, and upon inferring the neural network, the binarization of the output vector of the first layer comprises a binarization operation.
  • a second aspect relates to an apparatus comprising: a plurality of neural network layers comprising: a first layer configured to map an N-dimensional vector to a sub-region of an N-dimensional space; and a second layer configured to binarize an output vector of the first layer.
  • the further aspects relate to apparatuses and computer readable storage media stored with code thereon, which are arranged to carry out the above methods and one or more of the embodiments related thereto.
  • Figure 1 shows schematically an electronic device employing embodiments of the invention
  • Figure 2 shows schematically a user equipment suitable for employing embodiments of the invention
  • FIG. 3 further shows schematically electronic devices employing embodiments of the invention connected using wireless and wired network connections;
  • Figure 4 shows a flow chart of a method for training a neural network according to an embodiment of the invention
  • Figures 5a, 5b and 5c show simplified examples of activation mappings according to embodiments of the invention.
  • Figure 6 shows a simplified example of mapping activations pairwise on a sphere according to an embodiment of the invention.
  • Figures 7a and 7b illustrate the operation in a training phase and in an inference phase, correspondingly, according to some embodiments of the invention.
  • Figure 1 shows an example block diagram of an apparatus 50.
  • the apparatus may be an Internet of Things (IoT) apparatus configured to perform various functions, such as for example, gathering information by one or more sensors, receiving or transmitting information, analyzing information gathered or received by the apparatus, or the like.
  • the apparatus may comprise a video coding system, which may incorporate a codec.
  • Figure 2 shows a layout of an apparatus according to an example embodiment. The elements of Figs. 1 and 2 will be explained next.
  • the electronic device 50 may for example be a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or other lower power device.
  • a sensor device for example, a sensor device, a tag, or other lower power device.
  • embodiments of the invention may be implemented within any electronic device or apparatus which may process data by neural networks.
  • the apparatus 50 may comprise a housing 30 for incorporating and protecting the device.
  • the apparatus 50 further may comprise a display 32 in the form of a liquid crystal display.
  • the display may be any suitable display technology suitable to display an image or video.
  • the apparatus 50 may further comprise a keypad 34.
  • any suitable data or user interface mechanism may be employed.
  • the user interface may be
  • the apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input.
  • the apparatus 50 may further comprise an audio output device which in embodiments of the invention may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection.
  • the apparatus 50 may also comprise a battery (or in other embodiments of the invention the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator).
  • the apparatus may further comprise a camera capable of recording or capturing images and/or video.
  • the apparatus 50 may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB/firewire wired connection.
  • the apparatus 50 may comprise a controller 56, processor or processor circuitry for controlling the apparatus 50.
  • the controller 56 may be connected to memory 58 which in embodiments of the invention may store both data in the form of image and audio data and/or may also store instructions for implementation on the controller 56.
  • the controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and/or decoding of audio and/or video data or assisting in coding and/or decoding carried out by the controller.
  • the apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.
  • the apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network.
  • the apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and/or for receiving radio frequency signals from other apparatus(es).
  • the apparatus 50 may comprise a camera capable of recording or detecting individual frames which are then passed to the codec 54 or the controller for processing.
  • the apparatus may receive the video image data for processing from another device prior to transmission and/or storage.
  • the apparatus 50 may also receive either wirelessly or by a wired connection the image for coding/decoding.
  • the structural elements of apparatus 50 described above represent examples of means for performing a corresponding function.
  • the system 10 comprises multiple communication devices which can communicate through one or more networks.
  • the system 10 may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, CDMA, 4G, 5G network etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.
  • a wireless cellular telephone network such as a GSM, UMTS, CDMA, 4G, 5G network etc.
  • WLAN wireless local area network
  • the system 10 may include both wired and wireless communication devices and/or apparatus 50 suitable for implementing embodiments of the invention.
  • the system shown in Figure 3 shows a mobile telephone network 11 and a representation of the internet 28.
  • Connectivity to the internet 28 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.
  • the example communication devices shown in the system 10 may include, but are not limited to, an electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22.
  • PDA personal digital assistant
  • IMD integrated messaging device
  • the apparatus 50 may be stationary or mobile when carried by an individual who is moving.
  • the apparatus 50 may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle or any similar suitable mode of transport.
  • the embodiments may also be implemented in a set-top box; i.e. a digital TV receiver, which may/may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and/or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and/or embedded systems offering hardware/software based coding.
  • a set-top box i.e. a digital TV receiver, which may/may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and/or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and/or embedded systems offering hardware/software based coding.
  • Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24.
  • the base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 11 and the internet 28.
  • the system may include additional communication devices and communication devices of various types.
  • the communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, 3GPP Narrowband IoT and any similar wireless communication technology.
  • CDMA code division multiple access
  • GSM global systems for mobile communications
  • UMTS universal mobile telecommunications system
  • TDMA time divisional multiple access
  • FDMA frequency division multiple access
  • TCP-IP transmission control protocol-internet protocol
  • SMS short messaging service
  • MMS multimedia messaging service
  • email instant messaging service
  • IMS instant messaging service
  • Bluetooth IEEE 802.11, 3GPP Narrowband IoT and any similar wireless communication technology.
  • a communications device involved in implementing various embodiments of the present invention may communicate using various media including, but not
  • a channel may refer either to a physical channel or to a logical channel.
  • a physical channel may refer to a physical transmission medium such as a wire
  • a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels.
  • a channel may be used for conveying an information signal, for example a bitstream, from one or several senders (or transmitters) to one or several receivers.
  • the embodiments may also be implemented in so-called IoT devices.
  • the Internet of Things may be defined, for example, as an interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure.
  • the convergence of various technologies has and will enable many fields of embedded systems, such as wireless sensor networks, control systems, home/building automation, etc. to be included the Internet of Things (IoT).
  • IoT devices In order to utilize Internet IoT devices are provided with an IP address as a unique identifier.
  • IoT devices may be provided with a radio transmitter, such as WLAN or Bluetooth transmitter or a RFID tag.
  • a radio transmitter such as WLAN or Bluetooth transmitter or a RFID tag.
  • IoT devices may have access to an IP-based network via a wired network, such as an Ethernet-based network or a power- line connection (PLC).
  • a wired network such as an Ethernet-based network or a power- line connection (PLC).
  • PLC power- line connection
  • computation graphs comprising units and connections.
  • the units may be arranged in successive layers, and in some neural network architectures only units in adjacent layers are connected.
  • Each connection has an associated parameter or weight, which defines the strength of the connection. The weight gets multiplied by the incoming signal in that connection.
  • each unit in a layer is connected to each unit in the following layer. So, the signal which is output by a certain unit gets multiplied by the connections connecting that unit to another unit in the following layer. The latter unit then may perform a simple operation such as a sum of the weighted signals.
  • the input layer receives the input data, such as images, and the output layer is task-specific and outputs an estimate of the desired data, for example a vector whose values represent a class distribution in the case of image classification.
  • The“quality” of the neural network’s output is evaluated by comparing it to ground-truth output data.
  • the comparison may include a loss or cost function, run on the neural network’s output and the ground-truth data. This comparison would then provide a“loss” or“cost” value.
  • weights of the connections represent the biggest part of the leamable parameters of a neural network.
  • model“ and“neural network” are used interchangeably, as well as the weights of neural networks are sometimes referred to as leamable parameters or simply as parameters.
  • the parameters are learned by means of a training algorithm, where the goal is to minimize the loss value on a training dataset and on a held-out validation dataset.
  • the network is run on a training dataset, a loss value is computed for the whole training dataset or for part of it, and the leamable parameters are modified in order to minimize the loss value on the training dataset.
  • the training dataset is regarded as a representative sample of the whole data.
  • One learning approach is based on iterative local methods, where the loss on the training dataset is minimized by following the negative gradient direction.
  • the gradient is understood to be the gradient of the loss with respect to the leamable parameters of the neural network.
  • the loss may be represented by the reconstructed prediction error.
  • Computing the gradient on the whole training dataset may be computationally too heavy, thus learning is performed in sub-steps, where at each step a mini-batch of data is sampled and gradients are computed from the mini-batch. This is referred to as stochastic gradient descent.
  • the gradients are usually computed by back-propagation algorithm, where errors are propagated from the output layer to the input layer, by using the chain rule for differentiation. If the loss function or some operations performed by the neural network are not differentiable, it is still possible to estimate the gradient of the loss by using policy gradient methods, such as those used in reinforcement learning.
  • the computed gradients are then used by one of the available optimization routines (such as stochastic gradient descent, Adam, RMSprop, etc.), to compute a weight update, which is then applied to update the weights of the network.
  • a convergence criterion is met, usually a generalization criterion.
  • a generalization criterion may be derived from the loss value on the held-out validation dataset, for example by stopping the training when the loss value on the held-out validation dataset is less than a certain threshold.
  • the gradients of the loss i.e., the gradients of the reconstructed prediction error with respect to the weights of the neural network, may be referred to as the training signal.
  • Training a neural network is an optimization process, but as a difference to a typical optimization where the only goal is to minimize a function, the goal of the optimization or training process in machine learning is to make the model to leam the properties of the data distribution.
  • the goal is to leam to generalize to previously unseen data, i.e., data which was not used for training the model. This is usually referred to as generalization.
  • data is usually split into two (or more) sets, the training set and the validation set.
  • the training set is used for training the network, i.e., to modify its leamable parameters to minimize the loss.
  • the validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the model.
  • the errors on the training set and on the validation set are monitored during the training process to understand the following issues:
  • the training set error should decrease, otherwise we are in the regime of underfitting.
  • the validation set error needs to decrease and to be not too much higher than the training set error. If the training set error is low, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the model is in the regime of overfitting. This means that the model has just memorized the training set’s properties and performs well only on that set, but performs poorly on a set not used for tuning its parameters.
  • Binarization in neural networks has many applications such as weight binarization of neural networks, where the aim is to have neural networks of very small size. Another application is binarization of activations, i.e., binarization of the output of one or more layers of a neural network. This is particularly important for applications like image compression where the codes are aimed to be quantized/binarized for high compression rates.
  • Binarization as a Noise [0054] This approach considers the binarization as a noise process on top of the real valued activation. For example, the binarization of the value 0.7 outputs the value 1. This can be considered as adding a 0.3 valued noise on top of the actual activation.
  • a neural network can be trained to be robust within a certain noise interval where the actual binarization is also included. This is achieved as follows. During forward propagation of the training process, the neural network prediction is evaluated with the added noise on the activation to be binarized. So, the addition of noise during training simulates the real binarization operation.
  • noise may be added such that the i-th activation x_i is mapped to an interval between [0,x_i ] if x is closer to 0 than 1, otherwise it is mapped to an interval between [x_i,l]
  • This approach considers binarization as a random variable having a Bernoulli distribution.
  • the distribution involves a variable p such that with probability p the distribution generates number 1 and with probability (l-p), the distribution generates number 0.
  • variable p is directly selected as the real valued activation. For example, if the i-th activation’s value is 0.8, then with 0.8 probability it is mapped to 1, and with 0.2 probability it is mapped to 0.
  • An example of a method which is depicted in the flow chart of Figure 4, comprises training or inferring (400) a neural network comprising a plurality of layers to produce binarized activations and/or weights, said training or inferring comprising:
  • mapping an N-dimensional vector to a sub-region of an N-dimensional space in a first layer
  • binarizing an output vector of the first layer in a second layer.
  • inference stage a.k.a. testing stage, or utilization stage
  • a given task e.g. image or video compression.
  • the noise to be simulated by the neural network is more suitable to leam binarization approximation better.
  • the activations do not need to be outputs of sigmoid function.
  • a further advantage is obtained in the case where the binarization simulation according to the embodiments is used at the end of an encoder neural network, where the encoder is part of an auto-encoder network trained end-to-end to compress data.
  • the decoder is trained on activations and/or weights which are more suitable for approximation of binarization.
  • the sub-region of the N-dimensional space comprises a surface at the N-dimensional space.
  • the learning to approximate binarization is more effective and the noise to be simulated by the neural network is more suitable.
  • the M layers may be the layers of a neural auto-encoder.
  • the initial L layers may be the layers of the encoder part.
  • the binarization simulation is applied on the output activations of the encoder. In such an assumption, the amount of this noise is important. If the noise is too high, the rest of the network can be insufficient to achieve robustness.
  • each i-th activation of the n-dimensional activations vector that is obtained from a sigmoid layer takes values in the range [0,1].
  • Cartesian coordinates are (0,0), (0,1), (1,1), (1,0). This is illustrated in Figure 5a, which shows the worst case scenario for such method, where the maximum noise amount is depicted by the length of the arrow.
  • the simulated interval should be well suited.
  • the surface comprises a hypersphere having radius of 0.5 and centre point located at 0.5 x 1, wherein 1 is an N-dimensional vector of ones .
  • the method further comprises calculating the mapping by
  • the method further comprises dividing the N- dimensional input vector into pairs of 2-dimensional activations; and mapping each 2- dimensional activation to a circle that intersects comers of a square having side length of one.
  • the activations are mapped on surface that intersects the comers of the hypercube.
  • the sphere surface mapping according to the above embodiment greatly reduces the noise to be simulated, the activations can never have binary values, in other words, the sphere in Figures 5b and 5c does not cross the square comers which are the binary vectors.
  • An approach to alleviate this issue is to map the activations to the surface of a sphere that crosses the hypercube comers, i.e., to the surface of a sphere which inscribes the hypercube.
  • the activations may be mapped to the surface of a hypersphere of center [0.5 , 0.5 , ... , 0.5] and a radius of where N is the dimension of the activations.
  • the method further comprises calculating the mapping of the 2-dimensional activation z by
  • z comprises a pair of N-dimensional activation vectors.
  • the embodiment provides a solution that both enables the activations to have binary values and also stay in the interval that is required to perform binarization with rounding operation.
  • the embodiment may be constrained such that the N is a multiple of 2.
  • the activations are separated to pairs such that there will be N/2 pairs of 2-dimensional activations. Then, every pair of 2-d activations are mapped to a circle that intersects the comers of the square of side-length of 1.
  • Such pairwise sphere process is illustrated in Figure 6, where the N-dimensional vector is first separated into pairs (xi, x 2 ), (x 3 , x 4 ).. (XN-I , XN), and then each pair is mapped to a point on the circle.
  • each pair of the N-dimensional activation vectors is created from neighbouring/adjacent activations x t and x i+1 within the activation vector.
  • the embodiments of pairwise sphere approach are not limited to the activations of having N dimensions where N is divisible by 2 only. In fact, the embodiments can be generalized to odd number of dimensions by separating the vector into a 3-dimensional activation and (iV— 3)/2 activation pairs.
  • Another alternative is reflective padding, i.e. padding the vector from the ends with the same activation value at the end of the vector.
  • the training and inference strategies of the conventional binarization as a noise approach may be adopted, as shown in Figures 7a and 7b.
  • no sigmoid function is used to obtain the activation vectors and the activation vectors are either mapped to a N-dimensional sphere surface with radius 0.5 and centre [0.5 ,0.5 ,...,0.5] (i.e. so-called fitting sphere approach) or first separated into pairs and each pair is mapped to a circle of radius VN/2 and of centre [0.5 ,0.5 ] (i.e. so-called pairwise sphere approach).
  • the binarization of the output vector of the first layer comprises adding noise
  • the binarization of the output vector of the first layer comprises a binarization operation.
  • the binarization as referred to at training stage is actually an
  • the binarization in the case of inference stage refers to actual binarization.
  • the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof.
  • some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller,
  • microprocessor or other computing device although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
  • the embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware.
  • any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions.
  • the software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.
  • the memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as
  • the data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multi-core processor architecture, as non-limiting examples.
  • general purpose computers special purpose computers
  • microprocessors microprocessors
  • DSPs digital signal processors
  • processors based on multi-core processor architecture, as non-limiting examples.
  • Embodiments of the inventions may be practiced in various components such as integrated circuit modules.
  • the design of integrated circuits is by and large a highly automated process.
  • Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
  • Programs such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules.
  • the resultant design in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)

Abstract

Procédé comprenant : l'entraînement ou l'inférence d'un réseau neuronal comprenant une pluralité de couches pour produire des activations et/ou des pondérations binarisées (400), ledit entraînement ou ladite inférence comprenant : la mise en correspondance d'un vecteur N-dimensionnel avec une sous-région d'un espace N-dimensionnel dans une première couche (402); et la binarisation d'un vecteur de sortie de la première couche dans une seconde couche (404).
PCT/FI2019/050495 2018-07-05 2019-06-25 Appareil, procédé et programme informatique pour l'entraînement d'un réseau neuronal Ceased WO2020008108A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FI20185624 2018-07-05
FI20185624 2018-07-05

Publications (1)

Publication Number Publication Date
WO2020008108A1 true WO2020008108A1 (fr) 2020-01-09

Family

ID=69060434

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2019/050495 Ceased WO2020008108A1 (fr) 2018-07-05 2019-06-25 Appareil, procédé et programme informatique pour l'entraînement d'un réseau neuronal

Country Status (1)

Country Link
WO (1) WO2020008108A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115485696A (zh) * 2020-04-17 2022-12-16 微软技术许可有限责任公司 机器学习模型的对抗预训练
CN116897358A (zh) * 2021-02-23 2023-10-17 三星电子株式会社 使用在多个应用之间共享的机器学习模型的设备和方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160148078A1 (en) * 2014-11-20 2016-05-26 Adobe Systems Incorporated Convolutional Neural Network Using a Binarized Convolution Layer
US20170286830A1 (en) * 2016-04-04 2017-10-05 Technion Research & Development Foundation Limited Quantized neural network training and inference
WO2018048907A1 (fr) * 2016-09-06 2018-03-15 Neosensory, Inc. C/O Tmc+260 Procédé et système visant à fournir des informations sensorielles complémentaires à un utilisateur
WO2018091333A1 (fr) * 2016-11-18 2018-05-24 Robert Bosch Gmbh Procédé pour calculer une sortie d'un réseau de neurones artificiels

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160148078A1 (en) * 2014-11-20 2016-05-26 Adobe Systems Incorporated Convolutional Neural Network Using a Binarized Convolution Layer
US20170286830A1 (en) * 2016-04-04 2017-10-05 Technion Research & Development Foundation Limited Quantized neural network training and inference
WO2018048907A1 (fr) * 2016-09-06 2018-03-15 Neosensory, Inc. C/O Tmc+260 Procédé et système visant à fournir des informations sensorielles complémentaires à un utilisateur
WO2018091333A1 (fr) * 2016-11-18 2018-05-24 Robert Bosch Gmbh Procédé pour calculer une sortie d'un réseau de neurones artificiels

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
CAGLAR AYTEKIN ET AL.: "Block-optimized Variable Bit Rate Neural Image Compression", ARXIV.ORG, 28 May 2018 (2018-05-28), XP080883107, [retrieved on 20191204] *
MATTHIEU COURBARIAUX ET AL.: "Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1", IN: ARXIV.ORG, 17 March 2016 (2016-03-17), XP055405835, Retrieved from the Internet <URL:https://arxiv.org/abs/1602.02830> [retrieved on 20191204] *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115485696A (zh) * 2020-04-17 2022-12-16 微软技术许可有限责任公司 机器学习模型的对抗预训练
CN116897358A (zh) * 2021-02-23 2023-10-17 三星电子株式会社 使用在多个应用之间共享的机器学习模型的设备和方法

Similar Documents

Publication Publication Date Title
CN111352965A (zh) 序列挖掘模型的训练方法、序列数据的处理方法及设备
EP3938965A1 (fr) Appareil, procédé et programme d&#39;ordinateur pour l&#39;apprentissage d&#39;un réseau neuronal
CN118152581A (zh) 知识图谱补全方法及装置、电子设备、计算机可读介质
CN114375462A (zh) 用于视图合成的多视图多尺度方法和装置
CN117975211A (zh) 一种基于多模态信息的图像处理方法和装置
CN115937020A (zh) 图像处理方法、装置、设备、介质和程序产品
Tomić et al. Performance of Post‐Training Two‐Bits Uniform and Layer‐Wise Uniform Quantization for MNIST Dataset from the Perspective of Support Region Choice
WO2020008108A1 (fr) Appareil, procédé et programme informatique pour l&#39;entraînement d&#39;un réseau neuronal
US20240104915A1 (en) Long duration structured video action segmentation
CN111639198A (zh) 媒体文件识别方法、装置、可读介质及电子设备
CN111582456B (zh) 用于生成网络模型信息的方法、装置、设备和介质
WO2019141905A1 (fr) Appareil, procédé et programme informatique pour faire fonctionner un réseau neuronal
CN117409486B (zh) 基于视觉的动作生成方法、装置、电子设备及存储介质
Globa et al. The Modified Approach to Internet of Things Data Transmission Based on a Combined Neural Network Autoencoder
CN119089442A (zh) 一种恶意代码的分类方法、装置及存储介质
CN119537916A (zh) 多源异构流量融合的特征提取方法及系统
CN118097293A (zh) 基于残差图卷积网络和自注意力的小样本数据分类方法及系统
CN112364938A (zh) 对抗样本生成方法、装置和电子设备
Samarathunga et al. Semantic communication based complexity scalable image transmission system for resource constrained devices
CN119169326A (zh) 数据处理方法、模型训练方法、电子设备、介质及产品
CN118228255A (zh) 一种应用程序的风险检测方法、装置及设备
CN119106709A (zh) 二值神经网络装置及其训练方法
Liu et al. Cluster channel equalization using adaptive sensing and reinforcement learning for UAV communication
WO2019141896A1 (fr) Procédé pour réseaux neuronaux
Qi Computationally Efficient Approach for 6G‐AI‐IoT Network Slicing and Error‐Free Transmission

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19831509

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19831509

Country of ref document: EP

Kind code of ref document: A1