WO2024082679A1 - 计算图的处理方法和装置 - Google Patents

计算图的处理方法和装置 Download PDF

Info

Publication number
WO2024082679A1
WO2024082679A1 PCT/CN2023/102301 CN2023102301W WO2024082679A1 WO 2024082679 A1 WO2024082679 A1 WO 2024082679A1 CN 2023102301 W CN2023102301 W CN 2023102301W WO 2024082679 A1 WO2024082679 A1 WO 2024082679A1
Authority
WO
WIPO (PCT)
Prior art keywords
subgraph
operators
tasks
subgraphs
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/102301
Other languages
English (en)
French (fr)
Inventor
焦建兵
许世峰
林嘉树
周剑
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP23878676.8A priority Critical patent/EP4597362A4/en
Publication of WO2024082679A1 publication Critical patent/WO2024082679A1/zh
Priority to US19/182,292 priority patent/US20250306991A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G06F8/41—Compilation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G06F8/41—Compilation
    • G06F8/44—Encoding
    • G06F8/443—Optimisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
    • G06N3/105—Shells for specifying net layout

Definitions

  • the present application relates to neural network technology, and more particularly to a method and device for processing a computational graph.
  • each dimension of the static shape is a known value, for example, [10,10,20,30].
  • the present application provides a method and device for processing a computational graph to solve the software compilation and efficient execution of a dynamic shape network model, and implements a program execution method with a computational graph as the core under a dynamic shape.
  • the present application provides a method for processing a computational graph, comprising: obtaining a computational graph to be compiled, wherein the input data of the computational graph adopts a dynamic shape, and the computational graph includes multiple operators; dividing the computational graph to obtain multiple subgraphs, any one of the subgraphs including at least one of the operators in the computational graph; compiling and generating multiple executable tasks according to the multiple subgraphs; and running the computational graph based on the multiple executable tasks.
  • a graph scheduling software solution is provided with subgraph as the basic scheduling execution unit (different from the stream scheduling mechanism with node as the unit) to solve the software compilation and efficient execution of the dynamic shape network model, and realize the program execution method with computational graph as the core under dynamic shape.
  • the input/output of the computation graph uses a dynamic shape, that is, in the input/output, the shape used to represent the pixel composition structure is a dynamic shape, in which one or more dimensions have a value of -1, and the dimension corresponding to the -1 is unknown during the compilation phase.
  • the computation graph can include multiple operators (i.e., Nodes). For example, in the computation graph shown in Figure 1, each layer corresponds to an operator, so the computation graph includes 11 operators.
  • the input/output can be split.
  • the convolution operator can support continuous multi-equal splitting of input/outputinput/output, and can also support jump splitting of input/outputinput/output.
  • the input/outputinput/output can be split into multiple copies of data, thereby reducing the amount of data in each copy, increasing the number of operators, and improving execution efficiency through concurrent execution.
  • the strategy for dividing subgraphs in the embodiment of the present application may include classifying operators with the same and continuous segmentation methods into the same subgraph, and may also classify operators with different and continuous segmentation methods into the same subgraph.
  • multiple subgraphs can be statically compiled to obtain multiple thread tasks; then the data to be processed is obtained; and the multiple thread tasks are dynamically compiled according to the data to be processed to obtain multiple executable tasks.
  • the compilation process of the computation graph can include two parts: static compilation and dynamic compilation.
  • the computation graph can be divided into several subgraphs according to the hardware resource situation in the compilation state, so as to realize the static processing of the computation graph irrelevant to the shape and generate thread tasks.
  • the execution steps related to the static graph are executed in advance before the operation, which can reduce the runtime compilation overhead.
  • connection operator can be inserted at the beginning and end of the first subgraph.
  • the connection operator can include inlabel, AT-start, AT-end, outlabel, etc. These operators can be customized to achieve the connection function.
  • the m operators included in the first subgraph may be optimized, including various fusion optimizations, single operator optimizations, constant folding optimizations, dtype, format optimizations, and the like.
  • cache operations can be performed on N thread tasks.
  • Cache operations (for example, prefetch, invalid, writeback, etc.) can be performed on multiple operators in a subgraph.
  • the dynamic shape of the data to be processed is obtained; the unknown parameters in multiple thread tasks are updated according to the dynamic shape of the data to be processed to obtain multiple executable tasks.
  • the concurrency parameters of the thread task can be calculated according to the actual dynamic shape, and the dynamic shape and thread concurrency related parameters in the thread task can be updated (for example, the part related to -1 in the shape in the thread task is refreshed), and finally an executable task is generated.
  • host tasks and device tasks can be executed in the host and device pipelines to improve execution efficiency.
  • threadnum can be calculated in real time according to the dynamic shape of the data to be processed, and threadnum represents the number of segments of the data to be processed.
  • the dynamic shape is substituted into a preset formula to obtain threadnum, and the preset formula is used to express the corresponding relationship between the dynamic shape and threadnum.
  • the cache strategy can also be called a cache strategy.
  • all data will not be cached, and the cache bandwidth will be used to improve the subgraph execution performance.
  • threadnum is obtained with the goal of fully operating the engine in the first subgraph.
  • each engine runs computing resources at full capacity.
  • the blockdim of engines of the same type integer multiples of chip cores; the computing resources of the engine bound run at full capacity, and the computing resources of the engine non-bound do not block the operation of bound computing resources; in the compilation state, based on the fitted shape, the best pipeline of engine concurrency is evaluated, and the recommended value of threadnum that meets the best pipeline is given; combined with the cache resource strategy, the threadnum value is actually synthesized in the running state.
  • threadnum subtasks included in the first subgraph are scheduled based on multiple executable tasks, threadnum represents the number of divisions of the data to be processed in the first subgraph, each subtask includes m operators, m ⁇ 1, and the first subgraph is any one of the multiple subgraphs.
  • the embodiment of the present application issues executable tasks in units of subgraphs, and concurrently schedules and executes tasks in units of subgraphs, which can improve the concurrent efficiency of resources.
  • bandwidth is allocated to multiple executable tasks with the goal of minimizing the total running time.
  • the Qos strategy It can also be called the Qos strategy. It adjusts the Qos value of the thread subtask according to the optimal pipeline mentioned above. When the subgraphs are executed concurrently, the priority of each computing resource and bandwidth is coordinated to ensure the efficient execution of the entire graph.
  • the present application provides a processing device, including: an acquisition module, used to obtain a calculation graph to be compiled, the input data of the calculation graph adopts a dynamic shape, and the calculation graph includes multiple operators; a partitioning module, used to partition the calculation graph to obtain multiple sub-graphs, any one of the sub-graphs includes at least one of the operators in the calculation graph; a compilation module, used to compile and generate multiple executable tasks according to the multiple sub-graphs; and a running module, used to run the calculation graph based on the multiple executable tasks.
  • the partitioning module is specifically used to obtain partitioning information of the multiple operators, where the partitioning information is used to indicate input data partitioning methods supported by corresponding operators; and to obtain the multiple subgraphs according to the partitioning information of the multiple operators.
  • the first subgraph when the first subgraph includes n operators, the n operators are arranged consecutively, and the input data segmentation methods supported by the n operators are the same, n>1, and the first subgraph is any one of the multiple subgraphs.
  • the compilation module is specifically used to statically compile the multiple subgraphs respectively to obtain multiple thread tasks; obtain data to be processed; and dynamically compile the multiple thread tasks according to the data to be processed to obtain the multiple executable tasks.
  • the compilation module is specifically used to obtain the total number of engines in a first subgraph, where the first subgraph is any one of the multiple subgraphs, and the first subgraph includes m operators, where m ⁇ 1; determine N based on the total number of engines in the first subgraph, where N>1, and N represents the number of threads that can run concurrently; obtain N thread tasks, where any one of the thread tasks includes m structures, and the N thread tasks correspond to N threads.
  • the compiling module is specifically used to obtain the dynamic shape of the data to be processed; and update the unknown parameters in the multiple thread tasks according to the dynamic shape of the data to be processed to obtain the multiple executable tasks.
  • the compiling module is further configured to obtain threadnum according to the dynamic shape of the data to be processed, where threadnum represents the number of divisions of the data to be processed.
  • the compiling module is specifically configured to substitute the dynamic shape into a preset formula to obtain the threadnum, where the preset formula is configured to represent a corresponding relationship between the dynamic shape and the threadnum.
  • the compiling module is specifically configured to obtain the threadnum with the goal of fully operating the engine in the first subgraph.
  • the compiling module is further configured to insert a connection operator at the beginning and the end of the first subgraph.
  • the compilation module is further used to optimize the m operators included in the first subgraph.
  • the compiling module is further configured to perform a cache operation on the N thread tasks.
  • the running module is specifically used to schedule threadnum subtasks included in the first subgraph based on the multiple executable tasks in a manner of multiplexing executable tasks, where threadnum represents the number of divisions of the data to be processed in the first subgraph, and each of the subtasks includes m operators, m ⁇ 1, and the first subgraph is any one of the multiple subgraphs.
  • the running module is further configured to allocate bandwidth to the multiple executable tasks with the goal of minimizing the total running time.
  • the present application provides a device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement a method as described in any one of the above-mentioned first aspects.
  • the present application provides a computer-readable storage medium, comprising a computer program, wherein when the computer program is executed on a computer, the computer executes any one of the methods described in the first aspect.
  • the present application provides a computer program product, wherein the computer program product comprises a computer program code, and when the computer program code is executed on a computer, the computer executes any one of the methods in the first aspect.
  • FIG1 is an exemplary schematic diagram of a computation graph
  • FIG2 is an exemplary structural diagram of the hardware system of the present application.
  • FIG3 is a flow chart of a process 300 of a method for processing a computational graph of the present application
  • FIG4 is a schematic diagram of sub-graph segmentation
  • Fig. 5 is a schematic diagram of a connection operator
  • Fig. 6 is a schematic diagram of dynamic compilation
  • FIG7 is a schematic diagram of the implementation process of the subgraph operation
  • FIG. 8 is a schematic diagram of the structure of a processing device 800 according to an embodiment of the present application.
  • At least one (item) means one or more, and “plurality” means two or more.
  • “And/or” is used to describe the association relationship of associated objects, indicating that three relationships may exist.
  • a and/or B can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural.
  • the character “/” generally indicates that the objects associated before and after are in an “or” relationship.
  • At least one of the following” or similar expressions refers to any combination of these items, including any combination of single or plural items.
  • At least one of a, b or c can mean: a, b, c, "a and b", “a and c", “b and c", or "a and b and c", where a, b, c can be single or multiple.
  • a neural network is a machine learning model.
  • a neural network can be composed of neural units.
  • a neural unit can refer to an operation unit with xs and intercept 1 as input.
  • the output of the operation unit can be:
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer.
  • the activation function can be a nonlinear function such as ReLU.
  • a neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be an area composed of several neural units.
  • Multi-layer perception (MLP)
  • MLP is a simple deep neural network (DNN) (different layers are fully connected), also known as a multi-layer neural network, which can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated.
  • DNN simple deep neural network
  • the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter.
  • W the weight parameter
  • more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks.
  • Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.
  • a convolutional neural network contains a feature extractor consisting of a convolutional layer and a pooling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve an input image or convolution feature map.
  • a convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal.
  • a convolutional layer can include many convolution operators, also known as kernels. Their role in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator can essentially be a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is usually processed horizontally on the input image one pixel after another (or two pixels after two pixels... depending on the value of the stride) to complete the work of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image.
  • the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (row ⁇ column) are applied, that is, multiple isotype matrices.
  • the output of each weight matrix is stacked up to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image, for example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to blur unnecessary noise points in the image.
  • the multiple weight matrices have the same size (rows ⁇ columns), and the size of the feature maps extracted by the multiple weight matrices of the same size is also the same.
  • the extracted multiple feature maps of the same size are then merged to form the output of the convolution operation.
  • the weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • the weight matrices formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network can make correct predictions.
  • the initial convolutional layer often extracts more general features, which can also be called low-level features; as the depth of the convolutional neural network deepens, the features extracted by the later convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.
  • the pooling layer can include an average pooling operator and/or a maximum pooling operator to sample the input image to obtain an image of smaller size.
  • the average pooling operator can calculate the pixel values in the image within a specific range to produce an average value as the result of average pooling.
  • the maximum pooling operator can take the pixel with the largest value in the range as the result of maximum pooling within a specific range.
  • the operator in the pooling layer should also be related to the image size.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer.
  • Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolution neural network After being processed by the convolution layer/pooling layer, the convolution neural network is not sufficient to output the required output information. Because as mentioned above, the convolution layer/pooling layer will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.
  • an output layer of the entire convolutional neural network is also included.
  • the output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • Recurrent neural networks are used to process sequence data.
  • the layers are fully connected, and the nodes within each layer are disconnected.
  • this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output.
  • RNN can process sequence data of any length.
  • the training of RNN is the same as the training of traditional CNN or DNN.
  • the same error back propagation algorithm is used, but there is one difference: that is, if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above.
  • the output of each step depends not only on the network at the current step, but also on the state of the network at the previous steps. This learning algorithm is called the Back Propagation Through Time (BPTT).
  • BPTT Back Propagation Through Time
  • Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller.
  • BP error back propagation
  • the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges.
  • the back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the super-resolution model, such as the weight matrix.
  • Generative adversarial networks are a deep learning model.
  • the model includes at least two modules: one module is the generative model, and the other module is the discriminative model. Through the game learning between these two modules, better output can be produced.
  • Both the generative model and the discriminative model can be neural networks, specifically deep neural networks or convolutional neural networks.
  • the basic principle of GAN is as follows: Taking the GAN that generates pictures as an example, suppose there are two networks, G (Generator) and D (Discriminator), where G is a network that generates pictures. It receives a random noise z and generates pictures through this noise, denoted as G(z); D is a discriminative network, which is used to determine whether a picture is "real".
  • Its input parameter is x
  • x represents a picture
  • the output D(x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if it is 0, it means that it cannot be a real picture.
  • the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. In this way, G and D constitute a dynamic "game” process, which is also the "adversary" in the "generative adversarial network”.
  • a neural network model can use a computational graph to express each layer and the connection relationship between layers.
  • Figure 1 is an exemplary schematic diagram of a computational graph.
  • the computational graph includes 6 convolutional layers, 5 activation layers ReLU, and the size of the convolution kernel is 3 ⁇ 3. That is, the computational graph includes a total of 11 operators (also referred to as nodes), and one operator corresponds to one of the aforementioned layers (for example, a convolutional layer or an activation layer).
  • the output of the previous layer can be used as the input of the next layer, where the input of the first operator is also the input of the computational graph, and the output of the last operator is also the output of the computational graph.
  • the computational graph can adopt any of the topological structures of the neural network mentioned above, and there is no specific limitation on this.
  • the computational graph can realize some functions in use, such as image recognition, intelligent translation, intelligent recommendation, etc. Before implementing the aforementioned functions, the computational graph needs to be compiled to generate an executable expression, and then the aforementioned executable expression is run in the hardware environment to execute the function of the computational graph. It should be noted that the executable expression can be a file or a data structure of a program, etc., and the embodiments of this application do not specifically limit this.
  • the input/output of the computational graph can use a dynamic shape.
  • a dynamic shape also called dynamic tensor/dynamic shape/unknown shape
  • the dynamic shape is unknown at compile time, and the specific dim value is only known at actual runtime. For example:
  • the concept of shape can be used to represent the arrangement of input/output in memory, and can also be understood as the composition structure of input/outputinput/output.
  • input/output can be a picture, or it can be voice, text, etc., and there is no specific limitation on this.
  • the size of a two-dimensional picture can be expressed as 100*50
  • a three-dimensional picture can be expressed as 3*100*50 using red, green and blue.
  • the number of pictures 100 can be expressed as 100*3*100*50.
  • Shape can be considered as the arrangement structure of the pixels of the input picture in memory.
  • Dynamic shape can be understood as the pixels of the image to be processed each time are different, for example, this time it is 100*50, next time it is 10*5, and next time it is 1000*500. It can also be understood as the sentences to be processed each time are different, for example, the length and characters of the sentences are different. Also, the specifications of the objects to be processed are not fixed.
  • 100 images each composed of red, yellow and blue, with a size of 100*50, can be represented as shape[100,3,100,50].
  • the input data of the calculation graph includes the actual data (the pixel values of all pixels) and the aforementioned shape (indicating the emission structure of all multi-pixel points).
  • the output data of the calculation graph also includes the actual data (the pixel values of all pixels) and shape (indicating the emission structure of all multi-pixel points).
  • the shape can be the same as or different from the shape of the input data, for example, shape[50,3,100,50].
  • FIG2 is an exemplary structural diagram of the hardware system of the present application.
  • the processing program on the host sends the compiled model to the device, and the neural network processor/chip/central processing unit (CPU) on the device executes the neural network model.
  • the model can be compiled offline on other personal computers (PC) machines or on the host machine.
  • the host and device of the present application can be integrated into the same electronic device to jointly complete memory operations, and the electronic device has storage capabilities; the host and device can also be respectively set on different electronic devices, for example, the host is set on a user device such as a mobile phone or tablet computer, and the device is set on a neural network processor.
  • FIG3 is a flowchart of process 300 of the method for processing a computational graph of the present application. As shown in FIG3 , process 300 may be performed by the host and the device described above. Process 300 is described as a series of steps or operations. It should be understood that process 300 may be performed in various orders and/or occur simultaneously, and is not limited to the execution order shown in FIG3 . Process 300 includes the following steps:
  • Step 301 Obtain the computation graph to be compiled.
  • the input/output of the computational graph uses a dynamic shape, that is, in the input/output, the shape used to represent the pixel composition structure is a dynamic shape, in which one or more dimensions have a value of -1, and the dimension corresponding to the -1 is unknown during the compilation stage.
  • a computation graph may include multiple operators (ie, Nodes).
  • each layer corresponds to an operator, and the computation graph includes 11 operators.
  • Step 302 Divide the computation graph to obtain multiple subgraphs.
  • segmentation information of multiple operators included in a computational graph may be obtained, and the segmentation information is used to indicate the input/output segmentation method supported by the corresponding operator; and then the graph is divided into multiple subgraphs according to the segmentation information features of the multiple operators.
  • the input/output can be split.
  • the convolution operator can support continuous multiple equal splitting of the input/output, and can also support jump splitting of the input/output.
  • the input/output can be split into multiple copies of data, thereby reducing the amount of data in each copy, and then increasing the number of operators to improve execution efficiency through concurrent execution.
  • the strategy for dividing the subgraphs in the embodiment of the present application may include classifying operators with the same segmentation method and being continuous into the same subgraph, or classifying operators with different segmentation methods and being continuous into the same subgraph.
  • the computation graph has 10 operators 1-10, among which: Operators 1-3 support the same segmentation method, so operators 1-3 can be included in the same subgraph.
  • Operators 4 and 6 have the same segmentation method, they are not continuous, so operator 4 can only be included in one subgraph, operator 5 in another subgraph, and operator 6 in another subgraph.
  • Operators 7-10 support the same segmentation method, so operators 7-10 can be included in the same subgraph.
  • Step 303 compile and generate multiple executable tasks according to the multiple subgraphs.
  • multiple subgraphs can be statically compiled to obtain multiple thread tasks; then the data to be processed is obtained; and the multiple thread tasks are dynamically compiled according to the data to be processed to obtain multiple executable tasks.
  • the compilation process of the computation graph can include two parts: static compilation and dynamic compilation.
  • the computation graph can be divided into several subgraphs according to the hardware resource situation in the compilation state, so as to realize the static processing of the computation graph irrelevant to the shape and generate thread tasks.
  • the execution steps related to the static graph are executed in advance before the operation, which can reduce the runtime compilation overhead.
  • connection operator can be inserted at the beginning and end of the first subgraph.
  • the connection operator can include inlabel, AT-start, AT-end, outlabel, etc. These operators can be customized to achieve the connection function.
  • the m operators included in the first subgraph may be optimized, including various fusion optimizations, single operator optimizations, constant folding optimizations, dtype, format optimizations, and the like.
  • cache operations can be performed on N thread tasks.
  • Cache operations (for example, prefetch, invalid, writeback, etc.) can be performed on multiple operators in a subgraph.
  • the dynamic shape of the data to be processed is obtained; the unknown parameters in multiple thread tasks are updated according to the dynamic shape of the data to be processed to obtain multiple executable tasks.
  • the concurrency parameters of the thread task can be calculated according to the actual dynamic shape, and the dynamic shape and thread concurrency related parameters in the thread task can be updated (for example, the part related to -1 in the shape in the thread task is refreshed), and finally an executable task is generated, and the final executable task is generated.
  • host tasks and device tasks can be executed in the host and device pipelines to improve execution efficiency.
  • the number of slices (threadnum) of the data to be processed can be calculated in real time according to the dynamic shape of the data to be processed.
  • the dynamic shape is substituted into a preset formula to obtain threadnum, and the preset formula is used to express the corresponding relationship between the dynamic shape and threadnum.
  • the cache strategy can also be called a cache strategy.
  • all data will not be cached, and the cache bandwidth will be used to improve the subgraph execution performance.
  • threadnum is obtained with the goal of fully operating the engine in the first subgraph.
  • each engine runs computing resources at full specification.
  • the blockdim of engines of the same type integer multiples of chip cores; the bound computing resources of the engine run at full capacity, and the non-bound computing resources of the engine do not block the operation of bound computing resources; in the compilation state, based on the fitted shape, the best pipeline of engine concurrency is evaluated, and the recommended value of threadnum that meets the best pipeline is given; combined with the cache resource strategy, the threadnum value is actually synthesized in the running state.
  • Step 304 Run the computation graph based on multiple executable tasks.
  • threadnum subtasks included in the first subgraph are scheduled based on multiple executable tasks, threadnum represents the number of divisions of the data to be processed in the first subgraph, each subtask includes m operators, m ⁇ 1, and the first subgraph is any one of the multiple subgraphs.
  • the embodiment of the present application issues executable tasks in units of subgraphs, and concurrently schedules and executes tasks in units of subgraphs, which can improve the concurrent efficiency of resources.
  • bandwidth is allocated to multiple executable tasks with the goal of minimizing the total running time.
  • the Qos strategy It can also be called the Qos strategy. It adjusts the Qos value of the thread subtask according to the optimal flow mentioned above. When the subgraphs are executed concurrently, the priority of each computing resource and bandwidth is coordinated to ensure the efficient execution of the entire graph.
  • a graph scheduling software solution is provided with subgraph as the basic scheduling execution unit (different from the stream scheduling mechanism with node as the unit) to solve the software compilation and efficient execution of the dynamic shape network model, and realize the program execution method with computational graph as the core under dynamic shape.
  • the subgraph scope strategy is defined. Given a computational graph, the scope is determined according to the segmentation information (the range of the consistent segmentation axis is found according to the segmentation information), and the graph is segmented according to the automatic threading method (continuous equal segmentation method. If the graph cannot be evenly segmented, the tail block can be different from the non-tail block).
  • the window size (the number of threads executed simultaneously on the runtime devcie) is solidified, and threadnum is used as a variable to perform fitting evaluation on resources.
  • a fast formula is evaluated in the compilation state, and the threadnum of the segmentation is calculated by substituting the actual shape into the fast formula in the running state.
  • the evaluated resources include but are not limited to: cache resources, computing resources, etc.
  • Figure 4 is a schematic diagram of subgraph segmentation. As shown in Figure 4, the subgraph in this example includes 4 operators (conv, vec, conv, vec from top to bottom). The input/output of the subgraph can be divided into 8 parts (the method for obtaining the number of parts can be referred to below), and the processing of 8 parts of data can be processed by 8 threads, and the operator corresponding to each thread is the same.
  • the thread subtask context task under automatic threading is expressed by thread windows size threads. Even if the actual threadnum is unknown, the subgraph level task is expressed by automatic threading. When it exceeds or falls below the thread windows size, the tasks in the subgraph are adjusted during dynamic compilation.
  • each engine runs the computing resources at full specification.
  • the blockdim of the same type of engine is an integer multiple of the chip core; the bound computing resources of the engine run at full load, and the non-bound computing resources of the engine do not block the operation of the bound computing resources; in the compiled state, the optimal pipeline of the engine concurrency is evaluated according to the fitted shape, and the recommended value of threadnum that meets the optimal pipeline is given; combined with the cache resource strategy, the threadnum value is actually synthesized in the running state. Since the full load of each engine is different, the best situation can be found.
  • threadnum can be obtained by using only (2) or (3), or by using (2) and (3) to obtain threadnum.
  • connection operators such as inlabel, AT-start, AT-end, and outlabel at the beginning and end of the subgraph.
  • inlabel and outlabel indicate the beginning and end of the subgraph.
  • the operator name can be named arbitrarily.
  • AT-start and AT-end are the start and end operators of each thread.
  • the AT-start and AT-end descriptors contain threadid (the number of threads) and threadnum (the total number of threads).
  • Outlabel records the total number of threadnums that need to be executed. After each thread is executed, outlabel is reduced by 1. When threadnum in outlabel is reduced to 0, the subgraph is executed.
  • threadmun tasks can be fixedly expressed in the compiled state. When a thread is completed, a thread in threadnum-thread window size is replaced with the thread that has been executed in thread window size to achieve the purpose of reuse.
  • Shape-related parameter updates are implemented in dynamic compilation.
  • the subgraph is first dynamically compiled on the host to complete the derivation of shape information.
  • the number of threads in the subgraph is determined according to the aforementioned algorithm, memory is allocated, and the information that has not been obtained in the compiled state in the subgraph task is updated (the actual value can only be calculated when the actual shape is obtained during dynamic compilation).
  • the information required by the device is transferred through host to device, and the launch of the thread task is initiated to stimulate the device to start executing the task.
  • Multiple subgraphs on the host can be executed concurrently in a pipelined manner. After the previous subgraph infershapes the shape of the last node, the next subgraph obtains the shape of the first node, so that the host of the second subgraph begins to execute. Therefore, multiple subgraphs can be executed in a pipeline.
  • the host side refreshes the unknown part of the dynamic shape in the subgraph in the compiled state.
  • the dynamic shape is in the compiled state, because the shape is unknown, a lot of execution information cannot be determined. After obtaining the shape, the actual value of the execution information can be refreshed to the task.
  • Dynamic shape operators (the shape of the next operator depends on the output value of the previous operator, and this operator is called a dynamic shape operator) are not placed in the subgraph range, and need to obtain shape and other information through the interaction between the host and the device. They are executed as a single operator or an independent subgraph.
  • the execution process of the subgraph is completed by the cooperation of software and hardware.
  • the software is responsible for the scheduling of the subgraph.
  • the specific process is as follows:
  • the hardware preloads the subgraph into the cache and notifies the device scheduling software to start execution.
  • the device scheduling software reads the initial ready task (the preceding node count is 0).
  • the device scheduling software pushes the ready tasks to the hardware execution device for execution.
  • the hardware execution device reads the task descriptor from the cache and executes the ready tasks, including scheduling them to multiple heterogeneous engines for execution.
  • the device scheduling software takes the subsequent task of the currently completed task and counts the predecessor node of the subsequent task -. When the count is 0, the task is pushed to the hardware execution device for execution.
  • Steps 2 to 5 are repeated until the subgraph is executed.
  • the device scheduler After the device scheduler completes the execution of a thread task sequence, it will replace the entire thread with a new thread and modify the thread id and other information in the new thread context until all threads are executed.
  • the QoS value of thread subtask is adjusted.
  • the priority of each computing resource and bandwidth is coordinated to ensure the efficient execution of the whole graph.
  • Eight threads can run in parallel at the same time, which will grab the memory bandwidth.
  • QoS can allocate bandwidth according to the strategy. Bandwidth means that the computer has many buses. The higher the bus bandwidth, the better the performance.
  • FIG8 is a schematic diagram of the structure of a processing device 800 according to an embodiment of the present application.
  • the processing device 800 can be used in the hardware system described above, and includes: an acquisition module 801, a division module 802, a compilation module 803, and an operation module 804.
  • An acquisition module 801 is used to acquire a computational graph to be compiled, wherein the input data of the computational graph adopts a dynamic shape, and the computational graph includes multiple operators; a partitioning module 802 is used to partition the computational graph to obtain multiple subgraphs, wherein any of the subgraphs includes at least one of the operators in the computational graph; a compilation module 803 is used to compile and generate multiple executable tasks according to the multiple subgraphs; and a running module 804 is used to run the computational graph based on the multiple executable tasks.
  • the partitioning module 802 is specifically used to obtain partitioning information of the multiple operators, where the partitioning information is used to indicate input data partitioning methods supported by corresponding operators; and to obtain the multiple subgraphs according to the partitioning information of the multiple operators.
  • the first subgraph when the first subgraph includes n operators, the n operators are arranged consecutively, and the input data segmentation methods supported by the n operators are the same, n>1, and the first subgraph is any one of the multiple subgraphs.
  • the compilation module 803 is specifically used to statically compile the multiple subgraphs respectively to obtain multiple thread tasks; obtain data to be processed; and dynamically compile the multiple thread tasks according to the data to be processed to obtain the multiple executable tasks.
  • the compilation module 803 is specifically used to obtain the total number of engines in a first subgraph, where the first subgraph is any one of the multiple subgraphs, and the first subgraph includes m operators, where m ⁇ 1; determine N according to the total number of engines in the first subgraph, where N>1, and N represents the number of threads that can run concurrently; obtain N thread tasks, where any one of the thread tasks includes m structures, and the N thread tasks correspond to N threads.
  • the compiling module 803 is specifically configured to obtain a dynamic shape of the data to be processed; and update unknown parameters in the multiple thread tasks according to the dynamic shape of the data to be processed to obtain the multiple executable tasks.
  • the compiling module 803 is further configured to obtain threadnum according to the dynamic shape of the data to be processed, where threadnum indicates the number of divisions of the data to be processed.
  • the compiling module 803 is specifically configured to substitute the dynamic shape into a preset formula to obtain the threadnum, where the preset formula is used to represent a corresponding relationship between the dynamic shape and the threadnum.
  • the compiling module 803 is specifically configured to obtain the threadnum with the goal of fully operating the engine in the first subgraph.
  • the compiling module 803 is further configured to insert a connection operator at the beginning and the end of the first subgraph.
  • the compiling module 803 is further configured to optimize the m operators included in the first subgraph.
  • the compiling module 803 is further configured to perform a cache operation on the N thread tasks.
  • the running module 804 is specifically used to schedule threadnum subtasks included in the first subgraph based on the multiple executable tasks in a manner of multiplexing executable tasks, where threadnum represents the number of divisions of the data to be processed in the first subgraph, and each of the subtasks includes m operators, m ⁇ 1, and the first subgraph is any one of the multiple subgraphs.
  • the running module 804 is further configured to allocate bandwidth to the multiple executable tasks with the goal of minimizing the total running time.
  • the device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 3. Its implementation principle and technical effects are similar and will not be repeated here.
  • each step of the above method embodiment can be completed by the hardware integrated logic circuit in the processor or the instruction in the form of software.
  • the processor can be a general processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • the general processor can be a microprocessor or the processor can also be any conventional processor, etc.
  • the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to be executed, or the hardware and software modules in the coding processor are combined to be executed.
  • the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc.
  • the storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
  • the memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories.
  • the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.
  • the volatile memory may be a random access memory (RAM), which is used as an external cache.
  • RAM random access memory
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDR SDRAM double data rate SDRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM synchlink DRAM
  • DR RAM direct rambus RAM
  • the disclosed systems, devices and methods can be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
  • Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.
  • the technical solution of the present application or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Neurology (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Image Analysis (AREA)

Abstract

本申请提供一种计算图的处理方法和装置。本申请计算图的处理方法，包括：获取待编译的计算图，所述计算图的输入数据采用动态shape，所述计算图包括多个算子；对所述计算图进行划分以得到多个子图，任意一个所述子图包括所述计算图中的至少一个所述算子；根据所述多个子图编译生成多个可执行任务；基于所述多个可执行任务运行所述计算图。本申请可以解决动态shape网络模型的软件编译和高效执行，实现了动态shape下以计算图为核心的程序执行方法。

Description

计算图的处理方法和装置
本申请要求于2022年10月19日提交中国专利局、申请号为202211277782.4、申请名称为“计算图的处理方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及神经网络技术，尤其涉及一种计算图的处理方法和装置。
背景技术
动态shape的某一个或者某几个维度dim＝-1，这在编译时是未知的，只有实际运行时才知道具体的dim值。例如：[10,-1,20,30]和[10,-1,-1,30]，其中分别有一个或一个以上的维度是-1。与此相对应的，静态shape的每一个维度均是已知值，例如，[10,10,20,30]。
神经网络模型中，当输入输出是动态shape时，如何高效的编译和运行神经网络模型，是业界努力解决的一个课题。
发明内容
本申请提供一种计算图的处理方法和装置，以解决动态shape网络模型的软件编译和高效执行，实现了动态shape下以计算图为核心的程序执行方法。
第一方面，本申请提供一种计算图的处理方法，包括：获取待编译的计算图，所述计算图的输入数据采用动态shape，所述计算图包括多个算子；对所述计算图进行划分以得到多个子图，任意一个所述子图包括所述计算图中的至少一个所述算子；根据所述多个子图编译生成多个可执行任务；基于所述多个可执行任务运行所述计算图。
本申请实施例，在动态shape下，以子图为基本调度执行单位的图调度软件解决方案(区别以node为单位的stream调度机制)，以解决动态shape网络模型的软件编译和高效执行，实现了动态shape下以计算图为核心的程序执行方法。
计算图的input/output采用动态shape，即，input/output中，用于表示像素组成结构的shape为动态shape，其中有一个或多个维度的值为-1，在编译阶段该-1对应的维度是未知的。计算图可以包括多个算子(亦即Node)，例如图1所示计算图中，每一层对应一个算子，则该计算图包括11个算子。
根据算子的特性，可以对input/output进行切分，例如，卷积算子可以支持对input/outputinput/output进行连续多等分切分，也可以支持对input/outputinput/output进行跳跃切分，这样可以将input/outputinput/output切分成多份数据，从而减少每一份的数据量，再增加算子的个数，通过并发执行提升执行效率。
本申请实施例划分子图的策略可以包括将切分方式相同且连续的算子归入同一子图中，也可以将切分方式不同且连续的算子归入同一个子图。以切分方式相同为例，计算图共有10个算子1-10，其中，算子1-3所支持的切分方式相同，那么算子1-3可以归入同一个子图，算子4和6虽然切分方式相同，但是算子4和6不是连续的，因此只能将算子4归入一个子图，算子5归入另一个子图，算子6归入又一个子图，算子7-10所支持的切分方式相同，那么算子7-10又可以归入同一个子图。由此可知，计算机划分得到的多个子图中，可以有只包含一个算子的子图，也可以有包含多个算子的子图，其中当第一子图包括n个算子时，n个算子连续排列，且n个算子所支持的输入数据切分方式相同，n＞＝1，第一子图为多个子图中的任意一个。
本申请实施例中，可以对多个子图分别进行静态编译，以得到多个线程任务；然后获取待处理数据；根据待处理数据对多个线程任务进行动态编译，以得到多个可执行任务。由此可见，对计算图的编译过程可以包括静态编译和动态编译两部分，其中，
静态编译过程中：获取第一子图内的引擎总数，第一子图为多个子图中的任意一个，第一子图包括m个算子，m≥1；根据第一子图内的引擎总数确定N，N＞1且>＝引擎总数，N表示可并发运行的线程数量；获取N个线程任务，任意一个线程任务包括m个算子的切分块，N个线程任务对应N个线程。
本申请实施例中，可以在编译态根据硬件资源情况，将计算图划分成几个子图，实现shape无关的计算图静态处理，生成线程任务。将静态图相关的执行步骤提前在运行前执行，可以降低运行时编译开销。
可选的，可以在第一子图的开头和结尾插入连接算子。连接算子可以包括inlabel，AT-start，AT-end，outlabel等，这些算子可以自定义，只是达成连接功能。
可选的，可以对第一子图包括的m个算子进行优化。该优化可以包括各种融合优化，单算子优化，常量折叠优化，dtype，format优化等。
可选的，可以对N个线程任务进行缓存操作。可以对子图内的多个算子，进行Cache操作(例如，prefetch，invalid，writeback等)。
动态编译过程中：获取待处理数据的动态shape；根据待处理数据的动态shape对多个线程任务中的未知参数进行更新，以得到多个可执行任务。
本申请实施例中，可以根据实际的动态shape计算线程任务的并发参数，并更新线程任务中的动态shape和线程(thread)并发相关参数(例如刷新线程任务里跟shape中-1相关的部分)，最终生成可执行的task，生成最终的可执行任务。动态编译和动态执行在运行时，可以将host任务和device任务在host和device流水执行，以提升执行效率。
此外，可以根据待处理数据的动态shape实时计算获取threadnum，threadnum表示待处理数据的切分数量。
在一种可能的实现方式中，将动态shape代入预设公式以得到threadnum，预设公式用于表示动态shape和threadnum之间的对应关系。
亦可以称作cache策略，在子图并发执行时，所有数据不出cache，利用cache的带宽提升子图执行性能。子图内所有node thread并发时，每个thread上节点的input+output+workspace+prefetch的内存消耗最大值之和<＝cache大小。由于编译态不知道shape实际大小，设置shape为变量使用拟合或者插值等方式，预估出L2cache资源的变量公式，以便在运行获取实际shape时，快速带入变量公式，算出切分的threadnum的最小值。
在一种可能的实现方式中，以第一子图内的引擎满负荷工作为目的获取threadnum。
亦可以称作计算资源策略，在子图并发执行时，各个引擎满规格跑计算资源。同一类型引擎的blockdim＝芯片core的整数倍；引擎bound的计算资源满负荷跑，引擎非bound的计算资源不阻塞bound计算资源的运行；在编译态，根据拟合的shape，评估出引擎并发的最佳流水，给出符合最佳流水的threadnum的建议值；结合cache资源的策略，在运行态实际综合出threadnum值。
以复用可执行任务的方式，基于多个可执行任务调度第一子图包括的threadnum个子任务，threadnum表示第一子图的待处理数据的切分数量，每个子任务包括m个算子，m≥1，第一子图为多个子图中的任意一个。
本申请实施例以子图为单位下发可执行任务，并发调度执行以子图为单位的任务，可以提升资源的并发效率。
在一种可能的实现方式中，以运行总时长最短为目的，给多个可执行任务分配带宽。
亦可以称作Qos策略，根据上文最佳流水调整thread subtask的Qos值，子图并发执行时，通过协调各计算资源和带宽的优先级，保证整图的高效执行。
第二方面，本申请提供一种处理装置，包括：获取模块，用于获取待编译的计算图，所述计算图的输入数据采用动态shape，所述计算图包括多个算子；划分模块，用于对所述计算图进行划分以得到多个子图，任意一个所述子图包括所述计算图中的至少一个所述算子；编译模块，用于根据所述多个子图编译生成多个可执行任务；运行模块，用于基于所述多个可执行任务运行所述计算图。
在一种可能的实现方式中，所述划分模块，具体用于获取所述多个算子的切分信息，所述切分信息用于指示对应算子所支持的输入数据切分方式；根据所述多个算子的切分信息获取所述多个子图。
在一种可能的实现方式中，当第一子图包括n个算子时，所述n个算子连续排列，且所述n个算子所支持的输入数据切分方式相同，n＞1，所述第一子图为所述多个子图中的任意一个。
在一种可能的实现方式中，所述编译模块，具体用于对所述多个子图分别进行静态编译，以得到多个线程任务；获取待处理数据；根据所述待处理数据对所述多个线程任务进行动态编译，以得到所述多个可执行任务。
在一种可能的实现方式中，所述编译模块，具体用于获取第一子图内的引擎总数，所述第一子图为所述多个子图中的任意一个，所述第一子图包括m个算子，m≥1；根据所述第一子图内的引擎总数确定N，N＞1，N表示可并发运行的线程数量；获取N个线程任务，任意一个所述线程任务包括m个结构体，所述N个线程任务对应N个线程。
在一种可能的实现方式中，所述编译模块，具体用于获取所述待处理数据的动态shape；根据所述待处理数据的动态shape对所述多个线程任务中的未知参数进行更新，以得到所述多个可执行任务。
在一种可能的实现方式中，所述编译模块，还用于根据所述待处理数据的动态shape获取threadnum，threadnum表示所述待处理数据的切分数量。
在一种可能的实现方式中，所述编译模块，具体用于将所述动态shape代入预设公式以得到所述threadnum，所述预设公式用于表示所述动态shape和所述threadnum之间的对应关系。
在一种可能的实现方式中，所述编译模块，具体用于以所述第一子图内的引擎满负荷工作为目的获取所述threadnum。
在一种可能的实现方式中，所述编译模块，还用于在所述第一子图的开头和结尾插入连接算子。
在一种可能的实现方式中，所述编译模块，还用于对所述第一子图包括的所述m个算子进行优化。
在一种可能的实现方式中，所述编译模块，还用于对所述N个线程任务进行缓存操作。
在一种可能的实现方式中，所述运行模块，具体用于以复用可执行任务的方式，基于所述多个可执行任务调度第一子图包括的threadnum个子任务，threadnum表示所述第一子图的待处理数据的切分数量，每个所述子任务包括m个算子，m≥1，所述第一子图为所述多个子图中的任意一个。
在一种可能的实现方式中，所述运行模块，还用于以运行总时长最短为目的，给所述多个可执行任务分配带宽。
第三方面，本申请提供一种设备，包括：一个或多个处理器；存储器，用于存储一个或多个程序；当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如上述第一方面中任一项所述的方法。
第四方面，本申请提供一种计算机可读存储介质，包括计算机程序，所述计算机程序在计算机上被执行时，使得所述计算机执行上述第一方面中任一项所述的方法。
第五方面，本申请提供一种计算机程序产品，所述计算机程序产品包括计算机程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行上述第一方面中任一项所述的方法。
附图说明
图1为计算图的一个示例性的示意图；
图2为本申请的硬件系统的一个示例性的结构图；
图3为本申请计算图的处理方法的过程300的流程图；
图4为子图切分的示意图；
图5为连接算子示意图；
图6为动态编译示意图；
图7为子图运行的实施过程示意图；
图8为本申请实施例的处理装置800的结构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合本申请中的附图，对本申请中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请的说明书实施例和权利要求书及附图中的术语“第一”、“第二”等仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元。方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
应当理解，在本申请中，“至少一个(项)”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，用于描述关联对象的关联关系，表示可以存在三种关系，例如，“A和/或B”可以表示：只存在A，只存在B以及同时存在A和B三种情况，其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)”或其类似表达，是指这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b或c中的至少一项(个)，可以表示：a，b，c，“a和b”，“a和c”，“b和c”，或“a和b和c”，其中a，b，c可以是单个，也可以是多个。
本申请实施例涉及神经网络的应用，为了便于理解，下面先对相关名词或术语进行解释说明：
1、神经网络
神经网络(neural network，NN)是机器学习模型，神经网络可以是由神经单元组成的，神经单元可以是指以xs和截距1为输入的运算单元，该运算单元的输出可以为：
其中，s＝1、2、……n，n为大于1的自然数，Ws为xs的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入。激活函数可以是ReLU等非线性函数。神经网络是将许多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部感受野(local receptive field)相连，来提取局部感受野的特征，局部感受野可以是由若干个神经单元组成的区域。
2、多层感知器(multi-layer perception，MLP)
MLP是一种简单的深度神经网络(deep neural network，DNN)(不同层之间是全连接的)，也称多层神经网络，可以理解为具有很多层隐含层的神经网络，这里的“很多”并没有特别的度量标准。从DNN按不同层的位置划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：其中，是输入向量，是输出向量，是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量经过如此简单的操作得到输出向量由于DNN层数多，则系数W和偏移向量的数量也就很多了。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。总结就是：第L-1层的第k个神经元到第L层的第j个神经元的系数定义为需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
3、卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。作为一种深度学习架构，CNN是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。卷积神经网络包含了一个由卷积层和池化层构成的特征抽取器。该特征抽取器可以看作是滤波器，卷积过程可以看作是使用一个可训练的滤波器与一个输入的图像或者卷积特征平面(feature map)做卷积。
卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。卷积层可以包括很多个卷积算子，卷积算子也称为核，其在图像处理中的作用相当于一个从输入图像矩阵中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义，在对图像进行卷积操作的过程中，权重矩阵通常在输入图像上沿着水平方向一个像素接着一个像素(或两个像素接着两个像素……这取决于步长stride的取值)的进行处理，从而完成从图像中提取特定特征的工作。该权重矩阵的大小应该与图像的大小相关，需要注意的是，权重矩阵的纵深维度(depth dimension)和输入图像的纵深维度是相同的，在进行卷积运算的过程中，权重矩阵会延伸到输入图像的整个深度。因此，和一个单一的权重矩阵进行卷积会产生一个单一纵深维度的卷积化输出，但是大多数情况下不使用单一权重矩阵，而是应用多个尺寸(行×列)相同的权重矩阵，即多个同型矩阵。每个权重矩阵的输出被堆叠起来形成卷积图像的纵深维度，这里的维度可以理解为由上面所述的“多个”来决定。不同的权重矩阵可以用来提取图像中不同的特征，例如一个权重矩阵用来提取图像边缘信息，另一个权重矩阵用来提取图像的特定颜色，又一个权重矩阵用来对图像中不需要的噪点进行模糊化等。该多个权重矩阵尺寸(行×列)相同，经过该多个尺寸相同的权重矩阵提取后的特征图的尺寸也相同，再将提取到的多个尺寸相同的特征图合并形成卷积运算的输出。这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以用来从输入图像中提取信息，从而使得卷积神经网络进行正确的预测。当卷积神经网络有多个卷积层的时候，初始的卷积层往往提取较多的一般特征，该一般特征也可以称之为低级别的特征；随着卷积神经网络深度的加深，越往后的卷积层提取到的特征越来越复杂，比如高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在图像处理过程中，池化层的唯一目的就是减少图像的空间大小。池化层可以包括平均池化算子和/或最大池化算子，以用于对输入图像进行采样得到较小尺寸的图像。平均池化算子可以在特定范围内对图像中的像素值进行计算产生平均值作为平均池化的结果。最大池化算子可以在特定范围内取该范围内值最大的像素作为最大池化的结果。另外，就像卷积层中用权重矩阵的大小应该与图像尺寸相关一样，池化层中的运算符也应该与图像的大小相关。通过池化层处理后输出的图像尺寸可以小于输入池化层的图像的尺寸，池化层输出的图像中每个像素点表示输入池化层的图像的对应子区域的平均值或最大值。
在经过卷积层/池化层的处理后，卷积神经网络还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层只会提取特征，并减少输入图像带来的参数。然而为了生成最终的输出信息(所需要的类信息或其他相关信息)，卷积神经网络需要利用神经网络层来生成一个或者一组所需要的类的数量的输出。因此，在神经网络层中可以包括多层隐含层，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括图像识别，图像分类，图像超分辨率重建等等。
可选的，在神经网络层中的多层隐含层之后，还包括整个卷积神经网络的输出层，该输出层具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络的前向传播完成，反向传播就会开始更新前面提到的各层的权重值以及偏差，以减少卷积神经网络的损失，及卷积神经网络通过输出层输出的结果和理想结果之间的误差。
4、循环神经网络
循环神经网络(recurrent neural networks，RNN)是用来处理序列数据的。在传统的神经网络模型中，是从输入层到隐含层再到输出层，层与层之间是全连接的，而对于每一层层内之间的各个节点是无连接的。这种普通的神经网络虽然解决了很多难题，但是却仍然对很多问题却无能无力。例如，你要预测句子的下一个单词是什么，一般需要用到前面的单词，因为一个句子中前后单词并不是独立的。RNN之所以称为循环神经网路，即一个序列当前的输出与前面的输出也有关。具体的表现形式为网络会对前面的信息进行记忆并应用于当前输出的计算中，即隐含层本层之间的节点不再无连接而是有连接的，并且隐含层的输入不仅包括输入层的输出还包括上一时刻隐含层的输出。理论上，RNN能够对任何长度的序列数据进行处理。对于RNN的训练和对传统的CNN或DNN的训练一样。同样使用误差反向传播算法，不过有一点区别：即，如果将RNN进行网络展开，那么其中的参数，如W，是共享的；而如上举例上述的传统神经网络却不是这样。并且在使用梯度下降算法中，每一步的输出不仅依赖当前步的网络，还依赖前面若干步网络的状态。该学习算法称为基于时间的反向传播算法(Back propagation Through Time，BPTT)。
既然已经有了卷积神经网络，为什么还要循环神经网络？原因很简单，在卷积神经网络中，有一个前提假设是：元素之间是相互独立的，输入与输出也是独立的，比如猫和狗。但现实世界中，很多元素都是相互连接的，比如股票随时间的变化，再比如一个人说了：我喜欢旅游，其中最喜欢的地方是云南，以后有机会一定要去。这里填空，人类应该都知道是填“云南”。因为人类会根据上下文的内容进行推断，但如何让机器做到这一步？RNN就应运而生了。RNN旨在让机器像人一样拥有记忆的能力。因此，RNN的输出就需要依赖当前的输入信息和历史的记忆信息。
5、损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断的调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
6、反向传播算法
卷积神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的超分辨率模型中参数的大小，使得超分辨率模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的超分辨率模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的超分辨率模型的参数，例如权重矩阵。
7、生成式对抗网络
生成式对抗网络(generative adversarial networks，GAN)是一种深度学习模型。该模型中至少包括两个模块：一个模块是生成模型(Generative Model)，另一个模块是判别模型(Discriminative Model)，通过这两个模块互相博弈学习，从而产生更好的输出。生成模型和判别模型都可以是神经网络，具体可以是深度神经网络，或者卷积神经网络。GAN的基本原理如下：以生成图片的GAN为例，假设有两个网络，G(Generator)和D(Discriminator)，其中G是一个生成图片的网络，它接收一个随机的噪声z，通过这个噪声生成图片，记做G(z)；D是一个判别网络，用于判别一张图片是不是“真实的”。它的输入参数是x，x代表一张图片，输出D(x)代表x为真实图片的概率，如果为1，就代表100％是真实的图片，如果为0，就代表不可能是真实的图片。在对该生成式对抗网络进行训练的过程中，生成网络G的目标就是尽可能生成真实的图片去欺骗判别网络D，而判别网络D的目标就是尽量把G生成的图片和真实的图片区分开来。这样，G和D就构成了一个动态的“博弈”过程，也即“生成式对抗网络”中的“对抗”。最后博弈的结果，在理想的状态下，G可以生成足以“以假乱真”的图片G(z)，而D难以判定G生成的图片究竟是不是真实的，即D(G(z))＝0.5。这样就得到了一个优异的生成模型G，它可以用来生成图片。
8、计算图
本申请实施例中，神经网络模型可以使用计算图表达各层(layer)以及layer之间的连接关系。图1为计算图的一个示例性的示意图，如图1所示，该计算图包括6个卷积层，5个激活层ReLU，卷积核的尺寸为3×3。亦即，该计算图总共包括11个算子(亦称作节点(Node))，一个算子对应前述一个层(例如卷积层或激活层)。按照从左向右的顺序，前一层的输出可以作为后一层的输入，其中第一个算子的输入亦即该计算图的输入，最后一个算子的输出亦即该计算图的输出。需要说明的是，计算图可以采用上文任意一种神经网络的拓扑结构，对此不做具体限定。
计算图在使用中可以实现一些功能，例如，识别图像、智能翻译、智能推荐等。而在实施前述功能之前，需要对计算图进行编译，生成可执行表达，进而在硬件环境中运行前述可执行表达，以执行计算图的功能。需要说明的是，可执行表达可以是文件，也可以是程序的数据结构等，本申请实施例对此不做具体限定。
9、动态shape
本申请实施例中，计算图的输入/输出可以采用动态shape。动态shape(亦称作dynamic tensor/dynamic shape/unknown shape)，其表示输入/输出的shape中，某一个或者某几个维度为-1(即dim＝-1)，动态shape在编译时是未知的，只有在实际运行时才知道具体的dim值。例如：
1)[10,-1,20,30],[10,-1,-1,30]，其中有一个或一个以上的维度是-1，则是动态shape；
2)[10,10,20,30]，其中每一个维度都是已知值，则是静态(known)shape。
在计算机图中，可以使用shape的概念，用以表示输入/输出(input/output)在内存中的排布，亦可以理解为是input/outputinput/output的组成结构，本申请实施例中，input/output可以是图片，也可以是语音、文字等，对此不做具体限定。示例性的，以图片输入为例，二维图片的尺寸可以表示成100*50，三维图片用红绿蓝表示可以表示成3*100*50，此外加上图片的数量100，可以表示成100*3*100*50。shape可以认为是输入的图片的像素点在内存中的排布结构。
动态shape，可以理解为每次需要处理的图片的像素是不同的，例如，这次是100*50，下次是10*5，再下次是1000*500。也可以理解为每次需要处理的语句不同，例如，句子的长度、字符等不同。以及，待处理对象的规格是不固定的。
示例性的，100张图片，每张图片由红黄蓝组成，尺寸为100*50，则可以表示成shape[100,3,100,50]，此时计算图的输入数据包括实际数据(所有像素点的像素值)，以及前述shape(表示所有多像素点的排放结构)。计算图的输出数据也包括实际数据(所有像素点的像素值)，以及shape(表示所有多像素点的排放结构)，此时shape与输入数据的shape可以相同，也可以不同，例如，shape[50,3,100,50]。
图2为本申请的硬件系统的一个示例性的结构图，如图2所示，主机(host)上的处理程序将编译好的模型下发到设备(device)，由device上的神经网络处理器/芯片/中央处理器(central processing unit，CPU)进行神经网络模型的执行。模型的编译可以离线在其他个人计算机(personal computer，PC)机器上进行编译，也可以在host机器上进行编译。
需要说明的是，本申请的host和device可以集成于同一个电子设备中，共同完成存储器操作，该电子设备具备存储能力；host和device也可以分别设置于不同的电子设备上，例如，host设置于手机、平板电脑等用户设备上，device设置于神经网络处理器上。
图3为本申请计算图的处理方法的过程300的流程图，如图3所示，过程300可以由上述主机和设备共同执行。过程300描述为一系列的步骤或操作，应当理解的是，过程300可以以各种顺序执行和/或同时发生，不限于图3所示的执行顺序。过程300包括如下步骤：
步骤301、获取待编译的计算图。
本申请实施例中，计算图的输入/输出(input/output)采用动态shape，即，input/output中，用于表示像素组成结构的shape为动态shape，其中有一个或多个维度的值为-1，在编译阶段该-1对应的维度是未知的。
如上文所述，计算图可以包括多个算子(亦即Node)，例如图1所示计算图中，每一层对应一个算子，则该计算图包括11个算子。
步骤302、对计算图进行划分以得到多个子图。
本申请实施例中，可以获取计算图中包括的多个算子的切分信息，该切分信息用于指示对应算子所支持的input/output切分方式；再根据多个算子的切分信息特征划分成多个子图。
根据算子的特性，可以对input/output进行切分，例如，卷积算子可以支持对input/output进行连续多等分切分，也可以支持对input/output进行跳跃切分，这样可以将input/output切分成多份数据，从而减少每一份的数据量，再增加算子的个数，通过并发执行提升执行效率。
本申请实施例划分子图的策略可以包括将切分方式相同且连续的算子归入同一子图中，也可以将切分方式不同且连续的算子归入同一个子图。以切分方式相同为例，计算图共有10个算子1-10，其中， 算子1-3所支持的切分方式相同，那么算子1-3可以归入同一个子图，算子4和6虽然切分方式相同，但是算子4和6不是连续的，因此只能将算子4归入一个子图，算子5归入另一个子图，算子6归入又一个子图，算子7-10所支持的切分方式相同，那么算子7-10又可以归入同一个子图。由此可知，计算机划分得到的多个子图中，可以有只包含一个算子的子图，也可以有包含多个算子的子图，其中当第一子图包括n个算子时，n个算子连续排列，且n个算子所支持的输入数据切分方式相同，n＞＝1，第一子图为多个子图中的任意一个。
步骤303、根据多个子图编译生成多个可执行任务。
本申请实施例中，可以对多个子图分别进行静态编译，以得到多个线程任务；然后获取待处理数据；根据待处理数据对多个线程任务进行动态编译，以得到多个可执行任务。由此可见，对计算图的编译过程可以包括静态编译和动态编译两部分，其中，
静态编译过程中：获取第一子图内的引擎总数，第一子图为多个子图中的任意一个，第一子图包括m个算子，m≥1；根据第一子图内的引擎总数确定N，N＞1且>＝引擎总数，N表示可并发运行的线程数量；获取N个线程任务，任意一个线程任务包括m个算子的切分块，N个线程任务对应N个线程。
本申请实施例中，可以在编译态根据硬件资源情况，将计算图划分成几个子图，实现shape无关的计算图静态处理，生成线程任务。将静态图相关的执行步骤提前在运行前执行，可以降低运行时编译开销。
可选的，可以在第一子图的开头和结尾插入连接算子。连接算子可以包括inlabel，AT-start，AT-end，outlabel等，这些算子可以自定义，只是达成连接功能。
可选的，可以对第一子图包括的m个算子进行优化。该优化可以包括各种融合优化，单算子优化，常量折叠优化，dtype，format优化等。
可选的，可以对N个线程任务进行缓存操作。可以对子图内的多个算子，进行缓存(Cache)操作(例如，prefetch，invalid，writeback等)。
动态编译过程中：获取待处理数据的动态shape；根据待处理数据的动态shape对多个线程任务中的未知参数进行更新，以得到多个可执行任务。
本申请实施例中，可以根据实际的动态shape计算线程任务的并发参数，并更新线程任务中的动态shape和线程(thread)并发相关参数(例如刷新线程任务里跟shape中-1相关的部分)，最终生成可执行的task，生成最终的可执行任务。动态编译和动态执行在运行时，可以将host任务和device任务在host和device流水执行，以提升执行效率。
此外，可以根据待处理数据的动态shape实时计算获取待处理数据的切分数量(threadnum)。
在一种可能的实现方式中，将动态shape代入预设公式以得到threadnum，预设公式用于表示动态shape和threadnum之间的对应关系。
亦可以称作cache策略，在子图并发执行时，所有数据不出cache，利用cache的带宽提升子图执行性能。子图内所有node thread并发时，每个thread上节点的input+output+workspace+prefetch的内存消耗最大值之和<＝cache大小。由于编译态不知道shape实际大小，设置shape为变量使用拟合或者插值等方式，预估出L2cache资源的变量公式，以便在运行获取实际shape时，快速带入变量公式，算出切分的threadnum的最小值。
在一种可能的实现方式中，以第一子图内的引擎满负荷工作为目的获取threadnum。
亦可以称作计算资源策略，在子图并发执行时，各个引擎满规格跑计算资源。同一类型引擎的blockdim＝芯片core的整数倍；引擎bound的计算资源满负荷跑，引擎非bound的计算资源不阻塞bound计算资源的运行；在编译态，根据拟合的shape，评估出引擎并发的最佳流水，给出符合最佳流水的threadnum的建议值；结合cache资源的策略，在运行态实际综合出threadnum值。
步骤304、基于多个可执行任务运行计算图。
以复用可执行任务的方式，基于多个可执行任务调度第一子图包括的threadnum个子任务，threadnum表示第一子图的待处理数据的切分数量，每个子任务包括m个算子，m≥1，第一子图为多个子图中的任意一个。
本申请实施例以子图为单位下发可执行任务，并发调度执行以子图为单位的任务，可以提升资源的并发效率。
在一种可能的实现方式中，以运行总时长最短为目的，给多个可执行任务分配带宽。
亦可以称作Qos策略，根据上文最佳流水调整thread subtask的Qos值，子图并发执行时，通过协调各计算资源和带宽的优先级，保证整图的高效执行。
本申请实施例，在动态shape下，以子图为基本调度执行单位的图调度软件解决方案(区别以node为单位的stream调度机制)，以解决动态shape网络模型的软件编译和高效执行，实现了动态shape下以计算图为核心的程序执行方法。
以下采用一个具体的实施例对上述方案进行说明。
一、静态编译
静态编译中，圈定子图范围(scope)策略，给定一个计算图，根据切分信息确定scope范围(根据切分信息找到一致切分轴的范围)，按照自动线程化(threading)方式(连续等分方式，在不能均分的情况下，尾快可以和非尾快不相同)进行切分，固化窗口尺寸(window size)大小(运行时devcie上同时执行的thread数)，将threadnum作为变量，对资源进行拟合评估，在编译态评估出快速公式，在运行态根据实际的shape代入快速公式算出切分的threadnum；评估资源包括但不限于：cache资源，计算资源等。
图4为子图切分的示意图，如图4所示，该示例中的子图包括4个算子(从上到下依次为conv、vec、conv、vec)，该子图的input/output可以切分成8份(份数的获取方式可参照下文)，则可以由8个thread处理8份数据的处理，每个线程对应的算子相同。
1、预估策略：针对划定的scope子图范围，在编译态进行预估，减少运行态计算时长消耗：
(1)固化thread window size＝N(N>＝子图内引擎总数)
通过thread windows size个thread表达自动threading下的线程子任务结构(thread subtask context)任务，即便在不知道实际threadnum情况下，利用自动threading的方法将子图级task表达出来。超过或低于thread windows size时，在动态编译时对子图内的task进行调整。
(2)cache策略：子图并发时，所有数据不出cache，利用cache的带宽提升子图执行性能。子图内所有节点(node)thread并发时，每个thread上节点的input+output+workspace+prefetch的内存消耗最大值之和<＝cache大小。由于编译态不知道shape实际大小，设置shape为变量使用拟合或者插值等方式，预估出L2cache资源的变量公式，以便在运行获取实际shape时，快速带入变量公式，算出切分的threadnum的最小值。由于从内存读写性能比较差，希望输入输出可以从cache进行读写，提高读写效率。
(3)计算资源策略：子图并发执行时，各个引擎满规格跑计算资源。同一类型引擎的blockdim＝芯片core的整数倍；引擎bound的计算资源满负荷跑，引擎非bound的计算资源不阻塞bound计算资源的运行；在编译态，根据拟合的shape，评估出引擎并发的最佳流水，给出符合最佳流水的threadnum的建议值；结合cache资源的策略，在运行态实际综合出threadnum值。由于各个引擎的满负荷度不一样，可以找出最佳的一种情况，该方法可以包括寻优找到最佳值，应理解还可以采用其他方法(例如，对所有任务进行流水排布方式；根据引擎数确定计算资源并发数，并发thread数>＝引擎数，等策略)，对此不做限定。
上述(2)和(3)是和/或关系，即可以只采用(2)或(3)得到threadnum，也可以采用(2)和(3)得到threadnum。
2、Task编译和生成
(1)以子图为单位，进行子图内算子优化，包括各种融合优化，单算子优化，常量折叠优化，dtype，format优化等；
(2)对动态shape子图内的多thread节点，进行Cache操作(prefetch，invalid，writeback)；
(3)虽然thread数尚未确定，发明了自动threadding技术，
a)在子图的开头和结尾插入inlabel，AT-start，AT-end，outlabel等连接算子。如图5(图5为连接算子示意图)所示，inlabel和outlabel表示子图的开始和结束，算子名称可以任意命名，AT-start，AT-end是每个thread的开始和结束算子，AT-start，AT-end描述符中包含threadid(第几个thread)和threadnum(总共几个thread)，outlabel里记录了总共需要执行多少各threadnum，每执行完一个thread，对outlabel中减1，当outlabel中threadnum减为0时，该子图执行完成。
b)只需要表达thread window size个thread，多余的thread数＝threadnum-thread window size，通过执行时复用，从而可以在编译态固定表达出threadmun个task任务。当某一个thread执行完成后，将threadnum-thread window size中的一个thread替换进thread window size里执行完成的thread，以达成复用目的。
(4)shape相关的参数更新动态编译中实现。
二、动态编译
如图6所示，动态编译
(1)Host侧Subgraph子图间任务并发流水。子图在host先进行动态编译，完成推导shape信息，根据前述算法确定子图内thread数，分配内存，更新子图任务中在编译态尚未获取的信息(只有在动态编译，获取到实际shape时，才可以计算出实际的值)，将device需要的信息通过host to device进行搬运，并发起线程任务的launch动态，激发device开始执行任务。Host上多个子图间可以流水的方式并发执行。在前一个子图infershape出最后一个node的shape后，下一个子图就获取到了第一个node的shape，这样第二个子图的host就开始执行下去了。因此多个子图可以流水执行下去。
(2)Host侧subgraph子图内node任务并发流水。子图内存在多个算子，每个算子在host执行的操作类似，只是下一个算子需要依赖上一个算子的shape，因此在获取到上一个算子的shape后，下一个算子就可以流水执行下去。
(3)执行时host侧刷新subgraph内动态shape在编译态未知部分。动态shape在编译态时，因为未知shape，所以很多执行的信息无法确定，在拿到shape后，可以将执行信息的实际值刷新到task里面。
(4)动态shape算子(下一个算子的shape依赖以前一个算子的输出值才可以确定，此时这个算子称为动态shape算子)不放在子图范围内，需要通过host和device的交互获取shape等信息。以单算子或者独立子图的方式执行。
三、动态执行
如图7所示，通过软硬件配合的方式完成子图的执行过程，软件负责子图的调度，具体流程如下：
(1)硬件将子图预加载进cache，并通知device调度软件启动执行，device调度软件读取初始就绪(前序节点计数为0)的任务。
(2)device调度软件将就绪的任务推送到硬件执行装置中执行。
(3)硬件执行装置从cache中读取任务描述符，执行就绪任务，包括调度到多个异构引擎的执行。
(4)硬件执行子图任务完成，通知mcu软件。
(5)device调度软件取当前执行完成任务的后序任务，将后序任务的前序节点计数--，当计数为0，推送该任务到硬件执行装置执行。
(6)2～5循环往复，直到子图执行完成。
对于子图编译提供的thread windows size个task任务序列，device调度在执行完某个thread task任务序列后，将整个thread替换新的thread，并修改新thread context里的thread id等信息。直至所有thread执行完成。
此外，根据上面的最佳流水调整thread subtask的Qos值，子图并发执行时，通过协调各计算资源和带宽的优先级，保证整图的高效执行，8个thread可以同时并行，这样会抢内存的带宽，QoS可以按照策略分配带宽，带宽是指计算机有很多总线，总线带宽越高，性能越好。
图8为本申请实施例的处理装置800的结构示意图。该处理装置800可以用于上文的硬件系统，其包括：获取模块801、划分模块802、编译模块803和运行模块804。其中，
获取模块801，用于获取待编译的计算图，所述计算图的输入数据采用动态shape，所述计算图包括多个算子；划分模块802，用于对所述计算图进行划分以得到多个子图，任意一个所述子图包括所述计算图中的至少一个所述算子；编译模块803，用于根据所述多个子图编译生成多个可执行任务；运行模块804，用于基于所述多个可执行任务运行所述计算图。
在一种可能的实现方式中，所述划分模块802，具体用于获取所述多个算子的切分信息，所述切分信息用于指示对应算子所支持的输入数据切分方式；根据所述多个算子的切分信息获取所述多个子图。
在一种可能的实现方式中，当第一子图包括n个算子时，所述n个算子连续排列，且所述n个算子所支持的输入数据切分方式相同，n＞1，所述第一子图为所述多个子图中的任意一个。
在一种可能的实现方式中，所述编译模块803，具体用于对所述多个子图分别进行静态编译，以得到多个线程任务；获取待处理数据；根据所述待处理数据对所述多个线程任务进行动态编译，以得到所述多个可执行任务。
在一种可能的实现方式中，所述编译模块803，具体用于获取第一子图内的引擎总数，所述第一子图为所述多个子图中的任意一个，所述第一子图包括m个算子，m≥1；根据所述第一子图内的引擎总数确定N，N＞1，N表示可并发运行的线程数量；获取N个线程任务，任意一个所述线程任务包括m个结构体，所述N个线程任务对应N个线程。
在一种可能的实现方式中，所述编译模块803，具体用于获取所述待处理数据的动态shape；根据所述待处理数据的动态shape对所述多个线程任务中的未知参数进行更新，以得到所述多个可执行任务。
在一种可能的实现方式中，所述编译模块803，还用于根据所述待处理数据的动态shape获取threadnum，threadnum表示所述待处理数据的切分数量。
在一种可能的实现方式中，所述编译模块803，具体用于将所述动态shape代入预设公式以得到所述threadnum，所述预设公式用于表示所述动态shape和所述threadnum之间的对应关系。
在一种可能的实现方式中，所述编译模块803，具体用于以所述第一子图内的引擎满负荷工作为目的获取所述threadnum。
在一种可能的实现方式中，所述编译模块803，还用于在所述第一子图的开头和结尾插入连接算子。
在一种可能的实现方式中，所述编译模块803，还用于对所述第一子图包括的所述m个算子进行优化。
在一种可能的实现方式中，所述编译模块803，还用于对所述N个线程任务进行缓存操作。
在一种可能的实现方式中，所述运行模块804，具体用于以复用可执行任务的方式，基于所述多个可执行任务调度第一子图包括的threadnum个子任务，threadnum表示所述第一子图的待处理数据的切分数量，每个所述子任务包括m个算子，m≥1，所述第一子图为所述多个子图中的任意一个。
在一种可能的实现方式中，所述运行模块804，还用于以运行总时长最短为目的，给所述多个可执行任务分配带宽。
本实施例的装置，可以用于执行图3所示方法实施例的技术方案，其实现原理和技术效果类似，此处不再赘述。
在实现过程中，上述方法实施例的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。处理器可以是通用处理器、数字信号处理器(digital signal processor,DSP)、特定应用集成电路(application-specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array,FPGA)或其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。本申请实施例公开的方法的步骤可以直接体现为硬件编码处理器执行完成，或者用编码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法的步骤。
上述各实施例中提及的存储器可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(dynamic RAM，DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。应注意，本文描述的系统和方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (31)

  1. 一种计算图的处理方法，其特征在于，包括：
    获取待编译的计算图，所述计算图的输入数据采用动态shape，所述计算图包括多个算子；
    对所述计算图进行划分以得到多个子图，任意一个所述子图包括所述计算图中的至少一个所述算子；
    根据所述多个子图编译生成多个可执行任务；
    基于所述多个可执行任务运行所述计算图。
  2. 根据权利要求1所述的方法，其特征在于，所述对所述计算图进行划分以得到多个子图，包括：
    获取所述多个算子的切分信息，所述切分信息用于指示对应算子所支持的输入数据切分方式；
    根据所述多个算子的切分信息获取所述多个子图。
  3. 根据权利要求2所述的方法，其特征在于，当第一子图包括n个算子时，所述n个算子连续排列，且所述n个算子所支持的输入数据切分方式相同，n＞1，所述第一子图为所述多个子图中的任意一个。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，所述根据所述多个子图编译生成多个可执行任务，包括：
    对所述多个子图分别进行静态编译，以得到多个线程任务；
    获取待处理数据；
    根据所述待处理数据对所述多个线程任务进行动态编译，以得到所述多个可执行任务。
  5. 根据权利要求4所述的方法，其特征在于，所述对所述多个子图分别进行静态编译，以得到多个线程任务，包括：
    获取第一子图内的引擎总数，所述第一子图为所述多个子图中的任意一个，所述第一子图包括m个算子，m≥1；
    根据所述第一子图内的引擎总数确定N，N＞1，N表示可并发运行的线程数量；
    获取N个线程任务，任意一个所述线程任务包括m个结构体，所述N个线程任务对应N个线程。
  6. 根据权利要求4或5所述的方法，其特征在于，所述根据所述待处理数据对所述多个线程任务进行动态编译，以得到所述多个可执行任务，包括：
    获取所述待处理数据的动态shape；
    根据所述待处理数据的动态shape对所述多个线程任务中的未知参数进行更新，以得到所述多个可执行任务。
  7. 根据权利要求6所述的方法，其特征在于，还包括：
    根据所述待处理数据的动态shape获取threadnum，threadnum表示所述待处理数据的切分数量。
  8. 根据权利要求7所述的方法，其特征在于，所述根据所述待处理数据的动态shape获取threadnum，包括：
    将所述动态shape代入预设公式以得到所述threadnum，所述预设公式用于表示所述动态shape和所述threadnum之间的对应关系。
  9. 根据权利要求7或8所述的方法，其特征在于，所述根据所述待处理数据的动态shape获取threadnum，包括：
    以所述第一子图内的引擎满负荷工作为目的获取所述threadnum。
  10. 根据权利要求5-9中任一项所述的方法，其特征在于，还包括：
    在所述第一子图的开头和结尾插入连接算子。
  11. 根据权利要求5-10中任一项所述的方法，其特征在于，还包括：
    对所述第一子图包括的所述m个算子进行优化。
  12. 根据权利要求5-11中任一项所述的方法，其特征在于，还包括：
    对所述N个线程任务进行缓存操作。
  13. 根据权利要求1-12中任一项所述的方法，其特征在于，所述基于所述多个可执行任务运行所述计算图，包括：
    以复用可执行任务的方式，基于所述多个可执行任务调度第一子图包括的threadnum个子任务，threadnum表示所述第一子图的待处理数据的切分数量，每个所述子任务包括m个算子，m≥1，所述第一子图为所述多个子图中的任意一个。
  14. 根据权利要求13所述的方法，其特征在于，还包括：
    以运行总时长最短为目的，给所述多个可执行任务分配带宽。
  15. 一种处理装置，其特征在于，包括：
    获取模块，用于获取待编译的计算图，所述计算图的输入数据采用动态shape，所述计算图包括多个算子；
    划分模块，用于对所述计算图进行划分以得到多个子图，任意一个所述子图包括所述计算图中的至少一个所述算子；
    编译模块，用于根据所述多个子图编译生成多个可执行任务；
    运行模块，用于基于所述多个可执行任务运行所述计算图。
  16. 根据权利要求15所述的装置，其特征在于，所述划分模块，具体用于获取所述多个算子的切分信息，所述切分信息用于指示对应算子所支持的输入数据切分方式；根据所述多个算子的切分信息获取所述多个子图。
  17. 根据权利要求16所述的装置，其特征在于，当第一子图包括n个算子时，所述n个算子连续排列，且所述n个算子所支持的输入数据切分方式相同，n＞1，所述第一子图为所述多个子图中的任意一个。
  18. 根据权利要求15-17中任一项所述的装置，其特征在于，所述编译模块，具体用于对所述多个子图分别进行静态编译，以得到多个线程任务；获取待处理数据；根据所述待处理数据对所述多个线程任务进行动态编译，以得到所述多个可执行任务。
  19. 根据权利要求18所述的装置，其特征在于，所述编译模块，具体用于获取第一子图内的引擎总数，所述第一子图为所述多个子图中的任意一个，所述第一子图包括m个算子，m≥1；根据所述第一子图内的引擎总数确定N，N＞1，N表示可并发运行的线程数量；获取N个线程任务，任意一个所述线程任务包括m个结构体，所述N个线程任务对应N个线程。
  20. 根据权利要求18或19所述的装置，其特征在于，所述编译模块，具体用于获取所述待处理数据的动态shape；根据所述待处理数据的动态shape对所述多个线程任务中的未知参数进行更新，以得到所述多个可执行任务。
  21. 根据权利要求20所述的装置，其特征在于，所述编译模块，还用于根据所述待处理数据的动态shape获取threadnum，threadnum表示所述待处理数据的切分数量。
  22. 根据权利要求21所述的装置，其特征在于，所述编译模块，具体用于将所述动态shape代入预设公式以得到所述threadnum，所述预设公式用于表示所述动态shape和所述threadnum之间的对应关系。
  23. 根据权利要求21或22所述的装置，其特征在于，所述编译模块，具体用于以所述第一子图内的引擎满负荷工作为目的获取所述threadnum。
  24. 根据权利要求19-23中任一项所述的装置，其特征在于，所述编译模块，还用于在所述第一子图的开头和结尾插入连接算子。
  25. 根据权利要求19-24中任一项所述的装置，其特征在于，所述编译模块，还用于对所述第一子图包括的所述m个算子进行优化。
  26. 根据权利要求19-25中任一项所述的装置，其特征在于，所述编译模块，还用于对所述N个线程任务进行缓存操作。
  27. 根据权利要求15-26中任一项所述的装置，其特征在于，所述运行模块，具体用于以复用可执行任务的方式，基于所述多个可执行任务调度第一子图包括的threadnum个子任务，threadnum表示所述第一子图的待处理数据的切分数量，每个所述子任务包括m个算子，m≥1，所述第一子图为所述多个子图中的任意一个。
  28. 根据权利要求27所述的装置，其特征在于，所述运行模块，还用于以运行总时长最短为目的，给所述多个可执行任务分配带宽。
  29. 一种设备，其特征在于，包括：
    一个或多个处理器；
    存储器，用于存储一个或多个程序；
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-14中任一项所述的方法。
  30. 一种计算机可读存储介质，其特征在于，包括计算机程序，所述计算机程序在计算机上被执行时，使得所述计算机执行权利要求1-14中任一项所述的方法。
  31. 一种计算机程序产品，其特征在于，所述计算机程序产品包括计算机程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行权利要求1-14中任一项所述的方法。
PCT/CN2023/102301 2022-10-19 2023-06-26 计算图的处理方法和装置 Ceased WO2024082679A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23878676.8A EP4597362A4 (en) 2022-10-19 2023-06-26 METHOD AND APPARATUS FOR PROCESSING COMPUTING GRAPHS
US19/182,292 US20250306991A1 (en) 2022-10-19 2025-04-17 Computational graph processing method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211277782.4 2022-10-19
CN202211277782.4A CN117908894A (zh) 2022-10-19 2022-10-19 计算图的处理方法和装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/182,292 Continuation US20250306991A1 (en) 2022-10-19 2025-04-17 Computational graph processing method and apparatus

Publications (1)

Publication Number Publication Date
WO2024082679A1 true WO2024082679A1 (zh) 2024-04-25

Family

ID=90684411

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/102301 Ceased WO2024082679A1 (zh) 2022-10-19 2023-06-26 计算图的处理方法和装置

Country Status (4)

Country Link
US (1) US20250306991A1 (zh)
EP (1) EP4597362A4 (zh)
CN (1) CN117908894A (zh)
WO (1) WO2024082679A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120430354A (zh) * 2025-07-03 2025-08-05 沪渝人工智能研究院 一种动静态合并的算子调度与高低位扫描轻量化加速方法
US20250291732A1 (en) * 2024-03-14 2025-09-18 Nvidia Corporation Data transfer technique

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119902771B (zh) * 2024-12-28 2025-10-24 腾讯科技（深圳）有限公司 计算图的处理方法、装置、设备及存储介质
CN120723249B (zh) * 2025-09-02 2026-01-06 北京清微智能科技有限公司 基于动态shape场景的编译方法、装置、存储介质及电子装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190303762A1 (en) * 2018-03-30 2019-10-03 Xilinx, Inc. Methods of optimization of computational graphs of neural networks
CN111338635A (zh) * 2020-02-20 2020-06-26 腾讯科技（深圳）有限公司 计算图的图编译方法、装置、设备及存储介质
CN114385181A (zh) * 2021-12-17 2022-04-22 飞腾信息技术有限公司 一种数据处理方法、装置、设备及计算机存储介质
CN114580653A (zh) * 2022-01-12 2022-06-03 阿里云计算有限公司 机器学习计算优化方法和编译器

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190303762A1 (en) * 2018-03-30 2019-10-03 Xilinx, Inc. Methods of optimization of computational graphs of neural networks
CN111338635A (zh) * 2020-02-20 2020-06-26 腾讯科技（深圳）有限公司 计算图的图编译方法、装置、设备及存储介质
CN114385181A (zh) * 2021-12-17 2022-04-22 飞腾信息技术有限公司 一种数据处理方法、装置、设备及计算机存储介质
CN114580653A (zh) * 2022-01-12 2022-06-03 阿里云计算有限公司 机器学习计算优化方法和编译器

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4597362A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20250291732A1 (en) * 2024-03-14 2025-09-18 Nvidia Corporation Data transfer technique
CN120430354A (zh) * 2025-07-03 2025-08-05 沪渝人工智能研究院 一种动静态合并的算子调度与高低位扫描轻量化加速方法

Also Published As

Publication number Publication date
EP4597362A1 (en) 2025-08-06
EP4597362A4 (en) 2026-01-14
US20250306991A1 (en) 2025-10-02
CN117908894A (zh) 2024-04-19

Similar Documents

Publication Publication Date Title
US12406488B2 (en) Neural network model training method, image processing method, and apparatus
CN110175671B (zh) 神经网络的构建方法、图像处理方法及装置
US12555362B2 (en) Neural network model training method, image processing method, and apparatus
CN111295675B (zh) 用于使用内核来处理卷积运算的设备和方法
CN111882031B (zh) 一种神经网络蒸馏方法及装置
EP4428754A1 (en) Neural network model processing method and device
US11093225B2 (en) High parallelism computing system and instruction scheduling method thereof
EP4597362A1 (en) Method and apparatus for processing computational graph
TW202026858A (zh) 在深度神經網路中利用啟動稀疏性
CN111368993A (zh) 一种数据处理方法及相关设备
CN110689115A (zh) 神经网络模型处理方法、装置、计算机设备及存储介质
CN111797983A (zh) 一种神经网络构建方法以及装置
WO2022007867A1 (zh) 神经网络的构建方法和装置
CN111488177A (zh) 数据处理方法、装置、计算机设备和存储介质
CN111783937A (zh) 一种神经网络构建方法以及系统
WO2020062299A1 (zh) 一种神经网络处理器、数据处理方法及相关设备
CN113449859A (zh) 一种数据处理方法及其装置
CN115169548B (zh) 基于张量的持续学习方法和装置
US11461662B1 (en) Compilation time reduction for memory and compute bound neural networks
WO2022156475A1 (zh) 神经网络模型的训练方法、数据处理方法及装置
CN114008636A (zh) 优化机器学习模型性能
WO2021243489A1 (zh) 一种神经网络的数据处理方法及装置
KR102714536B1 (ko) 신경 프로세싱 유닛 및 이의 동작 방법
CN114298289B (zh) 一种数据处理的方法、数据处理设备及存储介质
Zhu et al. A Wireless Collaborated Inference Acceleration Framework for Plant Disease Recognition

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23878676

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023878676

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2023878676

Country of ref document: EP

Effective date: 20250429

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 2023878676

Country of ref document: EP