WO2023035722A1 - Rdma数据传输系统、rdma数据传输方法及网络设备 - Google Patents

Rdma数据传输系统、rdma数据传输方法及网络设备 Download PDF

Info

Publication number
WO2023035722A1
WO2023035722A1 PCT/CN2022/099788 CN2022099788W WO2023035722A1 WO 2023035722 A1 WO2023035722 A1 WO 2023035722A1 CN 2022099788 W CN2022099788 W CN 2022099788W WO 2023035722 A1 WO2023035722 A1 WO 2023035722A1
Authority
WO
WIPO (PCT)
Prior art keywords
message
network device
data transmission
identifier
completion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/099788
Other languages
English (en)
French (fr)
Inventor
曲会春
邱军
武雪平
张锦彬
吴沛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP22866197.1A priority Critical patent/EP4387201A4/en
Publication of WO2023035722A1 publication Critical patent/WO2023035722A1/zh
Priority to US18/598,357 priority patent/US20240275740A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L49/00—Packet switching elements
    • H04L49/30—Peripheral units, e.g. input or output ports
    • H04L49/3036—Shared queuing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00—Digital computers in general; Data processing equipment in general
    • G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
    • G06F15/163—Interprocessor communication
    • G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
    • G06F15/17306—Intercommunication techniques
    • G06F15/17331—Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L49/00—Packet switching elements
    • H04L49/10—Packet switching elements characterised by the switching fabric construction
    • H04L49/104—Asynchronous transfer mode [ATM] switching fabrics
    • H04L49/105—ATM switching elements
    • H04L49/107—ATM switching elements using shared medium
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Definitions

  • the present application relates to the field of communication technologies, and in particular to an RDMA data transmission system, an RDMA data transmission method, and network equipment.
  • Remote direct memory access is a technology that bypasses the core of the remote host CPU to access the data in the memory of the CPU. Because it does not go through the CPU core, it not only saves a lot of CPU core resources, but also improves the system performance. The throughput reduces the network communication delay of the system, and is especially suitable for wide application in large-scale parallel computer clusters.
  • the network card of the local host When the application process in the local host executes the RDMA data transmission message, the network card of the local host reads the RDMA data transmission message from the sending queue in the memory of the host or the network card to the buffer of the network card, and then transmits the message through the network The RDMA data transfer message is sent to the network card of the peer host. Afterwards, the network card of the peer host sends feedback information to the network card of the local host, and the network card of the local host notifies the application process that the RDMA data transmission message has been processed according to the feedback information.
  • the feedback information cannot quickly notify the corresponding application process of the completion of the RDMA data transmission message.
  • Embodiments of the present application provide a data transmission system, a data transmission method, and a network device, which are used to quickly notify a corresponding application process of the completion of the RDMA data transmission message.
  • the first network device can create a shared send queue for use by multiple processes running on the first host, and the first network device can obtain the RDMA data transfer message of the first process from the shared send queue, and encapsulate the RDMA data transfer message in a message (referred to as the first message) encapsulates the first identifier corresponding to the first process at the same time, and sends the first message to the second network device through RDMA.
  • the second network device may simultaneously encapsulate the first identifier in a message encapsulating the feedback message (called the second message), and then send the second message to the first network device through RDMA.
  • the first network device does not need to determine that the feedback message in the second message corresponds to the first process according to the Identifying and determining that the feedback message corresponds to the first process is beneficial to efficiently notifying the first process that the RDMA data transmission message has been processed, thereby improving the operating efficiency of the first process.
  • the first message is obtained by encapsulating the first network device according to the RDMA protocol
  • the second message is obtained by encapsulating the second network device according to the RDMA protocol.
  • the RDMA protocol can be, for example, an infiniband protocol, or The RDMA (RDMA over Converged Ethernet version 1, RoCE1) protocol using the converged Ethernet version 1, or the RDMA (RDMA over Converged Ethernet version 2, RoCE2) protocol using the converged Ethernet version 2, or the Internet wide area RDMA protocol (internet wide area RDMA protocol, iWARP) etc.
  • the first network device may send the first packet to the second network device through the transmission channel bound to the shared sending queue.
  • the transmission channel is used to realize the network connection between the first network device and the second network device, and the transmission channel may be, for example, an input-output channel (channel-IO).
  • the second network device may also send the second message to the second network device through the transmission channel.
  • the first network device may receive the second packet from the second network device through the transmission channel.
  • the first message and the second message further include a second identifier, and the second identifier is used to determine the RDMA data transmission from multiple RDMA data transmission messages of the first process message: the first network device is specifically configured to notify the first process of completion of the RDMA data transmission message according to the first identifier, the feedback message, and the second identifier.
  • the first network device is specifically configured to notify the first process of completion of the RDMA data transmission message according to the first identifier, the feedback message, and the second identifier.
  • the shared sending queue is specifically configured to store work requests from the multiple processes
  • the first network device is specifically configured to acquire the first work request from the first process from the shared sending queue , Acquire the RDMA data transmission message according to the first work request, where the first work request describes the RDMA data transmission message.
  • the shared sending queue may be set in the memory of the first network device.
  • the shared send queue can be set in the memory of the first host.
  • this embodiment of the present application obtains the first identifier through the second message, which is beneficial to avoid reading the first work request from the shared send queue as much as possible. Fetching the work request is beneficial to save the cache resource of the first network device and reduce the delay in the process of completing the work request.
  • the first network device is specifically configured to determine a first completion queue corresponding to the first process from a plurality of completion queues according to the first identifier, and according to the feedback message in the first completion queue A work completion element is written in , and the work completion element describes the completion of the RDMA data transmission message.
  • the first identifier can be used to determine the first completion queue.
  • the first completion queue can be efficiently and accurately determined through the first identifier carried in the first message and the second message, which is conducive to efficient and accurate notification of the first completion queue.
  • the process has already processed the RDMA data transmission message, which is beneficial to improving the operating efficiency of the first process.
  • an embodiment of the present application provides an RDMA data transmission method, including: a first network device acquires an RDMA data transmission message of a first process from a shared sending queue, the first network device is set on a first host, and the The first process is any one of a plurality of processes running on the first host and using the shared sending queue; the first network device sends the first message to the second network device, and the second network The device is set on the second host, and the first message includes the RDMA data transmission message and the first identifier corresponding to the first process; the first network device receives the second message, the second message includes the first identifier and a feedback message, the feedback message indicates the completion of the RDMA data transmission message; the first network device according to the information in the second message The first identifier and the feedback message notify the first process of the completion of the RDMA data transmission message.
  • the first message is obtained by encapsulating the first network device according to the RDMA protocol
  • the second message is obtained by encapsulating the second network device according to the RDMA protocol.
  • the RDMA protocol can be, for example, an infiniband protocol, or The RDMA (RDMA over Converged Ethernet version 1, RoCE1) protocol using the converged Ethernet version 1, or the RDMA (RDMA over Converged Ethernet version 2, RoCE2) protocol using the converged Ethernet version 2, or the Internet wide area RDMA protocol (internet wide area RDMA protocol, iWARP) etc.
  • the first network device may send the first packet to the second network device through the transmission channel bound to the shared sending queue.
  • the transmission channel is used to realize the network connection between the first network device and the second network device, and the transmission channel may be, for example, an input-output channel (channel-IO).
  • the first network device may receive the second packet from the second network device through the transmission channel.
  • the first network device By carrying the first identifier in the first packet, the first network device is beneficial to instruct the second network device to carry the first identifier in the second packet. In this way, after receiving the second packet, the first network device does not need to According to the context of the shared sending queue, it is determined that the feedback message in the second message corresponds to the first process, but it can be determined according to the first identifier in the second message that the feedback message corresponds to the first process, which is conducive to efficient notification The first process has already processed the RDMA data transmission message, which is beneficial to improving the operating efficiency of the first process.
  • the first message and the second message further include a second identifier, and the second identifier is used to determine the RDMA data transmission from multiple RDMA data transmission messages of the first process message; the first network device notifies the first process of the completion of the RDMA data transmission message according to the first identifier in the second message and the feedback message includes: the first network device Specifically, it is used to notify the first process of the completion of the RDMA data transmission message according to the first identifier, the feedback message and the second identifier.
  • the first network device is beneficial to instruct the second network device to carry the second identifier in the second packet, which in turn facilitates the first network device to carry the second identifier in the second packet. To identify and efficiently and accurately notify the first process of the completion of the RDMA data transmission message.
  • the shared sending queue is specifically used to store work requests from the plurality of processes; obtaining the RDMA data transmission message of the first process from the shared sending queue by the first network device includes: the first network device Obtain a first work request from the first process from the shared sending queue, the first work request describes the RDMA data transmission message; the first network device obtains the RDMA according to the first work request Data transfer messages.
  • the shared sending queue may be set in the memory of the first network device.
  • the shared send queue can be set in the memory of the first host.
  • the first network device Compared with determining the first identifier by reading the first work request from the shared send queue according to the context of the shared send queue, in the embodiment of the present application, the first network device carries the first identifier in the first message, which is beneficial to Instruct the second network device to carry the first identifier in the second message. In this way, after receiving the second message, the first network device obtains the first identifier through the second message, which is beneficial to avoid as much as possible from the shared sending queue. Reading the work request is beneficial to save the cache resource of the first network device and reduce the delay in the process of completing the work request.
  • the first network device notifies the first process of the completion of the RDMA data transmission message according to the first identifier in the second packet and the feedback message, including: the first process A network device determines a first completion queue corresponding to the first process from multiple completion queues according to the first identifier; the first network device writes work in the first completion queue according to the feedback message A completion element, the work completion element is used to notify the first process of the completion of the RDMA data transmission message.
  • the first identifier can be used to determine the first completion queue.
  • the first network device can instruct the second network device to carry the first identifier in the second packet by carrying the first identifier in the first packet.
  • the first network device can efficiently and accurately determine the first completion queue through the first identifier carried in the second packet, which is conducive to efficiently and accurately notifying the first process that it has processed the
  • the RDMA data transfer message is beneficial to improving the operating efficiency of the first process.
  • the embodiment of the present application provides an RDMA data transmission method, including: the second network device receives the first message from the first network device, the first network device is set on the first host, and the first network device The second network device is set on the second host, the first message includes the RDMA data transmission message of the first process and the first identifier corresponding to the first process, and the RDMA data transmission message is the first network device Acquired from the shared send queue, the first process is any one of the multiple processes running on the first host that use the shared send queue; the second network device sends the message to The first network device sends a second message, the second message includes the first identifier and a feedback message, and the first identifier and the feedback message indicate that the first network device notifies the first The completion status of the RDMA data transfer message described by the process.
  • the second network device After the second network device parses the first identifier in the first packet, it can carry the first identifier in the second packet. In this way, after receiving the second packet, the first network device does not need to The context determines that the feedback message in the second message corresponds to the first process, but it can be determined that the feedback message corresponds to the first process according to the first identifier in the second message, which is conducive to efficiently notifying the first process that it has been processed
  • the RDMA data transmission message is beneficial to improving the operating efficiency of the first process.
  • the first message is obtained by encapsulating the first network device according to the RDMA protocol
  • the second message is obtained by encapsulating the second network device according to the RDMA protocol.
  • the RDMA protocol can be, for example, an infiniband protocol, or The RDMA (RDMA over Converged Ethernet version 1, RoCE1) protocol using the converged Ethernet version 1, or the RDMA (RDMA over Converged Ethernet version 2, RoCE2) protocol using the converged Ethernet version 2, or the Internet wide area RDMA protocol (internet wide area RDMA protocol, iWARP) etc.
  • the first packet is sent by the first network device to the second network device through the transmission channel bound to the shared sending queue.
  • the transmission channel is used to realize the network connection between the first network device and the second network device, and the transmission channel may be, for example, an input-output channel (channel-IO).
  • the second network device may also send the second message to the second network device through the transmission channel.
  • the first message and the second message further include a second identifier, and the second identifier is used to determine the RDMA data transmission from multiple RDMA data transmission messages of the first process message, the first identifier, the feedback message and the second identifier instruct the first network device to notify the first process of the completion of the RDMA data transmission message.
  • the second network device After the second network device parses the second identifier in the first packet, it can carry the second identifier in the second packet, so that after receiving the second packet, the first network device Carrying the second identifier in the message is helpful for the first network device to efficiently and accurately notify the first process of the completion of the RDMA data transmission message.
  • the embodiment of the present application provides a network device, including: an acquisition unit configured to acquire an RDMA data transmission message of a first process from a shared sending queue, the network device is set on a first host, and the first The process is any one of multiple processes running on the first host and using the shared sending queue; the sending unit is configured to send the first message to the second network device, and the second network device is set at On the second host, the first message includes the RDMA data transmission message and the first identifier corresponding to the first process; the receiving unit is configured to receive the second message from the second network device, the The second message includes the first identifier and a feedback message, where the feedback message indicates the completion of the RDMA data transmission message; a completion unit, configured to, according to the first identifier and the feedback message in the second message, The feedback message notifies the first process of the completion of the RDMA data transmission message.
  • the first message is obtained by encapsulating the first network device according to the RDMA protocol
  • the second message is obtained by encapsulating the second network device according to the RDMA protocol.
  • the RDMA protocol can be, for example, an infiniband protocol, or The RDMA (RDMA over Converged Ethernet version 1, RoCE1) protocol using the converged Ethernet version 1, or the RDMA (RDMA over Converged Ethernet version 2, RoCE2) protocol using the converged Ethernet version 2, or the Internet wide area RDMA protocol (internet wide area RDMA protocol, iWARP) etc.
  • the sending unit may send the first packet to the second network device through the transmission channel bound to the shared sending queue.
  • the transmission channel is used to realize the network connection between the first network device and the second network device, and the transmission channel may be, for example, an input-output channel (channel-IO).
  • the receiving unit may receive the second message through the transmission channel.
  • the first message and the second message further include a second identifier, and the second identifier is used to determine the RDMA data transmission from multiple RDMA data transmission messages of the first process message; the completion unit is specifically configured to notify the first process of the completion of the RDMA data transmission message according to the first identifier, the feedback message and the second identifier.
  • the completion unit is specifically configured to: determine a first completion queue corresponding to the first process from a plurality of completion queues according to the first identifier; A work completion element is written, and the work completion element is used to notify the first process of the completion of the RDMA data transmission message.
  • the embodiment of the present application provides a network device, including: a receiving unit, configured to receive a first message from a first network device, the first network device is set on a first host, and the network device Set on the second host, the first message includes the RDMA data transmission message of the first process and the first identifier corresponding to the first process, and the RDMA data transmission message is sent by the first network device from the shared acquired by the queue, the first process is any one of multiple processes running on the first host that use the shared sending queue; the sending unit is configured to send the first message to the first The network device sends a second message, where the second message includes the first identifier and a feedback message, and the first identifier and the feedback message indicate that the first network device notifies the first process of the RDMA Completion of the data transfer message.
  • the first message is obtained by encapsulating the first network device according to the RDMA protocol
  • the second message is obtained by encapsulating the second network device according to the RDMA protocol.
  • the RDMA protocol can be, for example, an infiniband protocol, or The RDMA (RDMA over Converged Ethernet version 1, RoCE1) protocol using the converged Ethernet version 1, or the RDMA (RDMA over Converged Ethernet version 2, RoCE2) protocol using the converged Ethernet version 2, or the Internet wide area RDMA protocol (internet wide area RDMA protocol, iWARP) etc.
  • the first packet may be sent by the first network device to the second network device through the transmission channel bound to the shared sending queue.
  • the transmission channel is used to realize the network connection between the first network device and the second network device, and the transmission channel may be, for example, an input-output channel (channel-IO).
  • the receiving unit may receive the first message through the transmission channel.
  • the sending unit may send the second packet to the second network device through the transmission channel.
  • the first message and the second message further include a second identifier, and the second identifier is used to determine the RDMA data transmission from multiple RDMA data transmission messages of the first process message, the first identifier, the feedback message and the second identifier instruct the first network device to notify the first process of the completion of the RDMA data transmission message.
  • the present application provides a computing device, the computing device includes a processor and a memory, the processor is coupled to the memory, the memory is used to store program codes, and when the processor executes the program codes stored in the memory, The method described in the second aspect or any possible implementation manner of the second aspect or the third aspect or any possible implementation manner of the third aspect can be executed.
  • the computing device may further include a communication interface, through which the processor can, for example, receive or send messages.
  • the seventh aspect of the present application provides a chip system, the chip system includes a processor and an interface circuit, the processor is coupled with the memory through the interface circuit, and the processor is used to execute the program code in the memory to implement the second aspect Or any possible implementation of the second aspect, or the third aspect, or the method described in any possible implementation of the third aspect.
  • the system-on-a-chip may consist of chips, or may include chips and other discrete devices.
  • the eighth aspect of the present application provides a computer-readable storage medium, the computer-readable storage medium stores program codes, and when these program codes are run on a computer device, the computer device executes the second aspect or the first aspect of the present application. Any possible implementation of the second aspect or the third aspect or the method described in any possible implementation of the third aspect.
  • the ninth aspect of the present application provides a computer program product.
  • the program code contained in the computer program product is executed by a computer device, it can execute the second aspect or any possible implementation of the second aspect or the third aspect or the third aspect.
  • each device provided in this application can be used to execute the above-mentioned corresponding method
  • the technical effect obtained by each device of this application can refer to the technical effect obtained by the above-mentioned corresponding method, and will not be repeated here.
  • FIG. 1A, FIG. 1B and FIG. 1C respectively show a possible application scenario of the embodiment of the present application
  • FIG. 2A and FIG. 2B respectively show a possible structure of the computer system
  • Fig. 3 shows another possible structure of the computer system of the embodiment of the present application
  • Fig. 4 shows a possible structure of the first work request
  • FIG. 5A and FIG. 5B respectively show the processing procedure of the RDMA data transmission message by the first network device
  • FIG. 6A shows the transmission process of the first message and the second message corresponding to the sending operation
  • FIG. 6B shows the transmission process of the first message and the second message corresponding to the write operation
  • FIG. 6C shows the transmission process of the first message and the second message corresponding to the atomic operation
  • FIG. 6D shows the transmission process of the first message and the second message corresponding to the read operation
  • FIG. 7 shows a possible flow of the RDMA data transmission method of the embodiment of the present application.
  • FIG. 8 shows a possible structure of a network device according to an embodiment of the present application.
  • FIG. 9 shows a possible structure of a network device in the embodiment of the present application.
  • Fig. 1A shows a possible application scenario of the embodiment of the present application.
  • the computer system shown in FIG. 1A includes at least node #1 and node #2.
  • Node #1 or Node #2 may be any device connected to one or more links of a communication network and forming the origin and/or destination of data transmission messages (messages for short) within the communication network.
  • One or more application processes (referred to as processes) can run in node #1 or node #2.
  • FIG. 1A takes node #1 running process #1 and process #2 as an example, and node #2 running process #3 as an example. It is understood that node #1 or node #2 may run more or fewer processes. Taking node #1 as an example, the message sent by node #1 can come from process #1 or process #2, and the message received by node #1 can go to process #1 or process #2.
  • Node #1 and node #2 can communicate through the data transmission system.
  • the data transmission system shown in FIG. 1A may at least include network device #1 in node #1 and network device #2 in node #2.
  • network device #1 or network device #2 may be a network adapter (network interface card, NIC) or a network card.
  • the data transmission system shown in FIG. 1A may be a remote direct memory access (remote direct memory access, RDMA) system, and the messages transmitted in the data transmission system may be RDMA data transmission messages.
  • RDMA remote direct memory access
  • the data transmission system can create a data transmission channel (channel) based on processes run by nodes. Assuming that process #1 and process #2 need to send messages to process #3 respectively, network device #1 and network device #2 need to create data transmission channels for process #1 and process #2, respectively, as shown in Figure 1A p1-p3 and channels p2-p3. The message of process #1 is sent to process #3 through channel p1-p3, and the message of process #2 is sent to process #3 through channel p2-p3.
  • the data transmission system shown in FIG. 1A needs to transmit messages for a large number of processes, and correspondingly needs to create a large number of data transmission channels.
  • the computer system shown in FIG. 1A includes n nodes, and each node runs m processes, then for any node, the data transmission system needs to create m*n data transmission channels. Since each data transmission channel needs to create at least one queue pair (queue pair, QP) in the corresponding node, and a large number of QPs will occupy too much storage resources of the node, therefore, the data transmission system creates data transmission channels based on the processes running on the nodes Will affect the scalability of the data transmission system.
  • queue pair queue pair
  • FIG. 1B shows another possible application scenario of the embodiment of the present application.
  • the computer system shown in Fig. 1B can be understood with reference to the computer system shown in Fig. 1A, but different from Fig. 1A, in the computer system shown in Fig. 1B, the data transmission system is based on the nodes in the computer system to create data transmission channel. Still assuming that process #1 and process #2 need to send messages to process #3 respectively, in the data transmission system shown in Figure 1B, network device #1 and network device #2 create a data transmission for node #1 and node #2
  • the channels are, for example, channels n1-n2 shown in FIG. 1B. Both process #1's message and process #2's message are sent to process #3 through channels n1-n2.
  • the data transmission system needs to create n-1 data transmission channels at most.
  • the data transmission system shown in Figure 1A by reducing the number of data transmission channels, it is beneficial to reduce the storage resources occupied by QP. It can be seen that creating data transmission channels based on nodes in the data transmission system will reduce the scalability of the data transmission system. sexual influence.
  • FIG. 1C shows another possible application scenario of the embodiment of the present application.
  • the computer system shown in FIG. 1C not only includes node #1 and node #2, but also includes node #3 and node #4.
  • the data transmission system shown in Figure 1C creates data transmission channels based on nodes, for example, channels n1-n2, channels n1-n3, channels n1-n4, channels n2-n3, channels n2-n4 and channel n3 shown in Figure 1C -n4.
  • the data transmission system creates three data transmission channels for it.
  • computer system can comprise node #1 and node #2 at least, and node #1 comprises host computer 11, supports the network adapter (network interface card that supports RDMA, RNIC) 12 and memory 13 of RMDA, and node #2 comprises host computer 21 and RNIC 22.
  • RNIC 22 can communicate with host 21, and RNIC 12 is connected with host 11 and memory 13 respectively.
  • RNIC 12 among Fig. 2A can be regarded as Fig. 1B or network device #1 in Fig. 1C, perhaps, RNIC 12 and memory 13 among Fig. 2A can be regarded as Fig. 1B or network device #1 among Fig. 1C.
  • RNIC 22 among Fig. 2A can be regarded as Fig. 1B or network device #2 among Fig. 1C.
  • Fig. 2A takes the network device #1 shown in Fig. 1B or Fig. 1C to include RNIC 12, and the network device includes RNIC 22 as an example, the embodiment of the present application does not limit the network device #1 or network device #2 shown in Fig. 1B or Fig. 1C specific type of .
  • the host 11 or the host 21 may include: a processor, a communication interface, and a memory, and the processor, the communication interface, and the memory are connected to each other through an internal bus.
  • the processor may be composed of one or more general-purpose processors, such as a central processing unit (central processing unit, CPU), or a combination of a CPU and a hardware chip.
  • the memory of the host 11 or the host 21 may store codes of system applications and/or application processes, and the processor may execute the codes to realize the functions of the CPU core 113 and/or the functions of the processes.
  • the RNIC 12 may include a processor 122 and a cache 121
  • the RNIC 22 may include a processor 222 and a cache 221.
  • the processor 122 or the processor 222 may be composed of one or more general-purpose processors, such as a CPU, or a combination of a CPU and a hardware chip.
  • the processor 122 and the cache 121 may be connected through a bus (not shown in FIG. 2A ), or may be connected in other ways.
  • the processor 222 and the cache 221 may be connected through a bus (not shown in FIG. 2A ), or may be connected in other ways.
  • the aforementioned hardware chip may be an application-specific integrated circuit (application-specific integrated circuit, ASIC), a programmable logic device (programmable logic device, PLD) or a combination thereof.
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • the aforementioned PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a general array logic (generic array logic, GAL) or any combination thereof.
  • memory or memory 13 can comprise volatile memory (volatile memory), such as random access memory (random access memory, RAM); Memory or memory 13 can also comprise nonvolatile memory (non-volatile memory), such as Read-only memory (read-only memory, ROM), flash memory (flash memory), hard disk (hard disk drive, HDD) or solid-state drive (solid-state drive, SSD) or double rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDR SDRAM); The memory or memory 13 can also include a combination of the above types. Among them, DDR SDRAM can be referred to as DDR for short.
  • Cache 121 or cache 221 may be one or more levels of cache, for example, may be implemented using registers and/or static random access memory (static random access memory, SRAM).
  • the RNIC 12 and the memory 13 may be integrated on the same chip, or in other words, the memory 13 may be the memory in the network device #1 corresponding to the RNIC 12.
  • the memory 13 may be the memory in the network device #1 corresponding to the RNIC 12.
  • DDR DDR
  • this DDR can support multi-channel technology, and RNIC 12 can access memory 13 through multiple channels.
  • the memory 13 may be a memory in the host 11.
  • the RNIC 12 and the host 11 can be connected through an input-output (IO) interface, and the IO interface can include but not limited to IO fabric interfaces such as peripheral component interconnect express (PCIe). If the memory 13 is the memory in the host computer 11, then the RNIC 12 can access the memory memory 13 through the IO interface with the host computer 11.
  • IO input-output
  • PCIe peripheral component interconnect express
  • RNIC 12 can create at least one shared sending queue (Shard Send Queue, SSQ) of binding channel n1-n2 in memory 13, and SSQ can be used for processing from multiple processes in node #1, to be transmitted to node #2 information.
  • SSQ can be used for processing from multiple processes in node #1, to be transmitted to node #2 information.
  • Fig. 2A exemplarily shows SSQ#1 among the at least one SSQ.
  • the at least one SSQ also includes other SSQs (such as SSQ#2) except SSQ#1, SSQ#1 and SSQ#2 may correspond to different priorities, and RNIC 12 may transmit to node 2 according to The priority of the message uses the corresponding SSQ.
  • SSQ#2 other SSQs
  • Host 11 and/or host 21 may run one or more processes.
  • FIG. 2A schematically shows two processes (process #1 and process #2) running on the host 11 and two processes (process #3 and process #4) running on the host 21.
  • the message to be transmitted by RNIC 12 to node 2 may come from process #1 and/or process #2.
  • process #1 and process #2 can submit corresponding work requests (work request, WR) to RNIC 12 respectively.
  • a message to be sent by process #1 is called M1
  • a message to be sent by process #2 is called M2
  • the WR corresponding to M1 is called WR1
  • the WR corresponding to M2 is called WR2.
  • process #1 and process #2 can call the program interface in the host 11 and the driver program of RNIC 12 to submit WR1 and WR2 to RNIC 12 respectively.
  • M1 and M2 may be RDMA messages.
  • WQE work queue element 1 and WQE2, respectively.
  • WQE1 and WQE2 describe M1 and M2 respectively.
  • WQE1 includes storage address 1 of M1 in memory 111
  • WQE2 includes storage address 2 of M2 in memory 13 .
  • the memory 13 also includes a context of SSQ#1 (context S1 for short), and context S1 records usage information of SSQ#1.
  • the context S1 may include a write index (PI of SSQ#1 for short) and a read index (CI of SSQ#1 for short) of SSQ#1.
  • RNIC 12 can determine the write position of WQE in SSQ#1 according to the value of PI of SSQ#1, and determine the read position of WQE in SSQ#1 according to the value of CI of SSQ#1.
  • Fig. 2A exemplarily shows three storage positions for placing WQEs in SSQ#1, SSQ#1 may also include more or less WQEs, and, for ease of understanding, Fig. 2A uses SSQ#1 for The three storage locations holding the WQE are drawn as three rectangles arranged sequentially from top to bottom.
  • the three storage locations used to store the WQE in SSQ#1 shown in FIG. 2A are respectively called storage location T, storage location M, and storage location B from top to bottom.
  • the embodiment of the present application does not limit the storage manner of each WQE in SSQ#1.
  • SSQ#1 may correspond to a continuous storage space in the memory 13, or may correspond to a discrete storage space in the memory 13.
  • the RNIC 12 after the RNIC 12 receives WR1, it can point to the storage location T of SSQ#1 according to the PI of SSQ#1 in the context S1, and then write WQE1 into the storage location T, and update the value of the PI of SSQ#1, Make it point to memory location M of SSQ#1.
  • RNIC 12 After receiving WR2, RNIC 12 can write WQE2 into storage location M according to the value of PI of SSQ#1 in context S1, and update the value of PI of SSQ#1 to point to storage location B of SSQ#1.
  • Fig. 2A shows the storage location pointed to by CI and PI in memory 13 after RNIC 12 writes WQE1 and WQE2 in SSQ#1.
  • SSQ#1 is used to sequentially process WQEs written in SSQ#1. Assuming that WQE1 is written to SSQ#1 before WQE2, then the value of CI of SSQ#1 can point to WQE1 first, and after RNIC 12 processes WQE1, update the value of CI of SSQ#1 to point to WQE2 (refer to Figure 2B) , so that RNIC 12 continues to process WQE2.
  • Step 1.RNIC 12 reads WQE1 in SSQ#1 into cache 121;
  • RNIC 12 can determine the value of the CI of SSQ#1 according to the context S1, determine that CI points to the storage position T in SSQ#1 according to the value of CI, and then take out WQE (that is, WQE1) from storage position T.
  • WQE that is, WQE1
  • Step 2.RNIC 12 accesses memory location 1 indicated by WQE1 in memory 111;
  • WQE1 may include storage address 1 of M1 in memory 111. After RNIC 12 reads WQE1 into cache 121, it may resolve storage address 1 from WQE1, and then may access storage location 1 in memory 111.
  • Step 3.RNIC 12 reads M1 in storage location 1 to cache 121;
  • the RNIC 12 can read the data in the storage location 1 (that is, M1) to the cache 121.
  • Step 4.RNIC 12 encapsulates M1 as message 1, and message 1 is sent to RNIC 22 through channel n1-n2;
  • RNIC 12 may encapsulate M1 into message 1 according to the RDMA protocol, and send message 1 to RNIC 22 through channels n1-n2 bound to SSQ#1.
  • the RDMA protocol may be, for example, a wireless bandwidth (infiniband) protocol, or an RDMA (RDMA over Converged Ethernet version 1, RoCE1) protocol using Converged Ethernet version 1, or an RDMA (RDMA over Converged Ethernet version 1, RoCE1) protocol using Converged Ethernet version 2. Converged Ethernet version2, RoCE2) protocol, or Internet wide area RDMA protocol (internet wide area RDMA protocol, iWARP), etc.
  • RNIC 22 After receiving message 1, RNIC 22 can decapsulate message 1 to obtain M1.
  • Step 5.RNIC 22 stores M1 in the host computer 21;
  • the RNIC 22 can store M1 in the host 21. Further, assuming that M1 is used to write into the memory 211 of process #3, as an example, after the RNIC 22 decapsulates the message 1, it can also obtain the writing position of M1 in the memory 211, and the RNIC 22 can use the writing position Write M1 into memory 211 .
  • Step 4 and step 5 will be introduced in detail later in combination with the RDMA send model, write model, read model, and atomic model, which will not be introduced here.
  • the computer system shown in FIG. 2B describes the process of the data transmission system provided by the embodiment of the present application notifying process #1 of the completion of M1 through the dotted line representing the data transmission process and its serial number.
  • the structure in FIG. 2B can be interpreted as the corresponding structure in the computer system shown in FIG. 2A , which will not be repeated here.
  • RNIC 12 can create a complete queue (complete queue, CQ) in memory 13 for process #1 and process #2 respectively.
  • CQ#1 is used to feed back to process #1 the completion of the message (such as M1) submitted by process #1 to RNIC 12
  • CQ#2 is used to feed back to process #2 the message (such as M2) submitted by process #2 to RNIC 12 completion status.
  • Step 6.RNIC 22 sends message 2 carrying R1 to RNIC 12 through channel n1-n2;
  • RNIC 22 After RNIC 22 receives message 1, it can send message 2 to RNIC 12 through channel n1-n2.
  • Message 2 carries a feedback message (abbreviated as R1), and R1 describes the completion of the message.
  • R1 describes the completion of the message.
  • the completion of the message can be that the RNIC 22 successfully receives the message, or the RNIC 22 successfully writes the message to the host 21, or the RNIC 22 does not receive the message, or the message write fails.
  • RNIC 12 After receiving message 2, RNIC 12 can decapsulate message 2 to obtain R1.
  • RNIC 12 can determine the completion of the message described by R1.
  • R1 is a certain field (referred to as field R) in the message 2, and different values of the field R correspond to different completion states of the message.
  • the RNIC 12 can determine that the completion of the message is successful, for example, the RNIC 22 successfully receives the message, and when the value of the field R is 1, the RNIC 12 can determine that the completion of the message is failure , specifically for example, the RNIC 22 has not received the message.
  • Step 7.RNIC 12 reads WQE1 in SSQ#1 into cache memory 121, to obtain the mark of CQ#1;
  • RNIC 12 After RNIC 12 obtains message 2 from channel n1-n2, although it can be determined that R1 in message 2 corresponds to WQE in SSQ#1, because SSQ#1 corresponds to multiple processes running on host 11, RNIC 12 cannot Determine which process the completion status corresponding to R1 corresponds to, and then it is impossible to confirm which CQ should handle the R1. Therefore, after confirming that the SSQ of the bound channel n1-n2 is SSQ#1, the RNIC 12 can determine that R1 corresponds to WQE1 in SSQ#1 according to the context S1 of SSQ#1. In this embodiment of the present application, the identifier of CQ#1 may be carried in WQE1.
  • RNIC 12 reads WQE1 from SSQ#1 into cache 121, it can parse WQE1 to obtain the identifier of WQE1CQ#1, so as to determine that the completion status corresponding to R1 needs to be handed over to CQ#1 for processing.
  • Step 8.RNIC 12 writes CQE1 in CQ#1 according to the identification of CQ#1 and R1;
  • Step 9.RNIC 12 processes the CQE in CQ#1, and when processing CQE1, notifies process #1 of the completion of M1.
  • Step 8 and Step 9 are introduced below.
  • the RNIC 12 After the RNIC 12 obtains the identifier of CQ#1, it can write completion queue element (CQ element, CQE) 1 into CQ#1 according to R1, and CQE1 is used to determine the completion status of M1. During the process of processing the CQE in CQ#1, the RNIC 12 can notify the process #1 of the completion of M1 when CQE1 is processed.
  • CQ element CQE
  • RNIC 12 can refer to that process #1 notifies the completion of M1 to process #1, and the driver program of process #1 calling program interface and RNIC 12 retrieves the CQE in CQ#1, and when retrieving CQE1, process #1 can Get the completion status of M1.
  • the CQE indicates the completion of the corresponding message through a part of the field carried (for example, called an error code).
  • an error code For example, when process #1 parses CQE1 and determines that the value of the error code in CQE1 is 0, process #1 may determine that the completion of M1 is successful, for example, the data transmission task corresponding to M1 has been completed. For example, when process #1 parses CQE1 and determines that the value of the error code in CQE1 is 1, process #1 may determine that the completion of M1 is a failure, for example, the data transmission task corresponding to M1 is not completed.
  • RNIC 12 may determine the value of the error code in CQE1 according to R1.
  • the completion status described by the value of the error code in CQE1 is consistent with the completion status described by R1. For example, if R1 describes the completion of the message as successful, then the value of the error code in CQE1 can be 0; if R1 describes the completion of the message as failure, then the value of the error code in CQE1 can be 1. Or, optionally, the completion status described by the value of the error code in CQE1 is inconsistent with the completion status described by R1. For example, if R1 describes the completion of the message as success, but RNIC 12 may not correctly encapsulate message 1 (M1 error or destination address error, etc.) due to failure, then the value of the error code in CQE1 can be 1.
  • the following example introduces the process of RNIC 12 writing CQE and reading CQE in CQ#1.
  • the memory 13 also includes the context of CQ#1 (context C1 for short), and the context C1 records the usage information of CQ#1.
  • the context C1 may include the writing of CQ#1 Index (referred to as PI of CQ#1) and read index (referred to as CI of CQ#1).
  • RNIC 12 can determine the writing position of CQE in CQ#1 according to the value of PI, and determine the reading position of CQE in CQ#1 according to the value of CI.
  • Figure 2B only exemplarily shows three storage locations for placing CQEs in CQ#1, and CQ#1 may also include more or less CQEs, and, for ease of understanding, Figure 2B uses CQEs in CQ#1
  • the three storage locations where the CQE is saved are drawn as three rectangles arranged in sequence from top to bottom.
  • the three storage locations used to store the CQE in CQ#1 shown in FIG. 2B are respectively called storage location T, storage location M, and storage location B from top to bottom.
  • the embodiment of the present application does not limit the storage manner of each CQE in CQ#1.
  • CQ#1 may correspond to a continuous storage space in the memory 13, or may correspond to a discrete storage space in the memory 13.
  • the RNIC 12 after the RNIC 12 generates CQE4, it can determine that the PI of CQ#1 points to the storage location T of CQ#1 according to the context C1, then write CQE4 in the storage location T, and update the fetching of the PI of CQ#1 value so that it points to memory location M of CQ#1. Afterwards, after RNIC 12 generates CQE3, it can write CQE3 in storage location M according to the value of PI of CQ#1 in context C1, and update the value of PI of CQ#1 so that it points to storage location B.
  • Fig. 2B shows the storage location pointed to by CI and PI in memory 13 after RNIC 12 writes CQE4 and CQE2 in CQ#1.
  • CQ#1 is used to sequentially process the CQEs written in CQ#1.
  • RNIC 12 After RNIC 12 generates CQE1, it writes CQE1 from the value of context C1 to storage location B of CQ#1 according to the identifier of CQ#1. After RNIC 12 processes CQE4 and CQE3 in turn, update the value of CI of CQ#1 to point to storage location B of CQ#1, so that RNIC 12 continues to process CQE3 in storage location B of CQ#1, and sends to process # 1 Notify the completion of M1.
  • step 6 The data transmission process corresponding to step 1 to step 9 is introduced above with reference to FIG. 2A and FIG. 2B .
  • step 6 because the SSQ bound by channel n1-n2 preserves the WQEs of multiple processes in the host 11, RNIC 12 cannot determine which process in the host 11 the message 2 returned by channel n1-n2 corresponds to. It is also impossible to determine which CQ in memory 13 needs to be used to process R1 in message 2.
  • RNIC 12 needs to obtain the identification of corresponding CQ (namely the identification of CQ#1) through step 7, specifically, first confirm that the SSQ of binding channel n1-n2 is SSQ#1, then according to the context S1 of SSQ#1 Determine that R1 corresponds to WQE1 in SSQ#1, read WQE1 from SSQ#1 into cache 121, and parse WQE1 to obtain the identifier of CQ#1. Since step 7 includes the process of RNIC 12 reading WQE1 from memory 13, step 7 increases the read delay of RNIC 12 to memory 13, reducing the efficiency of RNIC 12 notifying process #1 of the completion of M1. Moreover, due to the limited space of the cache 121, reading WQE1 from the memory 13 into the cache 121 may cause the WQE to be sent prefetched into the cache 121 to be aged and deleted, resulting in a delay in sending services.
  • this embodiment of the application optimizes part of the content in steps 1 to 9.
  • the optimization scheme is introduced in detail below.
  • step 4 Before step 4 is optimized, message 1 includes M1. After step 4 is optimized, refer to the content in brackets in step 4 in FIG. 2A , and add identification information in message 1 on the basis of M1. Optionally, the identification information may be the identification of CQ#1 and/or the identification of process #1.
  • step 6 Before step 6 is optimized, message 2 includes R1. After step 6 is optimized, refer to the content in brackets in step 6 in FIG. 2B, and add all or part of the identification information carried in message 1 to message 2.
  • Step 7 is not performed. Since the message 2 carries the identifier of CQ#1, referring to the “ ⁇ ” on the dotted line corresponding to step 7 in 2B, the RNIC 12 does not need to perform step 7 to obtain the identifier of CQ#1.
  • FIG. 3 shows another possible embodiment of the computer system of the present application.
  • the computer system 3 may include a data transmission system 31 , a first host 32 and a second host 33 .
  • the data transmission system 31 includes a first network device 311 and a second network device 312, the first network device 311 is set on the first host 32, and the second network device 312 is set on the second host.
  • the data transmission system 31 may be an RDMA data transmission system, and an RDMA data transmission message may be transmitted between the first network device 311 and the second network device 312 .
  • the computer system 3 can be interpreted as the computer system shown in FIG. 1B or 1C or 2A or 2B.
  • the data transmission system 31 may be interpreted as the data transmission system shown in FIG. 1B or FIG. 1C or FIG. 2A or FIG. 2B.
  • the first host 32 may be interpreted as including the host 11 shown in FIG. 2A or FIG. 2B .
  • the second host 33 can be interpreted as including the host 21 shown in FIG. 2A or FIG. 2B .
  • the first network device 311 may be interpreted as including the network device #1 shown in FIG. 1B or 1C, or as including the RNIC 12 shown in FIG. 2A or 2B, or as including the network device #1 shown in FIG. 2A or 2B. RNIC 12 and memory 13 shown.
  • the second network device 312 may be interpreted as including the network device #2 shown in FIG. 1B or 1C, or as including the RNIC 22 shown in FIG. 2A or 2B.
  • the first host 32 may run one or more processes, and the first network device 311 may create SSQs used by multiple processes in the one or more processes.
  • one or more processes run by the first host 32 may include, for example, process #1 and process #2, and the SSQ created by the first network device 311 may be, for example, a memory SSQ #1 in 13.
  • the SSQ may be set in the memory of the first network device, or may be set in the memory of the first host.
  • the SSQ can be set in the memory 13 shown in FIG. 2A or FIG. 2B.
  • one of the multiple processes using the SSQ is referred to as a first process.
  • the first process may be interpreted as process #1 in the embodiment corresponding to FIG. 2A or FIG. 2B .
  • the following takes the process of the data transmission system 31 processing the data transmission message of the first process as an example to introduce the functions of the first network device 311 and the second network device 312 in the data transmission system 31 .
  • the first network device 311 may be configured to obtain the data transmission message of the first process from the SSQ.
  • the data transmission message may be interpreted as M1 in the embodiment corresponding to FIG. 2A or FIG. 2B .
  • the SSQ is specifically used to save work requests from multiple processes, and the first network device 311 obtaining a data transmission message from the SSQ may specifically refer to that the first network device 311 is used to obtain the first process request from the first process from the SSQ.
  • a work request and then obtain a data transmission message according to the first work request.
  • the first work request describes a data transfer message.
  • the work requests from multiple processes stored in the SSQ can be understood with reference to WQE1 and WQE2 shown in FIG. 2A or FIG. 2B respectively, and the first work request can be understood with reference to WQE1 shown in FIG. 2A or FIG. 2B .
  • Fig. 4 shows a possible structure of the first work request.
  • the first work request may include a control field, a task field, and a memory description field.
  • the task field may be used to describe the format information of the first work request.
  • the task field may include indication information for the first network device 311 to process the data transmission message.
  • the task field may include identification information.
  • the identification information may include a first identification corresponding to the first process.
  • the first identifier in the identifier information may be interpreted as the identifier of CQ#1 and/or the identifier of process #1 in the embodiment corresponding to FIG. 2A or FIG. 2B .
  • the identification information may also include other information, and the possible content of the other information will be introduced later, which will not be introduced here.
  • the first work request may also include a memory description field.
  • the memory description field may be used to describe the memory space registered by the first network device 311 and/or the second network device 312 .
  • the first network device 311 may obtain the data transmission message from the first host 32 according to the memory description field.
  • the memory description field may include an address field, and the address field may be used to determine the starting location of the memory space.
  • the memory description field may further include a length (length) field, which is used to determine the length of the memory space.
  • the memory description field may also include a key (key) field, which is used to uniquely identify the memory space.
  • the first network device 311 may also encapsulate the first packet according to the data transmission message and the identification information, and then send the first packet to the second network device.
  • the first network device 311 may encapsulate the first packet according to the RDMA protocol.
  • the first packet may be interpreted as packet 1 in the optimized embodiment corresponding to FIG. 2A or FIG. 2B , that is, the first packet may include a data transmission message and identification information.
  • the second network device 312 may be configured to receive the first packet, and decapsulate the first packet to obtain the data transmission message and identification information. Afterwards, the second network device 312 may generate a feedback message according to the data transmission message to indicate the completion of the data transmission message. Optionally, after obtaining the data transmission message, the second network device 312 generates a feedback message to notify the first network device 311 that the data transmission message has been successfully received. Alternatively, the second network device 312 may generate a corresponding feedback message according to whether the data transmission message is successfully written to the second host 33 .
  • the completion status indicated by the feedback message may be success; if the data transmission message is failed to be written to the second host 33, the completion status indicated by the feedback message may be failure.
  • the feedback message may be interpreted as R1 in the embodiment corresponding to FIG. 2B .
  • the second network device 312 may also be configured to encapsulate the second packet according to the feedback message and the identification information, and send the second packet to the first network device 311 .
  • the second message may be interpreted as message 2 in the optimized embodiment corresponding to FIG. 2B , that is, the second message may include the first identifier and the feedback message.
  • the first network device 311 may also be configured to receive the second packet, and decapsulate the second packet to obtain the feedback message and identification information. Afterwards, the first network device 311 may notify the first process of the completion of the data transmission message according to the identification information and the feedback message in the second packet.
  • the first network device 311 may create multiple CQs for multiple processes running on the first host 32, each process corresponds to a part of the CQs (for example, a CQ) in the multiple CQs, and each CQ is used to send information to the corresponding process Notify the completion of the message.
  • the CQ corresponding to the first process among the multiple CQs is called the first CQ.
  • the CQ created by the first network device 311 may be interpreted as CQ#1 or CQ#2 in the embodiment corresponding to FIG. 2B .
  • the first network device 311 may create CQ#1 for process #1, and create CQ#2 for process #2.
  • CQ#1 is used to notify process #1 of the completion of the message submitted by process #1
  • CQ#2 is used to notify process #2 of the completion of the message submitted by process #2.
  • the first network device 311 notifies the first process of the completion of the data transmission message according to the first identifier in the second message and the feedback message may refer to that the first network device 311 determines the CQ corresponding to the first identifier from multiple CQs (called as the first CQ), write the CQE (called the first CQE) in the first CQ according to the feedback message, and the first CQE describes the completion of the data transmission message.
  • the first CQE may be interpreted as CQE1 in the embodiment corresponding to FIG. 2B .
  • the first CQE indicates the completion of the data transmission message through a part of the field carried (for example, called an error code).
  • the first process parses the first CQE, and if the value of the error code in the first CQE is 0, the first process may determine that the completion of the data transmission message is successful, for example, the transmission of the data transmission message has been completed.
  • the first process parses the first CQE, and if the value of the error code in the first CQE1 is 1, the first process may determine that the completion of the data transmission message is a failure, for example, the transmission of the data transmission message has not been completed.
  • the completion status of the data transmission message notified by the first network device 311 may be consistent with the completion status described in the feedback message. For example, if the feedback message describes that the completion of the data transmission message is successful, then the first network device 311 may notify the first process that the data transmission message is completed or transmitted successfully. If the feedback message describes that the completion of the data transmission message is failure, then the first network device 311 may notify the first process that the data transmission message is not completed or the transmission fails.
  • the completion status of the data transmission message notified by the first network device 311 may be inconsistent with the completion status described in the feedback message. For example, if the feedback message describes the completion of the data transmission message as successful, but the first network device 311 may not correctly encapsulate the first packet due to a fault (for example, the encapsulated data transmission message is wrong or the destination address is wrong, etc.), then the first The network device 311 may notify the first process that the data transmission message is not completed or the transmission fails.
  • the first network device 311 can create an SSQ used by multiple processes running on the first host 32. After the first network device 311 obtains the data transmission message of the first process from the SSQ, it can encapsulate the data The first identifier corresponding to the first process is also encapsulated in the packet (that is, the first packet) of the transmission message. After the second network device 312 receives the first message, it may simultaneously encapsulate the first identifier in the message encapsulating the feedback message (that is, the second message).
  • the first network device 312 After the first network device 312 receives the second message , it is not necessary to determine that the feedback message in the second message corresponds to the first process according to the context of the SSQ, but it can be determined that the feedback message corresponds to the first process according to the first identifier in the second message, which is conducive to efficient notification
  • the first process has already processed the data transmission message, which is beneficial to improving the operating efficiency of the first process.
  • the first identifier may be interpreted as the identifier of CQ#1 and/or process #1 in the embodiment corresponding to FIG. 2A or FIG. 2B .
  • this embodiment of the present application does not limit the specific meaning of the first identifier, as long as the first identifier can be used to determine the first CQ.
  • the method for the first network device 311 to determine the first CQ according to the identifier of the first process is introduced.
  • the first network device 311 may save a mapping table, and the mapping table records the correspondence between processes and CQs.
  • the first network device 311 may search the mapping table for a corresponding relationship between the first process and the first CQ, and then determine the identifier of the first CQ.
  • RNIC 12 obtains the identification of process #1 that message 2 carries, can obtain the identification of CQ#1 from the mapping table, and then write CQE1 into CQ#1.
  • the first identifier may include the identifier of the first process and the identifier of the first CQ
  • the first CQE may include the identifier of the first process.
  • the first process and other processes may share the same CQ
  • the first process may determine the first CQE as its own CQE through the identifier of the first process carried by the first CQE. In this way, although the CQs created by the first network device 311 are still not shared by all processes, by allowing some processes to share a CQ, it is beneficial to reduce the number of created CQs, thereby saving memory space.
  • Figure 5A exemplarily shows the process of the first network device 311 obtaining the data transmission message of the process through the solid line segment with the arrow, and exemplarily shows the process of the first network device 311 notifying the process through the dotted line segment with the arrow The process of the completion of a data transfer message.
  • the process described in FIG. 5A is introduced below.
  • the SSQ#2 created by RNIC 12 can bind the channel n1-n3 between node #1 and node #3, SSQ#2 is used to get messages destined for node#3.
  • the messages to be transmitted by process #1 include M4, M3 and M1 in sequence, wherein M4 and M3 are used to send to node #3.
  • the messages transmitted by process #2 include M5 and M2 in sequence, wherein M5 is used to send to node #3.
  • RNIC 12 can sequentially write WQE5 corresponding to M5, WQE4 corresponding to M4, and WQE3 corresponding to M3 into SSQ#2 according to the value of PI of SSQ#2, and sequentially write the corresponding WQE5 of M1 according to the value of PI of SSQ#1 WQE2 corresponding to WQE1 and M2 is written into SSQ#1.
  • RNIC 12 can sequentially process WQE in SSQ#2 according to the value of CI of SSQ#2, and send a corresponding message, and then RNIC 12 can process WQE in SSQ#1 sequentially according to the value of CI of SSQ#1, Send the appropriate message. Assume that RNIC 12 sends M5, M4, M3, M1 and M2 sequentially from first to last.
  • the RNIC 12 may determine the corresponding CQ according to the first identifier carried in the message. Assume that RNIC 12 receives message a corresponding to M5, message b corresponding to M4, message c corresponding to M3, message d corresponding to M1, and message e corresponding to M2 from first to last. RNIC 12 adds CQE-a in CQ#2 according to the identifier of CQ#2 carried in message a, and adds CQE-b in CQ#1 according to the identifier of CQ#1 carried in message b. Added WQE-c and CQE-d in #1, and added CQE-e in CQ#2.
  • Process #1 can respectively determine that CQE-b corresponds to M4, CQE-c corresponds to M3, and CQE-d corresponds to M1 according to the writing sequence of CQEs in CQ#1.
  • process #2 can respectively determine that CQE-a corresponds to M5 and CQE-e corresponds to M2 according to the writing sequence of CQEs in CQ#2.
  • the identification information in the first message and the second message may also include a second identification, and the second identification is used for multiple data transmissions from the first process
  • the data transfer message is identified in the message.
  • the first network device is specifically configured to notify the first process of the completion of the data transmission message according to the first identifier, the feedback message, and the second identifier.
  • the first CQE may include the second identifier.
  • the second identifiers corresponding to M4, M3, and M1 may be sequence numbers of corresponding messages, respectively.
  • the second identifiers of M4, M3 and M1 may be 1, 2 and 3 respectively.
  • the second identifiers carried by CQE-b, CQE-c, and CQE-d are 1, 2, and 3, respectively.
  • Process #1 may determine that CQE-b corresponds to M4, CQE-c corresponds to M3, and CQE-d corresponds to M1 according to the second identification of each CQE in CQ#1.
  • process #1 submits WR4 describing M4, WR3 describing M3, and WR1 describing M1 to RNIC 12 in sequence by calling the driver program of the program interface and RNIC 12, and process #2 submits WR4 describing M4 and WR1 describing M1 through the driver program of the calling program interface and RNIC 12.
  • the program submits WR5 describing M5 and WR2 describing M2 to RNIC 12 in turn.
  • RNIC 12 adds corresponding WQEs in SSQ#2 and SSQ#1 according to WR5, WR4, WR3, WR1 and WR2 in turn.
  • the second identifier of the message may be the identifier of the corresponding WR (for example, the serial number of the WR).
  • process #1 can determine the message corresponding to CQE according to the second identification of each CQE in CQ#1, for example, by determining that CQE-b corresponds to WR4, in turn, determines that CQE-b corresponds to M4.
  • the data transmission message of the first process may include the first data to be written into the second host 33 .
  • the second network 312 device is configured to store the first data on the second host after obtaining the first data.
  • the first network device 311 may divide the first data into multiple data segments (segments), and may sequentially encapsulate the multiple segments into multiple packets (called a first packet sequence).
  • the first packet involved in the embodiment corresponding to FIG. 3 may refer to the first packet sequence.
  • the first network device 311 may encapsulate the identification information in each packet in the first packet sequence, or only encapsulate the identification information in the last packet in the first packet sequence.
  • the identification information may be encapsulated in an extension header of the message, and the first data or data slice may be used as payload data of the message.
  • the data transmission system 31 may use a send (send) operation, a write (write) operation, or an atomic (atomic) operation to execute the work request corresponding to the data transmission message.
  • FIG. 6A shows the transmission process of the first packet and the second packet corresponding to the sending operation.
  • the first message sequence includes message #1, message #2, ..., message #k, and the message #1, message #2, ..., message #k carry the first Data slice #1, data slice #2, ..., data slice #k in the data, where k is any positive integer.
  • each packet in the first packet sequence carries identification information.
  • the identification information may include the first identification described above, and may also include the second identification.
  • the identification information may also be a third identification.
  • the third identifier may include the identifier of the second process and/or the identifier of the RQ corresponding to the second process.
  • FIG. 6A exemplarily shows the first identifier and the RQ identifier in the identifier information.
  • the second network device 312 After the second network device 312 receives the first packet sequence sent by the first network device 311, it can read the WQE from the RQ corresponding to the third identifier, and the WQE Describe the storage space corresponding to the second process. Afterwards, the second network device 312 may save the first data in the storage space described by the WQE. After the second network device 312 receives the first message sequence, it can send an acknowledgment message to the first network device 311.
  • the acknowledgment message carries an acknowledgment character (acknowledge character, ACK). Therefore, in the embodiment of the present application, the acknowledgment message will be It is called an ACK message.
  • the ACK message may carry the feedback message introduced above (for example, an acknowledgment character), and may also carry all or part of the identification information in the first message sequence. For example, the ACK message may also carry the first identifier.
  • FIG. 6B shows the transmission process of the first packet and the second packet corresponding to the write operation.
  • the first message sequence includes message #1, message #2, ..., message #k, and the message #1, message #2, ..., message #k carry the first Data slice #1, data slice #2, . . . , data slice #k in the data, where k is a positive integer greater than or equal to 3.
  • each packet in the first packet sequence carries identification information.
  • the identification information may include the first identification described above, and optionally, may further include the second identification.
  • Fig. 6B exemplarily shows the first identifier in the identifier information.
  • the writing location of the first data in the second host 33 may also be encapsulated in the message #1.
  • the second network device 312 may store the first data in the second host 33 according to the write location.
  • the second network device 312 may send an ACK packet to the first network device 311.
  • the ACK message may carry the feedback message introduced above (for example, an acknowledgment character), and may also carry all or part of the identification information in the first message sequence.
  • the ACK message may also carry the first identifier.
  • FIG. 6C shows the transmission process of the first packet and the second packet corresponding to the atomic operation.
  • the first packet includes first data and identification information.
  • the identification information may include the first identification described above, and optionally, may further include the second identification.
  • Fig. 6C exemplarily shows the first identifier in the identifier information.
  • the second network device 312 may store the first data in the second host 33 .
  • the second network device 312 After the second network device 312 receives the first message, it can send a second message to the first network device 311.
  • the second message can be, for example, an atomic feedback (atomic response) message or an atomic acknowledgment (atomic acknowledge) message. .
  • the second message may carry the feedback message introduced above (for example, a confirmation character), and may also carry all or part of the identification information in the first message sequence.
  • the second packet may also carry the first identifier.
  • the data transfer message may include a source address and a destination address of the first data, and the source address of the first data points to the second host 33, and the destination address of the first data points to the first host 32, therefore, the data The transmission message may not carry the first data.
  • the second network device 312 may be configured to read the first data on the second host 33 according to the source address after obtaining the source address, destination address and identification information encapsulated in the first packet, and to read the first data according to the first data, identification information, The feedback message and the destination address encapsulate the second packet, and send the second packet to the first host 32 .
  • Fig. 6B exemplarily shows the first identifier in the identifier information.
  • the identification information may be encapsulated in the extension header of the message.
  • the first network device 311 is configured to store the first data on the first host according to the destination address after obtaining the first data and the destination address in the second packet.
  • the first network device 311 is further configured to notify the first process of the completion of the data transmission message according to the first identifier and the feedback message in the second packet.
  • the second network device 312 may divide the first data into multiple fragments, and may sequentially encapsulate the multiple fragments into multiple packets (called a second packet sequence).
  • the second packet involved in the embodiment corresponding to FIG. 3 may refer to a second packet sequence.
  • the second network device 312 may encapsulate part or all of the contents of the identification information in each packet in the second packet sequence, or only in the last packet in the second packet sequence Encapsulate part or all of the content in the identification information.
  • the identification information may be encapsulated in an extension header of the message, and the first data or data slice may be used as payload data of the message.
  • FIG. 6D shows the transmission process of the first packet and the second packet corresponding to the read (read) operation.
  • the first message includes a source address, a destination address, and a first identifier.
  • the second network device 312 can read the first data from the second host 33 according to the source address, and then The first data is divided into slices, and data slice #1, data slice #2, . . . , and data slice #k are sequentially obtained, where k is a positive integer greater than or equal to 3.
  • the second network device 312 can encapsulate and send a second packet sequence, the second packet sequence includes packet #1, packet #2, ..., packet #k, packet #1, packet #2, ...
  • each message in the second message sequence may further include a destination address, a first identifier, and a feedback message.
  • the first network device 311 may store the first data in the first host 32 according to the destination address, and notify the first process of the completion of the data transmission message according to the first identifier and the feedback message.
  • the embodiment of the present application also provides a data transmission method, which may be an RDMA data transmission method.
  • the data transmission method may include steps S701 to S704.
  • the first network device obtains the RDMA data transmission message of the first process from the SSQ;
  • the first network device may be set on the first host, the first process is any one of multiple processes running on the first host and using the shared sending queue, and the data transmission message may be an RDMA data transmission message.
  • the first network device sends the first packet to the second network device
  • the second network device is set on the second host, and the first message includes a data transmission message and a first identifier corresponding to the first process.
  • the second network device receives the first message from the first network device, the first network device is set on the first host, the second network device is set on the second host, and the first message includes the data transmission message of the first process
  • the first identifier corresponding to the first process, the data transmission message is obtained by the first network device from the shared sending queue, and the first process is any process among the multiple processes running on the first host and using the shared sending queue.
  • the second network device sends a second message to the first network device according to the first message, where the second message includes the first identification and the feedback message;
  • the first network device notifies the first process of the completion of the data transmission message according to the first identifier and the feedback message in the second packet.
  • the first network device receives the second packet from the second network device, the second packet includes the first identification and a feedback message, and the feedback message indicates the completion of the data transmission message.
  • the first network device notifies the first process of the completion of the data transmission message according to the first identifier in the second packet and the feedback message.
  • step S701 , step S702 and step S704 are methods performed by the first network device, and this method may be regarded as the method performed by the first network device in the embodiment corresponding to FIG. 3 .
  • step S701 , step S702 , and step S704 performed by the first network device shown in FIG. 7 refer to the relevant description of the first network device in the embodiment shown in FIG. 3 .
  • step S703 For the methods corresponding to step S701, step S702, and step S704, the embodiment of the present application does not limit the method specifically executed by the second network device in step S703, as long as the first network device sends the first message to the second network device, the second The network device may just send the second packet including the first identifier to the first network device.
  • step S703 is a method performed by the second network device, and this method may be regarded as the method performed by the second network device in the embodiment corresponding to FIG. 3 .
  • step S703 performed by the second network device shown in FIG. 7
  • the embodiment of the present application does not limit the steps performed by the first network device, as long as the first network device can send the first packet including the first identifier to the second network device.
  • FIG. 8 is a schematic structural diagram of a network device provided in an embodiment of the present application.
  • the network device may be the first network device in the embodiment corresponding to FIG. 3 or FIG. 7, and may execute the first network device in the corresponding embodiment. A method and steps executed by a network device.
  • the network device 800 may include an acquisition unit 801, configured to acquire the RDMA data transmission message of the first process from the shared send queue, the network device is set on the first host, and the first process is run by the first host Any one of multiple processes using a shared send queue.
  • the network device 800 may further include a sending unit 802, configured to send a first message to a second network device, the second network device is set on the second host, and the first message includes an RDMA data transmission message and a first message corresponding to the first process. a logo.
  • a receiving unit 803 may also be included, configured to receive a second message from the second network device, where the second message includes the first identification and a feedback message, and the feedback message indicates the completion of the RDMA data transmission message.
  • a completion unit 804 may also be included, configured to notify the first process of the completion of the RDMA data transmission message according to the first identifier and the feedback message in the second message.
  • the first packet and the second packet further include a second identifier, and the second identifier is used to determine the RDMA data transmission message from the multiple RDMA data transmission messages of the first process.
  • the completion unit 804 is specifically configured to notify the first process of the completion of the RDMA data transmission message according to the first identifier, the feedback message and the second identifier.
  • the shared send queue is specifically used to store work requests from multiple processes.
  • the acquiring unit 801 is specifically configured to acquire a first work request from the first process from the shared sending queue, the first work request describes an RDMA data transmission message, and acquires the RDMA data transmission message according to the first work request.
  • the completion unit 804 is specifically configured to determine the first completion queue corresponding to the first process from multiple completion queues according to the first identifier, and write the work completion queue in the first completion queue according to the feedback message. element, the work completion element is used to notify the first process of the completion of the RDMA data transmission message.
  • each unit included in the network device 800 may be a software module, may also be a hardware module, or may be partly a software module and partly a hardware module.
  • the above structure of the network device 800 is merely an example and should not be construed as a specific limitation, and various units in the network device may be added, reduced or combined as required.
  • the operation and/or function of each unit in the network device 800 is to realize the function or method of the first network device described in FIG. 3 or FIG. 7 above, and for the sake of brevity, details are not repeated here.
  • FIG. 9 is a schematic structural diagram of another network device provided by an embodiment of the present application.
  • the network device may be the second network device in the above-mentioned embodiment corresponding to FIG. 3 or FIG. The method and steps performed by the second network device.
  • the network device 900 may include a receiving unit 901 configured to receive a first message from a first network device, the first network device is set on the first host, and the network device is set on the second host,
  • the first message includes the RDMA data transmission message of the first process and the first identifier corresponding to the first process.
  • the RDMA data transmission message is obtained by the first network device from the shared sending queue.
  • the first process is run by the first host using the shared Any one of multiple processes in the send queue.
  • It may also include a sending unit 902, configured to send a second message to the first network device according to the first message, the second message includes a first identification and a feedback message, and the first identification and the feedback message instruct the first network device to notify the first network device The completion status of a process RDMA data transfer message.
  • a sending unit 902 configured to send a second message to the first network device according to the first message, the second message includes a first identification and a feedback message, and the first identification and the feedback message instruct the first network device to notify the first network device The completion status of a process RDMA data transfer message.
  • the first message and the second message further include a second identifier
  • the second identifier is used to determine the RDMA data transmission message from the multiple RDMA data transmission messages of the first process
  • the first identifier , the feedback message, and the second identifier instruct the first network device to notify the first process of the completion of the RDMA data transmission message.
  • each unit included in the network device 900 may be a software module, may also be a hardware module, or may be partly a software module and partly a hardware module.
  • the above structure of the network device 900 is merely an example and should not be construed as a specific limitation, and various units in the network device may be added, reduced or combined as required.
  • the operation and/or function of each unit in the network device 900 is to realize the function or method of the second network device described above in FIG. 3 or FIG. 7 , and details are not repeated here for brevity.
  • the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, part or all of any one of the above-mentioned method embodiments can be realized. step.
  • An embodiment of the present invention also provides a computer program, the computer program includes instructions, and when the computer program is executed by a computer, the computer can execute some or all steps of any method.
  • the disclosed device can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the above units is only a logical function division.
  • there may be other division methods for example, multiple units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical or other forms.
  • the units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • multiple refers to two or more, which is not limited in the present application.
  • "/" can indicate that the objects associated before and after are in an "or” relationship, for example, A/B can indicate A or B; “and/or” can be used to describe that there are three types of associated objects A relationship, for example, A and/or B, may mean: A exists alone, A and B exist simultaneously, and B exists independently, where A and B may be singular or plural.
  • words such as “first” and “second” may be used to distinguish technical features with the same or similar functions.
  • words “first” and “second” do not limit the number and execution order, and the words “first” and “second” do not necessarily mean that they must be different.
  • words such as “exemplary” or “for example” are used to represent examples, illustrations or illustrations, and any embodiment or design described as “exemplary” or “for example” should not be interpreted It is more preferred or more advantageous than other embodiments or design solutions.
  • the use of words such as “exemplary” or “for example” is intended to present related concepts in a specific manner for easy understanding.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computer And Data Communications (AREA)

Abstract

一种RDMA数据传输系统、RDMA数据传输方法及网络设备，例如可以用于智能网卡(Smart NIC)或数据处理单元(DPU)等网络设备，例如可以应用于RoCEv1、RoCEv2以及其他新RDMA场景。该RDMA数据传输系统可以包括第一主机中的第一网络设备和第二主机中的第二网络设备。第一网络设备可以创建供第一主机运行的多个进程使用的SSQ，从SSQ获取第一进程的RDMA数据传输消息，并且在封装RDMA数据传输消息的第一报文中同时封装第一进程对应的第一标识。第二网络设备在封装反馈消息的第二报文中同时封装第一标识，这样，第一网络设备在接收到第二报文后，可以根据第二报文中的第一标识高效的通知第一进程已处理该RDMA数据传输消息，进而有利于提高第一进程的运行效率。

Description

RDMA数据传输系统、RDMA数据传输方法及网络设备
本申请要求于2021年9月7日提交中国国家知识产权局、申请号为202111045146.4、发明名称为“RDMA数据传输系统、RDMA数据传输方法及网络设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及通信技术领域，尤其涉及一种RDMA数据传输系统、RDMA数据传输方法及网络设备。
背景技术
远程直接内存访问(remote direct memory access，RDMA)是一种绕过远程主机CPU的核心访问CPU的内存中数据的技术，由于不经过CPU核心，不仅节省了大量CPU核心资源，同样也提高了系统吞吐量、降低了系统的网络通信延迟，尤其适合在大规模并行计算机集群中有广泛应用。
本端主机中的应用进程执行RDMA数据传输消息时，本端主机的网卡从主机或网卡的内存中的发送队列中将该RDMA数据传输消息读取到网卡的缓冲区中，之后通过网络将该RDMA数据传输消息发送到对端主机的网卡。之后，对端主机的网卡发送反馈信息给本端主机的网卡，本端主机的网卡根据反馈信息通知应用进程该RDMA数据传输消息已被处理完。
但是，当本端主机的网卡的发送队列被多个应用进程共享的时候，反馈信息无法快速将该RDMA数据传输消息的完成情况通知对应的应用进程。
发明内容
本申请实施例提供一种数据传输系统、数据传输方法及网络设备，用于快速将该RDMA数据传输消息的完成情况通知对应的应用进程。
为解决上述技术问题，本申请实施例提供以下技术方案：
第一网络设备可以创建供第一主机运行的多个进程使用的共享发送队列，第一网络设备可以从共享发送队列获取第一进程的RDMA数据传输消息，并且在封装RDMA数据传输消息的报文(称作第一报文)中同时封装第一进程对应的第一标识，通过RDMA方式向第二网络设备发送第一报文。第二网络设备接收到第一报文后，可以在封装反馈消息的报文(称作第二报文)中同时封装第一标识，之后通过RDMA方式向第一网络设备发送第二报文。这样，第一网络设备在接收到第二报文后，无需根据共享发送队列的上下文来确定第二报文中的反馈消息对应于第一进程，而是可以根据第二报文中的第一标识确定该反馈消息对应于第一进程，有利于高效的通知第一进程已处理该RDMA数据传输消息，进而有利于提高第一进程的运行效率。
可选的，第一报文为第一网络设备根据RDMA协议封装得到的，第二报文为第二网络设备根据RDMA协议封装得到的，该RDMA协议可以例如为无线带宽(infiniband)协议、或采用聚合以太网络版本1的RDMA(RDMA over Converged Ethernet version 1,RoCE1)协议、或采用聚合以太网络版本2的RDMA(RDMA over Converged Ethernet version 2,RoCE2)协议、 或互联网广域RDMA协议(internet wide area RDMA protocol,iWARP)等。
可选的，第一网络设备可以通过该共享发送队列所绑定的传输通道向第二网络设备发送第一报文。该传输通道用于实现第一网络设备和第二网络设备之间的网络连接，该传输通道可以例如为输入输出通道(channel-IO)。
可选的，第二网络设备通过该传输通道接收第一报文后，还可以通过该传输通道向第二网络设备发送第二报文。
可选的，第一网络设备可以通过该传输通道接收来自第二网络设备的第二报文。
可选的，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息；所述第一网络设备具体用于根据所述第一标识、所述反馈消息和所述第二标识通知所述第一进程所述RDMA数据传输消息的完成情况。通过在第一报文和第二报文中携带第二标识，有利于第一网络设备高效、准确的通知第一进程RDMA数据传输消息的完成情况。
可选的，所述共享发送队列具体用于保存来自所述多个进程的工作请求，所述第一网络设备具体用于从所述共享发送队列获取来自所述第一进程的第一工作请求，根据所述第一工作请求获取所述RDMA数据传输消息，所述第一工作请求描述所述RDMA数据传输消息。
可选的，共享发送队列可以设置在第一网络设备的内存中。或者，可选的，共享发送队列可以设置在第一主机的内存中。
和根据共享发送队列的上下文来从共享发送队列读取第一工作请求来确定第一标识相比，本申请实施例通过第二报文获取第一标识，有利于尽量避免从共享发送队列中读取工作请求，有利于节约第一网络设备的缓存资源，减少工作请求的完成过程中的延时。
可选的，所述第一网络设备具体用于根据所述第一标识从多个完成队列中确定所述第一进程对应的第一完成队列，根据所述反馈消息在所述第一完成队列中写入工作完成元素，所述工作完成元素描述所述RDMA数据传输消息的完成情况。
第一标识可以用于确定第一完成队列，本申请实施例可以通过第一报文和第二报文携带的第一标识来高效准确的确定第一完成队列，有利于高效准确的通知第一进程已处理该RDMA数据传输消息，进而有利于提高第一进程的运行效率。
第二方面，本申请实施例提供一种RDMA数据传输方法，包括：第一网络设备从共享发送队列获取第一进程的RDMA数据传输消息，所述第一网络设备设置在第一主机上，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程；所述第一网络设备向第二网络设备发送第一报文，所述第二网络设备设置在第二主机上，所述第一报文包括所述RDMA数据传输消息和所述第一进程对应的第一标识；所述第一网络设备接收来自所述第二网络设备的第二报文，所述第二报文包括所述第一标识和反馈消息，所述反馈消息指示所述RDMA数据传输消息的完成情况；所述第一网络设备根据所述第二报文中的所述第一标识和所述反馈消息通知所述第一进程所述RDMA数据传输消息的完成情况。
可选的，第一报文为第一网络设备根据RDMA协议封装得到的，第二报文为第二网络设备根据RDMA协议封装得到的，该RDMA协议可以例如为无线带宽(infiniband)协议、或采用聚合以太网络版本1的RDMA(RDMA over Converged Ethernet version 1,RoCE1)协议、或采用聚合以太网络版本2的RDMA(RDMA over Converged Ethernet version 2,RoCE2)协议、 或互联网广域RDMA协议(internet wide area RDMA protocol,iWARP)等。
可选的，第一网络设备可以通过该共享发送队列所绑定的传输通道向第二网络设备发送第一报文。该传输通道用于实现第一网络设备和第二网络设备之间的网络连接，该传输通道可以例如为输入输出通道(channel-IO)。
可选的，第一网络设备可以通过该传输通道接收来自第二网络设备的第二报文。
第一网络设备通过在第一报文中携带第一标识，有利于指示第二网络设备在第二报文中携带第一标识，这样，第一网络设备在接收到第二报文后，无需根据共享发送队列的上下文来确定第二报文中的反馈消息对应于第一进程，而是可以根据第二报文中的第一标识确定该反馈消息对应于第一进程，有利于高效的通知第一进程已处理该RDMA数据传输消息，进而有利于提高第一进程的运行效率。
可选的，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息；所述第一网络设备根据所述第二报文中的所述第一标识和所述反馈消息通知所述第一进程所述RDMA数据传输消息的完成情况包括：所述第一网络设备具体用于根据所述第一标识、所述反馈消息和所述第二标识通知所述第一进程所述RDMA数据传输消息的完成情况。
第一网络设备通过在第一报文中携带第二标识，有利于指示第二网络设备在第二报文中携带第二标识，进而有利于第一网络设备根据第二报文中的第二标识，高效、准确的通知第一进程RDMA数据传输消息的完成情况。
可选的，所述共享发送队列具体用于保存来自所述多个进程的工作请求；所述第一网络设备从共享发送队列获取第一进程的RDMA数据传输消息包括：所述第一网络设备从所述共享发送队列获取来自所述第一进程的第一工作请求，所述第一工作请求描述所述RDMA数据传输消息；所述第一网络设备根据所述第一工作请求获取所述RDMA数据传输消息。
可选的，共享发送队列可以设置在第一网络设备的内存中。或者，可选的，共享发送队列可以设置在第一主机的内存中。
和根据共享发送队列的上下文来从共享发送队列读取第一工作请求来确定第一标识相比，本申请实施例中，第一网络设备通过在第一报文中携带第一标识，有利于指示第二网络设备在第二报文中携带第一标识，这样，第一网络设备在接收到第二报文后，通过第二报文获取第一标识，有利于尽量避免从共享发送队列中读取工作请求，有利于节约第一网络设备的缓存资源，减少工作请求的完成过程中的延时。
可选的，所述第一网络设备根据所述第二报文中的所述第一标识和所述反馈消息通知所述第一进程所述RDMA数据传输消息的完成情况，包括：所述第一网络设备根据所述第一标识从多个完成队列中确定所述第一进程对应的第一完成队列；所述第一网络设备根据所述反馈消息在所述第一完成队列中写入工作完成元素，所述工作完成元素用于通知所述第一进程所述RDMA数据传输消息的完成情况。
第一标识可以用于确定第一完成队列，本申请实施例中，第一网络设备通过在第一报文中携带第一标识，有利于指示第二网络设备在第二报文中携带第一标识，这样，第一网络设备在接收到第二报文后，可以通过第二报文携带的第一标识来高效准确的确定第一完成队列，有利于高效准确的通知第一进程已处理该RDMA数据传输消息，进而有利于提高第一进程的运 行效率。
第三方面，本申请实施例提供一种RDMA数据传输方法，包括：第二网络设备接收来自第一网络设备的第一报文，所述第一网络设备设置于第一主机上，所述第二网络设备设置于第二主机上，所述第一报文包括第一进程的RDMA数据传输消息和所述第一进程对应的第一标识，所述RDMA数据传输消息为所述第一网络设备从共享发送队列获取的，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程；所述第二网络设备根据所述第一报文向所述第一网络设备发送第二报文，所述第二报文包括所述第一标识和反馈消息，所述第一标识和所述反馈消息指示所述第一网络设备通知所述第一进程所述RDMA数据传输消息的完成情况。
第二网络设备解析到第一报文中的第一标识后，可以在第二报文中携带第一标识，这样，第一网络设备在接收到第二报文后，无需根据共享发送队列的上下文来确定第二报文中的反馈消息对应于第一进程，而是可以根据第二报文中的第一标识确定该反馈消息对应于第一进程，有利于高效的通知第一进程已处理该RDMA数据传输消息，进而有利于提高第一进程的运行效率。
可选的，第一报文为第一网络设备根据RDMA协议封装得到的，第二报文为第二网络设备根据RDMA协议封装得到的，该RDMA协议可以例如为无线带宽(infiniband)协议、或采用聚合以太网络版本1的RDMA(RDMA over Converged Ethernet version 1,RoCE1)协议、或采用聚合以太网络版本2的RDMA(RDMA over Converged Ethernet version 2,RoCE2)协议、或互联网广域RDMA协议(internet wide area RDMA protocol,iWARP)等。
可选的，第一报文为第一网络设备通过该共享发送队列所绑定的传输通道向第二网络设备发送的。该传输通道用于实现第一网络设备和第二网络设备之间的网络连接，该传输通道可以例如为输入输出通道(channel-IO)。
可选的，第二网络设备通过该传输通道接收第一报文后，还可以通过该传输通道向第二网络设备发送第二报文。
可选的，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息，所述第一标识、所述反馈消息和所述第二标识指示所述第一网络设备通知所述第一进程所述RDMA数据传输消息的完成情况。
第二网络设备解析到第一报文中的第二标识后，可以在第二报文中携带第二标识，这样，第一网络设备在接收到第二报文后，通过在第二报文中携带第二标识，有利于第一网络设备高效、准确的通知第一进程RDMA数据传输消息的完成情况。
第四方面，本申请实施例提供一种网络设备，包括：获取单元，用于从共享发送队列获取第一进程的RDMA数据传输消息，所述网络设备设置在第一主机上，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程；发送单元，用于向第二网络设备发送第一报文，所述第二网络设备设置在第二主机上，所述第一报文包括所述RDMA数据传输消息和所述第一进程对应的第一标识；接收单元，用于接收来自所述第二网络设备的第二报文，所述第二报文包括所述第一标识和反馈消息，所述反馈消息指示所述RDMA数据传输消息的完成情况；完成单元，用于根据所述第二报文中的所述第一标识和所述反馈消息 通知所述第一进程所述RDMA数据传输消息的完成情况。
可选的，第一报文为第一网络设备根据RDMA协议封装得到的，第二报文为第二网络设备根据RDMA协议封装得到的，该RDMA协议可以例如为无线带宽(infiniband)协议、或采用聚合以太网络版本1的RDMA(RDMA over Converged Ethernet version 1,RoCE1)协议、或采用聚合以太网络版本2的RDMA(RDMA over Converged Ethernet version 2,RoCE2)协议、或互联网广域RDMA协议(internet wide area RDMA protocol,iWARP)等。
可选的，发送单元可以通过该共享发送队列所绑定的传输通道向第二网络设备发送第一报文。该传输通道用于实现第一网络设备和第二网络设备之间的网络连接，该传输通道可以例如为输入输出通道(channel-IO)。
可选的，接收单元可以通过该传输通道接收第二报文。
可选的，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息；所述完成单元具体用于根据所述第一标识、所述反馈消息和所述第二标识通知所述第一进程所述RDMA数据传输消息的完成情况。
可选的，所述共享发送队列具体用于保存来自所述多个进程的工作请求；所述获取单元具体用于：从所述共享发送队列获取来自所述第一进程的第一工作请求，所述第一工作请求描述所述RDMA数据传输消息；根据所述第一工作请求获取所述RDMA数据传输消息。
可选的，所述完成单元具体用于：根据所述第一标识从多个完成队列中确定所述第一进程对应的第一完成队列；根据所述反馈消息在所述第一完成队列中写入工作完成元素，所述工作完成元素用于通知所述第一进程所述RDMA数据传输消息的完成情况。
第五方面，本申请实施例提供一种网络设备，包括：接收单元，用于接收来自第一网络设备的第一报文，所述第一网络设备设置于第一主机上，所述网络设备设置于第二主机上，所述第一报文包括第一进程的RDMA数据传输消息和所述第一进程对应的第一标识，所述RDMA数据传输消息为所述第一网络设备从共享发送队列获取的，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程；发送单元，用于根据所述第一报文向所述第一网络设备发送第二报文，所述第二报文包括所述第一标识和反馈消息，所述第一标识和所述反馈消息指示所述第一网络设备通知所述第一进程所述RDMA数据传输消息的完成情况。
可选的，第一报文为第一网络设备根据RDMA协议封装得到的，第二报文为第二网络设备根据RDMA协议封装得到的，该RDMA协议可以例如为无线带宽(infiniband)协议、或采用聚合以太网络版本1的RDMA(RDMA over Converged Ethernet version 1,RoCE1)协议、或采用聚合以太网络版本2的RDMA(RDMA over Converged Ethernet version 2,RoCE2)协议、或互联网广域RDMA协议(internet wide area RDMA protocol,iWARP)等。
可选的，第一报文可以是第一网络设备通过该共享发送队列所绑定的传输通道向第二网络设备发送的。该传输通道用于实现第一网络设备和第二网络设备之间的网络连接，该传输通道可以例如为输入输出通道(channel-IO)。
可选的，接收单元可以通过该传输通道接收第一报文。
可选的，发送单元可以通过该传输通道向第二网络设备发送第二报文。
可选的，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息，所述第一标识、所述反馈消息和所述第二标识指示所述第一网络设备通知所述第一进程所述RDMA数据传输消息的完成情况。
第六方面，本申请提供一种计算设备，该计算设备包括处理器和存储器，该处理器和该存储器耦合，该存储器用于存储程序代码，该处理器执行该存储器中存储的程序代码时，能够执行第二方面或第二方面的任意一种可能的实现方式或第三方面或第三方面的任意一种可能的实现方式所描述的方法。
在一种可能的实现方式中，该计算设备还可以包括通信接口，该处理器通过该通信接口例如能够接收或发送报文等。
本申请第七方面提供了一种芯片系统，该芯片系统包括处理器和接口电路，该处理器通过该接口电路与存储器耦合，该处理器用于执行该存储器中的程序代码，以执行第二方面或第二方面的任意一种可能的实现方式或第三方面或第三方面的任意一种可能的实现方式所描述的方法。该芯片系统可以由芯片构成，也可以包含芯片和其他分立器件。
本申请第八方面提供一种计算机可读存储介质，该计算机可读存储介质中存储了程序代码，这些程序代码在计算机设备上运行时，使得计算机设备执行如本申请能够执行第二方面或第二方面的任意一种可能的实现方式或第三方面或第三方面的任意一种可能的实现方式所描述的方法。
本申请第九方面提供一种计算机程序产品，该计算机程序产品包含的程序代码被计算机设备执行时，以执行第二方面或第二方面的任意一种可能的实现方式或第三方面或第三方面的任意一种可能的实现方式所描述的方法。
由于本申请提供的各装置可用于执行前述对应的方法，因此本申请各装置所能获得到的技术效果可参考前述对应的方法所获得的技术效果，此处不再赘述。
附图说明
图1A、图1B和图1C分别示出了本申请实施例一种可能的应用场景；
图2A和图2B分别示出了计算机系统一种可能的结构；
图3示出了本申请实施例计算机系统另一种可能的结构；
图4示出了第一工作请求一种可能的结构；
图5A和图5B分别示出了第一网络设备对RDMA数据传输消息的处理过程；
图6A示出了发送操作对应的第一报文和第二报文的传输过程；
图6B示出了写操作对应的第一报文和第二报文的传输过程；
图6C示出了原子操作对应的第一报文和第二报文的传输过程；
图6D示出了读操作对应的第一报文和第二报文的传输过程；
图7示出了本申请实施例RDMA数据传输方法一种可能的流程；
图8示出了本申请实施例网络设备一种可能的结构；
图9示出了本申请实施例网络设备一种可能的结构。
具体实施方式
下面，首先对本申请实施例的应用场景进行举例介绍。
图1A示出了本申请实施例一种可能的应用场景。图1A所示的计算机系统至少包括节点#1和节点#2。节点#1或节点#2可以为连接到通信网络的一个或多个链路并且形成通信网络内数据传输消息(简称消息)的起源和/或目的地的任意设备。节点#1或节点#2中可以运行一个或多个应用进程(简称进程)。图1A以节点#1运行进程#1和进程#2为例，以节点#2运行进程#3为例。可以理解的是，节点#1或节点#2可以运行更多或更少的进程。以节点#1为例，节点#1发出的消息可以来自进程#1或进程#2，节点#1接收的消息可以去往进程#1或进程#2。
节点#1和节点#2可以通过数据传输系统进行通信连接。图1A所示的数据传输系统可以至少包括节点#1中的网络设备#1和节点#2中的网络设备#2。可选的，网络设备#1或网络设备#2可以为网络适配器(network interface card，NIC)或称网卡。可选的，图1A所示的数据传输系统可以为远程直接内存访问(remote direct memory access，RDMA)系统，该数据传输系统中传输的消息可以为RDMA数据传输消息。
图1A所示的计算机系统中，数据传输系统可以基于节点运行的进程来创建数据传输通道(channel)。假设进程#1和进程#2需要分别向进程#3发送消息，网络设备#1和网络设备#2需要分别为进程#1和进程#2创建数据传输通道，分别为如图1A所示的通道p1-p3和通道p2-p3。进程#1的消息通过通道p1-p3发送给进程#3，进程#2的消息通过通道p2-p3发送给进程#3。
随着计算机系统中节点数目的增加以及节点中进程数目的增加，图1A所示的数据传输系统需要为大量进程传输消息，相应的需要创建大量数据传输通道。假设图1A所示的计算机系统包括n个节点，每个节点运行m个进程，那么对于任意一个节点，数据传输系统需要创建m*n个数据传输通道。由于每个数据传输通道需要在相应节点中创建至少一个队列对(queue pair，QP)，而大量QP将占用节点过多的存储资源，因此，数据传输系统基于节点运行的进程来创建数据传输通道将影响数据传输系统的可伸缩性。
图1B示出了本申请实施例另一种可能的应用场景。图1B所示的计算机系统可以参考图1A所示的计算机系统进行理解，但是和图1A不同的是，在图1B所示的计算机系统中，数据传输系统是基于计算机系统中的节点来创建数据传输通道。仍然假设进程#1和进程#2需要分别向进程#3发送消息，在图1B所示的数据传输系统中，网络设备#1和网络设备#2为节点#1和节点#2创建一个数据传输通道，例如为图1B所示的通道n1-n2。进程#1的消息和进程#2的消息均通过通道n1-n2发送给进程#3。
假设图1B所示的计算机系统包括n个节点，每个节点运行m个进程，那么对于任意一个节点，数据传输系统最多需要创建n-1个数据传输通道。和图1A所示的数据传输系统相比，通过减少数据传输通道的数目，有利于减少QP占用的存储资源，可见，数据传输系统基于节点来创建数据传输通道将减少对数据传输系统的可伸缩性的影响。
以n为4为例，图1C示出了本申请实施例另一种可能的应用场景。和图1B相比，图1C所示的计算机系统除了包括节点#1和节点#2，还包括节点#3和节点#4。图1C所示的数据传输系统基于节点创建数据传输通道，分别例如为图1C所示的通道n1-n2、通道n1-n3、通道 n1-n4、通道n2-n3、通道n2-n4和通道n3-n4。对于图1C所示4个节点中的任意一个节点，数据传输系统为其创建3条数据传输通道。
下面，以节点#1和节点#2为例，进一步介绍图1B或图1C所示的计算机系统。
参考图2A，计算机系统可以至少包括节点#1和节点#2，节点#1包括主机11、支持RMDA的网络适配器(network interface card that supports RDMA，RNIC)12和内存13，节点#2包括主机21和RNIC 22。RNIC 22可以与主机21通信相连，RNIC 12分别与主机11和内存13通信相连。
图2A中的RNIC 12可以视为图1B或图1C中的网络设备#1，或者，图2A中的RNIC 12和内存13可以视为图1B或图1C中的网络设备#1。图2A中的RNIC 22可以视为图1B或图1C中的网络设备#2。图2A以图1B或图1C所示的网络设备#1包括RNIC 12、网络设备包括RNIC 22为例，本申请实施例不限定图1B或图1C所示的网络设备#1或网络设备#2的具体类型。
其中，主机11或主机21可以包括：处理器、通信接口以及存储器，处理器、通信接口以及存储器通过内部总线相互连接。其中，处理器可以由一个或者多个通用处理器构成，例如中央处理器(central processing unit，CPU)，或者CPU和硬件芯片的组合。主机11或主机21的存储器可以存储系统应用和/或应用进程的代码，处理器执行这些代码可以实现CPU核心113和/或进程的功能的功能。
其中，RNIC 12可以包括处理器122和缓存121，RNIC 22可以包括处理器222和缓存221。其中，处理器122或处理器222可以由一个或者多个通用处理器构成，例如CPU，或者CPU和硬件芯片的组合。处理器122和缓存121之间可以通过总线相连(图2A未示出)，也可以通过其他方式相连。处理器222和缓存221之间可以通过总线相连(图2A未示出)，也可以通过其他方式相连。
上述硬件芯片可以是专用集成电路(application-specific integrated circuit，ASIC)、可编程逻辑器件(programmable logic device，PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)、现场可编程逻辑门阵列(field-programmable gate array，FPGA)、通用阵列逻辑(generic array logic，GAL)或其任意组合。
其中，存储器或内存13可以包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)；存储器或内存13也可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)、快闪存储器(flash memory)、硬盘(hard disk drive，HDD)或固态硬盘(solid-state drive，SSD)或为双倍速率同步动态随机存储器(Double Data Rate Synchronous Dynamic Random Access Memory，DDR SDRAM)；存储器或内存13还可以包括上述种类的组合。其中，DDR SDRAM可以简称为DDR。缓存121或缓存221可以为一级或多级缓存，例如，可以采用寄存器和/或静态随机存取存储器(static random access memory，SRAM)实现。
在一种可能的实现方式中，RNIC 12和内存13可以集成在同一芯片上，或者说，该内存13可以为RNIC 12对应的网络设备#1中的内存。以内存13为DDR为例，可选的，该DDR可 以支持多通道技术，RNIC 12可以通过多个通道访存内存13。
或者，在一种可能的实现方式中，该内存13可以为主机11中的内存。RNIC 12与主机11之间可以通过输入输出(input output，IO)接口相连，该IO接口可以包括但不限于周边装置互连高速(peripheral component interconnect express，PCIe)等IO结构(fabric)接口。若内存13为主机11中的内存，那么RNIC 12可以通过与主机11之间的IO接口访存内存13。
RNIC 12和RNIC 22之间创建有数据传输通道，该数据传输通道可以理解为图1B或图1C或图2A所示的通道n1-n2，RNIC 12和RNIC 22通过通道n1-n2传输节点1和节点2之间的消息。RNIC 12可以在内存13中创建绑定通道n1-n2的至少一个共享发送队列(Shard Send Queue，SSQ)，SSQ可以用于处理来自节点#1中多个进程的、待传输至节点#2的消息。图2A示例性示出了该至少一个SSQ中的SSQ#1。可选的，若该至少一个SSQ还包括除SSQ#1以外的其他SSQ(例如SSQ#2)，SSQ#1和SSQ#2可以对应于不同的优先级，RNIC 12可以根据待传输至节点2的消息的优先级使用相应的SSQ。
主机11和/或主机21可以运行一个或多个进程。图2A示意性示出主机11运行的两个进程(进程#1和进程#2)和主机21运行的两个进程(进程#3和进程#4)。RNIC 12待传输至节点2的消息可以来自进程#1和/或进程#2。
假设进程#1和进程#2需要向主机21发送消息，进程#1和进程#2可以分别向RNIC 12提交相应的工作请求(work request，WR)。为了便于区分，本申请实施例将进程#1待发送的一个消息称作M1，将进程#2待发送的一个消息称作M2，将M1对应的WR称作WR1，将M2对应的WR称作WR2。作为举例，进程#1和进程#2可以分别调用主机11中的程序接口和RNIC 12的驱动程序向RNIC 12提交WR1和WR2。在一种可能的实现方式中，M1和M2可以为RDMA消息。
由于M1和M2均用于发送至节点2，因此，RNIC 12接收到WR1和WR2后，可以分别将WR1和WR2写入SSQ#1中。本申请实施例将写入SSQ#1中的WR1和WR2分别称作工作队列元素(work queue element，WQE)1和WQE2。WQE1和WQE2分别描述了M1和M2，例如，WQE1包括M1在内存111中的存储地址1，WQE2包括M2在内存13中的存储地址2。
在一种可能的实现方式中，内存13中还包括SSQ#1的上下文(简称上下文S1)，上下文S1记录了SSQ#1的使用信息。示例性的，上下文S1可以包括SSQ#1的写入索引(简称SSQ#1的PI)和读取索引(简称SSQ#1的CI)。RNIC 12可以根据SSQ#1的PI的取值确定WQE在SSQ#1中的写入位置，根据SSQ#1的CI的取值确定WQE在SSQ#1中的读取位置。
图2A示例性示出了SSQ#1中的三个放置WQE的存储位置，SSQ#1还可以包括更多或更少的WQE，并且，为了便于理解，图2A将SSQ#1中的用于保存WQE的三个存储位置绘制为从上至下依次排列的三个矩形。本申请实施例将图2A所示的SSQ#1中用于保存WQE的三个存储位置从上至下分别称作存储位置T、存储位置M和存储位置B。本申请实施例不限定SSQ#1中各WQE的存储方式，例如，SSQ#1可以对应于内存13中的一块连续存储空间，或者，可以对应于内存13中离散的存储空间。
作为举例，RNIC 12接收到WR1后，可以根据上下文S1中SSQ#1的PI指向SSQ#1的存 储位置T，之后可以将WQE1写入存储位置T，并更新SSQ#1的PI的取值，使其指向SSQ#1的存储位置M。RNIC 12接收到WR2后，可以根据上下文S1中SSQ#1的PI的取值将WQE2写入存储位置M，并更新SSQ#1的PI的取值，使其指向SSQ#1的存储位置B。
图2A示出了RNIC 12在SSQ#1中写入WQE1和WQE2后CI和PI在内存13中指向的存储位置。在一种可能的实现方式中，SSQ#1用于顺序处理写入SSQ#1中的WQE。假设WQE1先于WQE2写入SSQ#1，那么SSQ#1的CI的取值可以先指向WQE1，待RNIC 12处理WQE1后，将SSQ#1的CI的取值更新为指向WQE2(参考图2B)，以使得RNIC 12继续处理WQE2。
下面结合图2A中代表数据传输过程的虚线及其序号，介绍本申请实施例提供的M1发送至主机21的过程。
步骤1.RNIC 12将SSQ#1中的WQE1读取到缓存121中；
示例性的，RNIC 12可以根据上下文S1确定SSQ#1的CI的取值，根据CI的取值确定CI指向SSQ#1中的存储位置T，进而从存储位置T取出WQE(即WQE1)。
步骤2.RNIC 12访问内存111中WQE1所指示的存储位置1；
WQE1可以包括M1在内存111中的存储地址1，RNIC 12将WQE1读取到缓存121后，可以从WQE1中解析到存储地址1，之后可以访问内存111中的存储位置1。
步骤3.RNIC 12将存储位置1中的M1读取到缓存121；
RNIC 12可以将存储位置1中的数据(即M1)读取到缓存121。
步骤4.RNIC 12将M1封装为报文1,并将报文1通过通道n1-n2发送到RNIC 22；
可选的，RNIC 12可以根据RDMA协议将M1封装为报文1，并将报文1通过SSQ#1所绑定的通道n1-n2发送到RNIC 22。示例性的，该RDMA协议可以例如为无线带宽(infiniband)协议、或采用聚合以太网络版本1的RDMA(RDMA over Converged Ethernet version 1,RoCE1)协议、或采用聚合以太网络版本2的RDMA(RDMA over Converged Ethernet version2,RoCE2)协议、或互联网广域RDMA协议(internet wide area RDMA protocol,iWARP)等。
RNIC 22接收到报文1后，可以对报文1进行解封装，得到M1。
步骤5.RNIC 22将M1存储在主机21中；
假设M1为待存储在主机21中的数据，那么RNIC 22可以将M1存储在主机21中。进一步的，假设M1用于写入进程#3的内存211，作为举例，RNIC 22对报文1进行解封装后还可以得到M1在内存211中的写入位置，RNIC 22可以根据该写入位置将M1写入内存211中。
后文将结合RDMA的发送(send)模型、写(write)模型、读(read)模型和原子(atomic)模型具体介绍步骤4和步骤5，此处暂不展开介绍。
图2B所示的计算机系统通过代表数据传输过程的虚线及其序号，描述了本申请实施例提供的数据传输系统向进程#1通知M1的完成情况的过程。图2B中的结构可以解释为图2A所示的计算机系统中的相应结构，此处不再赘述。
参考图2B，RNIC 12可以在内存13中分别为进程#1和进程#2创建完成队列(complete queue，CQ)。本申请实施例将进程#1对应的CQ称作CQ#1，将进程#2对应的CQ称作CQ#2。CQ#1用于向进程#1反馈进程#1向RNIC 12提交的消息(例如M1)的完成情况，CQ#2用于向 进程#2反馈进程#2向RNIC 12提交的消息(例如M2)的完成情况。
下面结合图2B介绍本申请实施例提供的RNIC 12和RNIC 22向进程#1通知M1的完成情况的过程。
步骤6.RNIC 22通过通道n1-n2向RNIC 12发送携带R1的报文2；
RNIC 22接收到报文1后，可以通过通道n1-n2向RNIC 12发送报文2，报文2携带反馈消息(简称R1)，R1描述消息的完成情况。例如，消息的完成情况可以为RNIC 22成功接收消息，或RNIC 22成功将消息写入主机21，或RNIC 22未接收到消息，或消息写入失败等。RNIC 12接收到报文2后，可以对报文2进行解封装，得到R1。RNIC 12可以确定R1所描述的消息的完成情况。
可选的，R1为报文2中的某个字段(称作字段R)，字段R的不同取值对应于消息的不同完成情况。例如，字段R的取值为0时，RNIC 12可以确定消息的完成情况为成功，具体例如为RNIC 22成功接收消息，字段R的取值为1时，RNIC 12可以确定消息的完成情况为失败，具体例如为RNIC 22未接收到消息。
步骤7.RNIC 12将SSQ#1中的WQE1读取到缓存121中，以获得CQ#1的标识；
RNIC 12从通道n1-n2获取报文2后，虽然可以确定报文2中的R1对应于SSQ#1中的WQE，但是由于SSQ#1对应于主机11运行的多个进程，因此RNIC 12无法确定R1对应的完成情况对应于哪个进程，进而无法确认该R1应该交由哪个CQ来处理。为此，RNIC 12在确认绑定通道n1-n2的SSQ为SSQ#1之后，可以根据SSQ#1的上下文S1确定R1对应于SSQ#1中的WQE1。本申请实施例中，可以在WQE1中携带CQ#1的标识。相应的，RNIC 12将WQE1从SSQ#1中读取到缓存121中后，可以解析WQE1以获取WQE1CQ#1的标识，从而确定需要将R1对应的完成情况交由CQ#1来处理。
步骤8.RNIC 12根据CQ#1的标识和R1向CQ#1中写入CQE1；
步骤9.RNIC 12处理CQ#1中的CQE，当处理到CQE1时，向进程#1通知M1的完成情况。
下面对步骤8和步骤9进行介绍。
RNIC 12获取CQ#1的标识后，可以根据R1向CQ#1中写入完成队列元素(CQ element，CQE)1，CQE1用于确定M1的完成情况。RNIC 12在处理CQ#1中的CQE的过程中，当处理到CQE1时，可以向进程#1通知M1的完成情况。
可选的，RNIC 12向进程#1通知M1的完成情况可以指，进程#1调用程序接口和RNIC 12的驱动程序对CQ#1中的CQE进行检索，当检索到CQE1时，进程#1能够获取M1的完成情况。
可选的，CQE通过携带的部分字段(例如称作错误码)来指示相应消息的完成情况。例如，进程#1解析CQE1，确定CQE1中错误码的取值为0时，进程#1可以确定M1的完成情况为成功，具体例如为M1对应的数据传输任务已完成。例如，进程#1解析CQE1，确定CQE1中错误码的取值为1时，进程#1可以确定M1的完成情况为失败，具体例如为M1对应的数据传输任务未完成。
可选的，RNIC 12可以根据R1确定CQE1中错误码的取值。可选的，CQE1中错误码的取值所描述的完成情况与R1所描述的完成情况一致。例如，若R1描述消息的完成情况为成功，那么CQE1中错误码的取值可以为0，若R1描述消息的完成情况为失败，那么CQE1中错误码的取值可以为1。或者，可选的，CQE1中错误码的取值所描述的完成情况与R1描述的完成情 况不一致。例如，若R1描述消息的完成情况为成功，但是RNIC 12因故障可能未正确的封装报文1(M1错误或目的地址错误等)，那么CQE1中错误码的取值可以为1。
下面举例介绍RNIC 12在CQ#1中写入CQE和读取CQE的过程。
在一种可能的实现方式中，内存13中还包括CQ#1的上下文(简称上下文C1)，上下文C1记录了CQ#1的使用信息，示例性的，上下文C1可以包括CQ#1的写入索引(简称CQ#1的PI)和读取索引(简称CQ#1的CI)。RNIC 12可以根据PI的取值确定CQE在CQ#1中的写入位置，根据CI的取值确定CQE在CQ#1中的读取位置。
图2B仅示例性示出了CQ#1中的三个放置CQE的存储位置，CQ#1还可以包括更多或更少的CQE，并且，为了便于理解，图2B将CQ#1中的用于保存CQE的三个存储位置绘制为从上至下依次排列的三个矩形。本申请实施例将图2B所示的CQ#1中用于保存CQE的三个存储位置从上至下分别称作存储位置T、存储位置M和存储位置B。本申请实施例不限定CQ#1中各CQE的存储方式，例如，CQ#1可以对应于内存13中的一块连续存储空间，或者，可以对应于内存13中离散的存储空间。
参考图2B，作为举例，RNIC 12生成CQE4后，可以根据上下文C1确定CQ#1的PI指向CQ#1的存储位置T，之后在存储位置T写入CQE4，并更新CQ#1的PI的取值，以使其指向CQ#1的存储位置M。之后，RNIC 12生成CQE3后，可以根据上下文C1中CQ#1的PI的取值在存储位置M写入CQE3，并更新CQ#1的PI的取值，使其指向存储位置B。
图2B示出了RNIC 12在CQ#1中写入CQE4和CQE2后CI和PI在内存13中指向的存储位置。在一种可能的实现方式中，CQ#1用于顺序处理写入CQ#1中的CQE。假设CQE4先于CQE3写入CQ#1，那么CQ#1的CI的取值可以先指向CQE4，RNIC 12处理CQE4，向进程#1通知CQE4对应的消息(称作M4)的完成情况，之后，将CQ#1的CI的取值更新为指向CQE3，以使得RNIC 12继续处理CQE3，向进程#1通知CQE3对应的消息(称作M3)的完成情况。
参考图2B，假设RNIC 12生成CQE1后，根据CQ#1的标识从上下文C1的取值向CQ#1的存储位置B写入CQE1。待RNIC 12依次处理CQE4和CQE3后，将CQ#1的CI的取值更新为指向CQ#1的存储位置B，以使得RNIC 12继续处理CQ#1的存储位置B中的CQE3，向进程#1通知M1的完成情况。
以上结合图2A和图2B介绍了步骤1至步骤9对应的数据传输流程。在上述步骤6后，由于通道n1-n2所绑定的SSQ保存主机11中多个进程的WQE，因此，RNIC 12无法确定通道n1-n2返回的报文2对应于主机11中的哪个进程，也无法确定需要通过内存13中的哪个CQ来处理报文2中的R1。为此，RNIC 12需要通过步骤7来获取相应CQ的标识(即CQ#1的标识)，具体的，先确认绑定通道n1-n2的SSQ为SSQ#1，之后根据SSQ#1的上下文S1确定R1对应于SSQ#1中的WQE1，将WQE1从SSQ#1中读取到缓存121中，解析WQE1以获取CQ#1的标识。由于步骤7包括RNIC 12从内存13中读取WQE1的过程，因此步骤7增加了RNIC 12对内存13的读时延，降低了RNIC 12向进程#1通知M1的完成情况的效率。并且，由于缓存121的空间有限，将WQE1从内存13中读取到缓存121中，可能导致预取到缓存121中的待发送的WQE被老化删除，导致发送业务的延时。
在分析发现了上述延时以及导致延时的原因后，本申请实施例对步骤1至步骤9中的部 分内容进行优化。下面具体介绍优化方案。
1、对步骤4进行优化。对步骤4优化前，报文1包括M1，对步骤4优化后，参考图2A步骤4的括号中的内容，在M1的基础上，在报文1中增加标识信息。可选的，该标识信息可以为CQ#1的标识和/或进程#1的标识。
2、对步骤6进行优化。对步骤6优化前，报文2包括R1，对步骤6优化后，参考图2B中步骤6的括号中的内容，在报文2中增加报文1所携带的全部或部分标识信息。
3、不执行步骤7。由于报文2携带CQ#1的标识，因此，参考2B中步骤7对应的虚线上的“×”，RNIC 12可以无需执行步骤7来获取CQ#1的标识。
基于上述步骤1至步骤9以及优化内容的构思，图3示出了本申请计算机系统另一种可能的实施例。
参考3，计算机系统3可以包括数据传输系统31、第一主机32和第二主机33。数据传输系统31包括第一网络设备311和第二网络设备312，第一网络设备311设置于第一主机32上，第二网络设备312设置于第二主机上。该数据传输系统31可以为RDMA数据传输系统，第一网络设备311和第二网络设备312之间可以传输RDMA数据传输消息。
可选的，计算机系统3可以解释为图1B或图1C或图2A或图2B所示的计算机系统。可选的，数据传输系统31可以解释为图1B或图1C或图2A或图2B所示的数据传输系统。可选的，第一主机32可以解释为包括图2A或图2B所示的主机11。可选的，第二主机33可以解释为包括图2A或图2B所示的主机21。可选的，第一网络设备311可以解释为包括图1B或图1C所示的网络设备#1，或者解释为包括图2A或图2B所示的RNIC 12，或者解释为包括图2A或图2B所示的RNIC 12和内存13。可选的，第二网络设备312可以解释为包括图1B或图1C所示的网络设备#2，或者解释为包括图2A或图2B所示的RNIC 22。
其中，第一主机32可以运行一个或多个进程，第一网络设备311可以创建供这一个或多个进程中的多个进程使用的SSQ。参考图2A或图2B所示的计算机系统，可选的，第一主机32运行的一个或多个进程可以例如包括进程#1和进程#2，第一网络设备311创建的SSQ可以例如为内存13中的SSQ#1。
可选的，SSQ可以设置在第一网络设备的内存中，或者可以设置在第一主机的内存中。示例性的，SSQ可以设置在图2A或图2B所示的内存13中。
为了便于描述，本申请实施例将使用该SSQ的多个进程中的一个进程称作第一进程，例如，第一进程可以解释为图2A或图2B对应的实施例中的进程#1。下面以数据传输系统31处理第一进程的数据传输消息的过程为例，介绍数据传输系统31中第一网络设备311和第二网络设备312的功能。
第一网络设备311可以用于从SSQ获取第一进程的数据传输消息。可选的，该数据传输消息可以解释为图2A或图2B对应的实施例中的M1。可选的，该SSQ具体用于保存来自多个进程的工作请求，第一网络设备311从SSQ获取数据传输消息具体可以指，第一网络设备311用于从SSQ获取来自第一进程的第一工作请求，之后根据第一工作请求获取数据传输消息。其中，第一工作请求描述数据传输消息。
可选的，SSQ所保存的来自多个进程的工作请求可以例如分别参考图2A或图2B所示的WQE1和WQE2进行理解，第一工作请求可以参考图2A或图2B所示的WQE1进行理解。
图4示出了第一工作请求一种可能的结构。参考图4，第一工作请求可以包括控制字段、任务字段和内存描述字段。
其中，任务字段可以用于描述第一工作请求的格式信息。其中，任务字段可以包括第一网络设备311处理数据传输消息的指示信息，例如，任务字段可以包括标识信息，可选的，该标识信息可以包括第一进程对应的第一标识。可选的，该标识信息中的第一标识可以解释为图2A或图2B对应的实施例中的CQ#1的标识和/或进程#1的标识。该标识信息还可以包括其他信息，后文将介绍该其他信息的可能内容，此处暂不展开介绍。
可选的，第一工作请求还可以包括内存描述字段。内存描述字段可以用于描述第一网络设备311和/或第二网络设备312注册的内存空间。第一网络设备311可以根据内存描述字段从第一主机32中获取数据传输消息。可选的，内存描述字段可以包括地址字段，地址字段可以用于确定该内存空间的起始位置。可选的，内存描述字段还可以包括长度(length)字段，用于确定该内存空间的长度。可选的，内存描述字段还可以包括钥匙(key)字段，用于唯一标识该内存空间。
第一网络设备311还可以根据该数据传输消息和标识信息封装第一报文，之后向第二网络设备发送第一报文。可选的，第一网络设备311可以根据RDMA协议封装第一报文。
可选的，该第一报文可以解释为图2A或图2B对应的优化后的实施例中的报文1，即第一报文可以包括数据传输消息和标识信息。
第二网络设备312可以用于接收第一报文，对第一报文进行解封装，以获得数据传输消息和标识信息。之后，第二网络设备312可以根据数据传输消息生成反馈消息，用以指示数据传输消息的完成情况。可选的，第二网络设备312在获得数据传输消息后生成反馈消息，用以通知第一网络设备311已经成功接收数据传输消息。或者，第二网络设备312可以根据数据传输消息是否成功写入第二主机33来生成相应的反馈消息。例如，若数据传输消息成功写入第二主机33，该反馈消息指示的完成情况可以为成功，若数据传输消息写入第二主机33失败，该反馈消息指示的完成情况可以为失败。可选的，该反馈消息可以解释为图2B对应的实施例中的R1。
第二网络设备312还可以用于根据反馈消息和标识信息封装第二报文，向第一网络设备311发送第二报文。可选的，第二报文可以解释为图2B对应的优化后的实施例中的报文2，即第二报文可以包括第一标识和反馈消息。
第一网络设备311还可以用于接收第二报文，对第二报文进行解封装，以获得反馈消息和标识信息。之后，第一网络设备311可以根据第二报文中的标识信息和反馈消息通知第一进程数据传输消息的完成情况。
可选的，第一网络设备311可以为第一主机32运行的多个进程创建多个CQ，每个进程对应多个CQ中的部分CQ(例如一个CQ)，每个CQ用于向相应进程通知消息的完成情况。本 申请实施例将该多个CQ中对应于第一进程的CQ称作第一CQ。
可选的，第一网络设备311所创建的CQ可以解释为图2B对应的实施例中的CQ#1或CQ#2。示例性的，参考图2A或图2B对应的实施例，第一网络设备311可以为进程#1创建CQ#1，为进程#2创建CQ#2。其中，CQ#1用于向进程#1通知进程#1提交的消息的完成情况，CQ#2用于向进程#2通知进程#2提交的消息的完成情况。
第一网络设备311根据第二报文中的第一标识和反馈消息通知第一进程数据传输消息的完成情况可以指，第一网络设备311从多个CQ中确定第一标识对应的CQ(称作第一CQ)，根据反馈消息在第一CQ中写入CQE(称作第一CQE)，第一CQE描述数据传输消息的完成情况。可选的，第一CQE可以解释为图2B对应的实施例中的CQE1。
可选的，第一CQE通过携带的部分字段(例如称作错误码)来指示数据传输消息的完成情况。例如，第一进程解析第一CQE，若第一CQE中错误码的取值为0，第一进程可以确定数据传输消息的完成情况为成功，具体例如为数据传输消息已完成传输等。例如，第一进程解析第一CQE，若第一CQE1中错误码的取值为1，第一进程可以确定该数据传输消息的完成情况为失败，具体例如为数据传输消息未完成传输等。
可选的，第一网络设备311通知的数据传输消息的完成情况可以与反馈消息所描述的完成情况一致。例如，若反馈消息描述数据传输消息的完成情况为成功，那么第一网络设备311可以通知第一进程数据传输消息已完成或传输成功。若反馈消息描述数据传输消息的完成情况为失败，那么第一网络设备311可以通知第一进程数据传输消息未完成或传输失败。
或者，可选的，第一网络设备311通知的数据传输消息的完成情况可以与反馈消息所描述的完成情况不一致。例如，若反馈消息描述数据传输消息的完成情况为成功，但是第一网络设备311因故障可能未正确的封装第一报文(例如封装的数据传输消息错误或目的地址错误等)，那么第一网络设备311可以通知第一进程数据传输消息未完成或传输失败。
图3对应的实施例中，第一网络设备311可以创建供第一主机32运行的多个进程使用的SSQ，第一网络设备311从SSQ获取第一进程的数据传输消息后，可以在封装数据传输消息的报文(即第一报文)中同时封装第一进程对应的第一标识。第二网络设备312接收到第一报文后，可以在封装反馈消息的报文(即第二报文)中同时封装第一标识，这样，第一网络设备312在接收到第二报文后，无需根据SSQ的上下文来确定第二报文中的反馈消息对应于第一进程，而是可以根据第二报文中的第一标识确定该反馈消息对应于第一进程，有利于高效的通知第一进程已处理该数据传输消息，进而有利于提高第一进程的运行效率。
下面对图3对应的实施例进行补充介绍。
一、上述图3对应的实施例提到，第一标识可以解释为图2A或图2B对应的实施例中的CQ#1的标识和/或进程#1。可选的，本申请实施例不限定第一标识的具体含义，只要第一标识能够用于确定第一CQ即可。
以第一标识为第一进程的标识为例，介绍第一网络设备311根据第一进程的标识确定第一CQ的方法。第一网络设备311可以保存映射表，该映射表记录进程与CQ之间的对应关系。第一网络设备311解封装第二报文得到第一进程的标识后，可以在映射表中查找第一进程与第一CQ对应关系，进而确定第一CQ的标识。继续参考图2B对应的实施例，RNIC 12在获取 报文2携带的进程#1的标识后，可以从映射表中获取CQ#1的标识，进而将CQE1写入CQ#1。
可选的，第一标识可以包括第一进程的标识和第一CQ的标识，第一CQE可以包括第一进程的标识。可选的，第一进程和其他进程可以共享同一CQ，第一进程通过第一CQE携带的第一进程的标识可以确定第一CQE为自己的CQE。这样，虽然第一网络设备311创建的CQ仍然不被所有进程共享，但是通过让部分进程共享一个CQ，有利于减少创建的CQ的数目，从而有利于节约内存空间。
二、图5A通过带有箭头的实线线段示例性示出了第一网络设备311获取进程的数据传输消息的过程，通过带有箭头的虚线线段示例性示出了第一网络设备311通知进程数据传输消息的完成情况的过程。结合图2A或图2B对应的实施例，下面介绍图5A所描述的过程。
参考图1C对应的实施例，假设图2A或图2B所示的计算机系统还包括节点#3，RNIC 12创建的SSQ#2可以绑定节点#1和节点#3之间的通道n1-n3，SSQ#2用于获取发往节点#3的消息。假设进程#1待传输的消息依次包括M4、M3和M1，其中，M4和M3用于发往节点#3。假设进程#2带传输的消息依次包括M5和M2，其中，M5用于发往节点#3。
RNIC 12可以根据SSQ#2的PI的取值依次将M5对应的WQE5、M4对应的WQE4和M3对应的WQE3写入SSQ#2中，并且根据SSQ#1的PI的取值依次将M1对应的WQE1和M2对应的WQE2写入SSQ#1。之后，RNIC 12可以根据SSQ#2的CI的取值依次处理SSQ#2中的WQE，发送相应的消息，之后RNIC 12可以根据SSQ#1的CI的取值依次处理SSQ#1中的WQE，发送相应的消息。假设RNIC 12从先至后依次发送M5、M4、M3、M1和M2。
RNIC 12在接收到携带反馈消息的报文后，可以根据报文携带的第一标识确定相应的CQ。假设RNIC 12从先至后依次接收M5对应的报文a、M4对应的报文b、M3对应的报文c、M1对应的报文d和M2对应的报文e。RNIC 12根据报文a携带的CQ#2的标识在CQ#2中新增CQE-a，根据报文b携带的CQ#1的标识在CQ#1中新增CQE-b，类似的在CQ#1中新增WQE-c和CQE-d，在CQ#2中新增CQE-e。
进程#1根据CQ#1中各CQE的写入顺序，可以分别确定CQE-b对应于M4、CQE-c对应于M3、CQE-d对应于M1。类似的，进程#2根据CQ#2中各CQE的写入顺序，可以分别确定CQE-a对应于M5、CQE-e对应于M2。
为了更加高效准确的通知进程消息的完成情况，可选的，第一报文和第二报文中的标识信息还可以包括第二标识，第二标识用于从第一进程的多个数据传输消息中确定数据传输消息。相应的，第一网络设备具体用于根据第一标识、反馈消息和第二标识通知第一进程数据传输消息的完成情况。可选的，第一CQE可以包括第二标识。
继续参考图5A，可选的，M4、M3和M1对应的第二标识可以分别为相应消息的序号。例如，M4、M3和M1的第二标识可以分别为1、2和3。CQE-b、CQE-c和CQE-d携带的第二标识分别为1、2和3。进程#1可以根据CQ#1中各CQE的第二标识确定CQE-b对应于M4、CQE-c对应于M3、CQE-d对应于M1。
参考图5B，假设进程#1通过调用程序接口和RNIC 12的驱动程序依次向RNIC 12提交描述M4的WR4、描述M3的WR3和描述M1的WR1，进程#2通过调用程序接口和RNIC 12的驱动程序依次向RNIC 12提交描述M5的WR5和描述M2的WR2。RNIC 12依次根据WR5、WR4、WR3、 WR1和WR2在SSQ#2和SSQ#1中新增相应的WQE。那么，可选的，消息的第二标识可以为相应WR的标识(例如WR的序号)。RNIC 12在CQ#1和CQ#2中分别新增相应消息的CQE后，进程#1可以根据CQ#1中各CQE的第二标识确定CQE对应的消息，例如，通过确定CQE-b对应于WR4，进而确定CQE-b对应于M4。
三、下面对图3对应的实施例所涉及的数据传输消息、第一报文和第二报文进行补充介绍。
1)可选的，第一进程的数据传输消息可以包括待写入第二主机33中的第一数据。第二网络312设备用于在获得第一数据后，在第二主机上存储第一数据。
可选的，第一网络设备311可以将第一数据分为多个数据片(segment)，并且可以将多个片段依次封装至多个报文(称作第一报文序列)中。相应的，图3对应的实施例中涉及的第一报文可以指第一报文序列。
可选的，第一网络设备311可以在第一报文序列中的每个报文中均封装标识信息，或者，仅在第一报文序列中的最后一个报文中封装标识信息。可选的，标识信息可以封装在报文的扩展头中，第一数据或数据片可以作为报文的净荷数据。下面，以第一网络设备311在第一报文序列中的每个报文中均封装标识信息为例，介绍第一报文和第二报文的发送过程。
示例性的，图3对应的实施例中，数据传输系统31可以采用发送(send)操作或写(write)操作或原子(atomic)操作来执行数据传输消息对应的工作请求。
图6A示出了发送操作对应的第一报文和第二报文的传输过程。参考图6A，第一报文序列包括报文#1、报文#2、……、报文#k，报文#1、报文#2、……、报文#k中分别携带第一数据中的数据片#1、数据片#2、……、数据片#k，k为任意一个正整数。并且，第一报文序列中的每个报文均携带标识信息。可选的，该标识信息可以包括前文介绍的第一标识，还可以包括第二标识。此外，该标识信息还可以第三标识。该第三标识可以包括第二进程的标识和/或第二进程对应的RQ的标识。图6A示例性示出了标识信息中的第一标识和RQ的标识。
以第三标识包括第二进程对应的RQ为例，第二网络设备312接收到第一网络设备311发送的第一报文序列后，可以从第三标识对应的RQ中读取WQE，该WQE描述第二进程对应的存储空间。之后第二网络设备312可以将第一数据保存在WQE描述的存储空间。第二网络设备312接收第一报文序列后，可以向第一网络设备311发送确认报文，该确认报文携带确认字符(acknowledge character，ACK)，因此，本申请实施例将该确认报文称作ACK报文。该ACK报文可以携带前文介绍的反馈消息(例如为确认字符)，还可以携带第一报文序列中标识信息中的全部或部分内容。例如，ACK报文还可以携带第一标识。
图6B示出了写操作对应的第一报文和第二报文的传输过程。参考图6B，第一报文序列包括报文#1、报文#2、……、报文#k，报文#1、报文#2、……、报文#k中分别携带第一数据中的数据片#1、数据片#2、……、数据片#k，k为大于或等于3的正整数。并且，第一报文序列中的每个报文均携带标识信息。可选的，该标识信息可以包括前文介绍的第一标识，可选的，还可以包括第二标识。图6B示例性示出了标识信息中的第一标识。
此外，报文#1中还可以封装有第一数据在第二主机33中的写入位置。第二网络设备312接收到第一报文序列后，可以根据该写入位置将第一数据存储在第二主机33。第二网络 设备312接收第一报文序列后，可以向第一网络设备311发送ACK报文。该ACK报文可以携带前文介绍的反馈消息(例如为确认字符)，还可以携带第一报文序列中标识信息中的全部或部分内容。例如，ACK报文还可以携带第一标识。
图6C示出了原子操作对应的第一报文和第二报文的传输过程。参考图6C，第一报文包括第一数据和标识信息。可选的，该标识信息可以包括前文介绍的第一标识，可选的，还可以包括第二标识。图6C示例性示出了标识信息中的第一标识。第二网络设备312接收到第一报文后，可以将第一数据存储在第二主机33。第二网络设备312接收第一报文后，可以向第一网络设备311发送第二报文，该第二报文可以例如为原子反馈(atomic response)报文或原子确认(atomic acknowledge)报文。第二报文可以携带前文介绍的反馈消息(例如为确认字符)，还可以携带第一报文序列中标识信息中的全部或部分内容。例如，第二报文还可以携带第一标识。
2)可选的，数据传输消息可以包括第一数据的源地址和目的地址，并且，第一数据的源地址指向第二主机33，第一数据的目的地址指向第一主机32，因此，数据传输消息可以不携带第一数据。
第二网络设备312可以用于在获得第一报文中封装的源地址、目的地址和标识信息后，根据源地址在第二主机33上读取第一数据，根据第一数据、标识信息、反馈消息和目的地址封装第二报文，并向第一主机32发送第二报文。图6B示例性示出了标识信息中的第一标识。
可选的，标识信息可以封装在报文的扩展头中。第一网络设备311用于在获得第二报文中的第一数据和目的地址后，根据目的地址在第一主机上存储第一数据。第一网络设备311还用于根据第二报文中的第一标识和反馈消息通知第一进程数据传输消息的完成情况。
可选的，第二网络设备312可以将第一数据分为多个片段，并且可以将多个片段依次封装至多个报文(称作第二报文序列)中。相应的，图3对应的实施例中涉及的第二报文可以指第二报文序列。
可选的，第二网络设备312可以在第二报文序列中的每个报文中均封装标识信息中的部分或全部内容，或者，仅在第二报文序列中的最后一个报文中封装标识信息中的部分或全部内容。可选的，标识信息可以封装在报文的扩展头中，第一数据或数据片可以作为报文的净荷数据。下面，以第一网络设备311在第一报文序列中的每个报文中均封装第一标识为例，介绍第一报文和第二报文的发送过程。
图6D示出了读(read)操作对应的第一报文和第二报文的传输过程。参考图6D，第一报文包括源地址、目的地址和第一标识，第二网络设备312接收到第一报文后，可以根据源地址从第二主机33中读取第一数据，之后对第一数据进行分片，依次得到数据片#1、数据片#2、……、数据片#k，k为大于或等于3的正整数。之后第二网络设备312可以封装并发送第二报文序列，第二报文序列包括报文#1、报文#2、……、报文#k，报文#1、报文#2、……、报文#k中分别携带数据片#1、数据片#2、……、数据片#k。此外，第二报文序列中的每个报文还可以包括目的地址、第一标识和反馈消息。第一网络设备311接收到第二报文后，可以根据目的地址将第一数据存储在第一主机32中，根据第一标识和反馈消息通知第一进程该数据传输消息的完成情况。
以上介绍了本申请实施例提供的计算机系统和数据传输系统，基于同一发明构思，本申请实施例还提供一种数据传输方法，该方法可以为RDMA数据传输方法。参考图7，该数据传输方法可以包括步骤S701至步骤S704。
S701、第一网络设备从SSQ获取第一进程的RDMA数据传输消息；
第一网络设备可以设置在第一主机上，第一进程是第一主机运行的使用共享发送队列的多个进程中的任意一个进程，该数据传输消息可以为RDMA数据传输消息。
S702、第一网络设备向第二网络设备发送第一报文；
第二网络设备设置在第二主机上，第一报文包括数据传输消息和第一进程对应的第一标识。
第二网络设备接收来自第一网络设备的第一报文，第一网络设备设置于第一主机上，第二网络设备设置于第二主机上，第一报文包括第一进程的数据传输消息和第一进程对应的第一标识，数据传输消息为第一网络设备从共享发送队列获取的，第一进程是第一主机运行的使用共享发送队列的多个进程中的任意一个进程。
S703、第二网络设备根据第一报文向第一网络设备发送第二报文，第二报文包括第一标识和反馈消息；
S704、第一网络设备根据第二报文中的第一标识和反馈消息通知第一进程数据传输消息的完成情况。
第一网络设备接收来自第二网络设备的第二报文，第二报文包括第一标识和反馈消息，反馈消息指示数据传输消息的完成情况。第一网络设备根据第二报文中的第一标识和反馈消息通知第一进程数据传输消息的完成情况。
需要说明的是，步骤S701、步骤S702和步骤S704对应的方法为第一网络设备所执行的方法，该方法可以视为图3对应实施例中第一网络设备所执行的方法。图7所示的第一网络设备所执行的步骤S701、步骤S702和步骤S704中未详尽描述的实现方式和技术效果可以参见图3所示的实施例中第一网络设备的相关描述。对于步骤S701、步骤S702和步骤S704对应的方法，本申请实施例不限定步骤S703中第二网络设备具体执行的方法，只要第一网络设备向第二网络设备发送第一报文后，第二网络设备可以向第一网络设备发送包括第一标识的第二报文即可。
需要说明的是，步骤S703对应的方法为第二网络设备所执行的方法，该方法可以视为图3对应实施例中第二网络设备所执行的方法。图7所示的第二网络设备所执行的步骤S703中未详尽描述的实现方式和技术效果可以参见图3所示的实施例中第二网络设备的相关描述。对于步骤S702对应的方法，本申请实施例不限定第一网络设备所执行的步骤，只要第一网络设备能够向第二网络设备发送包括第一标识的第一报文即可。
上述详细阐述了本申请实施例的方法，为了便于更好的实施本申请实施例的上述方案，相应地，下面还提供用于配合实施上述方案的相关设备。
参见图8，图8是本申请实施例提供的一种网络设备的结构示意图，该网络设备可以是上述图3或图7对应的实施例中的第一网络设备，可以执行相应实施例中第一网络设备所执 行的方法和步骤。
如图8所示，该网络设备800可以包括获取单元801，用于从共享发送队列获取第一进程的RDMA数据传输消息，网络设备设置在第一主机上，第一进程是第一主机运行的使用共享发送队列的多个进程中的任意一个进程。网络设备800还可以包括发送单元802，用于向第二网络设备发送第一报文，第二网络设备设置在第二主机上，第一报文包括RDMA数据传输消息和第一进程对应的第一标识。还可以包括接收单元803，用于接收来自第二网络设备的第二报文，第二报文包括第一标识和反馈消息，反馈消息指示RDMA数据传输消息的完成情况。还可以包括完成单元804，用于根据第二报文中的第一标识和反馈消息通知第一进程RDMA数据传输消息的完成情况。
在一种可能的实现方式中，第一报文和第二报文还包括第二标识，第二标识用于从第一进程的多个RDMA数据传输消息中确定RDMA数据传输消息。完成单元804具体用于根据第一标识、反馈消息和第二标识通知第一进程RDMA数据传输消息的完成情况。
在一种可能的实现方式中，共享发送队列具体用于保存来自多个进程的工作请求。获取单元801具体用于从共享发送队列获取来自第一进程的第一工作请求，第一工作请求描述RDMA数据传输消息，根据第一工作请求获取RDMA数据传输消息。
在一种可能的实现方式中，完成单元804具体用于，根据第一标识从多个完成队列中确定第一进程对应的第一完成队列，根据反馈消息在第一完成队列中写入工作完成元素，工作完成元素用于通知第一进程RDMA数据传输消息的完成情况。
应理解，网络设备800包括的各单元可以软件模块、也可以为硬件模块、或部分为软件模块部分为硬件模块。
关于网络设备800的可能实现方式及有益效果可以参考图3或图7对应的实施例的相关内容，此处不再赘述。
需要说明的是，上述网络设备800的结构仅仅作为一种示例，不应构成具体限定，可以根据需要对网络设备中的各个单元进行增加、减少或合并。此外，该网络设备800中的各个单元的操作和/或功能是为了实现上述图3或图7所描述的第一网络设备的功能或方法，为了简洁，在此不再赘述。
参见图9，图9是本申请实施例提供的另一种网络设备的结构示意图，该网络设备可以是上述图3或图7对应的实施例中的第二网络设备，可以执行相应实施例中第二网络设备所执行的方法和步骤。
如图9所示，该网络设备900可以包括接收单元901，用于接收来自第一网络设备的第一报文，第一网络设备设置于第一主机上，网络设备设置于第二主机上，第一报文包括第一进程的RDMA数据传输消息和第一进程对应的第一标识，RDMA数据传输消息为第一网络设备从共享发送队列获取的，第一进程是第一主机运行的使用共享发送队列的多个进程中的任意一个进程。还可以包括发送单元902，用于根据第一报文向第一网络设备发送第二报文，第二报文包括第一标识和反馈消息，第一标识和反馈消息指示第一网络设备通知第一进程RDMA数据传输消息的完成情况。
在一种可能的实现方式中，第一报文和第二报文还包括第二标识，第二标识用于从第一进程的多个RDMA数据传输消息中确定RDMA数据传输消息，第一标识、反馈消息和第二标识 指示第一网络设备通知第一进程RDMA数据传输消息的完成情况。
应理解，网络设备900包括的各单元可以软件模块、也可以为硬件模块、或部分为软件模块部分为硬件模块。
关于网络设备900的可能实现方式及有益效果可以参考图3或图7对应的实施例的相关内容，此处不再赘述。
需要说明的是，上述网络设备900的结构仅仅作为一种示例，不应构成具体限定，可以根据需要对网络设备中的各个单元进行增加、减少或合并。此外，该网络设备900中的各个单元的操作和/或功能是为了实现上述图3或图7所描述第二网络设备的功能或方法，为了简洁，在此不再赘述。
本申请还提供一种计算机可读存储介质，其中，计算机可读存储介质存储有计算机程序，当该计算机程序被处理器执行时，可以实现上述方法实施例中记载的任意一种的部分或全部步骤。
本发明实施例还提供一种计算机程序，该计算机程序包括指令，当该计算机程序被计算机执行时，使得计算机可以执行任意一种方法的部分或全部步骤。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其它实施例的相关描述。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某些步骤可能可以采用其它顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置，可通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如上述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性或其它的形式。
上述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
本申请实施例中，多个是指两个或两个以上，本申请不做限制。在本申请实施例中，“/”可以表示前后关联的对象是一种“或”的关系，例如，A/B可以表示A或B；“和/或”可以用于描述关联对象存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况，其中A，B可以是单数或者复数。为了便于描述本申请实施例的技术方案，在本申请实施例中，可以采用“第一”、“第二”等字样对功能相同或相似的技术特 征进行区分。该“第一”、“第二”等字样并不对数量和执行次序进行限定，并且“第一”、“第二”等字样也并不限定一定不同。在本申请实施例中，“示例性的”或者“例如”等词用于表示例子、例证或说明，被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念，便于理解。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于系统实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的范围。这样，倘若本申请的这些修改和变型属于本发明权利要求的范围之内，则本发明也意图包括这些改动和变型在内。

Claims (18)

  1. 一种远程直接内存访问RDMA数据传输系统，其特征在于，所述RDMA数据传输系统包括第一网络设备和第二网络设备，所述第一网络设备设置于第一主机上，所述第二网络设备设置于第二主机上；
    所述第一网络设备，用于从共享发送队列获取第一进程的RDMA数据传输消息，向所述第二网络设备发送第一报文，其中，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程，所述第一报文包括所述RDMA数据传输消息和所述第一进程对应的第一标识；
    所述第二网络设备，用于接收所述第一报文，根据所述第一报文向所述第一网络设备发送第二报文，其中，所述第二报文包括所述第一标识和反馈消息，所述反馈消息指示所述RDMA数据传输消息的完成情况；
    所述第一网络设备，还用于接收所述第二报文，根据所述第二报文中的所述第一标识和所述反馈消息通知所述第一进程所述RDMA数据传输消息的完成情况。
  2. 根据权利要求1所述的系统，其特征在于，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息；
    所述第一网络设备具体用于根据所述第一标识、所述反馈消息和所述第二标识通知所述第一进程所述RDMA数据传输消息的完成情况。
  3. 根据权利要求1或2所述的系统，其特征在于，所述共享发送队列具体用于保存来自所述多个进程的工作请求；
    所述第一网络设备具体用于从所述共享发送队列获取来自所述第一进程的第一工作请求，根据所述第一工作请求获取所述RDMA数据传输消息，其中，所述第一工作请求描述所述RDMA数据传输消息。
  4. 根据权利要求1至3中任一项所述的系统，其特征在于，所述第一网络设备具体用于根据所述第一标识从多个完成队列中确定所述第一进程对应的第一完成队列，根据所述反馈消息在所述第一完成队列中写入工作完成元素，其中，所述工作完成元素描述所述RDMA数据传输消息的完成情况。
  5. 一种远程直接内存访问RDMA数据传输方法，其特征在于，包括：
    第一网络设备从共享发送队列获取第一进程的RDMA数据传输消息，所述第一网络设备设置在第一主机上，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程；
    所述第一网络设备向第二网络设备发送第一报文，所述第二网络设备设置在第二主机上，所述第一报文包括所述RDMA数据传输消息和所述第一进程对应的第一标识；
    所述第一网络设备接收来自所述第二网络设备的第二报文，所述第二报文包括所述第一标识和反馈消息，所述反馈消息指示所述RDMA数据传输消息的完成情况；
    所述第一网络设备根据所述第二报文中的所述第一标识和所述反馈消息通知所述第一进程所述RDMA数据传输消息的完成情况。
  6. 根据权利要求5所述的方法，其特征在于，所述第一报文和所述第二报文还包括第二 标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息；
    所述第一网络设备根据所述第二报文中的所述第一标识和所述反馈消息通知所述第一进程所述RDMA数据传输消息的完成情况包括：
    所述第一网络设备根据所述第一标识、所述反馈消息和所述第二标识通知所述第一进程所述RDMA数据传输消息的完成情况。
  7. 根据权利要求5或6所述的方法，其特征在于，所述共享发送队列具体用于保存来自所述多个进程的工作请求；
    所述第一网络设备从共享发送队列获取第一进程的RDMA数据传输消息包括：
    所述第一网络设备从所述共享发送队列获取来自所述第一进程的第一工作请求，所述第一工作请求描述所述RDMA数据传输消息；
    所述第一网络设备根据所述第一工作请求获取所述RDMA数据传输消息。
  8. 根据权利要求5至7中任一项所述的方法，其特征在于，所述第一网络设备根据所述第二报文中的所述第一标识和所述反馈消息通知所述第一进程所述RDMA数据传输消息的完成情况，包括：
    所述第一网络设备根据所述第一标识从多个完成队列中确定所述第一进程对应的第一完成队列；
    所述第一网络设备根据所述反馈消息在所述第一完成队列中写入工作完成元素，所述工作完成元素用于通知所述第一进程所述RDMA数据传输消息的完成情况。
  9. 一种远程直接内存访问RDMA数据传输方法，其特征在于，包括：
    第二网络设备接收来自第一网络设备的第一报文，所述第一网络设备设置于第一主机上，所述第二网络设备设置于第二主机上，所述第一报文包括第一进程的RDMA数据传输消息和所述第一进程对应的第一标识，所述RDMA数据传输消息为所述第一网络设备从共享发送队列获取的，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程；
    所述第二网络设备根据所述第一报文向所述第一网络设备发送第二报文，所述第二报文包括所述第一标识和反馈消息，所述第一标识和所述反馈消息指示所述第一网络设备通知所述第一进程所述RDMA数据传输消息的完成情况。
  10. 根据权利要求9所述的方法，其特征在于，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息，所述第一标识、所述反馈消息和所述第二标识指示所述第一网络设备通知所述第一进程所述RDMA数据传输消息的完成情况。
  11. 一种网络设备，其特征在于，包括：
    获取单元，用于从共享发送队列获取第一进程的RDMA数据传输消息，所述网络设备设置在第一主机上，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程；
    发送单元，用于向第二网络设备发送第一报文，所述第二网络设备设置在第二主机上，所述第一报文包括所述RDMA数据传输消息和所述第一进程对应的第一标识；
    接收单元，用于接收来自所述第二网络设备的第二报文，所述第二报文包括所述第一标识和反馈消息，所述反馈消息指示所述RDMA数据传输消息的完成情况；
    完成单元，用于根据所述第二报文中的所述第一标识和所述反馈消息通知所述第一进程所述RDMA数据传输消息的完成情况。
  12. 根据权利要求11所述的网络设备，其特征在于，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息；
    所述完成单元具体用于根据所述第一标识、所述反馈消息和所述第二标识通知所述第一进程所述RDMA数据传输消息的完成情况。
  13. 根据权利要求11或12所述的网络设备，其特征在于，所述共享发送队列具体用于保存来自所述多个进程的工作请求；
    所述获取单元具体用于：
    从所述共享发送队列获取来自所述第一进程的第一工作请求，所述第一工作请求描述所述RDMA数据传输消息；
    根据所述第一工作请求获取所述RDMA数据传输消息。
  14. 根据权利要求11至13中任一项所述的网络设备，其特征在于，所述完成单元具体用于：
    根据所述第一标识从多个完成队列中确定所述第一进程对应的第一完成队列；
    根据所述反馈消息在所述第一完成队列中写入工作完成元素，所述工作完成元素用于通知所述第一进程所述RDMA数据传输消息的完成情况。
  15. 一种网络设备，其特征在于，包括：
    接收单元，用于接收来自第一网络设备的第一报文，所述第一网络设备设置于第一主机上，所述网络设备设置于第二主机上，所述第一报文包括第一进程的RDMA数据传输消息和所述第一进程对应的第一标识，所述RDMA数据传输消息为所述第一网络设备从共享发送队列获取的，所述第一进程是所述第一主机运行的使用所述共享发送队列的多个进程中的任意一个进程；
    发送单元，用于根据所述第一报文向所述第一网络设备发送第二报文，所述第二报文包括所述第一标识和反馈消息，所述第一标识和所述反馈消息指示所述第一网络设备通知所述第一进程所述RDMA数据传输消息的完成情况。
  16. 根据权利要求15所述的网络设备，其特征在于，所述第一报文和所述第二报文还包括第二标识，所述第二标识用于从所述第一进程的多个RDMA数据传输消息中确定所述RDMA数据传输消息，所述第一标识、所述反馈消息和所述第二标识指示所述第一网络设备通知所述第一进程所述RDMA数据传输消息的完成情况。
  17. 一种网络设备，其特征在于，包括：处理器以及存储器，
    所述处理器用于执行所述存储器中存储的指令，使得所述网络设备执行权利要求5至8中任一项所述的方法。
  18. 一种网络设备，其特征在于，包括：处理器以及存储器，
    所述处理器用于执行所述存储器中存储的指令，使得所述网络设备执行权利要求9或10 所述的方法。
PCT/CN2022/099788 2021-09-07 2022-06-20 Rdma数据传输系统、rdma数据传输方法及网络设备 Ceased WO2023035722A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP22866197.1A EP4387201A4 (en) 2021-09-07 2022-06-20 Rdma data transmission system, rdma data transmission method, and network device
US18/598,357 US20240275740A1 (en) 2021-09-07 2024-03-07 RDMA Data Transmission System, RDMA Data Transmission Method, and Network Device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111045146.4A CN115776434A (zh) 2021-09-07 2021-09-07 Rdma数据传输系统、rdma数据传输方法及网络设备
CN202111045146.4 2021-09-07

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/598,357 Continuation US20240275740A1 (en) 2021-09-07 2024-03-07 RDMA Data Transmission System, RDMA Data Transmission Method, and Network Device

Publications (1)

Publication Number Publication Date
WO2023035722A1 true WO2023035722A1 (zh) 2023-03-16

Family

ID=85387701

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/099788 Ceased WO2023035722A1 (zh) 2021-09-07 2022-06-20 Rdma数据传输系统、rdma数据传输方法及网络设备

Country Status (4)

Country Link
US (1) US20240275740A1 (zh)
EP (1) EP4387201A4 (zh)
CN (1) CN115776434A (zh)
WO (1) WO2023035722A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120710949A (zh) * 2023-08-28 2025-09-26 华为技术有限公司 通信方法和通信装置

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118869393A (zh) * 2023-04-28 2024-10-29 华为技术有限公司 跨网络访问固态盘ssd的方法和装置
CN118869775A (zh) * 2023-04-28 2024-10-29 华为技术有限公司 报文传输的方法和装置
CN116361037B (zh) * 2023-05-18 2023-08-18 之江实验室 一种分布式通信系统及方法
CN120224482A (zh) * 2023-12-25 2025-06-27 华为技术有限公司 通信方法、通信装置及通信系统
CN120390042A (zh) * 2024-01-26 2025-07-29 华为技术有限公司 数据传输方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103999068A (zh) * 2011-12-23 2014-08-20 英特尔公司 共享的发送队列
US20160072906A1 (en) * 2014-09-04 2016-03-10 Mellanox Technologies Ltd. Hybrid tag matching
CN108009032A (zh) * 2017-12-14 2018-05-08 郑州云海信息技术有限公司 一种Infiniband网络通信方法、节点、设备及存储介质
CN111782417A (zh) * 2020-06-22 2020-10-16 北京计算机技术及应用研究所 一种基于消息的多进程共享串口资源的实现方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8213315B2 (en) * 2009-11-19 2012-07-03 Mellanox Technologies Ltd. Dynamically-connected transport service

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103999068A (zh) * 2011-12-23 2014-08-20 英特尔公司 共享的发送队列
US20160072906A1 (en) * 2014-09-04 2016-03-10 Mellanox Technologies Ltd. Hybrid tag matching
CN108009032A (zh) * 2017-12-14 2018-05-08 郑州云海信息技术有限公司 一种Infiniband网络通信方法、节点、设备及存储介质
CN111782417A (zh) * 2020-06-22 2020-10-16 北京计算机技术及应用研究所 一种基于消息的多进程共享串口资源的实现方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120710949A (zh) * 2023-08-28 2025-09-26 华为技术有限公司 通信方法和通信装置

Also Published As

Publication number Publication date
US20240275740A1 (en) 2024-08-15
CN115776434A (zh) 2023-03-10
EP4387201A1 (en) 2024-06-19
EP4387201A4 (en) 2024-12-04

Similar Documents

Publication Publication Date Title
CN115776434A (zh) Rdma数据传输系统、rdma数据传输方法及网络设备
CN105579987B (zh) 通用pci express端口
US10521283B2 (en) In-node aggregation and disaggregation of MPI alltoall and alltoallv collectives
CN116569154B (zh) 数据传输方法和相关装置
CN106487896B (zh) 用于处理远程直接内存访问请求的方法和装置
WO2018137217A1 (zh) 一种数据处理的系统、方法及对应装置
WO2017000593A1 (zh) 报文处理方法及装置
JP6189898B2 (ja) アクティブメッセージによるｒｍａａｐｉのサポート
CN113973091A (zh) 一种报文处理方法、网络设备以及相关设备
CN111026324A (zh) 转发表项的更新方法及装置
CN111064680B (zh) 一种通信装置及数据处理方法
CN116266800A (zh) 组播传输方法、装置和系统
EP4686169A1 (en) Packet transmission method and device
CN116760504B (zh) 会话同步方法、装置、服务节点、终端及可读存储介质
US12093571B1 (en) Accelerating request/response protocols
WO2025002161A1 (zh) 一种数据处理方法、交换节点及相关系统
CN120658696B (zh) 一种数据包发送方法、数据处理单元、主机及网卡
CN118714183B (zh) 报文传输的方法和装置
EP4607897A1 (en) Data processing method and apparatus
WO2026056710A1 (zh) 一种数据处理方法及相关设备
CN118426673A (zh) 数据存取方法及相关装置
CN112764666B (zh) 用于存储管理的方法、设备和计算机程序产品
WO2023202241A1 (zh) 一种通信方法及相关产品
WO2024140068A2 (zh) 报文传输方法及系统、网络装置
CN120658696A (zh) 一种数据包发送方法、数据处理单元、主机及网卡

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22866197

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022866197

Country of ref document: EP

Effective date: 20240311

NENP Non-entry into the national phase

Ref country code: DE