WO2021018267A1 - 虚拟机热迁移方法和通信设备 - Google Patents

虚拟机热迁移方法和通信设备 Download PDF

Info

Publication number
WO2021018267A1
WO2021018267A1 PCT/CN2020/106044 CN2020106044W WO2021018267A1 WO 2021018267 A1 WO2021018267 A1 WO 2021018267A1 CN 2020106044 W CN2020106044 W CN 2020106044W WO 2021018267 A1 WO2021018267 A1 WO 2021018267A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual machine
network card
hot migration
virtual
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/106044
Other languages
English (en)
French (fr)
Inventor
曹水
刘耀崇
邱军
欧阳长春
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP20846078.2A priority Critical patent/EP3992805B1/en
Publication of WO2021018267A1 publication Critical patent/WO2021018267A1/zh
Priority to US17/204,108 priority patent/US12405816B2/en
Anticipated expiration legal-status Critical
Priority to US19/285,364 priority patent/US20260023590A1/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
    • G06F9/4856—Task life-cycle, e.g. stopping, restarting, resuming execution resumption being on a different machine, e.g. task migration, virtual machine migration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
    • G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45575—Starting, stopping, suspending or resuming virtual machine instances
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45579—I/O management, e.g. providing access to device drivers or storage
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45595—Network integration; Enabling network access in virtual machine instances

Definitions

  • This application relates to the field of communication technology, and in particular to a virtual machine hot migration method and communication equipment.
  • the hot migration of virtual machines is also called live migration or online migration, which refers to the migration of a virtual machine from one physical server to another physical server while ensuring the normal operation of application services, in order to ensure the client service during the migration process.
  • live migration refers to the migration of a virtual machine from one physical server to another physical server while ensuring the normal operation of application services, in order to ensure the client service during the migration process.
  • the availability of the migration process can only have a very short downtime. Relying on the migration of virtual machines between different hosts, the cluster can effectively allocate and schedule resources.
  • Single-root I/O virtualization is a kind of input/output (I/O) virtualization technology, which can be used in a physical and fast peripheral component interconnection standard (peripheral component interconnection express, PCIE) devices virtualize multiple virtual PCIE devices, which are called virtual functions (VF).
  • PCIE peripheral component interconnection express
  • VF virtual functions
  • the SR-IOV network card can virtualize multiple virtual network cards.
  • the virtual machine performs services through the virtual network card. Each virtual machine can bypass the virtual machine manager and directly perform network I/O data with one or more virtual network cards corresponding to it. Interactively, the state information of the virtual network card is stored in the storage medium of the network card, and the virtual machine manager cannot obtain the state information in the storage medium of the SR-IOV network card.
  • the status information of the virtual machine corresponding to the virtual network card needs to be migrated from the source server to the destination server. Since the status information of the virtual network card is stored in the storage medium of the physical network card, the virtual machine manager does not perceive the SR-IOV network card. The storage medium cannot obtain the state information, so the state information of the virtual network card corresponding to the virtual machine cannot be migrated.
  • the embodiment of the present application provides a virtual machine hot migration method, which can realize the SR-IOV network card state hot migration, so that the physical machine using the SR-IOV network card can also perform the virtual machine hot migration process.
  • the first aspect of the embodiments of the present application provides a virtual machine hot migration method, which is applied to a source server running a first virtual machine.
  • the method is used to migrate the first virtual machine to a destination server.
  • the source server includes a single root input. / Output a virtualized SR-IOV network card.
  • the source server runs a first virtual machine, a first virtual network card used by the first virtual machine, a virtual machine manager, and a hot migration module.
  • the first virtual network card is based on the SR -The IOV network card is virtualized, the method includes: the virtual machine manager issues a heat migration command to the hot migration module; the hot migration module obtains the information to be migrated by the first virtual machine according to the hot migration command, and the first virtual machine The information to be migrated by the machine includes the state information of the first virtual network card and the memory data of the first virtual machine, and is reported to the virtual machine manager; the virtual machine manager sends the state information and the first virtual machine to the destination server The memory data of the machine.
  • the source server obtains the hot migration request, the source server establishes a connection with the destination host, and sends a hot migration message to the destination server according to the hot migration request, the hot migration message Used to instruct the first virtual machine running on the host device to migrate to the destination server; the hot migration module receives the hot migration command issued by the virtual machine manager according to the hot migration request, which can be obtained through the newly added hot migration module The state information of the first virtual network card corresponding to the first virtual machine; then, the virtual machine manager sends the state information to the destination server. In this way, the state information can be migrated to the destination server, thereby realizing the migration of the first virtual machine.
  • the hot migration module acquiring the information that the first virtual machine needs to migrate according to the hot migration command includes: the hot migration module copies the state information to the The first memory space allocated by the virtual machine manager is not included in the virtual memory allocated for the first virtual machine.
  • the hot migration module can copy the acquired state information to the first memory space allocated by the virtual machine manager, and the first memory space is not included in the first memory space allocated to the first virtual machine. In the virtual memory, the first virtual machine does not perceive the migration process.
  • the hot migration module copying the state information to the first memory space allocated by the virtual machine manager according to the hot migration command includes: the hot migration module downloads from the SR -Copy the status information from the base address register, microprocessor and microcode of the IOV network card to the first memory space.
  • the hot migration module obtains the state information including copying from the base address register, the microprocessor, and the microcode of the physical network card, which provides a feasible way to obtain the state information. Way to achieve.
  • the method further includes: the hot migration module locks the first virtual network card The first task queue, so that the first task queue cannot receive new tasks, the first task queue is used to place the tasks issued by the first virtual machine; the first virtual function driver of the first virtual machine clears the task All existing tasks in the first task queue.
  • the hot migration module can lock the first task queue, and the first virtual function driver can execute the tasks in the first task queue until the task is emptied. queue. In this way, the user will not be able to modify the status information of the virtual network card corresponding to the first virtual machine, and business exceptions can also be avoided.
  • the method further includes: the hot migration module performs an iterative copy of the memory data; In the case that the memory data can be copied at one time, after the first virtual function driver clears all the existing tasks in the first task queue, the virtual machine manager suspends the first virtual machine; the hot migration module executes the last time Memory data copy.
  • the virtual machine hot migration method limits the hot migration module to lock and empty the channel for executing user commands when iterative copy of memory data and the remaining memory can be copied at one time. Because the channel for executing user commands is locked, No longer responding to new user commands, the remaining memory can be locked when one copy is completed, which can reduce the duration of the phase of not responding to user commands.
  • the second aspect of the embodiments of the present application provides a virtual machine hot migration method, which is applied to a destination server for virtual machine migration.
  • the destination server includes an SR-IOV network card, and the destination server runs a virtual machine manager and a hot migration module.
  • the method includes: the virtual machine manager obtains a hot migration command, the hot migration command is used to indicate that the first virtual machine of the source server will be migrated to the second virtual machine of the destination server; the virtual machine manager receives the source server The sent state information of the first virtual network card corresponding to the first virtual machine; the hot migration module restores the state information to the SR-IOV network card according to the instruction of the virtual machine manager.
  • the state information of the first network card corresponding to the first virtual machine needs to also be migrated to the physical network card of the destination server.
  • the virtual machine manager receives the state information of the virtual network card corresponding to the first virtual machine sent by the source server. After receiving the state information, the virtual machine manager of the destination server needs to restore the state information to the physical network card through the hot migration module in. Makes the successful migration of the first virtual machine.
  • the hot migration module restoring the state information to the SR-IOV network card according to the instruction of the virtual machine manager includes: the hot migration module restoring the state information to The base address register, microcode and microprocessor of the SR-IOV network card.
  • the virtual machine hot migration method provided by the embodiment of the present application can write state information into the base address register, microcode, and microprocessor of the physical network card, which enhances the feasibility of the solution.
  • the hot migration module restores the state information to the base address register, microcode, and microprocessor of the SR-IOV network card, respectively, including: the hot migration module restores the status
  • the first information in the information is copied to the microprocessor.
  • the first information is the information obtained from the read-only base address register of the source server.
  • the microprocessor is used to restore the first information to the SR-IOV network card.
  • the hot migration module cannot directly write to the read-only base address register of the destination server, but can be restored by copying it to the microprocessor. Higher, it can solve the problem that the read-only base address register cannot be written directly.
  • the method before the virtual machine manager receives the state information of the first virtual network card corresponding to the first virtual machine from the source server, the method further includes: the virtual machine manager The identification information of the second virtual network card corresponding to the second virtual machine is allocated through the hot migration module.
  • the virtual machine hot migration method provided by the embodiment of the present application performs global resource allocation before receiving the state information of the virtual network card corresponding to the first virtual machine sent by the source server, which can avoid conflicts caused by global resource occupation.
  • the virtual machine manager receiving the state information of the first virtual network card corresponding to the first virtual machine from the source server includes: the virtual machine manager receiving the state information from the source server
  • the second information is the lock status information of the second task queue of the second virtual network card corresponding to the second virtual machine, the second virtual network card is virtualized based on the SR-IOV network card, the second The task queue is used to place tasks issued for the second virtual machine; the method further includes: the hot migration module unlocks the second task queue.
  • the virtual machine hot migration method provided by the embodiment of the present application can unlock the locked channel for executing user commands, so that the commands issued by the user can continue to be executed after the network card status is restored.
  • the method further includes: the virtual machine manager receives the memory data corresponding to the first virtual machine sent by the source server; the virtual machine manager restores the memory data corresponding to the first virtual machine.
  • the third aspect of the embodiments of the present application provides a virtual machine hot migration method, which is applied to a source server running a first virtual machine.
  • the method is used to migrate the first virtual machine to a destination server.
  • the source server includes a single root input. / Output virtualized SR-IOV network card, the source server runs the first virtual machine, the first virtual network card used by the first virtual machine, the virtual machine manager and the physical function driver, and the first virtual network card is based on the SR -IOV network card is virtualized, the method includes: the virtual machine manager sends a heat migration command to the virtual function driver; the virtual function driver obtains the information to be migrated by the first virtual machine according to the hot migration command, and the first virtual machine
  • the information that the machine needs to migrate includes the state information of the first virtual network card corresponding to the first virtual machine and the memory data of the first virtual machine, and is reported to the virtual machine manager; the virtual machine manager sends the state to the destination server Information and memory data of the first virtual machine.
  • the virtual machine manager does not perceive the storage medium of the SR-IOV network card and cannot obtain the status information of the virtual network card.
  • the physical function driver can interact with the SR-IOV network card.
  • This application is implemented In the example, the virtual machine manager is connected to the physical function driver through VFIO, and then the state information of the virtual network card can be obtained through the physical function driver, so as to realize the migration of the first virtual machine.
  • the virtual function driver acquiring the information that the first virtual machine needs to migrate according to the hot migration command includes: the virtual function driver copying the state information to the first virtual machine according to the hot migration command
  • the first memory space allocated by the virtual machine manager is not included in the virtual memory allocated for the first virtual machine.
  • the virtual function driver copying the state information to the first memory space allocated by the virtual machine manager according to the hot migration command includes: the virtual function driver receives from the SR -Copy the status information from the base address register, microprocessor and microcode of the IOV network card to the first memory space.
  • the method before the virtual function driver copies the state information to the first memory space allocated by the virtual machine manager according to the hot migration command, the method further includes: the virtual The function driver locks the first task queue of the first virtual network card so that the first task queue cannot receive new tasks, and the first task queue is used to place tasks issued by the first virtual machine; the first virtual machine The first virtual function running on the computer drives all existing tasks in the first task queue to be cleared.
  • the method further includes: the virtual function driver performs an iterative copy of the memory data; and the remaining memory data can be copied once When completed, after the first virtual function driver empties all existing tasks in the first task queue, the virtual machine manager suspends the first virtual machine; the virtual function driver executes the last memory data copy.
  • the virtual function driver is connected to the virtual machine manager through the user mode driver framework VFIO.
  • the virtual function driver includes a hot migration module, and the hot migration module is configured to obtain the status information of the first virtual network card according to an instruction of the virtual machine manager.
  • a new hot migration module can be used to assist in realizing the virtual machine hot migration in the physical function drive.
  • the virtual machine manager does not perceive the storage medium of the SR-IOV network card, and cannot obtain the status information of the virtual network card.
  • the physical function driver can interact with the SR-IOV network card.
  • the hot migration is added to the physical function driver.
  • the virtual machine manager is connected to the hot migration module through VFIO, and then the state information of the virtual network card can be obtained through the hot migration module, so as to realize the migration of the first virtual machine.
  • the fourth aspect of the embodiments of the present application provides a virtual machine hot migration method, which is characterized in that it is applied to a destination server for virtual machine migration, the destination server includes an SR-IOV network card, and the destination server runs a virtual machine manager and Driven by physical functions, the method includes: the virtual machine manager obtains a hot migration command, the hot migration command is used to indicate that the first virtual machine of the source server will migrate to the second virtual machine of the destination server; the virtual machine manager Receiving the state information of the first virtual network card corresponding to the first virtual machine sent by the source server; the virtual function driver restores the state information to the SR-IOV network card according to the instruction of the virtual machine manager.
  • the virtual function driver restoring the state information to the SR-IOV network card according to the instruction of the virtual machine manager includes: the virtual function driver restoring the state information to The base address register, microcode and microprocessor of the SR-IOV network card.
  • the virtual function driver restores the state information to the base address register, microcode, and microprocessor of the SR-IOV network card, respectively, including: the virtual function driver restores the state The first information in the information is copied to the microprocessor, and the first information is the information obtained from the read-only base address register of the source server; the microprocessor restores the first information to the SR-IOV network card Read-only base address register.
  • the method before the virtual machine manager receives the state information of the first virtual network card corresponding to the first virtual machine from the source server, the method further includes: the virtual machine manager The identification information of the second virtual network card corresponding to the second virtual machine is allocated through the virtual function driver.
  • the virtual machine manager receiving the status information of the first virtual network card corresponding to the first virtual machine from the source server includes: the virtual machine manager receives second information sent by the source server, where the second information is the second The lock status information of the second task queue of the second virtual network card corresponding to the virtual machine, the second virtual network card is virtualized based on the SR-IOV network card, and the second task queue is used to place the second virtual machine to issue The task; the method further includes: the virtual function driver unlocks the second task queue.
  • the virtual function driver includes a hot migration module, and the hot migration module is configured to restore the state information to the SR-IOV network card according to an instruction of the virtual machine manager.
  • the physical function driver can be used to assist in realizing the virtual machine hot migration through a newly-added hot migration module.
  • the fifth aspect of the embodiments of the present application provides a communication device, the communication device is the source end of the virtual machine hot migration, the communication device runs a first virtual machine, a first virtual network card used by the first virtual machine, and a virtual The first virtual network card is virtualized based on the SR-IOV network card, and the method includes: the virtual machine manager is configured to issue a heat migration command to the hot migration module; the hot migration module, It is used to obtain the information to be migrated by the first virtual machine according to the hot migration command.
  • the information to be migrated by the first virtual machine includes the state information of the first virtual network card and the memory data of the first virtual machine, and reports the virtual Machine manager; the virtual machine manager is also used to send the status information and the memory data of the first virtual machine to the destination communication device.
  • the hot migration module is specifically configured to: according to the hot migration command, copy the state information to the first memory space allocated by the virtual machine manager, and the first memory The space is not included in the virtual memory allocated to the first virtual machine.
  • the hot migration module is specifically configured to: copy the state information from the base address register, the microprocessor, and the microcode of the SR-IOV network card to the first memory space.
  • the hot migration module is also used to lock the first task queue of the first virtual network card so that the first task queue cannot receive new tasks, and the first task The queue is used to place tasks issued for the first virtual machine; the communication device further includes: a first virtual function driver of the first virtual machine, which is used to clear all existing tasks in the first task queue.
  • the hot migration module is further used to: iteratively copy the memory data; in the case that the remaining memory data can be copied at one time, the first virtual function driver is cleared After all the existing tasks in the first task queue, the virtual machine manager is also used to suspend the first virtual machine; the hot migration module is also used to execute the last memory data copy.
  • the sixth aspect of the embodiments of the present application provides a communication device.
  • the communication device is a destination communication device for virtual machine hot migration.
  • the communication device includes an SR-IOV network card.
  • the communication device runs a virtual machine manager and hot migration.
  • the communication device includes: the virtual machine manager for obtaining a hot migration command, the hot migration command is used to instruct the first virtual machine of the source communication device to migrate to the second virtual machine of the destination communication device
  • the virtual machine manager is also used to receive the status information of the first virtual network card corresponding to the first virtual machine sent by the source-end communication device; the hot migration module is also used to according to the instructions of the virtual machine manager, Restore the state information to the SR-IOV network card.
  • the hot migration module is specifically configured to restore the state information to the base address register, microcode, and microprocessor of the SR-IOV network card, respectively.
  • the hot migration module is specifically configured to: copy the first information in the state information to the microprocessor, and the first information is read-only from the source communication device The information obtained in the base address register is used by the microprocessor to restore the first information to the read-only base address register of the SR-IOV network card.
  • the virtual machine manager is further configured to: allocate the identification information of the second virtual network card corresponding to the second virtual machine through the hot migration module.
  • the virtual machine manager is specifically configured to: receive second information sent by the source communication device, where the second information is the information of the second virtual network card corresponding to the second virtual machine
  • the lock status information of the second task queue, the second virtual network card is virtualized based on the SR-IOV network card, and the second task queue is used to place tasks issued for the second virtual machine; the hot migration module also Used to unlock the second task queue.
  • the seventh aspect of the embodiments of the present application provides a communication device, which is a source-end communication device for virtual machine hot migration, including: a processor, a memory, and an SR-IOV network card; the processor is used to run a first virtual machine, and virtual machine management And a hot migration module, the first virtual machine runs a first virtual function driver, and the processor executes the implementation manners provided in the foregoing first aspect through the virtual machine manager, the hot migration module, and the first virtual function driver Methods.
  • a communication device which is a source-end communication device for virtual machine hot migration, including: a processor, a memory, and an SR-IOV network card; the processor is used to run a first virtual machine, and virtual machine management And a hot migration module, the first virtual machine runs a first virtual function driver, and the processor executes the implementation manners provided in the foregoing first aspect through the virtual machine manager, the hot migration module, and the first virtual function driver Methods.
  • the processor, the memory and the SR-IOV network card are electrically connected.
  • the communication device further includes a bus, and the processor, the memory and the SR-IOV network card pass The bus is connected.
  • the eighth aspect of the embodiments of the present application provides a communication device, which is a destination communication device for virtual machine hot migration, including: a processor, a memory, and an SR-IOV network card; the processor is used to run a virtual machine manager and a hot migration module , And execute the methods of each implementation manner provided in the foregoing second aspect through the virtual machine manager and the hot migration module.
  • a communication device which is a destination communication device for virtual machine hot migration, including: a processor, a memory, and an SR-IOV network card; the processor is used to run a virtual machine manager and a hot migration module , And execute the methods of each implementation manner provided in the foregoing second aspect through the virtual machine manager and the hot migration module.
  • the processor, the memory, and the SR-IOV network card are electrically connected.
  • the communication device further includes a bus, and the processor, the memory, and the SR-IOV network card pass through The bus is connected.
  • the ninth aspect of the embodiments of the present application provides a communication device, which is a source-end communication device for virtual machine hot migration, including: a processor, a memory, and an SR-IOV network card; the processor is used to run a first virtual machine, and virtual machine management The first virtual machine runs the first virtual function driver, and the processor executes the implementation of the aforementioned third aspect through the virtual machine manager, the physical function driver, and the first virtual function driver. method.
  • the processor, the memory, and the SR-IOV network card are electrically connected.
  • the communication device further includes a bus, and the processor, the memory, and the SR-IOV network card pass through The bus is connected.
  • the tenth aspect of the embodiments of the present application provides a communication device, which is a destination communication device for virtual machine hot migration, including: a processor, a memory, and an SR-IOV network card; the processor is used to run a virtual machine manager and physical functions The virtual machine manager and the physical function are used to drive the implementation of the methods provided in the foregoing fourth aspect.
  • the processor, the memory, and the SR-IOV network card are electrically connected.
  • the communication device further includes a bus, and the processor, the memory, and the SR-IOV network card pass through The bus is connected.
  • the eleventh aspect of the embodiments of the present application provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the steps of the implementation manners provided in the foregoing first to fourth aspects.
  • a twelfth aspect of the embodiments of the present application provides a computer-readable storage medium, including instructions, which when run on a computer, cause the computer to execute the steps of the implementation manners provided in the first to fourth aspects.
  • a thirteenth aspect of the embodiments of the present application provides a virtual machine hot migration system, which includes the source-end communication device provided in the foregoing fifth aspect and the destination-end communication device provided in the foregoing sixth aspect.
  • the status information of the first virtual network card in the SR-IOV network card can be obtained through the hot migration module, and the status information can be migrated to the destination server.
  • the virtual machine hot migration method provided by the embodiment of the present application can make the state information migrate to the second virtual machine side, thereby ensuring the successful migration of the first virtual machine.
  • Figure 1 is a schematic diagram of a migration architecture for virtual machine hot migration
  • FIG. 2 is a schematic diagram of a system architecture of virtual machine hot migration according to an embodiment of the application
  • FIG. 3 is a schematic diagram of an embodiment of a virtual machine hot migration method in an embodiment of the application
  • FIG. 4 is a schematic diagram of another embodiment of a method for hot migration of a virtual machine in an embodiment of the application
  • FIG. 5 is an interactive schematic diagram of an embodiment of a virtual machine hot migration method in an embodiment of this application.
  • FIG. 6 is a schematic diagram of an embodiment of locking and emptying the task queue in an embodiment of the application
  • FIG. 7 is a schematic diagram of an embodiment of a source communication device in an embodiment of this application.
  • FIG. 8 is a schematic diagram of an embodiment of a destination communication device in an embodiment of this application.
  • FIG. 9 is a schematic diagram of another embodiment of a source communication device in an embodiment of this application.
  • FIG. 10 is a schematic diagram of another embodiment of a destination communication device in an embodiment of this application.
  • the embodiment of the present application provides a virtual machine hot migration method, which can realize the migration of the state information of the network card, so that the physical machine using the SR-IOV network card can also perform the virtual machine hot migration process.
  • Figure 1 is a schematic diagram of a migration architecture for virtual machine hot migration.
  • the migration architecture includes a source server 100 before the virtual machine migration and a destination server 110 after the virtual machine migration.
  • the running state of the virtual machine 1 running in the source server 100 can be migrated to any virtual machine in the destination server 110, for example, the virtual machine 1 is migrated to the virtual machine 2.
  • the source server and the destination server are both host machines deployed with a virtual machine manager, which can provide physical resources for the virtual machine.
  • a virtual machine manager which can provide physical resources for the virtual machine.
  • multiple virtual machines can be created on a host machine, and the number of virtual machines created depends on the hardware configuration of the host machine and the size of the virtual machine to be created.
  • the number N of virtual machines running on the source server 100 and the number M of virtual machines running on the destination server are not limited here.
  • the host may be a server that provides various cloud services.
  • FIG. 2 is a schematic diagram of a virtual machine hot migration system according to an embodiment of the application.
  • the system includes a source server before virtual machine migration and a destination server after virtual machine hot migration.
  • the source server includes a host and an SR-IOV network card. Specifically, at least one virtual machine runs on the host (HOST). FIG. 2 only shows a first virtual machine VM1 before the hot migration.
  • the host also runs a virtual machine manager of the active server.
  • the virtual machine manager can manage one or more virtual machine VMs, and each virtual machine VM can run an operating system and various applications running on the operating system.
  • the virtual machine manager may be a management module running on a host device, such as a hypervisor or a virtual machine monitor (virtual machine monitor, VMM), or it may be implemented by a customized logic chip.
  • the virtual machine manager is responsible for the management and control of the virtual machine, including operations such as the creation, migration, power-on and power-off of the virtual machine.
  • the virtual machine manager may specifically be qemu (Quick EMUlator), VMWare, ESXi, or a kernel-based virtual machine (kernel-based virtual machine, KVM), etc., which is not specifically limited here.
  • the hot migration module is a newly added module in this embodiment and provides a hot migration interface for the virtual machine manager. It can assist in realizing hot migration according to the instructions of the virtual machine manager.
  • the hot migration module in this embodiment of the application can be used to obtain virtual machines. The status information of the corresponding virtual network card.
  • the physical function driver can be used with the SR-IOV network card. Interactive.
  • the virtual machine manager is connected to the hot migration module through the VFIO, and can obtain the state information of the virtual network card through the hot migration module, so as to realize the migration of the first virtual machine.
  • the SR-IOV network card includes at least one physical function (PF) and at least one virtual function VF.
  • the virtual function of the SR-IOV network card is the virtual network card.
  • Figure 1 only shows one physical function and one virtual function.
  • One virtual machine corresponds to at least one virtual network card, and the state information of the virtual network card can be stored in the base address register (BAR), management unit, and data processing unit of the SR-IOV network card.
  • the management unit includes a microprocessor
  • the data processing unit includes microcode.
  • the host device also runs the PF driver that manages the SR-IOV network card.
  • the PF driver can directly access all PF resources, and is responsible for configuring and managing all VFs.
  • Each virtual machine VM runs a VF driver to manage its corresponding VF .
  • the destination server includes a host (HOST) and an SR-IOV network card.
  • HOST host
  • SR-IOV network card A virtual machine manager and a hot migration module of the destination server run on the HOST.
  • the host device is connected to the SR-IOV network card.
  • the hot migration module of the destination server is a newly added module in this embodiment, which provides a hot migration interface for the virtual machine manager, and can assist in realizing hot migration according to the instructions of the virtual machine manager.
  • the hot migration module in this embodiment of the application can be used for Restore the acquired state information of the virtual network card corresponding to the virtual machine to the SR-IOV network card.
  • the hot migration module is added to the physical function driver, and the virtual machine manager is connected to the hot migration module through VFIO.
  • a server may serve as the source and destination of virtual machine migration. Therefore, the functions of the software modules of the source server and the destination server provided in this embodiment can be The integration is implemented in a server.
  • the host device also includes hardware related to running a virtual machine VM, such as a central processing unit (CPU) and at least one computer-readable storage medium memory, including a hard disk or a memory.
  • the host device may also include a communication unit, an input unit, a display unit, and so on.
  • each virtual machine can directly perform network I/O data interaction with its corresponding one or more virtual network cards, there is no need to go through the virtual machine manager, which not only reduces the I/O performance overhead in the virtual machine manager, but also ensures I/O throughput efficiency of each VM.
  • the virtual machine VM sends and receives data packets from the virtual network card, it can achieve complete independence of sending and receiving, and the virtual machine manager cannot perceive it during the process of direct memory access (DMA) writing.
  • DMA direct memory access
  • the state information of the virtual network card corresponding to the virtual machine is stored in the storage medium of the SR-IOV network card, and the virtual machine manager cannot perceive it either.
  • the key technical points in virtual machine hot migration include memory state migration and network card state migration.
  • the migration of virtual machine memory can be divided into three stages:
  • Iterative pre-copy phase After the virtual machine migration process is started, it is still running on the source server, and the virtual machine business is not interrupted. First, all the memory of the virtual machine is copied to the destination server, and then the virtual machine is copied iteratively to the destination server. Changes in memory data. After iterating until the memory that needs to be copied is less than a certain threshold, it enters the shutdown copy phase.
  • Downtime copy phase the virtual machine is down, the business is interrupted, and the residual memory data of the virtual machine is copied.
  • Recovery phase The destination virtual machine completes the recovery processing before running, starts the destination virtual machine, and the entire migration process is completed.
  • the network card state migration refers to the migration of the state information of the virtual network card corresponding to the virtual machine to be migrated from the source server to the destination server.
  • the status information of the virtual network card that needs to be migrated during the virtual machine migration process mainly includes the following categories:
  • Device information including status information and interrupt table, etc., is stored in the BAR0/1 and BAR2/3 spaces of the BAR register.
  • the network card business context information is mainly used to manage the functions of VF and PF, including: function table, MAC-VLAN table and RSS template table, etc., which can be obtained and saved by the microprocessor.
  • the virtual machine management of the source server cannot obtain the status information of the virtual network card corresponding to the first virtual machine in the storage medium of the SR-IOV network card.
  • the state information of the first virtual network card also needs to be migrated to the side of the second virtual machine, so that the second virtual network card corresponding to the second virtual machine resumes work.
  • the newly-added hot migration module in the embodiment of this application can obtain the status information of the first virtual network card in the network card, and the virtual machine manager can copy the status information to the destination server, and finally guarantee the first virtual network card. The successful migration of the machine.
  • FIG. 3 is a schematic diagram of an embodiment of a virtual machine hot migration method in an embodiment of the application.
  • a virtual machine manager issues a heat migration command to the hot migration module
  • the source server obtains the hot migration request, the source server establishes a connection with the destination host according to the hot migration request, and requests to send the hot migration request to the destination server, indicating
  • the first virtual machine in the source server is migrated to the destination server, and the destination server may perform preparatory work before the hot migration.
  • the destination server pre-reserves virtual machine resources for the first virtual machine.
  • the virtual machine manager can obtain the hot migration request.
  • the user sends the hot migration request through the source server, or the server determines the first virtual machine to perform the hot migration according to a preset rule, which is not limited here.
  • the virtual machine manager issues a hot migration command to the hot migration module. It should be noted that the virtual machine manager can send multiple hot migration commands to the hot migration module during the migration of the first virtual machine. The number is not limited. Optionally, the virtual machine manager sends different hot migration commands to the hot migration module at different stages of the hot migration through a user-mode driver framework (virtual function I/O, VFIO).
  • a user-mode driver framework virtual function I/O, VFIO
  • the hot migration module obtains the information that the first virtual machine needs to migrate according to the hot migration command.
  • the information to be migrated by the first virtual machine includes the state information of the first virtual network card and the memory data of the first virtual machine, and is reported to the virtual machine manager.
  • the state information of the first virtual network card is stored in the SR-IOV network card, specifically, stored in the BAR register, management unit and data processing unit of the SR-IOV network card.
  • the virtual machine manager obtains the state information of the first virtual network card stored in the BAR register, the management unit, and the data processing unit of the SR-IOV network card through the hot migration module.
  • the hot migration module obtains the state information of the first virtual network card from the BAR register, the microprocessor, and the microcode of the network card according to the storage location of the state information, and combines The state information is saved in the first memory space.
  • the first memory space does not belong to the memory space corresponding to the first virtual machine, and is directly allocated and controlled by the virtual machine manager.
  • the virtual machine manager sends a control command to the hot migration module through VFIO.
  • the virtual machine manager sends the status information and the memory data of the first virtual machine to the destination server.
  • the virtual machine manager sends the memory data of the first virtual machine acquired in step 302 to the destination server for recovery.
  • the virtual machine manager also sends the state information acquired in step 302 to the destination server, so that the destination server
  • the server may perform network card state recovery on the migrated second virtual machine according to the state information, thereby realizing hot migration.
  • the source server can obtain the status information of the first virtual network card in the network card through the newly added hot migration module, and migrate the status information to the second virtual machine.
  • the state information can be migrated to the second virtual machine side, and finally the successful migration of the first virtual machine is guaranteed.
  • FIG. 4 is a schematic diagram of another embodiment of a virtual machine hot migration method in an embodiment of the application.
  • the virtual machine manager receives status information of the virtual network card corresponding to the first virtual machine sent by the source server.
  • the state information of the first network card corresponding to the first virtual machine needs to also be migrated to the SR-IOV network card of the destination server.
  • the virtual machine manager receives the status information sent by the source server.
  • the virtual machine manager restores the state information to the SR-IOV network card through the hot migration module.
  • the virtual machine manager of the destination server After receiving the status information, the virtual machine manager of the destination server needs to restore the status information to the BAR register, microcode, and microprocessor of the SR-IOV network card.
  • the status information obtained from the microprocessor and microcode of the source server can be directly written into the microprocessor and microcode of the destination server;
  • the BAR register includes BAR read-only registers and BAR writable registers
  • the status information obtained from the BAR register of the source SR-IOV network card needs to be processed in two cases when restoring, which are described below:
  • the status information copied from the BAR writable register can be directly written into the BAR writable register of the destination server;
  • the status information copied from the BAR read-only register needs to be restored by the microprocessor and written into the BAR read-only register of the destination server SR-IOV network card.
  • the virtual machine manager of the destination server restores the state information, it can enable the network card, and then enable the second virtual machine to complete the hot migration of the first virtual machine.
  • the virtual machine manager sends a control command to the hot migration module through VFIO.
  • the destination server receives the status information of the first virtual network card sent by the source server, and can realize the restoration of the status information in the SR-IOV network card through the newly added hot migration module, so that the first virtual machine can be successfully migrated .
  • FIG. 5 is an interactive schematic diagram of an embodiment of the virtual machine hot migration method in the embodiment of the application.
  • Virtual machine hot migration technology is often used in scenarios such as maintaining server load balance or virtual machine disaster recovery and backup.
  • the migration may be performed according to the virtual machine hot migration solution provided in the embodiment of the present application.
  • the source server establishes a connection with the destination server
  • the source server obtains the hot migration request, the source server establishes a connection with the destination host according to the hot migration request, and requests to send the hot migration request to the destination server, indicating
  • the first virtual machine in the source server is migrated to the destination server, and the destination server may perform preparatory work before the hot migration.
  • the destination server pre-reserves virtual machine resources for the first virtual machine.
  • the source server receives the user's migration request, and the source server establishes a connection with the destination host according to the host IP address of the destination server.
  • the destination server performs global resource allocation
  • the destination server performs global resource allocation, that is, initializes VF resources to ensure the availability of VF resources.
  • the virtual machine manager queries and applies for available global resources in the operating system through the hot migration module, including the identification information of the second virtual network card corresponding to the second virtual machine.
  • the global resource is the virtual network card identification information saved in the operating system for the virtual network card corresponding to the virtual machine. For example, index number (func_id), RSS template table identifier or q_cfg, etc.
  • the index number of the first virtual network card corresponding to the first virtual machine is 1, and it is migrated to the second virtual machine of the destination server.
  • the index number 1 may be occupied by other virtual machines.
  • the index number assigned by the server to the second virtual network card corresponding to the second virtual machine may be 2.
  • VF resources are not allocated independently, such as the RSS template table.
  • the RSS used by each VF is globally allocated by the microprocessor.
  • the rss_id at the source end is 1, but 1 may have been allocated to other VFs at the destination end. Up.
  • the destination virtual machine manager allocates global resources through the pre_active interface.
  • the step in which the virtual machine manager allocates the identification information of the virtual network card is called the post_active process.
  • the hot migration module reallocates the global resources corresponding to the VF through the microprocessor, and then executes the state restoration process.
  • the hot migration module sets the chip status information and VM corresponding resource configuration through the microprocessor and microcode, and executes the virtual machine active operation after the process returns.
  • the virtual machine hot migration method provided by the embodiment of the present application realizes the configuration consistency of the migration source VF and the migration destination target VF, and ensures the normal operation of the virtual machine after the migration.
  • step 502 is executed after step 501 and before step 508, and the execution order of steps 503 to 507 is not limited.
  • the source server iteratively copies memory data
  • the virtual machine manager may execute the iterative copy of memory data through the hot migration module according to the hot migration request.
  • the first virtual machine may migrate memory data, which are not limited here.
  • the copy of memory data needs to be executed multiple times. After iterative copying until the memory that needs to be copied is less than the preset threshold, it will enter the shutdown copy stage, and usually the remaining memory can be copied in one time.
  • the source server locks and drains the first task queue
  • the first task queue is used to place tasks issued by the first virtual machine.
  • the tasks are issued by the user through the application layer.
  • the first task queue is called a command queue for executing user commands in the actual application process. cmdq).
  • it includes tasks related to the status information of the virtual network card corresponding to the first virtual machine
  • the task issued by the user is a hardware operation
  • a hardware operation may be non-atomic, that is, it involves a combination of multiple operations.
  • the virtual function driver may be operating the hardware during the hot migration of the VM.
  • completing a hardware operation is a combination of multiple registers or hardware interactions.
  • the hardware operation is non-atomic, for example: the VF driver sends a Mailbox to the PF driver When sending a message, first write the Header register, then write the data register, and finally write the request register to trigger the VF hardware to start reading the header and send the data to the destination PF, and the PF returns the processing result to the asynchronous event queue (AEQ) of the VF.
  • AEQ asynchronous event queue
  • the VFD starts to process the result in the AEQ through an interrupt.
  • the source server locking and emptying the first task queue may be referred to as a pre_deactivate process.
  • the virtual network card state information corresponding to the first virtual machine If it has been migrated to the target virtual machine, abnormalities such as command timeout may occur, causing service interruption.
  • the hot migration module locks the channel that executes the user's command, so that the channel cannot receive the new task issued by the user. After the channel for executing user commands is locked, the user will not be able to modify the status information of the virtual network card corresponding to the first virtual machine, but user services can still continue.
  • the first virtual function running on the first virtual machine drives emptying the channel for executing user commands, which means that the user commands that existed when the queue is locked are executed, the queue is empty, and the emptying complete represents the execution of the issued user command. carry out.
  • the hot migration module can obtain information about the emptying of the channel for executing the user command.
  • the state of the channel that executes the user command belongs to the state information of the first virtual network card to be migrated. After the source server locks the channel that executes the user command, the locked state will be migrated to the destination server, and the destination server restores the state information Later, the channel for executing user commands can be unlocked.
  • FIG. 6 is a schematic diagram of an embodiment of locking and emptying the task queue in the embodiment of the present application.
  • Hot migration module side :
  • the hot migration module sets the task queue to a locked state.
  • the hot migration module may set the task queue to the locked state before acquiring the status information of the first virtual network card. Optionally, before the hot migration module performs the last memory copy, set the task queue to the locked state.
  • the locked state of the task queue can be migrated to the destination server side, and the locked state can be released on the destination server side for continuing to execute user commands.
  • the hot migration module judges whether the task queue is emptied, if yes, execute step 613, and if not, execute step 612 repeatedly.
  • hot migration module determines that the task queue is emptied, perform other operations.
  • the hot migration module can perform other operations, such as reporting to the virtual machine manager, which is not limited here.
  • the virtual function driver obtains the task issued by the user.
  • the virtual function driver judges whether the task queue is in a locked state.
  • the virtual function driver adds the task to the task queue.
  • the hot migration module sets the task queue to a locked state, after that, the task queue can no longer receive new tasks. If the user issues a new task, the virtual function driver places the task in the waiting channel. After the virtual machine migration is completed, on the destination server side, after the locked state of the task queue is released, the tasks in the waiting channel can be added to the task queue to continue execution.
  • the virtual function drives the execution of the tasks in the task queue.
  • the virtual function driver empties the task queue.
  • busy_state parameters when the hot migration module locks the task queue, set busy_state to 1, no new tasks are added to the task queue, and busy_state is 0 when the lock is released. Continue to receive tasks.
  • own is 1, and when the task is cleared, own is 0, and func_ref is the number of commands to be executed.
  • the pre_deactivate process ensures that the VFD message channel suspends.
  • the return of pre_deactivate means that all sent messages have been processed, and the VM can pause.
  • the destination VM resumes pass the PFD clear busy_state.
  • the suspended channel resumes after detecting that the channel is not in the busy state, and continues to send out the backlog of VFD messages at the source end.
  • the migration source process set busy_state to 1 in the Pre_deactivate process, and then check Own until Own is 0 to exit; in the VFD Uninterrupt flow process, if the busy_state state is 1, then perform message processing until func_ref is 0 and clear own bit. Then wait for the busy_state state to become 0.
  • the virtual function drives the tasks in the task queue until the tasks in the task queue are emptied.
  • the hot migration module can obtain whether the tasks in the task queue are emptied.
  • step 504 can be executed at any time between step 501 and step 505. After the channel for executing user commands is locked, no new user commands will be responded to. Optionally, after step 503, when the iterative copying of the memory ends and the remaining memory can be copied at one time (log_sync time), step 504 is executed, which can reduce the duration of the phase of not responding to user commands.
  • the first virtual machine running on the source server is down;
  • the virtual machine manager invokes the virtual machine shutdown process to suspend the first virtual machine.
  • the first virtual network card is deactivated, the service is interrupted, and the first virtual machine is shut down.
  • the source server executes the last memory copy to the destination server
  • the virtual machine manager will perform the last memory copy and migrate the remaining uncopyed memory in step 503 to the destination server.
  • the source server obtains the status information of the first virtual network card.
  • the virtual machine manager obtains the state information of the first virtual network card, and copies the state information to the virtual machine manager of the destination server.
  • the virtual machine manager obtains the state information of the first virtual network card corresponding to the first virtual machine through the hot migration module.
  • the hot migration module works in coordination with the management unit and the data processing unit.
  • the first memory space allocated by the virtual machine manager is used by the hot migration module from the BAR register, microprocessor, and microcode of the network card according to the storage location of the state information. Acquire the state information of the first virtual network card, and save the state information in the first memory space. It should be noted that the first memory space does not belong to the memory space corresponding to the first virtual machine, and is directly allocated and controlled by the virtual machine manager.
  • the source server sends the status information of the virtual network card to the destination server for recovery;
  • the virtual machine manager of the source server copies the state information stored in the first memory space to the virtual machine manager of the destination server.
  • the virtual machine manager of the destination server After receiving the status information, the virtual machine manager of the destination server needs to restore the status information to the BAR register, microcode, and microprocessor of the SR-IOV network card.
  • the status information obtained from the microprocessor and microcode of the source server can be directly written into the microprocessor and microcode of the destination server;
  • the BAR register includes BAR read-only registers and BAR writable registers
  • the status information obtained from the BAR register of the source SR-IOV network card needs to be processed in two cases when restoring, which are described below:
  • the status information copied from the BAR writable register can be directly written into the BAR writable register of the destination server;
  • the status information copied from the BAR read-only register needs to be restored by the microprocessor and written into the BAR read-only register of the destination server SR-IOV network card.
  • Exemplary, such as the message sending notification (Doorbell) and configuration control notification (Mailbox Req) registers cannot be restored by the source reading the destination writing, so the chip status residing in the BAR read-only register cannot be directly restored Use QEMU to copy MMIO space to migrate.
  • the virtual machine manager receives second information sent by the source server, where the second information is lock status information of the task queue of the second virtual network card corresponding to the second virtual machine, and the second virtual network card is also based on Virtually generated by the SR-IOV network card, the second task queue is used to place tasks issued for the second virtual machine; the hot migration module unlocks the second task queue.
  • the destination server may unlock the task queue before enabling the second virtual machine, for example, by setting busy_state to 0, unlock the locked state of the task queue.
  • the destination server enables the second virtual machine
  • the destination virtual machine manager After the state information of the first virtual network card is migrated to the destination server, the destination virtual machine manager opens the locked channel for executing user commands, so that user commands can be executed normally.
  • the virtual machine manager of the destination server can enable the network card and enable the second virtual machine.
  • the virtual machine manager enables the network card through the activate interface.
  • the destination server sends a broadcast message to resume communication
  • the virtual machine manager of the destination server sends a broadcast message, so that the communication can be quickly restored.
  • the virtual machine manager sends a broadcast message through the post_resume interface.
  • step 510 is an optional step, which may or may not be performed, and is not limited here.
  • the source server can obtain the status information of the first virtual network card in the network card through the newly added hot migration module, and copy the status information to the memory allocated by the virtual machine manager, and then send it to the second virtual machine.
  • the destination server receives the status information of the first virtual network card sent by the source server, and can realize the restoration of the status information in the SR-IOV network card through the newly added hot migration module.
  • the virtual machine migration system can realize the status of the first virtual network card Information migration enables the successful migration of the first virtual machine.
  • FIG. 7 is a schematic diagram of an embodiment of the source-end communication device in the embodiment of the present application.
  • the communication device is a source server for virtual machine hot migration, and the source server runs a first virtual machine, a first virtual network card used by the first virtual machine,
  • the virtual machine manager 701 and the hot migration module 702 the first virtual network card is virtualized based on the SR-IOV network card, including:
  • the virtual machine manager 701 is configured to issue a hot migration command to the hot migration module 702;
  • the hot migration module 702 is configured to obtain information to be migrated by the first virtual machine according to the hot migration command.
  • the information to be migrated by the first virtual machine includes the state information of the first virtual network card and the memory of the first virtual machine Data and report to the virtual machine manager 701;
  • the virtual machine manager 701 is also used to send the state information and the memory data of the first virtual machine to the destination server.
  • the hot migration module 702 is specifically configured to: according to the hot migration command, copy the state information to the first memory space allocated by the virtual machine manager 701, and the first memory space is not included in the allocation to the first virtual machine. Within the virtual memory.
  • the hot migration module 702 is specifically used to: copy the state information from the base address register, the microprocessor, and the microcode of the SR-IOV network card to the first memory space.
  • the hot migration module 702 is also used to lock the first task queue of the first virtual network card so that the first task queue cannot receive new tasks, and the first task queue is used to place the first virtual machine to issue
  • the server also includes: the first virtual function driver of the first virtual machine, used to clear all existing tasks in the first task queue.
  • the hot migration module 702 is also used to: iteratively copy the memory data; in the case that the remaining memory data can be copied at one time, after the first virtual function driver clears all existing tasks in the first task queue
  • the virtual machine manager 701 is also used to suspend the first virtual machine; the hot migration module 702 is also used to execute the last memory data copy.
  • FIG. 8 is a schematic diagram of an embodiment of a destination communication device in an embodiment of this application.
  • the embodiment of the present application provides a communication device.
  • the communication device is a target server for virtual machine hot migration.
  • the target server includes an SR-IOV network card.
  • the target server runs a virtual machine manager 801 and a hot migration module. 802.
  • the server includes: the virtual machine manager 801, configured to obtain a hot migration command, where the hot migration command is used to indicate that the first virtual machine of the source server will be migrated to the second virtual machine of the destination server; the virtual machine
  • the manager 801 is further configured to receive the status information of the first virtual network card corresponding to the first virtual machine sent by the source server;
  • the hot migration module 802 is further configured to change the status according to the instructions of the virtual machine manager 801 The information is restored to the SR-IOV network card.
  • the hot migration module 802 is specifically used to: restore the state information to the base address register, microcode, and microprocessor of the SR-IOV network card, respectively.
  • the hot migration module 802 is specifically used for: copying the first information in the state information to the microprocessor, the first information being the information obtained from the read-only base address register of the source server, and the microprocessor for Restore the first information to the read-only base address register of the SR-IOV network card.
  • the virtual machine manager 801 is further configured to: allocate the identification information of the second virtual network card corresponding to the second virtual machine through the hot migration module 802.
  • the virtual machine manager 801 is specifically configured to: receive second information sent by the source server, the second information being the lock status information of the second task queue of the second virtual network card corresponding to the second virtual machine, and the second virtual
  • the network card is virtualized based on the SR-IOV network card, the second task queue is used to place tasks issued for the second virtual machine; the hot migration module 802 is also used to unlock the second task queue.
  • FIG. 9 is a schematic diagram of another embodiment of a communication device in an embodiment of this application.
  • the communication device provided in this embodiment is a source server for the hot migration of virtual machines, and the specific device form is not limited in the embodiment of this application.
  • the source communication device 900 may have relatively large differences due to different configurations or performances, and may include one or more processors 901 and a memory 902, and the memory 902 stores programs or data.
  • the memory 902 may be volatile storage or non-volatile storage.
  • the processor 901 is one or more central processing units (CPU, Central Processing Unit).
  • the CPU may be a single-core CPU or a multi-core CPU.
  • the processor 901 may communicate with the memory 902 and communicate with the device at the source end. A series of instructions in memory 902 are executed on 900.
  • the source communication device 900 also includes a physical network card 903, which is a PCIE device, that is, an SR-IOV network card.
  • the source communication device 900 may also include one or more power supplies; one or more wired or wireless network interfaces, such as Ethernet interfaces; one or more input and output interfaces, input
  • the output interface can be used to connect a display, a mouse, a keyboard, a touch screen device or a sensor device, etc.
  • the input and output interface is an optional component, which may or may not exist, and is not limited here.
  • FIG. 10 is a schematic diagram of another embodiment of a communication device in an embodiment of this application.
  • the communication device provided in this embodiment is a target server for virtual machine hot migration, and its specific device form is not limited in this embodiment of the application.
  • the destination communication device 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors 1001 and a memory 1002, and the memory 1002 stores programs or data.
  • the memory 1002 may be volatile storage or non-volatile storage.
  • the processor 1001 is one or more central processing units (CPU, Central Processing Unit).
  • the CPU may be a single-core CPU or a multi-core CPU.
  • the processor 1001 may communicate with the memory 1002 and communicate with the device at the destination. A series of instructions in memory 1002 are executed on 1000.
  • the destination communication device 1000 further includes a physical network card 1003, and the physical network card 1003 is a PCIE device, that is, an SR-IOV network card.
  • the destination communication device 1000 may also include one or more power supplies; one or more wired or wireless network interfaces, such as Ethernet interfaces; one or more input and output interfaces, input
  • the output interface can be used to connect a display, a mouse, a keyboard, a touch screen device or a sensor device, etc.
  • the input and output interface is an optional component, which may or may not exist, and is not limited here.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , Including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Stored Programmes (AREA)
  • Hardware Redundancy (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Computer And Data Communications (AREA)

Abstract

本申请实施例公开了一种虚拟机热迁移方法，可实现SR-IOV网卡状态热迁移。应用于运行有第一虚拟机的源服务器，该方法用于向目的服务器迁移该第一虚拟机，该源服务器包括单根输入/输出虚拟化SR-IOV网卡，该源服务器中运行有第一虚拟机、该第一虚拟机使用的第一虚拟网卡、虚拟机管理器和热迁移模块，该第一虚拟网卡是基于该SR-IOV网卡虚拟出的，该方法包括：该虚拟机管理器向该热迁移模块下发热迁移命令；该热迁移模块根据该热迁移命令获取该第一虚拟机需迁移的信息，该第一虚拟机需迁移的信息包括该第一虚拟网卡的状态信息以及该第一虚拟机的内存数据，并上报该虚拟机管理器；该虚拟机管理器向该目的服务器发送该状态信息以及该第一虚拟机的内存数据。

Description

虚拟机热迁移方法和通信设备
本申请要求于2019年7月31日提交中国专利局、申请号为201910704642.2、发明名称为“虚拟机热迁移方法和通信设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及通信技术领域，特别涉及一种虚拟机热迁移方法和通信设备。
背景技术
虚拟机的热迁移又称动态迁移或在线迁移，是指在保证应用服务正常运行的同时，将一个虚拟机从一台物理服务器迁移到另一台物理服务器上，为了保证迁移过程中客户机服务的可用性，迁移过程只能有非常短暂的停机时间。依靠虚拟机在不同的主机之间迁移，集群能够有效地进行资源的分配和调度。
单根输入/输出虚拟化(single-root I/O virtualization，SR-IOV)是一种输入输出(input/output，I/O)虚拟化技术，该技术可以在一个物理快捷外围部件互连标准(peripheral component interconnection express，PCIE)设备上虚拟化出多个虚拟的PCIE设备，称之为虚拟功能(virtual function，VF)。SR-IOV网卡可以虚拟化出多个虚拟网卡，虚拟机通过虚拟网卡执行业务，每个虚拟机可以绕开虚拟机管理器，直接与其对应的一个或多个虚拟网卡进行网络I/O的数据交互，虚拟网卡的状态信息保存在网卡的存储介质中，虚拟机管理器无法获取SR-IOV网卡的存储介质中的状态信息。
虚拟机热迁移时，需要将虚拟机对应虚拟网卡的状态信息由源服务器迁移至目的服务器，由于虚拟网卡的状态信息保存在物理网卡的存储介质中，虚拟机管理器不感知SR-IOV网卡的存储介质，无法获取该状态信息，因此无法实现虚拟机对应的虚拟网卡的状态信息迁移。
发明内容
本申请实施例提供了一种虚拟机热迁移方法，可以实现SR-IOV网卡状态热迁移，从而使得使用SR-IOV网卡的物理机同样可以执行虚拟机的热迁移过程。
本申请实施例第一方面提供了一种虚拟机热迁移方法，应用于运行有第一虚拟机的源服务器，该方法用于向目的服务器迁移该第一虚拟机，该源服务器包括单根输入/输出虚拟化SR-IOV网卡，该源服务器中运行有第一虚拟机、该第一虚拟机使用的第一虚拟网卡、虚拟机管理器和热迁移模块，该第一虚拟网卡是基于该SR-IOV网卡虚拟出的，该方法包括：该虚拟机管理器向该热迁移模块下发热迁移命令；该热迁移模块根据该热迁移命令获取该第一虚拟机需迁移的信息，该第一虚拟机需迁移的信息包括该第一虚拟网卡的状态信息以及该第一虚拟机的内存数据，并上报该虚拟机管理器；该虚拟机管理器向该目的服务器发送该状态信息以及该第一虚拟机的内存数据。
当源服务器中的第一虚拟机从源服务器迁移至目的服务器时，源服务器获取热迁移请求，源服务器与目的主机建立连接，并根据热迁移请求向目的服务器发送热迁移消息，该热迁移消息用于指示运行于该主机设备的第一虚拟机迁移至该目的服务器；热迁移模块接收该虚拟 机管理器根据该热迁移请求下发的热迁移命令，可以通过新增的该热迁移模块获取该第一虚拟机对应的第一虚拟网卡的状态信息；然后，该虚拟机管理器将该状态信息发送给该目的服务器。由此，可以将该状态信息迁移至目的服务器，进而实现第一虚拟机的迁移。
在第一方面的一种可能的实现方式中，该热迁移模块根据该热迁移命令获取该第一虚拟机需迁移的信息包括：该热迁移模块根据该热迁移命令，将该状态信息拷贝至该虚拟机管理器分配的该第一内存空间，该第一内存空间不包括在给该第一虚拟机分配的虚拟内存内。
本申请实施例提供的虚拟机热迁移方法，热迁移模块可以将获取的状态信息拷贝至虚拟机管理器分配的第一内存空间，该第一内存空间不包括在给该第一虚拟机分配的虚拟内存内，第一虚拟机不感知该迁移过程。
在第一方面的一种可能的实现方式中，该热迁移模块根据该热迁移命令，将该状态信息拷贝至该虚拟机管理器分配的该第一内存空间包括：该热迁移模块从该SR-IOV网卡的基地址寄存器、微处理器和微码中拷贝该状态信息至该第一内存空间。
本申请实施例提供的虚拟机热迁移方法，热迁移模块获取该状态信息的包括了分别从物理网卡的基地址寄存器、微处理器和微码中进行拷贝，提供了获取状态信息的一种可行的实现方式。
在第一方面的一种可能的实现方式中，该热迁移模块根据该热迁移命令获取该第一虚拟机需迁移的信息之前，该方法还包括：该热迁移模块锁定该第一虚拟网卡的第一任务队列，以使得该第一任务队列无法接收新的任务，该第一任务队列用于放置为该第一虚拟机下发的任务；该第一虚拟机的第一虚拟功能驱动清空该第一任务队列中所有已有的任务。
在第一虚拟机的迁移过程中，若用户通过该第一虚拟机下发用户命令改变对应的虚拟网卡的状态信息，将可能发生在命令执行完成之前，第一虚拟机对应的虚拟网卡状态信息已经迁移到目标虚拟机的情况，有可能产生命令超时等异常，导致业务中断。本申请实施例提供的虚拟机热迁移方法，在获取虚拟网卡的状态信息之前，该热迁移模块可以锁定第一任务队列，第一虚拟功能驱动可执行第一任务队列中的任务直至排空该队列。这样用户将无法修改第一虚拟机对应的虚拟网卡的状态信息，还可以避免业务异常。
在第一方面的一种可能的实现方式中，该虚拟机管理器向该热迁移模块下发热迁移命令之后，该方法还包括：该热迁移模块对该内存数据进行迭代拷贝；在剩余的该内存数据可一次拷贝完成的情况下，该第一虚拟功能驱动清空该第一任务队列中所有已有的任务之后，该虚拟机管理器挂起该第一虚拟机；该热迁移模块执行最后一次内存数据拷贝。
本申请实施例提供的虚拟机热迁移方法，限定了热迁移模块在内存数据迭代拷贝剩余内存可一次拷贝完成时，锁定并排空执行用户命令的通道，由于锁定执行用户命令的通道后，将不再响应新的用户命令，剩余内存可一次拷贝完成时执行锁定操作，可以减少不响应用户命令阶段的时长。
本申请实施例第二方面提供了一种虚拟机热迁移方法，应用于虚拟机迁移的目的服务器，该目的服务器包括SR-IOV网卡，该目的服务器中运行有虚拟机管理器和热迁移模块，该方法包括：该虚拟机管理器获取热迁移命令，该热迁移命令用于指示源服务器的第一虚拟机将将迁移至该目的服务器的第二虚拟机；该虚拟机管理器接收该源服务器发送的该第一虚拟机对应的第一虚拟网卡的状态信息；该热迁移模块根据该虚拟机管理器的指示，将该状态信息恢 复至该SR-IOV网卡。
当源服务器中的第一虚拟机从源服务器迁移至目的服务器的第二虚拟机时，需要将第一虚拟机对应的第一网卡的状态信息也迁移至目的服务器的物理网卡中。该虚拟机管理器接收源服务器发送的第一虚拟机对应的虚拟网卡的状态信息，目的服务器的虚拟机管理器接收到该状态信息后，通过该热迁移模块需要将该状态信息恢复至物理网卡中。使得第一虚拟机的成功迁移。
在第二方面的一种可能的实现方式中，该热迁移模块根据该虚拟机管理器的指示，将该状态信息恢复至该SR-IOV网卡包括：该热迁移模块将该状态信息分别恢复至该SR-IOV网卡的基地址寄存器、微码和微处理器中。
本申请实施例提供的虚拟机热迁移方法，可以将状态信息分别写入该物理网卡的基地址寄存器、微码和微处理器中，增强了方案的可实现性。
在第二方面的一种可能的实现方式中，该热迁移模块将该状态信息分别恢复至该SR-IOV网卡的基地址寄存器、微码和微处理器中包括：该热迁移模块将该状态信息中的第一信息拷贝至该微处理器，该第一信息为从该源服务器的只读基地址寄存器中获取的信息，该微处理器用于将该第一信息恢复至该SR-IOV网卡的只读基地址寄存器。
对于从源服务器的只读基地址寄存器中获取的第一信息，热迁移模块不能直接写入目的服务器的只读基地址寄存器，而是通过拷贝至微处理器中进行恢复，由于微处理器权限较高，可以解决只读基地址寄存器中不能直接写入的问题。
在第二方面的一种可能的实现方式中，该虚拟机管理器接收该源服务器发送的该第一虚拟机对应的第一虚拟网卡的状态信息之前，该方法还包括：该虚拟机管理器通过该热迁移模块分配该第二虚拟机对应的第二虚拟网卡的标识信息。
本申请实施例提供的虚拟机热迁移方法，在接收源服务器发送的第一虚拟机对应的虚拟网卡的状态信息之前，进行全局资源分配，可以避免全局资源被占用带来的冲突。
在第二方面的一种可能的实现方式中，该虚拟机管理器接收该源服务器发送的该第一虚拟机对应的第一虚拟网卡的状态信息包括：该虚拟机管理器接收该源服务器发送的第二信息，该第二信息为该第二虚拟机对应的第二虚拟网卡的第二任务队列的锁定状态信息，该第二虚拟网卡是基于该SR-IOV网卡虚拟出的，该第二任务队列用于放置为该第二虚拟机下发的任务；该方法还包括：该热迁移模块解锁该第二任务队列。
本申请实施例提供的虚拟机热迁移方法，可以将锁定的执行用户命令的通道解锁，这样可以在网卡状态恢复后继续执行用户下发的命令。
该方法还包括：该虚拟机管理器接收该源服务器发送的第一虚拟机对应的内存数据；该虚拟机管理器恢复该第一虚拟机对应的内存数据。
本申请实施例第三方面提供了一种虚拟机热迁移方法，应用于运行有第一虚拟机的源服务器，该方法用于向目的服务器迁移该第一虚拟机，该源服务器包括单根输入/输出虚拟化SR-IOV网卡，该源服务器中运行有第一虚拟机、该第一虚拟机使用的第一虚拟网卡、虚拟机管理器和物理功能驱动，该第一虚拟网卡是基于该SR-IOV网卡虚拟出的，该方法包括：该虚拟机管理器向该虚拟功能驱动下发热迁移命令；该虚拟功能驱动根据该热迁移命令获取该第一虚拟机需迁移的信息，该第一虚拟机需迁移的信息包括该第一虚拟机对应的第一虚拟网卡 的状态信息以及该第一虚拟机的内存数据，并上报该虚拟机管理器；该虚拟机管理器向该目的服务器发送该状态信息以及该第一虚拟机的内存数据。
本申请实施例提供的虚拟机热迁移方法，虚拟机管理器不感知SR-IOV网卡的存储介质，无法获取虚拟网卡的状态信息，物理功能驱动则可以与SR-IOV网卡进行交互，本申请实施例中虚拟机管理器通过VFIO与物理功能驱动连接，进而可以通过物理功能驱动获取虚拟网卡的状态信息，进而实现第一虚拟机的迁移。
在第三方面的一种可能的实现方式中，该虚拟功能驱动根据该热迁移命令获取该第一虚拟机需迁移的信息包括：该虚拟功能驱动根据该热迁移命令，将该状态信息拷贝至该虚拟机管理器分配的该第一内存空间，该第一内存空间不包括在给该第一虚拟机分配的虚拟内存内。
在第三方面的一种可能的实现方式中，该虚拟功能驱动根据该热迁移命令，将该状态信息拷贝至该虚拟机管理器分配的该第一内存空间包括：该虚拟功能驱动从该SR-IOV网卡的基地址寄存器、微处理器和微码中拷贝该状态信息至该第一内存空间。
在第三方面的一种可能的实现方式中，该虚拟功能驱动根据该热迁移命令，将该状态信息拷贝至该虚拟机管理器分配的该第一内存空间之前，该方法还包括：该虚拟功能驱动锁定该第一虚拟网卡的第一任务队列，以使得该第一任务队列无法接收新的任务，该第一任务队列用于放置为该第一虚拟机下发的任务；该第一虚拟机运行的第一虚拟功能驱动清空该第一任务队列中所有已有的任务。
在第三方面的一种可能的实现方式中，该虚拟机管理器获取热迁移请求之后，该方法还包括：该虚拟功能驱动对该内存数据进行迭代拷贝；在剩余的该内存数据可一次拷贝完成的情况下，该第一虚拟功能驱动清空该第一任务队列中所有已有的任务之后，该虚拟机管理器挂起该第一虚拟机；该虚拟功能驱动执行最后一次内存数据拷贝。
在第三方面的一种可能的实现方式中，该虚拟功能驱动通过用户态驱动框架VFIO与虚拟机管理器连接。
在第三方面的一种可能的实现方式中，该虚拟功能驱动包括热迁移模块，该热迁移模块用于根据虚拟机管理器的指示获取第一虚拟网卡的状态信息。
本申请实施例提供的虚拟机热迁移方法，在物理功能驱动可以通过新增的热迁移模块，用于辅助实现虚拟机热迁移。虚拟机管理器不感知SR-IOV网卡的存储介质，无法获取虚拟网卡的状态信息，物理功能驱动则可以与SR-IOV网卡进行交互，本申请实施例中通过在物理功能驱动中新增热迁移模块，虚拟机管理器通过VFIO与该热迁移模块连接，进而可以通过热迁移模块获取虚拟网卡的状态信息，进而实现第一虚拟机的迁移。
本申请实施例第四方面提供了一种虚拟机热迁移方法，其特征在于，应用于虚拟机迁移的目的服务器，该目的服务器包括SR-IOV网卡，该目的服务器中运行有虚拟机管理器和物理功能驱动，该方法包括：该虚拟机管理器获取热迁移命令，该热迁移命令用于指示源服务器的第一虚拟机将将迁移至该目的服务器的第二虚拟机；该虚拟机管理器接收该源服务器发送的该第一虚拟机对应的第一虚拟网卡的状态信息；该虚拟功能驱动根据该虚拟机管理器的指示，将该状态信息恢复至该SR-IOV网卡。
在第四方面的一种可能的实现方式中，该虚拟功能驱动根据该虚拟机管理器的指示，将该状态信息恢复至该SR-IOV网卡包括：该虚拟功能驱动将该状态信息分别恢复至该SR-IOV 网卡的基地址寄存器、微码和微处理器中。
在第四方面的一种可能的实现方式中，该虚拟功能驱动将该状态信息分别恢复至该SR-IOV网卡的基地址寄存器、微码和微处理器中包括：该虚拟功能驱动将该状态信息中的第一信息拷贝至该微处理器，该第一信息为从该源服务器的只读基地址寄存器中获取的信息；该微处理器将该第一信息恢复至该SR-IOV网卡的只读基地址寄存器。
在第四方面的一种可能的实现方式中，该虚拟机管理器接收该源服务器发送的该第一虚拟机对应的第一虚拟网卡的状态信息之前，该方法还包括：该虚拟机管理器通过该虚拟功能驱动分配该第二虚拟机对应的第二虚拟网卡的标识信息。
该虚拟机管理器接收该源服务器发送的该第一虚拟机对应的第一虚拟网卡的状态信息包括：该虚拟机管理器接收该源服务器发送的第二信息，该第二信息为该第二虚拟机对应的第二虚拟网卡的第二任务队列的锁定状态信息，该第二虚拟网卡是基于该SR-IOV网卡虚拟出的，该第二任务队列用于放置为该第二虚拟机下发的任务；该方法还包括：该虚拟功能驱动解锁该第二任务队列。
在第四方面的一种可能的实现方式中，该虚拟功能驱动包括热迁移模块，该热迁移模块用于根据虚拟机管理器的指示将该状态信息恢复至该SR-IOV网卡。
本申请实施例提供的虚拟机热迁移方法，在物理功能驱动可以通过新增的热迁移模块用于辅助实现虚拟机热迁移。
本申请实施例第五方面提供了一种通信设备，该通信设备为虚拟机热迁移的源源端，该通信设备中运行有第一虚拟机、该第一虚拟机使用的第一虚拟网卡、虚拟机管理器和热迁移模块，该第一虚拟网卡是基于SR-IOV网卡虚拟出的，该方法包括：该虚拟机管理器，用于向该热迁移模块下发热迁移命令；该热迁移模块，用于根据该热迁移命令获取该第一虚拟机需迁移的信息，该第一虚拟机需迁移的信息包括该第一虚拟网卡的状态信息以及该第一虚拟机的内存数据，并上报该虚拟机管理器；该虚拟机管理器，还用于向该目的端通信设备发送该状态信息以及该第一虚拟机的内存数据。
在第五方面的一种可能的实现方式中，该热迁移模块具体用于：根据该热迁移命令，将该状态信息拷贝至该虚拟机管理器分配的该第一内存空间，该第一内存空间不包括在给该第一虚拟机分配的虚拟内存内。
在第五方面的一种可能的实现方式中，该热迁移模块具体用于：从该SR-IOV网卡的基地址寄存器、微处理器和微码中拷贝该状态信息至该第一内存空间。
在第五方面的一种可能的实现方式中，该热迁移模块，还用于锁定该第一虚拟网卡的第一任务队列，以使得该第一任务队列无法接收新的任务，该第一任务队列用于放置为该第一虚拟机下发的任务；该通信设备还包括：该第一虚拟机的第一虚拟功能驱动，用于清空该第一任务队列中所有已有的任务。
在第五方面的一种可能的实现方式中，该热迁移模块还用于：对该内存数据进行迭代拷贝；在剩余的该内存数据可一次拷贝完成的情况下，该第一虚拟功能驱动清空该第一任务队列中所有已有的任务之后，该虚拟机管理器还用于，挂起该第一虚拟机；该热迁移模块，还用于执行最后一次内存数据拷贝。
本申请实施例第六方面提供了一种通信设备，该通信设备为虚拟机热迁移的目的端通信 设备，该通信设备包括SR-IOV网卡，该通信设备中运行有虚拟机管理器和热迁移模块，该通信设备包括：该虚拟机管理器，用于获取热迁移命令，该热迁移命令用于指示源端通信设备的第一虚拟机将将迁移至该目的端通信设备的第二虚拟机；该虚拟机管理器，还用于接收该源端通信设备发送的该第一虚拟机对应的第一虚拟网卡的状态信息；该热迁移模块，还用于根据该虚拟机管理器的指示，将该状态信息恢复至该SR-IOV网卡。
在第六方面的一种可能的实现方式中，该热迁移模块具体用于：将该状态信息分别恢复至该SR-IOV网卡的基地址寄存器、微码和微处理器中。
在第六方面的一种可能的实现方式中，该热迁移模块具体用于：将该状态信息中的第一信息拷贝至该微处理器，该第一信息为从该源通信设备的只读基地址寄存器中获取的信息，该微处理器用于将该第一信息恢复至该SR-IOV网卡的只读基地址寄存器。
在第六方面的一种可能的实现方式中，该虚拟机管理器还用于：通过该热迁移模块分配该第二虚拟机对应的第二虚拟网卡的标识信息。
在第六方面的一种可能的实现方式中，该虚拟机管理器具体用于：接收该源通信设备发送的第二信息，该第二信息为该第二虚拟机对应的第二虚拟网卡的第二任务队列的锁定状态信息，该第二虚拟网卡是基于该SR-IOV网卡虚拟出的，该第二任务队列用于放置为该第二虚拟机下发的任务；该热迁移模块，还用于解锁该第二任务队列。
本申请实施例第七方面提供了一种通信设备，为虚拟机热迁移的源端通信设备，包括：处理器、存储器和SR-IOV网卡；该处理器用于运行第一虚拟机、虚拟机管理器和热迁移模块，该第一虚拟机运行第一虚拟功能驱动，该处理器通过该虚拟机管理器、该热迁移模块和该第一虚拟功能驱动执行如前述第一方面提供的各实施方式的方法。
本申请实施例第七方面提供的一种通信设备中，该处理器、存储器和SR-IOV网卡电连接，可选地，该通信设备还包括总线，该处理器、存储器和SR-IOV网卡通过该总线相连。
本申请实施例第八方面提供了一种通信设备，为虚拟机热迁移的目的端通信设备，包括：处理器、存储器和SR-IOV网卡；该处理器用于运行虚拟机管理器和热迁移模块，并通过该虚拟机管理器和热迁移模块执行如前述第二方面提供的各实施方式的方法。
本申请实施例第八方面提供的一种通信设备中，该处理器、存储器和SR-IOV网卡电连接，可选地，该通信设备还包括总线，该处理器、存储器和SR-IOV网卡通过该总线相连。
本申请实施例第九方面提供了一种通信设备，为虚拟机热迁移的源端通信设备，包括：处理器、存储器和SR-IOV网卡；该处理器用于运行第一虚拟机、虚拟机管理器和物理功能驱动，该第一虚拟机运行第一虚拟功能驱动，该处理器通过该虚拟机管理器、该物理功能驱动和该第一虚拟功能驱动执行前述第三方面提供的各实施方式的方法。
本申请实施例第九方面提供的一种通信设备中，该处理器、存储器和SR-IOV网卡电连接，可选地，该通信设备还包括总线，该处理器、存储器和SR-IOV网卡通过该总线相连。
本申请实施例第十方面提供了一种通信设备，为虚拟机热迁移的目的端通信设备，包括：处理器、存储器和SR-IOV网卡；该处理器用于运行虚拟机管理器，和物理功能驱动，并通过该虚拟机管理器和该物理功能驱动执行前述第四方面提供的各实施方式的方法。
本申请实施例第十方面提供的一种通信设备中，该处理器、存储器和SR-IOV网卡电连接，可选地，该通信设备还包括总线，该处理器、存储器和SR-IOV网卡通过该总线相连。
本申请实施例第十一方面提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得该计算机执行前述第一方面至第四方面提供的各实施方式的步骤。
本申请实施例第十二方面提供了一种计算机可读存储介质，包括指令，当该指令在计算机上运行时，使得计算机执行前述第一方面至第四方面提供的各实施方式的步骤。
本申请实施例第十三方面提供了一种虚拟机热迁移系统，包括前述第五方面提供的源端通信设备，和前述第六方面提供的目的端通信设备。
从以上技术方案可以看出，本申请实施例具有以下优点：
本申请实施例提供的虚拟机热迁移方法，通过热迁移模块可获取SR-IOV网卡中的第一虚拟网卡的状态信息，并将该状态信息迁移至目的服务器中。本申请实施例提供的虚拟机热迁移方法可以使得该状态信息迁移到第二虚拟机侧，进而保证第一虚拟机的成功迁移。
附图说明
图1为虚拟机热迁移的一种迁移架构示意图；
图2为本申请实施例的虚拟机热迁移的系统架构示意图；
图3为本申请实施例中虚拟机热迁移方法的一个实施例示意图；
图4为本申请实施例中虚拟机热迁移方法的另一个实施例示意图；
图5为本申请实施例中虚拟机热迁移方法的一个实施例交互示意图；
图6为本申请实施例中锁定并排空任务队列的一个实施例示意图；
图7为本申请实施例中源端通信设备的一个实施例示意图；
图8为本申请实施例中目的端通信设备的一个实施例示意图；
图9为本申请实施例中源端通信设备的另一个实施例示意图；
图10为本申请实施例中目的端通信设备的另一个实施例示意图。
具体实施方式
本申请实施例提供了一种虚拟机热迁移方法，可以实现网卡的状态信息的迁移，从而使得使用SR-IOV网卡的物理机同样可以执行虚拟机的热迁移过程。
请参阅图1，为虚拟机热迁移的一种迁移架构示意图。
该迁移架构中包括虚拟机迁移前的源服务器100和虚拟机迁移后的目的服务器110。在一种可能的应用场景中，可以将运行在源服务器100中的虚拟机1的运行状态迁移到目的服务器110中任一虚拟机，例如，将虚拟机1迁移至虚拟机2。
源服务器和目的服务器均为部署了虚拟机管理器的宿主机，可以为虚拟机提供物理资源。通常在一个宿主机上可以创建多个虚拟机，创建虚拟机的个数取决于宿主机本身的硬件配置和需要创建的虚拟机规格大小。此处对于源服务器100上运行的虚拟机数量N和目的服务器上运行的虚拟机数量M不做限定。在实际应用场景中，宿主机可以是提供各种云服务的服务器。
请参阅图2，为本申请实施例的虚拟机热迁移的系统示意图。
该系统中包括虚拟机迁移前的源服务器和虚拟机热迁移后的目的服务器。
源服务器包括主机和SR-IOV网卡，具体地，主机(HOST)中运行至少一个虚拟机，图2中仅示出一个热迁移前的第一虚拟机VM1。
该主机还运行有源服务器的虚拟机管理器，该虚拟机管理器可以管理一个或多个虚拟机VM，每个虚拟机VM上可以运行操作系统以及运行在操作系统上的各种应用。
该虚拟机管理器为可以是运行在主机设备上的管理模块，例如hypervisor或虚拟机监控器(virtual machine monitor，VMM)，也可以由定制的逻辑芯片实现。该虚拟机管理器负责虚拟机的管理和控制，包括虚拟机的创建、迁移、上电和下电等操作。该虚拟机管理器具体可以是qemu(Quick EMUlator)、VMWare、ESXi或基于内核的虚拟机(kernel-based virtual machine，KVM)等，具体此处不做限定。
热迁移模块，是本实施例中新增模块，为虚拟机管理器提供热迁移接口，可以根据虚拟机管理器的指示辅助实现热迁移，本申请实施例中热迁移模块可以用于获取虚拟机对应的虚拟网卡的状态信息。可选的，在物理功能驱动中新增该热迁移模块，由于虚拟机管理器不感知SR-IOV网卡的存储介质，无法获取虚拟网卡的状态信息，物理功能驱动则可以与SR-IOV网卡进行交互。虚拟机管理器通过VFIO与该热迁移模块连接，可以通过热迁移模块获取虚拟网卡的状态信息，进而实现第一虚拟机的迁移。
SR-IOV网卡包括至少一个物理功能(physical function，PF)以及至少一个虚拟功能VF，本实施例中SR-IOV网卡的虚拟功能即虚拟网卡，图1中仅示出了一个物理功能与一个第一虚拟网卡。一个虚拟机对应至少一个虚拟网卡，虚拟网卡的状态信息可以保存在SR-IOV网卡的基地址寄存器(base address register，BAR)、管理单元和数据处理单元中。本实施例中管理单元包括微处理器，数据处理单元包括微码。
该主机设备上还运行有管理SR-IOV网卡的PF驱动，该PF驱动可以直接访问PF的所有资源，并负责配置和管理所有VF，每个虚拟机VM运行有VF驱动来管理各自对应的VF。
目的服务器包括主机(HOST)和SR-IOV网卡，该HOST上运行有目的服务器的虚拟机管理器和热迁移模块，主机设备与SR-IOV网卡连接。
目的服务器的热迁移模块，为本实施例中新增模块，为虚拟机管理器提供热迁移接口，可以根据虚拟机管理器的指示辅助实现热迁移，本申请实施例中热迁移模块可以用于将获取的虚拟机对应的虚拟网卡的状态信息恢复至SR-IOV网卡。可选的，在物理功能驱动中新增该热迁移模块，虚拟机管理器通过VFIO与该热迁移模块连接。
可以理解的是，在实际应用中，一个服务器既可能作为虚拟机迁移的源端，也可以作为虚拟机迁移的目的端，因此本实施例中提供的源服务器和目的服务器的软件模块的功能可以集成在一个服务器中实现。
需要说明的是，尽管图2中未示出，但虚拟机管理器和各虚拟机之间存在有通信链路。尽管图2中未示出，该主机设备还包括与运行虚拟机VM相关的硬件，例如中央处理器(central processing unit，CPU)及至少一个计算机可读存储介质的存储器，包括硬盘或内存等。除此之外，该主机设备还可以包括通信单元、输入单元、显示单元等。
由于每个虚拟机可以与其对应的一个或多个虚拟网卡直接进行网络I/O的数据交互，无需通过虚拟机管理器，既降低了虚拟机管理器中的I/O性能开销，又保证了每个VM的I/O吞吐效率。具体的，虚拟机VM从虚拟网卡收发数据报文时，可实现收发完全独立，在进行直接内存访问(direct memory access，DMA)写入的过程中虚拟机管理器并不能感知。虚拟机对应的虚拟网卡的状态信息存储在SR-IOV网卡的存储介质中，虚拟机管理器也不能感知。
虚拟机热迁移中的关键技术点包括内存状态迁移和网卡状态迁移。
通常虚拟机内存的迁移可为三个阶段：
迭代预拷贝阶段:虚拟机迁移过程启动后，其依然运行在源服务器上，虚拟机业务不中断，首先把虚拟机的全部内存复制到目的服务器上，然后向目的服务器不停的迭代复制虚拟机中变化的内存数据。当迭代到需要复制的内存小到一定的门限后，进入停机拷贝阶段。
停机拷贝阶段:虚拟机停机，业务中断，拷贝虚拟机的残余内存数据。
恢复阶段：目的虚拟机完成运行前的恢复处理，启动目的虚拟机，整个迁移过程完成。
虚拟机内存的迁移方式有多种，本申请实施例中并不限定。
网卡状态迁移是指将进行迁移的虚拟机对应的虚拟网卡的状态信息由源服务器迁移至目的服务器。
虚拟机迁移过程中需进行迁移的虚拟网卡的状态信息主要包括以下几类:
(1)、设备信息，包括：状态信息和中断表等，存储在BAR寄存器的BAR0/1和BAR2/3空间中。
(2)、芯片与主机交互制信息和报文上下文，包括：响应事件队列、Mailbox、控制队列、报文描述队列和队列状态等。部分存储在BAR寄存器BAR0/1空间；部分存储在微码中网卡芯片内部表项中，可以通过微码获取和保存。
(3)、网卡业务上下文信息，主要用于管理VF、PF的功能，包括：函数表，MAC-VLAN表和RSS模板表等，可以通过微处理器获取和保存。
由于虚拟网卡的状态信息存储在SR-IOV网卡的BAR寄存器、微码和微处理器中，当源服务器的第一虚拟机要迁移到目的服务器的第二虚拟机时，源服务器的虚拟机管理器无法获取SR-IOV网卡的存储介质中该第一虚拟机对应的虚拟网卡的状态信息。要实现第一虚拟机的成功迁移，该第一虚拟网卡的状态信息也需要迁移到第二虚拟机侧，以使得第二虚拟机对应的第二虚拟网卡恢复工作。为解决此问题，本申请实施例中通过新增的热迁移模块，可以获取网卡中的第一虚拟网卡的状态信息，虚拟机管理器可以将该状态信息拷贝至目的服务器，最终保证第一虚拟机的成功迁移。
基于图2提供的虚拟机热迁移的系统架构，请参阅图3，为本申请实施例中虚拟机热迁移方法的一个实施例示意图。
301、虚拟机管理器向所述热迁移模块下发热迁移命令；
当源服务器中的第一虚拟机从源服务器迁移至目的服务器时，源服务器获取热迁移请求，源服务器根据该热迁移请求与目的主机建立连接，并求向该目的服务器发送热迁移请求，指示源服务器中第一虚拟机迁移至该目的服务器，目的服务器可以进行热迁移前的准备工作，例如，目的服务器为该第一虚拟机预保留虚拟机资源。
虚拟机管理器获取热迁移请求的方式有多种，可选的，用户通过源服务器发送热迁移请求，或者服务器根据预设的规则确定第一虚拟机进行热迁移，具体此处不做限定。
虚拟机管理器向所述热迁移模块下发热迁移命令，需要说明的是，虚拟机管理器在第一虚拟机迁移过程中可以向热迁移模块发送多个热迁移命令，此处对于热迁移命令的数量不做限定。可选的，虚拟机管理器通过用户态驱动框架(virtual function I/O，VFIO)，在热迁移的不同阶段，向热迁移模块发送不同的热迁移命令。
302、热迁移模块根据所述热迁移命令获取所述第一虚拟机需迁移的信息；
第一虚拟机需迁移的信息包括该第一虚拟网卡的状态信息以及该第一虚拟机的内存数据，并上报所述虚拟机管理器。
其中，第一虚拟网卡的状态信息存储在SR-IOV网卡中，具体地，存储在SR-IOV网卡的BAR寄存器、管理单元和数据处理单元中。
虚拟机管理器根据该热迁移请求，通过该热迁移模块获取存储在SR-IOV网卡的BAR寄存器、管理单元和数据处理单元中的第一虚拟网卡的状态信息。
可选的，由虚拟机管理器分配的第一内存空间，热迁移模块根据状态信息存储位置分别从网卡的BAR寄存器、微处理器和微码中获取第一虚拟网卡的状态信息，并将该状态信息保存到该第一内存空间。需要说明的是，该第一内存空间不属于第一虚拟机对应的内存空间，由虚拟机管理器直接分配控制。
可选的，虚拟机管理器通过VFIO向热迁移模块发送控制命令。
303、虚拟机管理器向所述目的服务器发送所述状态信息以及所述第一虚拟机的内存数据；
虚拟机管理器将步骤302中获取的第一虚拟机的内存数据发送给该目的服务器进行恢复，此外，虚拟机管理器还将步骤302中获取的该状态信息发送给该目的服务器，以使得目的服务器可以根据该状态信息对迁移后的第二虚拟机进行网卡状态恢复，进而实现热迁移。
本申请实施例中通过源服务器通过新增的热迁移模块，可以获取网卡中的第一虚拟网卡的状态信息，并将该状态信息迁移至第二虚拟机。使得这些状态信息能够迁移到第二虚拟机侧，最终保证第一虚拟机的成功迁移。
基于图2提供的虚拟机热迁移的系统架构，请参阅图4，为本申请实施例中虚拟机热迁移方法的另一个实施例示意图。
401、虚拟机管理器接收源服务器发送的第一虚拟机对应的虚拟网卡的状态信息；
当源服务器中的第一虚拟机从源服务器迁移至目的服务器的第二虚拟机时，需要将第一虚拟机对应的第一网卡的状态信息也迁移至目的服务器的SR-IOV网卡中。
虚拟机管理器接收源服务器发送的该状态信息。
402、虚拟机管理器通过该热迁移模块，将该状态信息恢复至该SR-IOV网卡；
目的服务器的虚拟机管理器接收到该状态信息后，需要将该状态信息分别恢复至SR-IOV网卡的BAR寄存器、微码和微处理器中。
从源服务器的微处理器和微码获取的该状态信息可以直接写入目的服务器的微处理器和微码中；
由于BAR寄存器中包括BAR只读寄存器和BAR可写寄存器，从源端SR-IOV网卡BAR寄存器获取的该状态信息，恢复时需分两种情况进行处理，下面分别进行介绍：
1、从BAR可写寄存器中拷贝的该状态信息可以直接写入目的服务器的BAR可写寄存器；
2、从BAR只读寄存器中拷贝的该状态信息需要通过微处理器进行恢复，写入目的服务器SR-IOV网卡BAR只读寄存器中。
目的服务器的虚拟机管理器对该状态信息进行恢复后，可以使能网卡，进而启用第二虚拟机，完成该第一虚拟机的热迁移。
可选的，虚拟机管理器通过VFIO向热迁移模块发送控制命令。
本申请实施例中目的服务器接收源服务器发送的第一虚拟网卡的状态信息，可以通过新增的热迁移模块，实现该状态信息在SR-IOV网卡中的恢复，使得第一虚拟机的成功迁移。
基于图2提供的虚拟机热迁移的系统架构，请参阅图5，为本申请实施例中虚拟机热迁移方法的一个实施例交互示意图。
虚拟机热迁移技术常用于保持服务器的负载均衡或虚拟机的容灾、备份等场景中。当源服务器中的第一虚拟机从源服务器迁移至目的服务器时，可以根据本申请实施例提供的虚拟机热迁移方案进行迁移。
501、源服务器与目的服务器建立连接；
当源服务器中的第一虚拟机从源服务器迁移至目的服务器时，源服务器获取热迁移请求，源服务器根据该热迁移请求与目的主机建立连接，并求向该目的服务器发送热迁移请求，指示源服务器中第一虚拟机迁移至该目的服务器，目的服务器可以进行热迁移前的准备工作，例如，目的服务器为该第一虚拟机预保留虚拟机资源。
可选的，源服务器接收用户的迁移请求，源服务器根据目的服务器的主机IP地址与目的主机建立连接。
502、目的服务器执行全局资源分配；
目的服务器执行全局资源分配，即初始化VF资源，保证VF资源的可用性。目的服务器中，虚拟机管理器通过热迁移模块在操作系统中查询并申请可用的全局资源，包括第二虚拟机对应的第二虚拟网卡的标识信息。
全局资源是虚拟机对应的虚拟网卡在操作系统中保存的虚拟网卡标识信息。例如索引号(func_id)、RSS模板表标识或q_cfg等。
示例性的，源服务器中，第一虚拟机对应的第一虚拟网卡的索引号是1，迁移到目的服务器的第二虚拟机，目的服务器中，该索引号1可能被其他虚拟机占用，目的服务器为第二虚拟机对应的第二虚拟网卡分配的索引号可能是2。
示例性的，VF资源并不是独立分配的，例如RSS模板表，每个VF使用的RSS是微处理器全局分配的，在源端的rss_id是1，但1在目的端可能已经分配给其他的VF了。
可选的，目的端虚拟机管理器通过pre_active接口进行全局资源的分配。
可选的，目的服务器中，虚拟机管理器分配虚拟网卡标识信息的步骤被称为post_active流程。
可选的，热迁移模块通过微处理器重新分配VF对应的全局资源，然后执行状态恢复流程。热迁移模块通过微处理器和微码设置芯片状态信息和VM对应资源的配置，流程返回后执行虚拟机active操作。本申请实施例提供的虚拟机热迁移方法，实现了迁移源端VF和迁移目的端目标VF的配置一致性，保证了虚拟机在迁移后的正常工作。
需要说明的是，步骤502在步骤501之后，步骤508之前执行，与步骤503至507的执行顺序不做限定。
503、源服务器迭代拷贝内存数据；
虚拟机管理器可以根据该热迁移请求，通过该热迁移模块执行内存数据迭代拷贝。第一虚拟机迁移内存数据的方式有多种，此处不做限定。通常，内存数据的拷贝需要执行多次，迭代拷贝至需要复制的内存小于预设门限后，将进入停机拷贝阶段，通常剩余的内存可一次 拷贝完成。
504、源服务器锁定并排空第一任务队列；
第一任务队列中用于放置为第一虚拟机下发的任务，该任务由用户通过应用层下发，该第一任务队列在实际应用过程中被称为执行用户命令的通道(command queue，cmdq)。
可选的，其中包括与第一虚拟机对应的虚拟网卡的状态信息相关的任务；
可选的，用户下发的任务为硬件操作，一次硬件操作可能是非原子性的，即涉及多次操作的组合。示例性的，VM在热迁移时虚拟功能驱动可能正在操作硬件，然而完成一次硬件操作是多次寄存器或硬件交互的组合，该硬件操作为非原子性的，例如：VF驱动向PF驱动发送Mailbox消息时，先写Header寄存器，再写数据寄存器，最后写request寄存器触发VF硬件开始读取头和数据发到目的PF，PF把处理结果返回给VF的异步事件队列(asynchronous event queue，AEQ)，VFD通过中断开始处理到AEQ中的结果。VM迁移时被暂停在上述任何一个过程中，都会导致迁移到目的服务器时恢复出不同的状态，可能出现Mailbox消息丢失，或者发送了一个非法的Mailbox给目的端的PF等异常情况。
可选的，源服务器锁定并排空第一任务队列可称为pre_deactivate流程。
在第一虚拟机的迁移过程中，若用户通过该第一虚拟机下发用户命令改变对应的虚拟网卡的状态信息，将可能发生在命令执行完成之前，第一虚拟机对应的虚拟网卡状态信息已经迁移到目标虚拟机的情况，有可能产生命令超时等异常，导致业务中断。
为解决这一问题，本申请实施例中，热迁移模块将执行用户命令的通道锁定，以使得该通道无法接收用户下发的新任务。执行用户命令的通道被锁定后，用户将无法修改第一虚拟机对应的虚拟网卡的状态信息，但用户业务仍可持续进行。
该第一虚拟机运行的第一虚拟功能驱动排空执行用户命令的通道，是指将锁定队列时已经存在的用户命令执行完成，将队列排空，排空完成代表已下发的用户命令执行完成。热迁移模块可以获取该执行用户命令的通道排空的信息。
需要说明的是，执行用户命令的通道的状态属于待迁移的第一虚拟网卡的状态信息，源服务器将执行用户命令的通道锁定后，该锁定状态将迁移至目的服务器，目的服务器恢复该状态信息后，可以解锁该执行用户命令的通道。
为了进一步介绍源服务器锁定并排空第一任务队列的实现过程，请参阅图6，本申请实施例中锁定并排空任务队列的一个实施例示意图。
一、热迁移模块侧：
611、热迁移模块设置任务队列为锁定状态。
热迁移模块可以在获取第一虚拟网卡的状态信息之前，设置任务队列为锁定状态，可选的，在热迁移模块执行最后一次内存拷贝前，设置任务队列为锁定状态。
需要说明的是，该任务队列的锁定状态可以迁移至目的服务器侧，并在目的服务器侧解除锁定状态，用于继续执行用户命令。
612、热迁移模块判断任务队列是否清空，若是，则执行步骤613，若否，则重复执行步骤612。
613、若热迁移模块判断任务队列清空，则执行其他操作。
若任务队列清空，则热迁移模块可执行其他操作，例如，上报虚拟机管理器，具体此处 不做限定。
二、虚拟功能驱动侧：
621、虚拟功能驱动获取用户下发的任务。
622、虚拟功能驱动判断任务队列是否处于锁定状态。
623、若虚拟功能驱动任务队列不处于锁定状态，则虚拟功能驱动将该任务加入任务队列。
624、若虚拟功能驱动任务队列处于锁定状态，则将该任务加入任务队列。
若热迁移模块设置任务队列为锁定状态，此后，该任务队列不能再接收新的任务，若用户下发了新的任务，则虚拟功能驱动将该任务置于等待通道中。当虚拟机迁移完成后，在目的服务器侧，任务队列的锁定状态解除后，等待通道中的任务可以被加入任务队列继续执行。
625、虚拟功能驱动执行任务队列中的任务。
626、虚拟功能驱动将任务队列清空。
在具体的实现过程中，可以通过设置busy_state参数、own参数等实现，其中，热迁移模块锁定任务队列时，设置busy_state为1，任务队列中不再加新任务，解除锁定时busy_state为0，可以继续接收任务。虚拟功能驱动处理任务时own为1，任务清空时own为0，func_ref为待执行命令数。
源端VM pause前，通过pre_deactivate流程确保VFD消息通道suspend，pre_deactivate返回则表示所有下发的消息均已处理完成，此时VM可pause。目的端VM resume前，通过PFD clear busy_state。然后当目的VM resume后，suspend的通道在检测到通道不处于busy状态后恢复，继续下发源端VFD积压的消息。迁移源端流程中：Pre_deactivate流程中设置busy_state为1，然后检查Own，直到Own为0退出；在VFD Uninterrupt flow流程中，busy_state状态为1，则执行消息处理，直到func_ref为0后clear own bit，然后一直等待busy_state状态变为0。
虚拟功能驱动执行任务队列中的任务，直至任务队列中的任务被清空。热迁移模块可以获取该任务队列的任务是否清空。
需要说明的是，步骤504可以在步骤501至步骤505之间任一时刻执行，由于锁定执行用户命令的通道后，将不再响应新的用户命令。可选的，在步骤503之后，当迭代拷贝内存结束，剩余内存可一次拷贝完成时(log_sync时刻)执行步骤504，可以减少不响应用户命令阶段的时长。
505、源服务器运行的第一虚拟机停机；
源服务器确认执行用户命令的通道排空之后，虚拟机管理器调用虚拟机停机流程，挂起第一虚拟机。第一虚拟网卡停用，业务中断，第一虚拟机停机。
506、源服务器向目的服务器执行最后一次内存拷贝；
第一虚拟机停机之后，虚拟机管理器将执行最后一次内存拷贝，将步骤503中剩余的未拷贝内存迁移至目的服务器。
507、源服务器获取第一虚拟网卡的状态信息；
虚拟机管理器获取第一虚拟网卡的状态信息，并将该状态信息拷贝到目的服务器的虚拟机管理器中。
虚拟机管理器通过热迁移模块获取该第一虚拟机对应的第一虚拟网卡的状态信息。具体 地，热迁移模块与管理单元、数据处理单元协同工作，由虚拟机管理器分配的第一内存空间，热迁移模块根据状态信息存储位置分别从网卡的BAR寄存器、微处理器和微码中获取第一虚拟网卡的状态信息，并将该状态信息保存到该第一内存空间。需要说明的是，该第一内存空间不属于第一虚拟机对应的内存空间，由虚拟机管理器直接分配控制。
508、源服务器将虚拟网卡的状态信息发送至目的服务器进行恢复；
源服务器的虚拟机管理器将存储在该第一内存空间的该状态信息拷贝到目的服务器的虚拟机管理器。
目的服务器的虚拟机管理器接收到该状态信息后，需要将该状态信息分别恢复至SR-IOV网卡的BAR寄存器、微码和微处理器中。
从源服务器的微处理器和微码获取的该状态信息可以直接写入目的服务器的微处理器和微码中；
由于BAR寄存器中包括BAR只读寄存器和BAR可写寄存器，从源端SR-IOV网卡BAR寄存器获取的该状态信息，恢复时需分两种情况进行处理，下面分别进行介绍：
1、从BAR可写寄存器中拷贝的该状态信息可以直接写入目的服务器的BAR可写寄存器；
2、从BAR只读寄存器中拷贝的该状态信息需要通过微处理器进行恢复，写入目的服务器SR-IOV网卡BAR只读寄存器中。示例性的，例如报文发送通知(Doorbell)和配置控制通知(Mailbox Req)寄存器不能通过源端读取目的端写入的方式来恢复值，因此驻留在BAR只读寄存器的芯片状态不能直接借助QEMU拷贝MMIO空间的方式来迁移。
可选的，该虚拟机管理器接收所述源服务器发送的第二信息，该第二信息为第二虚拟机对应的第二虚拟网卡的任务队列的锁定状态信息，该第二虚拟网卡也是基于SR-IOV网卡虚拟出的，该第二任务队列用于放置为所述第二虚拟机下发的任务；所述热迁移模块解锁所述第二任务队列。
具体地，与步骤504对应地，目的服务器在启用第二虚拟机之前可将任务队列解锁，例如，通过设置busy_state为0，将任务队列的锁定状态解锁。
509、目的服务器启用第二虚拟机；
第一虚拟网卡的状态信息迁移到目的服务器后，目的端虚拟机管理器将锁定的执行用户命令的通道打开，由此可以正常执行用户命令。
目的服务器的虚拟机管理器可以使能网卡，启用第二虚拟机。可选的，虚拟机管理器通过activate接口使能网卡。
510、目的服务器发送广播报文，恢复通信；
为缩短中断时延，目的服务器的虚拟机管理器发送广播报文，使得通信快速恢复。
可选的，虚拟机管理器通过post_resume接口发送广播报文。
需要说明的是，步骤510为可选步骤，可以执行，也可以不执行，此处不做限定。
本申请实施例中通过源服务器通过新增的热迁移模块，可以获取网卡中的第一虚拟网卡的状态信息，并将该状态信息拷贝至虚拟机管理器分配的内存中，进而发送至第二虚拟机。目的服务器接收源服务器发送的第一虚拟网卡的状态信息，可以通过新增的热迁移模块，实现该状态信息在SR-IOV网卡中的恢复，该虚拟机迁移系统可以实现第一虚拟网卡的状态信息迁移，使得第一虚拟机的成功迁移。
上面介绍了本申请实施例提供的虚拟机热迁移方法，下面对实现该方法的通信设备进行介绍，请参阅图7，为本申请实施例中源端通信设备的一个实施例示意图。
本申请实施例提供了一种通信设备，可选的，该通信设备为虚拟机热迁移的源服务器，该源服务器中运行有第一虚拟机、该第一虚拟机使用的第一虚拟网卡、虚拟机管理器701和热迁移模块702，该第一虚拟网卡是基于SR-IOV网卡虚拟出的，包括：
该虚拟机管理器701，用于向该热迁移模块702下发热迁移命令；
该热迁移模块702，用于根据该热迁移命令获取该第一虚拟机需迁移的信息，该第一虚拟机需迁移的信息包括该第一虚拟网卡的状态信息以及该第一虚拟机的内存数据，并上报该虚拟机管理器701；
该虚拟机管理器701，还用于向该目的服务器发送该状态信息以及该第一虚拟机的内存数据。
该热迁移模块702具体用于：根据该热迁移命令，将该状态信息拷贝至该虚拟机管理器701分配的该第一内存空间，该第一内存空间不包括在给该第一虚拟机分配的虚拟内存内。
该热迁移模块702具体用于：从该SR-IOV网卡的基地址寄存器、微处理器和微码中拷贝该状态信息至该第一内存空间。
该热迁移模块702，还用于锁定该第一虚拟网卡的第一任务队列，以使得该第一任务队列无法接收新的任务，该第一任务队列用于放置为该第一虚拟机下发的任务；该服务器还包括：该第一虚拟机的第一虚拟功能驱动，用于清空该第一任务队列中所有已有的任务。
该热迁移模块702还用于：对该内存数据进行迭代拷贝；在剩余的该内存数据可一次拷贝完成的情况下，该第一虚拟功能驱动清空该第一任务队列中所有已有的任务之后，该虚拟机管理器701还用于，挂起该第一虚拟机；该热迁移模块702，还用于执行最后一次内存数据拷贝。
请参阅图8，为本申请实施例中目的端通信设备的一个实施例示意图。
本申请实施例提供一种通信设备，可选的，该通信设备为虚拟机热迁移的目的服务器，该目的服务器包括SR-IOV网卡，该目的服务器中运行有虚拟机管理器801和热迁移模块802，该服务器包括：该虚拟机管理器801，用于获取热迁移命令，该热迁移命令用于指示源服务器的第一虚拟机将将迁移至该目的服务器的第二虚拟机；该虚拟机管理器801，还用于接收该源服务器发送的该第一虚拟机对应的第一虚拟网卡的状态信息；该热迁移模块802，还用于根据该虚拟机管理器801的指示，将该状态信息恢复至该SR-IOV网卡。
该热迁移模块802具体用于：将该状态信息分别恢复至该SR-IOV网卡的基地址寄存器、微码和微处理器中。
该热迁移模块802具体用于：将该状态信息中的第一信息拷贝至该微处理器，该第一信息为从该源服务器的只读基地址寄存器中获取的信息，该微处理器用于将该第一信息恢复至该SR-IOV网卡的只读基地址寄存器。
该虚拟机管理器801还用于：通过该热迁移模块802分配该第二虚拟机对应的第二虚拟网卡的标识信息。
该虚拟机管理器801具体用于：接收该源服务器发送的第二信息，该第二信息为该第二虚拟机对应的第二虚拟网卡的第二任务队列的锁定状态信息，该第二虚拟网卡是基于该SR- IOV网卡虚拟出的，该第二任务队列用于放置为该第二虚拟机下发的任务；该热迁移模块802，还用于解锁该第二任务队列。
请参阅图9，为本申请实施例中一种通信设备的另一个实施例示意图。本实施例提供的通信设备为虚拟机热迁移的源服务器，本申请实施例中对其具体设备形态不做限定。
该源端通信设备900可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上处理器901和存储器902，该存储器902中存储有程序或数据。
其中，存储器902可以是易失性存储或非易失性存储。可选地，处理器901是一个或多个中央处理器(CPU，Central Processing Unit，该CPU可以是单核CPU，也可以是多核CPU。处理器901可以与存储器902通信，在源端通信设备900上执行存储器902中的一系列指令。
该源端通信设备900还包括物理网卡903，该物理网卡903为PCIE设备，即SR-IOV网卡。
可选地，尽管图9中未示出，源端通信设备900还可以包括一个或一个以上电源；一个或一个以上有线或无线网络接口，例如以太网接口；一个或一个以上输入输出接口，输入输出接口可以用于连接显示器、鼠标、键盘、触摸屏设备或传感设备等，输入输出接口为可选部件，可以存在也可以不存在，此处不做限定。
本实施例中源端通信设备900中的处理器901所执行的流程可以参考前述方法实施例中描述的方法流程，此处不加赘述。
请参阅图10，为本申请实施例中一种通信设备的另一个实施例示意图。本实施例提供的通信设备为虚拟机热迁移的目的服务器，本申请实施例中对其具体设备形态不做限定。
该目的端通信设备1000可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上处理器1001和存储器1002，该存储器1002中存储有程序或数据。
其中，存储器1002可以是易失性存储或非易失性存储。可选地，处理器1001是一个或多个中央处理器(CPU，Central Processing Unit，该CPU可以是单核CPU，也可以是多核CPU。处理器1001可以与存储器1002通信，在目的端通信设备1000上执行存储器1002中的一系列指令。
该目的端通信设备1000还包括物理网卡1003，该物理网卡1003为PCIE设备，即SR-IOV网卡。
可选地，尽管图10中未示出，目的端通信设备1000还可以包括一个或一个以上电源；一个或一个以上有线或无线网络接口，例如以太网接口；一个或一个以上输入输出接口，输入输出接口可以用于连接显示器、鼠标、键盘、触摸屏设备或传感设备等，输入输出接口为可选部件，可以存在也可以不存在，此处不做限定。
本实施例中目的端通信设备1000中的处理器1001所执行的流程可以参考前述方法实施例中描述的方法流程，此处不加赘述。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结 合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (25)

  1. 一种虚拟机热迁移方法，其特征在于，应用于运行有第一虚拟机的源服务器，所述方法用于向目的服务器迁移所述第一虚拟机，所述源服务器包括单根输入/输出虚拟化SR-IOV网卡，所述源服务器中运行有第一虚拟机、所述第一虚拟机使用的第一虚拟网卡、虚拟机管理器和热迁移模块，所述第一虚拟网卡是基于所述SR-IOV网卡虚拟出的，所述方法包括：
    所述虚拟机管理器向所述热迁移模块下发热迁移命令；
    所述热迁移模块根据所述热迁移命令获取所述第一虚拟机需迁移的信息，所述第一虚拟机需迁移的信息包括所述第一虚拟网卡的状态信息以及所述第一虚拟机的内存数据，并上报所述虚拟机管理器；
    所述虚拟机管理器向所述目的服务器发送所述状态信息以及所述第一虚拟机的内存数据。
  2. 根据权利要求1所述的方法，其特征在于，所述热迁移模块根据所述热迁移命令获取所述第一虚拟机需迁移的信息包括：
    所述热迁移模块根据所述热迁移命令，将所述状态信息拷贝至所述虚拟机管理器分配的所述第一内存空间，所述第一内存空间不包括在给所述第一虚拟机分配的虚拟内存内。
  3. 根据权利要求2所述的方法，其特征在于，所述热迁移模块根据所述热迁移命令，将所述状态信息拷贝至所述虚拟机管理器分配的所述第一内存空间包括：
    所述热迁移模块从所述SR-IOV网卡的基地址寄存器、微处理器和微码中拷贝所述状态信息至所述第一内存空间。
  4. 根据权利要求1至3中任一项所述的方法，其特征在于，所述热迁移模块根据所述热迁移命令获取所述第一虚拟机需迁移的信息之前，所述方法还包括：
    所述热迁移模块锁定所述第一虚拟网卡的第一任务队列，以使得所述第一任务队列无法接收新的任务，所述第一任务队列用于放置为所述第一虚拟机下发的任务；
    所述第一虚拟机的第一虚拟功能驱动清空所述第一任务队列中所有已有的任务。
  5. 根据权利要求4所述的方法，其特征在于，所述虚拟机管理器向所述热迁移模块下发热迁移命令之后，所述方法还包括：
    所述热迁移模块对所述内存数据进行迭代拷贝；
    在剩余的所述内存数据可一次拷贝完成的情况下，所述第一虚拟功能驱动清空所述第一任务队列中所有已有的任务之后，所述虚拟机管理器挂起所述第一虚拟机；
    所述热迁移模块执行最后一次内存数据拷贝。
  6. 一种虚拟机热迁移方法，其特征在于，应用于虚拟机迁移的目的服务器，所述目的服务器包括SR-IOV网卡，所述目的服务器中运行有虚拟机管理器和热迁移模块，所述方法包括：
    所述虚拟机管理器获取热迁移命令，所述热迁移命令用于指示源服务器的第一虚拟机将将迁移至所述目的服务器的第二虚拟机；
    所述虚拟机管理器接收所述源服务器发送的所述第一虚拟机对应的第一虚拟网卡的状态信息；
    所述热迁移模块根据所述虚拟机管理器的指示，将所述状态信息恢复至所述SR-IOV网卡。
  7. 根据权利要求6所述的方法，其特征在于，所述热迁移模块根据所述虚拟机管理器的指示，将所述状态信息恢复至所述SR-IOV网卡包括：
    所述热迁移模块将所述状态信息分别恢复至所述SR-IOV网卡的基地址寄存器、微码和微处理器中。
  8. 根据权利要求7所述的方法，其特征在于，所述热迁移模块将所述状态信息分别恢复至所述SR-IOV网卡的基地址寄存器、微码和微处理器中包括：
    所述热迁移模块将所述状态信息中的第一信息拷贝至所述微处理器，所述第一信息为从所述源服务器的只读基地址寄存器中获取的信息，所述微处理器用于将所述第一信息恢复至所述SR-IOV网卡的只读基地址寄存器。
  9. 根据权利要求6至8中任一项所述的方法，其特征在于，所述虚拟机管理器接收所述源服务器发送的所述第一虚拟机对应的第一虚拟网卡的状态信息之前，所述方法还包括：
    所述虚拟机管理器通过所述热迁移模块分配所述第二虚拟机对应的第二虚拟网卡的标识信息。
  10. 根据权利要求6至9中任一项所述的方法，其特征在于，所述虚拟机管理器接收所述源服务器发送的所述第一虚拟机对应的第一虚拟网卡的状态信息包括：
    所述虚拟机管理器接收所述源服务器发送的第二信息，所述第二信息为所述第二虚拟机对应的第二虚拟网卡的第二任务队列的锁定状态信息，所述第二虚拟网卡是基于所述SR-IOV网卡虚拟出的，所述第二任务队列用于放置为所述第二虚拟机下发的任务；
    所述方法还包括：所述热迁移模块解锁所述第二任务队列。
  11. 一种通信设备，其特征在于，该通信设备为虚拟机热迁移的源服务器，所述源服务器中运行有第一虚拟机、所述第一虚拟机使用的第一虚拟网卡、虚拟机管理器和热迁移模块，所述第一虚拟网卡是基于SR-IOV网卡虚拟出的，所述通信设备包括：
    所述虚拟机管理器，用于向所述热迁移模块下发热迁移命令；
    所述热迁移模块，用于根据所述热迁移命令获取所述第一虚拟机需迁移的信息，所述第一虚拟机需迁移的信息包括所述第一虚拟网卡的状态信息以及所述第一虚拟机的内存数据，并上报所述虚拟机管理器；
    所述虚拟机管理器，还用于向所述目的服务器发送所述状态信息以及所述第一虚拟机的内存数据。
  12. 根据权利要求11所述的通信设备，其特征在于，所述热迁移模块具体用于：
    根据所述热迁移命令，将所述状态信息拷贝至所述虚拟机管理器分配的所述第一内存空间，所述第一内存空间不包括在给所述第一虚拟机分配的虚拟内存内。
  13. 根据权利要求12所述的通信设备，其特征在于，所述热迁移模块具体用于：
    从所述SR-IOV网卡的基地址寄存器、微处理器和微码中拷贝所述状态信息至所述第一内存空间。
  14. 根据权利要求11至13中任一项所述的通信设备，其特征在于，
    所述热迁移模块，还用于锁定所述第一虚拟网卡的第一任务队列，以使得所述第一任务队列无法接收新的任务，所述第一任务队列用于放置为所述第一虚拟机下发的任务；
    所述通信设备还包括：
    所述第一虚拟机的第一虚拟功能驱动，用于清空所述第一任务队列中所有已有的任务。
  15. 根据权利要求14所述的通信设备，其特征在于，所述热迁移模块还用于：
    对所述内存数据进行迭代拷贝；
    在剩余的所述内存数据可一次拷贝完成的情况下，所述第一虚拟功能驱动清空所述第一任务队列中所有已有的任务之后，所述虚拟机管理器还用于，挂起所述第一虚拟机；
    所述热迁移模块，还用于执行最后一次内存数据拷贝。
  16. 一种通信设备，其特征在于，该通信设备为虚拟机热迁移的目的服务器，所述目的服务器包括SR-IOV网卡，所述目的服务器中运行有虚拟机管理器和热迁移模块，所述通信设备包括：
    所述虚拟机管理器，用于获取热迁移命令，所述热迁移命令用于指示源服务器的第一虚拟机将将迁移至所述目的服务器的第二虚拟机；
    所述虚拟机管理器，还用于接收所述源服务器发送的所述第一虚拟机对应的第一虚拟网卡的状态信息；
    所述热迁移模块，还用于根据所述虚拟机管理器的指示，将所述状态信息恢复至所述SR-IOV网卡。
  17. 根据权利要求16所述的通信设备，其特征在于，所述热迁移模块具体用于：
    将所述状态信息分别恢复至所述SR-IOV网卡的基地址寄存器、微码和微处理器中。
  18. 根据权利要求17所述的通信设备，其特征在于，所述热迁移模块具体用于：
    将所述状态信息中的第一信息拷贝至所述微处理器，所述第一信息为从所述源服务器的只读基地址寄存器中获取的信息，所述微处理器用于将所述第一信息恢复至所述SR-IOV网卡的只读基地址寄存器。
  19. 根据权利要求16至18中任一项所述的通信设备，其特征在于，所述虚拟机管理器还用于：
    通过所述热迁移模块分配所述第二虚拟机对应的第二虚拟网卡的标识信息。
  20. 根据权利要求16至19中任一项所述的通信设备，其特征在于，所述虚拟机管理器具体用于：
    接收所述源服务器发送的第二信息，所述第二信息为所述第二虚拟机对应的第二虚拟网卡的第二任务队列的锁定状态信息，所述第二虚拟网卡是基于所述SR-IOV网卡虚拟出的，所述第二任务队列用于放置为所述第二虚拟机下发的任务；
    所述热迁移模块，还用于解锁所述第二任务队列。
  21. 一种通信设备，包括：处理器、存储器和SR-IOV网卡；
    所述处理器用于基于所述存储器中的代码运行第一虚拟机、虚拟机管理器和热迁移模块，所述第一虚拟机包括第一虚拟功能驱动，所述处理器通过所述虚拟机管理器、所述热迁移模块和所述第一虚拟功能驱动执行如权利要求1至5中任一项所述的方法。
  22. 一种通信设备，包括：处理器、存储器和SR-IOV网卡；
    所述处理器用于基于所述存储器中的代码运行虚拟机管理器和热迁移模块，并通过所述虚拟机管理器和热迁移模块执行如权利要求6至10中任一项所述的方法。
  23. 一种包含指令的计算机程序产品，其特征在于，当其在计算机上运行时，使得所述计算机执行如权利要求1至10任一项所述的方法。
  24. 一种计算机可读存储介质，包括指令，其特征在于，当所述指令在计算机上运行时， 使得计算机执行如权利要求1至10中任一项所述的方法。
  25. 一种虚拟机热迁移系统，包括如权利要求11至15中任一项所述源服务器，和如权利要求16至20中任一项所述目的服务器。
PCT/CN2020/106044 2019-07-31 2020-07-31 虚拟机热迁移方法和通信设备 Ceased WO2021018267A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP20846078.2A EP3992805B1 (en) 2019-07-31 2020-07-31 Live migration method for virtual machine and communication device
US17/204,108 US12405816B2 (en) 2019-07-31 2021-03-17 Virtual machine live migration method and communications device
US19/285,364 US20260023590A1 (en) 2019-07-31 2025-07-30 Virtual machine live migration method and communications device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910704642.2 2019-07-31
CN201910704642.2A CN112306650B (zh) 2019-07-31 2019-07-31 虚拟机热迁移方法和通信设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/204,108 Continuation US12405816B2 (en) 2019-07-31 2021-03-17 Virtual machine live migration method and communications device

Publications (1)

Publication Number Publication Date
WO2021018267A1 true WO2021018267A1 (zh) 2021-02-04

Family

ID=74230300

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/106044 Ceased WO2021018267A1 (zh) 2019-07-31 2020-07-31 虚拟机热迁移方法和通信设备

Country Status (4)

Country Link
US (2) US12405816B2 (zh)
EP (1) EP3992805B1 (zh)
CN (2) CN118312284A (zh)
WO (1) WO2021018267A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116302326A (zh) * 2023-02-10 2023-06-23 济南浪潮数据技术有限公司 一种虚拟机迁移方法、系统、计算机设备及存储介质

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11561821B1 (en) * 2020-05-07 2023-01-24 Amazon Technologies, Inc. Controlling placement of virtualized resources based on desirability index for host pools
CN114416293B (zh) * 2021-12-31 2025-06-27 阿里巴巴（中国）有限公司 一种虚拟化设备的故障恢复方法和装置
CN114398153A (zh) * 2022-01-21 2022-04-26 平安科技（深圳）有限公司 虚拟机热迁移的控制方法、系统、电子设备及存储介质
CN114691300B (zh) * 2022-03-25 2026-03-31 阿里巴巴（中国）有限公司 一种虚拟机实例的热迁移方法
CN114679461B (zh) * 2022-03-30 2023-02-14 浙江大学 面向虚拟机热迁移场景的云网络会话连通性保障方法及系统
US12360794B2 (en) * 2022-04-07 2025-07-15 VMware LLC Lifecycle and recovery for virtualized DPU management operating systems
CN115599494A (zh) * 2022-09-21 2023-01-13 阿里巴巴（中国）有限公司（Cn） 虚拟机迁移方法以及装置、升级方法和服务器
DE112022007644T5 (de) * 2022-11-08 2025-07-31 Intel Corporation Adaptive Live-Migration einer virtuellen Maschine für eine physische Speicherungsvorrichtungssteuerung
US12159051B2 (en) * 2023-01-13 2024-12-03 Hitachi, Ltd. Migration of application on container platform
CN116112412B (zh) * 2023-02-28 2024-10-15 苏州浪潮智能科技有限公司 一种虚拟网卡绑定冗余功能测试方法、系统、装置及介质
US20240311168A1 (en) * 2023-03-17 2024-09-19 Vmware, Inc. Virtual machine (vm) migration with smart network interface cards (nics)
US20240370288A1 (en) * 2023-05-04 2024-11-07 Dish Wireless L.L.C. Dynamic reassignment of hardware access to virtualized systems in 5g networks
CN116737324B (zh) * 2023-08-14 2023-11-03 无锡沐创集成电路设计有限公司 硬件Virtio-net设备的热迁移方法、装置、设备及介质
KR20250050538A (ko) 2023-10-06 2025-04-15 삼성전자주식회사 스토리지 장치, 스토리지 장치의 동작 방법, 및 스토리지 시스템의 동작 방법
CN119292969B (zh) * 2024-08-30 2025-09-16 西安电子科技大学 一种支持以太网的rdma网卡单根虚拟化硬件实现方法
CN119519930B (zh) * 2024-11-15 2026-01-09 航天信息股份有限公司 云密码机数据迁移方法、装置及电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120011508A1 (en) * 2010-07-12 2012-01-12 Vmware, Inc. Multiple time granularity support for online classification of memory pages based on activity level
CN103955399A (zh) * 2014-04-30 2014-07-30 华为技术有限公司 一种虚拟机迁移方法和装置及物理主机
CN106464579A (zh) * 2014-06-20 2017-02-22 谷歌公司 用于虚拟化网络堆栈的动态迁移的系统和方法
CN108205506A (zh) * 2013-12-31 2018-06-26 华为技术有限公司 虚拟机热迁移的方法、虚拟机内存数据处理方法、服务器和虚拟机系统

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7484029B2 (en) * 2006-02-09 2009-01-27 International Business Machines Corporation Method, apparatus, and computer usable program code for migrating virtual adapters from source physical adapters to destination physical adapters
US8533713B2 (en) * 2011-03-29 2013-09-10 Intel Corporation Efficent migration of virtual functions to enable high availability and resource rebalance
WO2014032233A1 (zh) * 2012-08-29 2014-03-06 华为技术有限公司 虚拟机热迁移的系统和方法
CN103888417B (zh) * 2012-12-21 2017-05-10 中国电信股份有限公司 虚拟机系统迁移的网络切换方法与系统
US9405642B2 (en) * 2013-01-29 2016-08-02 Red Hat Israel, Ltd. Providing virtual machine migration reliability using an intermediary storage device
WO2015100622A1 (zh) 2013-12-31 2015-07-09 华为技术有限公司 虚拟机热迁移的方法和服务器
US9361145B1 (en) 2014-06-27 2016-06-07 Amazon Technologies, Inc. Virtual machine state replication using DMA write records
CN104636186B (zh) * 2015-01-27 2018-02-13 华为技术有限公司 虚拟机内存管理方法、物理主机、pcie设备及其配置方法以及迁移管理设备
US10042720B2 (en) * 2016-02-22 2018-08-07 International Business Machines Corporation Live partition mobility with I/O migration
US10002018B2 (en) * 2016-02-23 2018-06-19 International Business Machines Corporation Migrating single root I/O virtualization adapter configurations in a computing system
US9785451B1 (en) * 2016-10-21 2017-10-10 International Business Machines Corporation Migrating MMIO from a source I/O adapter of a computing system to a destination I/O adapter of the computing system
US11556363B2 (en) * 2017-03-31 2023-01-17 Intel Corporation Techniques for virtual machine transfer and resource management
US10198299B1 (en) 2017-10-30 2019-02-05 Vmware, Inc. Enabling live migration of virtual machines with passthrough PCI devices
TWI645293B (zh) * 2017-11-30 2018-12-21 財團法人工業技術研究院 在多根輸入輸出虛擬化環境中虛擬機器即時遷移的方法、系統及其計算主機
CN108874506B (zh) 2018-06-08 2020-07-24 北京百度网讯科技有限公司 虚拟机直通设备的热迁移方法和装置
US10782754B2 (en) * 2018-09-21 2020-09-22 Quanta Computer Inc. Thermal management via virtual BMC manager

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120011508A1 (en) * 2010-07-12 2012-01-12 Vmware, Inc. Multiple time granularity support for online classification of memory pages based on activity level
CN108205506A (zh) * 2013-12-31 2018-06-26 华为技术有限公司 虚拟机热迁移的方法、虚拟机内存数据处理方法、服务器和虚拟机系统
CN103955399A (zh) * 2014-04-30 2014-07-30 华为技术有限公司 一种虚拟机迁移方法和装置及物理主机
CN106464579A (zh) * 2014-06-20 2017-02-22 谷歌公司 用于虚拟化网络堆栈的动态迁移的系统和方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3992805A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116302326A (zh) * 2023-02-10 2023-06-23 济南浪潮数据技术有限公司 一种虚拟机迁移方法、系统、计算机设备及存储介质

Also Published As

Publication number Publication date
CN112306650B (zh) 2024-03-26
EP3992805A1 (en) 2022-05-04
EP3992805B1 (en) 2024-06-19
CN118312284A (zh) 2024-07-09
US20210200573A1 (en) 2021-07-01
CN112306650A (zh) 2021-02-02
EP3992805A4 (en) 2022-08-31
US20260023590A1 (en) 2026-01-22
US12405816B2 (en) 2025-09-02

Similar Documents

Publication Publication Date Title
US12405816B2 (en) Virtual machine live migration method and communications device
EP4050477B1 (en) Virtual machine migration techniques
KR102140730B1 (ko) Gpu 기반의 딥러닝 개발 환경 제공 시스템 및 방법
US9317314B2 (en) Techniques for migrating a virtual machine using shared storage
US9760408B2 (en) Distributed I/O operations performed in a continuous computing fabric environment
US8381002B2 (en) Transparently increasing power savings in a power management environment
CN107479943B (zh) 基于工业互联网操作系统的多操作系统运行方法及装置
US20150205542A1 (en) Virtual machine migration in shared storage environment
US20160092261A1 (en) Method and system for physical computer system virtualization
KR20070100367A (ko) 하나의 가상 머신에서 다른 가상 머신으로 메모리를동적으로 재할당하기 위한 방법, 장치 및 시스템
CN101493781A (zh) 一种虚拟机系统及其启动方法
CN108027747A (zh) 在存储阵列系统中执行为单处理器环境设计的控制器固件的多处理
CN111213127B (zh) 用于直接分配的设备的虚拟化操作
TW200817920A (en) Method, apparatus, and computer usable program code for migrating virtual adapters from source physical adapters to destination physical adapters
CN111679889B (zh) 一种虚拟机的转换迁移方法和系统
CN107111530B (zh) 一种容灾方法、系统和装置
CN102609324B (zh) 一种虚拟机死锁后的恢复方法、装置及系统
CN114115703B (zh) 裸金属服务器在线迁移方法以及系统
CN103810015A (zh) 虚拟机创建方法和设备
CN106777394B (zh) 一种集群文件系统
CN106815067B (zh) 带i/o虚拟化的虚拟机在线迁移方法、装置
CN108762891A (zh) 一种云平台资源调度方法和装置
US10152341B2 (en) Hyper-threading based host-guest communication
US20250036438A1 (en) System and method for enabling operations for virtual computing instances with physical passthru devices
US20250315286A1 (en) Transparently servicing a host compute layer of a virtual machine

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20846078

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020846078

Country of ref document: EP

Effective date: 20210301