CN115834603A - Data synchronization method and device, storage medium and processor - Google Patents

Abstract

The application discloses a data synchronization method and device, a storage medium and a processor. The method includes: obtaining a plurality of target data and a plurality of first transfer nodes through a second transfer node; determining the quantity of the plurality of target data and the efficiency of processing data of each first transfer node through the second transfer node; The node allocates each target data to each first transfer node according to the quantity of multiple target data and the efficiency of processing data by each first transfer node, so that each first transfer node processes each target data, The processed multiple target data are obtained, and the processed multiple target data are synchronized to the destination server. Through this application, the method of statically binding the source server in the cluster environment and the transfer node for processing data in the related technology is solved, and the data of the source server in the cluster environment is synchronized to the destination server, resulting in poor data synchronization effect The problem.

Description

Data synchronization method and device, storage medium and processor

technical field

The present application relates to the field of computer technology, and in particular, to a data synchronization method and device, a storage medium, and a processor.

Background technique

Cluster (cluster) technology is a relatively new technology. Through cluster technology, relatively high benefits in terms of performance, reliability, and flexibility can be obtained at a lower cost. Its task scheduling is a cluster system core technology in.

Moreover, some cluster-deployed servers usually need to synchronize data to the target network. Because the interfaces provided by the data source server and the destination server are incompatible, or the required data formats are inconsistent, the transfer node needs to process the data before synchronizing. The transfer node here can be a large server, or a data processing software installed on a PC (Personal Computer, personal computer). Three application scenarios are listed below:

(1) In the audit and control of network security, security vendors often deploy proxy software in the internal network to synchronize users from authentication servers deployed in clusters, and the proxy software forwards user data to network security devices to help network security devices Complete real-name audit and control of traffic.

(2) In a complex audit system, the logs of various devices are usually different. In order to aggregate the logs to the log platform for unified management, a transfer node can be set up to process and format the logs uniformly.

(3) In a small network, server 1 deployed in a cluster stores resources accessible to users, server 2 is used to reverse proxy user requests and provide resources for users, and a transfer node is set between server 1 and server 2 to transfer data Decoding, classification, formatting and other processing operations.

However, the above-mentioned transit nodes may have limited ability to process data and relatively limited stability. In order to ensure synchronization efficiency and stability, we usually need to perform load balancing and redundant deployment, use multiple transfer nodes to connect to multiple source servers in the cluster environment, and process and forward data to the destination server.

In related technologies, data is synchronized in a one-to-many manner, that is, a transit node is used to connect to multiple source servers in a cluster environment, and the data of the source server is synchronized to the target network. However, the disadvantage of this method is that the processing capacity of a single transit node is insufficient, or it does not have stability, which may cause congestion or even complete paralysis of the data synchronization node.

In addition, in related technologies, a simple N-to-N method is also used to synchronize data, that is, there are multiple transit nodes, each transit node is bound to a source server, and the data of the source server is synchronized to the target network. However, the disadvantage of this method is that due to the binding relationship, it is inconvenient to deploy redundantly. In order to prevent a transit node from being paralyzed, a backup redundant node must be set for it. When the number of nodes is large, the synchronization will be interrupted or partially interrupted. In addition, due to the amount of data that each source server needs to synchronize, the actual carrying capacity of each transfer node is different, and the static binding relationship will make the distribution of synchronization resources uneven. Evenly, some transit nodes are too congested, and some transit nodes are too idle.

Aiming at the problem that the source server in the cluster environment is statically bound to the transfer node used to process data in the related technology, and the data of the source server in the cluster environment is synchronized to the destination server, resulting in poor data synchronization effect, there is currently no Come up with effective solutions.

Contents of the invention

The main purpose of this application is to provide a data synchronization method and device, storage medium and processor to solve the problem of statically binding the source server in the cluster In the environment, the data of the source server is synchronized to the destination server, resulting in poor data synchronization effect.

In order to achieve the above purpose, according to one aspect of the present application, a method for synchronizing data is provided. The method is applied in a data synchronization system, and the data synchronization system includes at least: a plurality of first transfer nodes and a second transfer node, the first transfer nodes are used for processing data, and the second transfer nodes are used for scheduling For the multiple first transit nodes, if there is a faulty node among the multiple first transit nodes, use a transit node other than the faulty node among the first transit nodes to replace the faulty node, if the first transitive node If the second transit node fails, using any one of the plurality of first transit nodes to replace the second transit node includes: acquiring multiple target data and the multiple first transit nodes through the second transit node A transfer node, wherein each target data is the data to be synchronized from multiple source servers to the destination server in the cluster environment; the quantity of the multiple target data and the processing of each first transfer node are determined by the second transfer node Data efficiency: through the second transit node, each target data is assigned to each first transit node according to the quantity of the plurality of target data and the efficiency of processing data by each first transit node, so that each The first transfer node processes each target data, obtains multiple processed target data, and synchronizes the processed multiple target data to the destination server.

Further, assigning each target data to each first transit node by the second transit node according to the quantity of the plurality of target data and the data processing efficiency of each first transit node includes: according to the multiple The quantity of target data and the efficiency of each first transit node to process data are calculated to obtain the amount of data allocated to each first transit node; based on the amount of data allocated to each first transit node, each target data is allocated for each first transit node.

Further, before the second transit node determines the quantity of the plurality of target data and the data processing efficiency of each first transit node, the method further includes: determining a plurality of target transit nodes, wherein the The plurality of target transit nodes at least include the plurality of first transit nodes; determining the second transit node from the plurality of target transit nodes, so as to determine the plurality of target data through the second transit node The quantity and the data processing efficiency of each first transit node, wherein the second transit node is a transit node in the plurality of target transit nodes except the plurality of first transit nodes.

Further, after the second transit node allocates each target data to each first transit node according to the quantity of the plurality of target data and the data processing efficiency of each first transit node, the method It also includes: judging whether there is a faulty node in the plurality of first transit nodes; if there is a faulty node in the plurality of first transit nodes, then determining from the plurality of first transit nodes other than the faulty node A plurality of third transfer nodes, wherein the third transfer node is used to replace the first transfer node to process the plurality of target data; obtain the data processing efficiency of each third transfer node; according to the multiple The quantity of each target data and the efficiency of each third transit node to process data, assign each target data to each third transit node.

Further, after the second transit node is determined from the plurality of target transit nodes, the method further includes: judging whether the second transit node fails; if the second transit node fails, then Determining a fourth transit node from the plurality of first transit nodes, wherein the fourth transit node is used to replace the second transit node in scheduling the plurality of first transit nodes except the fourth transit node other transit nodes.

Further, assigning each target data to each first transit node by the second transit node according to the quantity of the plurality of target data and the data processing efficiency of each first transit node includes: judging the multiple Whether the data volume of each target data is greater than the preset data volume; if the data volume of the multiple target data is greater than the preset data volume, then through the second transit node according to the number of the multiple target data and each the efficiency of data processing by a first transit node, assign each target data to each first transit node, so that the first transit node obtains the plurality of target data from the plurality of source servers, and assigns each target data to each first transit node. Processing the plurality of target data; if the data volume of the multiple target data is not greater than the preset data volume, acquiring the multiple target data from the multiple source servers through the second transit node Data: assigning each target data obtained from the multiple source servers to each first transit node according to the quantity of the multiple target data and the data processing efficiency of each first transit node.

Further, determining the second transit node from the multiple target transit nodes includes: acquiring number information corresponding to each target transit node; and determining the multiple target transit nodes based on the number information corresponding to each target transit node The transit node with the smallest number among the nodes; the transit node with the smallest number among the plurality of target transit nodes is used as the second transit node.

Further, the method further includes: acquiring the amount of data to be synchronized from each source server to the destination server within unit time; calculating the amount of data for each source server to be synchronized to the destination server within unit time; An average queue length of a transit node, wherein the average queue length is used to represent the accumulation of the plurality of target data within the first time period; determine the real-time queue length of each first transit node, wherein the The real-time queue length is used to represent the load pressure situation of each first transit node in the second time period, the length of the second time period is less than the length of the first time period; based on the average of each first transit node The queue length and the real-time queue length send reminder information to the target object, wherein the reminder information is used to remind the target object to adjust a strategy for synchronizing the plurality of target data to the destination server.

Further, after acquiring multiple target data and multiple first transit nodes, the method further includes: judging whether a fifth transit node among the multiple first transit nodes is in an idle state; if the multiple first transit nodes a fifth transit node among the transit nodes is in an idle state, allocating the first data among the plurality of target data to the fifth transit node, so that the fifth transit node processes the first data, Obtaining the processed first data, and synchronizing the processed first data to the destination server; if the fifth transit node among the multiple first transit nodes is not Whether the sixth transit node among the first transit nodes is in an idle state; if the sixth transit node among the plurality of first transit nodes is in an idle state, then distribute the first data among the plurality of target data to the A sixth transit node, so that the sixth transit node processes the first data to obtain processed first data, and synchronizes the processed first data to the destination server.

In order to achieve the above purpose, according to another aspect of the present application, a data synchronization device is provided. The device is applied in a data synchronization system, and the data synchronization system at least includes: a plurality of first transfer nodes and a second transfer node, the first transfer nodes are used for processing data, and the second transfer nodes are used for scheduling For the multiple first transit nodes, if there is a faulty node among the multiple first transit nodes, use a transit node other than the faulty node among the first transit nodes to replace the faulty node, if the first transitive node If the second transit node fails, use any one of the multiple first transit nodes to replace the second transit node, including: a first acquisition module, configured to acquire multiple targets through the second transit node data and the plurality of first transfer nodes, wherein each target data is data to be synchronized from multiple source servers to a destination server in a cluster environment; a first determination module is configured to determine the The quantity of multiple target data and the data processing efficiency of each first transit node; the first processing module is configured to process data according to the quantity of multiple target data and each first transit node through the second transit node efficiency, assigning each target data to each first transit node, so that each first transit node processes each target data, obtains a plurality of processed target data, and distributes the processed multi-target data The target data is synchronized to the target server.

Further, the first processing module includes: a first computing unit, configured to calculate and obtain the value assigned to each first transit node according to the quantity of the plurality of target data and the efficiency of processing data by each first transit node. Data volume; a first allocation unit configured to allocate each target data to each first transit node based on the data volume allocated to each first transit node.

Further, the device further includes: a second determination module, configured to determine the multiple targets before determining the quantity of the multiple target data and the data processing efficiency of each first transit node through the second transit node A transit node, wherein the plurality of target transit nodes include at least the plurality of first transit nodes; a third determining module, configured to determine the second transit node from the plurality of target transit nodes, to pass The second transit node determines the quantity of the plurality of target data and the data processing efficiency of each first transit node, wherein the second transit node is one of the plurality of target transit nodes except the plurality of first transit nodes A transit node other than a transit node.

Further, the device further includes: a first judging module, configured to convert each target data to After being assigned to each first transit node, it is judged whether there is a faulty node in the plurality of first transit nodes; the fourth determining module is configured to, if there is a faulty node in the plurality of first transit nodes, then from the Determining a plurality of third transit nodes other than the faulty node among the plurality of first transit nodes, wherein the third transit node is used to replace the first transit node to process the plurality of target data; the second An acquisition module, configured to acquire the efficiency of data processing by each third transfer node; a first distribution module, configured to assign each target data to Assigned to each third transit node.

Further, the device further includes: a second judging module, configured to judge whether the second transit node fails after determining the second transit node from the plurality of target transit nodes; a fifth determining module , used to determine a fourth transit node from the plurality of first transit nodes if the second transit node fails, wherein the fourth transit node is used to replace the second transit node to schedule the A transit node other than the fourth transit node among the plurality of first transit nodes.

Further, the first processing module includes: a first judging unit, configured to judge whether the data volume of the plurality of target data is greater than a preset data volume; a second allocation unit, configured to determine whether the data volume of the multiple target data If the amount of data is greater than the preset amount of data, then the second transit node distributes each target data to each first a transfer node, so that the first transfer node obtains the multiple target data from the multiple source servers, and processes the multiple target data; the first obtaining unit is configured to if the multiple If the data volume of the target data is not greater than the preset data volume, the multiple target data are obtained from the multiple source servers through the second transit node; the third allocation unit is configured to The quantity of target data and the data processing efficiency of each first transit node, assigning each target data obtained from the plurality of source servers to each first transit node.

Further, the third determining module includes: a second acquiring unit, configured to acquire number information corresponding to each target transit node; a first determining unit, configured to determine the number information corresponding to each target transit node A transit node with the smallest number among the multiple target transit nodes; a second determination unit configured to use the transit node with the smallest number among the multiple target transit nodes as the second transit node.

Further, the device further includes: a third acquisition module, configured to acquire the amount of data to be synchronized to the destination server by each source server within a unit time; The amount of data to be synchronized to the destination server is calculated to obtain the average queue length of each first transit node, wherein the average queue length is used to represent the accumulation of the plurality of target data within the first time period; sixth A determining module, configured to determine the real-time queue length of each first transit node, wherein the real-time queue length is used to represent the load pressure situation of each first transit node within a second time period, and the second time period The length is less than the length of the first time period; the first sending module is configured to send reminder information to the target object based on the average queue length and real-time queue length of each first transit node, wherein the reminder information is used for Reminding the target object to adjust a strategy for synchronizing the plurality of target data to the target server.

Further, the device further includes: a third judging module, configured to judge whether a fifth transit node among the plurality of first transit nodes is in an idle state after acquiring a plurality of target data and a plurality of first transit nodes a second allocation module, configured to allocate the first data among the plurality of target data to the fifth transit node if the fifth transit node among the plurality of first transit nodes is in an idle state, so that all The fifth transit node processes the first data to obtain the processed first data, and synchronizes the processed first data to the destination server; the fourth judging module is configured to If the fifth transit node among the first transit nodes is not in an idle state, it is judged whether the sixth transit node among the plurality of first transit nodes is in an idle state; A sixth transit node among the transit nodes is in an idle state, allocating the first data among the plurality of target data to the sixth transit node, so that the sixth transit node processes the first data, The processed first data is obtained, and the processed first data is synchronized to the destination server.

In order to achieve the above object, according to another aspect of the present application, a computer-readable storage medium is provided, the storage medium stores a program, wherein the program executes the data synchronization method described in any one of the above.

In order to achieve the above object, according to another aspect of the present application, a processor is provided, the processor is used to run a program, wherein, when the program is running, the method for synchronizing data described in any one of the above items is executed.

Through this application, the following steps are adopted: obtain a plurality of target data and a plurality of first transfer nodes through the second transfer node, wherein each target data is data to be synchronized from multiple source servers to the destination server in the cluster environment; through the second transfer node The second transit node determines the quantity of multiple target data and the data processing efficiency of each first transit node; through the second transit node, each target The data is distributed to each first transit node, so that each first transit node processes each target data, obtains multiple processed target data, and synchronizes the processed multiple target data to the destination server, solving The method of statically binding the source server in the cluster environment and the transfer node for processing data in the related technology is used to synchronize the data of the source server in the cluster environment to the destination server, resulting in poor data synchronization effect. According to the determined number of target data and the data processing efficiency of each first transit node, the second transit node assigns each target data to each first transit node, so that each first transit node The target data is processed to obtain multiple target data after processing, and the multiple target data after processing are synchronized to the destination server, thereby improving the efficiency and stability of data synchronization, and further improving the effect of data synchronization.

Description of drawings

The drawings constituting a part of the application are used to provide further understanding of the application, and the schematic embodiments and descriptions of the application are used to explain the application, and do not constitute an improper limitation to the application. In the attached picture:

Fig. 1 is a flowchart of a data synchronization method provided according to an embodiment of the present application;

Fig. 2 is a schematic diagram of deploying a data synchronization device according to an embodiment of the present application;

Fig. 3 is a schematic diagram of scheduling nodes performing task data forwarding when the data to be processed occupies a large amount of bandwidth in the embodiment of the present application;

Fig. 4 is a schematic diagram of scheduling nodes performing task data forwarding when the data to be processed does not need to occupy a large amount of bandwidth in the embodiment of the present application;

FIG. 5 is a flow chart 1 of a data synchronization method provided according to an embodiment of the present application;

FIG. 6 is a flowchart of an optional data synchronization method provided according to an embodiment of the present application;

Fig. 7 is a schematic diagram of an apparatus for synchronizing data according to an embodiment of the present application.

Detailed ways

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is an embodiment of a part of the application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

It should be noted that the terms "first" and "second" in the description and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It should be understood that the data so used may be interchanged under appropriate circumstances for the embodiments of the application described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Example 1

The present invention will be described below in conjunction with preferred implementation steps. Fig. 1 is a flow chart of the data synchronization method provided according to the embodiment of the application. As shown in Fig. 1, the method is applied in a data synchronization system, and in the data synchronization system At least include: a plurality of first transfer nodes and a second transfer node, the first transfer nodes are used to process data, the second transfer nodes are used to schedule the plurality of first transfer nodes, if the plurality of first transfer nodes If there is a faulty node in a transit node, use a transit node other than the faulty node in the first transit node to replace the faulty node, and if the second transit node fails, use the multiple first transit nodes Any one of the transit nodes in the nodes replaces the second transit node, and includes the following steps:

In step S101, a plurality of target data and a plurality of first transfer nodes are acquired through a second transfer node, wherein each target data is data to be synchronized from a plurality of source servers to a destination server in a cluster environment.

For example, Fig. 2 is a schematic diagram of deploying a data synchronization device according to the embodiment of the application. As shown in Fig. 2, the number of destination servers in the data synchronization device can be multiple, since the embodiment of the application does not focus on the destination server and each transit The binding relationship of nodes, so the destination server can be regarded as one.

For example, suppose there are m source servers and n transit nodes, and the transit nodes synchronize data from the source server, process it, and send it to the destination server.

In addition, among all transit nodes, one of the transit nodes is regarded as a scheduling node (the above-mentioned second transit node), and the scheduling node is responsible for the scheduling of all transit nodes, and other nodes are responsible for data processing, also known as data processing nodes (the above-mentioned first transit point). And for any transit node, its internal structure is composed of a network module and a multi-process data processing module. The former is responsible for sending and receiving data, and the latter is responsible for data processing. There is no difference between a scheduling node and a data processing node, but it is elected during actual operation to schedule other nodes. If necessary, any transit node can be used as a scheduling node. That is, if one or more data processing nodes fail, the scheduling node can record the failed data processing nodes, and select a non-faulty data processing node to take over the work of the failed data processing nodes. If the current scheduling node fails, a new scheduling node can be elected from the data processing nodes according to the order of numbers. For example, if the scheduling node numbered 1 fails, the node numbered 2 will be the new scheduling node. If the scheduling node fails, the node numbered 3 is the new scheduling node.

For example, the scheduling node determines the data that needs to be synchronized from multiple source servers in the cluster environment to the destination server (the aforementioned multiple target data), and determines which data processing nodes exist in the data synchronization device.

In step S102, the quantity of multiple target data and the data processing efficiency of each first transit node are determined by the second transit node.

For example, determine the amount of data that needs to be synchronized from multiple source servers in the cluster environment to the destination server through the scheduling node, and calculate the amount of data processed by each data processing node per unit time, that is, the average processing rate (each of the above-mentioned The efficiency of a transit node to process data).

Step S103, distribute each target data to each first transit node through the second transit node according to the quantity of multiple target data and the data processing efficiency of each first transit node, so that each first transit node is Each target data is processed to obtain multiple processed target data, and the processed multiple target data are synchronized to the target server.

For example, the scheduling node allocates the data to be synchronized to each data processing nodes for processing. After each data processing node receives the data allocated to it, it processes the data allocated to it, obtains the processed data, and then synchronizes the processed data to the destination server.

Through the above-mentioned steps S101 to S103, the second transit node assigns each target data to each first transit node according to the determined number of target data and the data processing efficiency of each first transit node, so that Each first transit node processes each target data, obtains multiple processed target data, and synchronizes the processed multiple target data to the destination server, thereby improving the efficiency and stability of data synchronization, and further Improved the effect of data synchronization.

In order to quickly and accurately determine the second transit node, in the data synchronization method provided by the embodiment of the present application, the second transit node may also be determined through the following steps: determine multiple target transit nodes, wherein at least including a plurality of first transfer nodes; determining a second transfer node from the plurality of target transfer nodes, so as to determine the quantity of multiple target data and the efficiency of processing data by each first transfer node through the second transfer node, wherein, the second The transit node is a transit node in the multiple target transit nodes except the multiple first transit nodes.

For example, as shown in Figure 2, there may be multiple transit nodes in the data synchronization device (that is, corresponding to transit node 1, transit node 2, transit node 3, transit node 4, etc. in Fig. 2), and the transit nodes in Fig. 2 1. Transit node 2, transit node 3, transit node 4, etc. may be the above-mentioned target transit nodes. Then, any one of the transit nodes can be selected from transit nodes 1, 2, 3, 4, etc. The number of data synchronized from multiple source servers to the destination server, and calculate the amount of data processed by each data processing node per unit time, that is, the average processing rate (the above-mentioned efficiency of processing data by each first transfer node). In addition, if transit node 1 among transit nodes 1, 2, 3, and 4 is used as a scheduling node, the rest of transit nodes 2, 3, 4, etc. The node serves as a data processing node (the aforementioned multiple first transit nodes).

To sum up, a multi-node synchronization device is designed, and one node is selected as the scheduling node. The scheduling node is used to connect to each server in the cluster in a unified manner. The processing node completes the processing work based on load balancing, which is easy to deploy and avoids insufficient load on a single node.

In order to quickly and accurately determine the second transit node, in the data synchronization method provided in the embodiment of the present application, the second transit node can also be determined through the following steps: obtain the serial number information corresponding to each target transit node; The number information corresponding to the node determines the transit node with the smallest number among the multiple target transit nodes; the transit node with the smallest number among the multiple target transit nodes is used as the second transit node.

For example, the transfer nodes can be automatically numbered according to the time of joining the system, and the transfer node numbered 1 can be automatically elected as a scheduling node. The scheduling node is responsible for node scheduling and task assignment, not data processing.

Through the above solution, the scheduling node can be quickly and accurately determined from multiple transit nodes.

In order to quickly and accurately assign each target data to each first transit node, in the data synchronization method provided in the embodiment of the present application, each target data can also be assigned to each first transit node through the following steps: Judging whether the data volume of the multiple target data is greater than the preset data volume; if the data volume of the multiple target data is greater than the preset data volume, the second transit node processes according to the quantity of multiple target data and each first transit node Data efficiency, assigning each target data to each first transit node, so that the first transit node obtains multiple target data from multiple source servers, and processes multiple target data; if multiple target data If the amount of data is not greater than the preset data amount, multiple target data will be obtained from multiple source servers through the second transfer node; according to the number of multiple target data and the efficiency of processing data at each first transfer node, the Each target data obtained from source servers is assigned to each first transit node.

For example, when the data to be processed occupies a large amount of bandwidth and the transmission capacity of the scheduling node is limited, when the scheduling node allocates tasks, the process is as follows: the scheduling node obtains task-related information from the source server, such as the serial number of the data, without actually synchronizing the data . When the data synchronization work needs to be assigned to other transit nodes, a message is sent to inform other transit nodes of the source server corresponding to the task, the data number, quantity and other information, and the transit node obtains the data from the source server by itself. And when the data to be processed occupies a large amount of bandwidth, a schematic diagram of scheduling nodes forwarding task data is shown in Figure 3 .

For example, when the data to be processed does not need to occupy a large amount of bandwidth and the transmission capacity of the scheduling node is sufficient, when the scheduling node allocates tasks, the process is: the scheduling node obtains task-related information and the data itself from the source server. When the data synchronization work needs to be assigned to other transit nodes, a message is sent to forward the data to other transit nodes, and there is no need for other transit nodes to establish a connection with the source server at this time. And when the data to be processed does not need to occupy a large amount of bandwidth, the schematic diagram of the scheduling node forwarding task data is shown in Figure 4.

To sum up, two deployment methods can be set according to the type of data to be synchronized. When the data occupies a large amount of bandwidth, the scheduling node assigns tasks. When the data does not occupy a large amount of bandwidth, the scheduling node distributes the actual task data.

In order to quickly and accurately obtain the processed first data, in the data synchronization method provided in the embodiment of the present application, the processed first data may also be obtained through the following steps: determine the fifth transit among multiple first transit nodes Whether the node is in an idle state; if the fifth transit node in a plurality of first transit nodes is in an idle state, then the first data in the plurality of target data will be assigned to the fifth transit node, so that the fifth transit node is to the first transit node The data is processed to obtain the processed first data, and the processed first data is synchronized to the destination server; if the fifth transit node among the multiple first transit nodes is not in an idle state, then it is judged that the multiple first transit nodes Whether the sixth transit node in the node is in an idle state; if the sixth transit node in a plurality of first transit nodes is in an idle state, the first data in the plurality of target data is allocated to the sixth transit node, so that the sixth transit node The six transit nodes process the first data, obtain the processed first data, and synchronize the processed first data to the destination server.

For example, when the system is initially running, a relatively simple round-robin load balancing method is adopted. The process is as follows: the scheduling node obtains tasks from each source server, traverses other transit nodes each time, and assigns a fixed amount of tasks to each transit node. Before allocation, query whether the target transit node is free. If it is free, assign the task to the transit node. If it is not free, query the next transit node.

Through the above-mentioned solution, the initial scheduling node can quickly distribute the data to be synchronized to each source server for processing according to the round-robin load balancing method.

Fig. 5 is a flow chart 1 of the data synchronization method provided according to the embodiment of the present application. As shown in Fig. 5, in the data synchronization method provided in the embodiment of the present application, according to the number of multiple target data through the second transit node and the efficiency of data processing by each first transit node, distributing each target data to each first transit node includes:

Step S501, calculating the amount of data allocated to each first transfer node according to the quantity of multiple target data and the data processing efficiency of each first transfer node;

Step S502, based on the amount of data allocated to each first transit node, assign each target data to each first transit node.

For example, when performing index monitoring, the idle time can be calculated first, specifically, for any transit node, accept the task assigned by the scheduling node at intervals. The transfer node accepts the task assigned by the scheduling node, processes it after accepting the task, and waits for the next task to arrive after processing. During this period, the scheduling node may ask whether the node is idle multiple times, because the current task has not been processed, and the scheduling node will not Assign it a new task. Now assume that the time to accept a task is t1, the time when this task is completed is t2, and the time to accept the task next time is t3. t2 is between t1 and t3, and during the time from t1 to t3, the idle time Tidle=t3-t2.

Then calculate the average processing rate, and set μ to be the amount of data processed by the transit node per unit time, that is, the average processing rate. In addition, different types of data usually use different units of measurement. For example, when processing source server logs through a transfer node in this embodiment of the application, the number of log entries can be used as the unit of measurement. The calculation formula is:

Among them, p is the total amount of data processed by the transit node within the statistical time, Tidle is the total idle time of the transit node within the statistical time (that is, the time period when the data is not actually processed), and T is the statistical time.

For example, the scheduling node obtains tasks from each source server, and uses μ as the weight to assign tasks weightedly. Assume that the scheduling node obtains n pieces of data from each source server at a time, and the μ values of each processing node are μ1, μ2, ..., μn-1.

Then the amount of data allocated to transit node k is:

Then, the scheduling node can allocate the data to be synchronized to the destination server to each data processing intermediate node according to the calculated data volume allocated to each data processing intermediate node.

To sum up, dynamic load balancing based on statistical indicators can distribute synchronization tasks according to the load capacity of nodes and reduce the load of each node as much as possible.

In order to quickly and accurately synchronize multiple target data to the destination server, in the data synchronization method provided by the embodiment of the present application, multiple target data can also be synchronized to the target server through the following steps: Obtain each source server at a unit time The amount of data to be synchronized to the destination server within; according to the amount of data to be synchronized to the destination server by each source server per unit time, the average queue length of each first transit node is calculated, wherein the average queue length is used to represent the The accumulation of a plurality of target data in the first time period; determine the real-time queue length of each first transit node, wherein the real-time queue length is used to represent the load pressure situation of each first transit node in the second time period, The length of the second time period is less than the length of the first time period; based on the average queuing length and real-time queuing length of each first transit node, a reminder message is sent to the target object, wherein the reminder message is used to remind the target object to adjust multiple The strategy for synchronizing target data to the target server.

For example, the scheduling node can record the following indicators: λ, the amount of data that needs to be synchronized per unit time for all source servers, that is, the average arrival rate, and its measurement unit is the same as μ. The calculation formula of λ is:

Among them, T is the statistical time, and q is the sum of data volumes that need to be synchronized by all source servers within the statistical time.

Then, the scheduling node can pay attention to whether the current load situation is too heavy, calculate relevant indicators and provide them to users. Its calculation indicators include the real-time queue length L1 and the average queue length L2. The real-time queue length L1 is the amount of data recorded by the scheduling node that cannot be allocated at present, and the real-time queue length reflects the current load pressure of the transit node in a short period of time.

The average queuing length L2 reflects the accumulation of data over a long period of time, and reflects whether the setting of transit nodes is reasonable. If the value is large, it is recommended to add transit nodes. Its calculation formula is as follows:

In the formula, λ: the amount of data that needs to be synchronized per unit time of all source servers, μ: the sum of the amount of data processed by the transit node per unit time, that is

Through the above solution, when the data in the source server is synchronized to the destination server, the user can be promptly reminded to adjust the data synchronization method, thereby improving the stability and efficiency of data synchronization.

In order to quickly and accurately troubleshoot the transit node, in the data synchronization method provided in the embodiment of the present application, the transit node can also be troubleshooted through the following steps: determine whether the second transit node fails; if the second transit node When a failure occurs, a fourth transit node is determined from the plurality of first transit nodes, wherein the fourth transit node is used to replace the second transit node to schedule transit nodes other than the fourth transit node among the plurality of first transit nodes.

For example, in addition to realizing dynamic load balancing according to the queuing model, the device designed in the embodiment of the present application may also have a fault replacement function.

That is, when any transit node in the device fails, the device can quickly replace the fault.

If the current scheduling node fails, the device performs the following process:

(1) Elect a new scheduling node according to the order of numbers. For example, if the scheduling node numbered 1 fails, the node numbered 2 will be the new scheduling node; if the scheduling node numbered 2 fails, the node numbered 3 will be new scheduling node;

(2) Re-execute the initial deployment process, that is, redeploy the data synchronization device, and adopt a new transit node as a scheduling node;

(3) Re-execute the indicator monitoring process, that is, after the initialization is completed, the system runs and starts to observe, and each processing node records μ, which is the amount of data processed by the transit node per unit time, that is, the average processing rate, and the scheduling node records λ, which is The amount of data that all source servers need to synchronize per unit time, that is, the average arrival rate;

(4) Re-execute the dynamic adjustment process, that is, the scheduling node dynamically adjusts the data allocated to each source server according to the changes in the indicators monitored during the indicator monitoring process.

To sum up, the scheduling node and the statistical node have the same structure and can replace each other. If either of them fails, they can be quickly replaced.

In order to quickly and accurately troubleshoot the transit node, in the data synchronization method provided in the embodiment of the present application, the transit node can also be troubleshooted through the following steps: judging whether there is a faulty node among the multiple first transit nodes; If there is a faulty node among the multiple first transit nodes, multiple third transit nodes other than the faulty node are determined from the multiple first transit nodes, wherein the third transit node is used to replace the first transit node for multiple The target data is processed; the data processing efficiency of each third transit node is obtained; and each target data is allocated to each third transit node according to the quantity of multiple target data and the data processing efficiency of each third transit node.

For example, if one or more processing nodes fail, the device performs the following process:

The scheduling node records the faulty processing node, continues to execute the above indicator monitoring process, and updates the above dynamic adjustment process as follows:

The amount of data allocated to a transit node k (the transit node is a non-faulty node) is changed to:

where b represents the number of faulty nodes.

In addition, regardless of whether the scheduling node fails or the processing node fails, when the node fails, the real-time queue length L1 and the average queue length L2 may increase. Add a device, and the newly added device will automatically act as a processing node, and the joining process is as follows:

(1) Re-execute the initial deployment process, that is, re-deploy the data synchronization device, and use a new transfer node as a scheduling node or a data processing transfer node;

(2) Re-execute the indicator monitoring process, that is, after the initialization is completed, the system runs and starts to observe, and each processing node records μ, which is the amount of data processed by the transit node per unit time, that is, the average processing rate, and the scheduling node records λ, which is The amount of data that all source servers need to synchronize per unit time, that is, the average arrival rate;

(3) Re-execute the dynamic adjustment process, that is, the scheduling node dynamically adjusts the data allocated to each source server according to the changes in the indicators monitored during the indicator monitoring process, and the n value in the above formula will include the newly added transit node .

To sum up, the scheduling node and the statistical node can replace each other, and if either of them fails, it can be quickly replaced.

For example, FIG. 6 is a flowchart of an optional data synchronization method provided according to an embodiment of the present application. As shown in FIG. 6, the optional data synchronization method includes the following steps:

1. Deploy the device

The embodiment of the present application is assumed to be deployed as follows, where the number of destination servers may be multiple, and the solution of the embodiment of the present application does not pay attention to the binding relationship between the destination server and each transit node, so the destination server is regarded as one.

Assume that there are m source servers and n transit nodes, and the transit nodes synchronize data from the source server, process it, and send it to the destination server.

Among all the transit nodes, one of the transit nodes is regarded as a scheduling node, and the scheduling node is responsible for the scheduling of all transit nodes, and the other nodes are responsible for data processing, also known as data processing nodes.

For any transit node, its internal structure is composed of a network module and a multi-process data processing module. The former is responsible for sending and receiving data, and the latter is responsible for data processing. There is no difference between a scheduling node and a data processing node, but it is elected during actual operation to schedule other nodes. If necessary, any transit node can be used as a scheduling node.

Moreover, according to the time of joining the system, the transfer nodes are automatically numbered, and the transfer node numbered 1 is automatically elected as a scheduling node. The scheduling node is responsible for node scheduling and task assignment, not data processing.

When the data to be processed occupies a large amount of bandwidth and the transmission capacity of the scheduling node is limited, when the scheduling node allocates tasks, the process is as follows: the scheduling node obtains task-related information from the source server, such as the serial number of the data, without actually synchronizing the data. When the data synchronization work needs to be assigned to other transit nodes, a message is sent to inform other transit nodes of the source server corresponding to the task, the data number, quantity and other information, and the transit node obtains the data from the source server by itself.

When the data to be processed does not need to occupy a large amount of bandwidth and the transmission capacity of the scheduling node is sufficient, when the scheduling node allocates tasks, the process is as follows: the scheduling node obtains task-related information and the data itself from the source server. When the data synchronization work needs to be assigned to other transit nodes, a message is sent to forward the data to other transit nodes, and there is no need for other transit nodes to establish a connection with the source server at this time.

When the system is initially running, a relatively simple round-robin load balancing method is adopted. The process is as follows: the scheduling node obtains tasks from each source server, traverses other transit nodes each time, and assigns a fixed amount of tasks to each transit node. Before allocation, query whether the target transit node is free. If it is free, assign the task to the transit node. If it is not free, query the next transit node.

2. Indicator monitoring

(1) Calculation of free time

For any transit node, it accepts tasks assigned by the scheduling node at intervals.

The transfer node accepts the task assigned by the scheduling node, processes it after accepting the task, and waits for the next task to arrive after processing. During this period, the scheduling node may ask whether the node is idle multiple times, because the current task has not been processed, and the scheduling node will not Assign new tasks to it. Now assume that the time to accept a task is t1, the time when this task is completed is t2, and the time to accept the task next time is t3. t2 is between t1 and t3, and during the time from t1 to t3, the idle time Tidle=t3-t2.

(2) Calculation of average processing rate and average arrival rate

After the initialization in Section 1 is completed, the system runs and starts to observe, and each processing node records the following indicators:

μ: the amount of data processed by the transfer node per unit time, that is, the average processing rate. Different types of data usually use different measurement units. For example, when the source server logs are processed by the transfer node in the embodiment of this application, the number of logs can be used as the measurement unit. Its calculation formula is:

Among them, p is the total amount of data processed by the transit node within the statistical time, Tidle is the total idle time of the transit node within the statistical time (that is, the time period when the data is not actually processed), and T is the statistical time.

The scheduling node records the following indicators:

λ: The amount of data that all source servers need to synchronize per unit time, that is, the average arrival rate, and its measurement unit is the same as μ. The calculation formula of λ is:

Among them, T is the statistical time, and q is the sum of data volumes that need to be synchronized by all source servers within the statistical time.

3. Dynamic adjustment

The purpose of dynamic adjustment based on detection indicators is to allocate tasks according to the load capacity of nodes, reduce the burden of each node, reduce the average data processing time, and avoid data retention. When the amount of data that the system needs to process is too large for each transit node to bear, the scheduling node will give an early warning.

If the polling method in Section 1 is used all the time, it is likely to cause a node to be overloaded for a long time. Therefore, it is necessary to make dynamic adjustments according to the monitoring indicators in Section 2.

The scheduling node obtains tasks from each source server, and assigns tasks weighted by μ as a weight. Assume that the scheduling node obtains n pieces of data from each source server at a time, and the μ values of each processing node are μ1, μ2, ..., μn-1.

Then the amount of data allocated to transit node k is:

The transit node counts μ and λ at a certain period, and when there is a significant difference between μ and λ through the t test, the values of the two are updated.

The scheduling node pays attention to whether the current load is too heavy, calculates relevant indicators and provides them to users. Its calculation indicators include the real-time queue length L1 and the average queue length L2. The real-time queue length L1 is the amount of data recorded by the scheduling node that cannot be allocated at present, and the real-time queue length reflects the current load pressure of the transit node in a short period of time.

The average queuing length L2 reflects the accumulation of data over a long period of time, and reflects whether the setting of transit nodes is reasonable. If the value is large, it is recommended to add transit nodes. Its calculation formula is as follows:

In the formula, λ: the amount of data that all source servers need to synchronize per unit time

μ: the total amount of data processed by the transit node per unit time, that is,

4. Fault replacement

The device designed in the embodiment of the present application not only realizes dynamic load balancing according to the queuing model, but also has a fault replacement function.

That is, when any transit node in the device fails, the device can quickly replace the fault.

If the current scheduling node fails, the device performs the following process:

(1) Elect a new scheduling node according to the order of numbers. For example, if the scheduling node numbered 1 fails, the node numbered 2 will be the new scheduling node; if the scheduling node numbered 2 fails, the node numbered 3 will be The new scheduling node.

(2) Re-execute the initialization process described in Section 1.

(3) Re-execute the indicator monitoring process described in Section 2.

(4) Re-execute the dynamic adjustment process described in Section 3.

If one or more processing nodes fail, the device performs the following process:

The scheduling node records the faulty processing node, continues to execute the indicator monitoring process described in Section 2, and updates the dynamic adjustment process described in Section 3 as follows:

The amount of data allocated to a transit node k (the transit node is a non-faulty node) is changed to:

where b represents the number of faulty nodes.

Regardless of whether the scheduling node fails or the processing node fails, when the node fails, the real-time queue length L1 and the average queue length L2 may increase. At this time, the user can increase the number of transit nodes and add new transit nodes to the device , the newly added device automatically acts as a processing node, and its joining process is:

(1) Re-execute the initialization process described in Section 1.

(2) Re-execute the indicator monitoring process described in Section 2.

(3) Re-execute the dynamic adjustment process described in Section 3, and the value of n in the above formula will include the newly added transit node.

For example, in the audit and control of network security, security vendors often deploy proxy software in the internal network to synchronize users from authentication servers deployed in clusters, and the proxy software forwards data to network security devices to help network security devices complete traffic real-name audit and control. In network security auditing, the logs of different network security devices are usually different. In order to aggregate the logs to the log platform for unified management, a transfer node can be set up to process and format the logs uniformly. In a small network, server 1 deployed in a cluster stores resources that can be accessed by users. Server 2 is used to reverse proxy user requests and provide resources for users. A transfer node is set between server 1 and server 2 to decode and classify data. , formatting and other processing operations.

Therefore, through the method provided in the embodiment of the present application, multiple transfer nodes can be used to dynamically connect to multiple source servers in the cluster environment, and data is processed and forwarded to the destination server. And based on the processing efficiency of the transit node and the data volume of the source server, dynamically adjust the binding relationship between the two, which can improve the efficiency and stability of data synchronization.

To sum up, the data synchronization method provided by the embodiment of the present application obtains multiple target data and multiple first transit nodes through the second transit node, wherein each target data is multiple source servers in the cluster environment to be synchronized to the destination The data of the server; determine the quantity of multiple target data and the efficiency of processing data by each first transit node through the second transit node; process the data according to the quantity of multiple target data and the efficiency of each first transit node through the second transit node Efficiency, assigning each target data to each first transit node, so that each first transit node processes each target data, obtains processed multiple target data, and distributes the processed multiple target data Synchronization to the destination server solves the problem of statically binding the source server in the cluster environment with the transfer node used to process data in the related technology, and synchronizes the data of the source server in the cluster environment to the destination server, resulting in a relatively low effect of data synchronization. bad question. According to the determined number of target data and the data processing efficiency of each first transit node, the second transit node assigns each target data to each first transit node, so that each first transit node The target data is processed to obtain multiple target data after processing, and the multiple target data after processing are synchronized to the destination server, thereby improving the efficiency and stability of data synchronization, and further improving the effect of data synchronization.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be performed in a computer system, such as a set of computer-executable instructions, and that although a logical order is shown in the flowcharts, in some cases, The steps shown or described may be performed in an order different than here.

Example 2

The embodiment of the present application also provides a data synchronization device. It should be noted that the data synchronization device in the embodiment of the present application can be used to execute the data synchronization method provided in the embodiment of the present application. The device for synchronizing data provided by the embodiment of the present application is introduced below.

Fig. 7 is a schematic diagram of a data synchronization device according to an embodiment of the present application. As shown in Figure 7, the device is applied in a data synchronization system. The data synchronization system at least includes: a plurality of first transfer nodes and second transfer nodes, the first transfer nodes are used for processing data, and the second transfer nodes are used for scheduling multiple first transit nodes, if there is a faulty node among the multiple first transit nodes, then use a transit node other than the faulty node among the first transit nodes to replace the faulty node; if the second transit node fails, then use multiple Any one of the first transit nodes replaces the second transit node, and the apparatus includes: a first acquiring module 701 , a first determining module 702 and a first processing module 703 .

Specifically, the first acquisition module 701 is configured to acquire a plurality of target data and a plurality of first transfer nodes through the second transfer node, wherein each target data is data to be synchronized from a plurality of source servers to a destination server in a cluster environment ;

The first determining module 702 is configured to determine the quantity of multiple target data and the efficiency of processing data by each first transit node through the second transit node;

The first processing module 703 is configured to distribute each target data to each first transit node through the second transit node according to the quantity of multiple target data and the data processing efficiency of each first transit node, so that each The first transfer node processes each target data, obtains multiple processed target data, and synchronizes the processed multiple target data to the destination server.

To sum up, in the data synchronization device provided by the embodiment of the present application, the first acquisition module 701 acquires multiple target data and multiple first transit nodes through the second transit node, wherein each target data is multiple sources in the cluster environment The server is to be synchronized to the data of the destination server; the first determination module 702 determines the quantity of multiple target data and the efficiency of each first transfer node to process the data through the second transfer node; the first processing module 703 uses the second transfer node according to multiple The number of target data and the efficiency of each first transfer node to process data, each target data is allocated to each first transfer node, so that each first transfer node processes each target data, and after processing multiple target data, and synchronize the processed multiple target data to the destination server, which solves the problem of statically binding the source server in the cluster environment with the transfer node for data processing in the related technology, and the The data of the source server is synchronized to the destination server, resulting in poor data synchronization effect. According to the determined number of target data and the data processing efficiency of each first transit node, the second transit node assigns each target data to each first transit node, so that each first transit node The target data is processed to obtain multiple target data after processing, and the multiple target data after processing are synchronized to the destination server, thereby improving the efficiency and stability of data synchronization, and further improving the effect of data synchronization.

Optionally, in the data synchronization device provided in the embodiment of the present application, the first processing module includes: a first calculation unit, configured to calculate the The amount of data allocated to each first transit node is obtained; the first allocating unit is configured to allocate each target data to each first transit node based on the amount of data allocated to each first transit node.

Optionally, in the data synchronization device provided in the embodiment of the present application, the device further includes: a second determination module, configured to determine the quantity of multiple target data through the second transit node and process each first transit node Before the data efficiency, a plurality of target transit nodes are determined, wherein the plurality of target transit nodes at least include a plurality of first transit nodes; a third determination module is configured to determine a second transit node from the plurality of target transit nodes, to The quantity of multiple target data and the data processing efficiency of each first transit node are determined by the second transit node, wherein the second transit node is a transit node among the multiple target transit nodes except the multiple first transit nodes.

Optionally, in the data synchronization device provided in the embodiment of the present application, the device further includes: a first judging module, configured to process the target data according to the quantity of multiple target data and each first transit node through the second transit node For data efficiency, after distributing each target data to each first transit node, it is judged whether there is a faulty node among the multiple first transit nodes; the fourth determination module is used for if there is a faulty node among the multiple first transit nodes , then determine a plurality of third transit nodes except the faulty node from the plurality of first transit nodes, wherein the third transit node is used to replace the first transit node to process multiple target data; the second acquisition module, Used to obtain the efficiency of data processing by each third transfer node; the first allocation module is used to allocate each target data to each third transfer node according to the quantity of multiple target data and the efficiency of processing data by each third transfer node Three transit nodes.

Optionally, in the data synchronization device provided in the embodiment of the present application, the device further includes: a second judging module, configured to judge whether the second transit node is A failure occurs; the fifth determination module is configured to determine a fourth transit node from multiple first transit nodes if the second transit node fails, wherein the fourth transit node is used to replace the second transit node to schedule multiple first transit nodes Transit nodes in a transit node other than the fourth transit node.

Optionally, in the data synchronization device provided in the embodiment of the present application, the first processing module includes: a first judging unit, configured to judge whether the data volume of multiple target data is greater than a preset data volume; a second distributing unit, If the data volume of the multiple target data is greater than the preset data volume, the second transit node distributes each target data to each The first transfer node, so that the first transfer node obtains a plurality of target data from a plurality of source servers, and processes the plurality of target data; If the amount of data is set, a plurality of target data is obtained from multiple source servers through the second transfer node; the third distribution unit is used to process the data from the Each target data obtained from multiple source servers is assigned to each first transit node.

Optionally, in the data synchronization device provided in the embodiment of the present application, the third determining module includes: a second acquiring unit, configured to acquire number information corresponding to each target transit node; a first determining unit, configured to The number information corresponding to each target transit node is used to determine the transit node with the smallest number among the multiple target transit nodes; the second determination unit is configured to use the transit node with the smallest number among the multiple target transit nodes as the second transit node.

Optionally, in the data synchronization device provided in the embodiment of the present application, the device further includes: a third acquisition module, configured to acquire the amount of data to be synchronized to the destination server by each source server within a unit time; the first calculation The module is used to calculate the average queue length of each first transit node according to the amount of data to be synchronized to the destination server by each source server within a unit time, wherein the average queue length is used to indicate how many accumulation of target data; the sixth determination module is used to determine the real-time queue length of each first transfer node, wherein the real-time queue length is used to represent the load pressure situation of each first transfer node in the second time period , the length of the second time period is less than the length of the first time period; the first sending module is configured to send reminder information to the target object based on the average queue length and real-time queue length of each first transit node, wherein the reminder information is used To remind the target object to adjust the strategy of synchronizing multiple target data to the target server.

Optionally, in the data synchronization device provided in the embodiment of the present application, the device further includes: a third judging module, configured to judge the multiple first transit nodes after acquiring multiple target data and multiple first transit nodes Whether the fifth transit node among the nodes is in an idle state; the second distribution module is configured to allocate the first data among the plurality of target data to the first transit node if the fifth transit node among the plurality of first transit nodes is in an idle state Five transit nodes, so that the fifth transit node processes the first data, obtains the processed first data, and synchronizes the processed first data to the destination server; the fourth judging module is used for if multiple first If the fifth transit node among the transit nodes is not in an idle state, it is judged whether the sixth transit node among the plurality of first transit nodes is in an idle state; When the transit node is in an idle state, the first data among the plurality of target data is allocated to the sixth transit node, so that the sixth transit node processes the first data, obtains the processed first data, and distributes the processed The first data is synchronized to the destination server.

The data synchronization device includes a processor and a memory. The above-mentioned first acquisition module 701, first determination module 702, and first processing module 703 are all stored in the memory as program units, and the above-mentioned program units stored in the memory are executed by the processor. to realize the corresponding function.

The processor includes a kernel, and the kernel fetches corresponding program units from the memory. One or more kernels can be set to improve the effect of data synchronization by adjusting kernel parameters.

Memory may include non-permanent memory in computer-readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM), memory includes at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, the data synchronization method is implemented.

An embodiment of the present invention provides a processor, where the processor is used to run a program, wherein the method for synchronizing data is executed when the program is running.

An embodiment of the present invention provides an electronic device. The device includes a processor, a memory, and a program stored on the memory and operable on the processor. When the processor executes the program, the following steps are implemented: Obtain multiple targets through the second transit node data and the plurality of first transfer nodes, wherein each target data is data to be synchronized from multiple source servers to the destination server in a cluster environment; the quantity and the number of the plurality of target data are determined by the second transfer node The data processing efficiency of each first transit node; the second transit node assigns each target data to each first transit node according to the quantity of the plurality of target data and the data processing efficiency of each first transit node A transfer node, so that each first transfer node processes each target data, obtains multiple processed target data, and synchronizes the processed multiple target data to the destination server.

When the processor executes the program, the following steps are also implemented: according to the quantity of the plurality of target data and the data processing efficiency of each first transit node through the second transit node, assign each target data to each first transit node The node includes: calculating the amount of data allocated to each first transit node according to the quantity of the plurality of target data and the efficiency of processing data of each first transit node; based on the amount of data allocated to each first transit node , distributing each target data to each first transit node.

When the processor executes the program, the following steps are also implemented: before the quantity of the multiple target data and the efficiency of processing data by each first transit node are determined by the second transit node, the method further includes: determining multiple target data A transit node, wherein at least the plurality of first transit nodes are included in the plurality of target transit nodes; the second transit node is determined from the plurality of target transit nodes, so as to be determined by the second transit node The quantity of the plurality of target data and the data processing efficiency of each first transfer node, wherein the second transfer node is a transfer of the plurality of target transfer nodes except the plurality of first transfer nodes node.

When the processor executes the program, the following steps are also implemented: according to the quantity of the plurality of target data and the data processing efficiency of each first transit node through the second transit node, assign each target data to each first transit node After the transfer node, the method further includes: judging whether there is a fault node among the plurality of first transfer nodes; if there is a fault node among the plurality of first transfer nodes, selecting the Determining a plurality of third transit nodes other than the faulty node, wherein the third transit nodes are used to replace the first transit nodes to process the plurality of target data; acquiring each third transit node to process data efficiency; assigning each target data to each third transfer node according to the quantity of the plurality of target data and the data processing efficiency of each third transfer node.

When the processor executes the program, the following steps are further implemented: after determining the second transit node from the plurality of target transit nodes, the method further includes: judging whether the second transit node fails; if the second transit node If the second transit node fails, then determine a fourth transit node from the plurality of first transit nodes, where the fourth transit node is used to replace the second transit node to schedule among the plurality of first transit nodes A transit node other than the fourth transit node.

When the processor executes the program, the following steps are also implemented: according to the quantity of the plurality of target data and the data processing efficiency of each first transit node through the second transit node, assign each target data to each first transit node The node includes: judging whether the data volume of the plurality of target data is greater than the preset data volume; if the data volume of the multiple target data is greater than the preset data volume, then through the second transit node according to the multiple the number of target data and the data processing efficiency of each first transfer node, assign each target data to each first transfer node, so that the first transfer node obtains the described first transfer node from the plurality of source servers a plurality of target data, and process the plurality of target data; if the data volume of the multiple target data is not greater than the preset data volume, then from the multiple source servers through the second transit node Acquiring the plurality of target data; according to the quantity of the plurality of target data and the data processing efficiency of each first transit node, assign each target data acquired from the plurality of source servers to each first transfer node A transit node.

When the processor executes the program, the following steps are also implemented: determining the second transit node from the plurality of target transit nodes includes: obtaining number information corresponding to each target transit node; based on the number information corresponding to each target transit node, Determine the transit node with the smallest number among the multiple target transit nodes; use the transit node with the smallest number among the multiple target transit nodes as the second transit node.

When the processor executes the program, the following steps are also implemented: the method also includes: obtaining the amount of data to be synchronized to the destination server by each source server within unit time; amount, calculate the average queuing length of each first transit node, wherein, the average queuing length is used to represent the accumulation of the plurality of target data in the first time period; determine the real-time queue length, wherein the real-time queue length is used to represent the load pressure situation of each first transit node in a second time period, the length of the second time period is less than the length of the first time period; based on each The average queuing length and real-time queuing length of a first transit node, and send reminder information to the target object, wherein the reminder information is used to remind the target object to adjust the strategy of synchronizing the multiple target data to the destination server. .

When the processor executes the program, the following steps are also implemented: after acquiring multiple target data and multiple first transit nodes, the method further includes: judging whether the fifth transit node among the multiple first transit nodes is in an idle state ; If the fifth transit node among the plurality of first transit nodes is in an idle state, assigning the first data among the plurality of target data to the fifth transit node, so that the fifth transit node is process the first data, obtain the processed first data, and synchronize the processed first data to the destination server; if the fifth transit node among the plurality of first transit nodes is not idle state, it is judged whether the sixth transit node among the plurality of first transit nodes is in an idle state; if the sixth transit node among the plurality of first transit nodes is in an idle state, then the The first data is distributed to the sixth transit node, so that the sixth transit node processes the first data, obtains the processed first data, and synchronizes the processed first data to the The destination server.

The devices in this article can be servers, PCs, PADs, mobile phones, etc.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. The memory is an example of a computer readable medium.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The above are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (10)

1. A data synchronization method is applied to a data synchronization system, wherein the data synchronization system at least comprises a plurality of first transit nodes and a second transit node, the first transit nodes are used for processing data, and the second transit nodes are used for scheduling the plurality of first transit nodes; if a fault node exists in the plurality of first transit nodes, adopting transit nodes except the fault node in the first transit nodes to replace the fault node; if the second transit node fails, adopting any one of the plurality of first transit nodes to replace the second transit node; the synchronization method specifically comprises the following steps:

acquiring a plurality of target data and the plurality of first transit nodes through the second transit node, wherein each target data is data to be synchronized to a destination server by a plurality of source servers in a cluster environment;

determining, by the second transit node, the number of the plurality of target data and the efficiency of processing data by each first transit node;

and distributing each target data to each first transfer node through the second transfer node according to the number of the target data and the data processing efficiency of each first transfer node, so that each first transfer node processes each target data to obtain a plurality of processed target data, and synchronizing the plurality of processed target data to the target server.

2. The method of claim 1, wherein assigning, by the second transit node, each target data to each first transit node as a function of the number of the plurality of target data and the efficiency with which each first transit node processes data comprises:

calculating to obtain the data volume distributed to each first transfer node according to the quantity of the target data and the data processing efficiency of each first transfer node;

each target data is assigned to each first transit node based on the amount of data assigned to each first transit node.

3. The method of claim 1, wherein prior to determining, by the second transit node, the quantity of the plurality of target data and the efficiency with which each first transit node processes data, the method further comprises:

determining a plurality of target transit nodes, wherein the plurality of target transit nodes at least include the plurality of first transit nodes;

determining the second transit node from the plurality of target transit nodes to determine the amount of the plurality of target data and the efficiency of processing data by each first transit node through the second transit node, wherein the second transit node is a transit node other than the plurality of first transit nodes from the plurality of target transit nodes.

4. The method of claim 1, wherein after assigning each target data to each first transit node by the second transit node depending on the number of the plurality of target data and the efficiency with which each first transit node processes data, the method further comprises:

judging whether a fault node exists in the plurality of first transfer nodes or not;

if a fault node exists in the first transit nodes, determining a plurality of third transit nodes except the fault node from the first transit nodes, wherein the third transit nodes are used for processing the target data instead of the first transit nodes;

acquiring the data processing efficiency of each third transit node;

and distributing each target data to each third transfer node according to the quantity of the plurality of target data and the data processing efficiency of each third transfer node.

5. The method of claim 3, wherein after determining the second transit node from the plurality of target transit nodes, the method further comprises:

judging whether the second transit node has a fault;

and if the second transit node fails, determining a fourth transit node from the plurality of first transit nodes, wherein the fourth transit node is used for replacing the second transit node to schedule transit nodes except the fourth transit node in the plurality of first transit nodes.

6. The method of claim 1, wherein assigning, by the second transit node, each target data to each first transit node as a function of the number of the plurality of target data and the efficiency with which each first transit node processes data comprises:

judging whether the data volume of the target data is larger than a preset data volume or not;

if the data volume of the target data is larger than the preset data volume, distributing each target data to each first transfer node through the second transfer node according to the number of the target data and the data processing efficiency of each first transfer node, so that the first transfer node obtains the target data from the source servers and processes the target data;

if the data volume of the target data is not larger than the preset data volume, obtaining the target data from the source servers through the second transit node;

and distributing each target data acquired from the plurality of source servers to each first transfer node according to the quantity of the plurality of target data and the efficiency of processing data by each first transfer node.

7. The method of claim 3, wherein determining the second transit node from the plurality of target transit nodes comprises:

acquiring the number information corresponding to each target transfer node;

determining a transfer node with the minimum number in the plurality of target transfer nodes based on the number information corresponding to each target transfer node;

and taking the transfer node with the minimum number in the target transfer nodes as the second transfer node.

8. A data synchronization apparatus, wherein the apparatus is applied in a data synchronization system, and the data synchronization system at least comprises: a plurality of first transit nodes and a second transit node, the first transit nodes being configured to process data, the second transit node being configured to schedule the plurality of first transit nodes, if a failure node exists in the plurality of first transit nodes, replacing the failure node with a transit node other than the failure node in the first transit nodes, and if a failure occurs in the second transit node, replacing the second transit node with any transit node in the plurality of first transit nodes, including:

a first obtaining module, configured to obtain, by the second transit node, a plurality of target data and the plurality of first transit nodes, where each target data is data to be synchronized to a destination server by a plurality of source servers in a cluster environment;

a first determining module, configured to determine, by the second transit node, the number of the plurality of target data and an efficiency of processing data by each first transit node;

and the first processing module is used for distributing each target data to each first transfer node through the second transfer node according to the number of the target data and the data processing efficiency of each first transfer node, so that each first transfer node processes each target data to obtain a plurality of processed target data, and synchronizing the plurality of processed target data to the target server.

9. A computer-readable storage medium characterized in that the storage medium stores a program, wherein the program executes the synchronization method of data according to any one of claims 1 to 7.

10. A processor for a method of synchronizing data, the processor being configured to run a program, wherein the program is configured to perform the method of synchronizing data according to any one of claims 1 to 7 when the program is run.

Images