CN115834603B

CN115834603B - Data synchronization method and device, storage medium and processor

Info

Publication number: CN115834603B
Application number: CN202211466310.3A
Authority: CN
Inventors: 余祖愿; 万腾飞; 代应浪; 沈亚琪; 卢梦梦
Original assignee: Hillstone Networks Co Ltd
Current assignee: Beijing Hillstone Networks Information Technology Co ltd; Hillstone Networks Co Ltd
Priority date: 2022-11-22
Filing date: 2022-11-22
Publication date: 2025-09-23
Anticipated expiration: 2042-11-22
Also published as: CN115834603A

Abstract

The present application discloses a data synchronization method and device, storage medium and processor. The method includes: obtaining multiple target data and multiple first transfer nodes through a second transfer node; determining the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node; allocating each target data to each first transfer node based on the number of multiple target data and the efficiency of each first transfer node in processing data through the second transfer node, so that each first transfer node processes each target data to obtain multiple processed target data, and synchronizes the processed multiple target data to the destination server. Through this application, the problem of poor data synchronization effect caused by the method of static binding of the source server in the cluster environment and the transfer node for processing data in the related art is solved.

Description

Data synchronization method and device, storage medium and processor

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data synchronization method and apparatus, a storage medium, and a processor.

Background

Cluster (cluster) technology is a newer technology, by which relatively high gains in performance, reliability, flexibility can be achieved at lower cost, and task scheduling is the core technology in a clustered system.

Moreover, some servers deployed in clusters generally need to synchronize data to a target network, and because the interfaces provided by the data source server and the target server are not compatible, or the required data forms are inconsistent, the data needs to be processed by a transit node and then synchronized. The transit node may be a large server or data processing software installed on a PC (Personal Computer ). Three application scenarios are listed below:

(1) In the auditing and control of network security, security vendors often synchronize users from a cluster-deployed authentication server by deploying proxy software in the internal network, which forwards user data to the network security device to help the network security device complete real-name auditing and control of traffic.

(2) In a complex audit system, logs of various devices are generally different, and in order to collect the logs to a log platform for unified management, a transfer node can be set to perform unified processing and formatting on the logs.

(3) In a small network, a cluster-deployed server 1 stores resources which can be accessed by users, a server 2 is used for reversely acting user requests to provide resources for the users, and a transfer node is arranged between the server 1 and the server 2 to perform processing operations such as data decoding, classification, formatting and the like.

However, the above-mentioned transit nodes may have limited capability to process data and relatively limited stability. In order to ensure synchronization efficiency and synchronization stability, load balancing and redundancy deployment are generally needed, and a plurality of transfer nodes are used for interfacing a plurality of source servers in a cluster environment, so that data are processed and forwarded to a destination server.

In the related art, data is synchronized in a one-to-many manner, that is, one transit node is used to interface with a plurality of source servers in a cluster environment, so as to synchronize the data of the source servers to a target network. However, this method has the disadvantage that the single transit node has insufficient processing capability or is not stable, and may cause congestion or even complete paralysis of the data synchronization node.

In addition, in the related art, data is synchronized in a simple N-to-N manner, that is, a plurality of transit nodes exist, each transit node binds one source server, and the data of the source server is synchronized to the target network. However, this method has the disadvantage that it is inconvenient to perform redundant deployment because of the binding relationship, in order to prevent paralysis of a certain transit node, a backup redundant node must be set for the transit node, and when the number of the paralyzed nodes is greater than that of the redundant nodes, synchronization will be interrupted or partially interrupted.

Aiming at the problem that the effect of data synchronization is poor because the data of the source server in the cluster environment is synchronized to the target server by adopting a method for statically binding the source server in the cluster environment with the transfer node for processing the data in the related art, no effective solution is proposed at present.

Disclosure of Invention

The application mainly aims to provide a data synchronization method and device, a storage medium and a processor, which are used for solving the problem that the effect of data synchronization is poor because the data of a source server in a cluster environment is synchronized to a target server by adopting a method for statically binding the source server in the cluster environment with a transfer node for processing the data in the related art.

To achieve the above object, according to one aspect of the present application, there is provided a data synchronization method. The method is applied to a data synchronization system, and at least comprises a plurality of first transfer nodes and a plurality of second transfer nodes, wherein the first transfer nodes are used for processing data, the second transfer nodes are used for scheduling the plurality of first transfer nodes, if the plurality of first transfer nodes have fault nodes, the transfer nodes except the fault nodes in the first transfer nodes are adopted to replace the fault nodes, if the second transfer nodes have faults, any one of the plurality of first transfer nodes is adopted to replace the second transfer nodes, the method comprises the steps of acquiring a plurality of target data and the plurality of first transfer nodes through the second transfer nodes, wherein each target data is data to be synchronized to a target server in a cluster environment, determining the number of the plurality of target data and the efficiency of processing the data of each first transfer node through the second transfer nodes, distributing the second transfer nodes to each target data according to the number of the plurality of target data and the processing efficiency of each first transfer node, and processing each target data to each target server after the plurality of target data are processed, and processing the target data to each target server.

Further, allocating each target data to each first transit node by the second transit node according to the number of the plurality of target data and the efficiency of each first transit node for processing data comprises calculating the data amount allocated to each first transit node according to the number of the plurality of target data and the efficiency of each first transit node for processing data, and allocating each target data to each first transit node based on the data amount allocated to each first transit node.

Further, before determining the number of the plurality of target data and the efficiency of each first transit node to process the data by the second transit node, the method further comprises determining a plurality of target transit nodes, wherein at least the plurality of first transit nodes are included in the plurality of target transit nodes, and determining the second transit node from the plurality of target transit nodes to determine the number of the plurality of target data and the efficiency of each first transit node to process the data by the second transit node, wherein the second transit node is a transit node of the plurality of target transit nodes other than the plurality of first transit nodes.

Further, after each target data is allocated to each first transit node through the second transit node according to the number of the target data and the efficiency of each first transit node in processing the data, the method further comprises judging whether a fault node exists in the first transit nodes, determining a plurality of third transit nodes except the fault node from the first transit nodes if the fault node exists in the first transit nodes, wherein the third transit nodes are used for processing the target data instead of the first transit nodes, acquiring the efficiency of each third transit node in processing the data, and allocating each target data to each third transit node according to the number of the target data and the efficiency of each third transit node in processing the data.

Further, after determining the second transit node from the plurality of target transit nodes, the method further includes judging whether the second transit node fails, and if the second transit node fails, determining a fourth transit node from the plurality of first transit nodes, wherein the fourth transit node is used for scheduling transit nodes, except the fourth transit node, in the plurality of first transit nodes instead of the second transit node.

Further, the allocation of each target data to each first transit node through the second transit node according to the number of the plurality of target data and the efficiency of each first transit node in processing data comprises judging whether the data amount of the plurality of target data is larger than a preset data amount or not, if the data amount of the plurality of target data is larger than the preset data amount, allocating each target data to each first transit node through the second transit node according to the number of the plurality of target data and the efficiency of each first transit node in processing data, so that the first transit node obtains the plurality of target data from the plurality of source servers and processes the plurality of target data, and if the data amount of the plurality of target data is not larger than the preset data amount, obtaining the plurality of target data from the plurality of source servers through the second transit node, and allocating each obtained target data from the plurality of source servers to each first transit node according to the number of the plurality of target data and the efficiency of each first transit node in processing data.

Further, determining the second transit node from the plurality of target transit nodes comprises obtaining number information corresponding to each target transit node, determining the transit node with the smallest number in the plurality of target transit nodes based on the number information corresponding to each target transit node, and taking the transit node with the smallest number in the plurality of target transit nodes as the second transit node.

The method further comprises the steps of obtaining the data quantity to be synchronized to the destination server in unit time of each source server, calculating the average queuing length of each first transfer node according to the data quantity to be synchronized to the destination server in unit time of each source server, determining the real-time queuing length of each first transfer node, wherein the real-time queuing length is used for representing the load pressure condition of each first transfer node in a second time period, the length of the second time period is smaller than the length of the first time period, and sending reminding information to a target object based on the average queuing length and the real-time queuing length of each first transfer node, wherein the reminding information is used for reminding the target object to adjust a strategy for synchronizing the plurality of target data to the destination server.

Further, after the plurality of target data and the plurality of first transit nodes are acquired, the method further comprises the steps of judging whether a fifth transit node in the plurality of first transit nodes is in an idle state or not, distributing the first data in the plurality of target data to the fifth transit node if the fifth transit node in the plurality of first transit nodes is in the idle state, enabling the fifth transit node to process the first data to obtain processed first data, synchronizing the processed first data to the destination server, judging whether a sixth transit node in the plurality of first transit nodes is in the idle state if the fifth transit node in the plurality of first transit nodes is not in the idle state, distributing the first data in the plurality of target data to the sixth transit node if the sixth transit node in the plurality of first transit nodes is in the idle state, enabling the sixth transit node to process the first data to obtain the processed first data, and synchronizing the processed first data to the destination server.

To achieve the above object, according to another aspect of the present application, there is provided a data synchronizing apparatus. The device is applied to a data synchronization system, the data synchronization system at least comprises a plurality of first transfer nodes and a plurality of second transfer nodes, the first transfer nodes are used for processing data, the second transfer nodes are used for scheduling the plurality of first transfer nodes, if the plurality of first transfer nodes have fault nodes, the transfer nodes except the fault nodes in the first transfer nodes are adopted to replace the fault nodes, if the second transfer nodes have faults, any one of the plurality of first transfer nodes is adopted to replace the second transfer nodes, the device comprises a first acquisition module, a first determination module and a first processing module, the first processing module is used for acquiring a plurality of target data and the plurality of first transfer nodes through the second transfer nodes, each target data is data to be synchronized to a target server in a cluster environment, the first determination module is used for determining the quantity of the plurality of target data and the efficiency of each first transfer node for processing the data through the second transfer nodes, and the first processing module is used for distributing the second transfer nodes to the plurality of target data to the first transfer nodes according to the quantity of the second transfer nodes and the efficiency of each target data to each target server for processing the data after the plurality of target data is processed.

Further, the first processing module comprises a first calculating unit and a first distributing unit, wherein the first calculating unit is used for calculating the data quantity distributed to each first transit node according to the quantity of the target data and the data processing efficiency of each first transit node, and the first distributing unit is used for distributing each target data to each first transit node based on the data quantity distributed to each first transit node.

Further, the device further comprises a second determining module for determining a plurality of target transit nodes before determining the number of the plurality of target data and the efficiency of each first transit node for processing data by the second transit node, wherein at least the plurality of first transit nodes are included in the plurality of target transit nodes, and a third determining module for determining the second transit node from the plurality of target transit nodes so as to determine the number of the plurality of target data and the efficiency of each first transit node for processing data by the second transit node, wherein the second transit node is a transit node except the plurality of first transit nodes in the plurality of target transit nodes.

Further, the device comprises a first judging module, a fourth determining module, a first allocating module and a first allocating module, wherein the first judging module is used for judging whether a fault node exists in the first transfer nodes after each target data is allocated to each first transfer node through the second transfer node according to the quantity of the target data and the data processing efficiency of each first transfer node, the fourth determining module is used for determining a plurality of third transfer nodes except the fault node from the first transfer nodes if the fault node exists in the first transfer nodes, the third transfer nodes are used for processing the target data instead of the first transfer nodes, the second acquiring module is used for acquiring the data processing efficiency of each third transfer node, and the first allocating module is used for allocating each target data to each third transfer node according to the quantity of the target data and the data processing efficiency of each third transfer node.

Further, the device further comprises a second judging module for judging whether the second transit node is faulty after the second transit node is determined from the target transit nodes, and a fifth determining module for determining a fourth transit node from the first transit nodes if the second transit node is faulty, wherein the fourth transit node is used for scheduling transit nodes except the fourth transit node in the first transit nodes instead of the second transit node.

Further, the first processing module comprises a first judging unit for judging whether the data amount of the plurality of target data is larger than a preset data amount, a second distributing unit for obtaining the plurality of target data from the plurality of source servers through the second transit node according to the number of the plurality of target data and the efficiency of processing the data by each first transit node, so that the first transit node obtains the plurality of target data from the plurality of source servers and processes the plurality of target data, and a first obtaining unit for obtaining the plurality of target data from the plurality of source servers through the second transit node if the data amount of the plurality of target data is not larger than the preset data amount, and a third distributing unit for distributing each target data obtained from the plurality of source servers to each first transit node according to the number of the plurality of target data and the efficiency of processing the data by each first transit node.

Further, the third determining module comprises a second obtaining unit, a first determining unit and a second determining unit, wherein the second obtaining unit is used for obtaining the number information corresponding to each target transfer node, the first determining unit is used for determining the transfer node with the smallest number in the plurality of target transfer nodes based on the number information corresponding to each target transfer node, and the second determining unit is used for taking the transfer node with the smallest number in the plurality of target transfer nodes as the second transfer node.

The device further comprises a third acquisition module, a first calculation module and a first sending module, wherein the third acquisition module is used for acquiring the data quantity to be synchronized to the destination server in unit time of each source server, the first calculation module is used for calculating the average queuing length of each first transit node according to the data quantity to be synchronized to the destination server in unit time of each source server, the average queuing length is used for representing the accumulation condition of the plurality of target data in a first time period, the sixth determination module is used for determining the real-time queuing length of each first transit node, the real-time queuing length is used for representing the load pressure condition of each first transit node in a second time period, the length of the second time period is smaller than the length of the first time period, and the first sending module is used for sending reminding information to a target object based on the average queuing length and the real-time queuing length of each first transit node, wherein the reminding information is used for reminding the target object to adjust the strategy of synchronizing the plurality of target data to the destination server.

The device further comprises a third judging module, a second distributing module and a third distributing module, wherein the third judging module is used for judging whether a fifth transit node in the first transit nodes is in an idle state after a plurality of target data and the first transit nodes are acquired, the third distributing module is used for distributing the first data in the target data to the fifth transit node if the fifth transit node in the first transit nodes is in the idle state so that the fifth transit node processes the first data to obtain processed first data and synchronizes the processed first data to the destination server, the fourth judging module is used for judging whether a sixth transit node in the first transit nodes is in the idle state if the fifth transit node in the first transit nodes is not in the idle state, and the third distributing module is used for distributing the first data in the target data to the sixth transit node if the sixth transit node in the first transit nodes is in the idle state so that the first data in the first transit nodes are distributed to the sixth transit node, and processing the processed first data to the destination server is synchronized.

In order to achieve the above object, according to another aspect of the present application, there is provided a computer-readable storage medium storing a program, wherein the program performs the data synchronization method of any one of the above.

To achieve the above object, according to another aspect of the present application, there is provided a processor for executing a program, wherein the program executes the data synchronization method according to any one of the above.

The method comprises the steps of obtaining a plurality of target data and a plurality of first transfer nodes through the second transfer nodes, wherein each target data is data to be synchronized to a target server by a plurality of source servers in a cluster environment, determining the quantity of the plurality of target data and the efficiency of processing the data by each first transfer node through the second transfer nodes, distributing each target data to each first transfer node through the second transfer nodes according to the quantity of the plurality of target data and the efficiency of processing the data by each first transfer node, so that each first transfer node processes each target data to obtain a plurality of processed target data, synchronizing the processed plurality of target data to the target servers, and solving the problem that the data of the source servers in the cluster environment are synchronized to the target servers by adopting a method of statically binding the source servers and the transfer nodes for processing the data in the related technology, so that the effect of data synchronization is poor. And distributing each target data to each first transit node through the second transit node according to the determined quantity of the plurality of target data and the data processing efficiency of each first transit node, so that each first transit node processes each target data to obtain a plurality of processed target data, and synchronizing the plurality of processed target data to a target server, thereby improving the high efficiency and stability of data synchronization and further improving the effect of data synchronization.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

FIG. 1 is a flow chart of a method of synchronizing data provided in accordance with an embodiment of the present application;

FIG. 2 is a schematic diagram of a deployment data synchronization apparatus provided in accordance with an embodiment of the present application;

FIG. 3 is a schematic diagram of task data forwarding performed by a scheduling node when data to be processed occupies a large amount of bandwidth in an embodiment of the present application;

FIG. 4 is a schematic diagram of task data forwarding performed by a scheduling node when data to be processed does not need to occupy a large amount of bandwidth in an embodiment of the present application;

FIG. 5 is a flowchart I of a method for synchronizing data provided in accordance with an embodiment of the present application;

FIG. 6 is a flow chart of an alternative method of synchronizing data provided in accordance with an embodiment of the present application;

Fig. 7 is a schematic diagram of a data synchronization device according to an embodiment of the present application.

Detailed Description

It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the application herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example 1

In the following, the present application will be described in conjunction with the preferred implementation steps, fig. 1 is a flowchart of a data synchronization method according to an embodiment of the present application, where, as shown in fig. 1, the method is applied in a data synchronization system, where the data synchronization system includes at least a plurality of first transit nodes and second transit nodes, the first transit nodes are used for processing data, the second transit nodes are used for scheduling the plurality of first transit nodes, if there is a fault node in the plurality of first transit nodes, transit nodes other than the fault node in the first transit nodes are used to replace the fault node, and if the second transit node fails, any one of the plurality of first transit nodes is used to replace the second transit node, and the following steps are included:

step S101, a plurality of target data and a plurality of first transfer nodes are obtained through a second transfer node, wherein each target data is data to be synchronized to a target server by a plurality of source servers in a cluster environment.

For example, fig. 2 is a schematic diagram of a deployment data synchronization device according to an embodiment of the present application, as shown in fig. 2, the number of destination servers in the data synchronization device may be multiple, and since the embodiment of the present application does not pay attention to the binding relationship between the destination servers and each transit node, the destination servers may be regarded as one.

For example, suppose there are m source servers and n transit nodes, and the transit nodes synchronize data from the source servers, process the data, and send the processed data to the destination server.

In addition, one of the transit nodes is regarded as a scheduling node (the second transit node described above), the scheduling node is responsible for scheduling of all the transit nodes, and the other nodes are responsible for data processing, also referred to as data processing nodes (the first transit node described above). And for any transit node, the internal structure of the transit node is composed of a network module and a multi-process data processing module, wherein the network module is responsible for receiving and transmitting data, and the multi-process data processing module is responsible for processing data. The scheduling node is not different from the data processing node, but is selected in actual operation for scheduling other nodes, and any transit node can be used as the scheduling node if necessary. That is, if one or more data processing nodes fail, the scheduling node may record the failed data processing node and select the data processing node without the failure to take over the work of the failed data processing node. If the current scheduling node fails, a new scheduling node may be selected from the data processing nodes according to the number sequence, for example, if the scheduling node with the number of 1 fails, the node with the number of 2 is the new scheduling node, and if the scheduling node with the number of 2 fails, the node with the number of 3 is the new scheduling node.

For example, the scheduling node determines the data (the plurality of target data described above) that needs to be synchronized from a plurality of source servers to a destination server in the cluster environment, and determines which data processing nodes are present in the data synchronization apparatus.

Step S102, determining the quantity of a plurality of target data and the efficiency of each first transit node for processing the data through the second transit nodes.

For example, the amount of data that needs to be synchronized from a plurality of source servers to a destination server in the cluster environment is determined by the scheduling node, and the amount of data processed per unit time of each data processing node, that is, the average processing rate (the efficiency of processing data per first transit node described above) is calculated.

Step S103, distributing each target data to each first transit node through the second transit node according to the quantity of the plurality of target data and the efficiency of each first transit node for processing the data, so that each first transit node processes each target data to obtain a plurality of processed target data, and synchronizing the plurality of processed target data to a target server.

For example, the scheduling node allocates data to be synchronized to each data processing node for processing according to the amount of data to be synchronized from a plurality of source servers to a destination server in the cluster environment and the amount of data processed per unit time (average processing rate) of each data processing node. And then each data processing node processes the data allocated to the data processing node after receiving the data allocated to the data processing node, so as to obtain processed data, and then synchronizes the processed data to the destination server.

Through the steps S101 to S103, each target data is distributed to each first transit node through the second transit node according to the determined number of the plurality of target data and the efficiency of processing the data by each first transit node, so that each first transit node processes each target data to obtain a plurality of processed target data, and the plurality of processed target data are synchronized to the destination server, thereby improving the efficiency and stability of data synchronization and further improving the effect of data synchronization.

In order to quickly and accurately determine the second transit node, in the data synchronization method provided by the embodiment of the application, the second transit node can be determined by determining a plurality of target transit nodes, wherein the plurality of target transit nodes at least comprise a plurality of first transit nodes, and determining the second transit node from the plurality of target transit nodes so as to determine the number of the plurality of target data and the efficiency of each first transit node for processing the data through the second transit node, wherein the second transit node is a transit node except for the plurality of first transit nodes in the plurality of target transit nodes.

For example, as shown in fig. 2, there may be a plurality of transit nodes in the data synchronization device (i.e., corresponding to transit node 1, transit node 2, transit node 3, transit node 4, etc. in fig. 2), and transit node 1, transit node 2, transit node 3, transit node 4, etc. in fig. 2 may be the target transit node described above. Any one of the transit nodes 1,2, 3,4, etc. may then be selected as a scheduling node (the second transit node described above), and the scheduling node determines the amount of data to be synchronized from the plurality of source servers to the destination server in the cluster environment, and calculates the amount of data processed per unit time of each data processing node, that is, the average processing rate (the efficiency of processing data by each first transit node described above). If the transit node 1 among the transit nodes 1,2, 3,4, etc. is used as the scheduling node, the remaining transit nodes 2,3, 4, etc. are used as the data processing nodes (the plurality of first transit nodes described above).

In summary, the multi-node synchronization device is designed, one node is selected as a scheduling node, the scheduling node is utilized to uniformly butt-joint each server of the cluster, the processing node completes processing work based on load balancing, deployment is easy, and meanwhile, the shortage of single node load can be avoided.

In order to quickly and accurately determine the second transfer node, in the data synchronization method provided by the embodiment of the application, the second transfer node can be determined by acquiring the number information corresponding to each target transfer node, determining the transfer node with the smallest number in the plurality of target transfer nodes based on the number information corresponding to each target transfer node, and taking the transfer node with the smallest number in the plurality of target transfer nodes as the second transfer node.

For example, the transit nodes may be automatically numbered according to the time of joining the system, and the transit node with the number 1 may be automatically selected as a scheduling node, where the scheduling node is responsible for scheduling of the node and allocation of tasks, and is not responsible for data processing.

By the scheme, the scheduling node can be determined from a plurality of transit nodes quickly and accurately.

In order to quickly and accurately allocate each target data to each first transit node, in the data synchronization method provided by the embodiment of the application, each target data can be allocated to each first transit node by judging whether the data amount of the plurality of target data is larger than the preset data amount, allocating each target data to each first transit node through the second transit node according to the number of the plurality of target data and the efficiency of processing the data of each first transit node if the data amount of the plurality of target data is larger than the preset data amount, so that the first transit node obtains the plurality of target data from the plurality of source servers and processes the plurality of target data, and allocating each target data obtained from the plurality of source servers to each first transit node according to the number of the plurality of target data and the efficiency of processing the data of each first transit node if the data amount of the plurality of target data is not larger than the preset data amount.

For example, when the data to be processed occupies a large amount of bandwidth and the transmission capacity of the scheduling node is limited, the scheduling node performs task allocation, and the scheduling node acquires task related information, such as the number of the data, from the source server, and does not actually synchronize the data. When the data synchronous work is required to be distributed to other transfer nodes, a message is sent to inform the other transfer nodes of the information such as the source server corresponding to the task, the number and the number of the data, and the transfer nodes acquire the data from the source server by themselves. And when the data to be processed occupies a large amount of bandwidth, the scheduling node performs a task data forwarding schematic diagram, as shown in fig. 3.

For example, when the data to be processed does not occupy a large amount of bandwidth and the transmission capacity of the scheduling node is sufficient, the scheduling node performs task allocation, the scheduling node acquires task related information and the data from the source server. When the data synchronization work is required to be distributed to other transfer nodes, a message is sent to transfer the data to the other transfer nodes, and no connection is required between the other transfer nodes and the source server. And when the data to be processed does not occupy a large amount of bandwidth, the scheduling node performs a task data forwarding schematic diagram, as shown in fig. 4.

In summary, two deployment modes can be set according to the synchronous data types, when the data occupy a large amount of bandwidth, the scheduling node performs task assignment, and when the data do not occupy a large amount of bandwidth, the scheduling node performs actual task data distribution.

In order to quickly and accurately obtain the processed first data, the method for synchronizing the data provided by the embodiment of the application can also obtain the processed first data by judging whether a fifth transfer node in a plurality of first transfer nodes is in an idle state, distributing the first data in a plurality of target data to the fifth transfer node if the fifth transfer node in the plurality of first transfer nodes is in the idle state, processing the first data by the fifth transfer node to obtain the processed first data, synchronizing the processed first data to a destination server, judging whether the sixth transfer node in the plurality of first transfer nodes is in the idle state if the fifth transfer node in the plurality of first transfer nodes is not in the idle state, distributing the first data in the plurality of target data to the sixth transfer node if the sixth transfer node in the plurality of first transfer nodes is in the idle state, and synchronizing the processed first data to the destination server.

For example, when the system is initially running, a relatively simple polling load balancing method is adopted, and the scheduling node acquires tasks from each source server, traverses other transfer nodes each time, and distributes a fixed amount of tasks to each transfer node. And inquiring whether the target transfer node is idle before distribution, if so, distributing the task to the transfer node, and if not, inquiring the next transfer node.

Through the scheme, the scheduling node can rapidly distribute the data to be synchronized to each source server for processing according to a polling load balancing method.

Fig. 5 is a flowchart of a method for synchronizing data according to an embodiment of the present application, as shown in fig. 5, in the method for synchronizing data according to an embodiment of the present application, allocating, by a second transit node, each target data to each first transit node according to the number of the plurality of target data and the efficiency of processing the data by each first transit node includes:

step S501, calculating the data quantity distributed to each first transit node according to the quantity of a plurality of target data and the data processing efficiency of each first transit node;

step S502, each target data is allocated to each first transit node based on the data amount allocated to each first transit node.

For example, when performing the index monitoring, the idle time may be calculated first, specifically, for any transit node, the task allocated by the scheduling node is accepted every time interval. The transfer node receives the task distributed by the scheduling node, processes the task after receiving the task, waits for the next arrival of the task after finishing the processing, and the scheduling node can inquire whether the node is idle for many times during the processing, because the current task is not processed, and the scheduling node cannot distribute a new task for the current task. Now, assume that the time for receiving a task at a time is t1, the time for completing the task at the time is t2, and the time for receiving the task at the next time is t 3. t2 is between t1 and t3, and its idle time tidle=t3-t 2 in the time t1 to t 3.

Then, the average processing rate is calculated, and μ is set as the amount of data processed per unit time of the transit node, that is, the average processing rate. In addition, different types of data generally adopt different measurement units, for example, when the embodiment of the application processes the log of the source server through the transit node, the number of the log can be used as the measurement unit, and the calculation formula is as follows:

wherein p is the sum of the data amounts processed by the transit node in the statistical time, tidle is the sum of idle time of the transit node in the statistical time (i.e. the time period of actually not processing the data), and T is the statistical time.

For example, the scheduling node acquires tasks from each source server, and assigns tasks with μ as a weight weighting. Assuming that the scheduling node obtains n pieces of data for each origin server at a time, the μ value for each processing node is μ1, μ2,..mu.n-1, respectively.

The amount of data allocated to the transit node k is:

then, the scheduling node may allocate data to be synchronized to the destination server to each data processing relay node according to the calculated data amount allocated to each data processing relay node.

In summary, dynamic load balancing is performed based on the statistical indexes, so that the synchronous tasks can be shared according to the load capacity of the nodes, and the load of each node can be reduced as much as possible.

In order to quickly and accurately synchronize a plurality of target data to a destination server, the method for synchronizing the plurality of target data to the destination server can further include the steps of obtaining data quantity to be synchronized to the destination server in unit time of each source server, calculating to obtain average queuing length of each first transfer node according to the data quantity to be synchronized to the destination server in unit time of each source server, wherein the average queuing length is used for representing accumulation conditions of the plurality of target data in a first time period, determining real-time queuing length of each first transfer node, wherein the real-time queuing length is used for representing load pressure conditions of each first transfer node in a second time period, the length of the second time period is smaller than that of the first time period, and sending reminding information to a target object based on the average queuing length and the real-time queuing length of each first transfer node, wherein the reminding information is used for adjusting strategies for synchronizing the plurality of target data to the destination server.

For example, the scheduling node may record an index of λ, the amount of data that needs to be synchronized per unit time for all source servers, i.e. the average arrival rate, in units of measure equal to μ. The formula for lambda is:

wherein T is the statistical time, q is the sum of the data amounts needed to be synchronized by all source servers in the statistical time.

The scheduling node may then focus on whether the current load situation is excessive, calculate the relevant index and provide it to the user. The calculation indexes comprise a real-time queuing length L1 and an average queuing length L2, wherein the real-time queuing length L1 is the data quantity recorded by a scheduling node and not capable of completing distribution currently, and reflects the load pressure condition of a transfer node in a current short time.

The average queuing length L2 reflects the data accumulation condition in a longer time, reflects whether the arrangement of the transit node is reasonable, and if the value is larger, suggests to increase the transit node. The calculation formula is as follows:

In the formula, lambda is the data quantity which needs to be synchronized in unit time of all source servers, mu is the sum of the data quantity processed in unit time of the transfer node, namely

Through the scheme, when the data in the source server is synchronized to the destination server, a user can be timely reminded of adjusting the data synchronization method, so that the stability and the high efficiency of the data synchronization are improved.

In order to rapidly and accurately perform fault processing on the relay node, the data synchronization method provided by the embodiment of the application can also perform fault processing on the relay node by judging whether the second relay node fails or not, and determining a fourth relay node from a plurality of first relay nodes if the second relay node fails, wherein the fourth relay node is used for replacing the second relay node to schedule the relay nodes except the fourth relay node in the plurality of first relay nodes.

For example, the device designed by the embodiment of the application not only can realize dynamic load balancing according to the queuing model, but also can have a fault replacement function.

I.e. when any one of the transit nodes in the device fails, the device is able to achieve a rapid failure override.

If the current dispatching node fails, the device executes the following processes:

(1) Selecting new scheduling nodes according to the number sequence, for example, if the scheduling node with the number of 1 fails, the node with the number of 2 is the new scheduling node, and if the scheduling node with the number of 2 fails, the node with the number of 3 is the new scheduling node;

(2) Re-executing the initialization deployment process, namely re-deploying the data synchronization device, and adopting a new transfer node as a scheduling node;

(3) The index monitoring process is re-executed, namely after initialization is finished, the system operates and starts to observe, each processing node records mu, the average processing rate is the data quantity processed by the transit node in unit time, and the scheduling node records lambda, the average arrival rate is the data quantity needed to be synchronized in unit time of all source servers;

(4) And re-executing the dynamic adjustment process, namely dynamically adjusting the data distributed to each source server by the scheduling node according to the index change monitored in the index monitoring process.

In summary, the scheduling node and the statistical node have the same structure, and can be replaced by each other, so that any one of the scheduling node and the statistical node has a fault and can be quickly replaced.

In order to rapidly and accurately perform fault processing on the relay node, the data synchronization method provided by the embodiment of the application can also perform fault processing on the relay node by judging whether a fault node exists in a plurality of first relay nodes, determining a plurality of third relay nodes except the fault node from the plurality of first relay nodes if the fault node exists in the plurality of first relay nodes, wherein the third relay nodes are used for processing a plurality of target data instead of the first relay nodes, acquiring the data processing efficiency of each third relay node, and distributing each target data to each third relay node according to the number of the plurality of target data and the data processing efficiency of each third relay node.

For example, if one or more processing nodes fail, the apparatus performs the following:

The dispatching node records the processing node of the fault, continuously executes the index monitoring process, and updates the dynamic adjustment process as follows:

the data amount allocated to a certain transit node k (the transit node is a non-faulty node) is changed to:

where b represents the number of failed nodes.

In addition, no matter the scheduling node is failed or the processing node is failed, when the node is failed, the real-time queuing length L1 and the average queuing length L2 may be increased, and at this time, the user may increase the number of the transfer nodes, add a new transfer node into the device, and the newly added device is automatically used as the processing node, where the adding process is as follows:

(1) Re-executing an initial deployment process, namely re-deploying the data synchronization device, and adopting a new transfer node as a scheduling node or a data processing transfer node;

(2) The index monitoring process is re-executed, namely after initialization is finished, the system operates and starts to observe, each processing node records mu, the average processing rate is the data quantity processed by the transit node in unit time, and the scheduling node records lambda, the average arrival rate is the data quantity needed to be synchronized in unit time of all source servers;

(3) And re-executing the dynamic adjustment process, namely dynamically adjusting the data distributed to each source server by the scheduling node according to the index change monitored in the index monitoring process, wherein the n value in the formula comprises a newly added transit node.

In summary, the scheduling node and the statistical node may replace each other, and any one of them fails and may be quickly replaced.

For example, fig. 6 is a flowchart of an alternative method for synchronizing data according to an embodiment of the present application, and as shown in fig. 6, the alternative method for synchronizing data includes the following steps:

1. deployment device

The embodiment of the application is deployed as follows, wherein the number of the destination servers can be multiple, and the binding relationship between the destination servers and each transit node is not concerned, so that the destination servers are regarded as one.

Assuming that there are m source servers and n transit nodes, the transit nodes synchronize data from the source servers, process the data, and send the data to the destination server.

Among all the transit nodes, one of the transit nodes is regarded as a scheduling node, the scheduling node is responsible for scheduling of all the transit nodes, and other nodes are responsible for data processing, which is also called a data processing node.

For any transit node, the internal structure of the transit node is composed of a network module and a multi-process data processing module, wherein the network module is responsible for receiving and transmitting data, and the multi-process data processing module is responsible for processing data. The scheduling node is not different from the data processing node, but is selected in actual operation for scheduling other nodes, and any transit node can be used as the scheduling node if necessary.

And according to the time of joining the system, the transfer nodes are automatically numbered, the transfer node with the number of 1 is automatically selected as a scheduling node, and the scheduling node is responsible for scheduling of the nodes and allocation of tasks and is not responsible for data processing.

When the data to be processed occupies a large amount of bandwidth and the transmission capacity of the scheduling node is limited, the scheduling node performs task allocation, and the process is that the scheduling node acquires task related information, such as the number of the data, from the source server, and the data is not actually synchronized. When the data synchronous work is required to be distributed to other transfer nodes, a message is sent to inform the other transfer nodes of the information such as the source server corresponding to the task, the number and the number of the data, and the transfer nodes acquire the data from the source server by themselves.

When the data to be processed does not occupy a large amount of bandwidth and the transmission capacity of the scheduling node is sufficient, the scheduling node performs task allocation, and the process is that the scheduling node acquires the task related information and the data from the source server. When the data synchronization work is required to be distributed to other transfer nodes, a message is sent to transfer the data to the other transfer nodes, and no connection is required between the other transfer nodes and the source server.

When the system is initially operated, a relatively simple polling load balancing method is adopted, and the method comprises the following steps that a scheduling node acquires tasks from each source server, traverses other transfer nodes each time, and distributes a fixed amount of tasks to each transfer node. And inquiring whether the target transfer node is idle before distribution, if so, distributing the task to the transfer node, and if not, inquiring the next transfer node.

2. Index monitoring

(1) Calculation of idle time

For any transit node, the receiving dispatch node allocates tasks every time a period of time.

The transfer node receives the task distributed by the scheduling node, processes the task after receiving the task, waits for the next arrival of the task after finishing the processing, and the scheduling node can inquire whether the node is idle for many times during the processing, because the current task is not processed, and the scheduling node cannot distribute a new task for the current task. Now, assume that the time for receiving a task at a time is t1, the time for completing the task at the time is t2, and the time for receiving the task at the next time is t 3. t2 is between t1 and t3, and its idle time tidle=t3-t 2 in the time t1 to t 3.

(2) Calculation of average processing Rate and average arrival Rate

After the initialization of the 1 st section is finished, the system operates and starts to observe, and each processing node records the following indexes:

Mu, the data amount processed by the transit node in unit time, namely the average processing rate, and different types of data usually adopt different measuring units, for example, when the embodiment of the application processes the log of the source server through the transit node, the log number can be used as the measuring unit. The calculation formula is as follows:

The scheduling node records the following indexes:

Lambda is the amount of data that all source servers need to synchronize per unit time, i.e. the average arrival rate, with the unit of measure being the same as mu. The formula for lambda is:

3. Dynamic adjustment

The purpose of dynamic adjustment according to the detection index is to distribute tasks according to the load capacity of the nodes, reduce the burden of each node, reduce the average processing time of data and avoid data retention. When the data volume required to be processed by the system is too large, each transit node is insufficient to bear the burden, the scheduling node gives an early warning.

If the polling method of section 1 is always used, there is a high possibility that a certain node is overloaded for a long time. Thus requiring dynamic adjustment in accordance with the monitoring index of section 2.

The scheduling node acquires tasks from each source server, and the tasks are weighted and allocated by taking mu as a weight. Assuming that the scheduling node obtains n pieces of data for each origin server at a time, the μ value for each processing node is μ1, μ2,..mu.n-1, respectively.

The amount of data allocated to the transit node k is:

Transit nodes count μ and λ at certain periods, and update the values of μ and λ when there is a significant difference in the t-test.

The scheduling node pays attention to whether the current load situation is overweight, calculates the related index and provides the related index for the user. The calculation indexes comprise a real-time queuing length L1 and an average queuing length L2, wherein the real-time queuing length L1 is the data quantity recorded by a scheduling node and not capable of completing distribution currently, and reflects the load pressure condition of a transfer node in a current short time.

In the formula, lambda is the data quantity which needs to be synchronized in unit time of all source servers

Mu, sum of data amounts processed per unit time of transit node, i.e.

4. Fault displacement

The device designed by the embodiment of the application not only realizes dynamic load balancing according to the queuing model, but also has a fault replacement function.

(1) The new scheduling node is selected according to the number sequence, for example, if the scheduling node with the number 1 fails, the node with the number 2 is the new scheduling node, and if the scheduling node with the number 2 fails, the node with the number 3 is the new scheduling node.

(2) The initialization procedure described in section 1 is re-performed.

(3) The index monitoring procedure described in section 2 is re-executed.

(4) The dynamic adjustment procedure described in section 3 is re-executed.

If one or more processing nodes fail, the device performs the following process:

the dispatching node records the processing node of the fault, continues to execute the index monitoring process described in the section 2, and updates the dynamic adjustment process described in the section 3 as follows:

where b represents the number of failed nodes.

Whether the scheduling node is faulty or the processing node is faulty, when the node is faulty, the real-time queuing length L1 and the average queuing length L2 may be increased, and at this time, the user may increase the number of the transfer nodes accordingly, add a new transfer node to the device, and the newly added device automatically serves as the processing node, where the adding process is as follows:

(1) The initialization procedure described in section 1 is re-performed.

(2) The index monitoring procedure described in section 2 is re-executed.

(3) The dynamic adjustment process described in section 3 is re-executed and the n value in the above formula will contain the newly added transit node.

For example, in the auditing and control of network security, security vendors often synchronize users from a cluster-deployed authentication server by deploying proxy software in the internal network that forwards data to the network security devices to help the network security devices complete real-name auditing and control of traffic. In network security audit, logs of different network security devices are generally different, and in order to collect the logs to a log platform for unified management, a transfer node can be set to perform unified processing and formatting on the logs. In a small network, a cluster-deployed server 1 stores resources which can be accessed by users, a server 2 is used for reversely acting user requests to provide resources for the users, and a transfer node is arranged between the server 1 and the server 2 to perform processing operations such as data decoding, classification, formatting and the like.

Therefore, by the method provided by the embodiment of the application, a plurality of transfer nodes can be used for dynamically interfacing with a plurality of source servers in the cluster environment, and data can be processed and forwarded to the destination server. And the binding relation between the processing efficiency of the transfer node and the data volume of the source server is dynamically adjusted based on the processing efficiency of the transfer node, so that the high efficiency and the stability of data synchronization can be improved.

In summary, the method for synchronizing data provided by the embodiment of the application acquires a plurality of target data and a plurality of first transfer nodes through the second transfer nodes, wherein each target data is data to be synchronized to a target server by a plurality of source servers in a cluster environment, determines the quantity of the plurality of target data and the efficiency of each first transfer node for processing the data through the second transfer nodes, distributes each target data to each first transfer node according to the quantity of the plurality of target data and the efficiency of each first transfer node for processing the data through the second transfer nodes, so that each first transfer node processes each target data to obtain a plurality of processed target data, and synchronizes the plurality of processed target data to the target servers. And distributing each target data to each first transit node through the second transit node according to the determined quantity of the plurality of target data and the data processing efficiency of each first transit node, so that each first transit node processes each target data to obtain a plurality of processed target data, and synchronizing the plurality of processed target data to a target server, thereby improving the high efficiency and stability of data synchronization and further improving the effect of data synchronization.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.

Example 2

The embodiment of the application also provides a data synchronization device, and the data synchronization device of the embodiment of the application can be used for executing the data synchronization method provided by the embodiment of the application. The following describes a data synchronization device provided by an embodiment of the present application.

Fig. 7 is a schematic diagram of a data synchronization apparatus according to an embodiment of the present application. As shown in fig. 7, the device is applied to a data synchronization system, the data synchronization system at least comprises a plurality of first transfer nodes and a second transfer node, the first transfer nodes are used for processing data, the second transfer nodes are used for scheduling the plurality of first transfer nodes, if a fault node exists in the plurality of first transfer nodes, the transfer nodes except the fault node in the first transfer nodes are used for replacing the fault node, if the second transfer node breaks down, any one of the plurality of first transfer nodes is used for replacing the second transfer node, and the device comprises a first acquisition module 701, a first determination module 702 and a first processing module 703.

Specifically, the first obtaining module 701 is configured to obtain, through the second transit node, a plurality of target data and a plurality of first transit nodes, where each target data is data to be synchronized to a destination server by a plurality of source servers in a cluster environment;

A first determining module 702, configured to determine, by the second transit node, the number of the plurality of target data and the efficiency of processing the data by each first transit node;

The first processing module 703 is configured to allocate each target data to each first transit node through the second transit node according to the number of the plurality of target data and the efficiency of processing the data by each first transit node, so that each first transit node processes each target data to obtain a plurality of processed target data, and synchronize the plurality of processed target data to the destination server.

In summary, in the data synchronization device provided by the embodiment of the application, the first acquisition module 701 acquires a plurality of target data and a plurality of first transfer nodes through the second transfer nodes, wherein each target data is data to be synchronized to a destination server by a plurality of source servers in a cluster environment, the first determination module 702 determines the number of the plurality of target data and the efficiency of processing the data by each first transfer node through the second transfer nodes, and the first processing module 703 distributes each target data to each first transfer node according to the number of the plurality of target data and the efficiency of processing the data by each first transfer node through the second transfer nodes, so that each first transfer node processes each target data to obtain a plurality of processed target data, and synchronizes the processed plurality of target data to the destination server, thereby solving the problem that in the related art, the data synchronization effect is poor due to the adoption of a method of static binding of the source servers in the cluster environment and the transfer nodes for processing the data. And distributing each target data to each first transit node through the second transit node according to the determined quantity of the plurality of target data and the data processing efficiency of each first transit node, so that each first transit node processes each target data to obtain a plurality of processed target data, and synchronizing the plurality of processed target data to a target server, thereby improving the high efficiency and stability of data synchronization and further improving the effect of data synchronization.

Optionally, in the data synchronization device provided by the embodiment of the application, the first processing module comprises a first calculating unit, a first distributing unit and a first distributing unit, wherein the first calculating unit is used for calculating the data quantity distributed to each first transit node according to the quantity of a plurality of target data and the efficiency of processing the data by each first transit node, and the first distributing unit is used for distributing each target data to each first transit node based on the data quantity distributed to each first transit node.

Optionally, in the data synchronization device provided by the embodiment of the application, the device further comprises a second determining module, configured to determine a plurality of target transit nodes before determining the number of the plurality of target data and the efficiency of each first transit node for processing the data by the second transit node, wherein the plurality of target transit nodes at least comprise a plurality of first transit nodes, and a third determining module, configured to determine the second transit node from the plurality of target transit nodes, and determine the number of the plurality of target data and the efficiency of each first transit node for processing the data by the second transit node, wherein the second transit node is a transit node except for the plurality of first transit nodes in the plurality of target transit nodes.

Optionally, in the data synchronization device provided by the embodiment of the application, the device further comprises a first judging module, a fourth determining module, a second acquiring module and a first distributing module, wherein the first judging module is used for judging whether a fault node exists in the first relay nodes or not after each target data is distributed to each first relay node through the second relay node according to the quantity of the target data and the data processing efficiency of each first relay node, the fourth determining module is used for determining a plurality of third relay nodes except the fault node from the first relay nodes if the fault node exists in the first relay nodes, the third relay nodes are used for replacing the first relay nodes to process the target data, the second acquiring module is used for acquiring the data processing efficiency of each third relay node, and the first distributing module is used for distributing each target data to each third relay node according to the quantity of the target data and the data processing efficiency of each third relay node.

Optionally, in the data synchronization device provided by the embodiment of the application, the device further comprises a second judging module, a fifth determining module and a fourth determining module, wherein the second judging module is used for judging whether the second transit node fails after the second transit node is determined from the plurality of target transit nodes, and the fifth determining module is used for determining the fourth transit node from the plurality of first transit nodes if the second transit node fails, wherein the fourth transit node is used for replacing the second transit node to schedule transit nodes except the fourth transit node in the plurality of first transit nodes.

Optionally, in the data synchronization device provided by the embodiment of the application, the first processing module comprises a first judging unit, a second distributing unit and a third distributing unit, wherein the first judging unit is used for judging whether the data volume of the plurality of target data is larger than the preset data volume, the second distributing unit is used for distributing each target data to each first transit node according to the number of the plurality of target data and the efficiency of processing the data of each first transit node through the second transit node, so that the first transit node can acquire the plurality of target data from the plurality of source servers and process the plurality of target data, the first acquiring unit is used for acquiring the plurality of target data from the plurality of source servers through the second transit node if the data volume of the plurality of target data is not larger than the preset data volume, and the third distributing unit is used for distributing each target data acquired from the plurality of source servers to each first transit node according to the number of target data and the efficiency of processing the data of each first transit node.

Optionally, in the data synchronization device provided by the embodiment of the application, the third determining module comprises a second obtaining unit, a first determining unit and a second determining unit, wherein the second obtaining unit is used for obtaining the number information corresponding to each target transfer node, the first determining unit is used for determining the transfer node with the smallest number in the plurality of target transfer nodes based on the number information corresponding to each target transfer node, and the second determining unit is used for taking the transfer node with the smallest number in the plurality of target transfer nodes as the second transfer node.

Optionally, the data synchronization device provided by the embodiment of the application further comprises a third acquisition module, a first calculation module and a first sending module, wherein the third acquisition module is used for acquiring the data quantity to be synchronized to the destination server in unit time of each source server, the first calculation module is used for calculating the average queuing length of each first transfer node according to the data quantity to be synchronized to the destination server in unit time of each source server, the average queuing length is used for representing the accumulation condition of a plurality of target data in a first time period, the sixth determination module is used for determining the real-time queuing length of each first transfer node, the real-time queuing length is used for representing the load pressure condition of each first transfer node in a second time period, the length of the second time period is smaller than the length of the first time period, and the first sending module is used for sending reminding information to a target object based on the average queuing length and the real-time queuing length of each first transfer node, and the reminding information is used for reminding the target object to adjust the strategy of synchronizing the plurality of target data to the destination server.

Optionally, in the data synchronization device provided by the embodiment of the application, the device further comprises a third judging module, a second distributing module and a third distributing module, wherein the third judging module is used for judging whether a fifth transit node in the first transit nodes is in an idle state or not after the plurality of target data and the plurality of first transit nodes are acquired, the second distributing module is used for distributing the first data in the plurality of target data to the fifth transit node if the fifth transit node in the plurality of first transit nodes is in the idle state so as to enable the fifth transit node to process the first data, the processed first data is obtained, the processed first data is synchronized to a destination server, the fourth judging module is used for judging whether a sixth transit node in the plurality of first transit nodes is in the idle state or not if the fifth transit node in the plurality of first transit nodes is not in the idle state, and the third distributing module is used for distributing the first data in the plurality of target data to the sixth transit node if the sixth transit node in the plurality of first transit nodes is in the idle state so as to enable the sixth transit node to process the first data, and the processed first data is synchronized to the destination server.

The data synchronizing device includes a processor and a memory, where the first acquiring module 701, the first determining module 702, the first processing module 703, and the like are stored as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one, and the effect of data synchronization is improved by adjusting kernel parameters.

The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium having stored thereon a program which, when executed by a processor, implements a method of synchronizing data.

The embodiment of the invention provides a processor which is used for running a program, wherein the program runs to execute a data synchronization method.

The embodiment of the invention provides electronic equipment, which comprises a processor, a memory and a program which is stored in the memory and can be run on the processor, wherein when the processor executes the program, the processor realizes the following steps of acquiring a plurality of target data and a plurality of first transfer nodes through a second transfer node, wherein each target data is data to be synchronized to a destination server of a plurality of source servers in a cluster environment, determining the quantity of the plurality of target data and the efficiency of each first transfer node for processing the data through the second transfer node, and distributing each target data to each first transfer node according to the quantity of the plurality of target data and the efficiency of each first transfer node for processing the data through the second transfer node so as to enable each first transfer node to process each target data, acquire the processed plurality of target data and synchronize the processed plurality of target data to the destination server.

The processor further performs the steps of assigning each target data to each first transit node by the second transit node in accordance with the number of the plurality of target data and the efficiency of each first transit node in processing the data, including calculating the amount of data assigned to each first transit node in accordance with the number of the plurality of target data and the efficiency of each first transit node in processing the data, and assigning each target data to each first transit node in accordance with the amount of data assigned to each first transit node.

The processor, when executing the program, further performs the steps of determining a plurality of target transit nodes, wherein at least the plurality of first transit nodes are included in the plurality of target transit nodes, and determining the second transit node from the plurality of target transit nodes, to determine the number of the plurality of target data and the efficiency of each first transit node to process the data by the second transit node, wherein the second transit node is a transit node of the plurality of target transit nodes other than the plurality of first transit nodes, before determining the number of the plurality of target data and the efficiency of each first transit node to process the data by the second transit node.

The processor further comprises the steps of judging whether a fault node exists in the first transfer nodes after each target data is distributed to each first transfer node through the second transfer nodes according to the quantity of the target data and the data processing efficiency of each first transfer node, determining a plurality of third transfer nodes except the fault node from the first transfer nodes if the fault node exists in the first transfer nodes, wherein the third transfer nodes are used for processing the target data instead of the first transfer nodes, acquiring the data processing efficiency of each third transfer node, and distributing each target data to each third transfer node according to the quantity of the target data and the data processing efficiency of each third transfer node.

The processor further performs the steps of determining whether the second transit node fails after determining the second transit node from the plurality of target transit nodes, and determining a fourth transit node from the plurality of first transit nodes if the second transit node fails, wherein the fourth transit node is configured to schedule transit nodes other than the fourth transit node in the plurality of first transit nodes instead of the second transit node.

The processor further comprises the steps of distributing each target data to each first transfer node through the second transfer node according to the quantity of the target data and the data processing efficiency of each first transfer node, judging whether the quantity of the target data is larger than a preset quantity of data or not, if the quantity of the target data is larger than the preset quantity of data, distributing each target data to each first transfer node through the second transfer node according to the quantity of the target data and the data processing efficiency of each first transfer node, enabling the first transfer node to acquire the target data from the source servers and process the target data, and if the quantity of the target data is not larger than the preset quantity of data, acquiring the target data from the source servers through the second transfer node, and distributing the target data to each first transfer node according to the quantity of the target data and the data processing efficiency of each first transfer node.

The processor further realizes the following steps when executing the program, wherein the step of determining the second transfer node from the plurality of target transfer nodes comprises the steps of obtaining the number information corresponding to each target transfer node, determining the transfer node with the smallest number in the plurality of target transfer nodes based on the number information corresponding to each target transfer node, and taking the transfer node with the smallest number in the plurality of target transfer nodes as the second transfer node.

The method further comprises the steps of obtaining data quantity to be synchronized to a destination server in unit time of each source server, calculating average queuing length of each first transfer node according to the data quantity to be synchronized to the destination server in unit time of each source server, determining real-time queuing length of each first transfer node, wherein the real-time queuing length is used for representing accumulation conditions of the plurality of target data in a first time period, the load pressure condition of each first transfer node in a second time period, the length of the second time period is smaller than that of the first time period, and sending reminding information to a target object based on the average queuing length and the real-time queuing length of each first transfer node, wherein the reminding information is used for reminding the target object to adjust a strategy for synchronizing the plurality of target data to the destination server.

The method further comprises the steps of judging whether a fifth transfer node in the first transfer nodes is in an idle state or not after the target data and the first transfer nodes are acquired, distributing the first data in the target data to the fifth transfer node if the fifth transfer node in the first transfer nodes is in the idle state, enabling the fifth transfer node to process the first data to obtain processed first data, synchronizing the processed first data to the destination server, judging whether a sixth transfer node in the first transfer nodes is in the idle state if the fifth transfer node in the first transfer nodes is not in the idle state, distributing the first data in the target data to the sixth transfer node if the sixth transfer node in the first transfer nodes is in the idle state, enabling the sixth transfer node to process the first data, synchronizing the processed first data to the destination server, and processing the first data to the destination server.

The device herein may be a server, PC, PAD, cell phone, etc.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims

1. The data synchronization method is applied to a data synchronization system, and is characterized in that the data synchronization system at least comprises a plurality of first transit nodes and second transit nodes, wherein the first transit nodes are used for processing data, the second transit nodes are used for scheduling the plurality of first transit nodes, if a fault node exists in the plurality of first transit nodes, transit nodes except the fault node in the first transit nodes are adopted to replace the fault node, if the second transit node breaks down, any transit node in the plurality of first transit nodes is adopted to replace the second transit node, and the synchronization method specifically comprises the following steps:

Acquiring a plurality of target data and the plurality of first transfer nodes through the second transfer nodes, wherein each target data is data to be synchronized to a target server by a plurality of source servers in a cluster environment;

Determining, by the second transit node, the number of the plurality of target data and the efficiency of each first transit node to process the data;

Distributing each target data to each first transit node through the second transit node according to the number of the plurality of target data and the data processing efficiency of each first transit node, so that each first transit node processes each target data to obtain a plurality of processed target data, and synchronizing the plurality of processed target data to the target server;

wherein assigning each target data to each first transit node by the second transit node in accordance with the number of the plurality of target data and the efficiency of processing data by each first transit node comprises:

judging whether the data volume of the plurality of target data is larger than a preset data volume or not;

If the data quantity of the plurality of target data is larger than the preset data quantity, distributing each target data to each first transit node through the second transit node according to the quantity of the plurality of target data and the efficiency of processing the data of each first transit node, so that the first transit node obtains the plurality of target data from the plurality of source servers and processes the plurality of target data;

If the data quantity of the plurality of target data is not larger than the preset data quantity, acquiring the plurality of target data from the plurality of source servers through the second transfer node;

And distributing each target data acquired from the plurality of source servers to each first transit node according to the quantity of the target data and the data processing efficiency of each first transit node.

2. The method of claim 1, wherein assigning each target data to each first transit node by the second transit node in accordance with the number of the plurality of target data and the efficiency with which each first transit node processes data comprises:

calculating to obtain the data quantity distributed to each first transit node according to the quantity of the target data and the data processing efficiency of each first transit node;

each target data is assigned to each first transit node based on the amount of data assigned to each first transit node.

3. The method of claim 1, wherein prior to determining the number of the plurality of target data and the efficiency of processing data by each first transit node by the second transit node, the method further comprises:

Determining a plurality of target transfer nodes, wherein the plurality of target transfer nodes at least comprise the plurality of first transfer nodes;

Determining the second transit node from the plurality of target transit nodes to determine the number of the plurality of target data and the efficiency of each first transit node to process the data through the second transit node, wherein the second transit node is a transit node of the plurality of target transit nodes except the plurality of first transit nodes.

4. The method of claim 1, wherein after each target data is assigned to each first transit node by the second transit node in accordance with the number of the plurality of target data and the efficiency of processing data by each first transit node, the method further comprises:

judging whether a fault node exists in the first transit nodes or not;

If a fault node exists in the plurality of first transfer nodes, determining a plurality of third transfer nodes except the fault node from the plurality of first transfer nodes, wherein the third transfer nodes are used for replacing the first transfer nodes to process the plurality of target data;

Acquiring the data processing efficiency of each third transit node;

and distributing each target data to each third transit node according to the quantity of the target data and the data processing efficiency of each third transit node.

5. The method of claim 3, wherein after determining the second transit node from the plurality of target transit nodes, the method further comprises:

judging whether the second transfer node fails or not;

And if the second transit node fails, determining a fourth transit node from the plurality of first transit nodes, wherein the fourth transit node is used for replacing the second transit node to schedule transit nodes except the fourth transit node in the plurality of first transit nodes.

6. The method of claim 3, wherein determining the second transit node from the plurality of target transit nodes comprises:

acquiring the number information corresponding to each target transfer node;

determining a transit node with the smallest number in the plurality of target transit nodes based on the number information corresponding to each target transit node;

and taking the transit node with the smallest number in the plurality of target transit nodes as the second transit node.

7. The data synchronization device is characterized by being applied to a data synchronization system, at least comprising a plurality of first transfer nodes and a second transfer node, wherein the first transfer nodes are used for processing data, the second transfer nodes are used for scheduling the plurality of first transfer nodes, if a fault node exists in the plurality of first transfer nodes, the transfer nodes except the fault node in the first transfer nodes are adopted to replace the fault node, and if the second transfer node breaks down, any one of the plurality of first transfer nodes is adopted to replace the second transfer node, and the device comprises the following steps:

The first acquisition module is used for acquiring a plurality of target data and the plurality of first transfer nodes through the second transfer nodes, wherein each target data is data to be synchronized to a target server by a plurality of source servers in a cluster environment;

a first determining module, configured to determine, by the second transit node, the number of the plurality of target data and an efficiency of processing data by each first transit node;

The first processing module is used for distributing each target data to each first transit node through the second transit node according to the quantity of the plurality of target data and the efficiency of each first transit node for processing the data, so that each first transit node processes each target data to obtain a plurality of processed target data, and the plurality of processed target data are synchronized to the target server;

The device is further used for judging whether the data quantity of the plurality of target data is larger than a preset data quantity, if the data quantity of the plurality of target data is larger than the preset data quantity, distributing each target data to each first transit node through the second transit node according to the quantity of the plurality of target data and the efficiency of processing the data of each first transit node, enabling the first transit node to acquire the plurality of target data from the plurality of source servers and process the plurality of target data, and if the data quantity of the plurality of target data is not larger than the preset data quantity, acquiring the plurality of target data from the plurality of source servers through the second transit node, and distributing each target data acquired from the plurality of source servers to each first transit node according to the quantity of the plurality of target data and the efficiency of processing the data of each first transit node.

8. A computer-readable storage medium storing a program, wherein the program performs the method of synchronizing data according to any one of claims 1 to 6.

9. A processor for a method of synchronizing data, characterized in that the processor is arranged to run a program, wherein the program runs to perform the method of synchronizing data according to any of claims 1 to 6.