WO2014015492A1 - Procédé, dispositif et système de distribution de données - Google Patents

Procédé, dispositif et système de distribution de données Download PDF

Info

Publication number
WO2014015492A1
WO2014015492A1 PCT/CN2012/079173 CN2012079173W WO2014015492A1 WO 2014015492 A1 WO2014015492 A1 WO 2014015492A1 CN 2012079173 W CN2012079173 W CN 2012079173W WO 2014015492 A1 WO2014015492 A1 WO 2014015492A1
Authority
WO
WIPO (PCT)
Prior art keywords
distribution
logical data
data table
column
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2012/079173
Other languages
English (en)
Chinese (zh)
Inventor
吴向阳
曹俊亮
曹莉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN201280002465.XA priority Critical patent/CN103748578B/zh
Priority to PCT/CN2012/079173 priority patent/WO2014015492A1/fr
Publication of WO2014015492A1 publication Critical patent/WO2014015492A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2471—Distributed queries
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2453—Query optimisation
    • G06F16/24532—Query optimisation of parallel queries

Definitions

  • the present invention relates to the field of information technology, and in particular, to a data distribution method, apparatus, and system. Background technique
  • Parallel database system is a data storage technology that stores data content distribution on multiple data nodes. It can distribute a logical data table according to algorithms such as hash, range, and round-bin. On each data node.
  • the parallel database system queries the data content required by the user in parallel on each data node, and the query speed is fast compared with the non-parallel database system, and the data content is easy to manage.
  • the logical data table will contain multiple fields.
  • the parallel database system stores the logical data table on the data node with the content of one (or more) of the fields as the argument of the above algorithm, and the field as an independent variable. It is called the distribution column of the logical data table.
  • Embodiments of the present invention provide a data distribution method, apparatus, and system, which can save query time and improve query efficiency.
  • an embodiment of the present invention provides a data distribution method, including:
  • the control node Before the data query, the control node sets a distribution table creation indication according to the creation rule, where the distribution table creation indication carries the identifier ID of the logical data table and the distribution column identifier of the selected distribution column, wherein the selected distribution column is the a distribution column in the logical data table, the logical data table being a logical data table that has been created in the control node;
  • an embodiment of the present invention further provides a control node, including:
  • a processing unit configured to: before the data query, set a distribution table creation indication according to the creation rule, where the distribution table creation indication carries an identifier ID of the logical data table and a distribution column identifier of the selected distribution column, wherein the selected distribution column For the distribution column in the logical data table, the logical data table is a logical data table that has been created in the control node;
  • a sending unit configured to send, to the data node, the distribution table creation indication set by the processing unit, so that the data node creates a distribution table indicating the creation of the logical data table according to the distribution table.
  • an embodiment of the present invention further provides a system for data distribution, including:
  • a control node configured to: before the data query, set a distribution table creation indication according to the creation rule, where the distribution table creation indication carries an identifier ID of the logical data table and a distribution column identifier of the selected distribution column, wherein the selected distribution column For the distribution column in the logical data table, the logical data table is a logical data table that has been created in the control node, and the distribution table creation indication is sent to the data node; the data node is used for data query And receiving the distribution table creation indication sent by the control node, and creating a distribution table indicating the creation of the logical data table according to the distribution table.
  • the method, device and system for distributing data can store a plurality of logical data table distributions on respective data nodes according to the same distribution column according to the creation rule before the data query, and distribute and store the data nodes.
  • the established distribution table is used for subsequent data queries. It can be avoided that in the process of distributed column joint query, when the distribution columns of multiple distribution tables with query relationships are different, the problem of redistributing multiple logical data tables on each data node is needed, and the distribution column can be avoided.
  • the query delay caused by massive data migration during the process of joint query can improve query efficiency.
  • Figure 1 is a schematic diagram of the architecture of a parallel database system
  • FIG. 2 is a schematic diagram of a logical data table in a parallel database system
  • Figure 3 is a schematic diagram of a data node creation distribution table
  • Figure 4 is a schematic diagram of another data node creating a distribution table
  • FIG. 5 is a flowchart of a method for data distribution according to an embodiment of the present invention.
  • FIG. 6 is a flowchart of another method for data distribution in an embodiment of the present invention.
  • FIG. 7 is a schematic diagram of an application scenario in an embodiment of the present invention.
  • FIG. 8 is a schematic structural diagram of a control node according to an embodiment of the present invention.
  • FIG. 9 is a schematic diagram of a system for data distribution in an embodiment of the present invention. detailed description
  • Figure 1 shows an architecture of a parallel database system.
  • the control node is connected to three data nodes.
  • the control node and each data node have separate central processing units (Centra l Proces s ing Uni t, CPU for short) and memory (memory). , hard disk), data communication between nodes and nodes through high-speed networks (such as Ethernet, fiber-switched networks).
  • the control node is mainly used to: 1. Store the to-be-managed data distribution to each data node according to a field of the logical data table and a preset algorithm, where the logical data table is data with data structure attributes stored in the control node.
  • the logical data table is a logical basis for creating a distribution table for the data node; 2.
  • the data node is mainly used for:
  • the controlled node controls the distribution table of the logical data table according to the distribution column, performs data migration with other data nodes, and realizes distribution as an independent database node. Table storage and query.
  • logical data table A (hereinafter referred to as table A) stores the data of the student number and name
  • logical data table B (hereinafter referred to as table B) stores the student number.
  • course identification and grade data
  • Logical Data Table C (hereinafter referred to as Table C) stores data for course identification and course name.
  • Any column (or field or attribute) in the logical data table can be stored as a distribution column of the logical data table distribution, for example, the student number in Table B, the course ID, and the three fields of the table B.
  • the control node uses the student number field as a distribution column, and stores the data distribution of Table A to three data nodes by a hash modulo 3 algorithm.
  • the name corresponding to the student number 1 and the student number 1 (that is, the first line of data in the table A) is stored on the data node 1, and the student number 2 is divided by 3 or more 2
  • the second row of data in Table A is stored on the data node 2
  • the student number 3 is divided by 3 and 0, then the third row of data in the table A is stored on the data node 0, where the divisor in the hash modulo 3 algorithm 3 is the number of data nodes in the parallel database system.
  • control node uses the student number field and the course identification field as distribution columns, and stores the data distribution of Table B and Table C on three data nodes.
  • the control node uses the student number field as the distribution column (j) of the table B to distribute the table B by the hash modulo 3 algorithm, and uses the course identification field as the distribution column (k) of the table C through the hash modulo 3 algorithm pair table. C is distributed.
  • the data table established on each data node is called the distribution table of the logical data table, and the name of the distribution table may be represented by the logical data table identifier + the data node identifier + the distribution column identifier, for example, the data node 1 B+mdtl+j in the middle indicates that the data node 1 creates the distribution table of the table B with the student number field (j) as the distribution column.
  • the client executes the query statement select stu.name, course.
  • the query statement represents a joint query through the distribution columns of Table A and Table B (from A, B ), query the data of the name, course ID and grade, and the correspondence between the three (select stu.name, course-id ), where Table A and Table B are stored in the data by the student number field as the distribution column.
  • the result of the query is that the data node 1 queries the three course identifiers corresponding to the student number 1 and the corresponding three-door scores, and the data node 2 queries the three course identifiers corresponding to the student number 2 and the corresponding three-door scores, and the data node 0 queries The three course IDs corresponding to student number 3 and the corresponding three grades.
  • the three data nodes respectively report the query result to the control node, and control The node reports the received query result to the client to complete the query.
  • the control node needs to re-distribute the table B with the course identification field as the distribution column (since the table C has no student number field, in order to make the table B and the table C are distributed according to the same distribution column, the course identification field needs to be used as the distribution column.
  • the table B is re-distributed according to the hash modulo 3 algorithm, and the distribution table of the table B is re-created).
  • the distribution table recreated for Table B on the three data nodes is shown in Figure 4, where k in the distribution table name is the distribution column identifier representing the course identification field as the distribution column.
  • the control node controls the three data nodes for data migration, so that the data node obtains the data corresponding to the newly added entries in the self-reconstructed distribution table from other data nodes.
  • the distribution table B+mdt l+k in the data node two courses of the course 1 and the course 1 respectively corresponding to the student number 2 and the student number 3 are added, and the data node 1 is creating the distribution table.
  • B+mdt l+j does not involve these two rows of entries, so data node 1 does not get the data of these two rows of entries when creating the distribution table B+mdt l+j.
  • the data node 1 obtains the data of the course 1 and the course 1 corresponding to the student number 2 from the data node 2 (ie, the data corresponding to the first row entry in B+mdt2+j), and obtains the student number 3 corresponding from the data node 0.
  • the data of course 1 and course 1 that is, the data corresponding to the first row of B+mdt O+j).
  • the corresponding steps are also performed according to the characteristics of the own table entries, which are not performed here.
  • the control node can control the data node to query the course names of the three courses corresponding to each student number according to the distribution table of Table C and the distribution table of the reconstructed Table B.
  • the sum of the distribution tables corresponding to the same logical data table on each data node can fully reflect the The contents of the logical data table.
  • An embodiment of the present invention provides a data distribution method. As shown in FIG. 5, the method includes the following steps:
  • control node Before the data query, the control node sets an indication of the distribution table according to the creation rule.
  • the distribution table creation indication carries an identifier (Identity, ID for short) of the logical data table and a distribution column identifier of the selected distribution column, wherein the selected distribution column is a distribution column in the logical data table.
  • the logical data table is a logical data table that has been created in the control node.
  • the ID of the logical data table is used to uniquely identify a logical data table
  • the distribution column identifier is used to uniquely identify the selected distribution column.
  • the distribution table creation indication may carry one or more logical data table IDs and one or more distribution column identifiers. When carrying multiple distribution column identifiers, the multiple distribution column identifiers may be in a logical data table. Multiple distribution column identifiers, or multiple distribution column identifiers in multiple logical data tables.
  • B is the ID of the table B
  • j is the distribution column identifier with the student number field as the distribution column
  • mdt l is the identifier of the data node.
  • the control node sends a distribution table creation indication to the data node.
  • the data node creates a distribution table of the logical data table according to the logical data table ID and the distribution column identifier in the distribution table creation indication, and completes data migration between the data nodes.
  • the data distribution method can select a distribution column according to a creation rule in a data storage phase, and the distribution table created by the data node based on the selected distribution column is used for subsequent distributed column joint query.
  • the data node directly performs the joint query of the distributed columns according to the plurality of distribution tables created according to the predetermined distribution column, which can save the query time and improve the query efficiency.
  • the embodiment of the present invention further provides a data distribution method. As shown in FIG. 6, the method includes the following steps: 601. Before the data query, the control node creates an indication by using a rule setting distribution table for the statistical result of the logical data table in the preset period.
  • the control node adds at least one field of the logical data table as a selected distribution column according to the statistical result of the data of the logical data table, and adds the distribution column identifier of the selected distribution column and the ID of the logical data table to the distribution table creation indication. .
  • the statistical object of the control node is all the created logical data tables in the parallel database system, namely Table A, Table B and Table C in Figure 2.
  • the statistical result includes: in a preset period, the number of times the logical data table is queried, the ratio of the table data of the logical data table being queried to the total data of the logical data table, and the distribution column in the logical data table are queried. The number of times.
  • the number of times the logical data table is called is the number of times the logical data table participates in the distributed column joint query in a preset period; the ratio of the table data of the logical data table being called to the total data of the logical data table is In a preset period, the cumulative access amount of any row of table data in the logical data table accounts for the proportion of the total data of the logical data table. For example, a logical data table has three rows of table item data, and the second threshold is 120%. In the 5 minutes, the first row of the data in the logical data table is queried 4 times, then the query quantity of the row entry data is 4 rows (cumulative value), and the query quantity of the row entry data occupies the logical data.
  • any row of table item data is not limited to the distributed column joint query.
  • the number of times the distributed column is queried in the logical data table is the number of times that any one of the logical data tables is queried as a distributed column within a preset period, and the query is not limited to the distributed column joint query.
  • control node adds the ID of the logical data table and the distribution column identifier of all the fields (as distribution columns) in the logical data table to Distribution table creation instructions.
  • control node may set a distribution table creation indication according to any one of the foregoing three statistical results, for example:
  • the control node When the number of times the logical data table is queried and the ratio of the table data of the logical data table being queried to the total data of the logical data table do not reach the respective thresholds, the control node does not add the ID of the logical data table to the distribution table to be created. In the indication. When any one of the two conditions of the logical data table being queried or the ratio of the table data of the logical data table being queried to the total data of the logical data table reaches the corresponding threshold, the control node will logical data table The ID and the distribution column ID of all the fields in the logical data table (as distribution columns) are added to the distribution table creation indication.
  • the control node When the number of times the distributed column is queried reaches the third threshold in the logical data table, the control node adds the distribution column identifier of the distribution column whose number of queries reaches the third threshold and the ID of the logical data table to the distribution table creation indication.
  • control node may also combine the foregoing three statistical results as a basis for adding the logical data table ID and the distribution column identifier. For example, in a preset period of 5 minutes, if a logical data table satisfies the expression (Join Times>6) and (Pacces sed Lines Percent >180%) and (CFrenqence>8), then the logical data table is The ID and corresponding distribution column identifier are added to the distribution table creation indication. (Join Times>6) indicates that the logical data table is queried more than 6 times, (Pacces sed Lines Percent ⁇ 180%), indicating that the query quantity of an item data in the logical data table accounts for the total of the logical data table.
  • the proportion of the item data is greater than 180%.
  • CFRPqence>8 indicates that the number of times a certain column in the logical data table is queried is greater than 8 times, and indicates that the relationship between the three decision conditions is AND, that is, the above three are satisfied at the same time. Judgment conditions.
  • the recurring expression of a logical data table may also be (Pacces sed L ines Percent>5/ Join Times>50%) and (CFrenqence>6 ), where "/" represents or relationship, two decision conditions One can be.
  • the statement that the control node controls the data node to create the distribution table is: [Distribution on KEY column-name [, column_name, ⁇ ] ... a llowMult ipleDistr ibution , where column-name is the distribution ⁇ 1 J identifier, column_name [, Column—name,...] indicates that multiple distribution columns can be used as arguments to the distribution algorithm, and allowMultipleDistribution indicates that the distribution table is allowed to be created.
  • control node may add the ID of the logical data table and the distributed column identifier of the selected distribution column of the logical data table to the distribution table creation indication according to the creation instruction of the client.
  • the creation indication of the control node reception includes the logical data table ID selected by the querier or database administrator and the distribution column identifier of at least one distribution column in the logical data table.
  • the control node adds the logical data table ID in the creation indication and the distribution column ID of the selected distribution column to the distribution table creation indication.
  • the control node may also periodically delete the distribution table of the data node that is not queried or rarely queried. Specifically, the control node counts the number of times each distribution table in the data node is queried. If a distribution table whose number of times of query is less than the fourth threshold exists within a preset period, the control node sends a distribution table to the data node to which the distribution table belongs. Deleting the indication, the distribution table deletion indication carries the distribution column identifier of the distribution column corresponding to the distribution table whose number of times of query is less than the fourth threshold, and the ID of the logical data table to which the distribution column belongs. After receiving the distribution table deletion indication, the data node deletes the corresponding distribution table according to the logical data table ID and the distribution column identifier carried therein.
  • the control node is setting the distribution table.
  • the indication it may also be determined whether the data node has created a distribution table of the logical data table according to the ID of the logical data table and the distribution column identifier of the selected distribution column, where the ID of the logical data table is added to the control node
  • the distribution table creates an ID of the logical data table in the indication, and the distribution column identifier of the selected distribution column is a distribution column identifier added to the distribution table creation indication by the control node.
  • the control node does not add the ID of the logical data table and the distribution column identifier of the selected distribution column. Go to the distribution table creation directive.
  • the control node sends a distribution table creation indication to the data node.
  • the control node sends a distribution table creation indication to the data node, so that the data node creates a distribution table for the logical data table according to the ID of the logical data table in the distribution table creation indication, the distribution column identifier, and the preset distribution algorithm.
  • the distribution table created for Table B is shown in the table B+mdt l+k, B+mdt 2+k, and B+mdt O+k in Figure 4.
  • a data node can create a distribution table by using one field of the logical data table as a distribution column (as shown in Figure 3), or you can create a distribution table by using two or more fields of the logical data table as distribution columns. For example, the student number field and the grade field are summed as arguments of the modulo 3 algorithm.
  • the data node can create a maximum number of distribution tables for the number of fields in the logical data table.
  • the maximum number of distribution tables that the data node can create is an arrangement combination of at least two fields of the logical data table, and the number of distribution tables that can be created is greater than the logical data. The number of fields in the table.
  • the distribution algorithm used when the data node performs distribution or creates a distribution table includes, but is not limited to, a hash algorithm, a range algorithm, and a rotation algorithm.
  • the data node After the data node creates the distribution table according to the distribution table creation, it also needs to obtain the data that is included in the new distribution table but not stored in the data node from other data nodes, thereby completing the data migration between the data nodes.
  • the data node 1 after creating the distribution table B+mdt l+k, acquires the B+mdt 2+j table and the B+mdt O+j table sent by the data node 2 and the data node 0. All table entry data. Further, in order to reduce the amount of data migration, the data node 1 may also obtain only the data of the first row entry in the B+mdt2+j sent by the data node 2, and the first in the B+mdt O+j sent by the data node 0. The data of the row entry.
  • the control node After the data node creates the distribution table and performs data migration, when the client initiates the query request, the control node sends a query indication to the data node, so that the data node performs the joint query of the distributed column.
  • the query may be selected to query the less expensive distribution table, including but not limited to the amount of data migration.
  • the data table participating in the distributed column joint query on data node 1 is B+ Mdt l+k and C+mdt l+k
  • the two tables are distributed according to the same distribution column (course ID k).
  • the data node 1 can find and report to the control node the scores of the course 1 corresponding to the three student numbers.
  • data node 2 and data node 0 can find and report to the control node the scores of course 2 and course 3 corresponding to the three student numbers.
  • the control node feeds back the query results reported by the three data nodes to the client to complete the query.
  • the two data tables participate in the joint query of the distributed column as an example, and the data table of the joint query in the actual application may also be three or more.
  • the distribution table is started according to the query condition and the data migration is performed.
  • the data migration during the query process consumes a large amount of query time and reduces the query efficiency.
  • the data distribution method provided by the embodiment of the present invention can select a distribution column according to a statistical result of the preset data or an indication of the client in the data storage stage, and the distribution table created by the data node based on the selected distribution column is used for subsequent distribution.
  • Column joint query When the client inputs the query statement, the data node directly performs the joint query of the distributed columns according to the plurality of distribution tables created according to the predetermined distribution column, which can save the query time and improve the query efficiency.
  • the control node may instruct the data node to create the distribution table of the table B again according to the client indication or the statistical result.
  • the client sends a distribution table creation indication to the control node, indicating that the distribution table is created for the table B in the parallel database system.
  • the control node separately sends a distribution table creation indication to the three data nodes, where the distribution table creation indication carries an ID of the table B and an identifier of the distribution column (course identification field). 703, three
  • the data nodes create a distribution table that creates a table B according to the distribution table. 704.
  • the control node sends a data migration indication to each of the three data nodes. 705.
  • the data node performs data migration according to the data migration indication. 706.
  • the data node sends a create success message to the control node. 707.
  • the control node After receiving the creation success message sent by the data node, the control node sends a creation success message to the client.
  • the application scenario shown in Figure 7 can create a distribution table on each data node and complete data migration between data nodes before the client queries. Since the distribution table is created before the client query, the distribution column of the distribution table cannot be determined according to the keywords in the query statement, so in the application scenario shown in FIG. 7 according to the statistical result of querying the data of the user or the client in advance
  • the set logical data table ID and distribution column create a distribution table, and the created distribution table is used for subsequent distributed column joint query.
  • the steps of creating the distribution table and data migration are transferred to the client query, which saves the delay caused by the creation of the distribution table and the data migration to the query during the query process, and can improve the query efficiency.
  • the data migration is specifically: the source data node copies the data stored by itself, and transfers the copied data to the destination data node.
  • the data stored in the source data node is still in existence after the data is migrated.
  • the data migration is a well-known technical means of the person skilled in the art, and the embodiment of the present invention does not introduce too much.
  • the embodiment of the present invention further provides a control node for implementing the method embodiment shown in FIG. 6.
  • the control node includes: a processing unit 81, a sending unit 82, and a receiving unit 83, where
  • the processing unit 81 is configured to: before the data query, set a distribution table creation indication according to the creation rule, where the distribution table creation indication carries an ID of the logical data table and a distribution column identifier of the selected distribution column, wherein the selection
  • the distribution column is a distribution column in the logical data table
  • the logical data table is a logical data table that has been created in the control node;
  • the sending unit 82 is configured to send, to the data node, the distribution table creation indication set by the processing unit 81, so that the data node creates a distribution table indicating the creation of the logical data table according to the distribution table.
  • the processing unit 81 is specifically configured to: collect data of the logical data table in a preset period, obtain a statistical result, and set an ID and a location of the logical data table according to the statistical result. A distribution column identifier of the selected distribution column is added to the distribution table creation indication.
  • the receiving unit 83 is configured to receive a creation indication of the client, where the creation indication carries an ID of the logical data table and a distribution column identifier of the selected distribution column;
  • the processing unit 81 is also specifically configured to:
  • processing unit 81 is further configured to: calculate, in a preset period, at least one of the following data: the number of times the logical data table is queried, and the table that is queried in the logical data table The ratio of the item data to the total data of the logical data table item and the number of times the distribution column in the logical data table is queried;
  • the ID of the logical data table and the distribution column identifier of the distribution column in the logical data table whose number of queries exceeds the third threshold Added to the distribution table creation indication.
  • the sending unit 82 is further configured to: when the processing unit 81 counts that the number of times the distribution table corresponding to the distribution column in the logical data table is queried is less than a fourth threshold, send a distribution table to the data node. Instructing, the distribution table deletion indication carries a distribution column identifier of a distribution column corresponding to a distribution table whose number of times of query is less than the fourth threshold, and an ID of the logical data table, where the distribution table deletion indication is used to indicate the data The node deletes the distribution table whose number of times of query is less than the fourth threshold.
  • processing unit 81 is further configured to: before the control node adds the ID of the logical data table and the distributed column identifier of the selected distribution column to the distribution table creation indication, Determining whether the data node has created a distribution table of the logical data table according to an ID of the logical data table and a distribution column identifier of the selected distribution column;
  • the data node does not create a distribution table of the logical data table according to the ID of the logical data table and the distribution column of the selected distribution column, the ID of the logical data table and the selected distribution The distribution column identifier of the column is added to the distribution table creation indication;
  • the data node When the data node has created a distribution table of the logical data table according to an ID of the logical data table and a distribution column of the selected distribution column, the ID of the logical data table and the selection are not selected
  • the distribution column identifier of the distribution column is added to the distribution table creation indication.
  • the control node provided by the embodiment of the present invention can select a distribution column according to a statistical result of the preset data or an indication of the client in the data storage phase, and the distribution table created by the data node based on the selected distribution column is used for subsequent distribution of the column association. Inquire.
  • the data node directly performs the joint query of the distributed columns according to the plurality of distribution tables created according to the predetermined distribution column, which can save the query time and improve the query efficiency.
  • the embodiment of the present invention further provides a system for data distribution.
  • the system includes a control node 91 and three data nodes 92, where
  • the control node 91 is configured to: before the data query, set a distribution table creation indication according to the creation rule, where the distribution table creation indication carries the identifier ID of the logical data table and the distribution column identifier of the selected distribution column, wherein the selection
  • the distribution is listed as a distribution column in the logical data table, the logical data table is a logical data table that has been created in the control node, and the distribution table creation indication is sent to the data node 92.
  • the data node 92 is configured to receive the distribution table creation indication sent by the control node 91 before the data query, and create a distribution table that is used to create the logical data table according to the distribution table.
  • the system for distributing data according to the embodiment of the present invention is described by taking three data nodes 92 as an example.
  • the number of data nodes 92 is not limited in practical applications.
  • the control node can select a distribution column according to the statistical result of the preset data or the indication of the client, and the distribution table created by the data node based on the selected distribution column is used for subsequent
  • the distribution column is a joint query.
  • the data node directly performs distributed column query according to multiple distribution tables created according to a predetermined distribution column, which can save query time and improve query efficiency.
  • the disclosed systems, devices, and methods may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the modules or units is only a logical function division.
  • there may be another division manner for example, multiple units or components may be used. Combined or can be integrated into another system, or some features can be ignored, or not executed.
  • the coupling or direct coupling or communication connection between the various components shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be electrical, mechanical or otherwise.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software function unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
  • the technical solution of the present invention may contribute to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium. , including a number of instructions to make a computer device (which can be a personal computer, A server, or network device, or the like, or a processor (proces sor) performs all or part of the steps of the method of the various embodiments of the present invention.
  • the foregoing storage medium includes: a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM, Random Acces s Memory), a magnetic disk, or an optical disk, and the like, which can store program codes. medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Mathematical Physics (AREA)
  • Fuzzy Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

La présente invention concerne le domaine technique de l'information. Un procédé, un dispositif et un système de distribution de données sont décrits, lesquels sont destinés à réduire le temps d'interrogation et à améliorer l'efficacité d'interrogation. Le procédé comprend les opérations suivantes : avant une interrogation de données, un nœud de commande configure une instruction d'établissement de table de distribution conformément à une règle d'établissement, l'instruction d'établissement de table de distribution comportant un identificateur (ID) d'une table de données logiques et un ID de colonne de distribution d'une colonne de distribution sélectionnée, la colonne de distribution sélectionnée étant une colonne de distribution dans la table de données logiques, et la table de données logiques étant une table de données logiques établie dans le nœud de commande; et le nœud de commande envoie l'instruction d'établissement de table de distribution à un nœud de données, de telle sorte que le nœud de données établit une table de distribution de la table de données logiques conformément à l'instruction d'établissement de table de distribution. La présente invention est principalement appliquée dans un processus de distribution de données dans un système de base de données parallèle.
PCT/CN2012/079173 2012-07-26 2012-07-26 Procédé, dispositif et système de distribution de données Ceased WO2014015492A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201280002465.XA CN103748578B (zh) 2012-07-26 2012-07-26 数据分布的方法、装置及系统
PCT/CN2012/079173 WO2014015492A1 (fr) 2012-07-26 2012-07-26 Procédé, dispositif et système de distribution de données

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/079173 WO2014015492A1 (fr) 2012-07-26 2012-07-26 Procédé, dispositif et système de distribution de données

Publications (1)

Publication Number Publication Date
WO2014015492A1 true WO2014015492A1 (fr) 2014-01-30

Family

ID=49996501

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2012/079173 Ceased WO2014015492A1 (fr) 2012-07-26 2012-07-26 Procédé, dispositif et système de distribution de données

Country Status (2)

Country Link
CN (1) CN103748578B (fr)
WO (1) WO2014015492A1 (fr)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220261389A1 (en) * 2021-02-18 2022-08-18 International Business Machines Corporation Distributing rows of a table in a distributed database system
WO2022240906A1 (fr) * 2021-05-11 2022-11-17 Strong Force Vcn Portfolio 2019, Llc Systèmes, procédés, kits et appareils de mémorisation et d'interrogation distribués en périphérie dans des réseaux à chaîne de valeur
US12039559B2 (en) 2021-04-16 2024-07-16 Strong Force Vcn Portfolio 2019, Llc Control tower encoding of cross-product data structure
US12153580B2 (en) 2021-05-11 2024-11-26 Strong Force Vcn Portfolio 2019, Llc Dynamic-ledger-enabled edge-device query processing
US12393915B2 (en) 2020-12-18 2025-08-19 Strong Force Vcn Portfolio 2019, Llc Variable-focus dynamic vision for robotic system
US12498680B2 (en) 2020-12-18 2025-12-16 Strong Force Vcn Portfolio 2019, Llc Robotic fleet configuration method for additive manufacturing systems
US12585231B2 (en) 2020-12-18 2026-03-24 Strong Force Vcn Portfolio 2019, Llc Digital-twin-enabled artificial intelligence system for distributed additive manufacturing

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114356969A (zh) * 2020-10-12 2022-04-15 中国移动通信集团辽宁有限公司 数据库查询优化方法、装置、设备及计算机可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101120340A (zh) * 2004-02-21 2008-02-06 数据迅捷股份有限公司 超无共享并行数据库
CN101916261A (zh) * 2010-07-28 2010-12-15 北京播思软件技术有限公司 一种分布式并行数据库系统的数据分区方法
CN102033889A (zh) * 2009-09-29 2011-04-27 熊凡凡 分布式数据库并行处理系统
CN102122306A (zh) * 2011-03-28 2011-07-13 中国人民解放军国防科学技术大学 一种数据处理方法及应用该方法的分布式文件系统

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7562090B2 (en) * 2002-12-19 2009-07-14 International Business Machines Corporation System and method for automating data partitioning in a parallel database
CN102375853A (zh) * 2010-08-24 2012-03-14 中国移动通信集团公司 分布式数据库系统、在其中建立索引的方法和查询方法
US8326825B2 (en) * 2010-11-05 2012-12-04 Microsoft Corporation Automated partitioning in parallel database systems
CN102323946B (zh) * 2011-09-05 2013-03-27 天津神舟通用数据技术有限公司 并行数据库中算子复用的实现方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101120340A (zh) * 2004-02-21 2008-02-06 数据迅捷股份有限公司 超无共享并行数据库
CN102033889A (zh) * 2009-09-29 2011-04-27 熊凡凡 分布式数据库并行处理系统
CN101916261A (zh) * 2010-07-28 2010-12-15 北京播思软件技术有限公司 一种分布式并行数据库系统的数据分区方法
CN102122306A (zh) * 2011-03-28 2011-07-13 中国人民解放军国防科学技术大学 一种数据处理方法及应用该方法的分布式文件系统

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12589498B2 (en) 2020-12-18 2026-03-31 Strong Force Vcn Portfolio 2019, Llc Deployment system for additive manufacturing robot fleet
US12585231B2 (en) 2020-12-18 2026-03-24 Strong Force Vcn Portfolio 2019, Llc Digital-twin-enabled artificial intelligence system for distributed additive manufacturing
US12552035B2 (en) 2020-12-18 2026-02-17 Strong Force Vcn Portfolio 2019, Ll Robotic fleet resource provisioning system
US12498680B2 (en) 2020-12-18 2025-12-16 Strong Force Vcn Portfolio 2019, Llc Robotic fleet configuration method for additive manufacturing systems
US12393915B2 (en) 2020-12-18 2025-08-19 Strong Force Vcn Portfolio 2019, Llc Variable-focus dynamic vision for robotic system
CN116868183A (zh) * 2021-02-18 2023-10-10 国际商业机器公司 将表的行分布在分布式数据库系统中
US20220261389A1 (en) * 2021-02-18 2022-08-18 International Business Machines Corporation Distributing rows of a table in a distributed database system
US12189600B2 (en) * 2021-02-18 2025-01-07 International Business Machines Corporation Distributing rows of a table in a distributed database system
US12491637B2 (en) 2021-04-16 2025-12-09 Strong Force Vcn Portfolio 2019, Llc Demand-responsive raw material management system
US12039559B2 (en) 2021-04-16 2024-07-16 Strong Force Vcn Portfolio 2019, Llc Control tower encoding of cross-product data structure
US12567082B2 (en) 2021-04-16 2026-03-03 Strong Force Vcn Portfolio 2019, Llc Autonomous smart contract execution platform
US12417464B2 (en) 2021-04-16 2025-09-16 Strong Force Vcn Portfolio 2019, Llc Autonomous contingency-responsive smart contract configuration system
US12153580B2 (en) 2021-05-11 2024-11-26 Strong Force Vcn Portfolio 2019, Llc Dynamic-ledger-enabled edge-device query processing
US12339848B2 (en) 2021-05-11 2025-06-24 Strong Force Vcn Portfolio 2019, Llc Edge device query processing of distributed database
US12271382B2 (en) 2021-05-11 2025-04-08 Strong Force Vcn Portfolio 2019, Llc Query prediction modeling for distributed databases
US12204543B2 (en) 2021-05-11 2025-01-21 Strong Force Vcn Portfolio 2019, Llc Dynamic edge-distributed storage in value chain network
US12585654B2 (en) 2021-05-11 2026-03-24 Strong Force Vcn Portfolio 2019, Llc Dynamic vision system for robot fleet management
US12189631B2 (en) 2021-05-11 2025-01-07 Strong Force Vcn Portfolio 2019, Llc Edge-distributed query processing in value chain networks
WO2022240906A1 (fr) * 2021-05-11 2022-11-17 Strong Force Vcn Portfolio 2019, Llc Systèmes, procédés, kits et appareils de mémorisation et d'interrogation distribués en périphérie dans des réseaux à chaîne de valeur

Also Published As

Publication number Publication date
CN103748578B (zh) 2017-10-10
CN103748578A (zh) 2014-04-23

Similar Documents

Publication Publication Date Title
CN109660607B (zh) 一种业务请求分发方法、接收方法、装置及服务器集群
CN104969213B (zh) 用于低延迟数据存取的数据流分割
CN103118084B (zh) 一种主节点的选举方法及节点
CN103748578A (zh) 数据分布的方法、装置及系统
CN106549988B (zh) 加入群组方法、装置及系统
US10270726B2 (en) Selective distribution of messages in a scalable, real-time messaging system
CN106572054B (zh) 分布式系统中分布式锁服务实现方法以及装置
US20200044881A1 (en) Managing channels in an open data ecosystem
US11953997B2 (en) Systems and methods for cross-regional back up of distributed databases on a cloud service
CN108920111B (zh) 数据共享方法及分布式数据共享系统
WO2019007010A1 (fr) Procédé et système de recherche distribuée et de mise à jour d'index, serveurs et dispositifs informatiques
CN106407011A (zh) 一种基于路由表的搜索系统集群服务管理的方法及系统
CN112685499A (zh) 一种工作业务流的流程数据同步方法、装置及设备
CN104809130A (zh) 数据查询的方法、设备及系统
CN105630419B (zh) 一种资源池的分区视图发送方法及管理节点
CN103377236B (zh) 一种用于分布式数据库的连接查询方法及系统
CN112910796B (zh) 流量管理方法、装置、设备、存储介质以及程序产品
CN112968788A (zh) 网络切片容量设置方法、控制方法、装置、设备及介质
CN106921512B (zh) 一种大数据集群租户带宽控制方法及装置
CN108156201B (zh) 统一管理应用系统配置数据的方法和装置
CN110019456A (zh) 数据导入方法、装置和系统
CN111221786B (zh) 对等式数据运算与存储架构构建方法
CN104978327B (zh) 一种查询数据的方法、管理控制节点及目标数据节点
CN107870924A (zh) 查询数据的方法和数据查询系统
CN116467336A (zh) 数据处理方法及装置、电子设备、存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12881616

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12881616

Country of ref document: EP

Kind code of ref document: A1