WO2018040488A1 - 一种处理连接查询的方法及装置 - Google Patents

一种处理连接查询的方法及装置 Download PDF

Info

Publication number
WO2018040488A1
WO2018040488A1 PCT/CN2017/071568 CN2017071568W WO2018040488A1 WO 2018040488 A1 WO2018040488 A1 WO 2018040488A1 CN 2017071568 W CN2017071568 W CN 2017071568W WO 2018040488 A1 WO2018040488 A1 WO 2018040488A1
Authority
WO
WIPO (PCT)
Prior art keywords
combination
same
table combination
cluster
connection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/071568
Other languages
English (en)
French (fr)
Inventor
王振华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP17844799.1A priority Critical patent/EP3499388B1/en
Publication of WO2018040488A1 publication Critical patent/WO2018040488A1/zh
Priority to US16/287,510 priority patent/US11030196B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2393—Updating materialised views
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2453—Query optimisation
    • G06F16/24534—Query rewriting; Transformation
    • G06F16/24542—Plan optimisation
    • G06F16/24544—Join order optimisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2228—Indexing structures
    • G06F16/2255—Hash tables
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G06F16/285—Clustering or classification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G06F7/22—Arrangements for sorting or merging computer data on continuous record carriers, e.g. tape, drum, disc
    • G06F7/36—Combined merging and sorting

Definitions

  • the present invention relates to the field of communications technologies, and in particular, to a method and apparatus for processing a connection query.
  • MapReduce Map Reduction
  • MapReduce-based distributed computing framework can be used for large data query and analysis tasks, however, due to MapReduce-based
  • MapReduce-based Map Reduction
  • OLAP On-Line Analytical Processing
  • SQL Structured Query Language
  • SQL Structured Query Language
  • the Join query in the SQL query can connect two tables in the database through the connection properties, so you can use distributed Hash Join technology to determine a column between two tables in a distributed environment.
  • the equivalence connection as shown in Figure 1, first needs to determine the storage node of the data of the table to be processed, start the map task on the determined node, perform a local hash process for each data block, respectively
  • the data in the block (block) is divided according to the hash value of the key value. For example, the data in each block in FIG. 1 is divided into three buckets, and the key of the data in the same bucket.
  • the hash value of the value is the same, and then the shuffle process is performed, and the records of the same bucket are transmitted to the same reduce node, and finally the connection operation is performed on the records with the same key value in the two tables in the reduce phase.
  • the embodiment of the invention provides a method and a device for processing a connection query, which can improve the efficiency of the connection query.
  • an embodiment of the present invention provides a method for processing a connection query, including: a primary node determines a frequent table combination, and a frequent table combination is a table combination in which a frequency of occurrence in a historical query record is greater than a preset value, and the table combination includes a connection.
  • the key and the table connected by the connection key and then create a cluster index according to the connection key information in the frequent table combination
  • the number of index columns in the cluster index is the same as the number of connection keys in the frequent table combination
  • the cluster index is used to indicate frequent table combination
  • the storage location of the record with the same index column value after which the master node controls the work node to perform the shuffling operation according to the index column in the cluster index, and stores the records with the same index column value in at least one number.
  • a table cluster corresponding to the frequent table combination is formed, wherein the records with the same index column value are records in the table connected by the connection key.
  • the records with the same connection key are stored in one data block as much as possible, so that the connection key value of the data in the data block is concentrated, which greatly reduces the calculation amount when the data is barreled in the map stage when the connection query is executed, and Since the data stored in one data block is basically the same data of the connection key, at this time, there is not too much bucket data in one data block, which can reduce the time required for the reduce phase to transmit data to the reduce node, and improve the time. The efficiency of connecting queries.
  • the connection query operation may be performed, and the query request is first received, the query request includes the table combination to be queried, and then the table to be queried is searched.
  • the corresponding table clusters to be queried respectively, respectively, the records with the same hash value of the index column values in each data block included in the node corresponding to the table cluster to be queried are divided into one bucket, and all the data in the table cluster to be queried are to be queried.
  • the records in the same bucket in the block are transferred to the same data node.
  • the map task can perform the bucketing without performing the local hash process, which reduces the calculation amount when the data is bucketed, saves the CPU overhead, and is in the table cluster. Only records with the same index column value are stored. For records with different index column values, no processing is required, which reduces the amount of data to be processed and the number of tasks started and disk I/O. After the shuffling process, due to the large Some data blocks correspond to only one bucket, which is equivalent to one index column value, and the reduce task is preferentially arranged locally in the map task. Most of the output data of the map task required by the reduce node does not need network transmission. Therefore, the network transmission overhead generated by the process of transmitting the data with the same index column value to the reduce node is greatly reduced, the transmission time is shortened, and the efficiency of the connection query is improved.
  • the frequent table combination when determining the frequent table combination, it is necessary to extract the table combination from the historical query record, generate the table combination set, and then filter out the table combination whose appearance frequency is greater than the preset value from the table combination set, and then appear from the table combination.
  • the redundancy table combination is deleted from the table combination whose frequency is greater than the preset value, and the remaining table combinations are determined to be frequent table combinations.
  • the last stored table By filtering the combination of the tables whose frequency is greater than the preset value, the last stored table can be combined into a common combination of the user, which can improve the query efficiency and delete the redundant table combination, which can save storage space and reduce subsequent processing.
  • the amount of data increases processing efficiency.
  • the specific method of deleting the redundant table combination from the combination of the table whose frequency is greater than the preset value is: when there are at least two table combinations with the same table but different connection keys, the connection is reserved.
  • the shuffling operation is performed according to the index column in the cluster index, and the records with the same index column value are stored in at least one data block in an implementation manner: when the record containing the same index column value is included The total size reaches the first preset ratio of the storage space size of one data block, and when the storage space size of one data block is not exceeded, the records containing the same index column value are stored in one data block; when the same index is included When the total size of the record of the column value exceeds the storage space of one data block, the record containing the same index column value is stored in multiple data blocks; when the total size of the record containing the same index column value is less than one data block When the second preset ratio of the storage space size is used, multiple records containing the same index column value are stored in one data block.
  • the representation of the table combination is:
  • an embodiment of the present invention provides a method for processing a connection query, including: a primary node receives a query request, the query request includes a combination of the to-be-queried table, and then searches for a combination of the table to be queried in the distributed file system DFS.
  • the table to be queried, the master node control working node respectively divides the records with the same hash value of the index column values in each data block included in the node corresponding to the table cluster to be queried into one bucket, and then the table to be queried Records in the same bucket in all data blocks in the cluster are transferred to the same data node.
  • the DFS stores the table cluster corresponding to the frequent table combination
  • the frequent table combination is a table combination whose appearance frequency in the historical query record is greater than a preset value
  • the table combination includes a connection key and a table connected by the connection key, and the frequent table
  • the number of join keys in the combination is the same as the number of index columns in the cluster index.
  • the cluster index is used to indicate the storage location of the records with the same index column value in the frequent table combination, wherein the records with the same index column value in the frequent table combination are stored in the cluster.
  • a table cluster corresponding to a frequent table combination is formed.
  • the map task can perform the bucketing without performing the local hash process, which reduces the calculation amount when the data is bucketed, saves the CPU overhead, and is in the table cluster. Only records with the same index column value are stored. For records with different index column values, no processing is required, which reduces the amount of data to be processed and the number of tasks started and disk I/O. After the shuffling process, due to the large Some data blocks correspond to only one bucket, which is equivalent to one index column value, and the reduce task is preferentially arranged locally in the map task. Most of the output data of the map task required by the reduce node does not need network transmission. Therefore, the network transmission overhead generated by the process of transmitting the data with the same index column value to the reduce node is greatly reduced, the transmission time is shortened, and the efficiency of the connection query is improved.
  • the embodiment of the present invention provides a device for processing a connection query, which can implement the function of the master node in the foregoing method embodiment, and the function can be implemented by using hardware or by executing corresponding software through hardware.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the apparatus includes a processor and a transceiver configured to support the master node in performing the corresponding functions of the above methods.
  • the transceiver is used to support communication between the primary node and other network elements.
  • the master node may also include a memory for coupling with the processor that holds the program instructions and data necessary for the device.
  • an embodiment of the present invention provides a big data analysis system, where the system includes the primary node, the metadata storage unit metastore for storing a table cluster, and a client for sending a connection query request, A distributed file system DFS for carrying data blocks in the above method, and a working node for reading data in the DFS and calculating the data.
  • an embodiment of the present invention provides a computer storage medium for storing computer software instructions for use by the master node, including a program designed to perform the above aspects.
  • records with the same connection key can be stored in one data block as much as possible, so that the connection key value of the data in the data block is concentrated, which is greatly reduced.
  • Data can reduce the time required for the reduce phase to transfer data to the reduce node, improving the efficiency of connection queries.
  • FIG. 1 is an exemplary schematic diagram of a method for processing a connection query provided by a background example
  • FIG. 2 is a schematic structural diagram of a clustered big data analysis system according to an embodiment of the present invention.
  • FIG. 3 is a flowchart of a method for processing a connection query according to an embodiment of the present invention
  • FIG. 4 is a flowchart of another method for processing a connection query according to an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of a method for processing a connection query according to an embodiment of the present invention.
  • FIG. 6 is a flowchart of another method for processing a connection query according to an embodiment of the present invention.
  • FIG. 7 is an exemplary schematic diagram of another method for processing a connection query according to an embodiment of the present invention.
  • FIG. 8 is a schematic diagram of a logical structure of an apparatus for processing a connection query according to an embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of another apparatus for processing a connection query according to an embodiment of the present disclosure.
  • FIG. 10 is a schematic diagram of a logical structure of another apparatus for processing a connection query according to an embodiment of the present invention.
  • FIG. 11 is a schematic diagram of a logical structure of another apparatus for processing a connection query according to an embodiment of the present invention.
  • the embodiment of the present invention provides a method for processing a connection query.
  • the method can be specifically applied to a clustered big data analysis system.
  • the system includes a client (client) and a metastore. (metadata storage unit), a master node, multiple worker nodes, and DFS (Distribute File System).
  • client client
  • metastore metadata storage unit
  • master node master node
  • worker nodes multiple worker nodes
  • DFS Distribute File System
  • the master includes a table cluster management module and a SQL engine, and the table cluster management module further includes a workload analysis submodule and a cluster index maintenance submodule.
  • a DFS contains a plurality of nodes, each of which contains at least one block.
  • a workload is a set of queries consisting of multiple historical SQL queries, which can be obtained by recording a historical SQL query submitted by the client within a preset time period.
  • the master node can construct the table cluster and the cluster index structure by analyzing the historical SQL query records in the workload, store the established table clusters in the metastore, and then the work nodes reorganize the data in the table clusters, and The reorganized data is stored in the corresponding node in the DFS.
  • the SQL engine executes the connection query process.
  • the SQL engine can call the working node to read the data stored in the DFS, and the working node can also complete some
  • the calculation work, the specific method of processing the connection query can refer to the following embodiments.
  • the embodiment of the present invention provides a method for processing a connection query. As shown in FIG. 3, the method includes:
  • the master node determines a frequent table combination.
  • the frequent table combination is a table combination in which the frequency of occurrence in the historical query record is greater than a preset value, and the table combination includes a connection key and a table connected by the connection key.
  • connection key is the connection property between the tables, that is, the properties of the public column between the tables.
  • the representation of the table combination is:
  • tab i is the ith table in the table combination
  • key j is the jth connection key
  • N is the number of tables in the table combination
  • M is the number of connection keys in the table combination.
  • the historical query record in this step records the workload reported by the terminal received by the host, and the workload is a query set composed of multiple historical SQL query records.
  • the master node can periodically determine the frequent table combination from the workload according to a preset time period.
  • the master node creates a cluster index according to the connection key information in the frequent table combination.
  • the number of index columns in the cluster index is the same as the number of connection keys in the frequent table combination.
  • a connection key corresponds to an index column.
  • a single column index is created, and when the number M of the connection keys is greater than or equal to 2, a composite index is created.
  • the cluster index is used to indicate the storage location of the record with the same index column value in the frequent table combination.
  • the frequent table combination includes Table A and Table B, and Table A and Table B have one common column, and the cluster index is used for the cluster index.
  • the storage location of the record indicating the same index column value in Table A and Table B.
  • the master node controls the working node to perform a shuffling operation according to the index column in the cluster index, and stores the records with the same index column value in at least one data block to form a table cluster corresponding to the frequent table combination.
  • the record with the same index column value is the record in the table connected by the connection key, and one table cluster is composed of a set of tables sharing the data block, and the common column of the table in the table combination is the cluster index of the table cluster, and the table is Records in all tables in the combination that contain the same index column value are stored in at least one data block, so that the records containing the same index column value are more concentrated.
  • the master node when the shuffling operation is required, the master node sends a control command to the working node, controls the working node to perform the shuffling operation, and stores the records with the same index column value in at least one data block in the DFS. .
  • steps 301 to 302 are specifically performed by the table cluster management module in the master node.
  • the method for processing a connection query determines a frequent table combination, and then creates a cluster index according to the connection key information in the frequent table combination, and then performs a shuffle operation according to the index column in the cluster index, and records the index column with the same value.
  • the data is stored in at least one data block to form a table cluster corresponding to the frequent table combination.
  • the records with the same connection key can be stored in one data block as much as possible, so that the connection key value of the data in the data block is It is more concentrated, which greatly reduces the amount of calculation when data is bucketed in the map phase when performing join query, and since the data stored in one data block is basically the same data of the connection key, at this time, it will not be in one data block. There are too many buckets of data, which can reduce the reduce phase to transfer data to the reduce node. The time required to improve the efficiency of join queries.
  • the frequent table combination in the embodiment of the present invention is determined by the primary node according to the workload reported by the client, and based on this, in another implementation manner provided by the embodiment of the present invention, The method of the frequent table combination is described in detail.
  • the master node determines the frequent table combination, and may be implemented as step 3011 to step 3013.
  • the master node extracts a table combination from the historical query record to generate a table combination set.
  • the master node performs a formal conversion on each query record in the workload, and extracts a table using the connection key to perform the join operation from the SQL query statement to form a table combination. If there are multiple query statements in a query record, a table combination is generated according to each query statement, and all generated table combinations are combined into a table combination set.
  • one of the query records is:
  • this query contains two join queries, and the first join query is formally converted, and the obtained table combination is:
  • web_sales and warehouse are two tables, and the connection key between the two tables is warehouse_sk.
  • catalog_sales and ship_mode are two tables, and the connection key between the two tables is ship_mode_sk.
  • the master node selects, from the table combination set, a table combination whose appearance frequency is greater than a preset value.
  • the frequent item set mining algorithm may be used to calculate a table combination whose frequency appears in the table combination set is greater than a preset value, and a table combination whose frequency appears to be greater than a preset value in the table combination set constitutes a frequent table combination set. .
  • the master node deletes the redundancy table combination from the table combination whose appearance frequency is greater than the preset value, and determines that the remaining table combination is a frequent table combination.
  • redundancy table combination may exist in the frequent table combination set generated in the foregoing step 3012, and the redundancy combination may be filtered by using any one or more of the following three rules.
  • TG1 and TG2 are catalog_sales and catalog_returns, but TG1 has only one connection key item_sk, and TG2 has two connection keys item_sk and order_number.
  • TG1 and TG2 contain the same table, but the connection key of TG2 is more than the connection key of TG1, so the table combination TG2 is retained, and the table combination TG1 is deleted.
  • TG1 and TG2 contain the same connection key, which is item_sk.
  • the set of tables in TG1 ⁇ catalog_sales, catalog_returns ⁇ is a subset of the set of tables in TG2 ⁇ catalog_sales, catalog_returns, item ⁇ , so delete
  • the table combination TG1 retains the table combination TG2.
  • FIG. 5 is an exemplary schematic diagram of a process for creating a distributed table cluster according to an embodiment of the present invention.
  • a cluster index needs to be created, and then a cluster is generated according to the cluster index and stored in the data.
  • the master node controls the working node to perform a shuffle operation according to the index column in the cluster index, and stores the record with the same index column value in the table connected by the connection key.
  • the at least one data block may be specifically implemented as step 3031 to step 3033.
  • the first preset ratio may be 80%.
  • connection key item_sk and order_number respectively correspond to an index column
  • the index column values of the index column corresponding to the connection item item_sk include A and B
  • the index column value of the index column corresponding to the connection key order_number includes 1, 2, 3, and 4.
  • the total size of the record containing the first index column value (A, 1) reaches 80% of the total size of block1, and does not exceed the total storage space of block1, so only records containing index column value 1 can be stored in block1. in.
  • the total size of a record containing the fourth index column value (B, 4) has exceeded a block store.
  • the record containing the index column value (B, 4) is stored in the two data blocks block3 and block4.
  • the second index will be included.
  • the record of the column value (A, 3) and the record containing the third index column value (B, 2) are stored in block 2.
  • the method for processing a connection query provided by the embodiment of the present invention can make the last stored table combination be a common combination of the table by filtering the table combination whose appearance frequency is greater than the preset value, which can improve the query efficiency and delete the redundancy table combination. To save storage space. Finally, store records containing the same index column values in one or more data blocks as much as possible based on the total size of records containing the same index column value and the size of one block storage space. In comparison, the storage is stored in multiple data blocks. Since the records containing the same index column values are stored in one or more data blocks, it can be considered that the data has been completed in the data block, and the reduction is completed. The time required to connect the buckets during the query process, so the time required to connect the query process can be reduced, and the efficiency of the join query can be improved.
  • the subsequent connection query operation can be performed.
  • the table cluster established according to the embodiment of the present invention is further provided in an implementation manner provided by the embodiment of the present invention.
  • the process of processing a connection query based on the table cluster established in the above embodiment, as shown in FIG. 6, the method includes:
  • the master node receives the query request, where the query request includes a combination of the tables to be queried.
  • the query compiler is required to determine the combination of the to-be-queried tables requested by the query request for the connection calculation.
  • the master node searches for a to-be-queried table cluster corresponding to the to-be-queried table combination.
  • the SQL engine in the primary node may determine, by the table cluster management module, the cluster to be queried corresponding to the combination of the tables to be queried.
  • the master node control working node respectively divides the records with the same hash value of the index column values in each data block included in the to-be-queried table cluster into one bucket.
  • the primary node may send a control instruction to the working node to control the working node to perform a subsequent mapreduce process.
  • the SQL engine can read the data about the cluster to be queried in the distributed file system through the working node, and the process of performing the connection query on the query table cluster is as shown in FIG. 7 , and the master node controls the working node.
  • the map program is started on two nodes respectively, and since the data with the same index column value (key value) has been stored together, the map program can complete the bucket operation without performing the local hash process, for example,
  • block1 only stores records with an index column value of 1, so all records stored in block1 are a bucket
  • block2 only stores records with an index column value of 1, so all records stored in block2 are A bucket
  • block3 stores a record with an index column value of 2 and a partial index column value of 3
  • records with an index column value of 2 and records with an index column value of 3 are stored independently, so no need to perform Local hash can also divide the data stored in block3 into two buckets.
  • Block4 stores records with index column value of 3, so block3 stores all records as one bucket.
  • the master node controls the working node to transmit the records in the same bucket in all the data blocks in the to-be-queried table cluster to the same data node.
  • the method for processing a connection query provided by the embodiment of the present invention, after receiving the query request after establishing the table cluster, determining the combination of the to-be-queried table in the query request, and then searching for the to-be-queried table cluster corresponding to the combination of the to-be-queried table,
  • the map task is executed on the node corresponding to the query table cluster. Since the data with the same index column value has been stored together, the map task can be completed without the local hash process, and the calculation of the data bucket is reduced. The amount of CPU overhead is saved, and only records with the same index column value are stored in the table cluster. For records with different index column values, no processing is required, which reduces the amount of data to be processed and the number of tasks started and disk I/.
  • each network element such as a device for connecting a query, etc.
  • each network element includes hardware structures and/or software modules corresponding to each function.
  • the present invention can be implemented in a combination of hardware or hardware and computer software in combination with the elements and algorithm steps of the various examples described in the embodiments disclosed herein. Whether a function is implemented in hardware or computer software to drive hardware depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.
  • the embodiment of the present invention may divide the function modules of the master node and the like shown in FIG. 2 according to the foregoing method example.
  • each function module may be divided according to each function, or two or more functions may be integrated into one process.
  • the above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of the module in the embodiment of the present invention is schematic, and is only a logical function division, and the actual implementation may have another division manner.
  • FIG. 8 is a schematic diagram showing a possible structure of the master node involved in the foregoing embodiment, and FIG. 8 is specifically a table cluster in the master node shown in FIG.
  • a schematic diagram of a management module, the master node includes: a determining unit 801, a creating unit 802, and a shuffling unit 803.
  • the determining unit 801 is configured to support the master node to perform step 301 in FIG. 3, steps 3011 to 3013 in FIG. 4;
  • the creating unit 802 is configured to support the master node to perform step 302 in FIG. 3;
  • the shuffling unit 803 is configured to support the master node.
  • the control working node performs step 303 in FIG. 3 to support the master node to perform steps 3031 to 3033 in FIG.
  • FIG. 9 is another schematic structural diagram of the primary node involved in the foregoing embodiment, and FIG. 9 is specifically shown in the primary node shown in FIG. 2 in the case of dividing each functional module by using corresponding functions.
  • a schematic diagram of the structure of the SQL engine includes a receiving unit 901, a searching unit 902, a bucket unit 903, and a transmitting unit 904.
  • the receiving unit 901 is used by the primary node to perform step 601 in FIG. 6;
  • the searching unit 902 is configured to support the primary node to perform step 602 in FIG. 6;
  • the bucket unit 903 is configured to support the primary node to control the working node to perform the operation in FIG. 6.
  • the transmission unit 904 is configured to support the master node to control the working node to perform step 604 in FIG. 6.
  • FIG. 10 shows a possible structural diagram of the master node involved in the above embodiment.
  • the master node includes a processing module 1002 and a communication module 1003.
  • the processing module 1002 is configured to perform control management on the actions of the master node.
  • the processing module 1002 is configured to support steps 301 to 303 in FIG. 3, steps 3011 to 3033 in FIG. 4, and steps 602 to 603 in FIG. 6;
  • the module 1003 is configured to support communication between the master node and other network entities.
  • the communication module 1003 is configured to support steps 601 and 604 in FIG. 6, and may be implemented between the function modules or network entities shown in FIG. 2 or FIG. Communication.
  • the apparatus also includes a storage module 1001 for storing program code and data of the primary node.
  • the processing module 1002 may be a processor or a controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), and an application-specific integrated circuit (Application-Specific). Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the present disclosure.
  • the processor may also be a combination of computing functions, for example, including one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like.
  • the communication module 1003 may be a transceiver, a transceiver circuit, a communication interface, or the like.
  • the storage module 1001 may be a memory.
  • FIG. 8 to FIG. 10 is a schematic structural diagram of the main node being implemented as software, and the master node may also exist in a hardware manner, that is, when the processing module 1002 is a processor, the communication module 1003 is a transceiver, and the storage module 1001 is a memory.
  • the master node involved in the embodiment of the present invention may be the master node shown in FIG.
  • the processor 1102, the transceiver 1103, the memory 1101, and the bus 1104 are included.
  • the transceiver 1103, the processor 1102, and the memory 1101 are connected to each other through a bus 1104.
  • the bus 1104 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Wait.
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • the bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in Figure 11, but it does not mean that there is only one bus or one type of bus.
  • the primary node and the working node in the embodiment of the present invention may be a single device or may exist in a software manner.
  • the function of the primary node and the function of the working node may be Different virtual machine implementations in the system.
  • the steps of a method or algorithm described in connection with the present disclosure may be implemented in a hardware, or may be implemented by a processor executing software instructions.
  • the software instructions may be composed of corresponding software modules, which may be stored in a random access memory (RAM), a flash memory, a read only memory (ROM), an erasable programmable read only memory ( Erasable Programmable ROM (EPROM), electrically erasable programmable read only memory (EEPROM), registers, hard disk, removable hard disk, compact disk read only (CD-ROM) or any other form of storage medium known in the art.
  • An exemplary storage medium is coupled to the processor to enable the processor to read information from, and write information to, the storage medium.
  • the storage medium can also be an integral part of the processor.
  • the processor and the storage medium can be located in an ASIC. Additionally, the ASIC can be located in a core network interface device.
  • the processor and the storage medium may also exist as discrete components in the core network interface device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Operations Research (AREA)
  • Computer Hardware Design (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种处理连接查询的方法及装置，涉及通信技术领域，能够解决连接查询的效率低的问题。方法包括：确定频繁表组合，频繁表组合为在历史查询记录中的出现频率大于预设值的表组合，表组合包括连接键以及通过连接键进行连接的表，然后根据频繁表组合中的连接键信息创建簇索引，再根据簇索引中的索引列进行洗牌操作，将索引列值相同的记录存放在至少一个数据块中，以形成频繁表组合对应的表簇。该方法适用于对表格进行连接查询。

Description

一种处理连接查询的方法及装置
本申请要求于2016年08月31日提交中国专利局、申请号为201610797295.9、发明名称为“一种处理连接查询的方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及通信技术领域，尤其涉及一种处理连接查询的方法及装置。
背景技术
网络技术的飞速发展使得数据量急剧增长，为了对大规模的数据进行高效的处理，可以采用基于MapReduce(映射归约)的分布式计算框架进行大数据的查询分析任务，然而由于在基于MapReduce的分布式计算框架中执行查询分析任务时，需要针对每个任务编写复杂的程序，对于OLAP(On-Line Analytical Processing，联机分析处理)这种复杂的查询，实现过程更为复杂，易用性较低。相比之下，SQL(Structured Query Language，结构化查询语言)的易用性较高，所以通常将SQL应用于基于MapReduce的分布式计算框架中以进行大数据的查询分析。
SQL查询中的Join(连接)查询可以将数据库中的两张表通过连接属性连接起来，所以可以采用分布式Hash Join(散列连接)技术在分布式环境下确定两个表之间对某一列的等值连接，如图1所示，首先需确定待处理表格的数据的存储节点，在确定的节点上启动map任务，对每个数据块执行一次本地hash(哈希)过程，分别将每个block(数据块)中的数据根据key值的哈希值进行分桶，例如，图1中的每个数据块中的数据分别被划分至三个分桶，同一分桶中的数据的key值的hash值相同，之后进行shuffle(洗牌)过程，将相同分桶的记录传输至同一reduce节点上，最终在reduce阶段对两张表中的key值相同的记录执行连接操作。
然而，在实现上述方法的过程中，当数据块中数据的key值较为分散时，在map阶段对数据进行分桶的过程需要进行大量的计算，所需的时间较长，且由于每个数据块中都存在多个分桶的数据，在shuffle过程中会产生大量的网络连接开销和数据传输开销，结合图1，每个数据块中的数据都分别需要传输至三个不同reduce节点，传输过程需要消耗一定的时间，最终导致连接查询的效率较低。
发明内容
本发明实施例提供一种处理连接查询的方法及装置，能够提高连接查询的效率。
第一方面，本发明实施例提供一种处理连接查询的方法，包括：主节点确定频繁表组合，频繁表组合为在历史查询记录中的出现频率大于预设值的表组合，表组合包括连接键以及通过连接键进行连接的表，然后根据频繁表组合中的连接键信息创建簇索引，簇索引中索引列的数量与频繁表组合中连接键的数量相同，簇索引用于指示频繁表组合中索引列值相同的记录的存储位置，之后主节点控制工作节点根据簇索引中的索引列进行洗牌操作，将索引列值相同的记录集中存放在至少一个数 据块中，以形成频繁表组合对应的表簇，其中索引列值相同的记录为通过连接键进行连接的表中的记录。可见将连接键相同的记录尽可能的存放在一个数据块中，使得数据块中数据的连接键值较为集中，大大减少了执行连接查询时，在map阶段对数据分桶时的计算量，且由于一个数据块中存储的基本都是连接键相同的数据，此时，一个数据块中就不会出现太多分桶的数据，可以减少reduce阶段将数据传输至reduce节点所需的时间，提高了连接查询的效率。
在一种可能的设计中，在完成洗牌操作，形成频繁表组合对应的表簇之后，可以进行连接查询的操作，首先接收查询请求，查询请求中包含待查询表组合，然后查找待查询表组合对应的待查询表簇，分别将待查询表簇对应的节点所包含的每个数据块中索引列值的哈希值相同的记录划分至一个分桶中，将待查询表簇中所有数据块中相同分桶中的记录传输至同一数据节点中。由于索引列值相同的数据已经被存储在一起，所以执行map任务时无需进行本地hash的过程也能够完成分桶，减少了对数据分桶时的计算量，节省了CPU开销，且表簇中只存储了包含相同索引列值的记录，对于索引列值不同的记录不需要进行处理，减少了需要处理的数据量以及启动的任务数量和磁盘I/O，之后在洗牌过程中，由于大部分数据块都只对应一个分桶，相当于一个数据块对应一个索引列值，且reduce任务优先安排在map任务的本地执行，reduce节点所需的map任务的输出数据大部分都不需要网络传输，所以大大减小了将索引列值相同的数据传输至reduce节点的过程产生的网络传输开销，缩短了传输时间，提高了连接查询的效率。
在一种可能的设计中，确定频繁表组合时，需从历史查询记录中提取表组合，生成表组合集合，然后从表组合集合中筛选出出现频率大于预设值的表组合，再从出现频率大于预设值的表组合中删除冗余表组合，确定剩余的表组合为频繁表组合。通过筛选出现频率大于预设值的表组合，可以使得最后存储的表组合为用户常用的表组合，可以提高查询效率，且将冗余表组合删除，可以节省存储空间，还能够减少后续处理的数据量，提高处理效率。
在一种可能的设计中，从出现频率大于预设值的表组合中删除冗余表组合的具体方法为：当存在包含的表相同，但是连接键不同的至少两个表组合时，保留连接键最多的表组合；当存在两个表组合所包含的连接键相同，且一个表组合中的表组成的集合是另一个表组合中表组成的集合的子集时，删除包含表较少的表组合；当存在相同表包含于至少两个表组合中时，只将所述相同表保留于所述至少两个表组合中出现频率最高的一个表组合中。
在一种可能的设计中，根据簇索引中的索引列进行洗牌操作，将索引列值相同的记录集中存放在至少一个数据块中的实现方式为：当包含同一个索引列值的记录的总大小达到一个数据块的存储空间大小的第一预设比例，且未超过一个数据块的存储空间大小时，将包含同一个索引列值的记录存储在一个数据块中；当包含同一个索引列值的记录的总大小超过一个数据块的存储空间大小时，将包含同一个索引列值的记录存储在多个数据块中；当包含同一个索引列值的记录的总大小小于一个数据块的存储空间大小的第二预设比例时，将多个包含同一个索引列值的记录存储在一个数据块中。可见，根据包含相同索引列值的记录的总大小和一个数据块存储 空间大小的关系，来尽可能地将包含相同索引列值的记录集中存储在一个或多个数据块中，相比较于分散存储在多个数据块中，可以减少连接查询过程中分桶所需的时间，所以可以减少连接查询过程所需的时间，提高连接查询的效率。
在一种可能的设计中，表组合的表示形式为：
TG＝(tab1,...,tabi,...,tabN)key＝(key1,...,keyj,...,keyM)，其中tabi为表组合中的第i张表，keyj为第j个连接键，N为表组合中表的个数，M为表组合中连接键的个数。
另一方面，本发明实施例提供了一种处理连接查询的方法，包括：主节点接收查询请求，查询请求中包含待查询表组合，然后在分布式文件系统DFS中查找与待查询表组合对应的待查询表簇，主节点控制工作节点分别将待查询表簇对应的节点所包含的每个数据块中索引列值的哈希值相同的记录划分至一个分桶中，再将待查询表簇中所有数据块中相同分桶中的记录传输至同一数据节点中。其中，DFS中存储了频繁表组合对应的表簇，频繁表组合为在历史查询记录中的出现频率大于预设值的表组合，表组合包括连接键以及通过连接键进行连接的表，频繁表组合中连接键的数量与簇索引中索引列的数量相同，簇索引用于指示频繁表组合中索引列值相同的记录的存储位置，其中，频繁表组合中索引列值相同的记录集中存放在至少一个数据块中，形成频繁表组合对应的表簇。由于索引列值相同的数据已经被存储在一起，所以执行map任务时无需进行本地hash的过程也能够完成分桶，减少了对数据分桶时的计算量，节省了CPU开销，且表簇中只存储了包含相同索引列值的记录，对于索引列值不同的记录不需要进行处理，减少了需要处理的数据量以及启动的任务数量和磁盘I/O，之后在洗牌过程中，由于大部分数据块都只对应一个分桶，相当于一个数据块对应一个索引列值，且reduce任务优先安排在map任务的本地执行，reduce节点所需的map任务的输出数据大部分都不需要网络传输，所以大大减小了将索引列值相同的数据传输至reduce节点的过程产生的网络传输开销，缩短了传输时间，提高了连接查询的效率。
另一方面，本发明实施例提供了一种处理连接查询的装置，该装置可以实现上述方法实施例中主节点的功能，所述功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个上述功能相应的模块。
在一种可能的设计中，该装置的结构中包括处理器和收发器，该处理器被配置为支持主节点执行上述方法中相应的功能。该收发器用于支持主节点与其他网元之间的通信。主节点还可以包括存储器，该存储器用于与处理器耦合，其保存该装置必要的程序指令和数据。
又一方面，本发明实施例提供了一种大数据分析系统，该系统包括上述方面所述的主节点、用于存储表簇的元数据存储单元metastore、用于发送连接查询请求的客户端、用于承载上述方法中的数据块的分布式文件系统DFS，以及用于读取DFS中数据，并对数据进行计算的工作节点。
再一方面，本发明实施例提供了一种计算机存储介质，用于储存为上述、主节点所用的计算机软件指令，其包含用于执行上述方面所设计的程序。
相比于现有技术，本发明实施例提供的技术方案中，可以将连接键相同的记录尽可能的存放在一个数据块中，使得数据块中数据的连接键值较为集中，大大减少 了执行连接查询时，在map阶段对数据分桶时的计算量，且由于一个数据块中存储的基本都是连接键相同的数据，此时，一个数据块中就不会出现太多分桶的数据，可以减少reduce阶段将数据传输至reduce节点所需的时间，提高了连接查询的效率。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图。
图1为背景技术例提供的一种处理连接查询的方法的示例性示意图；
图2为本发明实施例提供的一种集群式的大数据分析系统的结构示意图；
图3为本发明实施例提供的一种处理连接查询的方法的流程图；
图4为本发明实施例提供的另一种处理连接查询的方法的流程图；
图5为本发明实施例提供的一种处理连接查询的方法的示例性示意图；
图6为本发明实施例提供的另一种处理连接查询的方法的流程图；
图7为本发明实施例提供的另一种处理连接查询的方法的示例性示意图；
图8为本发明实施例提供的一种处理连接查询的装置的逻辑结构示意图；
图9为本发明实施例提供的另一种处理连接查询的装置的逻辑结构示意图；
图10为本发明实施例提供的另一种处理连接查询的装置的逻辑结构示意图；
图11为本发明实施例提供的另一种处理连接查询的装置的逻辑结构示意图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其它实施例，都属于本发明保护的范围。
为了提高连接查询的效率，本发明实施例提供一种处理连接查询的方法，该方法具体可以应用于集群式的大数据分析系统，如图2所示，该系统包括Client(客户端)、Metastore(元数据存储单元)、一个主节点(master)、多个工作节点(workers)以及DFS(Distribute File System，分布式文件系统)。
其中，master中包含表簇管理模块和SQL引擎，表簇管理模块中还包括工作负载分析子模块以及簇索引维护子模块。DFS中包含多个节点(node),每个节点中包含至少一个数据块(block)。
结合图2，工作负载(workload)是由多个历史SQL查询组成的一个查询集合，可以通过记录客户端在一段预设时间周期内提交的历史SQL查询来得到。主节点可以通过对工作负载中的历史SQL查询记录进行分析来构建表簇和簇索引结构，将建立好的表簇存储在Metastore中，之后由工作节点对表簇中的数据进行重组，并将重组后的数据存储在DFS中相应节点中。当主节点接收到客户端发送的SQL查询请求时，通过SQL引擎执行连接查询的过程，在连接查询的过程中，SQL引擎可以调用工作节点读取DFS中存储的数据，且工作节点还能够完成一些计算工作，具体的处理连接查询的方法可参考下述实施例。
结合图2所示的大数据分析系统，为了提高连接查询的效率，本发明实施例提供一种处理连接查询的方法，如图3所示，该方法包括：
301、主节点确定频繁表组合。
其中，频繁表组合为在历史查询记录中的出现频率大于预设值的表组合，表组合包括连接键以及通过连接键进行连接的表。
连接键为表格之间的连接属性，即表格之间公共列的属性。表组合的表示形式为：
TG＝(tab1,...,tabi,...,tabN)key＝(key1,...,keyj,...,keyM)
其中，tabi为表组合中的第i张表，keyj为第j个连接键，N为表组合中表的个数，M为表组合中连接键的个数。
需要说明的是，本步骤中的历史查询记录为主节点接收到的终端上报的工作负载，工作负载是由多个历史SQL查询记录组成的一个查询集合。主节点可以根据预设的时间段周期性地从工作负载中确定频繁表组合。
302、主节点根据频繁表组合中的连接键信息创建簇索引，簇索引中索引列的数量与频繁表组合中连接键的数量相同。
其中，一个连接键对应一个索引列，当连接键的个数M等于1时，创建单列索引，当连接键的个数M大于或等于2时，创建复合索引。
需要说明的是，簇索引用于指示频繁表组合中索引列值相同的记录的存储位置，例如，频繁表组合中包含表A和表B，表A和表B具有一个公共列，簇索引用于指示表A和表B中的索引列值相同的记录的存储位置。
303、主节点控制工作节点根据簇索引中的索引列进行洗牌操作，将索引列值相同的记录集中存放在至少一个数据块中，以形成频繁表组合对应的表簇。
其中，索引列值相同的记录为通过连接键进行连接的表中的记录，一个表簇由一组共享数据块的表组成，表组合中的表的公共列为表簇的簇索引，将表组合中的所有表中包含相同索引列值的记录存放在至少一个数据块中，使得包含相同索引列值的记录的存放位置较为集中。
例如，如果一个数据块的存储空间足以存储表组合中的所有表包含相同索引列值的记录，则将这些索引列值相同的记录集中存放在一个数据块中。
需要说明的是，当需要进行洗牌操作时，主节点会向工作节点发送控制指令，控制工作节点进行洗牌操作，并将索引列值相同的记录集中存放在DFS中的至少一个数据块中。
还需说明的是，步骤301至步骤302具体由主节点中的表簇管理模块来执行。
本发明实施例提供的处理连接查询的方法，确定频繁表组合，然后根据频繁表组合中的连接键信息创建簇索引，再根据簇索引中的索引列进行shuffle操作，将索引列值相同的记录集中存放在至少一个数据块中，从而形成频繁表组合对应的表簇，通过这种方法，可以将连接键相同的记录尽可能的存放在一个数据块中，使得数据块中数据的连接键值较为集中，大大减少了执行连接查询时，在map阶段对数据分桶时的计算量，且由于一个数据块中存储的基本都是连接键相同的数据，此时，一个数据块中就不会出现太多分桶的数据，可以减少reduce阶段将数据传输至reduce节点 所需的时间，提高了连接查询的效率。
结合图2，需要说明的是，本发明实施例中的频繁表组合是主节点根据客户端上报的工作负载确定的，基于此，在本发明实施例提供的另一种实现方式中，对确定频繁表组合的方法进行了详细说明，如图4所示，上述步骤301、主节点确定频繁表组合，具体可以实现为步骤3011至步骤3013。
3011、主节点从历史查询记录中提取表组合，生成表组合集合。
其中，主节点会对工作负载中的每一条查询记录进行形式转换，从SQL查询语句中提取出使用连接键进行连接操作的表，以形成一个表组合。如果一条查询记录中有多条查询语句，则根据每条查询语句生成一个表组合，将生成的所有表组合组成表组合集合。
例如，其中一条查询记录为：
web_sales ws JOIN warehouse w ON ws.ws_warehouse_sk＝w.w_warehouse_sk,
catalog_sales cs JOIN ship_mode sm ON cs.cs_mode_sk＝sm.sm_ship_mode_sk
可见，这条查询语句中包含两个连接查询，对第一个连接查询进行形式转换，得到的表组合为：
TQ1：(web_sales，warehouse)key＝warehouse_sk。
其中，web_sales和warehouse是两个表格，这两个表格之间的连接键为warehouse_sk。
对第二个连接查询进行形式转换，得到的表组合为：
TQ2：(catalog_sales，ship_mode)key＝ship_mode_sk。
其中，catalog_sales和ship_mode是两个表格，这两个表格之间的连接键为ship_mode_sk。
3012、主节点从表组合集合中筛选出出现频率大于预设值的表组合。
在生成表组合集合之后，可以利用频繁项集挖掘算法计算在表组合集合中出现频率大于预设值的表组合，将在表组合集合中出现频率大于预设值的表组合组成频繁表组合集合。
3013、主节点从出现频率大于预设值的表组合中删除冗余表组合，确定剩余的表组合为频繁表组合。
需要说明的是，上述步骤3012生成的频繁表组合集合中可能会存在冗余表组合，可以采用以下三种规则中的任一种或多种来过滤冗余组合。
规则一、
当存在包含的表相同，但是连接键不同的至少两个表组合时，保留连接键最多的表组合；
例如，存在表组合TG1：(catalog_sales，catalog_returns)key＝item_sk以及表组合TG2：(catalog_sales，catalog_returns)key＝item_sk and order_number。
可以看出，TG1和TG2中包含的表都是catalog_sales和catalog_returns，但是TG1只有一个连接键item_sk，而TG2有两个连接键item_sk和order_number。显然，TG1和TG2包含的表相同，但是TG2的连接键比TG1的连接键多，所以保留表组合TG2，将表组合TG1删除。
规则二、
当存在两个表组合所包含的连接键相同，且一个表组合中的表组成的集合是另一个表组合中的表组成的集合的子集时，删除包含表较少的表组合。
例如，存在表组合TG1：(catalog_sales，catalog_returns)key＝item_sk以及表组合TG2：(catalog_sales，catalog_returns，item)key＝item_sk。
可以看出，TG1和TG2包含的连接键相同，都是item_sk，TG1中的表组成的集合{catalog_sales，catalog_returns}是TG2中的表组成的集合{catalog_sales，catalog_returns，item}的子集，所以删除表组合TG1，保留表组合TG2。
规则三、
当存在相同表包含于至少两个表组合中时，只将相同表保留于至少两个表组合中出现频率最高的一个表组合中。
例如，存在表组合TG1：(catalog_sales，ship_mode)key＝item_sk、表组合TG2：(catalog_sales，web_sales，item)key＝item_sk and order_number，以及表组合TG3：(catalog_sales，catalog_returns，item)key＝item_sk。
可以看出，TG1、TG2和TG3中都包含catalog_sales，假设通过频繁表项挖掘算法确定TG3为出现频率最高的表组合，则将catalog_sales从TG1和TG2中删除，只保留在TG3中，删除之后，TG1变为(ship_mode)key＝item_sk，TG2变为(web_sales，item)key＝item_sk and order_number。
还需说明的是，如果将相同表从某个表组合删除之后，如果该表组合只剩下一个成员表，则无法构成表组合，此时直接将该表组合删除，例如TG1中的catalog_sales被删除后，只剩下一个表ship_mode，此时可以直接将表组合TG1删除。
如图5所示，图5为本发明实施例提供的分布式表簇的创建流程的示例性示意图，在生成表组合后，还需要创建簇索引，进而根据簇索引生成表簇并存储在数据块中，以下对表组合的存储方法进行说明，上述步骤303、主节点控制工作节点根据簇索引中的索引列进行shuffle操作，将通过连接键进行连接的表中索引列值相同的记录存放在至少一个数据块中，具体可以实现为步骤3031至步骤3033。
3031、当包含同一个索引列值的记录的总大小达到一个数据块的存储空间大小的第一预设比例，且未超过一个数据块的存储空间大小时，将包含同一个索引列值的记录存储在一个数据块中。
其中，第一预设比例可以为80％。
结合图5，以频繁表组合TG1：(catalog_sales，catalog_returns)key＝item_sk and order_number为例，连接键item_sk和order_number分别对应一个索引列，连接键item_sk对应的索引列的索引列值包括A和B，连接键order_number对应的索引列的索引列值包括1、2、3、4。
包含第一个索引列值(A，1)的记录的总大小达到了block1总大小的80％，且未超出block1的总存储空间大小，所以可以只将包含索引列值1的记录存储在block1中。
3032、当包含同一个索引列值的记录的总大小超过一个数据块的存储空间大小时，将包含同一个索引列值的记录存储在多个数据块中。
例如，包含第四个索引列值(B，4)的记录的总大小已经超出了一个block存储 空间大小，则将包含索引列值(B，4)的记录存储在block3和block4这两个数据块中。
3033、当包含同一个索引列值的记录的总大小小于一个数据块的存储空间大小的第二预设比例时，将多个包含同一个索引列值的记录存储在一个数据块中。
例如，包含第二个索引列值(A，3)的记录和包含第三个索引列值(B，2)的记录的总大小未超出block2的总存储空间大小，则将包含第二个索引列值(A，3)的记录和包含第三个索引列值(B，2)的记录均存储在block2中。
本发明实施例提供的处理连接查询的方法，通过筛选出现频率大于预设值的表组合，可以使得最后存储的表组合为用户常用的表组合，可以提高查询效率，且将冗余表组合删除，可以节省存储空间，最后，根据包含相同索引列值的记录的总大小和一个数据块存储空间大小的关系，来尽可能地将包含相同索引列值的记录集中存储在一个或多个数据块中，相比较于分散存储在多个数据块中，由于包含相同索引列值的记录集中存储在一个或多个数据块中，可以认为在数据存储在数据块中时已经完成了分桶，减少连接查询过程中分桶所需的时间，所以可以减少连接查询过程所需的时间，提高连接查询的效率。
结合图3所示的方法流程，在建立好表簇之后，即可进行后续的连接查询操作，基于本发明实施例建立的表簇，在本发明实施例提供的一种实现方式中，还提供了基于上述实施例中建立的表簇处理连接查询的过程，如图6所示，该方法包括：
601、主节点接收查询请求，查询请求中包含待查询表组合。
需要说明的是，在建立好表簇之后，当主节点中的SQL引擎接收到SQL查询请求时，需通过查询编译器来确定查询请求所请求进行连接计算的待查询表组合。
602、主节点查找待查询表组合对应的待查询表簇。
在确定待查询表组合之后，主节点中的SQL引擎可以通过表簇管理模块确定待查询表组合对应的待查询表簇。
603、主节点控制工作节点分别将待查询表簇所包含的每个数据块中索引列值的哈希值相同的记录划分至一个分桶中。
需要说明的是，当主节点中的SQL引擎接收到SQL查询请求时，主节点可向工作节点发送控制指令，控制工作节点执行后续的mapreduce过程。
在确定待查询表簇之后,SQL引擎可通过工作节点读取分布式文件系统中的关于待查询表簇的数据，对待查询表簇进行连接查询的过程如图7所示，主节点控制工作节点分别在两个节点上启动map程序，而由于索引列值(key值)相同的数据已经被集中存储在一起，所以map程序中无需进行本地hash的过程，即可完成分桶操作，例如，图8中，block1中只存储了索引列值为1的记录，所以block1中存储的所有记录就是一个分桶，block2中也只存储了索引列值为1的记录，所以block2中存储的所有记录就是一个分桶，block3中分别存储了索引列值为2的记录和部分索引列值为3的记录，且索引列值为2的记录和索引列值为3的记录是独立存储的，所以无需进行本地hash也能将block3中存储的数据分为两个分桶，block4中存储的都是索引列值为3的记录，所以block3中存储所有记录为一个分桶。
604、主节点控制工作节点将待查询表簇中所有数据块中相同分桶中的记录传输至同一数据节点中。
本发明实施例提供的处理连接查询的方法，在建立好表簇之后再接收到查询请求时，确定查询请求中的待查询表组合，然后查找待查询表组合对应的待查询表簇，在待查询表簇对应的节点上执行map任务，由于索引列值相同的数据已经被存储在一起，所以执行map任务时无需进行本地hash的过程也能够完成分桶，减少了对数据分桶时的计算量，节省了CPU开销，且表簇中只存储了包含相同索引列值的记录，对于索引列值不同的记录不需要进行处理，减少了需要处理的数据量以及启动的任务数量和磁盘I/O，之后在洗牌过程中，由于大部分数据块都只对应一个分桶，相当于一个数据块对应一个索引列值，且reduce任务优先安排在map任务的本地执行，reduce节点所需的map任务的输出数据大部分都不需要网络传输，所以大大减小了将索引列值相同的数据传输至reduce节点的过程产生的网络传输开销，缩短了传输时间，提高了连接查询的效率。
上述主要从各个网元之间交互的角度对本发明实施例提供的方案进行了介绍。可以理解的是，各个网元，例如连接查询的装置等为了实现上述功能，其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，本发明能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。
本发明实施例可以根据上述方法示例对图2所示的主节点等进行功能模块的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本发明实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
在采用对应各个功能划分各个功能模块的情况下，图8示出了上述实施例中所涉及的主节点的一种可能的结构示意图，图8具体为图2所示的主节点中的表簇管理模块的结构示意图，主节点包括：确定单元801，创建单元802，洗牌单元803。确定单元801用于支持主节点执行图3中的步骤301，图4中的步骤3011至3013；创建单元802用于支持主节点执行图3中的步骤302；洗牌单元803用于支持主节点控制工作节点执行图3中的步骤303，支持主节点执行图4中的步骤3031至3033。
在采用对应各个功能划分各个功能模块的情况下，图9还示出了上述实施例中所涉及的主节点的另一种可能的结构示意图，图9具体为图2所示的主节点中的SQL引擎的结构示意图，主节点中包括：接收单元901，查找单元902，分桶单元903，传输单元904。其中，接收单元901用于主节点执行图6中的步骤601；查找单元902用于支持主节点执行图6中的步骤602；分桶单元903用于支持主节点控制工作节点执行图6中的步骤603，传输单元904用于支持主节点控制工作节点执行图6中的步骤604。
其中，上述方法实施例涉及的各步骤的所有相关内容均可以援引到对应功能模块的功能描述，在此不再赘述。
在采用集成的单元的情况下，图10示出了上述实施例中所涉及的主节点的一种可能的结构示意图。主节点包括：处理模块1002和通信模块1003。处理模块1002用于对主节点的动作进行控制管理，例如，处理模块1002用于支持图3中的步骤301至303，图4中的步骤3011至3033，图6中的步骤602至603；通信模块1003用于支持主节点与其他网络实体的通信，例如，通信模块1003用于支持图6中的步骤601和604，可以实现与图2或图5中示出的功能模块或网络实体之间的通信。该装置还包括存储模块1001，用于存储主节点的程序代码和数据。
其中，处理模块1002可以是处理器或控制器，例如可以是中央处理器(Central Processing Unit，CPU)，通用处理器，数字信号处理器(Digital Signal Processor，DSP)，专用集成电路(Application-Specific Integrated Circuit，ASIC)，现场可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、晶体管逻辑器件、硬件部件或者其任意组合。其可以实现或执行结合本发明公开内容所描述的各种示例性的逻辑方框，模块和电路。所述处理器也可以是实现计算功能的组合，例如包含一个或多个微处理器组合，DSP和微处理器的组合等等。通信模块1003可以是收发器、收发电路或通信接口等。存储模块1001可以是存储器。
图8至图10为主节点实现为软件时的结构示意图，而主节点还可以以硬件的方式存在，即当处理模块1002为处理器，通信模块1003为收发器，存储模块1001为存储器时，本发明实施例所涉及的主节点可以为图11所示的主节点。
参阅图11所示，当主节点实现为硬件时，包括：处理器1102、收发器1103、存储器1101以及总线1104。其中，收发器1103、处理器1102以及存储器1101通过总线1104相互连接；总线1104可以是外设部件互连标准(Peripheral Component Interconnect，PCI)总线或扩展工业标准结构(Extended Industry Standard Architecture，EISA)总线等。所述总线可以分为地址总线、数据总线、控制总线等。为便于表示，图11中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
需要说明的是，本发明实施例中的主节点和工作节点可以分别为一个单机设备，也可以以软件的方式存在，例如在一个计算机集群系统中，主节点的功能和工作节点的功能可以由该系统中不同的虚拟机实现。
结合本发明公开内容所描述的方法或者算法的步骤可以硬件的方式来实现，也可以是由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(Random Access Memory，RAM)、闪存、只读存储器(Read Only Memory，ROM)、可擦除可编程只读存储器(Erasable Programmable ROM，EPROM)、电可擦可编程只读存储器(Electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、只读光盘(CD-ROM)或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于核心网接口设备中。当然，处理器和存储介质也可以作为分立组件存在于核心网接口设备中。
本领域技术人员应该可以意识到，在上述一个或多个示例中，本发明所描述的 功能可以用硬件、软件、固件或它们的任意组合来实现。当使用软件实现时，可以将这些功能存储在计算机可读介质中或者作为计算机可读介质上的一个或多个指令或代码进行传输。计算机可读介质包括计算机存储介质和通信介质，其中通信介质包括便于从一个地方向另一个地方传送计算机程序的任何介质。存储介质可以是通用或专用计算机能够存取的任何可用介质。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于设备实施例而言，由于其基本相似于方法实施例，所以描述得比较简单，相关之处参见方法实施例的部分说明即可。
以上所述，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到的变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应该以权利要求的保护范围为准。

Claims (14)

  1. 一种处理连接查询的方法，其特征在于，包括：
    确定频繁表组合，所述频繁表组合为在历史查询记录中的出现频率大于预设值的表组合，所述表组合包括连接键以及通过连接键进行连接的表；
    根据所述频繁表组合中的连接键信息创建簇索引，所述簇索引中索引列的数量与所述频繁表组合中连接键的数量相同，所述簇索引用于指示所述频繁表组合中索引列值相同的记录的存储位置；
    根据所述簇索引中的索引列进行洗牌操作，将索引列值相同的记录集中存放在至少一个数据块中，以形成所述频繁表组合对应的表簇，所述索引列值相同的记录为通过所述连接键进行连接的表中的记录。
  2. 根据权利要求1所述的处理连接查询的方法，其特征在于，在所述根据所述簇索引中的索引列进行洗牌操作，将索引列值相同的记录集中存放在至少一个数据块中之后，所述方法还包括：
    接收查询请求，所述查询请求中包含待查询表组合；
    查找所述待查询表组合对应的待查询表簇；
    分别将所述待查询表簇对应的节点所包含的每个数据块中索引列值的哈希值相同的记录划分至一个分桶中；
    将所述待查询表簇中所有数据块中相同分桶中的记录传输至同一数据节点中。
  3. 根据权利要求2所述的处理连接查询的方法，其特征在于，所述确定频繁表组合，包括：
    从所述历史查询记录中提取表组合，生成表组合集合；
    从所述表组合集合中筛选出出现频率大于预设值的表组合；
    从出现频率大于预设值的表组合中删除冗余表组合，确定剩余的表组合为频繁表组合。
  4. 根据权利要求3所述的处理连接查询的方法，其特征在于，所述从出现频率大于预设值的表组合中删除冗余表组合，包括：
    当存在包含的表相同，但是连接键不同的至少两个表组合时，保留连接键最多的表组合；
    当存在两个表组合所包含的连接键相同，且一个表组合中的表组成的集合是另一个表组合中表组成的集合的子集时，删除包含表较少的表组合；
    当存在相同表包含于至少两个表组合中时，只将所述相同表保留于所述至少两个表组合中出现频率最高的一个表组合中。
  5. 根据权利要求1至4任一项所述的处理连接查询的方法，其特征在于，所述根据所述簇索引中的索引列进行洗牌操作，将索引列值相同的记录集中存放在至少一个数据块中，包括：
    当包含同一个索引列值的记录的总大小达到一个数据块的存储空间大小的第一预设比例，且未超过一个数据块的存储空间大小时，将所述包含同一个索引列值的记录存储在一个数据块中；
    当包含同一个索引列值的记录的总大小超过一个数据块的存储空间大小时，将所 述包含同一个索引列值的记录存储在多个数据块中；
    当包含同一个索引列值的记录的总大小小于一个数据块的存储空间大小的第二预设比例时，将多个所述包含同一个索引列值的记录存储在一个数据块中。
  6. 根据权利要求1至5任一项所述的处理连接查询的方法，其特征在于，表组合的表示形式为：
    TG＝(tab1,...,tabi,...,tabN)key＝(key1,...,keyj,...,keyM)
    其中，tabi为表组合中的第i张表，keyj为第j个连接键，N为表组合中表的个数，M为表组合中连接键的个数。
  7. 一种处理连接查询的方法，其特征在于，包括：
    接收查询请求，所述查询请求中包含待查询表组合；
    在分布式文件系统DFS中查找与所述待查询表组合对应的待查询表簇，所述DFS中存储了频繁表组合对应的表簇，所述频繁表组合为在历史查询记录中的出现频率大于预设值的表组合，所述表组合包括连接键以及通过连接键进行连接的表，所述频繁表组合中连接键的数量与簇索引中索引列的数量相同，所述簇索引用于指示所述频繁表组合中索引列值相同的记录的存储位置，其中，所述频繁表组合中索引列值相同的记录集中存放在至少一个数据块中，形成所述频繁表组合对应的表簇；
    分别将所述待查询表簇对应的节点所包含的每个数据块中索引列值的哈希值相同的记录划分至一个分桶中；
    将所述待查询表簇中所有数据块中相同分桶中的记录传输至同一数据节点中。
  8. 一种处理连接查询的装置，其特征在于，包括：
    确定单元，用于确定频繁表组合，所述频繁表组合为在历史查询记录中的出现频率大于预设值的表组合，所述表组合包括连接键以及通过连接键进行连接的表；
    创建单元，用于根据所述确定单元确定的所述频繁表组合中的连接键信息创建簇索引，所述簇索引中索引列的数量与所述频繁表组合中连接键的数量相同；
    洗牌单元，用于根据所述创建单元创建的所述簇索引中的索引列进行洗牌操作，将索引列值相同的记录集中存放在至少一个数据块中，以形成所述频繁表组合对应的表簇，所述索引列值相同的记录为通过所述连接键进行连接的表中的记录。
  9. 根据权利要求8所述的处理连接查询的装置，其特征在于，所述装置还包括
    接收单元，用于接收查询请求，所述查询请求中包含待查询表组合；
    查找单元，用于查找所述待查询表组合对应的待查询表簇；
    分桶单元，用于分别将所述待查询表簇对应的节点所包含的每个数据块中索引列值的哈希值相同的记录划分至一个分桶中；
    传输单元，用于根据所述分桶单元的分桶结果，将所述待查询表簇中所有数据块中相同分桶中的记录传输至同一数据节点中。
  10. 根据权利要求9所述的处理连接查询的装置，其特征在于，
    所述确定单元，还用于从所述历史查询记录中提取表组合，生成表组合集合；从所述表组合集合中筛选出出现频率大于预设值的表组合；从出现频率大于预设值的表组合中删除冗余表组合，确定剩余的表组合为频繁表组合。
  11. 根据权利要求10所述的处理连接查询的装置，其特征在于，
    所述确定单元，还用于当存在包含的表相同，但是连接键不同的至少两个表组合时，保留连接键最多的表组合；当存在两个表组合所包含的连接键相同，且一个表组合中的表组成的集合是另一个表组合中表组成的集合的子集时，删除包含表较少的表组合；当存在相同表包含于至少两个表组合中时，只将所述相同表保留于所述至少两个表组合中出现频率最高的一个表组合中。
  12. 根据权利要求8至11任一项所述的处理连接查询的装置，其特征在于，
    所述洗牌单元，还用于当包含同一个索引列值的记录的总大小达到一个数据块的存储空间大小的第一预设比例，且未超过一个数据块的存储空间大小时，将所述包含同一个索引列值的记录存储在一个数据块中；当包含同一个索引列值的记录的总大小超过一个数据块的存储空间大小时，将所述包含同一个索引列值的记录存储在多个数据块中；当包含同一个索引列值的记录的总大小小于一个数据块的存储空间大小的第二预设比例时，将多个所述包含同一个索引列值的记录存储在一个数据块中。
  13. 根据权利要求8至12任一项所述的处理连接查询的装置，其特征在于，表组合的表示形式为：
    TG＝(tab1,...,tabi,...,tabN)key＝(key1,...,keyj,...,keyM)
    其中，tabi为表组合中的第i张表，keyj为第j个连接键，N为表组合中表的个数，M为表组合中连接键的个数。
  14. 一种连处理接查询的装置，其特征在于，包括：
    接收单元，用于接收查询请求，所述查询请求中包含待查询表组合；
    查找单元，用于在分布式文件系统DFS中查找与所述接收单元接收的所述待查询表组合对应的待查询表簇，所述DFS中存储了频繁表组合对应的表簇，所述频繁表组合为在历史查询记录中的出现频率大于预设值的表组合，所述表组合包括连接键以及通过连接键进行连接的表，所述频繁表组合中连接键的数量与簇索引中索引列的数量相同，所述簇索引用于指示所述频繁表组合中索引列值相同的记录的存储位置，其中，所述频繁表组合中索引列值相同的记录集中存放在至少一个数据块中，形成所述频繁表组合对应的表簇；
    分桶单元，用于分别将所述待查询表簇对应的节点所包含的每个数据块中索引列值的哈希值相同的记录划分至一个分桶中；
    传输单元，用于根据所述分桶单元的分桶结果将所述待查询表簇中所有数据块中相同分桶中的记录传输至同一数据节点中。
PCT/CN2017/071568 2016-08-31 2017-01-18 一种处理连接查询的方法及装置 Ceased WO2018040488A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP17844799.1A EP3499388B1 (en) 2016-08-31 2017-01-18 Method and device for processing join query
US16/287,510 US11030196B2 (en) 2016-08-31 2019-02-27 Method and apparatus for processing join query

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610797295.9 2016-08-31
CN201610797295.9A CN107784030B (zh) 2016-08-31 2016-08-31 一种处理连接查询的方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/287,510 Continuation US11030196B2 (en) 2016-08-31 2019-02-27 Method and apparatus for processing join query

Publications (1)

Publication Number Publication Date
WO2018040488A1 true WO2018040488A1 (zh) 2018-03-08

Family

ID=61299963

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/071568 Ceased WO2018040488A1 (zh) 2016-08-31 2017-01-18 一种处理连接查询的方法及装置

Country Status (4)

Country Link
US (1) US11030196B2 (zh)
EP (1) EP3499388B1 (zh)
CN (1) CN107784030B (zh)
WO (1) WO2018040488A1 (zh)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109063186A (zh) * 2018-08-27 2018-12-21 郑州云海信息技术有限公司 一种跨表查询方法及相关装置
US20200233882A1 (en) * 2019-01-18 2020-07-23 Huawei Technologies Co., Ltd. Bucketizing data into buckets for processing by code modules
CN111881145A (zh) * 2020-07-31 2020-11-03 北京致远互联软件股份有限公司 业务数据表的处理方法、装置、服务器及存储介质
CN112988801B (zh) * 2021-04-07 2024-08-20 拉卡拉支付股份有限公司 数据处理方法、装置、电子设备、存储介质及程序产品
CN113469801A (zh) * 2021-06-30 2021-10-01 建信金融科技有限责任公司 审核结果的确定方法和装置
CN113535756B (zh) * 2021-07-30 2023-05-30 上海达梦数据库有限公司 数据查询方法、装置、设备及存储介质
CN114329155A (zh) * 2021-12-30 2022-04-12 北京诺司时空科技有限公司 一种包含时序数据库的多模态存储缓存系统及查询方法
CN115544173B (zh) * 2022-11-29 2023-10-03 创意信息技术股份有限公司 可线性扩展的分布式数据库
CN115686799B (zh) * 2022-12-29 2023-04-07 中国华能集团清洁能源技术研究院有限公司 一种大数据平台中的任务调度方法、装置、设备及介质
CN119537468A (zh) * 2023-08-30 2025-02-28 华为技术有限公司 一种数据处理方法及相关设备
CN118626526B (zh) * 2024-08-15 2024-12-03 金篆信科有限责任公司 基于数据页索引的Hash Join查询方法、装置、系统及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090055370A1 (en) * 2008-10-10 2009-02-26 Business.Com System and method for data warehousing and analytics on a distributed file system
CN102426609A (zh) * 2011-12-28 2012-04-25 厦门市美亚柏科信息股份有限公司 一种基于MapReduce编程架构的索引生成方法和装置
CN103488657A (zh) * 2012-06-14 2014-01-01 华为技术有限公司 一种数据表关联方法及装置
CN105095515A (zh) * 2015-09-11 2015-11-25 北京金山安全软件有限公司 支持快速查询Map-Reduce输出结果的分桶方法、装置及设备
CN105357311A (zh) * 2015-11-23 2016-02-24 中国南方电网有限责任公司 一种云计算技术的二次设备大数据存储与处理方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7254574B2 (en) * 2004-03-08 2007-08-07 Microsoft Corporation Structured indexes on results of function applications over data
JP5484470B2 (ja) 2008-09-19 2014-05-07 オラクル・インターナショナル・コーポレイション オフロードされたブルームフィルタを伴うインテリジェントストレージにおける協調並列フィルタ処理を用いるハッシュジョイン
US20120246158A1 (en) 2011-03-25 2012-09-27 Microsoft Corporation Co-range partition for query plan optimization and data-parallel programming model
CN102323947B (zh) * 2011-09-05 2013-07-10 东北大学 环形架构数据库上预连接表的生成方法
CN102955843B (zh) * 2012-09-20 2015-07-22 北大方正集团有限公司 一种键值数据库的多键查找实现方法
US9195701B2 (en) * 2012-10-29 2015-11-24 Futurewei Technologies, Inc. System and method for flexible distributed massively parallel processing (MPP) database
CN104871153B8 (zh) * 2012-10-29 2019-02-01 华为技术有限公司 用于分布式大规模并行处理数据库的方法和系统
CN104850572B (zh) * 2014-11-18 2018-11-23 中兴通讯股份有限公司 HBase非主键索引构建与查询方法及其系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090055370A1 (en) * 2008-10-10 2009-02-26 Business.Com System and method for data warehousing and analytics on a distributed file system
CN102426609A (zh) * 2011-12-28 2012-04-25 厦门市美亚柏科信息股份有限公司 一种基于MapReduce编程架构的索引生成方法和装置
CN103488657A (zh) * 2012-06-14 2014-01-01 华为技术有限公司 一种数据表关联方法及装置
CN105095515A (zh) * 2015-09-11 2015-11-25 北京金山安全软件有限公司 支持快速查询Map-Reduce输出结果的分桶方法、装置及设备
CN105357311A (zh) * 2015-11-23 2016-02-24 中国南方电网有限责任公司 一种云计算技术的二次设备大数据存储与处理方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3499388A4 *

Also Published As

Publication number Publication date
CN107784030B (zh) 2020-04-28
EP3499388A4 (en) 2019-07-10
US20190197040A1 (en) 2019-06-27
EP3499388A1 (en) 2019-06-19
EP3499388B1 (en) 2023-03-01
CN107784030A (zh) 2018-03-09
US11030196B2 (en) 2021-06-08

Similar Documents

Publication Publication Date Title
US11030196B2 (en) Method and apparatus for processing join query
US9697254B2 (en) Graph traversal operator inside a column store
CN107463637B (zh) 一种分布式NewSQL数据库系统和数据储存方法
JP6697392B2 (ja) 半構造データスキーマのトランスペアレントディスカバリ
US8972405B1 (en) Storage resource management information modeling in a cloud processing environment
US9405855B2 (en) Processing diff-queries on property graphs
CN111221791A (zh) 一种多源异构数据导入数据湖的方法
US9734176B2 (en) Index merge ordering
CN115552390A (zh) 无服务器数据湖索引子系统及应用编程接口
US10810174B2 (en) Database management system, database server, and database management method
Tang et al. Toward coordination-free and reconfigurable mixed concurrency control
JPWO2017013701A1 (ja) 計算機システム及びデータベース管理方法
CN106569896A (zh) 一种数据分发及并行处理方法和系统
US9229969B2 (en) Management of searches in a database system
CN106815318B (zh) 一种时序数据库的集群化方法及系统
US10997160B1 (en) Streaming committed transaction updates to a data store
WO2024011932A1 (zh) 一种文件管理方法及相关设备
WO2020192225A1 (zh) 一种面向Spark的遥感数据索引方法、系统及电子设备
CN108932258B (zh) 数据索引处理方法及装置
Ye Research on the key technology of big data service in university library
CN106446039A (zh) 聚合式大数据查询方法及装置
CN113268483B (zh) 请求处理方法和装置、电子设备和存储介质
WO2024234405A1 (zh) 一种应用于大数据的实体增强规则挖掘方法及装置
Li et al. Evaluating spatial keyword queries under the mapreduce framework
Jamadagni et al. GoDB: From batch processing to distributed querying over property graphs

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17844799

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017844799

Country of ref document: EP

Effective date: 20190313