CN102968356A - Data processing method of cloud storage system - Google Patents

Abstract

The invention relates to a data processing method of a cloud storage system. Data storage and data extraction of the cloud storage system are respectively subjected to reed solomon coded data error-correction coding process and reed solomon error correction decoding process. According to the data processing method of the cloud storage system, disclosed by the invention, the safety and the restorability of data in the cloud storage system are improved, the number of backup data is reduced, the storage space of data can be saved, and the cost is reduced.

Description

Data processing method of cloud storage system

technical field

The invention relates to a cloud storage system, and is characterized by a data processing method based on the Solomon cloud storage system.

Background technique

In today's turbulent era, cloud storage, as the infrastructure and the most extensive application of the cloud, has received great attention. In the cloud storage system, user data is stored in the cloud of the system, and the storage nodes that make up the cloud are beyond the control of the user. User data may be mined and compared or maliciously tampered with by unauthorized third parties.

At the same time, when one or more storage nodes in the cloud are missing or fail (as the cloud expands, the probability of storage node failure will increase), the possibility of user data loss is extremely high. The above situation shows that the development of cloud storage urgently needs a security mechanism that can fully guarantee the integrity, privacy and reliability of user data.

At present, all cloud storage technologies are based on HDFS (Hadoop Distributed File System, that is, Hadoop Distributed File System) in a system similar to Hadoop (an open source software for cloud computing). This technology mainly divides a data file into several blocks according to the set size, and then improves reliability by fully backing up each data (for example, HDFS in Hadoop has 3 identical backups). The disadvantage of this technology is that it wastes storage space.

The Solomon error-correcting coding method in: the principle is to calculate the remainder after dividing the information code symbol polynomial by the check code generator polynomial, the specific formula is:

F mod D = C;

Among them, F is the original data, D is the generator polynomial, and C is the generated redundant error correction data. mod is the remainder operation.

In decoding, for the sake of simplicity, it is assumed that the stored original information symbols are m ₃ , m ₂ , m ₁ , m ₀ and the resulting check symbols Q ₁ , Q ₀ , and the read symbols are m ₃ ′ , m ₂ ′, m ₁ ′, m ₀ ′, Q ₁ ′ and Q ₀ ′, if the obtained syndromes s ₀ and s ₁ are not all 0, it means that there is an error, and then by calculating the error polynomial and the error value to correct the error.

This method is widely used in the processing of DVD disc data. This encoding method can improve the error correction ability of the original data very well, and can reduce the random error rate of the data from 2×10 ^-2 to 1×10 ^-15 . In the present invention, we use this coding method to carry out Solomon coding in a data block in the form of an array in the horizontal and vertical directions respectively to obtain horizontal and vertical error correction redundant data, so that the data has been doubled. Error correction protection, The error correction ability is improved, and these redundant data only account for 13% of the original data volume.

It is precisely because data error correction can be performed so efficiently in the case of low data redundancy, and the general cloud storage system does not adopt this error correction method, but only achieves the ability of data recovery through data backup. All storage systems must back up 3 or more copies, which will obviously waste data storage space and increase costs.

Contents of the invention

The technical problem to be solved by the present invention is to provide a data processing method for a cloud storage system, which will improve the security and recoverability of data in the cloud storage system, reduce the number of data backups, and greatly save data storage space and reduce costs.

Technical solution of the present invention is as follows:

A data processing method for a cloud storage system, which is characterized in that the data storage and data extraction of the cloud storage system adopt the Solomon coded data error correction encoding process and the Solomon error correction decoding process respectively.

The data storage method includes the following specific steps:

① Divide the original data to be stored into K frame data, each frame data contains the same fixed-length N-bit data, when the data length of the last original data frame is less than N, fill the last data frame with data '0' , to reach the length N, where K is a positive integer greater than or greater than 1, and the value range of N is: 200<N<2000:

2. each data frame is added numbering, i.e. ID, obtains a new data frame, and described ID length is 4 bytes, starts incrementing from 0001, so the length of new data frame is (N+ 4);

3. described new K data frames are recombined into W data blocks, each data block contains M data frames, forming a data matrix of M*(N+4), when the data frame of the last data block When the number is less than M, the data block is filled with '0' data frame, so that the last data block reaches a fixed data frame M, where the value ranges of M and W are: 200<M<2000, W= K/M;

4. the Solomon product code error correction coding method in the described data block is adopted to carry out error correction coding: the row and column of the ith data block are respectively added P ₀ , P _I error-corrected redundant data are encoded, converted into a Solomon data block containing (M+P ₀ )*(N+4+P _I ) pieces of data, where P ₀ and P _I are the redundancy for error correction added to one row and one column in the data block respectively The number of remaining data, and 0<P ₀ <M/2, 0<P _I <M/2, 1<i≤M;

⑤ Decompose each of the Solomon data blocks in each column into M+P ₀ data slices, and store the data of the M+P ₀ data slices of the same data block on multiple memories of the cloud storage system respectively , and in the same memory, the number of slices of the same data block ≤ P _I /2 slices;

The described data extraction method comprises the following steps:

① Read the data slices belonging to the same data block. If there are P ₀ /2 data reading errors in the i-th slice, according to the error correction decoding algorithm of the Solomon product code in the i-th slice, the data of the i-th slice is corrected. restore the original data;

② After reading all the data slices of the same data block, if less than P _I /2 slices are lost or cannot be read when reading the data block, the computer will decode the less than P I /2 slices according to the decoding algorithm of the Solomon product code in the computer Error correction in P _I /2 pieces of data to restore the original data;

③Repeat steps ① and ②, read and process all data blocks belonging to the same original data, remove error correction redundant data, arrange the new data frames of all data blocks in the order of the original number ID, and then remove the number to obtain the original stored data.

Technical effect of the present invention:

1. The most remarkable effect of the present invention is that the row and column of the original data block are all subjected to the Solomon error correction coding, and the ability of double error correction is achieved, and these error-corrected data blocks are re-sharded for storage to different storages in the cloud system, so that not only can error correction be performed on the slice data on each storage, but also when a certain number of storages in the cloud system fail and some slice data cannot be read, complete recovery can still be obtained. Recovery, which greatly improves the reliability of the system.

2. The general cloud storage system needs to copy multiple copies of the original data to ensure the safety of the data (the general cloud storage system must back up 3 copies or more), the present invention can reduce the amount of redundant storage used when ensuring the security of the stored data ( You only need to back up two copies or one copy), which will greatly save data storage space, reduce costs, and make more full use of the cloud storage system space.

3. Another notable feature of the present invention is that since the data is stored in multiple storages, when someone outside illegally invades a single storage in the cloud system, the obtained data will be incomplete, that is, Improved data security in the face of illegal intrusion from outside the system.

Detailed ways

Below in conjunction with example the present invention will be further described, but should not limit protection scope of the present invention with this.

This example is used to store a 100MB original data in the cloud. The specific implementation steps are as follows

Step 1: Divide a 100MB original data to be stored into 102401 frames of data, each frame of data has a fixed length of 1020 bits.

The data to be stored is divided into 102401 frames of data with a fixed length of 1020. When the length of the last original data frame is less than 255, the last data frame is filled with data '0' to reach the length of 1020.

Step 2, adding a number, ie ID, to each of the data frames to obtain a new data frame.

The ID is added to each frame of data. The ID is 4 bytes long and increments from 1. In the example, the ID number is from 1 to 102401.

Step 3, recombine the data frame into several data blocks, each data block contains 1024 data frames, and a total of 100 such data blocks are obtained.

The data frames are recombined into data blocks, and these data frames will form a 1024*1024 data matrix to form a data block. If the number of data frames in the last data block is less than 1024, fill the last data block with data' 0', to reach a fixed data block number of 1024.

Step 4: Encode the row and column of the i-th data block containing 1024*1024 data respectively through the Solomon error correction coding in , and convert it into a block containing (1024+PO)*(1024+PI) data Data block, where 0<i<=M. PO=PI=100 is the number of redundant data added for error correction in one row and one column in the data block. The specific formula is:

F mod D = C;

In this embodiment, F is 1024-bit data, D is a generator polynomial, and C is 100 redundant error correction data generated. mod is the remainder operation.

Step 5, decompose each data block obtained above into 1024+100 pieces by column, and store several pieces of data obtained from the same data block in the memory of several cloud storage systems, and the same data in the same memory The number of slices of a block cannot exceed 50 blocks.

Step 6: In the example, when reading data from the cloud storage system, the data to be extracted is decoded through the decoding algorithm in the RS-PC.

The reading of data from the cloud storage system refers to:

1) Read different slices belonging to the same data block, and if there are less than 50 data reading errors in the i-th slice, according to the Solomon error correction decoding algorithm in it, the entire slice of data can be corrected and restored to the original data;

2) After reading different slices of the same data block, if less than 50 slices are lost or cannot be read when reading these slices, according to the Solomon error correction decoding algorithm in the program, the data of this segment can be corrected and recovered .

3) After reading all the different data blocks belonging to the same original data, remove the error correction redundant data, arrange these data blocks in order of numbers (1 to 102401), and then remove the numbers, and finally obtain the original stored data.

Claims (3)

1. the data processing method of a cloud storage system is characterized in that, the data of cloud storage system is deposited in the Saloman coded data Error Correction of Coding of extracting in adopting respectively with data process and inner Saloman error correction decoding processing.

2. the data processing method of cloud storage system according to claim 1 is characterized in that described data store method, comprises following concrete steps:

1. raw data to be stored is divided into K frame data, each frame data comprises the data of identical regular length N bit, when the data length of last initial data frame during less than N, to last blocks of data frame padding data ' 0 ', to reach length N, wherein K is greater than greater than 1 positive integer, and the span of N is: 200＜N＜2000:

2. described each Frame is added numbering, namely ID obtains a new data frame, and described ID length is 4 bytes, progressively increases since 0001, so the length of new Frame is (N+4);

3. be W data block with a described new K Frame recombinant, each data block comprises M Frame, form the data matrix of a M* (N+4), when the number of the Frame of last data block during less than M, this data block is filled ' 0 ' Frame, so that last data block reaches fixing Frame M, wherein M, the span of W is respectively: 200＜M＜2000, W=K/M;

4. the Saloman product code error correction/encoding method in described data block being adopted carries out Error Correction of Coding: the row and column of i data block is added respectively P ₀, P _IThe redundant data of individual error correction is encoded, and changes into one and comprises (M+P ₀) * (N+4+P _I) individual data in the Saloman data block, wherein, P ₀, P _IBe respectively the redundant data number that is used for error correction that delegation in the data block and row increase, and 0＜P ₀＜M/2,0＜P _I＜M/2), 1＜i≤M;

5. described Saloman data block in each is resolved into M+P by row ₀Individual data slice is with the M+P of same data block ₀The data of individual data slice store into respectively on a plurality of storeies of cloud storage system, and in the same storer, the sheet number≤P of same data block _I/ 2;

3. the data processing method of described cloud storage system according to claim 1 is characterized in that described data extraction method comprises the following steps:

1. read out the data slice that belongs to same data block, if P occurs in the i sheet ₀/ 2 data read errors, the error correction decoding algorithm according to inner Saloman product code to the correcting data error of this i sheet, recovers raw data;

2. all data slice of same data block have been read, if the P of being less than is arranged when reading this data block _I/ 2 go out active or can't read, and computing machine is less than P by the decoding algorithm of inner Saloman product code to this _I/ 2 correcting data errors recover raw data;

3. 1., 2. repeating step reads and handles all data blocks that belong to same raw data, removes the error correction redundancy data, and the new data frame of all data blocks is arranged sequentially by former numbering ID, removes numbering again, obtains the data of original storage.