WO2020041928A1 - Procédé et système de stockage de données et dispositif terminal - Google Patents

Procédé et système de stockage de données et dispositif terminal Download PDF

Info

Publication number
WO2020041928A1
WO2020041928A1 PCT/CN2018/102490 CN2018102490W WO2020041928A1 WO 2020041928 A1 WO2020041928 A1 WO 2020041928A1 CN 2018102490 W CN2018102490 W CN 2018102490W WO 2020041928 A1 WO2020041928 A1 WO 2020041928A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
data block
block
file
index table
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/102490
Other languages
English (en)
Chinese (zh)
Inventor
王斌
刘文涛
归建章
苏晓天
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Streamax Technology Co Ltd
Original Assignee
Streamax Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Streamax Technology Co Ltd filed Critical Streamax Technology Co Ltd
Priority to CN201880001157.2A priority Critical patent/CN110149803B/zh
Priority to PCT/CN2018/102490 priority patent/WO2020041928A1/fr
Publication of WO2020041928A1 publication Critical patent/WO2020041928A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the present application belongs to the field of computer technology, and particularly relates to a data storage method, system, and terminal device.
  • the data of one channel is usually used as a file. Since all channels will generate data at the same time, multiple files will be written to the SMR disk at the same time. This will increase the write position and cause the write speed to decrease.
  • embodiments of the present application provide a data storage method, system, and terminal device to solve the problem that the writing speed decreases when multi-channel data is written to an SMR disk in the prior art.
  • the data block index table includes multiple data block index nodes
  • the in-block index table in the cache is written to the tail of the free data block, and the step of finding the free data block according to the data block index table is continued.
  • a second aspect of the embodiments of the present application provides a data storage system, including:
  • the idle data block search module is used to read the data block index table, find the free data block according to the data block index table, and update the data block index node corresponding to the idle data block.
  • the data block index table includes multiple data block index nodes;
  • a cache allocation module configured to allocate a cache for an in-block index table in an idle data block
  • Cyclic reading module which is used to read the data in the data buffer area of each channel cyclically;
  • a data writing module is configured to read data of a preset time length in the data buffer area of the target channel if the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, and set the preset time The length of data is written into the free data block, and the data block index node corresponding to the free data block and the in-block index table in the cache are updated, and the target channel is any one of all channels;
  • a third aspect of the embodiments of the present application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • the processor executes the computer program, the data storage method described above is implemented. step.
  • a fourth aspect of the embodiments of the present application provides a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program.
  • the computer program is executed by one or more processors, the steps of the data storage method described above are implemented.
  • the embodiment of the present application first reads a data block index table, searches for a free data block according to the data block index table, and updates a data block index node corresponding to the idle data block, allocates a buffer for the in-block index table in the free data block, and then reads the data circularly Take the data in the data buffer area of each channel. If the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, then read the data of the preset time length in the data buffer area of the target channel.
  • the target channel is any one of all channels, and finally if the free data block is written When it is full, the in-block index table in the cache is written to the end of the free data block, and the steps of finding the free data block according to the data block index table are continued.
  • the data in the data buffer area of each channel is read cyclically, and it is determined whether the time length of the data in the data buffer area of each channel is greater than or equal to the preset time length. If so, the preset time is set.
  • the length of data is written into the free data block, which can combine multi-channel data into one channel of data for write operations, reducing the write position and improving the disk write speed; it also uses the data block index table and the in-block index table in the data block Build a two-level index to speed up retrieval and reading.
  • FIG. 1 is a schematic flowchart of a data storage method provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of a data block index table provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of an intra-block index table in a data block provided by an embodiment of the present application.
  • FIG. 6 is a schematic block diagram of a data storage system according to an embodiment of the present application.
  • FIG. 7 is a schematic block diagram of a terminal device according to an embodiment of the present application.
  • FIG. 1 is a schematic diagram of an implementation process of a data storage method provided by an embodiment of the present application. For ease of description, only parts related to the embodiment of the present application are shown.
  • the execution subject in this embodiment of the present application may be a terminal device. As shown in Figure 1, the method may include the following steps:
  • Step S101 Read a data block index table, find a free data block according to the data block index table, and update a data block index node corresponding to the idle data block.
  • the data block index table includes a plurality of data block index nodes.
  • the storage space of the SMR disk is divided into a plurality of data blocks according to a preset size, and each data block may be numbered, and the numbering may start from 0 and increase sequentially.
  • the preset size can be 256 megabytes (MByte, MB).
  • MB megabytes
  • a 4T hard disk can be divided into 256MB to obtain 16,384 data blocks.
  • the 16384 data blocks are numbered in sequence, with numbers ranging from 0 to 16383.
  • owner_stamp is a file ownership stamp, which can also be referred to as a file identifier.
  • Different files have different file ownership stamps. You can use the file ownership stamp to find out which data blocks are stored in the same file.
  • the files in the embodiments of the present application are files with time attributes, such as video files, audio files, and so on. Because the monitoring channels are multiple channels, all channels will generate data at the same time. A file is made up of data from all channels that are continuous in time.
  • inner_num is the internal number of the file, which is used to indicate the order in which the files are stored.
  • the internal number of the same file can be a continuous natural number starting from 1.
  • a file is often stored in multiple data blocks. Assuming a file is stored in data blocks 1, 3, and 5, the internal number of the file corresponding to data block 1 is 3, the internal number of the file corresponding to data block 2 is 2, and the data block The internal number of the file corresponding to 5 is 1. Then the order of storing the file is first stored in data block 5, after data block 5 is full, then it is stored in data block 3. After data block 3 is full, Stored in data block 1.
  • end_time is the end time of the file, indicating the end time of the file in the data block. That is, the data stored in the data block is data in the file from the file start time to the file end time.
  • chn_info is channel information, which indicates which channels of data are stored in the data block. Bits are used to indicate whether data for the corresponding channel is stored. For example, if the 0th bit is 1, it means that the data of channel 0 is stored. If the 0th bit is 0, it means that the data of channel 0 is not stored. The other channels are similar and will not be described again.
  • block_type is a data block type and is used to distinguish the type of the data block. You can use 0 to indicate free data blocks, that is, data blocks that do not store data; use 1 to indicate data blocks that store data block index tables; and use 2 to indicate data blocks that store data.
  • reserve [7] is a reserved field.
  • check_num is a check digit, which is used to determine whether the data is legal.
  • the data block index table is fixedly stored in the data block numbered 0, and only the data block index table is stored in the data block numbered 0. Therefore, the data block numbered 0 corresponds to The data block type is 1, the other data blocks are used to store the data of each channel.
  • Each data block is divided into a data area and an index area, as shown in Figure 2.
  • the data area stores data of each channel
  • the index area stores an index table in the block.
  • the index table in the block is constructed by data of a preset time length of each channel, and the preset time length can be 1 second.
  • the intra-block index table includes multiple intra-block index nodes, and each intra-block index node stores information related to one second of data.
  • the index table in the block can accurately locate the data of a certain channel for a certain second.
  • the index area (the index table in the block) is buffered into memory.
  • the in-block index table in the memory is written into the index area, which is equivalent to the write operation of the data block as a sequential write.
  • the index table in the block records the relevant information of each second of data.
  • the definition of the index node in the block is as follows:
  • chn is the channel number, which indicates that the data corresponding to the index node in the block belongs to the channel number.
  • Hour, minute, and second are hours, minutes, and seconds, respectively, indicating that the data corresponding to the i-nodes in the block are the hours, minutes, and seconds.
  • offset is the data offset, which indicates the offset of the data corresponding to the index node in the block in the data block.
  • length is the data size, which indicates the size of the data corresponding to the inodes in the block.
  • flag is the index flag. If it is 0, it means idle, if it is 1, it means occupied.
  • reserve [3] is a reserved field.
  • the 256KB index area occupies 1/1024 of the 256MB data block, which means that the space utilization rate can reach 99.9%.
  • the data block index table is first read, and the free data block is found according to the data block index table. After finding the free data block, the data block index node corresponding to the free data block is found.
  • the data block type in is updated from 0 to 2, and the file internal number is updated.
  • Step S102 Allocate a cache for the in-block index table in the free data block.
  • Step S103 The data in the data buffer area of each channel is read cyclically.
  • Each channel has a corresponding data buffer area, and the monitoring data of the corresponding channel is stored in the data buffer area.
  • Read the data in the data buffer area of each channel cyclically, and determine whether the time length of the data in the data buffer area of each channel is greater than or equal to the preset time length.
  • the data in the data buffer area of each channel is read cyclically.
  • the data in the data buffer area of channel 0 can be read first, and then read.
  • Data in the data buffer area of channel 1 then read the data in the data buffer area of channel 2, then read the data in the data buffer area of channel 3, and then read the data in the data buffer area of channel 0 again, In this way, the data in the data buffer area of each channel is read cyclically.
  • Step S104 If the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, then read the data of the preset time length in the data buffer area of the target channel and write the data of the preset time length Into the idle data block, and update the data block index node corresponding to the idle data block and the in-block index table in the cache, the target channel is any one of all channels.
  • the preset time length may be 1 second.
  • the time length of the data in the data buffer area of the target channel is greater than or equal to 1 second, then the data in the data buffer area of the target channel with a length of 1 second is read, and the 1 second is The data is written into the data area of the free data block, and the data block index node corresponding to the free data block and the in-block index table in the cache are updated.
  • Updating the intra-block index table in the cache may be to find an idle intra-block index node in the intra-block index table, that is, an intra-block index node with an index identifier of 0, and updating the idle intra-block index node according to the channel number of the target channel.
  • the data offset of the inode is updated according to the data size of the 1 second data of the inode in the idle block, and the index identifier is updated to 1.
  • Step S105 if the free data block is full, write the index table in the block in the cache to the tail of the free data block, and continue to perform the steps of finding the free data block according to the data block index table.
  • the index table in the block in the cache is written to the tail of the free data block, that is, the data of the free data block is written.
  • the index structure is small, it can complete high efficiency with less memory overhead.
  • Data retrieval and reading of specified data by writing a buffer to the in-block index table in the free data block before writing the data, while writing the data, the in-block index table in the cache is updated and the data is full Then, the in-block index table in the cache is written to the free data block, and the sequential writing of the data area and the index area in the data block is realized, reducing the writing position, further High write speed.
  • the method After updating the data block index node corresponding to the free data block and the in-block index table in the cache, the method further includes:
  • the embodiments of the present application can query the file information list that meets the requirements according to the start time, end time and channel number to be queried.
  • the in-block index node whose channel number is the channel number to be read is referred to as the target in-block index node.
  • Step S507 if there is an index node in the target block, use the index node in the target block as a new index node in the positioning block, and continue to determine the position of the data to be read according to the data offset in the index node in the positioning block. A step of.
  • Step S508 if there is no index node in the target block, then in the data block index table, find whether a file attribution stamp is the same as the file attribution corresponding to the positioning data block, and the file internal number is the internal number of the file corresponding to the positioning data block. Increment the target data block inode.
  • the file home stamp is the same as the file home stamp corresponding to the positioning data block, and the internal number of the file is the data block index node plus the internal number of the file corresponding to the positioning data block plus one as the target data block index node.
  • the intra-block index table in the new positioning data block is used as the new intra-block index table.
  • Step S510 traverse the new index table in the positioning block in the memory, determine the new index node in the positioning block where the channel number to be read is located, and continue to perform determination based on the data offset in the index node in the positioning block to be read Steps to get the location of the data.
  • the index node within the block with the same channel number as the channel number to be read is used as the new index node within the positioning block, and the new index node within the positioning block is replaced.
  • the previous index node in the positioning block returns to step S504, and the step of determining the position of the data to be read according to the data offset in the index node in the positioning block is executed cyclically.
  • the embodiment of the present application can read data after a certain time node in a channel, and first locate the data block according to the data block index table, and then accurately locate the data through the index table in the block, which can efficiently locate Required data.
  • the data block index node includes a file start time and a file end time
  • FIG. 6 is a schematic block diagram of a data storage system according to an embodiment of the present application. For convenience of explanation, only a part related to the embodiment of the present application is shown.
  • the data storage system 6 includes:
  • the idle data block search module 61 is configured to read a data block index table, find a free data block according to the data block index table, and update a data block index node corresponding to the idle data block.
  • the data block index table includes multiple data block index nodes;
  • a cache allocation module 62 configured to allocate a cache for an in-block index table in an idle data block
  • the index table writing module 65 is configured to write an in-block index table in the cache to the tail of the free data block if the free data block is full, and continue to perform the steps of finding the free data block according to the data block index table.
  • the idle data block search module 61 is further configured to cache the data block index table into memory;
  • the idle data block search module 61 further includes:
  • the file ownership stamp obtaining unit is used to traverse the data block index table in the memory to obtain the largest file ownership stamp, and determine the file ownership stamp of the file to be stored according to the largest file ownership stamp;
  • a judging module configured to continue to judge whether the time length of the data in the data buffer area of the next channel is greater than or equal to the preset time length if the time length of the data in the data buffer area of the target channel is less than the preset time length;
  • the first loop module is configured to, if the free data block is not full, continue to perform the step of cyclically reading data in the data buffer area of each channel.
  • the data block index node includes a file belonging stamp
  • the information to be queried includes a start time to be queried, an end time to be queried, and a channel number to be queried
  • the file description information includes a file belonging stamp, a file start time, a file end time, and channel information contained in the file
  • the query result acquisition module is specifically configured to determine a list of file information corresponding to the start time, the end time of the query, and the channel number to be queried according to the file start time, the file end time, and the channel information contained in the file.
  • the data storage system 6 further includes:
  • a positioning data block determination module which is used to traverse a data block index table to obtain a positioning data block to which a positioning time belongs;
  • the index table reading module is used to read the index table in the positioning block in the positioning data block, and cache the index table in the positioning block into memory;
  • the index node search module is used to find whether there is a target in-block index node containing the channel number to be read in the in-block index node after the in-block index node in the index table in the positioning block;
  • a first processing module configured to use the index node in the target block as a new index node in the positioning block if the index node in the target block exists, and continue to execute the determination of the to-be-read based on the data offset in the index node in the positioning block; Steps to get the location of the data;
  • the second processing module is configured to find, in the data block index table, if there is no inode in the target block, the file attribution stamp is the same as the file attribution corresponding to the positioning data block, and the internal number of the file corresponds to the positioning data block The internal number of the file plus the target data block index node;
  • a third processing module configured to: if a target data block index node exists, use the data block corresponding to the target data block index node as a new positioning data block, and read a new index table within the positioning block in the new positioning data block; And cache the index table in the new positioning block into memory;
  • the second loop module is used to traverse the new index table in the positioning block in memory, determine the index node in the positioning block where the channel number to be read is located, and continue to execute the determination based on the data offset in the index node in the positioning block. Steps for the location of the data to be read;
  • a fourth processing module is configured to: if the target data block index node does not exist, the data reading ends.
  • the data block index node includes a file start time and a file end time
  • the positioning data block determination module is specifically used to traverse the data block index table and obtain the positioning data block to which the positioning time belongs according to the file start time and the file end time.
  • FIG. 7 is a schematic block diagram of a terminal device according to an embodiment of the present application.
  • the terminal device 7 of this embodiment includes one or more processors 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70.
  • the processor 70 executes the computer program 72
  • the steps in the foregoing data storage method embodiments are implemented, for example, steps S101 to S105 shown in FIG.
  • the processor 70 executes the computer program 72
  • the functions of the modules / units in the foregoing embodiment of the data storage system are implemented, for example, the functions of modules 61 to 65 shown in FIG. 6.
  • a cache allocation module configured to allocate a cache for an in-block index table in an idle data block
  • a data writing module is configured to read data of a preset time length in the data buffer area of the target channel if the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, and set the preset time The length of data is written into the free data block, and the data block index node corresponding to the free data block and the in-block index table in the cache are updated, and the target channel is any one of all channels;
  • the terminal device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server.
  • the terminal device 7 includes, but is not limited to, a processor 70 and a memory 71.
  • FIG. 7 is only an example of the terminal device, and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or combine some components or different components.
  • the terminal device 7 may further include an input device, an output device, a network access device, a bus, and the like.
  • the processor 70 may be a central processing unit (Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), and application-specific integrated circuits (Applications) Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • CPU Central Processing Unit
  • DSP digital signal processor
  • ASIC Application-specific integrated circuits
  • ASIC Applications Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the memory 71 may be an internal storage unit of the terminal device, such as a hard disk or a memory of the terminal device.
  • the memory 71 may also be an external storage device of the terminal device, for example, a plug-in hard disk, a smart memory card (Smart Media) Card, SMC), Secure Digital (SD) card, Flash Card, etc. Further, the memory 71 may further include both an internal storage unit of the terminal device and an external storage device.
  • the memory 71 is configured to store the computer program 72 and other programs and data required by the terminal device.
  • the memory 71 may also be used to temporarily store data that has been output or is to be output.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.
  • the integrated module / unit When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this application implements all or part of the processes in the method of the above embodiment, and can also be completed by a computer program instructing related hardware.
  • the computer program can be stored in a computer-readable storage medium.
  • the computer When the program is executed by a processor, the steps of the foregoing method embodiments can be implemented.
  • the computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file, or some intermediate form.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

La présente invention se rapporte au domaine technique des ordinateurs et concerne un procédé et un système de stockage de données et un dispositif terminal. Le procédé comprend les étapes consistant : à lire un tableau d'indices de blocs de données, à rechercher un bloc de données inactives selon le tableau d'indices de blocs de données et à mettre à jour un nœud d'indices de blocs de données correspondant au bloc de données inactives ; à attribuer un tampon à un tableau d'indices de blocs internes dans le bloc de données inactives ; à lire de manière cyclique les données dans un tampon de données de chaque canal ; si une durée de données dans un tampon de données d'un canal cible est supérieure ou égale à une durée prédéfinie, à lire les données de la durée prédéfinie dans le tampon de données du canal cible, à écrire les données de la durée prédéfinie dans le bloc de données inactives et à mettre à jour le nœud d'indices de blocs de données correspondant au bloc de données inactives et au tableau d'indices de blocs internes de données dans le tampon ; et si le bloc de données inactives est plein, à écrire le tableau d'indices de blocs internes dans le tampon jusqu'à une extrémité du bloc de données inactives et à poursuivre l'exécution de l'étape de recherche d'un bloc de données inactives selon le tableau d'indices de blocs de données.La présente invention réduit le nombre de positions d'écriture et améliore la vitesse d'écriture d'un disque.
PCT/CN2018/102490 2018-08-27 2018-08-27 Procédé et système de stockage de données et dispositif terminal Ceased WO2020041928A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201880001157.2A CN110149803B (zh) 2018-08-27 2018-08-27 数据存储方法、系统及终端设备
PCT/CN2018/102490 WO2020041928A1 (fr) 2018-08-27 2018-08-27 Procédé et système de stockage de données et dispositif terminal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/102490 WO2020041928A1 (fr) 2018-08-27 2018-08-27 Procédé et système de stockage de données et dispositif terminal

Publications (1)

Publication Number Publication Date
WO2020041928A1 true WO2020041928A1 (fr) 2020-03-05

Family

ID=67588283

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/102490 Ceased WO2020041928A1 (fr) 2018-08-27 2018-08-27 Procédé et système de stockage de données et dispositif terminal

Country Status (2)

Country Link
CN (1) CN110149803B (fr)
WO (1) WO2020041928A1 (fr)

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111651407A (zh) * 2020-04-16 2020-09-11 广州南方卫星导航仪器有限公司 原始观测数据记录方法、电子设备、存储介质及系统
CN111723056A (zh) * 2020-06-09 2020-09-29 北京青云科技股份有限公司 小文件的处理方法、装置、设备和存储介质
CN111782656A (zh) * 2020-06-30 2020-10-16 北京海益同展信息科技有限公司 数据读写方法及装置
CN111859033A (zh) * 2020-07-22 2020-10-30 北京金山云网络技术有限公司 Ip库查询方法、装置及ip库压缩方法、装置
CN113064860A (zh) * 2020-12-08 2021-07-02 上海金融期货信息技术有限公司 一种动态数据缓存系统和方法
CN113076458A (zh) * 2021-04-13 2021-07-06 鹏城实验室 一种网内缓存控制方法、系统、存储介质及终端设备
CN113220693A (zh) * 2021-06-02 2021-08-06 北京字节跳动网络技术有限公司 计算存储分离系统及其数据访问方法、介质和电子设备
CN113348450A (zh) * 2020-06-24 2021-09-03 智协慧同(北京)科技有限公司 一种车载数据存储方法及系统
CN114020791A (zh) * 2021-11-10 2022-02-08 西安热工研究院有限公司 一种提高趋势数据查询效率的池化结构及方法
CN114327942A (zh) * 2021-12-24 2022-04-12 凌云光技术股份有限公司 一种共享内存管理方法和缓存服务组件
CN114371810A (zh) * 2020-10-15 2022-04-19 中国移动通信集团设计院有限公司 Hdfs的数据存储方法及装置
CN114519129A (zh) * 2022-02-11 2022-05-20 北京易源兴华软件有限公司 地震大数据集群并行机高效分选方法和装置
CN114816322A (zh) * 2022-04-29 2022-07-29 中国人民解放军国防科技大学 Ssd的外部排序方法、装置和ssd存储器
CN114816258A (zh) * 2022-04-29 2022-07-29 中国人民解放军国防科技大学 Nvm的外部排序方法、装置和nvm存储器
CN114896268A (zh) * 2022-05-06 2022-08-12 山东浪潮科学研究院有限公司 一种时序数据存储和查询方法
CN115617746A (zh) * 2022-10-21 2023-01-17 南京中孚信息技术有限公司 文件传输方法、电子设备及存储介质
CN115905219A (zh) * 2022-11-04 2023-04-04 上海威固信息技术股份有限公司 多通道数据存云库用的数据存储驱动系统及其传输方法
CN116578589A (zh) * 2023-02-08 2023-08-11 成都锋卫科技有限公司 一种基于连续内存的高速数据查询方法及系统
CN117033388A (zh) * 2023-08-23 2023-11-10 上海英方软件股份有限公司 一种oracle数据库实时复制环境下的rowid映射数据的存储方法
CN117130986A (zh) * 2023-09-08 2023-11-28 中国银行股份有限公司 一种业务数据的管理方法、系统及电子设备
CN117931098A (zh) * 2024-03-22 2024-04-26 江西云眼视界科技股份有限公司 边缘侧事件存储方法、系统、存储介质及计算机
CN118069074A (zh) * 2024-04-22 2024-05-24 联想凌拓科技有限公司 一种数据处理方法及装置、存储介质、计算机程序产品
WO2026081672A1 (fr) * 2024-10-15 2026-04-23 腾讯科技（深圳）有限公司 Procédé de traitement de données de mémoire cache, appareil associé et support

Families Citing this family (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111399762B (zh) 2019-11-27 2021-06-18 杭州海康威视系统技术有限公司 数据存储方法、装置及存储系统
CN111241090B (zh) * 2019-12-23 2023-11-10 华为技术有限公司 存储系统中管理数据索引的方法和装置
CN111025068B (zh) * 2019-12-24 2022-05-20 宁波三星医疗电气股份有限公司 电表负荷曲线数据抄读方法、装置和电子设备
CN111124313A (zh) * 2019-12-24 2020-05-08 宁波三星医疗电气股份有限公司 电力采集终端数据读写方法、装置和电子设备
CN111190869A (zh) * 2019-12-27 2020-05-22 深圳市恒扬数据股份有限公司 文件存储方法及终端
CN113126882B (zh) * 2019-12-30 2026-03-17 深圳市中兴微电子技术有限公司 数据处理方法及装置、存储介质、电子装置
CN111274295B (zh) * 2020-01-12 2022-07-08 苏州浪潮智能科技有限公司 一种数据库快速加载数据的方法、装置、设备和介质
CN111309678B (zh) * 2020-02-22 2023-01-03 呼和浩特市奥祥电力自动化有限公司 一种数据循环存储方法及网络报文记录分析装置
CN111522810A (zh) * 2020-03-12 2020-08-11 视联动力信息技术股份有限公司 一种索引获取方法和装置
CN111651127B (zh) * 2020-06-10 2023-05-02 杭州海康威视数字技术股份有限公司 一种基于叠瓦式磁记录盘的监控数据存储方法及装置
CN114201647A (zh) * 2020-09-18 2022-03-18 上海航空电器有限公司 基于fpga的afdx接收数据内存索引方法
CN115221118A (zh) * 2021-04-14 2022-10-21 北京三快在线科技有限公司 查找无人车的数据文件的方法、存储介质以及电子设备
CN113704190B (zh) * 2021-07-19 2025-06-03 新华三技术有限公司成都分公司 一种数据写入方法以及装置
CN113568868B (zh) * 2021-07-28 2024-02-06 重庆紫光华山智安科技有限公司 文件系统管理方法、系统、电子设备及介质
CN114116795B (zh) * 2021-11-01 2022-12-20 北京力控元通科技有限公司 一种数据存储和查询方法、装置、存储介质及电子设备
CN114153394A (zh) * 2021-11-30 2022-03-08 浙江大华技术股份有限公司 一种数据存储管理方法、装置及设备
CN114385075A (zh) * 2021-12-01 2022-04-22 浙江大华技术股份有限公司 特征值的存储方法、查询方法、电子设备和存储介质
CN114816277B (zh) * 2022-06-30 2022-11-11 广东睿江云计算股份有限公司 文件数据块顺序性保障的控制方法及控制系统
CN115312084B (zh) * 2022-07-29 2025-04-25 深圳市锐明技术股份有限公司 磁盘存储系统、数据读写方法、恢复方法、设备和介质
CN115359825B (zh) * 2022-08-06 2026-02-10 惠尔丰信息系统有限公司 基于nand flash的文件数据存取方法、装置、设备及存储介质
CN116112456B (zh) * 2023-02-01 2024-02-13 镁佳(北京)科技有限公司 一种基于bap协议的数据缓存方法、装置、设备及介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101316334A (zh) * 2008-07-07 2008-12-03 中兴通讯股份有限公司 一种处理电视媒体数据的方法及电视手机
US20100174846A1 (en) * 2009-01-05 2010-07-08 Alexander Paley Nonvolatile Memory With Write Cache Having Flush/Eviction Methods
CN102332030A (zh) * 2011-10-17 2012-01-25 中国科学院计算技术研究所 用于分布式键-值存储系统的数据存储、管理和查询方法及系统
CN104393962A (zh) * 2014-12-12 2015-03-04 成都朗锐芯科技发展有限公司 一种多路e1解帧系统

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6252789B1 (en) * 2000-06-14 2001-06-26 Netlogic Microsystems, Inc. Inter-row configurability of content addressable memory
US8060481B1 (en) * 2005-06-30 2011-11-15 Symantec Operating Corporation Time indexed file system
WO2011020360A1 (fr) * 2009-08-19 2011-02-24 北京书生国际信息技术有限公司 Procédé de mémorisation de document
KR101954995B1 (ko) * 2011-04-27 2019-05-31 시게이트 테크놀로지 엘엘씨 저장 매체에 라이트 된 데이터 머지 방법, 저장 매체에 대한 라이트 동작 제어 방법, 이를 적용한 저장 장치, 컴퓨터 시스템 및 저장 매체
US8706971B1 (en) * 2012-03-14 2014-04-22 Netapp, Inc. Caching and deduplication of data blocks in cache memory
CN103310008B (zh) * 2013-06-28 2018-06-01 安科智慧城市技术（中国）有限公司 一种云控制服务器及其文件索引方法
CN106302604B (zh) * 2015-06-05 2020-09-11 腾讯科技（深圳）有限公司 数据传输方法及装置
CN107391023B (zh) * 2016-05-16 2021-01-22 深圳市中兴微电子技术有限公司 一种多通道数据存储器及其存取方法及装置
CN108255408B (zh) * 2016-12-28 2021-03-23 中国电信股份有限公司 数据存储方法以及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101316334A (zh) * 2008-07-07 2008-12-03 中兴通讯股份有限公司 一种处理电视媒体数据的方法及电视手机
US20100174846A1 (en) * 2009-01-05 2010-07-08 Alexander Paley Nonvolatile Memory With Write Cache Having Flush/Eviction Methods
CN102332030A (zh) * 2011-10-17 2012-01-25 中国科学院计算技术研究所 用于分布式键-值存储系统的数据存储、管理和查询方法及系统
CN104393962A (zh) * 2014-12-12 2015-03-04 成都朗锐芯科技发展有限公司 一种多路e1解帧系统

Cited By (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111651407A (zh) * 2020-04-16 2020-09-11 广州南方卫星导航仪器有限公司 原始观测数据记录方法、电子设备、存储介质及系统
CN111723056B (zh) * 2020-06-09 2024-04-30 北京青云科技股份有限公司 小文件的处理方法、装置、设备和存储介质
CN111723056A (zh) * 2020-06-09 2020-09-29 北京青云科技股份有限公司 小文件的处理方法、装置、设备和存储介质
CN113348450A (zh) * 2020-06-24 2021-09-03 智协慧同(北京)科技有限公司 一种车载数据存储方法及系统
CN111782656B (zh) * 2020-06-30 2024-04-12 京东科技信息技术有限公司 数据读写方法及装置
US12189992B2 (en) 2020-06-30 2025-01-07 Jingdong Technology Information Technology Co., Ltd. Method and apparatus for reading and writing data
CN111782656A (zh) * 2020-06-30 2020-10-16 北京海益同展信息科技有限公司 数据读写方法及装置
CN111859033A (zh) * 2020-07-22 2020-10-30 北京金山云网络技术有限公司 Ip库查询方法、装置及ip库压缩方法、装置
CN111859033B (zh) * 2020-07-22 2023-10-27 北京金山云网络技术有限公司 Ip库查询方法、装置及ip库压缩方法、装置
CN114371810A (zh) * 2020-10-15 2022-04-19 中国移动通信集团设计院有限公司 Hdfs的数据存储方法及装置
CN114371810B (zh) * 2020-10-15 2023-10-27 中国移动通信集团设计院有限公司 Hdfs的数据存储方法及装置
CN113064860A (zh) * 2020-12-08 2021-07-02 上海金融期货信息技术有限公司 一种动态数据缓存系统和方法
CN113076458A (zh) * 2021-04-13 2021-07-06 鹏城实验室 一种网内缓存控制方法、系统、存储介质及终端设备
CN113076458B (zh) * 2021-04-13 2024-02-06 鹏城实验室 一种网内缓存控制方法、系统、存储介质及终端设备
US12346269B2 (en) 2021-06-02 2025-07-01 Beijing Bytedance Network Technology Co., Ltd. Computing storage separation system and data access method therefor, medium, and electronic device
CN113220693A (zh) * 2021-06-02 2021-08-06 北京字节跳动网络技术有限公司 计算存储分离系统及其数据访问方法、介质和电子设备
CN113220693B (zh) * 2021-06-02 2023-10-20 北京火山引擎科技有限公司 计算存储分离系统及其数据访问方法、介质和电子设备
CN114020791A (zh) * 2021-11-10 2022-02-08 西安热工研究院有限公司 一种提高趋势数据查询效率的池化结构及方法
CN114327942A (zh) * 2021-12-24 2022-04-12 凌云光技术股份有限公司 一种共享内存管理方法和缓存服务组件
CN114519129A (zh) * 2022-02-11 2022-05-20 北京易源兴华软件有限公司 地震大数据集群并行机高效分选方法和装置
CN114816258A (zh) * 2022-04-29 2022-07-29 中国人民解放军国防科技大学 Nvm的外部排序方法、装置和nvm存储器
CN114816322A (zh) * 2022-04-29 2022-07-29 中国人民解放军国防科技大学 Ssd的外部排序方法、装置和ssd存储器
CN114896268A (zh) * 2022-05-06 2022-08-12 山东浪潮科学研究院有限公司 一种时序数据存储和查询方法
CN115617746A (zh) * 2022-10-21 2023-01-17 南京中孚信息技术有限公司 文件传输方法、电子设备及存储介质
CN115905219B (zh) * 2022-11-04 2023-09-19 上海威固信息技术股份有限公司 多通道数据存云库用的数据存储驱动系统及其传输方法
CN115905219A (zh) * 2022-11-04 2023-04-04 上海威固信息技术股份有限公司 多通道数据存云库用的数据存储驱动系统及其传输方法
CN116578589A (zh) * 2023-02-08 2023-08-11 成都锋卫科技有限公司 一种基于连续内存的高速数据查询方法及系统
CN117033388A (zh) * 2023-08-23 2023-11-10 上海英方软件股份有限公司 一种oracle数据库实时复制环境下的rowid映射数据的存储方法
CN117130986A (zh) * 2023-09-08 2023-11-28 中国银行股份有限公司 一种业务数据的管理方法、系统及电子设备
CN117931098A (zh) * 2024-03-22 2024-04-26 江西云眼视界科技股份有限公司 边缘侧事件存储方法、系统、存储介质及计算机
CN117931098B (zh) * 2024-03-22 2024-06-07 江西云眼视界科技股份有限公司 边缘侧事件存储方法、系统、存储介质及计算机
CN118069074A (zh) * 2024-04-22 2024-05-24 联想凌拓科技有限公司 一种数据处理方法及装置、存储介质、计算机程序产品
WO2026081672A1 (fr) * 2024-10-15 2026-04-23 腾讯科技（深圳）有限公司 Procédé de traitement de données de mémoire cache, appareil associé et support

Also Published As

Publication number Publication date
CN110149803B (zh) 2023-06-09
CN110149803A (zh) 2019-08-20

Similar Documents

Publication Publication Date Title
CN110149803B (zh) 数据存储方法、系统及终端设备
CN104346357B (zh) 一种嵌入式终端的文件存取方法及系统
US9047330B2 (en) Index compression in databases
CN110555001B (zh) 数据处理方法、装置、终端及介质
CN110879800B (zh) 数据写入、压缩和读取方法、数据处理方法及装置
WO2019062574A1 (fr) Procédé et dispositif d'interrogation de métadonnées
CN116257523B (zh) 一种基于非易失存储器的列式存储索引方法及装置
CN104462141A (zh) 一种数据存储与查询的方法、系统及存储引擎装置
CN106156070B (zh) 一种查询方法、文件合并方法与相关装置
CN114490737B (zh) 一种提高数据库深度分页查询效率的方法与终端
CN106469120A (zh) 碎片整理方法、装置及设备
CN110147203B (zh) 一种文件管理方法、装置、电子设备及存储介质
CN108763473A (zh) 一种分布式存储的本地对象存储方法及装置
CN104424219A (zh) 一种数据文件的管理方法及装置
CN115994148B (zh) 多表数据的更新方法、装置、电子设备及可读存储介质
WO2013075306A1 (fr) Procédé et dispositif d'accès aux données
CN109240607B (zh) 一种文件读取方法和装置
CN115878027A (zh) 一种存储对象的处理方法、装置、终端及存储介质
CN107391544A (zh) 列式存储数据的处理方法、装置、设备及计算机储存介质
CN104750432B (zh) 一种数据存储方法及装置
US12373337B2 (en) Method and system for in-memory metadata reduction in cloud storage system
CN110020001A (zh) 字符串数据的存储、查询方法以及相应的设备
CN108804571B (zh) 一种数据存储方法、装置以及设备
CN113535092B (zh) 用于减少内存元数据的存储引擎、方法和可读介质
CN112380174B (zh) 含删除文件的xfs文件系统解析方法、终端设备及存储介质

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18932250

Country of ref document: EP

Kind code of ref document: A1