基于民航数据特性的重删固定长度分块方法

taptap下载安装安卓学报 ›› taptap点点手机网页 , taptap点点手机网页 ›› Issue (4): 32-37.

基于民航数据特性的重删固定长度分块方法

丁建立，李慧，曹卫东

（taptap下载安装安卓计算机科学与技术学院，天津300300）

收稿日期:2021-04-13 修回日期:2021-05-13 出版日期:2022-08-15 发布日期:2023-10-28
作者简介:丁建立（1963—），男，河南洛阳人，教授，博士，研究方向为智能仿生算法、智能信息处理及民航应用.
基金资助:
国家自然科学基金项目（U1833114）；民航安全能力建设资金项目（SA2020280）

Deduplication fixed-length block method based on characteristics of civil aviation data

DING Jianli, LI Hui, CAO Weidong

（College of Computer Science and Technology, CAUC, Tianjin 300300, China）

Received:2021-04-13 Revised:2021-05-13 Online:2022-08-15 Published:2023-10-28

摘要/Abstract

摘要： 针对民航数据在容灾备份时存在存储数据重复的问题，提出一种基于民航数据特性的重删固定长度分块方法。该方法根据民航数据类型的一致性，结合固定长度分块与可变长度分块的优势，设计了一种分块策略索引表的数据结构，为同种类型的数据提供分块策略，节省了分块时寻找数据块边界的时间，将备份时重复数据的模拟重删率提高到97.8%~99.3%，比固定长度分块方法高11.8%~12.5%，比可变长度分块方法高2.5%~3.0%；同时，为新的数据类型建立新的分块策略，便于后续数据流匹配，提高命中精度。

关键词: 民航数据, 容灾备份, 重复数据删除, 类型一致性, 分块策略, 模拟重删率

Abstract: Aiming at the problem of data duplication in the storage of civil aviation data during disaster recovery and back-up. A deduplication fixed-length block method based on the characteristics of civil aviation data is proposed. According to the consistency of civil aviation data types, this method combines the advantages of fixed-length block and variable-length block, and designs a data structure of the block strategy index table. A block strategy is provided for the same type of data, and time is saved to find the boundary of the data block during block division, the simulated data deduplication rate during backup increases to 97.8%~99.3%, which is 11.8%~12.5% higher than that of the fixed-length block method, and 2.5%~3.0% higher than that of the variable length block method; at the same time, a new block strategy is established for new data types to facilitate subsequent data stream matching and improve hit accuracy.

Key words: civil aviation data, disaster recovery backup, data deduplication, type consistency, block strategy, simulated deduplication rate

中图分类号:

丁建立, 李慧, 曹卫东. 基于民航数据特性的重删固定长度分块方法[J]. taptap下载安装安卓学报, 2022, 40(4): 32-37.

DING Jianli, LI Hui, CAO Weidong. Deduplication fixed-length block method based on characteristics of civil aviation data[J]. Journal of Civil Aviation University of China, 2022, 40(4): 32-37.

参考文献 17

[1]	王龙翔, 董凯, 王鹏博, 等. R-dedup: 一种重复数据删除指纹计算的优化方法[J]. 西安交通大学学报, 2021, 55(1): 43-51.
[2]	何柯文, 张佳辰, 刘晓光, 等. 新型存储设备上重复数据删除指纹查找优化[J]. 计算机研究与发展, 2020, 57(2): 269-280.
[3]	LILLIBRIDGE M, ESHGHI K, BHAGWAT D, et al Sparse indexing: large scale , inline deduplication using sampling and locality [C]//7th USENIX Conference on File and Storage Technologies, San Francisco , CA, USA, February 24-27, 2009: 111-123.
[4]	DEBNATH B, SENGUPTA S, LI J. ChunkStash: speeding up inline storage deduplication using flash memory[C]//2010 USENIX Annual Technical Conference,June 23-25, 2010, Boston, MA, USA. CA: USENIX Assocation, 2010.
[5]	CAO Z, WEN H, WU F, et al ALACC: accelerating restore performance of data deduplication systems using adaptive look-ahead window assist－ ed chunk caching[C]//16th USENIX Conference on File and Storage Technologies (FAST′18), February 12-15, 2018, Oakland, CA, USA. CA: USENIX Assocation, 2018: 309-324.
[6]	廖海生. 基于重复数据删除技术的数据容灾系统的研究[D]. 广州: 华南理工大学, 2009.
[7]	张甲燃. 重复数据删除技术研究[D]. 济南: 山东大学, 2013.
[8]	韩英杰. 数据容灾系统重复数据删除关键技术研究 [D]. 长春: 长春理工大学, 2018.
[9]	阎芳. 重复数据删除系统数据组织研究[D]. 北京: 北京理工大学, 2015.
[10]	WANG G H, ZHAO Y L, LI T X, et al A method for organizing metada－ ta of storage nodes with data de-duplication[J]. Journal of Communica－ tional Information Systems, 2014, 10(9): 3845-3854.
[11]	贺秦禄, 边根庆, 邵必林, 等. 移动闪存的重复数据删除技术[J]. 西安电子科技大学学报, 2020, 47(1): 128-134.
[12]	XIA W, JIANG H, FENG D, et al Silo: a similarity -locality based near -exact deduplication scheme with low ram overhead and high throughput[C]//2011 USENIX Annual Technical Conference, June 15, 2011, Portland, OR , USA. CA: USENIX Assocation, 2011: 26-28.
[13]	DANG Q L, XIE Y, LI D H, et al Updatable block-level deduplication of encrypted data with efficient auditing in cloud storage[J]. The Journal of China Universities of Posts and Telecommunications, 2019(3): 56-72.
[14]	BHAGWAT D, ESHGHI K, LONG D D E, et al Extreme Binning: scal－ able, parallel deduplication for chunk-based file backup[C]//2009 IEEE International Symposium on Modeling, Analysis & Simulation of Com－ puter and Telecommunication Systems, September 21-23, 2009, Lon－ don, UK. IEEE, 2009.
[15]	WEN X, ZHOU Y K, JIANG H, et al FastCDC: a fast and efficient con－ tent-defined chunking approach for data deduplication[C]//2016 USENIX Annual Technical Conferenc, June 22-24, 2016, Denver, CO, USA. CA: USENIX Assocation, 2016.
[16]	DENG M Z, CHEN W, XIAO N, et al GLE-dedup: a globally-locally even deduplication by request-aware placement for better read perfor－ mance[J]. International Journal of Parallel Programming, 2017, 45(4): 946-964.
[17]	WU F H, CHEN J, DONG Y, et al A holistic energy-efficient approach for a processor -memory system [J]. Tsinghua Science and Technology, 2019, 24(4): 468-483.