ODCP: Optimizing Data Caching and Placement in Distributed File System Using Erasure Coding - Network and Parallel Computing
Conference Papers Year : 2021

ODCP: Optimizing Data Caching and Placement in Distributed File System Using Erasure Coding

Abstract

Many current distributed file systems use erasure-coding based data redundancy techniques to improve the reliability of data storage. Such techniques can significantly improve the effective storage utilization. However, there are several drawbacks to the above techniques. Firstly, they introduce non-negligible computation overhead for decoding. Secondly, traditional data caching and placement strategies become less effective in such cases. To solve the above drawbacks, this paper proposes a new data cache allocation mechanism based on simulated annealing and a new data placement strategy based on convex optimization, which effectively reduces data block transmission delay and decoding delay. We have implemented the proposed data placement strategy in the real-world distributed file system Alluxio, and evaluated the performance of our strategy. Experiment results show that our strategy can significantly reduce the file read delay compared to traditional data placement strategies.
Fichier principal
Vignette du fichier
511910_1_En_38_Chapter.pdf (639.02 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03768742 , version 1 (04-09-2022)

Licence

Identifiers

Cite

Shuhan Wu, Yunchun Li, Hailong Yang, Zerong Luan, Wei Li. ODCP: Optimizing Data Caching and Placement in Distributed File System Using Erasure Coding. 17th IFIP International Conference on Network and Parallel Computing (NPC), Sep 2020, Zhengzhou, China. pp.452-464, ⟨10.1007/978-3-030-79478-1_38⟩. ⟨hal-03768742⟩
31 View
29 Download

Altmetric

Share

More