Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning - Network and Parallel Computing
Conference Papers Year : 2019

Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning

Abstract

As storage hierarchies are getting deeper on modern high-performance computing systems, intelligent data placement strategies that can choose the optimal storage tier dynamically is the key to realize the potential of hierarchical storage architecture. However, providing a general solution that can be applied in different storage architectures and diverse applications is challenging. In this paper, we propose adaptive storage learner (ASL), which explores the idea of using machine learning techniques to mine the relationship between data placement strategies and I/O performance under varied workflow characteristics and system status, and uses the learned model to choose the optimal storage tier intelligently. We implement a prototype and integrate it into an existing data management system. Empirical comparison based on real scientific workflows tests shows that ASL is capable of combining workflow characteristics and real-time system status to make optimal data placement decisions.
Fichier principal
Vignette du fichier
486810_1_En_23_Chapter.pdf (340.5 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03770550 , version 1 (06-09-2022)

Licence

Identifiers

Cite

Peng Cheng, Yutong Lu, Yunfei Du, Zhiguang Chen, Yang Liu. Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning. 16th IFIP International Conference on Network and Parallel Computing (NPC), Aug 2019, Hohhot, China. pp.289-302, ⟨10.1007/978-3-030-30709-7_23⟩. ⟨hal-03770550⟩
47 View
175 Download

Altmetric

Share

More