Optimization of RDMA-Based HDFS Data Distribution Mechanism - Network and Parallel Computing Access content directly
Conference Papers Year : 2021

Optimization of RDMA-Based HDFS Data Distribution Mechanism

Abstract

Hadoop Distributed File System (short for HDFS) is a high availability file system designed to run on commodity hardware. It uses multiple replicas to ensure high reliability, and many data are transmitted between storage nodes. The performance of data transmission has a great impact on the latency of writing operations. Remote Direct Memory Access (short for RDMA) is a protocol with low latency and high through which is running on the Infiniband network. When HDFS runs on the Infiniband network, the default protocol IPoIB can not take advantage of the high-speed network. The latency of the writing process is similar to a TCP/IP network. In this paper, we present a new RDMA-based HDFS writing mechanism. It enables DataNodes to read data parallelly from the Client through RDMA. And by using RDMA primitive, the transmission latency is slower than the original TCP/IP protocol. The experiments show that our approach reduces the latency of the writing process by 10.11%–40.81% compared with the original HDFS.
Fichier principal
Vignette du fichier
511910_1_En_22_Chapter.pdf (2.1 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03768747 , version 1 (04-09-2022)

Licence

Attribution

Identifiers

Cite

Xiao Zhang, Binbin Liu, Junhao Zhao, Cong Dong. Optimization of RDMA-Based HDFS Data Distribution Mechanism. 17th IFIP International Conference on Network and Parallel Computing (NPC), Sep 2020, Zhengzhou, China. pp.251-262, ⟨10.1007/978-3-030-79478-1_22⟩. ⟨hal-03768747⟩
18 View
6 Download

Altmetric

Share

Gmail Facebook X LinkedIn More