DDP-B: A Distributed Dynamic Parallel Framework for Meta-genomics Binary Similarity - Network and Parallel Computing Access content directly
Conference Papers Year : 2019

DDP-B: A Distributed Dynamic Parallel Framework for Meta-genomics Binary Similarity

Abstract

Great efforts have been made on meta-genomics in the field of new species exploration in the past decades. With the development of next-generation sequencing technology, meta-genomics datasets have been produced as large as dozens of hundreds of gigabytes or even several terabytes, which brings a severe challenge to data analysis. Besides, conventional meta-genomics comparing algorithms may not take full advantage of powerful computing capacity from parallel computing techniques due to lack of parallelism. In this paper, we propose DDP-B, a distributed dynamic parallel framework for meta-genomics binary similarity analysis, to overcome these limitations. In this framework, we introduce a binary distance algorithm for meta-genomics similarity measurement and develop different levels of parallel granularity of the algorithm utilizing MPI, OpenMP, and SIMD techniques. Moreover, we establish a dynamic scheduling method to deliver asynchronous parallel computing tasks and design a distributed cluster to deploy the dynamic parallel system, which completes 2.97K pairs of meta-genomics vectors comparison per second and achieves an 134.79x speedup versus the baseline in the optimal condition. Our framework shows stable scalability when assigned larger workloads.
Fichier principal
Vignette du fichier
486810_1_En_12_Chapter.pdf (687.47 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03770527 , version 1 (06-09-2022)

Licence

Attribution

Identifiers

Cite

Mengxian Chi, Xu Jin, Feng Li, Hong An. DDP-B: A Distributed Dynamic Parallel Framework for Meta-genomics Binary Similarity. 16th IFIP International Conference on Network and Parallel Computing (NPC), Aug 2019, Hohhot, China. pp.143-155, ⟨10.1007/978-3-030-30709-7_12⟩. ⟨hal-03770527⟩
10 View
21 Download

Altmetric

Share

Gmail Facebook X LinkedIn More