M-DRL: Deep Reinforcement Learning Based Coflow Traffic Scheduler with MLFQ Threshold Adaption - Network and Parallel Computing Access content directly
Conference Papers Year : 2021

M-DRL: Deep Reinforcement Learning Based Coflow Traffic Scheduler with MLFQ Threshold Adaption

Abstract

The coflow scheduling in data-parallel clusters can improve application-level communication performance. The existing coflow scheduling method without prior knowledge usually uses Multi-Level Feedback Queue (MLFQ) with fixed threshold parameters, which is insensitive to coflow traffic characteristics. Manual adjustment of the threshold parameters for different application scenarios often has long optimization period and is coarse in optimization granularity. We propose M-DRL, a deep reinforcement learning based coflow traffic scheduler by dynamically setting thresholds of MLFQ to adapt to the coflow traffic characteristics, and reduces the average coflow completion time. Trace-driven simulations on the public dataset show that coflow communication stages using M-DRL complete 2.08$$\times $$×(6.48$$\times $$×) and 1.36$$\times $$×(1.25$$\times $$×) faster on average coflow completion time (95-th percentile) in comparison to per-flow fairness and Aalo, and is comparable to SEBF with prior knowledge.
Fichier principal
Vignette du fichier
511910_1_En_7_Chapter.pdf (525.38 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03768743 , version 1 (04-09-2022)

Licence

Attribution

Identifiers

Cite

Tianba Chen, Wei Li, Yukang Sun, Yunchun Li. M-DRL: Deep Reinforcement Learning Based Coflow Traffic Scheduler with MLFQ Threshold Adaption. 17th IFIP International Conference on Network and Parallel Computing (NPC), Sep 2020, Zhengzhou, China. pp.80-91, ⟨10.1007/978-3-030-79478-1_7⟩. ⟨hal-03768743⟩
15 View
7 Download

Altmetric

Share

Gmail Facebook X LinkedIn More