FLEX: A Slot Allocation Scheduling Optimizer for MapReduce Workloads - Middleware 2010 Access content directly
Conference Papers Year : 2010

FLEX: A Slot Allocation Scheduling Optimizer for MapReduce Workloads

Abstract

Originally, MapReduce implementations such as Hadoop employed First In First Out (fifo) scheduling, but such simple schemes cause job starvation. The Hadoop Fair Scheduler (hfs) is a slot-based MapReduce scheme designed to ensure a degree of fairness among the jobs, by guaranteeing each job at least some minimum number of allocated slots. Our prime contribution in this paper is a different, flexible scheduling allocation scheme, known as flex. Our goal is to optimize any of a variety of standard scheduling theory metrics (response time, stretch, makespan and Service Level Agreements (slas), among others) while ensuring the same minimum job slot guarantees as in hfs, and maximum job slot guarantees as well. The flex allocation scheduler can be regarded as an add-on module that works synergistically with hfs. We describe the mathematical basis for flex, and compare it with fifo and hfs in a variety of experiments.
Fichier principal
Vignette du fichier
middleware10-cr.pdf (1.04 Mo) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-01055274 , version 1 (12-08-2014)

Licence

Attribution

Identifiers

Cite

Joel Wolf, Deepak Rajan, Kirsten Hildrum, Rohit Khandekar, Vibhore Kumar, et al.. FLEX: A Slot Allocation Scheduling Optimizer for MapReduce Workloads. ACM/IFIP/USENIX 11th International Middleware Conference (MIDDLEWARE), Nov 2010, Bangalore, India. pp.1-20, ⟨10.1007/978-3-642-16955-7_1⟩. ⟨hal-01055274⟩
152 View
536 Download

Altmetric

Share

Gmail Facebook X LinkedIn More