Will Hadoop CapacityScheduler slot reservation scheme cost deadlock?

["Li, Shen" <sh...@illinois.edu>]

Hi,

Hadoop CapacityScheduler reserves slots on a TaskTracker if the available memory does not satisfy the need of the job being scheduled ( in TaskSchedulingMgr.getTaskFromQueue(...)). However, will this create any deadlock? Suppose, I have two different jobs from two different queues, each map task needs 3 slots. Each machine has only 4 map slots. At first, when job 1 being scheduled, there are 2 slots available on machine A, so job 1 reserves these 2 slots. Later, when job 2 being scheduled on machine A, another 2 slots become available, so job 2 reserves the remaining two slots. In this case, neither job 1 or job 2 will ever get enough slots to be executed on machine A.

Is there any scheme in the CapacityScheduler preventing this from happening?


Thanks,

Shen Li<http://www.cs.illinois.edu/homes/shenli3>