You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Radim Kolar <hs...@filez.com> on 2012/10/17 14:19:51 UTC
speculative execution in yarn
Is yarn smart enough to do speculative execution only if there are no
other jobs running or queued?
Re: speculative execution in yarn
Posted by Radim Kolar <hs...@filez.com>.
Dne 17.10.2012 20:28, Vinod Kumar Vavilapalli napsal(a):
> Speculative execution is a per-job concept, so in 2.* release line, it
> is MR AM's responsibility.
>
> Because there is a two level scheduling - one at RM and one at AM, AMs
> have not way of figuring out if there are other jobs are not.
AM can query RM if there are other jobs sitting in same queue as its
currently running job before enabling speculative execution.
Re: speculative execution in yarn
Posted by Radim Kolar <hs...@filez.com>.
Dne 17.10.2012 20:28, Vinod Kumar Vavilapalli napsal(a):
> Speculative execution is a per-job concept, so in 2.* release line, it
> is MR AM's responsibility.
>
> Because there is a two level scheduling - one at RM and one at AM, AMs
> have not way of figuring out if there are other jobs are not.
AM can query RM if there are other jobs sitting in same queue as its
currently running job before enabling speculative execution.
Re: speculative execution in yarn
Posted by Radim Kolar <hs...@filez.com>.
Dne 17.10.2012 20:28, Vinod Kumar Vavilapalli napsal(a):
> Speculative execution is a per-job concept, so in 2.* release line, it
> is MR AM's responsibility.
>
> Because there is a two level scheduling - one at RM and one at AM, AMs
> have not way of figuring out if there are other jobs are not.
AM can query RM if there are other jobs sitting in same queue as its
currently running job before enabling speculative execution.
Re: speculative execution in yarn
Posted by Radim Kolar <hs...@filez.com>.
Dne 17.10.2012 20:28, Vinod Kumar Vavilapalli napsal(a):
> Speculative execution is a per-job concept, so in 2.* release line, it
> is MR AM's responsibility.
>
> Because there is a two level scheduling - one at RM and one at AM, AMs
> have not way of figuring out if there are other jobs are not.
AM can query RM if there are other jobs sitting in same queue as its
currently running job before enabling speculative execution.
Re: speculative execution in yarn
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Speculative execution is a per-job concept, so in 2.* release line, it is MR AM's responsibility.
Because there is a two level scheduling - one at RM and one at AM, AMs have not way of figuring out if there are other jobs are not. In general, AMs send container requests in a greedy manner and leave the responsibility of cross-job fairness to the RM and the scheduler.
In sum, to answer your question, speculative execution will be triggered only based on task run times, and irrespective of other jobs.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Oct 17, 2012, at 5:19 AM, Radim Kolar wrote:
> Is yarn smart enough to do speculative execution only if there are no other jobs running or queued?
Re: speculative execution in yarn
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Speculative execution is a per-job concept, so in 2.* release line, it is MR AM's responsibility.
Because there is a two level scheduling - one at RM and one at AM, AMs have not way of figuring out if there are other jobs are not. In general, AMs send container requests in a greedy manner and leave the responsibility of cross-job fairness to the RM and the scheduler.
In sum, to answer your question, speculative execution will be triggered only based on task run times, and irrespective of other jobs.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Oct 17, 2012, at 5:19 AM, Radim Kolar wrote:
> Is yarn smart enough to do speculative execution only if there are no other jobs running or queued?
Re: speculative execution in yarn
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Speculative execution is a per-job concept, so in 2.* release line, it is MR AM's responsibility.
Because there is a two level scheduling - one at RM and one at AM, AMs have not way of figuring out if there are other jobs are not. In general, AMs send container requests in a greedy manner and leave the responsibility of cross-job fairness to the RM and the scheduler.
In sum, to answer your question, speculative execution will be triggered only based on task run times, and irrespective of other jobs.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Oct 17, 2012, at 5:19 AM, Radim Kolar wrote:
> Is yarn smart enough to do speculative execution only if there are no other jobs running or queued?
Re: speculative execution in yarn
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Speculative execution is a per-job concept, so in 2.* release line, it is MR AM's responsibility.
Because there is a two level scheduling - one at RM and one at AM, AMs have not way of figuring out if there are other jobs are not. In general, AMs send container requests in a greedy manner and leave the responsibility of cross-job fairness to the RM and the scheduler.
In sum, to answer your question, speculative execution will be triggered only based on task run times, and irrespective of other jobs.
Thanks,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Oct 17, 2012, at 5:19 AM, Radim Kolar wrote:
> Is yarn smart enough to do speculative execution only if there are no other jobs running or queued?