You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by Bhuvan Arumugam <bh...@apache.org> on 2015/02/16 07:14:06 UTC
reasonable preemption delay to use
Hello,
Recently, in one of our clusters we noticed production jobs go to
PENDING state, due to insufficient CPU. The non production jobs are
not preempted, as we haven't used --preemption_delay flag for
scheduler. The default value for this flag is 10mins. Why is it too
high? Is there any reasoning behind using 10mins as a default value?
We are thinking to to use 2mins for this flag. We wouldn't want to
wait beyond 2mins to run a prod job during resource constraint. Does
it sound reasonable? What's the typical preemption delay used by SREs?
--
Regards,
Bhuvan Arumugam
www.livecipher.com
Re: reasonable preemption delay to use
Posted by Maxim Khutornenko <ma...@apache.org>.
The watch_secs is triggered when a task enters RUNNING. In order for
the rolling update to not fail early the restart_threshold [1] needs
to be bumped up to account for the preemption delay.
As for the default preemption delay, it was implemented to avoid
unnecessary churn in the cluster. Larger/constraint-diverse tasks take
longer to bin-place, as such there could be occasional scheduling
delays when resources are tight. Hence, the grace buffer. You can
definitely dial it in given the specifics of your cluster.
Thanks,
Maxim
[1] - https://github.com/apache/incubator-aurora/blob/master/docs/configuration-reference.md#updateconfig-objects
On Tue, Feb 17, 2015 at 12:51 AM, Erb, Stephan
<St...@blue-yonder.com> wrote:
> If I remember correctly, you also have to make sure that your UpdateConfig watch_secs is larger than your preemption_delay. Otherwise a rolling update of a production job might not be able to get the resources it needs.
>
> Best Regards,
> Stephan
> ________________________________________
> From: Bhuvan Arumugam <bh...@apache.org>
> Sent: Monday, February 16, 2015 7:14 AM
> To: dev@aurora.incubator.apache.org
> Subject: reasonable preemption delay to use
>
> Hello,
> Recently, in one of our clusters we noticed production jobs go to
> PENDING state, due to insufficient CPU. The non production jobs are
> not preempted, as we haven't used --preemption_delay flag for
> scheduler. The default value for this flag is 10mins. Why is it too
> high? Is there any reasoning behind using 10mins as a default value?
>
> We are thinking to to use 2mins for this flag. We wouldn't want to
> wait beyond 2mins to run a prod job during resource constraint. Does
> it sound reasonable? What's the typical preemption delay used by SREs?
>
> --
> Regards,
> Bhuvan Arumugam
> www.livecipher.com
Re: reasonable preemption delay to use
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
If I remember correctly, you also have to make sure that your UpdateConfig watch_secs is larger than your preemption_delay. Otherwise a rolling update of a production job might not be able to get the resources it needs.
Best Regards,
Stephan
________________________________________
From: Bhuvan Arumugam <bh...@apache.org>
Sent: Monday, February 16, 2015 7:14 AM
To: dev@aurora.incubator.apache.org
Subject: reasonable preemption delay to use
Hello,
Recently, in one of our clusters we noticed production jobs go to
PENDING state, due to insufficient CPU. The non production jobs are
not preempted, as we haven't used --preemption_delay flag for
scheduler. The default value for this flag is 10mins. Why is it too
high? Is there any reasoning behind using 10mins as a default value?
We are thinking to to use 2mins for this flag. We wouldn't want to
wait beyond 2mins to run a prod job during resource constraint. Does
it sound reasonable? What's the typical preemption delay used by SREs?
--
Regards,
Bhuvan Arumugam
www.livecipher.com