You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@aurora.apache.org by "Joe Smith (JIRA)" <ji...@apache.org> on 2015/07/01 20:24:04 UTC
[jira] [Commented] (AURORA-1096) Scheduler updater should limit the
number of job/instance events
[ https://issues.apache.org/jira/browse/AURORA-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14610775#comment-14610775 ]
Joe Smith commented on AURORA-1096:
-----------------------------------
[~maximk] I'm leaning toward knowing ahead of time being preferred to allowing a job to (potentially and surprisingly) error out because it hit a scheduler-imposed limit vs. something that a user did.
If you're okay with that, my intention is to add a check inside [{{startJobUpdate}}|https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java#L1085]
> Scheduler updater should limit the number of job/instance events
> ----------------------------------------------------------------
>
> Key: AURORA-1096
> URL: https://issues.apache.org/jira/browse/AURORA-1096
> Project: Aurora
> Issue Type: Story
> Components: Scheduler
> Reporter: Maxim Khutornenko
> Assignee: Joe Smith
>
> Large/flapping scheduler job updates may generate too many events in the update store. The update settings are fully controlled by the user and there is a potential for a misconfigured job update to completely overwhelm our in-memory DB storage with job update instance events.
> For example, a large flapping update with {{max_per_shard_failures}} and {{max_total_failures}} set to max INT when left unattended can quickly consume all available RAM and kill the scheduler. A manual cleanup of the scheduler log would be needed to bring the scheduler up.
> This can be especially relevant with the introduction of update heartbeats (AURORA-690) that can further exacerbate the problem (e.g. when {{blockIfNoPulseAfterMs}} set too low wrt the external service pulse rate).
> We need to cap the max per-job lifetime count of {{JobUpdateEvent}} and {{JobInstanceUpdateEvent}} instances. A nice bonus would be providing a hint in the UI when the event sequence is cut off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)