[jira] [Commented] (AURORA-1837) Adding delay on pruning inactive jobs

["Stephan Erb (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AURORA-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705384#comment-15705384 ] 

Stephan Erb commented on AURORA-1837:
-------------------------------------

I am not sure if I understand the pruning process and your proposal correctly, but it seems like there is a performance bug in the code (as is today):

For _each terminated task instance_ we schedule:
* a [per task id instance pruning|https://github.com/apache/aurora/blob/99164834a3f23fa50576d2044743d2cb6d7da535/src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java#L156-L165]
* a [group based task instance pruning|https://github.com/apache/aurora/blob/99164834a3f23fa50576d2044743d2cb6d7da535/src/main/java/org/apache/aurora/scheduler/pruning/TaskHistoryPruner.java#L167-L188]

The second query/pruning is independent of the task id it is scheduled for. So in your example of scheduler restore it is run 100 times (assuming the default of 100 terminated tasks). A single run would be sufficient. This sounds like a potential two order of magnitude perf improvement.

> Adding delay on pruning inactive jobs
> -------------------------------------
>
>                 Key: AURORA-1837
>                 URL: https://issues.apache.org/jira/browse/AURORA-1837
>             Project: Aurora
>          Issue Type: Task
>            Reporter: Reza Motamedi
>            Priority: Minor
>              Labels: scheduler
>
> TaskHistoryPrunner registers all inactive tasks upon _state_ change for pruning. 
> TaskHistoryPrunner::registerInactiveTask uses delay executor to schedule the process of prunning _task_s and _job_s. This is totally reasonable since pruning in not critical and can be done when the load on the scheduler is low.
> Once pruning tasks, a delay is used in the first pruning phase (shutdownOnError) but in the second one seems to be instant. This has caused problems when lots of tasks are changing state and the load on the scheduler is high (for instance during scheduler restore).
> to do items:
> 1. investigate if we can add a delay to all executions, and what the delays should be.
> 2.  investigate if executions can be suppressed based on the load on the scheduler. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)