You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@aurora.apache.org by "Zameer Manji (JIRA)" <ji...@apache.org> on 2016/01/22 02:39:40 UTC
[jira] [Comment Edited] (AURORA-1582) Task History Pruning attempts
can fail silently
[ https://issues.apache.org/jira/browse/AURORA-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111758#comment-15111758 ]
Zameer Manji edited comment on AURORA-1582 at 1/22/16 1:39 AM:
---------------------------------------------------------------
{noformat}
commit c89fecbcd93aa6ddcf6af60f2a2cb6315b7a4d19
Author: Zameer Manji <zm...@apache.org>
Date: Thu Jan 21 17:38:25 2016 -0800
Turn TaskHistoryPruner into a service and trigger shutdown on pruning failure.
Task pruning is key to operating a large cluster and failure to prune should
trigger shutdown to prevent unbounded growth of storage. This patch turns
`TaskHistoryPruner` into a service which propagates failure from failed pruning
attempts towards the `ServiceManager`. Also completing a TODO which removes a
test for behaviour that is very awkward to test for.
Bugs closed: AURORA-1582
Reviewed at https://reviews.apache.org/r/42332/
3 files changed, 68 insertions(+), 111 deletions(-)
{noformat}
was (Author: zmanji):
````
commit c89fecbcd93aa6ddcf6af60f2a2cb6315b7a4d19
Author: Zameer Manji <zm...@apache.org>
Date: Thu Jan 21 17:38:25 2016 -0800
Turn TaskHistoryPruner into a service and trigger shutdown on pruning failure.
Task pruning is key to operating a large cluster and failure to prune should
trigger shutdown to prevent unbounded growth of storage. This patch turns
`TaskHistoryPruner` into a service which propagates failure from failed pruning
attempts towards the `ServiceManager`. Also completing a TODO which removes a
test for behaviour that is very awkward to test for.
Bugs closed: AURORA-1582
Reviewed at https://reviews.apache.org/r/42332/
3 files changed, 68 insertions(+), 111 deletions(-)
````
> Task History Pruning attempts can fail silently
> -----------------------------------------------
>
> Key: AURORA-1582
> URL: https://issues.apache.org/jira/browse/AURORA-1582
> Project: Aurora
> Issue Type: Bug
> Reporter: Zameer Manji
> Assignee: Zameer Manji
>
> As discovered in AURORA-1580, task history pruning attempts can fail and if they do fail, they fail silently. The root cause seems to be that AsyncModule's {{AsyncProcessor}} threads just log the unhandled exception if it exists:
> {noformat}
> private static void evaluateResult(Runnable runnable, Throwable throwable, Logger logger) {
> // See java.util.concurrent.ThreadPoolExecutor#afterExecute(Runnable, Throwable)
> // for more details and an implementation example.
> if (throwable == null) {
> if (runnable instanceof Future) {
> try {
> Future<?> future = (Future<?>) runnable;
> if (future.isDone()) {
> future.get();
> }
> } catch (InterruptedException ie) {
> Thread.currentThread().interrupt();
> } catch (ExecutionException ee) {
> logger.error(ee.toString(), ee);
> }
> }
> } else {
> logger.error(throwable.toString(), throwable);
> }
> }
> {noformat}
> I think instead of silently failing if work on these threads fail, we should shut down the scheduler, much like how if the preemptor or other guava service fails we shut down the scheduler. This way the scheduler does not enter an undefined state and operators are informed of the abnormal behaviour.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)