[jira] [Commented] (REEF-726) Race condition with completed Containers

["Andrew Chung (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737518#comment-14737518 ] 

Andrew Chung commented on REEF-726:
-----------------------------------

Another possible race condition is when the Driver has already received the {{DONE}} heartbeat but has not yet processed it. Before the Driver processes the {{DONE}} heartbeat, YARN reports that the Evaluator is {{DONE}}, and thus the Evaluator is in an unexpected state, as the Driver expects it to be {{RUNNING}}. The Driver then triggers an incorrect failed Evaluator exception.

> Race condition with completed Containers
> ----------------------------------------
>
>                 Key: REEF-726
>                 URL: https://issues.apache.org/jira/browse/REEF-726
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF Driver, REEF Runtime YARN, REEF-Runtime-YARN
>            Reporter: Andrew Chung
>            Assignee: Andrew Chung
>         Attachments: completed_containers_race.txt, restart_failure.txt
>
>
> There is a race condition in which it's possible for a Container to be removed twice in {{YarnContainerManager}}. i.e. {{removeAndGet()}} is called both in {{onContainersCompleted}} (without a lock) and by the {{EvaluatorManager}} (with a lock on the {{EvaluatorManager}}).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)