You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Pei Jiang (JIRA)" <ji...@apache.org> on 2017/11/03 19:03:02 UTC
[jira] [Created] (REEF-1949) Closing ThreadPoolStage before tasks
are finished
Pei Jiang created REEF-1949:
-------------------------------
Summary: Closing ThreadPoolStage before tasks are finished
Key: REEF-1949
URL: https://issues.apache.org/jira/browse/REEF-1949
Project: REEF
Issue Type: Bug
Components: REEF Driver
Affects Versions: 0.17
Reporter: Pei Jiang
Priority: Normal
In EvaluatorManager.onEvaluatorDone(),
// This relies on the dispatcher to call the CompletedEvaluator handler.
this.messageDispatcher.onEvaluatorCompleted(new CompletedEvaluatorImpl(this.evaluatorId));
// This will close the dispatcher, which in turns shut down the executor in ThreadPoolStage.
this.close();
Since in onEvaluatorCompleted the message sending task is submitted to an executor, there is no guarantee that the CompletedEvaluator message will be sent before the termination of the executor in this.close() call. When this happens, the CompletedEvaluator handler will not be triggered so the driver will think that some evaluators are alive and hence hang.
Relevant logs:
…
Nov 01, 2017 11:05:57 PM org.apache.reef.wake.impl.ThreadPoolStage close
SEVERE: Closing ThreadPoolStage EvaluatorMessageDispatcher:container_1508975419755_0006_01_000004: Executor did not terminate in 1,000 ms. Dropping 2 tasks
Nov 01, 2017 11:05:57 PM org.apache.reef.wake.impl.ThreadPoolStage close
SEVERE: Closing ThreadPoolStage EvaluatorMessageDispatcher:container_1508975419755_0006_01_000004: Executor failed to terminate.
End of LogType:driver.stderr
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)