You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Tyson Condie (JIRA)" <ji...@apache.org> on 2018/02/28 23:36:00 UTC
[jira] [Commented] (REEF-1981) Evaluators fail to heartbeat to
restarted driver
[ https://issues.apache.org/jira/browse/REEF-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381236#comment-16381236 ]
Tyson Condie commented on REEF-1981:
------------------------------------
As per offline discussions, it seems like AM/Driver discovery occurs via HDFS. I believe [~seanpo03] confirmed that the file used for the discover was in fact written, but we have not been able to identify within the REEF codebase where this information is read (by Evaluators) to rediscover.
> Evaluators fail to heartbeat to restarted driver
> ------------------------------------------------
>
> Key: REEF-1981
> URL: https://issues.apache.org/jira/browse/REEF-1981
> Project: REEF
> Issue Type: Bug
> Components: REEF Evaluator
> Reporter: Sean Po
> Priority: Major
>
> On driver failover, we are hitting the following exception:
> Feb 28, 2018 1:43:23 PM org.apache.reef.runtime.yarn.driver.YarnDriverRuntimeRestartManager informAboutEvaluatorFailures
> WARNING: Container [container_e4838_1519690816115_0025_01_000005] has failed during driver restart process, FailedEvaluatorHandler will be triggered, but no additional evaluator can be requested due to YARN-2433.
> Feb 28, 2018 1:43:23 PM org.apache.reef.runtime.common.driver.evaluator.EvaluatorManager onEvaluatorException
> WARNING: Failed evaluator: container_e4838_1519690816115_0025_01_000005
> org.apache.reef.exception.EvaluatorException: Evaluator [container_e4838_1519690816115_0025_01_000005] is assumed to be in state [ALLOCATED]. But the resource manager reports it to be in state [FAILED]. This most likely means that the Evaluator suffered a failure before being used.
> at org.apache.reef.runtime.common.driver.evaluator.EvaluatorManager.onResourceStatusMessage(EvaluatorManager.java:693)
> at org.apache.reef.runtime.common.driver.resourcemanager.ResourceStatusHandler.onNext(ResourceStatusHandler.java:91)
> at org.apache.reef.runtime.common.driver.resourcemanager.ResourceStatusHandler.onNext(ResourceStatusHandler.java:38)
> at org.apache.reef.runtime.yarn.driver.REEFEventHandlers.onResourceStatus(REEFEventHandlers.java:93)
> at org.apache.reef.runtime.yarn.driver.YarnDriverRuntimeRestartManager.informAboutEvaluatorFailures(YarnDriverRuntimeRestartManager.java:230)
> at org.apache.reef.driver.restart.DriverRestartManager.onDriverRestartCompleted(DriverRestartManager.java:282)
> at org.apache.reef.driver.restart.DriverRestartManager.access$000(DriverRestartManager.java:47)
> at org.apache.reef.driver.restart.DriverRestartManager$1.run(DriverRestartManager.java:136)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
> However, according to Yarn RM logs, these containers have not failed at this time. We suspect that the evaluators are failing to heartbeat into the new Driver.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)