You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Min Zhou (JIRA)" <ji...@apache.org> on 2014/10/24 00:15:35 UTC
[jira] [Created] (SPARK-4069) [SPARK-YARN] ApplicationMaster should
releases all executors' containers before unregistering itself from Yarn RM
Min Zhou created SPARK-4069:
-------------------------------
Summary: [SPARK-YARN] ApplicationMaster should releases all executors' containers before unregistering itself from Yarn RM
Key: SPARK-4069
URL: https://issues.apache.org/jira/browse/SPARK-4069
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.1.0
Reporter: Min Zhou
Curently, ApplciationMaster in yarn mode simply unregister itself from yarn master , a.k.a resourcemanager. Itnever release executors' containers before that. Yarn's master will make a decision to kill all the executors' containers if it face such scenario. so the log of resourcemanager is like below
{noformat}
2014-10-22 23:39:09,903 DEBUG org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Processing event for appattempt_1414003182949_0004_000001 of type UNREGISTERED
2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1414003182949_0004_000001 State change from RUNNING to FINAL_SAVING
2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1414003182949_0004 with final state: FINISHING
2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1414003182949_0004 State change from RUNNING to FINAL_SAVING
2014-10-22 23:39:09,903 DEBUG org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Processing event for appattempt_1414003182949_0004_000001 of type ATTEMPT_UPDATE_SAVED
2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1414003182949_0004
2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1414003182949_0004_000001 State change from FINAL_SAVING to FINISHING
2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1414003182949_0004 State change from FINAL_SAVING to FINISHING
2014-10-22 23:39:10,485 DEBUG org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Processing event for appattempt_1414003182949_0004_000001 of type CONTAINER_FINISHED
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1414003182949_0004_01_000001 Container Transitioned from RUNNING to COMPLETED
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1414003182949_0004_000001
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerApp: Completed container: container_1414003182949_0004_01_000001 in state: COMPLETED event:FINISHED
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore: Finish information of container container_1414003182949_0004_01_000001 is written
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1414003182949_0004_000001 State change from FINISHING to FINISHED
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=akim OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1414003182949_0004 CONTAINERID=container_1414003182949_0004_01_000001
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Stored the finish data of container container_1414003182949_0004_01_000001
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode: Released container container_1414003182949_0004_01_000001 of capacity <memory:3072, vCores:1> on host host1, which currently has 0 containers, <memory:0, vCores:0> used and <memory:241901, vCores:32> available, release resources=true
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1414003182949_0004 State change from FINISHING to FINISHED
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore: Finish information of application attempt appattempt_1414003182949_0004_000001 is written
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=akim OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1414003182949_0004
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1414003182949_0004_000001 released container container_1414003182949_0004_01_000001 on node: host: host2:8041 #containers=0 available=<memory:241901, vCores:32> used=<memory:0, vCores:0> with event: FINISHED
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Stored the finish data of application attempt appattempt_1414003182949_0004_000001
2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application appattempt_1414003182949_0004_000001 is done. finalState=FINISHED
2014-10-22 23:39:10,486 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore: Finish information of application application_1414003182949_0004 is written
2014-10-22 23:39:10,486 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1414003182949_0004_01_000019 Container Transitioned from RUNNING to KILLED
{noformat}
Although it won't affect the job's final succeed status, but the log will confuse users.
If we run a spark job on yarn 2.4.1 with timeline server enabled, we will get errors on the resourcemanager's log
{noformat}
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000019
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000017
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000009
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000010
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000012
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000003
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000005
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000004
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000015
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000018
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000013
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000008
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000014
2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000007
2014-10-22 23:39:10,638 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000002
{noformat}
This is because the application is finished before containers being terminated. Once the executors' containers being killed, resourcemanager will try to log something for containers' finsih event, but can't find a writer due to the application finished before that.
{noformat}
java.io.IOException: History file of application application_1414003182949_0003 is not opened
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getHistoryFileWriter(FileSystemApplicationHistoryStore.java:643)
org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.containerFinished(FileSystemApplicationHistoryStore.java:532)
org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter.handleWritingApplicationHistoryEvent(RMApplicationHistoryWriter.java:203)
org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter$ForwardingEventHandler.handle(RMApplicationHistoryWriter.java:297)
org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter$ForwardingEventHandler.handle(RMApplicationHistoryWriter.java:292)
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org