[jira] [Commented] (SPARK-4069) [SPARK-YARN] ApplicationMaster should release all executors' containers before unregistering itself from Yarn RM

["Andrew Or (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606701#comment-14606701 ] 

Andrew Or commented on SPARK-4069:
----------------------------------

Fixed in YARN-3415.

> [SPARK-YARN] ApplicationMaster should release all executors' containers before unregistering itself from Yarn RM
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-4069
>                 URL: https://issues.apache.org/jira/browse/SPARK-4069
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.1.0
>            Reporter: Min Zhou
>
> Curently,  ApplciationMaster in yarn mode simply unregister itself from yarn master , a.k.a resourcemanager.  Itnever release executors' containers before that.  Yarn's master will make a decision to kill all the executors' containers if it face such scenario.  so the log of resourcemanager is like below 
> {noformat}
> 2014-10-22 23:39:09,903 DEBUG org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Processing event for appattempt_1414003182949_0004_000001 of type UNREGISTERED
> 2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1414003182949_0004_000001 State change from RUNNING to FINAL_SAVING
> 2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1414003182949_0004 with final state: FINISHING
> 2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1414003182949_0004 State change from RUNNING to FINAL_SAVING
> 2014-10-22 23:39:09,903 DEBUG org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Processing event for appattempt_1414003182949_0004_000001 of type ATTEMPT_UPDATE_SAVED
> 2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1414003182949_0004
> 2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1414003182949_0004_000001 State change from FINAL_SAVING to FINISHING
> 2014-10-22 23:39:09,903 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1414003182949_0004 State change from FINAL_SAVING to FINISHING
> 2014-10-22 23:39:10,485 DEBUG org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Processing event for appattempt_1414003182949_0004_000001 of type CONTAINER_FINISHED
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1414003182949_0004_01_000001 Container Transitioned from RUNNING to COMPLETED
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1414003182949_0004_000001
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerApp: Completed container: container_1414003182949_0004_01_000001 in state: COMPLETED event:FINISHED
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore: Finish information of container container_1414003182949_0004_01_000001 is written
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1414003182949_0004_000001 State change from FINISHING to FINISHED
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=akim	OPERATION=AM Released Container	TARGET=SchedulerApp	RESULT=SUCCESS	APPID=application_1414003182949_0004	CONTAINERID=container_1414003182949_0004_01_000001
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Stored the finish data of container container_1414003182949_0004_01_000001
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode: Released container container_1414003182949_0004_01_000001 of capacity <memory:3072, vCores:1> on host host1, which currently has 0 containers, <memory:0, vCores:0> used and <memory:241901, vCores:32> available, release resources=true
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1414003182949_0004 State change from FINISHING to FINISHED
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore: Finish information of application attempt appattempt_1414003182949_0004_000001 is written
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=akim	OPERATION=Application Finished - Succeeded	TARGET=RMAppManager	RESULT=SUCCESS	APPID=application_1414003182949_0004
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1414003182949_0004_000001 released container container_1414003182949_0004_01_000001 on node: host: host2:8041 #containers=0 available=<memory:241901, vCores:32> used=<memory:0, vCores:0> with event: FINISHED
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Stored the finish data of application attempt appattempt_1414003182949_0004_000001
> 2014-10-22 23:39:10,485 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application appattempt_1414003182949_0004_000001 is done. finalState=FINISHED
> 2014-10-22 23:39:10,486 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore: Finish information of application application_1414003182949_0004 is written
> 2014-10-22 23:39:10,486 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1414003182949_0004_01_000019 Container Transitioned from RUNNING to KILLED
> {noformat}
> Although it won't affect the job's final succeed status, but the log will confuse users. 
> If we run a  spark job on yarn 2.4.1 with timeline server enabled,  we will get errors on the resourcemanager's log
> {noformat}
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000019
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000017
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000009
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000010
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000012
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000003
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000005
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000004
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000015
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000018
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000013
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000008
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000014
> 2014-10-22 23:39:10,637 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000007
> 2014-10-22 23:39:10,638 ERROR org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter: Error when storing the finish data of container container_1414003182949_0004_01_000002
> {noformat}
> This is because the application is finished before containers being terminated.  Once the executors' containers being killed,  resourcemanager will try to log something for containers' finsih event, but can't find a writer due to the application  finished before that.  
> {noformat}
> java.io.IOException: History file of application application_1414003182949_0003 is not opened
>     org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getHistoryFileWriter(FileSystemApplicationHistoryStore.java:643)
>     org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.containerFinished(FileSystemApplicationHistoryStore.java:532)
>     org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter.handleWritingApplicationHistoryEvent(RMApplicationHistoryWriter.java:203)
>     org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter$ForwardingEventHandler.handle(RMApplicationHistoryWriter.java:297)
>     org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter$ForwardingEventHandler.handle(RMApplicationHistoryWriter.java:292)
>     org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
>     org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
>     java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org