[GitHub] [incubator-livy] runzhiwang opened a new pull request #265: [LIVY-727] Fix session state always be idle though the yarn application has been killed after restart livy

[GitBox <gi...@apache.org>]

runzhiwang opened a new pull request #265: [LIVY-727] Fix session state always be idle though the yarn application has been killed after restart livy
URL: https://github.com/apache/incubator-livy/pull/265
 
 
   ## What changes were proposed in this pull request?
   
   [LIVY-727] Fix session state always be idle though the yarn application has been killed after restart livy.
   
   Follows are steps to reproduce the problem:
   1. Set livy.server.recovery.mode=recovery, and create a session in yarn-cluster
   2. Restart livy
   3. Kill the yarn application.
   4. The session state will always be idle and never change to killed or dead. Just as the image, livy-session-16 has been killed in yarn, but the state is still idle.
   ![image](https://user-images.githubusercontent.com/51938049/70371088-92695c80-1909-11ea-875c-73696db693ce.png)
   
   The cause of the problem are as follows:
   1. Because when recover session, livy will not startDriver again, so the driverProcess is None.
   2. SparkYarnApp will not be created in `driverProcess.map { _ => SparkApp.create(appTag, appId, driverProcess, livyConf, Some(this)) }` when driverProcess is None.
   3. So yarnAppMonitorThread of the session will never start, and the session state will never change.
   
   How to fix the bug:
   1. If livy run in yarn, SparkApp will create even though the driverProcess is None
   2. If not run in yarn, SparkApp will not create, because the code require driverProcess is not None, 
     and I don't want to change the behavior.
   
   ## How was this patch tested?
   
   1. Set livy.server.recovery.mode=recovery, and create a session in yarn-cluster
   2. Restart livy
   3. Kill the yarn application.
   4. The session state will change to killed
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services