[GitHub] [incubator-kyuubi] ulysses-you opened a new issue #1800: [Bug] Exit spark engine if spark context is stopped

[GitBox <gi...@apache.org>]

ulysses-you opened a new issue #1800:
URL: https://github.com/apache/incubator-kyuubi/issues/1800


   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Describe the bug
   
   If the spark engine get OOM issue or underlying bug, the SparkContext will stop itself. However the engine jvm is still alived and it become a zombie process.
   
   We should aware the status of SparkContext and stop the spark engine if it's already stopped.
   
   The error msg:
   ```
   
   (No active SparkContext.)
   
           at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
           at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:73)
           at org.apache.kyuubi.engine.spark.session.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:93)
           at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45)
           at org.apache.kyuubi.service.ThriftBinaryFrontendService.getSessionHandle(ThriftBinaryFrontendService.scala:199)
           at org.apache.kyuubi.service.ThriftBinaryFrontendService.OpenSession(ThriftBinaryFrontendService.scala:208)
           at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497)
           at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482)
           at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
           at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
           at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
           at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   Caused by: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
   ```
   
   ### Affects Version(s)
   
   1.3.1/1.4.0
   
   ### Kyuubi Server Log Output
   
   _No response_
   
   ### Kyuubi Engine Log Output
   
   _No response_
   
   ### Kyuubi Server Configurations
   
   _No response_
   
   ### Kyuubi Engine Configurations
   
   _No response_
   
   ### Additional context
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1800: [Bug] Exit spark engine if spark context is stopped

[GitBox <gi...@apache.org>]

yaooqinn commented on issue #1800:
URL: https://github.com/apache/incubator-kyuubi/issues/1800#issuecomment-1016181718


   have you checked the jstack to see which thread blocked termination?
   
   BTW, we seem to have a listener to handle it
   
   ```scala
     override def onApplicationEnd(event: SparkListenerApplicationEnd): Unit = {
       server.getServiceState match {
         case ServiceState.STOPPED =>
           debug("Received ApplicationEnd Message form Spark after the engine has stopped")
         case state =>
           info(s"Received ApplicationEnd Message from Spark at $state, stopping")
           server.stop()
       }
     }
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kyuubi] yaooqinn closed issue #1800: [Bug] Exit spark engine if spark context is stopped

[GitBox <gi...@apache.org>]

yaooqinn closed issue #1800:
URL: https://github.com/apache/incubator-kyuubi/issues/1800


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org