You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "kangzhilv (via GitHub)" <gi...@apache.org> on 2025/04/11 09:35:46 UTC

[I] [Bug] Spark Driver can not close where the task compact [kyuubi]

kangzhilv opened a new issue, #7019:
URL: https://github.com/apache/kyuubi/issues/7019

   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Describe the bug
   
   when i use k8s deploy spark task, the task driver pod is not compact.
   
   
   ### Affects Version(s)
   
   1.9.3
   
   ### Kyuubi Server Log Output
   
   ```logtalk
   
   ```
   
   ### Kyuubi Engine Log Output
   
   ```logtalk
   25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is initialized.
   25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is started.
   25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
   25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
   25/04/09 03:00:08 ERROR SparkTBinaryFrontendService: Error fetching results: 
   25/04/09 03:00:08 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [09f53cfd-2220-4e1e-8dd7-6b61dc8b2f52]
   25/04/09 03:00:08 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [09f53cfd-2220-4e1e-8dd7-6b61dc8b2f52]
   25/04/09 03:00:10 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
   25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [0b434ecc-111a-44f5-b9d5-9ad29f67ff83]
   25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [0b434ecc-111a-44f5-b9d5-9ad29f67ff83]
   25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
   25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [ab9e6ae9-0fef-4210-b432-286a9bf4885e]
   25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [ab9e6ae9-0fef-4210-b432-286a9bf4885e]
   25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
   25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [84a5120e-4238-4a8b-9b4b-20720df1eabc]
   25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [84a5120e-4238-4a8b-9b4b-20720df1eabc]
   25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
   25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [773ad9ba-687c-4004-a64a-0b0b7ef5089f]
   25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [773ad9ba-687c-4004-a64a-0b0b7ef5089f]
   25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
   25/04/09 03:04:16 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
   25/04/09 03:04:16 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
   25/04/09 03:04:16 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
   25/04/09 03:04:16 INFO SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
   25/04/09 03:08:19 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [7041cd34-ca41-4593-bbad-7979c9a071da]
   25/04/09 03:08:19 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [7041cd34-ca41-4593-bbad-7979c9a071da]
   ```
   
   ### Kyuubi Server Configurations
   
   ```yaml
   
   ```
   
   ### Kyuubi Engine Configurations
   
   ```yaml
   
   ```
   
   ### Additional context
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
   - [x] No. I cannot submit a PR at this time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug] Spark Driver can not close where the task compact [kyuubi]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7019:
URL: https://github.com/apache/kyuubi/issues/7019#issuecomment-2796370554

   Hello @kangzhilv,
   Thanks for finding the time to report the issue!
   We really appreciate the community's efforts to improve Apache Kyuubi.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug] Spark Driver cannot be shutdown at the end of a task [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #7019:
URL: https://github.com/apache/kyuubi/issues/7019#issuecomment-2822906010

   Unlike Spark on YARN, in Spark on K8s cluster mode, daemon threads would block your driver JVM exit, see technical details at https://github.com/apache/spark/pull/46889


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug] Spark Driver cannot be shutdown at the end of a task [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #7019:
URL: https://github.com/apache/kyuubi/issues/7019#issuecomment-2808692403

   Can you provide the jstack info to reveal if there is any daemon thread that prevents JVM from exiting?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [I] [Bug] Spark Driver can not close where the task compact [kyuubi]

Posted by "kangzhilv (via GitHub)" <gi...@apache.org>.
kangzhilv commented on issue #7019:
URL: https://github.com/apache/kyuubi/issues/7019#issuecomment-2796373986

   net information
   Active Internet connections (servers and established)
   Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
   tcp6       0      0 172.26.13.100:52060     172.26.255.198:9083     ESTABLISHED -                   
   tcp6       0      0 172.26.13.100:34905     172.26.255.175:8216     ESTABLISHED -      


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org