You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "kangzhilv (via GitHub)" <gi...@apache.org> on 2025/04/11 09:35:46 UTC
[I] [Bug] Spark Driver can not close where the task compact [kyuubi]
kangzhilv opened a new issue, #7019:
URL: https://github.com/apache/kyuubi/issues/7019
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
when i use k8s deploy spark task, the task driver pod is not compact.
### Affects Version(s)
1.9.3
### Kyuubi Server Log Output
```logtalk
```
### Kyuubi Engine Log Output
```logtalk
25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is initialized.
25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is started.
25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:06 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:08 ERROR SparkTBinaryFrontendService: Error fetching results:
25/04/09 03:00:08 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [09f53cfd-2220-4e1e-8dd7-6b61dc8b2f52]
25/04/09 03:00:08 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [09f53cfd-2220-4e1e-8dd7-6b61dc8b2f52]
25/04/09 03:00:10 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [0b434ecc-111a-44f5-b9d5-9ad29f67ff83]
25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [0b434ecc-111a-44f5-b9d5-9ad29f67ff83]
25/04/09 03:00:11 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [ab9e6ae9-0fef-4210-b432-286a9bf4885e]
25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [ab9e6ae9-0fef-4210-b432-286a9bf4885e]
25/04/09 03:00:12 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [84a5120e-4238-4a8b-9b4b-20720df1eabc]
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [84a5120e-4238-4a8b-9b4b-20720df1eabc]
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [773ad9ba-687c-4004-a64a-0b0b7ef5089f]
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [773ad9ba-687c-4004-a64a-0b0b7ef5089f]
25/04/09 03:00:13 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V7
25/04/09 03:04:16 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
25/04/09 03:04:16 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
25/04/09 03:04:16 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is stopped.
25/04/09 03:04:16 INFO SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped
25/04/09 03:08:19 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [7041cd34-ca41-4593-bbad-7979c9a071da]
25/04/09 03:08:19 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [7041cd34-ca41-4593-bbad-7979c9a071da]
```
### Kyuubi Server Configurations
```yaml
```
### Kyuubi Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [x] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Spark Driver can not close where the task compact [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7019:
URL: https://github.com/apache/kyuubi/issues/7019#issuecomment-2796370554
Hello @kangzhilv,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Spark Driver cannot be shutdown at the end of a task [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #7019:
URL: https://github.com/apache/kyuubi/issues/7019#issuecomment-2822906010
Unlike Spark on YARN, in Spark on K8s cluster mode, daemon threads would block your driver JVM exit, see technical details at https://github.com/apache/spark/pull/46889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Spark Driver cannot be shutdown at the end of a task [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #7019:
URL: https://github.com/apache/kyuubi/issues/7019#issuecomment-2808692403
Can you provide the jstack info to reveal if there is any daemon thread that prevents JVM from exiting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Spark Driver can not close where the task compact [kyuubi]
Posted by "kangzhilv (via GitHub)" <gi...@apache.org>.
kangzhilv commented on issue #7019:
URL: https://github.com/apache/kyuubi/issues/7019#issuecomment-2796373986
net information
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 172.26.13.100:52060 172.26.255.198:9083 ESTABLISHED -
tcp6 0 0 172.26.13.100:34905 172.26.255.175:8216 ESTABLISHED -
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org