You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "ruanwenjun (via GitHub)" <gi...@apache.org> on 2026/01/19 05:47:23 UTC
[I] [Bug] Kyuubi SparkSQLEngine hang in stopping [kyuubi]
ruanwenjun opened a new issue, #7309:
URL: https://github.com/apache/kyuubi/issues/7309
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
We use kyuubi to submit spark sql on k8s.
After the sql finished, the SparkSqlEngine pod doesn't deleted.
I find the below log
```log
26/01/07 06:40:14 INFO SparkSQLSessionManager: Session stopped due to shared level is Connection.
26/01/07 06:40:14 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
26/01/07 06:40:14 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
```
Seems the `EngineServiceDiscovery` doesn't stopped.
### Affects Version(s)
1.9.1
### Kyuubi Server Log Output
```logtalk
```
### Kyuubi Engine Log Output
```logtalk
26/01/07 06:40:14 INFO SparkSQLSessionManager: Session stopped due to shared level is Connection.
26/01/07 06:40:14 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
26/01/07 06:40:14 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
```
### Kyuubi Server Configurations
```yaml
```
### Kyuubi Engine Configurations
```yaml
```
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3776509482
by default, kyuubi uses a shaded zk 3.4 client which is very very old, if it's related to zk, does it have changes in new zk versions? if so, we can port such changes to our shaded zk 3.4.
(unfortunately, we still have some customers that uses zk server 3.4, so we'd like to keep using the old zk 3.4 client as much as possible to gain best compatibility)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3782184670
@cxzl25 Sorry, the send thread has exist due to OOM, OOM may be a common issue within the driver.
```
26/01/06 21:35:14 WARN ZooKeeperThread: Exception occurred from thread main-SendThread(172.28.16.195:2181)
java.lang.OutOfMemoryError: Java heap space
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3778152635
@ruanwenjun, you are allowed to build with zk-3.6 by passing `-Pzookeeper-3.6`, but as I said, I still prefer to keep using zk-3.4 by default (I apologize for being so extremely conservative here), if you like, you can follow the below PRs to port ZOOKEEPER-2251 to kyuubi's shaded zk-3.4
- https://github.com/apache/kyuubi-shaded/pull/46
- https://github.com/apache/kyuubi-shaded/pull/47
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3778022989
@cxzl25 No OOM here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3770699081
Seems this is zk bug, I submit jira: https://issues.apache.org/jira/browse/ZOOKEEPER-5016
Maybe we can use an asynchronous thread to stop the `EngineServiceDiscovery` and set the maximum wait time to avoid the engine hang.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3782404843
@pan3793 Or maybe we should directly exit when `KyuubiUncaughtExceptionHandler` catch an `OutOfMemoryError` to avoid missing this parameter when starting the engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3782365233
@pan3793 Would it make sense to add
```
-XX: ExitOnOutOfMemoryError
```
on the Engine jvm args? If OOM then the engine can exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3782207901
for oom cases, it's more like related to ZOOKEEPER-1100 and ZOOKEEPER-1375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3778011936
Judging from the call stack you provided, the `main-SendThread` seems to be missing. This may prevent the `packet` from `ClientCnxn.submitRequest` from being processed, ultimately resulting in a stuck wait state.
Is there thread-level OOM?
```java
main-SendThread
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.kyuubi.shaded.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun closed issue #7309: [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery
URL: https://github.com/apache/kyuubi/issues/7309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3782353867
@pan3793 The `OutOfMemoryError` has not been catched.
The error might happen at the catch code block
```
} catch (Throwable e) {
if (ClientCnxn.this.closing) {
if (ClientCnxn.LOG.isDebugEnabled()) {
ClientCnxn.LOG.debug("An exception was thrown while closing send thread for session 0x" + Long.toHexString(ClientCnxn.this.getSessionId()) + " : " + e.getMessage());
}
break;
}
if (e instanceof SessionExpiredException) {
ClientCnxn.LOG.info(e.getMessage() + ", closing socket connection");
} else if (e instanceof SessionTimeoutException) {
ClientCnxn.LOG.info(e.getMessage() + ", closing socket connection and attempting reconnect");
} else if (e instanceof EndOfStreamException) {
ClientCnxn.LOG.info(e.getMessage() + ", closing socket connection and attempting reconnect");
} else if (e instanceof RWServerFoundException) {
ClientCnxn.LOG.info(e.getMessage());
} else if (e instanceof SocketException) {
ClientCnxn.LOG.info("Socket error occurred: {}: {}", serverAddress, e.getMessage());
} else {
ClientCnxn.LOG.warn("Session 0x{} for server {}, unexpected error{}", new Object[]{Long.toHexString(ClientCnxn.this.getSessionId()), serverAddress, ", closing socket connection and attempting reconnect", e});
}
this.cleanup();
if (ClientCnxn.this.state.isAlive()) {
ClientCnxn.this.eventThread.queueEvent(new WatchedEvent(EventType.None, KeeperState.Disconnected, (String)null));
}
this.clientCnxnSocket.updateNow();
this.clientCnxnSocket.updateLastSendAndHeard();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun closed issue #7309: [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery
URL: https://github.com/apache/kyuubi/issues/7309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3778419734
@pan3793 Ok, got it. I can upgrade to zookeeper-3.6 on our internal version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3782371071
yes, we generally recommend adding `-XX:ExitOnOutOfMemoryError` to `spark.driver.extraJavaOptions`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3784121534
> Or maybe we should directly exit when `KyuubiUncaughtExceptionHandler` catch an `OutOfMemoryError`, ensuring that ExitOnOutOfMemoryError is not bypassed during engine startup
In the case of OOM, there is no guarantee that the exception will be handled correctly. I strongly recommend using [jvmquake](https://github.com/Netflix-Skunkworks/jvmquake) to avoid this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3776496404
cc @cxzl25
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3784618436
@cxzl25 Thanks, I will take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3776520274
This is a bit strange, can you provide the JDK version used and whether jemalloc is used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3777687635
@cxzl25 I use
```
openjdk version "1.8.0_322"
```
I don't use `jemalloc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #7309:
URL: https://github.com/apache/kyuubi/issues/7309#issuecomment-3777900904
@pan3793 Thanks for the key information, I find zk has solved this at the latest code
jira: ZOOKEEPER-2251
PR: https://github.com/apache/zookeeper/pull/119
but has not fixed at 3.4.x 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery [kyuubi]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun closed issue #7309: [Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery
URL: https://github.com/apache/kyuubi/issues/7309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org