You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "SGITLOGIN (via GitHub)" <gi...@apache.org> on 2024/11/02 10:12:34 UTC
[I] [Bug] kyuubi webui kill engine failed [kyuubi]
SGITLOGIN opened a new issue, #6790:
URL: https://github.com/apache/kyuubi/issues/6790
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
hello，I have killed the engine in kyuubi webui, and I have also seen that all sessions and operations have stopped, but in fact the engine has not stopped. Engine Log Output： EngineServiceDiscovery: 1 connection(s) are active, delay shutdown, The engine will not stop until kyuubi.session.engine.idle.timeout.
<img width="1679" alt="image" src="https://github.com/user-attachments/assets/8890f712-0a2a-488f-8cc8-652e911a29d2">
<img width="1337" alt="image" src="https://github.com/user-attachments/assets/a56fc0c9-becc-451e-b562-37e7aed4eb09">
### Affects Version(s)
1.9.2
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
```logtalk
24/11/01 17:54:06 INFO ZookeeperDiscoveryClient: Created a /kyuubi_1.9.2_USER_SPARK_SQL/changguowei/default/serverUri=ali-odp-test-01.huan.tv:46373;version=1.9.2;spark.driver.memory=3g;spark.executor.memory=3g;kyuubi.engine.id=application_1730447465039_0023;kyuubi.engine.url=ali-odp-test-01.huan.tv:44237;refId=6a07379c-4d18-49d7-b113-9a22c113351b;sequence=0000000137 on ZooKeeper for KyuubiServer uri: ali-odp-test-01.huan.tv:46373
24/11/01 17:54:06 INFO EngineServiceDiscovery: Registered EngineServiceDiscovery in namespace /kyuubi_1.9.2_USER_SPARK_SQL/changguowei/default.
24/11/01 17:54:06 INFO EngineServiceDiscovery: Service[EngineServiceDiscovery] is started.
24/11/01 17:54:06 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is started.
24/11/01 17:54:06 INFO SparkSQLEngine: Service[SparkSQLEngine] is started.
24/11/01 17:54:06 INFO SparkTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
24/11/01 17:54:06 WARN SparkTBinaryFrontendService: No matching Hive token found for engine metastore uris thrift://ali-odp-test-01.huan.tv:9083,thrift://ali-odp-test-02.huan.tv:9083
24/11/01 17:54:06 WARN SparkTBinaryFrontendService: Ignore token with earlier issue date: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ha-nn, Ident: (token for changguowei: HDFS_DELEGATION_TOKEN owner=changguowei, renewer=changguowei, realUser=hive/ali-odp-test-01.huan.tv@HUAN.TV, issueDate=1730453394322, maxDate=1731058194322, sequenceNumber=64314, masterKeyId=371)
24/11/01 17:54:06 INFO SparkSQLSessionManager: Opening session for changguowei@........
24/11/01 17:54:06 INFO ServerInfo: Adding filter to /kyuubi: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
24/11/01 17:54:06 INFO ServerInfo: Adding filter to /kyuubi/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
24/11/01 17:54:06 INFO ServerInfo: Adding filter to /kyuubi/session: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
24/11/01 17:54:06 INFO ServerInfo: Adding filter to /kyuubi/session/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
24/11/01 17:54:06 INFO ServerInfo: Adding filter to /kyuubi/stop: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
24/11/01 17:54:06 INFO ServerInfo: Adding filter to /kyuubi/gracefulstop: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
24/11/01 17:54:06 INFO SparkSQLEngine:
Spark application name: kyuubi_USER_SPARK_SQL_changguowei_default_6a07379c-4d18-49d7-b113-9a22c113351b
application ID: application_1730447465039_0023
application tags: KYUUBI,6a07379c-4d18-49d7-b113-9a22c113351b
application web UI: http://ali-odp-test-02.huan.tv:8088/proxy/application_1730447465039_0023,http://ali-odp-test-01.huan.tv:8088/proxy/application_1730447465039_0023
master: yarn
version: 3.4.2.1.2.2.0-130
driver: [cpu: 1, mem: 3g]
executor: [cpu: 1, mem: 3g, maxNum: 10]
Start time: Fri Nov 01 17:53:24 CST 2024
User: changguowei (shared mode: USER)
State: STARTED
24/11/01 17:54:06 INFO SparkSQLSessionManager: changguowei's SparkSessionImpl with SessionHandle [6a07379c-4d18-49d7-b113-9a22c113351b] is opened, current opening sessions 1
24/11/01 17:54:06 WARN SparkTBinaryFrontendService: No matching Hive token found for engine metastore uris thrift://ali-odp-test-01.huan.tv:9083,thrift://ali-odp-test-02.huan.tv:9083
24/11/01 17:54:06 WARN SparkTBinaryFrontendService: Ignore token with earlier issue date: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ha-nn, Ident: (token for changguowei: HDFS_DELEGATION_TOKEN owner=changguowei, renewer=changguowei, realUser=hive/ali-odp-test-01.huan.tv@HUAN.TV, issueDate=1730453394322, maxDate=1731058194322, sequenceNumber=64314, masterKeyId=371)
24/11/01 17:54:07 INFO OperationLog: Creating operation log file /hwdata/kyuubi/apache-kyuubi-1.9.2-bin/work/engine_operation_logs/6a07379c-4d18-49d7-b113-9a22c113351b/2c34f952-6a5b-4ebd-ab78-cc37670ccde6
24/11/01 17:54:07 INFO SetCurrentCatalog: Processing changguowei's query[2c34f952-6a5b-4ebd-ab78-cc37670ccde6]: INITIALIZED_STATE -> RUNNING_STATE, statement:
SetCurrentCatalog
24/11/01 17:54:07 INFO SetCurrentCatalog: Processing changguowei's query[2c34f952-6a5b-4ebd-ab78-cc37670ccde6]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.006 seconds
24/11/01 17:54:07 INFO SetCurrentCatalog: statementId=2c34f952-6a5b-4ebd-ab78-cc37670ccde6, operationRunTime=0 ms, operationCpuTime=0 ms
24/11/01 17:54:07 INFO DAGScheduler: Asked to cancel job group 2c34f952-6a5b-4ebd-ab78-cc37670ccde6
24/11/01 17:54:30 WARN ZookeeperDiscoveryClient: This Kyuubi instance ali-odp-test-01.huan.tv:46373 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
24/11/01 17:54:30 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:54:40 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:54:50 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:54:55 INFO YarnClientSchedulerBackend: Requesting to kill executor(s) 1
24/11/01 17:54:55 INFO YarnClientSchedulerBackend: Actual list of executor(s) to be killed is 1
24/11/01 17:54:55 INFO ExecutorAllocationManager: Executors 1 removed due to idle timeout.
24/11/01 17:54:58 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 1.
24/11/01 17:54:58 INFO DAGScheduler: Executor lost: 1 (epoch 0)
24/11/01 17:54:58 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
24/11/01 17:54:58 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, ali-odp-test-03.huan.tv, 43721, None)
24/11/01 17:54:58 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor
24/11/01 17:54:58 INFO DAGScheduler: Shuffle files lost for executor: 1 (epoch 0)
24/11/01 17:54:58 INFO YarnScheduler: Executor 1 on ali-odp-test-03.huan.tv killed by driver.
24/11/01 17:54:58 INFO ExecutorMonitor: Executor 1 is removed. Remove reason statistics: (gracefully decommissioned: 0, decommision unfinished: 0, driver killed: 1, unexpectedly exited: 0).
24/11/01 17:55:00 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:55:06 INFO SparkSQLSessionManager: Checking sessions timeout, current count: 1
24/11/01 17:55:10 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:55:17 INFO SparkTBinaryFrontendService: Received request of closing SessionHandle [6a07379c-4d18-49d7-b113-9a22c113351b]
24/11/01 17:55:17 INFO SparkSQLSessionManager: changguowei's SparkSessionImpl with SessionHandle [6a07379c-4d18-49d7-b113-9a22c113351b] is closed, current opening sessions 0
24/11/01 17:55:17 INFO SparkSessionImpl: sessionId=6a07379c-4d18-49d7-b113-9a22c113351b, sessionRunTime=0 ms, sessionCpuTime=0 ms
24/11/01 17:55:17 INFO SparkTBinaryFrontendService: Finished closing SessionHandle [6a07379c-4d18-49d7-b113-9a22c113351b]
24/11/01 17:55:20 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:55:30 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:55:40 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:55:50 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:56:00 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:56:06 INFO SparkSQLSessionManager: Checking sessions timeout, current count: 0
24/11/01 17:56:10 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:56:20 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:56:30 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:56:40 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:56:50 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:57:00 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:57:06 INFO SparkSQLSessionManager: Checking sessions timeout, current count: 0
24/11/01 17:57:10 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:57:20 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:57:30 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:57:40 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:57:50 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:58:00 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:58:06 INFO SparkSQLSessionManager: Checking sessions timeout, current count: 0
24/11/01 17:58:10 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:58:20 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:58:30 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:58:40 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:58:48 INFO BlockManagerInfo: Removed broadcast_0_piece0 on ali-odp-test-05.huan.tv:33342 in memory (size: 3.5 KiB, free: 1458.6 MiB)
24/11/01 17:58:48 INFO BlockManagerInfo: Removed broadcast_0_piece0 on ali-odp-test-01.huan.tv:40710 in memory (size: 3.5 KiB, free: 1458.6 MiB)
24/11/01 17:58:50 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:59:00 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:59:06 INFO SparkSQLSessionManager: Checking sessions timeout, current count: 0
24/11/01 17:59:10 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:59:20 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:59:30 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:59:40 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 17:59:50 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 18:00:00 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 18:00:06 INFO SparkSQLSessionManager: Checking sessions timeout, current count: 0
24/11/01 18:00:10 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 18:00:20 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 18:00:30 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 18:00:40 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 18:00:50 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 18:01:00 INFO EngineServiceDiscovery: 1 connection(s) are active, delay shutdown
24/11/01 18:01:06 INFO SparkSQLSessionManager: Checking sessions timeout, current count: 0
24/11/01 18:01:06 INFO SparkSQLSessionManager: Idled for more than 300000 ms, terminating
24/11/01 18:01:06 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
24/11/01 18:01:06 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.
```
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] kyuubi webui kill engine failed [kyuubi]
Posted by "SGITLOGIN (via GitHub)" <gi...@apache.org>.
SGITLOGIN commented on issue #6790:
URL: https://github.com/apache/kyuubi/issues/6790#issuecomment-2454059924
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] kyuubi webui kill engine failed [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei closed issue #6790: [Bug] kyuubi webui kill engine failed
URL: https://github.com/apache/kyuubi/issues/6790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] kyuubi webui kill engine failed [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #6790:
URL: https://github.com/apache/kyuubi/issues/6790#issuecomment-2453796073
This issue seems to be caused by the `activeSessionCount` not being updated which prevents the while loop from ending. Would you like to submit a pull request to fix this?
https://github.com/apache/kyuubi/blob/d3520ddbcea96ec55c525600126047c44c7adb35/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/ServiceDiscovery.scala#L70-L73
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] kyuubi webui kill engine failed [kyuubi]
Posted by "SGITLOGIN (via GitHub)" <gi...@apache.org>.
SGITLOGIN commented on issue #6790:
URL: https://github.com/apache/kyuubi/issues/6790#issuecomment-2453878326
> This issue seems to be caused by the `activeSessionCount` not being updated which prevents the while loop from ending. Would you like to submit a pull request to fix this?
>
> https://github.com/apache/kyuubi/blob/d3520ddbcea96ec55c525600126047c44c7adb35/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/ServiceDiscovery.scala#L70-L73
I'm sorry, I don't know how to submit a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] kyuubi webui kill engine failed [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #6790:
URL: https://github.com/apache/kyuubi/issues/6790#issuecomment-2453947511
@SGITLOGIN Thank you for reporting this issue. If you are interested in contributing to Kyuubi you can read https://kyuubi.readthedocs.io/en/master/contributing/doc/get_started.html to learn how to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org