You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "huangzhir (via GitHub)" <gi...@apache.org> on 2023/05/17 08:08:55 UTC
[GitHub] [kyuubi] huangzhir opened a new issue, #4847: [Bug] kyuubi session leak caused by engine stop
huangzhir opened a new issue, #4847:
URL: https://github.com/apache/kyuubi/issues/4847
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
While using the Kyuubi REST functionality, I encountered the following issues:
I created a session and submitted a statement. However, this statement had an issue that caused the Spark engine to crash. For example, I submitted a spark.stop Scala code snippet.
POST http://127.0.0.1:10099/api/v1/sessions/5536200f-cfbc-44ff-a5fd-05a34c9892ab/operations/statement
```
{
"statement": "spark.close",
"runAsync": true,
"confOverlay": {
"kyuubi.operation.language":"SCALA"
}
}
```
Here, because the engine crashed, the Spark session is closed. However, Kyuubi still maintains this session. When using the 'list session' and 'session count' interfaces, I can see that the session is still present.Even after half an hour, the situation remains the same.
GET http://127.0.0.1:10099/api/v1/sessions
return
```
[
{
"identifier": "5536200f-cfbc-44ff-a5fd-05a34c9892ab",
"user": "anonymous",
"ipAddr": "127.0.0.1",
"conf": {
"kyuubi.session.connection.url": "0.0.0.0:10099",
"kyuubi.server.ipAddress": "0.0.0.0",
"kyuubi.session.real.user": "anonymous",
"kyuubi.client.ipAddress": "127.0.0.1"
},
"createTime": 1684309480586,
"duration": 39452,
"idleTime": 0,
"exception": "",
"sessionType": "INTERACTIVE",
"kyuubiInstance": "0.0.0.0:10099",
"engineId": "local-1684309482569"
}
]
```
### Affects Version(s)
1.6,1.7,master
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2088768032
How about ignore the alive probe sessions in terminating checker? @tgravescs
https://github.com/apache/kyuubi/pull/6355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "HaoYang670 (via GitHub)" <gi...@apache.org>.
HaoYang670 commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2114284918
cc @tgravescs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2077195099
I'll see if I can test on the latest release as well maybe we missed something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "HaoYang670 (via GitHub)" <gi...@apache.org>.
HaoYang670 commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2114284381
@turboFei unfortunately your PR doesn't work on my local test. I find the _lastAccessTime and _lastIdleTime are updated periodically when engine alive probe is enabled, which leads to session can never be idel timeout.
I am afraid the bug is introduced by https://github.com/apache/kyuubi/pull/4848 (cc @huangzhir ) in which the function `checkEngineConnectionAlive` calls the `getInfo` function which internally updates the _lastAccessTime and _lastIdleTime. https://github.com/apache/kyuubi/blob/02e1aa30636f3c5bea36e54d90986935fcafa2e3/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala#L330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] huangzhir commented on issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "huangzhir (via GitHub)" <gi...@apache.org>.
huangzhir commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1566750089
I examined the code of Livy and found that it also has a session manager that inherits from SessionHeartbeatWatchdog. It has similar functionality, but the heartbeatTimeout parameter is controlled by the client. The heartbeat is not triggered periodically, but rather whenever a /sessions/:id REST call is made.
https://github.com/cloudera/livy/blob/9ae24d08738652ba5fd817780711d01b110d74a9/server/src/main/scala/com/cloudera/livy/server/interactive/SessionHeartbeat.scala#LL87C16-L87C16
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2169116939
> @turboFei unfortunately your PR doesn't work on my local test. I find the _lastAccessTime and _lastIdleTime are updated periodically when engine alive probe is enabled, which leads to session can never be idel timeout.
>
> I am afraid the bug is introduced by #4848 (cc @huangzhir ) in which the function `checkEngineConnectionAlive` calls the `getInfo` function which internally updates the _lastAccessTime and _lastIdleTime.
>
> https://github.com/apache/kyuubi/blob/02e1aa30636f3c5bea36e54d90986935fcafa2e3/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala#L330
Make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2072829821
> This means that the session never idle timeouts
KyuubiServer establish a new session dedicated for alive probe check, so I think it will not impact the original session idle timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] huangzhir commented on issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "huangzhir (via GitHub)" <gi...@apache.org>.
huangzhir commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1566734603
> What's the behavior of HiveServer2?
Good idea. I looked into the code of HiveServer2, and I found that the sessionManager does not handle whether the session is actually alive.If HiveServer2 is using the Tez engine, it will ultimately reference a TezClient (tezSession). Inside the TezClient, there is a heartbeat check thread that regularly checks if the heartbeat is normal. If a session is not alive, it only prints error logs without performing any other handling.
https://github.com/apache/tez/blame/ccad71dabceb850b58ff38e77fd206e68f62ce4b/tez-api/src/main/java/org/apache/tez/client/TezClient.java#L538
When executing SQL tasks, if TezClient encounters a SessionNotRunning exception, it will create a new tezclient and submit job .
https://github.com/apache/hive/blob/7bfc54ffc8baf5f3e7de326b750fe9355b11132b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java#L633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] huangzhir commented on issue #4847: [Bug] close the session immediately when engine corrupt
Posted by "huangzhir (via GitHub)" <gi...@apache.org>.
huangzhir commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1576085951
> OK, that makes sense then.
>
> Could you change the title to something like "close the session immediately when engine corrupt"? Because currently, the session will finally closed by idle session checker, it not a real leak IMO.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2072406530
I know this is an old issue but I ran into an issue with this. It seems like this functionality conflicts with the normal kyuubi.session.idle.timeout behavior. Was that intended? The behavior I"m seeing is that with
```
kyuubi.session.engine.alive.timeout: "PT10S"
kyuubi.session.engine.alive.probe.enabled: true
```
That causes the session to stay active because every 10 seconds its checking if engine is alive. This means that the session never idle timeouts which then of course also causes the engine to never idle timeout. It seems like since this check is internal it shouldn't affect the overall session/engine idle time.
Any thoughts on this or am I perhaps missing something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 closed issue #4847: [Bug] close the session immediately when engine corrupt
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #4847: [Bug] close the session immediately when engine corrupt
URL: https://github.com/apache/kyuubi/issues/4847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] huangzhir commented on issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "huangzhir (via GitHub)" <gi...@apache.org>.
huangzhir commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1566907880
InteractiveSession, Livy Server to Spark Driver
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1566768919
```
Client => Kyuubi Server => Kyuubi Engine
```
Kyuubi definitely has a configuration for removing idle sessions `kyuubi.session.idle.timeout`, it covers the case that client forgets to close session, but do nothing when the engine is broken.
The Hive case seems not suitable to Kyuubi, TEZ cluster is a long running daemon cluster, but Kyuubi engine is an application running on YARN, I don't think we should re-submit an new application if the existing one is broken. The Livy case you mentioned is not clear to me, what is the heartbeatTimeout represent? Client to Livy Server, or Livy Server to Spark Driver?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1552953833
Thanks for digging into this issue, I would not say the current behavior is wrong, because it returns the error state and message to the client, and the client has the responsibility to close the session, no matter if it's healthy or corrupt.
What's the behavior of HiveServer2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "tgravescs (via GitHub)" <gi...@apache.org>.
tgravescs commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2072945805
so I explicitly see this behavior that the session/engine never timeout. If I turn off kyuubi.session.engine.alive.probe.enabled then it idle times out. Unless there is some fix that was made that we don't have because we aren't on the latest code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] close the session immediately when engine corrupt [kyuubi]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-2076016630
thanks, I am oncall this week, will look into this issue later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1566934988
OK, that makes sense then.
Could you change the title to something like "close the session immediately when engine corrupt"? Because currently, the session will finally closed by idle session checker, it not a real leak IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1566876943
> Client to Livy Server, or Livy Server to Spark Driver?
You cut the question off, the unclear thing is the latter, which two roles' heartbeat of the above cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] huangzhir closed issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "huangzhir (via GitHub)" <gi...@apache.org>.
huangzhir closed issue #4847: [Bug] kyuubi session leak caused by engine stop
URL: https://github.com/apache/kyuubi/issues/4847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] huangzhir commented on issue #4847: [Bug] kyuubi session leak caused by engine stop
Posted by "huangzhir (via GitHub)" <gi...@apache.org>.
huangzhir commented on issue #4847:
URL: https://github.com/apache/kyuubi/issues/4847#issuecomment-1566865154
> The Livy case you mentioned is not clear to me, what is the heartbeatTimeout represent?
heartbeatTimeoutInSecond from CreateInteractiveRequest ,The request parameter "heartbeatTimeoutInSecond" has a default value of 0. If it is set to a value greater than 0, it enables heartbeat timeout checks during runtime.
https://github.com/apache/incubator-livy/blob/c3dd6457c45662de13be72919532ed0ef1b57ab3/server/src/main/scala/org/apache/livy/server/interactive/CreateInteractiveRequest.scala#LL37C7-L37C31
https://github.com/apache/incubator-livy/blob/c3dd6457c45662de13be72919532ed0ef1b57ab3/server/src/main/scala/org/apache/livy/server/interactive/SessionHeartbeat.scala#L38
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org