You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "A-little-bit-of-data (via GitHub)" <gi...@apache.org> on 2024/12/20 02:23:58 UTC
[I] [Bug] There are too many Spark sessions and the session timeout parameter is not effective [kyuubi]
A-little-bit-of-data opened a new issue, #6855:
URL: https://github.com/apache/kyuubi/issues/6855
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
When using kyuubi to start a spark SQL ON K8S cluster, the number of internal sessions increases. The parameter `kyuubi.engine.user.isolated.spark.session.idle.timeout=PT30M` is found in the documentation. The default value is 6H. After changing it to 30M, there are still sessions that exceed 30M and are not closed, and the SQL of the session has been executed. I would like to ask whether the creation of this session is based on a SQL or how it is created. I found that there are not so many SQLs in my cluster, but there are many sessions. I don't know how they are created.
### Affects Version(s)
1.9.1
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] There are too many Spark sessions and the session timeout parameter is not effective [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6855:
URL: https://github.com/apache/kyuubi/issues/6855#issuecomment-2558819422
you should read the docs carefully, the docs of `kyuubi.engine.user.isolated.spark.session.idle.timeout` say
> If kyuubi.engine.user.isolated.spark.session is false ...
and the default value of `kyuubi.engine.user.isolated.spark.session` is `true`, further, the docs of `kyuubi.engine.user.isolated.spark.session` say
> ... if the engine is running in a group or server share level ... Note that, it does not affect if the share level is connection or user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] There are too many Spark sessions and the session timeout parameter is not effective [kyuubi]
Posted by "shuyouZZ (via GitHub)" <gi...@apache.org>.
shuyouZZ commented on issue #6855:
URL: https://github.com/apache/kyuubi/issues/6855#issuecomment-2601882256
我们支持reset idle timeout吗？好像只在执行某些操作时会刷新lastAccessTime和lastIdleTime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] There are too many Spark sessions and the session timeout parameter is not effective [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6855:
URL: https://github.com/apache/kyuubi/issues/6855#issuecomment-2597813541
@holiday-zj no, yet. it's a server-side configuration that the client can not override, but the behavior can be discussed, in a quick thought, we can make changes to support that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] There are too many Spark sessions and the session timeout parameter is not effective [kyuubi]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #6855:
URL: https://github.com/apache/kyuubi/issues/6855#issuecomment-2559670927
> you should read the docs carefully, the docs of `kyuubi.engine.user.isolated.spark.session.idle.timeout` say
>
> > If kyuubi.engine.user.isolated.spark.session is false ...
>
> and the default value of `kyuubi.engine.user.isolated.spark.session` is `true`, further, the docs of `kyuubi.engine.user.isolated.spark.session` say和默认值 `kyuubi.engine.user.isolated.spark.session` 是`true` ，此外，文档 `kyuubi.engine.user.isolated.spark.session` 说
>
> > ... if the engine is running in a group or server share level ... Note that, it does not affect if the share level is connection or user.
>
> while the screenshot shows you are using USER share level
Thank you very much. Can I understand it this way? When using the user sharing level, `kyuubi.session.idle.timeout` is used to control the session timeout. When using the group or service sharing level, `kyuubi.engine.user.isolated.spark.session.idle.timeout` is used to control the session timeout. Whether it is spark, flink, or trino engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] There are too many Spark sessions and the session timeout parameter is not effective [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6855:
URL: https://github.com/apache/kyuubi/issues/6855#issuecomment-2558820361
for your cases, maybe you should change `kyuubi.session.idle.timeout`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] There are too many Spark sessions and the session timeout parameter is not effective [kyuubi]
Posted by "holiday-zj (via GitHub)" <gi...@apache.org>.
holiday-zj commented on issue #6855:
URL: https://github.com/apache/kyuubi/issues/6855#issuecomment-2597773942
kyuubi.session.idle.timeout可以放到创建session的rest请求中，针对某个特定的session设置吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org