You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/02/22 10:32:32 UTC
[GitHub] [incubator-kyuubi] yaooqinn opened a new issue #1960: [Improvement] Skip creating sparksession and starting engine if the max initialization time exceeds
yaooqinn opened a new issue #1960:
URL: https://github.com/apache/incubator-kyuubi/issues/1960
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
When a resource manager such as YARN is in high traffic, many engines will be pending and fail to get initialized in time. But sometimes, the pending engine is still waiting for YARN to accept it and run it. In the meanwhile, it already becomes orphaned and may get a re-submit at the server-side. This may become an endless loop.
We should avoid these kinds of engines be running.
### How should we improve?
One way to fix this is to add a timestamp that indicates the submit time of an engine, then when we actually before we actually create the SparkSession instance, we check whether the `current time - submit time` already exceeds the maximum engine initialization time. If true, we skip; otherwise, we go the current way
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1960: [Improvement] Skip creating sparksession and starting engine if the max initialization time exceeds
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1960:
URL: https://github.com/apache/incubator-kyuubi/issues/1960#issuecomment-1047678192
Assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1960: [Improvement] Skip creating sparksession and starting engine if the max initialization time exceeds
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1960:
URL: https://github.com/apache/incubator-kyuubi/issues/1960#issuecomment-1047665052
Sounds good, in this way, the orphaned engine will exit immediately without waiting for idle timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1960: [Improvement] Skip creating sparksession and starting engine if the max initialization time exceeds
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1960:
URL: https://github.com/apache/incubator-kyuubi/issues/1960#issuecomment-1047673597
@yaooqinn Could you please assign this to me? I want to try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1960: [Improvement] Skip creating sparksession and starting engine if the max initialization time exceeds
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1960:
URL: https://github.com/apache/incubator-kyuubi/issues/1960#issuecomment-1047652706
cc @wForget @turboFei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed issue #1960: [Improvement] Skip creating sparksession and starting engine if the max initialization time exceeds
Posted by GitBox <gi...@apache.org>.
ulysses-you closed issue #1960:
URL: https://github.com/apache/incubator-kyuubi/issues/1960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org