You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "liaoyt (via GitHub)" <gi...@apache.org> on 2023/11/19 13:57:49 UTC
[PR] [K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
liaoyt opened a new pull request, #5728:
URL: https://github.com/apache/kyuubi/pull/5728
# :mag: Description
## Issue References ๐
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #
## Describe Your Solution ๐ง
A new feature introduced from #5555 supports kyuubi to clean up spark driver pods automatically, but all pod would be clean up without considering app's terminated state.
This PR make user can chose which pods should be delete by setting up a cleanup strategy.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan ๐งช
Test locally.
---
# Checklists
## ๐ Author Self Checklist
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## ๐ Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
Posted by "liaoyt (via GitHub)" <gi...@apache.org>.
liaoyt commented on PR #5728:
URL: https://github.com/apache/kyuubi/pull/5728#issuecomment-1818287219
> @liaoyt thanks for catching this, you may refer the wrong ticket in PR description?
oh yes, should refer to #5714, allready fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5728:
URL: https://github.com/apache/kyuubi/pull/5728#issuecomment-1818205007
@liaoyt thanks for catching this, you may refer the wrong ticket in PR description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5728:
URL: https://github.com/apache/kyuubi/pull/5728#discussion_r1398648206
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -1231,13 +1231,20 @@ object KyuubiConf {
.checkValue(_ > 0, "must be positive number")
.createWithDefault(Duration.ofMinutes(5).toMillis)
- val KUBERNETES_SPARK_DELETE_DRIVER_POD_ON_TERMINATION_ENABLED: ConfigEntry[Boolean] =
- buildConf("kyuubi.kubernetes.spark.deleteDriverPodOnTermination.enabled")
- .doc("If set to true then Kyuubi server will delete the spark driver pod after " +
- s"the application terminates for ${KUBERNETES_TERMINATED_APPLICATION_RETAIN_PERIOD.key}.")
+ val KUBERNETES_SPARK_DELETE_DRIVER_POD_ON_TERMINATION: ConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.spark.deleteDriverPodOnTermination")
Review Comment:
let's rename the configuration name to `kyuubi.kubernetes.spark.cleanupTerminatedDriverPod`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5731][K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #5728: [KYUUBI #5731][K8S] Support to cleanup the spark driver pod with specified cleanup strategy
URL: https://github.com/apache/kyuubi/pull/5728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5731][K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5728:
URL: https://github.com/apache/kyuubi/pull/5728#issuecomment-1818438114
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5728?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `5 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`3478fc9`)](https://app.codecov.io/gh/apache/kyuubi/commit/3478fc9dfb4716dd4a808506f3dab26155eaba91?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.37% compared to head [(`d2cc8cb`)](https://app.codecov.io/gh/apache/kyuubi/pull/5728?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.36%.
> Report is 5 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/kyuubi/pull/5728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...kyuubi/engine/KubernetesApplicationOperation.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvS3ViZXJuZXRlc0FwcGxpY2F0aW9uT3BlcmF0aW9uLnNjYWxh) | 28.57% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/5728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #5728 +/- ##
============================================
- Coverage 61.37% 61.36% -0.01%
Complexity 23 23
============================================
Files 607 607
Lines 35820 35845 +25
Branches 4914 4920 +6
============================================
+ Hits 21983 21995 +12
- Misses 11456 11461 +5
- Partials 2381 2389 +8
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/5728?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
Posted by "liaoyt (via GitHub)" <gi...@apache.org>.
liaoyt commented on code in PR #5728:
URL: https://github.com/apache/kyuubi/pull/5728#discussion_r1398698477
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -1231,13 +1231,20 @@ object KyuubiConf {
.checkValue(_ > 0, "must be positive number")
.createWithDefault(Duration.ofMinutes(5).toMillis)
- val KUBERNETES_SPARK_DELETE_DRIVER_POD_ON_TERMINATION_ENABLED: ConfigEntry[Boolean] =
- buildConf("kyuubi.kubernetes.spark.deleteDriverPodOnTermination.enabled")
- .doc("If set to true then Kyuubi server will delete the spark driver pod after " +
- s"the application terminates for ${KUBERNETES_TERMINATED_APPLICATION_RETAIN_PERIOD.key}.")
+ val KUBERNETES_SPARK_DELETE_DRIVER_POD_ON_TERMINATION: ConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.spark.deleteDriverPodOnTermination")
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5731][K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5728:
URL: https://github.com/apache/kyuubi/pull/5728#issuecomment-1818492807
Thanks, merged to master/1.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [K8S] Support to cleanup the spark driver pod with specified cleanup strategy [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5728:
URL: https://github.com/apache/kyuubi/pull/5728#issuecomment-1818312529
@liaoyt please regenerate the docs, others LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org