You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "zwangsheng (via GitHub)" <gi...@apache.org> on 2023/08/15 10:28:06 UTC
[GitHub] [kyuubi] zwangsheng opened a new pull request, #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
zwangsheng opened a new pull request, #5168:
URL: https://github.com/apache/kyuubi/pull/5168
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. Print those pod name/prefix in kyuubi server log
2. Make sure driver pod name in length limit
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1294948502
##########
kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala:
##########
@@ -305,6 +305,49 @@ class SparkProcessBuilderSuite extends KerberizedTestHelper with MockitoSugar {
b1.toString.contains(
s"$CONF spark.$KYUUBI_ENGINE_LOG_PATH_KEY=${b1.engineLog.getAbsolutePath}"))
}
+
+ test("[KYUUBI #5165] Test SparkProcessBuilder#appendDriverPodPrefix") {
+ val engineRefId = "kyuubi-test-engine"
+ val appName = "test-app"
+ val b1 = new SparkProcessBuilder("kyuubi", conf, engineRefId)
Review Comment:
it only presents one time, better to use a meaningful name, e.g. `procBuilder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1295662576
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -208,6 +210,42 @@ class SparkProcessBuilder(
kubernetesNamespace())
}
+ def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
+ val map = mutable.Map.newBuilder[String, String]
+ def appendOptionPair(pairOpt: Option[(String, String)]): Unit = {
+ pairOpt match {
+ case Some(pair) => map += (pair._1 -> pair._2)
+ case _ =>
+ }
+ }
+ if (clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
+ appendOptionPair(getExecutorPodPrefix(conf))
+ if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
+ appendOptionPair(getDriverPodPrefix(conf))
+ }
+ }
+ map.result().toMap
+ }
+
+ def getDriverPodPrefix(conf: Map[String, String]): Option[(String, String)] = {
+ conf.get(KUBERNETES_DRIVER_POD_NAME) match {
+ // spark app name always be set
+ case None => Some(
+ KUBERNETES_DRIVER_POD_NAME ->
+ KubernetesUtils.generateDriverPodName(conf(APP_KEY), engineRefId))
+ case _ => None
+ }
+ }
+
+ def getExecutorPodPrefix(conf: Map[String, String]): Option[(String, String)] = {
+ conf.get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX) match {
+ case None => Some(
+ KUBERNETES_EXECUTOR_POD_NAME_PREFIX ->
+ KubernetesUtils.generateExecutorPodNamePrefix(conf(APP_KEY), engineRefId))
+ case _ => None
+ }
+ }
Review Comment:
```suggestion
def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
val map = mutable.Map.newBuilder[String, String]
if (clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
getExecutorPodPrefix(conf).foreach { case (k, v) => map += (k, v) }
if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
getDriverPodPrefix(conf).foreach { case (k, v) => map += (k, v) }
}
}
map.result().toMap
}
def getDriverPodPrefix(conf: Map[String, String]): Option[(String, String)] = {
conf.get(KUBERNETES_DRIVER_POD_NAME) match {
case None => // skip setting if provided
val name = KubernetesUtils.generateDriverPodName(conf(APP_KEY), engineRefId)
Some(KUBERNETES_DRIVER_POD_NAME -> name)
case _ => None
}
}
def getExecutorPodPrefix(conf: Map[String, String]): Option[(String, String)] = {
conf.get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX) match {
case None => // skip setting if provided
val prefix = KubernetesUtils.generateExecutorPodNamePrefix(conf(APP_KEY), engineRefId)
Some(KUBERNETES_EXECUTOR_POD_NAME_PREFIX -> prefix)
case _ => None
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1295319085
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -210,30 +210,30 @@ class SparkProcessBuilder(
}
def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
- var result = Map[String, String]()
+ val map = scala.collection.mutable.Map.newBuilder[String, String]
Review Comment:
`mutable.Map.newBuilder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1294461952
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkBatchProcessBuilder.scala:
##########
@@ -51,9 +51,13 @@ class SparkBatchProcessBuilder(
// tag batch application
KyuubiApplicationManager.tagApplication(batchId, "spark", clusterManager(), batchKyuubiConf)
- (batchKyuubiConf.getAll ++ sparkAppNameConf() ++ engineLogPathConf()).foreach { case (k, v) =>
- buffer += CONF
- buffer += s"${convertConfigKey(k)}=$v"
+ (batchKyuubiConf.getAll ++
+ sparkAppNameConf() ++
+ engineLogPathConf() ++
+ appendPodNameConf(batchConf)).foreach {
+ case (k, v) =>
+ buffer += CONF
+ buffer += s"${convertConfigKey(k)}=$v"
Review Comment:
```suggestion
appendPodNameConf(batchConf)).foreach { case (k, v) =>
buffer += CONF
buffer += s"${convertConfigKey(k)}=$v"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 closed pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
URL: https://github.com/apache/kyuubi/pull/5168
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1294940715
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -208,6 +209,34 @@ class SparkProcessBuilder(
kubernetesNamespace())
}
+ def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
+ var result = Map[String, String]()
+ if (clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
+ result = result ++ appendExecutorPodPrefix(conf)
+ if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
+ result = result ++ appendDriverPodPrefix(conf)
+ }
+ }
+ result
+ }
+
+ def appendDriverPodPrefix(conf: Map[String, String]): Map[String, String] = {
+ conf.get(KUBERNETES_DRIVER_POD_NAME) match {
+ // spark app name always be set
+ case None => Map(KUBERNETES_DRIVER_POD_NAME ->
+ KubernetesUtils.generateDriverPodName(conf(APP_KEY), engineRefId))
+ case _ => Map()
Review Comment:
`Map.empty` should be used to avoid creating unnecessary instance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1294950089
##########
kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala:
##########
@@ -305,6 +305,49 @@ class SparkProcessBuilderSuite extends KerberizedTestHelper with MockitoSugar {
b1.toString.contains(
s"$CONF spark.$KYUUBI_ENGINE_LOG_PATH_KEY=${b1.engineLog.getAbsolutePath}"))
}
+
+ test("[KYUUBI #5165] Test SparkProcessBuilder#appendDriverPodPrefix") {
+ val engineRefId = "kyuubi-test-engine"
+ val appName = "test-app"
+ val b1 = new SparkProcessBuilder("kyuubi", conf, engineRefId)
+ val conf1 = Map(SparkProcessBuilder.APP_KEY -> "test-app")
+ val map1 = b1.appendDriverPodPrefix(conf1)
+ assert(s"kyuubi-$appName-$engineRefId-driver" == map1(
+ SparkProcessBuilder.KUBERNETES_DRIVER_POD_NAME))
+ val conf2 =
+ conf1 ++ Map(SparkProcessBuilder.KUBERNETES_DRIVER_POD_NAME -> "kyuubi-test-1-driver")
+ val map2 = b1.appendDriverPodPrefix(conf2)
+ assert("kyuubi-test-1-driver" == map2(SparkProcessBuilder.KUBERNETES_DRIVER_POD_NAME))
+ val longAppName = "thisisalonglonglonglonglonglonglonglonglonglonglonglong" +
Review Comment:
Please add an app name starting with a Chinese char case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#issuecomment-1680578403
Thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] codecov-commenter commented on pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#issuecomment-1680566848
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5168?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#5168](https://app.codecov.io/gh/apache/kyuubi/pull/5168?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (290a468) into [master](https://app.codecov.io/gh/apache/kyuubi/commit/14d0dab697c314366750b8b80de057aee0155ffd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (14d0dab) will **not change** coverage.
> Report is 6 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #5168 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 566 566
Lines 31654 31699 +45
Branches 4124 4134 +10
======================================
- Misses 31654 31699 +45
```
| [Files Changed](https://app.codecov.io/gh/apache/kyuubi/pull/5168?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...kyuubi/engine/spark/SparkBatchProcessBuilder.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5168?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvc3BhcmsvU3BhcmtCYXRjaFByb2Nlc3NCdWlsZGVyLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...ache/kyuubi/engine/spark/SparkProcessBuilder.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5168?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvc3BhcmsvU3BhcmtQcm9jZXNzQnVpbGRlci5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
| [...scala/org/apache/kyuubi/util/KubernetesUtils.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5168?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS91dGlsL0t1YmVybmV0ZXNVdGlscy5zY2FsYQ==) | `0.00% <0.00%> (ø)` | |
... and [4 files with indirect coverage changes](https://app.codecov.io/gh/apache/kyuubi/pull/5168/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1294940338
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -208,6 +209,34 @@ class SparkProcessBuilder(
kubernetesNamespace())
}
+ def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
+ var result = Map[String, String]()
+ if (clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
+ result = result ++ appendExecutorPodPrefix(conf)
+ if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
+ result = result ++ appendDriverPodPrefix(conf)
+ }
+ }
+ result
Review Comment:
use builder here
```
val result = mutable.Map.newBuilder[String, String]
result += ("1", "A")
result += ("2", "B")
result.result()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1295318942
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -210,30 +210,30 @@ class SparkProcessBuilder(
}
def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
- var result = Map[String, String]()
+ val map = scala.collection.mutable.Map.newBuilder[String, String]
Review Comment:
why use full class name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #5168: [KYUUBI #5165][K8S][SPARK] Build Spark Driver/Executor Pod Name(Prefix) in process
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #5168:
URL: https://github.com/apache/kyuubi/pull/5168#discussion_r1295664564
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -208,6 +210,35 @@ class SparkProcessBuilder(
kubernetesNamespace())
}
+ def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
+ val map = mutable.Map.newBuilder[String, String]
+ if (clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
+ getExecutorPodPrefix(conf).foreach { case (k, v) => map += (k, v) }
+ if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
+ getDriverPodPrefix(conf).foreach { case (k, v) => map += (k, v) }
+ }
+ }
+ map.result().toMap
+ }
+
+ def getDriverPodPrefix(conf: Map[String, String]): Option[(String, String)] = {
+ conf.get(KUBERNETES_DRIVER_POD_NAME) match {
+ case None => // skip setting if provided
+ val name = KubernetesUtils.generateDriverPodName(conf(APP_KEY), engineRefId)
+ Some(KUBERNETES_DRIVER_POD_NAME -> name)
+ case _ => None
+ }
+ }
+
+ def getExecutorPodPrefix(conf: Map[String, String]): Option[(String, String)] = {
+ conf.get(KUBERNETES_EXECUTOR_POD_NAME_PREFIX) match {
+ case None => // skip setting if provided
Review Comment:
```suggestion
case None =>
```
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala:
##########
@@ -208,6 +210,35 @@ class SparkProcessBuilder(
kubernetesNamespace())
}
+ def appendPodNameConf(conf: Map[String, String]): Map[String, String] = {
+ val map = mutable.Map.newBuilder[String, String]
+ if (clusterManager().exists(cm => cm.toLowerCase(Locale.ROOT).startsWith("k8s"))) {
+ getExecutorPodPrefix(conf).foreach { case (k, v) => map += (k, v) }
+ if (deployMode().exists(_.toLowerCase(Locale.ROOT) == "cluster")) {
+ getDriverPodPrefix(conf).foreach { case (k, v) => map += (k, v) }
+ }
+ }
+ map.result().toMap
+ }
+
+ def getDriverPodPrefix(conf: Map[String, String]): Option[(String, String)] = {
+ conf.get(KUBERNETES_DRIVER_POD_NAME) match {
+ case None => // skip setting if provided
Review Comment:
```suggestion
case None =>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org