You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/10/19 09:12:18 UTC
[GitHub] [incubator-kyuubi] zwangsheng opened a new pull request, #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
zwangsheng opened a new pull request, #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Provided one way to build kubernetes client from pod env and service account when kyuubi run in pod.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #3663: [SPARK][K8S] Support auto build Kubernetes client when Kyuubi running in Pod
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#issuecomment-1289883548
refactor only change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1000300435
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala:
##########
@@ -135,6 +128,7 @@ class KubernetesApplicationOperation extends ApplicationOperation with Logging {
object KubernetesApplicationOperation extends Logging {
val LABEL_KYUUBI_UNIQUE_KEY = "kyuubi-unique-tag"
+ val SPARK_APP_SELECTOR = "spark-app-selector"
Review Comment:
please change the var name ti align w/ Spark `org.apache.spark.deploy.k8s.Constants`
```
val SPARK_APP_ID_LABEL = "spark-app-selector"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1000301616
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala:
##########
@@ -33,24 +34,16 @@ class KubernetesApplicationOperation extends ApplicationOperation with Logging {
private var jpsOperation: JpsApplicationOperation = _
override def initialize(conf: KyuubiConf): Unit = {
- info("Start Initialize Kubernetes Client.")
- val contextOpt = conf.get(KUBERNETES_CONTEXT)
- if (contextOpt.isEmpty) {
- warn("Skip Initialize Kubernetes Client, because of Context not set.")
- return
- }
jpsOperation = new JpsApplicationOperation
jpsOperation.initialize(conf)
- kubernetesClient =
- try {
- val client = new DefaultKubernetesClient(Config.autoConfigure(contextOpt.get))
- info(s"Initialized Kubernetes Client connect to: ${client.getMasterUrl}")
- client
- } catch {
- case e: KubernetesClientException =>
- error("Fail to init KubernetesClient for KubernetesApplicationOperation", e)
- null
- }
+
+ info("Start Initialize Kubernetes Client.")
Review Comment:
```suggestion
info("Start initializing Kubernetes Client.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3663: [SPARK][K8S] Support auto build Kubernetes client from env when Kyuubi running in Pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#issuecomment-1290042109
The CI failure caused by known flaky tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1001451352
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -828,6 +828,69 @@ object KyuubiConf {
.version("1.4.0")
.fallbackConf(FRONTEND_WORKER_KEEPALIVE_TIME)
+ val KUBERNETES_CONTEXT: OptionalConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.context")
+ .doc("The desired context from your kubernetes config file used to configure the K8S " +
+ "client for interacting with the cluster.")
+ .version("1.6.0")
+ .stringConf
+ .createOptional
+
+ val KUBERNETES_NAMESPACE: ConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.namespace")
+ .version("1.7.0")
+ .stringConf
+ .createWithDefault("default")
+
+ val KUBERNETES_MASTER: OptionalConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.master.address")
+ .doc("kubernetes master address for build kubernetes client")
+ .version("1.7.0")
+ .stringConf
+ .createOptional
+
+ val KUBERNETES_AUTHENTICATE_OAUTH_TOKEN_FILE: OptionalConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.authenticate.oauthTokenFile")
+ .doc("kubernetes client authenticate oauth token file")
+ .version("1.7.0")
+ .stringConf
+ .createOptional
+
+ val KUBERNETES_AUTHENTICATE_OAUTH_TOKEN: OptionalConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.authenticate.oauthToken")
+ .doc("kubernetes client authenticate oauth token value")
+ .version("1.7.0")
+ .stringConf
+ .createOptional
+
+ val KUBERNETES_AUTHENTICATE_CLIENT_KEY_FILE: OptionalConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.authenticate.clientKeyFile")
+ .doc("kubernetes client authenticate client key file")
+ .version("1.7.0")
+ .stringConf
+ .createOptional
+
+ val KUBERNETES_AUTHENTICATE_CLIENT_CERT_FILE: OptionalConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.authenticate.clientCertFile")
+ .doc("kubernetes client authenticate client cert file")
+ .version("1.7.0")
+ .stringConf
+ .createOptional
+
+ val KUBERNETES_AUTHENTICATE_CA_CERT_FILE: OptionalConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.authenticate.ca cert file")
Review Comment:
fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1002694966
##########
docker/helm/templates/kyuubi-configmap.yaml:
##########
@@ -47,5 +47,6 @@ data:
#
kyuubi.frontend.bind.host={{ .Values.server.bind.host }}
kyuubi.frontend.bind.port={{ .Values.server.bind.port }}
+ kyuubi.kubernetes.namespace= {{ .Release.Namespace }}
# Details in https://kyuubi.apache.org/docs/latest/deployment/settings.html
Review Comment:
nit: eol
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.util
+
+import java.io.File
+
+import com.fasterxml.jackson.databind.ObjectMapper
+import com.google.common.base.Charsets
+import com.google.common.io.Files
+import io.fabric8.kubernetes.client.{Config, ConfigBuilder, DefaultKubernetesClient, KubernetesClient}
+import io.fabric8.kubernetes.client.Config.autoConfigure
+import io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory
+import okhttp3.{Dispatcher, OkHttpClient}
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.config.KyuubiConf._
+
+object KubernetesUtils extends Logging {
+
+ def buildKubernetesClient(conf: KyuubiConf): Option[KubernetesClient] = {
+ val master = conf.get(KUBERNETES_MASTER)
+ val namespace = conf.get(KUBERNETES_NAMESPACE)
+ val serviceAccountToken =
+ Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)).filter(_.exists)
+ val serviceAccountCaCrt =
+ Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)).filter(_.exists)
+
+ val oauthTokenFile = conf.get(KUBERNETES_AUTHENTICATE_OAUTH_TOKEN_FILE)
+ .map(new File(_))
+ .orElse(serviceAccountToken)
+ val oauthTokenValue = conf.get(KUBERNETES_AUTHENTICATE_OAUTH_TOKEN)
+
+ KubernetesUtils.requireNandDefined(
+ oauthTokenFile,
+ oauthTokenValue,
+ s"Cannot specify OAuth token through both a oauth token file and a " +
+ s"oauth token value.")
+
+ val caCertFile = conf
+ .get(KUBERNETES_AUTHENTICATE_CA_CERT_FILE)
+ .orElse(serviceAccountCaCrt.map(_.getAbsolutePath))
+ val clientKeyFile = conf.get(KUBERNETES_AUTHENTICATE_CLIENT_KEY_FILE)
+ val clientCertFile = conf.get(KUBERNETES_AUTHENTICATE_CLIENT_CERT_FILE)
+
+ // Allow for specifying a context used to auto-configure from the users K8S config file
+ val kubeContext = conf.get(KUBERNETES_CONTEXT).filter(_.nonEmpty)
+ info("Auto-configuring K8S client using " +
+ kubeContext.map("context " + _).getOrElse("current context") +
+ " from users K8S config file")
+
+ val config = new ConfigBuilder(autoConfigure(kubeContext.orNull))
+ .withApiVersion("v1")
+ .withOption(master) {
+ (master, configBuilder) => configBuilder.withMasterUrl(master)
+ }.withNamespace(namespace)
+ .withTrustCerts(conf.get(KUBERNETES_TRUST_CERTIFICATES))
+ .withOption(oauthTokenValue) {
+ (token, configBuilder) => configBuilder.withOauthToken(token)
+ }.withOption(oauthTokenFile) {
+ (file, configBuilder) =>
+ configBuilder.withOauthToken(Files.asCharSource(file, Charsets.UTF_8).read())
+ }.withOption(caCertFile) {
+ (file, configBuilder) => configBuilder.withCaCertFile(file)
+ }.withOption(clientKeyFile) {
+ (file, configBuilder) => configBuilder.withClientKeyFile(file)
+ }.withOption(clientCertFile) {
+ (file, configBuilder) => configBuilder.withClientCertFile(file)
+ }.build()
+
+ (master, kubeContext, loadMasterAddressFromEnv) match {
+ case (None, None, Some(url)) =>
+ debug(s"Set kubernetes master address $url from env KUBERNETES_PORT")
+ config.setMasterUrl(url)
Review Comment:
you missed other cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1002895317
##########
docker/helm/templates/kyuubi-configmap.yaml:
##########
@@ -47,5 +47,6 @@ data:
#
kyuubi.frontend.bind.host={{ .Values.server.bind.host }}
kyuubi.frontend.bind.port={{ .Values.server.bind.port }}
+ kyuubi.kubernetes.namespace= {{ .Release.Namespace }}
Review Comment:
nit: remove space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#issuecomment-1285240496
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3663](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e7f9ec0) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/9d0b6909dbd45cc65595a23071a0cac8214df32e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9d0b690) will **decrease** coverage by `0.05%`.
> The diff coverage is `34.00%`.
> :exclamation: Current head e7f9ec0 differs from pull request most recent head a7f0b55. Consider uploading reports for the commit a7f0b55 to get more accurate results
```diff
@@ Coverage Diff @@
## master #3663 +/- ##
============================================
- Coverage 52.02% 51.96% -0.06%
Complexity 13 13
============================================
Files 485 486 +1
Lines 27202 27238 +36
Branches 3793 3795 +2
============================================
+ Hits 14151 14155 +4
- Misses 11685 11712 +27
- Partials 1366 1371 +5
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ain/scala/org/apache/kyuubi/util/ThreadUtils.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS91dGlsL1RocmVhZFV0aWxzLnNjYWxh) | `86.11% <0.00%> (-5.07%)` | :arrow_down: |
| [...scala/org/apache/kyuubi/util/KubernetesUtils.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS91dGlsL0t1YmVybmV0ZXNVdGlscy5zY2FsYQ==) | `12.50% <12.50%> (ø)` | |
| [...kyuubi/engine/KubernetesApplicationOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvS3ViZXJuZXRlc0FwcGxpY2F0aW9uT3BlcmF0aW9uLnNjYWxh) | `18.03% <57.14%> (+5.53%)` | :arrow_up: |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `97.35% <100.00%> (+0.01%)` | :arrow_up: |
| [...ache/kyuubi/operation/KyuubiOperationManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uTWFuYWdlci5zY2FsYQ==) | `79.45% <0.00%> (-2.74%)` | :arrow_down: |
| [...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWN0bC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jdGwvY21kL2xvZy9Mb2dCYXRjaENvbW1hbmQuc2NhbGE=) | `80.00% <0.00%> (-2.00%)` | :arrow_down: |
| [...mon/src/main/scala/org/apache/kyuubi/Logging.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9Mb2dnaW5nLnNjYWxh) | `52.63% <0.00%> (-1.32%)` | :arrow_down: |
| [...ain/scala/org/apache/kyuubi/engine/EngineRef.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvRW5naW5lUmVmLnNjYWxh) | `72.26% <0.00%> (-0.85%)` | :arrow_down: |
| [...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | `89.27% <0.00%> (-0.70%)` | :arrow_down: |
| [...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvUHJvY0J1aWxkZXIuc2NhbGE=) | `80.74% <0.00%> (-0.63%)` | :arrow_down: |
| ... and [4 more](https://codecov.io/gh/apache/incubator-kyuubi/pull/3663/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed pull request #3663: [SPARK][K8S] Support auto build Kubernetes client from env when Kyuubi running in Pod
Posted by GitBox <gi...@apache.org>.
pan3793 closed pull request #3663: [SPARK][K8S] Support auto build Kubernetes client from env when Kyuubi running in Pod
URL: https://github.com/apache/incubator-kyuubi/pull/3663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1001453636
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.util
+
+import java.io.File
+
+import com.fasterxml.jackson.databind.ObjectMapper
+import com.google.common.base.Charsets
+import com.google.common.io.Files
+import io.fabric8.kubernetes.client.{Config, ConfigBuilder, DefaultKubernetesClient, KubernetesClient}
+import io.fabric8.kubernetes.client.Config.autoConfigure
+import io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory
+import okhttp3.{Dispatcher, OkHttpClient}
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.config.KyuubiConf.{KUBERNETES_AUTHENTICATE_CA_CERT_FILE, KUBERNETES_AUTHENTICATE_CLIENT_CERT_FILE, KUBERNETES_AUTHENTICATE_CLIENT_KEY_FILE, KUBERNETES_AUTHENTICATE_OAUTH_TOKEN, KUBERNETES_AUTHENTICATE_OAUTH_TOKEN_FILE, KUBERNETES_CONTEXT, KUBERNETES_MASTER, KUBERNETES_NAMESPACE, KUBERNETES_TRUST_CERTIFICATES}
+
+object KubernetesUtils extends Logging {
+
+ def buildKubernetesClient(conf: KyuubiConf): KubernetesClient = {
Review Comment:
how about returning Option instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1001452986
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.util
+
+import java.io.File
+
+import com.fasterxml.jackson.databind.ObjectMapper
+import com.google.common.base.Charsets
+import com.google.common.io.Files
+import io.fabric8.kubernetes.client.{Config, ConfigBuilder, DefaultKubernetesClient, KubernetesClient}
+import io.fabric8.kubernetes.client.Config.autoConfigure
+import io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory
+import okhttp3.{Dispatcher, OkHttpClient}
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.config.KyuubiConf.{KUBERNETES_AUTHENTICATE_CA_CERT_FILE, KUBERNETES_AUTHENTICATE_CLIENT_CERT_FILE, KUBERNETES_AUTHENTICATE_CLIENT_KEY_FILE, KUBERNETES_AUTHENTICATE_OAUTH_TOKEN, KUBERNETES_AUTHENTICATE_OAUTH_TOKEN_FILE, KUBERNETES_CONTEXT, KUBERNETES_MASTER, KUBERNETES_NAMESPACE, KUBERNETES_TRUST_CERTIFICATES}
Review Comment:
```suggestion
import org.apache.kyuubi.config.KyuubiConf._
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3663: [SPARK][K8S] Support auto build Kubernetes client from env when Kyuubi running in Pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#issuecomment-1290041821
The PR description is updated and should cover the changes in the PR.
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1000300435
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala:
##########
@@ -135,6 +128,7 @@ class KubernetesApplicationOperation extends ApplicationOperation with Logging {
object KubernetesApplicationOperation extends Logging {
val LABEL_KYUUBI_UNIQUE_KEY = "kyuubi-unique-tag"
+ val SPARK_APP_SELECTOR = "spark-app-selector"
Review Comment:
please change the var name to align w/ Spark `org.apache.spark.deploy.k8s.Constants`
```
val SPARK_APP_ID_LABEL = "spark-app-selector"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3663: [SPARK][K8S] Kyuubi Kubernetes Application Operator add build kuebrnetes client from pod env when kyuubi in pod
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#discussion_r1001451611
##########
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:
##########
@@ -828,6 +828,69 @@ object KyuubiConf {
.version("1.4.0")
.fallbackConf(FRONTEND_WORKER_KEEPALIVE_TIME)
+ val KUBERNETES_CONTEXT: OptionalConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.context")
+ .doc("The desired context from your kubernetes config file used to configure the K8S " +
+ "client for interacting with the cluster.")
+ .version("1.6.0")
+ .stringConf
+ .createOptional
+
+ val KUBERNETES_NAMESPACE: ConfigEntry[String] =
+ buildConf("kyuubi.kubernetes.namespace")
+ .version("1.7.0")
Review Comment:
doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #3663: [SPARK][K8S] Add KubernetesUtils help build kubernetes client
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#issuecomment-1288909153
I try my best to understand what you are trying to say about this PR in the PR desc, but failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] zwangsheng commented on pull request #3663: [SPARK][K8S] Add KubernetesUtils help build kubernetes client
Posted by GitBox <gi...@apache.org>.
zwangsheng commented on PR #3663:
URL: https://github.com/apache/incubator-kyuubi/pull/3663#issuecomment-1288913490
> I try my best to understand what you are trying to say about this PR in the PR desc, but failed
This pr aims to add util object `KubernetesUtils` help kyuubi build kubernetes client in spark way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org