You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org> on 2026/03/12 05:25:53 UTC
[PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
LamiumAmplexicaule opened a new pull request, #7349:
URL: https://github.com/apache/kyuubi/pull/7349
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/master/contributing/code/index.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug, and what versions are affected.
-->
The engine and the service are separate components, so their authentication principal and keytab should also be separated.
Close: #7348
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Unit tests.
```shell
./build/mvn clean install -Dtest=none -DwildcardSuites=org.apache.kyuubi.ha.client.zookeeper.EmbeddedZookeeperDiscoveryClientSuite,org.apache.kyuubi.ha.client.zookeeper.ZookeeperClientProviderSuite,org.apache.kyuubi.engine.spark.SparkProcessBuilderSuite
```
### Was this patch authored or co-authored using generative AI tooling?
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2923587463
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication." +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
Review Comment:
Thank you for pointing that out.
I’ve fixed it in commit https://github.com/apache/kyuubi/pull/7349/changes/b8622214f2719fe82aff8b8ca122ae036f4aa93f.
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication." +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
+ "the engine's ZooKeeper authentication." +
+ "It fallback to `kyuubi.ha.zookeeper.auth.keytab`")
Review Comment:
Thank you for pointing that out.
I’ve fixed it in commit https://github.com/apache/kyuubi/pull/7349/changes/b8622214f2719fe82aff8b8ca122ae036f4aa93f.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r3020014204
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication. " +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
Review Comment:
I think you mean "engine"?
```suggestion
.doc("Location of the Kyuubi engine's keytab that is used for " +
```
do you assume that the keytab was pre-distributed to each machine that the engine (e.g., spark driver) might be run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#issuecomment-4052442944
Hi @LamiumAmplexicaule would you fix the test failure https://github.com/apache/kyuubi/actions/runs/22996851002/job/66896343535?pr=7349#step:7:429? It looks related to the patch.
```
- set up zookeeper auth for engine *** FAILED ***
java.lang.IllegalArgumentException: Can't get Kerberos realm
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:71)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366)
at org.apache.kyuubi.KerberizedTestHelper.tryWithSecurityEnabled(KerberizedTestHelper.scala:154)
at org.apache.kyuubi.KerberizedTestHelper.tryWithSecurityEnabled$(KerberizedTestHelper.scala:142)
at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClientSuite.tryWithSecurityEnabled(ZookeeperDiscoveryClientSuite.scala:66)
at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClientSuite.$anonfun$new$5(ZookeeperDiscoveryClientSuite.scala:142)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
...
Cause: java.lang.IllegalArgumentException: KrbException: Cannot locate default realm
at javax.security.auth.kerberos.KerberosPrincipal.<init>(KerberosPrincipal.java:159)
at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:120)
at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:69)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:315)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:366)
at org.apache.kyuubi.KerberizedTestHelper.tryWithSecurityEnabled(KerberizedTestHelper.scala:154)
at org.apache.kyuubi.KerberizedTestHelper.tryWithSecurityEnabled$(KerberizedTestHelper.scala:142)
at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClientSuite.tryWithSecurityEnabled(ZookeeperDiscoveryClientSuite.scala:66)
at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClientSuite.$anonfun$new$5(ZookeeperDiscoveryClientSuite.scala:142)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r3020014204
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication. " +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
Review Comment:
I think you mean "engine"?
```suggestion
.doc("Location of the Kyuubi engine's keytab that is used for " +
```
do you assume that the keytab was pre-distributed to each machine that the engine (e.g., spark driver) might run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2922689707
##########
docs/configuration/settings.md:
##########
@@ -310,33 +310,35 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Ha
-| Key | Default | Meaning | Type | Since |
-|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
-| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
-| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
-| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
-| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
-| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
-| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
-| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.engine.auth.type | NONE | The type of ZooKeeper authentication for the engine, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.namespace | kyuubi | (deprecated) The root directory for the service to deploy its instance uri | string | 1.0.0 |
-| kyuubi.ha.zookeeper.node.creation.timeout | PT2M | Timeout for creating ZooKeeper node | duration | 1.2.0 |
-| kyuubi.ha.zookeeper.publish.configs | false | When set to true, publish Kerberos configs to Zookeeper. Note that the Hive driver needs to be greater than 1.3 or 2.0 or apply HIVE-11581 patch. | boolean | 1.4.0 |
-| kyuubi.ha.zookeeper.quorum || (deprecated) The connection string for the ZooKeeper ensemble | string | 1.0.0 |
-| kyuubi.ha.zookeeper.session.timeout | 60000 | The timeout(ms) of a connected session to be idled | int | 1.0.0 |
+| Key | Default | Meaning | Type | Since |
+|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
+| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
+| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
+| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
+| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
+| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
+| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
+| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
+| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
+| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.engine.auth.principal | <undefined> | Kerberos principal name that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
+| kyuubi.ha.zookeeper.engine.auth.keytab | <undefined> | Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
Review Comment:
Thank you for your review.
I have added the document for fallback of `kyuubi.ha.zookeeper.engine.auth.principal` and `kyuubi.ha.zookeeper.engine.auth.keytab` in https://github.com/apache/kyuubi/pull/7349/changes/84da06276c3fb24e72203f6e161cc9a0d9dfc145.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2922700382
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/zookeeper/ZookeeperClientProvider.scala:
##########
@@ -110,53 +110,56 @@ object ZookeeperClientProvider extends Logging {
*/
@throws[Exception]
def setUpZooKeeperAuth(conf: KyuubiConf): Unit = {
- def setupZkAuth(): Unit = (conf.get(HA_ZK_AUTH_PRINCIPAL), getKeyTabFile(conf)) match {
- case (Some(principal), Some(keytab)) if UserGroupInformation.isSecurityEnabled =>
- if (!new File(keytab).exists()) {
- throw new IOException(s"${HA_ZK_AUTH_KEYTAB.key}: $keytab does not exists")
- }
- System.setProperty("zookeeper.sasl.clientconfig", "KyuubiZooKeeperClient")
- conf.get(HA_ZK_AUTH_SERVER_PRINCIPAL).foreach { zkServerPrincipal =>
- // ZOOKEEPER-1467 allows configuring SPN in client
- System.setProperty("zookeeper.server.principal", zkServerPrincipal)
- }
- val zkClientPrincipal = KyuubiHadoopUtils.getServerPrincipal(principal)
- val jaasConf = jaasConfigurationCache.computeIfAbsent(
- (principal, keytab),
- _ => {
- // HDFS-16591 makes breaking change on JaasConfiguration
- DynConstructors.builder()
- .impl( // Hadoop 3.3.5 and above
- "org.apache.hadoop.security.authentication.util.JaasConfiguration",
- classOf[String],
- classOf[String],
- classOf[String])
- .impl( // Hadoop 3.3.4 and previous
- // scalastyle:off
- "org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager$JaasConfiguration",
- // scalastyle:on
- classOf[String],
- classOf[String],
- classOf[String])
- .build[Configuration]()
- .newInstance("KyuubiZooKeeperClient", zkClientPrincipal, keytab)
- })
- Configuration.setConfiguration(jaasConf)
- case _ =>
- }
+ def setupZkAuth(
+ principalConfKey: ConfigEntry[Option[String]],
+ keytabConfKey: ConfigEntry[Option[String]]): Unit =
+ (conf.get(principalConfKey), getKeyTabFile(conf, keytabConfKey)) match {
+ case (Some(principal), Some(keytab)) if UserGroupInformation.isSecurityEnabled =>
+ if (!new File(keytab).exists()) {
+ throw new IOException(s"${keytabConfKey.key}: $keytab does not exists")
+ }
+ System.setProperty("zookeeper.sasl.clientconfig", "KyuubiZooKeeperClient")
+ conf.get(HA_ZK_AUTH_SERVER_PRINCIPAL).foreach { zkServerPrincipal =>
+ // ZOOKEEPER-1467 allows configuring SPN in client
+ System.setProperty("zookeeper.server.principal", zkServerPrincipal)
+ }
+ val zkClientPrincipal = KyuubiHadoopUtils.getServerPrincipal(principal)
+ val jaasConf = jaasConfigurationCache.computeIfAbsent(
+ (principal, keytab),
+ _ => {
+ // HDFS-16591 makes breaking change on JaasConfiguration
+ DynConstructors.builder()
+ .impl( // Hadoop 3.3.5 and above
+ "org.apache.hadoop.security.authentication.util.JaasConfiguration",
+ classOf[String],
+ classOf[String],
+ classOf[String])
+ .impl( // Hadoop 3.3.4 and previous
+ // scalastyle:off
+ "org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager$JaasConfiguration",
+ // scalastyle:on
+ classOf[String],
+ classOf[String],
+ classOf[String])
+ .build[Configuration]()
+ .newInstance("KyuubiZooKeeperClient", zkClientPrincipal, keytab)
+ })
+ Configuration.setConfiguration(jaasConf)
+ case _ =>
+ }
Review Comment:
I set it up according to [IntelliJ IDEA Setup Guide — Apache Kyuubi](https://kyuubi.readthedocs.io/en/master/contributing/code/idea_setup.html), so I understand that this indentation is the result of automatic formatting. (It’s possible that I misconfigured something, though.)
Do I need to intentionally disable this and adjust it manually?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#issuecomment-4115830984
> For unit tests that use JAAS, we need to save the current javax.security.auth.login.Configuration before the test and restore it afterward.
Thank you for the detailed analysis. Would you move the restore code using `beforeEach` or/and `afterEach` to ensure the original conf is restored for each test case? Otherwise the conf is not restored when the unit test fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#issuecomment-4052536131
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/7349?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `37 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 0.00%. Comparing base ([`f319921`](https://app.codecov.io/gh/apache/kyuubi/commit/f3199219bd8439c571c11e797170ec134b62db05?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b862221`](https://app.codecov.io/gh/apache/kyuubi/commit/b8622214f2719fe82aff8b8ca122ae036f4aa93f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 4 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/kyuubi/pull/7349?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../ha/client/zookeeper/ZookeeperClientProvider.scala](https://app.codecov.io/gh/apache/kyuubi/pull/7349?src=pr&el=tree&filepath=kyuubi-ha%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fha%2Fclient%2Fzookeeper%2FZookeeperClientProvider.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWhhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2hhL2NsaWVudC96b29rZWVwZXIvWm9va2VlcGVyQ2xpZW50UHJvdmlkZXIuc2NhbGE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/7349?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...la/org/apache/kyuubi/ha/HighAvailabilityConf.scala](https://app.codecov.io/gh/apache/kyuubi/pull/7349?src=pr&el=tree&filepath=kyuubi-ha%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fha%2FHighAvailabilityConf.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWhhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2hhL0hpZ2hBdmFpbGFiaWxpdHlDb25mLnNjYWxh) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/7349?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ache/kyuubi/engine/spark/SparkProcessBuilder.scala](https://app.codecov.io/gh/apache/kyuubi/pull/7349?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fengine%2Fspark%2FSparkProcessBuilder.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvc3BhcmsvU3BhcmtQcm9jZXNzQnVpbGRlci5zY2FsYQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/7349?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #7349 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 698 698
Lines 43657 43667 +10
Branches 5896 5896
======================================
- Misses 43657 43667 +10
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/7349?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r3020144626
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication. " +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
Review Comment:
Thank you for your review.
> I think you mean "engine"?
Yes. it is typo. I will fix it.
> do you assume that the keytab was pre-distributed to each machine that the engine (e.g., spark driver) might be run?
No. As before, the administrator just needs to place the Engine’s keytab used for Zookeeper authentication on Kyuubi.
- https://github.com/apache/kyuubi/blob/fd5ee95ffe81f648cf2294c0ceafe3f926d55096/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala#L211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#issuecomment-4174771357
Thank you @LamiumAmplexicaule and @pan3793. Merged the PR into master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2922689707
##########
docs/configuration/settings.md:
##########
@@ -310,33 +310,35 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Ha
-| Key | Default | Meaning | Type | Since |
-|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
-| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
-| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
-| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
-| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
-| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
-| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
-| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.engine.auth.type | NONE | The type of ZooKeeper authentication for the engine, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.namespace | kyuubi | (deprecated) The root directory for the service to deploy its instance uri | string | 1.0.0 |
-| kyuubi.ha.zookeeper.node.creation.timeout | PT2M | Timeout for creating ZooKeeper node | duration | 1.2.0 |
-| kyuubi.ha.zookeeper.publish.configs | false | When set to true, publish Kerberos configs to Zookeeper. Note that the Hive driver needs to be greater than 1.3 or 2.0 or apply HIVE-11581 patch. | boolean | 1.4.0 |
-| kyuubi.ha.zookeeper.quorum || (deprecated) The connection string for the ZooKeeper ensemble | string | 1.0.0 |
-| kyuubi.ha.zookeeper.session.timeout | 60000 | The timeout(ms) of a connected session to be idled | int | 1.0.0 |
+| Key | Default | Meaning | Type | Since |
+|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
+| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
+| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
+| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
+| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
+| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
+| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
+| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
+| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
+| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.engine.auth.principal | <undefined> | Kerberos principal name that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
+| kyuubi.ha.zookeeper.engine.auth.keytab | <undefined> | Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
Review Comment:
Thank you for your review.
I have added the document for fallback of `kyuubi.ha.zookeeper.engine.auth.principal` and `kyuubi.ha.zookeeper.engine.auth.keytab` in https://github.com/apache/kyuubi/pull/7349/changes/20b6f0d40838bf55fbd41dfa8e0c4f73554132ab.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r3020144626
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication. " +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
Review Comment:
Thank you for your review.
> I think you mean "engine"?
Yes. it is typo. I will fix it.
> do you assume that the keytab was pre-distributed to each machine that the engine (e.g., spark driver) might be run?
No. As before, the administrator just needs to place the Engine’s keytab used for authentication on Kyuubi.
- https://github.com/apache/kyuubi/blob/fd5ee95ffe81f648cf2294c0ceafe3f926d55096/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala#L211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#issuecomment-4167808822
the intention - server/engine zk credential separation, makes sense, change is straightforward, but the keytab management mechanism is unclear to me.
previouly, it's able to use `spark-submit --files <zk-keytab>` to auto distribute keytab so administrators just need to distribute zk keytab at the Kyuubi server host, instead of each YARN nodes.
you can check `SparkProcessBuilder.zkAuthKeytabFileConf` for more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2923219886
##########
docs/configuration/settings.md:
##########
@@ -310,33 +310,35 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Ha
-| Key | Default | Meaning | Type | Since |
-|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
-| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
-| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
-| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
-| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
-| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
-| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
-| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.engine.auth.type | NONE | The type of ZooKeeper authentication for the engine, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.namespace | kyuubi | (deprecated) The root directory for the service to deploy its instance uri | string | 1.0.0 |
-| kyuubi.ha.zookeeper.node.creation.timeout | PT2M | Timeout for creating ZooKeeper node | duration | 1.2.0 |
-| kyuubi.ha.zookeeper.publish.configs | false | When set to true, publish Kerberos configs to Zookeeper. Note that the Hive driver needs to be greater than 1.3 or 2.0 or apply HIVE-11581 patch. | boolean | 1.4.0 |
-| kyuubi.ha.zookeeper.quorum || (deprecated) The connection string for the ZooKeeper ensemble | string | 1.0.0 |
-| kyuubi.ha.zookeeper.session.timeout | 60000 | The timeout(ms) of a connected session to be idled | int | 1.0.0 |
+| Key | Default | Meaning | Type | Since |
+|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
+| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
+| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
+| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
+| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
+| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
+| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
+| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
+| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
+| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.engine.auth.principal | <undefined> | Kerberos principal name that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
+| kyuubi.ha.zookeeper.engine.auth.keytab | <undefined> | Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
Review Comment:
I realized that instead of updating it manually, I should use `dev/gen/gen_all_config_docs.sh`, so I’ve update that.
- https://kyuubi.readthedocs.io/en/master/contributing/code/developer.html#append-descriptions-of-new-configurations-to-settings-md
##########
docs/configuration/settings.md:
##########
@@ -310,33 +310,35 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Ha
-| Key | Default | Meaning | Type | Since |
-|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
-| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
-| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
-| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
-| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
-| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
-| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
-| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.engine.auth.type | NONE | The type of ZooKeeper authentication for the engine, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.namespace | kyuubi | (deprecated) The root directory for the service to deploy its instance uri | string | 1.0.0 |
-| kyuubi.ha.zookeeper.node.creation.timeout | PT2M | Timeout for creating ZooKeeper node | duration | 1.2.0 |
-| kyuubi.ha.zookeeper.publish.configs | false | When set to true, publish Kerberos configs to Zookeeper. Note that the Hive driver needs to be greater than 1.3 or 2.0 or apply HIVE-11581 patch. | boolean | 1.4.0 |
-| kyuubi.ha.zookeeper.quorum || (deprecated) The connection string for the ZooKeeper ensemble | string | 1.0.0 |
-| kyuubi.ha.zookeeper.session.timeout | 60000 | The timeout(ms) of a connected session to be idled | int | 1.0.0 |
+| Key | Default | Meaning | Type | Since |
+|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
+| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
+| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
+| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
+| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
+| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
+| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
+| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
+| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
+| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.engine.auth.principal | <undefined> | Kerberos principal name that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
+| kyuubi.ha.zookeeper.engine.auth.keytab | <undefined> | Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
Review Comment:
I realized that instead of updating it manually, I should use `dev/gen/gen_all_config_docs.sh`, so I’ve update it.
- https://kyuubi.readthedocs.io/en/master/contributing/code/developer.html#append-descriptions-of-new-configurations-to-settings-md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r3020144626
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication. " +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
Review Comment:
Thank you for your review.
> I think you mean "engine"?
No.
Since this keytab will be placed on the Kyuubi server, I think this description is correct.
> do you assume that the keytab was pre-distributed to each machine that the engine (e.g., spark driver) might be run?
No. As before, the administrator just needs to place the Engine’s keytab used for Zookeeper authentication on Kyuubi.
- https://github.com/apache/kyuubi/blob/fd5ee95ffe81f648cf2294c0ceafe3f926d55096/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala#L212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2923402728
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication." +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
Review Comment:
(nit) Need a whitespace between the 2 sentences.
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication." +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
+ "the engine's ZooKeeper authentication." +
+ "It fallback to `kyuubi.ha.zookeeper.auth.keytab`")
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2923219886
##########
docs/configuration/settings.md:
##########
@@ -310,33 +310,35 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Ha
-| Key | Default | Meaning | Type | Since |
-|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
-| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
-| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
-| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
-| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
-| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
-| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
-| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.engine.auth.type | NONE | The type of ZooKeeper authentication for the engine, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.namespace | kyuubi | (deprecated) The root directory for the service to deploy its instance uri | string | 1.0.0 |
-| kyuubi.ha.zookeeper.node.creation.timeout | PT2M | Timeout for creating ZooKeeper node | duration | 1.2.0 |
-| kyuubi.ha.zookeeper.publish.configs | false | When set to true, publish Kerberos configs to Zookeeper. Note that the Hive driver needs to be greater than 1.3 or 2.0 or apply HIVE-11581 patch. | boolean | 1.4.0 |
-| kyuubi.ha.zookeeper.quorum || (deprecated) The connection string for the ZooKeeper ensemble | string | 1.0.0 |
-| kyuubi.ha.zookeeper.session.timeout | 60000 | The timeout(ms) of a connected session to be idled | int | 1.0.0 |
+| Key | Default | Meaning | Type | Since |
+|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
+| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
+| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
+| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
+| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
+| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
+| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
+| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
+| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
+| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.engine.auth.principal | <undefined> | Kerberos principal name that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
+| kyuubi.ha.zookeeper.engine.auth.keytab | <undefined> | Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
Review Comment:
I realized that instead of updating it manually, I should be using `dev/gen/gen_all_config_docs.sh`, so I’ve updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2923219886
##########
docs/configuration/settings.md:
##########
@@ -310,33 +310,35 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Ha
-| Key | Default | Meaning | Type | Since |
-|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
-| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
-| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
-| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
-| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
-| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
-| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
-| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.engine.auth.type | NONE | The type of ZooKeeper authentication for the engine, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.namespace | kyuubi | (deprecated) The root directory for the service to deploy its instance uri | string | 1.0.0 |
-| kyuubi.ha.zookeeper.node.creation.timeout | PT2M | Timeout for creating ZooKeeper node | duration | 1.2.0 |
-| kyuubi.ha.zookeeper.publish.configs | false | When set to true, publish Kerberos configs to Zookeeper. Note that the Hive driver needs to be greater than 1.3 or 2.0 or apply HIVE-11581 patch. | boolean | 1.4.0 |
-| kyuubi.ha.zookeeper.quorum || (deprecated) The connection string for the ZooKeeper ensemble | string | 1.0.0 |
-| kyuubi.ha.zookeeper.session.timeout | 60000 | The timeout(ms) of a connected session to be idled | int | 1.0.0 |
+| Key | Default | Meaning | Type | Since |
+|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
+| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
+| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
+| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
+| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
+| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
+| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
+| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
+| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
+| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.engine.auth.principal | <undefined> | Kerberos principal name that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
+| kyuubi.ha.zookeeper.engine.auth.keytab | <undefined> | Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
Review Comment:
I realized that instead of updating it manually, I should be using `dev/gen/gen_all_config_docs.sh`, so I’ve updated it.
- https://kyuubi.readthedocs.io/en/master/contributing/code/developer.html#append-descriptions-of-new-configurations-to-settings-md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#issuecomment-4064611012
It turned out that the `Cannot locate default realm` error occurs for the following reason.
In `org.apache.kyuubi.KerberizedTestHelper#tryWithSecurityEnabled`, `System.clearProperty("java.security.krb5.conf")` is called in the finally block.
- https://github.com/apache/kyuubi/blob/52b038b93cb5e214fcd4a1ed9d1fa8cd3ae8abbc/kyuubi-common/src/test/scala/org/apache/kyuubi/KerberizedTestHelper.scala#L159
In `org.apache.kyuubi.ha.client.zookeeper.ZookeeperClientProvider`, `org.apache.hadoop.security.authentication.util.JaasConfiguration` or `org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager$JaasConfiguration` is used,
- https://github.com/apache/kyuubi/blob/52b038b93cb5e214fcd4a1ed9d1fa8cd3ae8abbc/kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/zookeeper/ZookeeperClientProvider.scala#L124-L144
and inside it there is a line:
```java
options.put("refreshKrb5Config", "true");
```
- https://github.com/apache/hadoop/blob/a178eb780f5c983ab7e5c5a4add8a3027060c68d/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/JaasConfiguration.java#L51
- https://github.com/apache/hadoop/blob/15621e34155ff41b3bfaefec066cde2b37eaff64/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java#L280
Because of this, `sun.security.krb5.Config#refresh` is invoked before `com.sun.security.auth.module.Krb5LoginModule#login`.
- https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/jdk.security.auth/share/classes/com/sun/security/auth/module/Krb5LoginModule.java#L533
- https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L150-L158
At that point, since `java.security.krb5.conf` does not exist,
- https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L213
- https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L867-L868
it attempts to read `/etc/krb5.conf`.
- https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L234
- https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L939
As `/etc/krb5.conf` does not exist, nothing is loaded, and the singleton instance maintained by `sun.security.krb5.Config` is overwritten (the stanzaTable becomes empty).
- https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L152
As a result, `get("libdefaults", "default_realm")` returns `null`, which causes a `KrbException("Cannot locate default realm")` to be thrown.
- https://github.com/openjdk/jdk17u-dev/blob/1e113f495f88a854db13af9341924c9f84647bff/src/java.security.jgss/share/classes/sun/security/krb5/Config.java#L1196
We need logic that either uses `tryWithSecurityEnabled` or, for unit tests that use JAAS (e.g., tests that `set up zookeeper auth` and `set up zookeeper auth for engine`), saves the current `javax.security.auth.login.Configuration` before the test and restores it afterwards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka closed pull request #7349: [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine
URL: https://github.com/apache/kyuubi/pull/7349
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r2922486250
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/client/zookeeper/ZookeeperClientProvider.scala:
##########
@@ -110,53 +110,56 @@ object ZookeeperClientProvider extends Logging {
*/
@throws[Exception]
def setUpZooKeeperAuth(conf: KyuubiConf): Unit = {
- def setupZkAuth(): Unit = (conf.get(HA_ZK_AUTH_PRINCIPAL), getKeyTabFile(conf)) match {
- case (Some(principal), Some(keytab)) if UserGroupInformation.isSecurityEnabled =>
- if (!new File(keytab).exists()) {
- throw new IOException(s"${HA_ZK_AUTH_KEYTAB.key}: $keytab does not exists")
- }
- System.setProperty("zookeeper.sasl.clientconfig", "KyuubiZooKeeperClient")
- conf.get(HA_ZK_AUTH_SERVER_PRINCIPAL).foreach { zkServerPrincipal =>
- // ZOOKEEPER-1467 allows configuring SPN in client
- System.setProperty("zookeeper.server.principal", zkServerPrincipal)
- }
- val zkClientPrincipal = KyuubiHadoopUtils.getServerPrincipal(principal)
- val jaasConf = jaasConfigurationCache.computeIfAbsent(
- (principal, keytab),
- _ => {
- // HDFS-16591 makes breaking change on JaasConfiguration
- DynConstructors.builder()
- .impl( // Hadoop 3.3.5 and above
- "org.apache.hadoop.security.authentication.util.JaasConfiguration",
- classOf[String],
- classOf[String],
- classOf[String])
- .impl( // Hadoop 3.3.4 and previous
- // scalastyle:off
- "org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager$JaasConfiguration",
- // scalastyle:on
- classOf[String],
- classOf[String],
- classOf[String])
- .build[Configuration]()
- .newInstance("KyuubiZooKeeperClient", zkClientPrincipal, keytab)
- })
- Configuration.setConfiguration(jaasConf)
- case _ =>
- }
+ def setupZkAuth(
+ principalConfKey: ConfigEntry[Option[String]],
+ keytabConfKey: ConfigEntry[Option[String]]): Unit =
+ (conf.get(principalConfKey), getKeyTabFile(conf, keytabConfKey)) match {
+ case (Some(principal), Some(keytab)) if UserGroupInformation.isSecurityEnabled =>
+ if (!new File(keytab).exists()) {
+ throw new IOException(s"${keytabConfKey.key}: $keytab does not exists")
+ }
+ System.setProperty("zookeeper.sasl.clientconfig", "KyuubiZooKeeperClient")
+ conf.get(HA_ZK_AUTH_SERVER_PRINCIPAL).foreach { zkServerPrincipal =>
+ // ZOOKEEPER-1467 allows configuring SPN in client
+ System.setProperty("zookeeper.server.principal", zkServerPrincipal)
+ }
+ val zkClientPrincipal = KyuubiHadoopUtils.getServerPrincipal(principal)
+ val jaasConf = jaasConfigurationCache.computeIfAbsent(
+ (principal, keytab),
+ _ => {
+ // HDFS-16591 makes breaking change on JaasConfiguration
+ DynConstructors.builder()
+ .impl( // Hadoop 3.3.5 and above
+ "org.apache.hadoop.security.authentication.util.JaasConfiguration",
+ classOf[String],
+ classOf[String],
+ classOf[String])
+ .impl( // Hadoop 3.3.4 and previous
+ // scalastyle:off
+ "org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager$JaasConfiguration",
+ // scalastyle:on
+ classOf[String],
+ classOf[String],
+ classOf[String])
+ .build[Configuration]()
+ .newInstance("KyuubiZooKeeperClient", zkClientPrincipal, keytab)
+ })
+ Configuration.setConfiguration(jaasConf)
+ case _ =>
+ }
Review Comment:
The indents look incorrect. Would you keep the current indents so that reviewers can easily review the diff.
##########
docs/configuration/settings.md:
##########
@@ -310,33 +310,35 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Ha
-| Key | Default | Meaning | Type | Since |
-|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
-| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
-| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
-| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
-| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
-| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
-| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
-| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
-| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
-| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
-| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
-| kyuubi.ha.zookeeper.engine.auth.type | NONE | The type of ZooKeeper authentication for the engine, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
-| kyuubi.ha.zookeeper.namespace | kyuubi | (deprecated) The root directory for the service to deploy its instance uri | string | 1.0.0 |
-| kyuubi.ha.zookeeper.node.creation.timeout | PT2M | Timeout for creating ZooKeeper node | duration | 1.2.0 |
-| kyuubi.ha.zookeeper.publish.configs | false | When set to true, publish Kerberos configs to Zookeeper. Note that the Hive driver needs to be greater than 1.3 or 2.0 or apply HIVE-11581 patch. | boolean | 1.4.0 |
-| kyuubi.ha.zookeeper.quorum || (deprecated) The connection string for the ZooKeeper ensemble | string | 1.0.0 |
-| kyuubi.ha.zookeeper.session.timeout | 60000 | The timeout(ms) of a connected session to be idled | int | 1.0.0 |
+| Key | Default | Meaning | Type | Since |
+|------------------------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
+| kyuubi.ha.addresses || The connection string for the discovery ensemble | string | 1.6.0 |
+| kyuubi.ha.client.class | org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient | Class name for service discovery client.<ul> <li>Zookeeper: org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient</li> <li>Etcd: org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient</li></ul> | string | 1.6.0 |
+| kyuubi.ha.etcd.lease.timeout | PT10S | Timeout for etcd keep alive lease. The kyuubi server will know the unexpected loss of engine after up to this seconds. | duration | 1.6.0 |
+| kyuubi.ha.etcd.ssl.ca.path | <undefined> | Where the etcd CA certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.certificate.path | <undefined> | Where the etcd SSL certificate file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.client.key.path | <undefined> | Where the etcd SSL key file is stored. | string | 1.6.0 |
+| kyuubi.ha.etcd.ssl.enabled | false | When set to true, will build an SSL secured etcd client. | boolean | 1.6.0 |
+| kyuubi.ha.namespace | kyuubi | The root directory for the service to deploy its instance uri | string | 1.6.0 |
+| kyuubi.ha.zookeeper.acl.enabled | false | (deprecated) Set to true if the ZooKeeper ensemble is kerberized | boolean | 1.0.0 |
+| kyuubi.ha.zookeeper.auth.digest | <undefined> | The digest auth string is used for ZooKeeper authentication, like: username:password. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.keytab | <undefined> | Location of the Kyuubi server's keytab that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.principal | <undefined> | Kerberos principal name that is used for ZooKeeper authentication. | string | 1.3.2 |
+| kyuubi.ha.zookeeper.auth.serverPrincipal | <undefined> | Kerberos principal name of ZooKeeper Server. It only takes effect when Zookeeper client's version at least 3.5.7 or 3.6.0 or applies ZOOKEEPER-1467. To use Zookeeper 3.6 client, compile Kyuubi with `-Pzookeeper-3.6`. | string | 1.8.0 |
+| kyuubi.ha.zookeeper.auth.type | NONE | The type of ZooKeeper authentication, all candidates are <ul><li>NONE</li><li> KERBEROS</li><li> DIGEST</li></ul> | string | 1.3.2 |
+| kyuubi.ha.zookeeper.connection.base.retry.wait | 1000 | Initial amount of time to wait between retries to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retries | 3 | Max retry times for connecting to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.max.retry.wait | 30000 | Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.retry.policy | EXPONENTIAL_BACKOFF | The retry policy for connecting to the ZooKeeper ensemble, all candidates are: <ul><li>ONE_TIME</li><li> N_TIME</li><li> EXPONENTIAL_BACKOFF</li><li> BOUNDED_EXPONENTIAL_BACKOFF</li><li> UNTIL_ELAPSED</li></ul> | string | 1.0.0 |
+| kyuubi.ha.zookeeper.connection.timeout | 15000 | The timeout(ms) of creating the connection to the ZooKeeper ensemble | int | 1.0.0 |
+| kyuubi.ha.zookeeper.engine.auth.principal | <undefined> | Kerberos principal name that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
+| kyuubi.ha.zookeeper.engine.auth.keytab | <undefined> | Location of the Kyuubi server’s keytab that is used for the engine's ZooKeeper authentication. | string | 1.12.0 |
Review Comment:
If they are unset, what are used by default? Would you document?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#issuecomment-4116619873
> Would you move the restore code using beforeEach or/and afterEach to ensure the original conf is restored for each test case?
I have moved the restore logic to beforeEach/afterEach in https://github.com/apache/kyuubi/pull/7349/changes/fd5ee95ffe81f648cf2294c0ceafe3f926d55096.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "LamiumAmplexicaule (via GitHub)" <gi...@apache.org>.
LamiumAmplexicaule commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r3020144626
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication. " +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
Review Comment:
Thank you for your review.
> I think you mean "engine"?
Yes. it is typo. I will fix it.
> do you assume that the keytab was pre-distributed to each machine that the engine (e.g., spark driver) might be run?
No. As before, the administrator just needs to place the Engine’s keytab used for Zookeeper authentication on Kyuubi.
- https://github.com/apache/kyuubi/blob/fd5ee95ffe81f648cf2294c0ceafe3f926d55096/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala#L212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7348] Separate the principal and keytab used for ZooKeeper authentication between the service and the engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7349:
URL: https://github.com/apache/kyuubi/pull/7349#discussion_r3021079022
##########
kyuubi-ha/src/main/scala/org/apache/kyuubi/ha/HighAvailabilityConf.scala:
##########
@@ -106,12 +106,27 @@ object HighAvailabilityConf {
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_PRINCIPAL)
+ val HA_ZK_ENGINE_AUTH_PRINCIPAL: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.principal")
+ .doc("Kerberos principal name that is used for the engine's ZooKeeper authentication. " +
+ "It fallback to `kyuubi.ha.zookeeper.auth.principal`")
+ .version("1.12.0")
+ .fallbackConf(HighAvailabilityConf.HA_ZK_AUTH_PRINCIPAL)
+
val HA_ZK_AUTH_KEYTAB: ConfigEntry[Option[String]] =
buildConf("kyuubi.ha.zookeeper.auth.keytab")
.doc("Location of the Kyuubi server's keytab that is used for ZooKeeper authentication.")
.version("1.3.2")
.fallbackConf(KyuubiConf.SERVER_KEYTAB)
+ val HA_ZK_ENGINE_AUTH_KEYTAB: ConfigEntry[Option[String]] =
+ buildConf("kyuubi.ha.zookeeper.engine.auth.keytab")
+ .doc("Location of the Kyuubi server's keytab that is used for " +
Review Comment:
oh, I missed your change in `SparkProcessBuilder`, that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org