You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/10/09 10:23:56 UTC
[GitHub] [incubator-kyuubi] wForget opened a new issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
wForget opened a new issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Support zookeeper sasl kerberos authentication of engine and more zookeeper ACL Schemes.
### Motivation
_No response_
### Describe the solution
Zookeeper 支持多种类型的 ACL Schemes，下面列出两种典型进行说明
1. 支持 SASL Kerberos 类型的 ACL
节点 ACLs 示例：
```
'world,'anyone
: r
'sasl,'test
: cdrwa
```
配置规划：
```
kyuubi.ha.zookeeper.acl.enabled=true
kyuubi.ha.zookeeper.auth.sasl.kerberos=true # 使用 sasl kerberos 类型认证
# 还需要 kerberos 相关配置
```
其它修改：
+ Yarn Cluster 模式运行 Engine 时，通过 --file 上传 keytab 文件，并指定相对路径访问 keytab 文件
2. 支持 Digest 类型的 ACL
节点 ACLs 示例：
```
'world,'anyone
: r
'digest,'test:V28q/NynI4JI3Rk54h0r8O5kMug=
: cdrwa
```
配置规划：
```
kyuubi.ha.zookeeper.acl.enabled=true
kyuubi.ha.zookeeper.auth=digest:test:test # 使用 auth string 进行认证，格式：scheme:expression:perms
```
其它修改：
+ 将 kyuubi.ha.zookeeper.auth 配置解析为 AuthInfo 对象，通过 CuratorFrameworkFactory.Builder#authorization 方法添加认证
参考：
+ [ZooKeeper access control using ACLs](http://zookeeper.apache.org/doc/r3.7.0/zookeeperProgrammers.html#sc_ZooKeeperAccessControl)
+ [ZooKeeper and SASL](https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+and+SASL)
+ [zookeeper（client-server身份认证-SASL配置）](https://www.liangzl.com/get-article-detail-167111.html)
### Additional context
I don’t have a deep understanding of the Zookeeper authentication mechanism. If you have any questions, please point them out.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940816783
It looks fine to me. However, can we merge some of these configurations, it is now very hard to explain and use.
```
kyuubi.ha.zookeeper.acl.enabled=true
kyuubi.ha.zookeeper.auth.sasl.kerberos=false
kyuubi.ha.zookeeper.auth=digest:hue:***
kyuubi.ha.zookeeper.acl.engine.enabled=true
```
cc @zhouyifan279, do you have any idea, if we can add some unit tests to test with a kerberied zookeeper and acls
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940827602
How about
```
kyuubi.ha.zookeeper.acl.enabled=true // deprecated this
kyuubi.ha.zookeeper.acl.engine.enabled=true // remove this as it's still under dev
kyuubi.ha.zookeeper.auth.type=none/kerberos/digest
kyuubi.ha.zookeeper.engine.auth.type=none/kerberos/digest, where none = kyuubi.ha.zookeeper.acl.enabled=false
# we can introduce these a new PR later to avoid staging service keytab in engine side, which is unsecure
kyuubi.ha.zookeeper.auth.principal
kyuubi.ha.zookeeper.auth.keytab
kyuubi.ha.zookeeper.auth.digest=digest contents?
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940609570
> --conf spark.kyuubi.kinit.keytab=kyuubi.keytab \
Looks like we have to detect the value of ```--conf spark.kyuubi.kinit.keytab=kyuubi.keytab \``` at engine side after spark gets instantiated not pass it from server to engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn closed issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-942116481
> It looks fine to me. However, can we merge some of these configurations, it is now very hard to explain and use.
>
> ```
> kyuubi.ha.zookeeper.acl.enabled=true
> kyuubi.ha.zookeeper.auth.sasl.kerberos=false
> kyuubi.ha.zookeeper.auth=digest:hue:***
> kyuubi.ha.zookeeper.acl.engine.enabled=true
> ```
>
> cc @zhouyifan279, do you have any idea, if we can add some unit tests to test with a kerberied zookeeper and acls
As we already have `org.apache.kyuubi.KerberizedTestHelper` to setup a KDC, and zookeeper integrates kerberos using JAAS, it should be easy to setup an kerberied embedde zookeeper server.
@wForget , would you mind to add these test cases ? I'm also glad to do the work if you have no time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 edited a comment on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
zhouyifan279 edited a comment on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-942116481
> It looks fine to me. However, can we merge some of these configurations, it is now very hard to explain and use.
>
> ```
> kyuubi.ha.zookeeper.acl.enabled=true
> kyuubi.ha.zookeeper.auth.sasl.kerberos=false
> kyuubi.ha.zookeeper.auth=digest:hue:***
> kyuubi.ha.zookeeper.acl.engine.enabled=true
> ```
>
> cc @zhouyifan279, do you have any idea, if we can add some unit tests to test with a kerberied zookeeper and acls
As we already have `org.apache.kyuubi.KerberizedTestHelper` to setup a KDC, and zookeeper uses JAAS to integrates with kerberos , it should be easy to setup an kerberied embedded zookeeper server.
@wForget , would you mind to add these test cases ?
I'm also glad to do the work if you have no time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-939728759
> That's true. But it will be difficult and hacky to let the Kyuubi server know whether the engine is in client mode or not.
After adding the --file configuration, the keytab path needs to be changed to a relative path. Is there a problem changing to a relative path in the yarn client mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-939719789
That's true. But it will be difficult and hacky to let the Kyuubi server know whether the engine is in client mode or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-939744096
> > Is there a problem changing to a relative path in the yarn client mode?
>
> I guess it is not a problem and as same as in the yarn cluster mode, maybe also other cluster managers
OK, thanks @yaooqinn . I will not distinguish between client and cluster modes and test them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940852636
@yaooqinn Looks good, Can we add the following fallback configuration?
```
kyuubi.ha.zookeeper.engine.auth.type fallback to kyuubi.ha.zookeeper.auth.type
kyuubi.ha.zookeeper.auth.principal fallback to kyuubi.kinit.principal
kyuubi.ha.zookeeper.auth.keytab fallback to kyuubi.kinit.keytab
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-939717546
> > Yarn Cluster 模式运行 Engine 时，通过 --file 上传 keytab 文件，并指定相对路径访问 keytab 文件
>
> we can use --file for both client and cluster mode
when running in spark yarn-client mode, the driver should be able to read the keytab file of the kyuubi server. is it necessary to add the --file ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-942125202
> As we already have `org.apache.kyuubi.KerberizedTestHelper` to setup a KDC, and zookeeper uses JAAS to integrates with kerberos , it should be easy to setup an kerberied embedded zookeeper server.
>
> @wForget , would you mind to add these test cases ?
>
> I'm also glad to do the work if you have no time.
Thanks @zhouyifan279 , I still have some configurations to be adjusted. After completion, I will improve the test cases according to your suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-939909134
Hi @yaooqinn , There is a problem with adding --file in the yarn-client mode and changing the path to a relative path.
spark conf:
```
--conf spark.master=yarn \
--conf spark.submit.deployMode=client \
--conf spark.files=/***/kyuubi.keytab \
--conf spark.kyuubi.kinit.keytab=kyuubi.keytab \
--conf spark.kyuubi.kinit.principal=*** \
```
error log:
```
Diagnostic: Failed to initialize SparkSQLEngine: kyuubi.kinit.keytab does not exists
org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: kyuubi.kinit.keytab does not exists
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:130)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:113)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:113)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:154)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: kyuubi.kinit.keytab does not exists
at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.setUpZooKeeperAuth(ZooKeeperClientProvider.scala:106)
at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.buildZookeeperClient(ZooKeeperClientProvider.scala:42)
at org.apache.kyuubi.ha.client.ServiceDiscovery.initialize(ServiceDiscovery.scala:73)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
at org.apache.kyuubi.service.AbstractFrontendService.initialize(AbstractFrontendService.scala:42)
at org.apache.kyuubi.service.ThriftBinaryFrontendService.initialize(ThriftBinaryFrontendService.scala:104)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:46)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:49)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:126)
... 22 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-939738985
> Is there a problem changing to a relative path in the yarn client mode?
I guess it is not a problem and as same as in the yarn cluster mode, maybe also other cluster managers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940628069
1. pass --conf spark.kyuubi.kinit.keytab=/the/absolute/path/of/kyuubi.keytab to engine side
2. at engine side, replace it with relative path of kyuubi.keytab, if needed and the relative kyuubi.keytab is present, otherwise leave it empty or the AS-IS absolute path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-939711558
> Yarn Cluster 模式运行 Engine 时，通过 --file 上传 keytab 文件，并指定相对路径访问 keytab 文件
we can use --file for both client and cluster mode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940807282
I have tested it. Please help me see if there are problems with the implementation and configuration. cc @yaooqinn
The results are as follows:
1\. sasl kerberos
kyuubi conf：
```
kyuubi.ha.zookeeper.acl.enabled=true
kyuubi.ha.zookeeper.auth.sasl.kerberos=true
kyuubi.authentication KERBEROS
kyuubi.kinit.principal hue/***@****
kyuubi.kinit.keytab /****/hue.keytab
kyuubi.ha.zookeeper.quorum=***:2181
kyuubi.ha.zookeeper.namespace=kyuubi_***-test
kyuubi.ha.zookeeper.acl.engine.enabled=true
```
acls:
2\. digest
kyuubi conf:
```
kyuubi.ha.zookeeper.acl.enabled=true
kyuubi.ha.zookeeper.auth.sasl.kerberos=false
kyuubi.ha.zookeeper.auth=digest:hue:***
kyuubi.ha.zookeeper.quorum=***:2181
kyuubi.ha.zookeeper.namespace=kyuubi_***-test
kyuubi.ha.zookeeper.acl.engine.enabled=true
```
acls:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-942116481
> It looks fine to me. However, can we merge some of these configurations, it is now very hard to explain and use.
>
> ```
> kyuubi.ha.zookeeper.acl.enabled=true
> kyuubi.ha.zookeeper.auth.sasl.kerberos=false
> kyuubi.ha.zookeeper.auth=digest:hue:***
> kyuubi.ha.zookeeper.acl.engine.enabled=true
> ```
>
> cc @zhouyifan279, do you have any idea, if we can add some unit tests to test with a kerberied zookeeper and acls
As we already have `org.apache.kyuubi.KerberizedTestHelper` to setup a KDC, and zookeeper integrates kerberos using JAAS, it should be easy to setup an kerberied embedde zookeeper server.
@wForget , would you mind to add these test cases ? I'm also glad to do the work if you have no time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940855223
> @yaooqinn Looks good, Can we add the following fallback configuration?
>
> ```
> kyuubi.ha.zookeeper.engine.auth.type fallback to kyuubi.ha.zookeeper.auth.type
> kyuubi.ha.zookeeper.auth.principal fallback to kyuubi.kinit.principal
> kyuubi.ha.zookeeper.auth.keytab fallback to kyuubi.kinit.keytab
> ```
SGTM also cc @turboFei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940629478
> 1. pass --conf spark.kyuubi.kinit.keytab=/the/absolute/path/of/kyuubi.keytab to engine side
> 2. at engine side, replace it with relative path of kyuubi.keytab, if needed and the relative kyuubi.keytab is present, otherwise leave it empty or the AS-IS absolute path
OK, thanks you for your guidance. I will implement it in this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 edited a comment on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
zhouyifan279 edited a comment on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-942116481
> It looks fine to me. However, can we merge some of these configurations, it is now very hard to explain and use.
>
> ```
> kyuubi.ha.zookeeper.acl.enabled=true
> kyuubi.ha.zookeeper.auth.sasl.kerberos=false
> kyuubi.ha.zookeeper.auth=digest:hue:***
> kyuubi.ha.zookeeper.acl.engine.enabled=true
> ```
>
> cc @zhouyifan279, do you have any idea, if we can add some unit tests to test with a kerberied zookeeper and acls
As we already have `org.apache.kyuubi.KerberizedTestHelper` to setup a KDC, and zookeeper integrates kerberos using JAAS, it should be easy to setup an kerberied embedde zookeeper server.
@wForget , would you mind to add these test cases ?
I'm also glad to do the work if you have no time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-942125202
> As we already have `org.apache.kyuubi.KerberizedTestHelper` to setup a KDC, and zookeeper uses JAAS to integrates with kerberos , it should be easy to setup an kerberied embedded zookeeper server.
>
> @wForget , would you mind to add these test cases ?
>
> I'm also glad to do the work if you have no time.
Thanks @zhouyifan279 , I still have some configurations to be adjusted. After completion, I will improve the test cases according to your suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] zhouyifan279 edited a comment on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
zhouyifan279 edited a comment on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-942116481
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] wForget commented on issue #1204: [FEATURE] Enhance zookeeper authentication and acls of Kyuubi HA module
Posted by GitBox <gi...@apache.org>.
wForget commented on issue #1204:
URL: https://github.com/apache/incubator-kyuubi/issues/1204#issuecomment-940623454
> Looks like we have to detect the value of `--conf spark.kyuubi.kinit.keytab=kyuubi.keytab \` at engine side after spark gets instantiated not pass it from server to engine
Sorry, @yaooqinn , I don't quite understand this reply. The current detection is on the Engine side.
This problem may be due to the fact that spark.files are not placed in the user.home directory in yarn-client mode, resulting in inaccessible relative paths. Should we distinguish between client and cluster modes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org