You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/08/22 14:09:04 UTC
[GitHub] [incubator-kyuubi] bowenliang123 opened a new pull request, #3308: [KYUUBI 3300] [FEATURE] [AUTHZ] WIP: Support overriding usergroup with UserStore in AccessRequest
bowenliang123 opened a new pull request, #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Fix https://github.com/apache/incubator-kyuubi/issues/3300
User group based policies from Ranger are not working as expectedly.
User can be binded to multiple user groups in Ranger Admin. But in privileage checking, user group is identified by plugin it self.
In Authz, User group param setting in AccessRequest currently relies on UserGroupInformation returned by getAuthzUgi method of AuthZUtils. RangerPlugin is plainly use the value of it to check policies conditions.
I suggest,
support overriding usergroup in AccessRequest by fetching user-group relation mapping from UserStore (which introduced since Ranger 2.3) in RangerBasePlugin. UserStore is loaded and refreshed by base plugin with user-group mapping and user/group attributes and used for policy evaluation.
add a switch config for it, disabled by default.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042973015
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerLocalClient.scala:
##########
@@ -45,6 +48,24 @@ class RangerLocalClient extends RangerAdminRESTClient with RangerClientHelper {
lastKnownVersion: Long): ServicePolicies = {
policies
}
+
+ override def getUserStoreIfUpdated(
Review Comment:
I have removed the ut which needs impleation for `getUserStoreIfUpdated` that would break compilation compatibility for Ranger 2.0 or below. And I will put the removed ut and necessary resources in rule json in this PR as remark, after it ismerged.
@yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
URL: https://github.com/apache/incubator-kyuubi/pull/3308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1040299932
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerLocalClient.scala:
##########
@@ -45,6 +48,24 @@ class RangerLocalClient extends RangerAdminRESTClient with RangerClientHelper {
lastKnownVersion: Long): ServicePolicies = {
policies
}
+
+ override def getUserStoreIfUpdated(
Review Comment:
`RangerLocalClient` is put in `ranger-spark-security.xml` as implementation of `RangerAdminRESTClient` interface, and it must override `getUserStoreIfUpdated` for Ranger 2.1+ but it will comes to error while overriding nothing for Ranger 2.0 and below.
I have difficulty in shim the version difference. What shall we do here? Make authz support 2.1+ only, or just remove support this use case ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1342573919
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042971769
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(
Review Comment:
addressed.
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(
+ s"ranger.plugin.${SparkRangerAdminPlugin.getConfig.getServiceType}" +
+ ".enable.override.usergroup.from.userstore",
+ false);
+ if (!enableOverrideUserGroupFromUserStore) {
+ user.getGroupNames.toSet.asJava
+ }
+
+ try {
+ val userGroupsFromUserStore: util.Set[String] = {
+ val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+ "getUserStoreEnricher")
+ getUserStoreEnricher.setAccessible(true)
+ val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+ val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+ getRangerUserStore.setAccessible(true)
+ val userStore = getRangerUserStore.invoke(storeEnricher)
+
+ val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+ getUserGroupMapping.setAccessible(true)
+
+ val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+ mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+ .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+ userGroupMappingMap.get(userName).orNull
Review Comment:
addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042886117
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(
Review Comment:
how define this in SparkRangerAdminPlugin like
```
def authorizeInSingleCall: Boolean = getRangerConf.getBoolean(
s"ranger.plugin.${getServiceType}.authorize.in.single.call",
false)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042886117
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(
Review Comment:
how define this in SparkRangerAdminPlugin like
```scala
def authorizeInSingleCall: Boolean = getRangerConf.getBoolean(
s"ranger.plugin.${getServiceType}.authorize.in.single.call",
false)
```
As getConfig breaks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042944206
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(
Review Comment:
OK, change to used `getRangerConf` instead.
```
val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getRangerConf.getBoolean(
s"ranger.plugin.${SparkRangerAdminPlugin.getServiceType}" +
".enable.override.usergroup.from.userstore",
false)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042884950
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(
+ s"ranger.plugin.${SparkRangerAdminPlugin.getConfig.getServiceType}" +
+ ".enable.override.usergroup.from.userstore",
+ false);
+ if (!enableOverrideUserGroupFromUserStore) {
+ user.getGroupNames.toSet.asJava
+ }
+
+ try {
+ val userGroupsFromUserStore: util.Set[String] = {
+ val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+ "getUserStoreEnricher")
+ getUserStoreEnricher.setAccessible(true)
+ val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+ val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+ getRangerUserStore.setAccessible(true)
+ val userStore = getRangerUserStore.invoke(storeEnricher)
+
+ val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+ getUserGroupMapping.setAccessible(true)
+
+ val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+ mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+ .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+ userGroupMappingMap.get(userName).orNull
Review Comment:
nit: user.getShortUserName
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1043051325
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,48 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroupsFromUgi(user: UserGroupInformation): util.Set[String] = {
+ user.getGroupNames.toSet.asJava
+ }
+
+ private def getUserGroupsFromUserStore(user: UserGroupInformation): Option[util.Set[String]] = {
+ try {
+ val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+ "getUserStoreEnricher")
+ getUserStoreEnricher.setAccessible(true)
+ val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+ val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+ getRangerUserStore.setAccessible(true)
+ val userStore = getRangerUserStore.invoke(storeEnricher)
+
+ val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+ getUserGroupMapping.setAccessible(true)
+
+ val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+ mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+ .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+ userGroupMappingMap.get(user.getShortUserName)
+ } catch {
+ case _: NoSuchMethodException =>
+ None
+ }
+ }
+
+ private def getUserGroups(user: UserGroupInformation): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getRangerConf.getBoolean(
Review Comment:
Good point.
Moved to SparkRangerAdminPlugin, and renamed to `ranger.plugin.spark.use.usergroups.from.userstore.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3308: [KYUUBI #3300] [FEATURE] [AUTHZ] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1225069992
cc @yaooqinn
Please have a look at this feature. The WIP tag removed and it should be ready for review.
Unit test is not added which will not increate much code coverage as it requires methods in Ranger 2.3. (realted to bump of ranger version : https://github.com/apache/incubator-kyuubi/pull/3282)
Docs could be enhanced after code review, for showing usage of the switch config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1338552867
cc @yaooqinn @pan3793 .
This PR passes the CI with ut added over Ranger 2.1+.
And I need some help to solve the compatibility for Ranger 2.0 and below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1043051325
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,48 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroupsFromUgi(user: UserGroupInformation): util.Set[String] = {
+ user.getGroupNames.toSet.asJava
+ }
+
+ private def getUserGroupsFromUserStore(user: UserGroupInformation): Option[util.Set[String]] = {
+ try {
+ val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+ "getUserStoreEnricher")
+ getUserStoreEnricher.setAccessible(true)
+ val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+ val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+ getRangerUserStore.setAccessible(true)
+ val userStore = getRangerUserStore.invoke(storeEnricher)
+
+ val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+ getUserGroupMapping.setAccessible(true)
+
+ val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+ mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+ .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+ userGroupMappingMap.get(user.getShortUserName)
+ } catch {
+ case _: NoSuchMethodException =>
+ None
+ }
+ }
+
+ private def getUserGroups(user: UserGroupInformation): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getRangerConf.getBoolean(
Review Comment:
Moved to SparkRangerAdminPlugin, and renamed to `ranger.plugin.spark.use.usergroups.from.userstore.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3308: [KYUUBI #3300] [FEATURE] [AUTHZ] WIP: Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1229166295
The logic change is ready and works correctly in practices. But ut is required some more investigation to activate UserStoreRefresher with RangerLocalClient to demostrate sample user groups mapping info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #3308: [KYUUBI 3300] [FEATURE] [AUTHZ] WIP: Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1222509782
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3308](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5d035c7) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/bc168527177899e2a72c50cb8f4477c15b4f9452?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bc16852) will **decrease** coverage by `0.05%`.
> The diff coverage is `20.00%`.
```diff
@@ Coverage Diff @@
## master #3308 +/- ##
============================================
- Coverage 51.45% 51.40% -0.06%
Complexity 13 13
============================================
Files 470 470
Lines 26242 26256 +14
Branches 3629 3630 +1
============================================
- Hits 13504 13498 -6
- Misses 11462 11476 +14
- Partials 1276 1282 +6
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...uubi/plugin/spark/authz/ranger/AccessRequest.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L3Jhbmdlci9BY2Nlc3NSZXF1ZXN0LnNjYWxh) | `69.56% <20.00%> (-24.19%)` | :arrow_down: |
| [...apache/kyuubi/engine/JpsApplicationOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvSnBzQXBwbGljYXRpb25PcGVyYXRpb24uc2NhbGE=) | `77.41% <0.00%> (-3.23%)` | :arrow_down: |
| [...pache/kyuubi/engine/YarnApplicationOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvWWFybkFwcGxpY2F0aW9uT3BlcmF0aW9uLnNjYWxh) | `64.81% <0.00%> (-1.86%)` | :arrow_down: |
| [...ache/kyuubi/operation/KyuubiOperationManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uTWFuYWdlci5zY2FsYQ==) | `80.82% <0.00%> (-1.37%)` | :arrow_down: |
| [...ain/scala/org/apache/kyuubi/engine/EngineRef.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvRW5naW5lUmVmLnNjYWxh) | `74.07% <0.00%> (-0.93%)` | :arrow_down: |
| [...che/kyuubi/server/KyuubiTHttpFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvS3l1dWJpVEh0dHBGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `64.23% <0.00%> (-0.73%)` | :arrow_down: |
| [...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvUHJvY0J1aWxkZXIuc2NhbGE=) | `82.60% <0.00%> (-0.63%)` | :arrow_down: |
| [...a/org/apache/kyuubi/service/TFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL1RGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `91.17% <0.00%> (-0.30%)` | :arrow_down: |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `96.97% <0.00%> (-0.09%)` | :arrow_down: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042973015
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerLocalClient.scala:
##########
@@ -45,6 +48,24 @@ class RangerLocalClient extends RangerAdminRESTClient with RangerClientHelper {
lastKnownVersion: Long): ServicePolicies = {
policies
}
+
+ override def getUserStoreIfUpdated(
Review Comment:
I have removed the ut which needs impleation for `getUserStoreIfUpdated` that would break compilation compatibility for Ranger 2.0 or below. And I will put the removed ut and necessary resources in rule json in this PR as remark, after it is merged.
@yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1043012175
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,48 @@ object AccessRequest {
}
req
}
+
+ private def getUserGroupsFromUgi(user: UserGroupInformation): util.Set[String] = {
+ user.getGroupNames.toSet.asJava
+ }
+
+ private def getUserGroupsFromUserStore(user: UserGroupInformation): Option[util.Set[String]] = {
+ try {
+ val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+ "getUserStoreEnricher")
+ getUserStoreEnricher.setAccessible(true)
+ val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+ val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+ getRangerUserStore.setAccessible(true)
+ val userStore = getRangerUserStore.invoke(storeEnricher)
+
+ val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+ getUserGroupMapping.setAccessible(true)
+
+ val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+ mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+ .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+ userGroupMappingMap.get(user.getShortUserName)
+ } catch {
+ case _: NoSuchMethodException =>
+ None
+ }
+ }
+
+ private def getUserGroups(user: UserGroupInformation): util.Set[String] = {
+ val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getRangerConf.getBoolean(
Review Comment:
how about moving this to `SparkRangerAdminPlugin`, so that we can group all possible configurations there from the present to the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1342675622
Remarks with removed unit test, which requires Ranger 2.1 or above, for further reference.
Patching RangerClientHelper trait
```
def getUserStoreIfUpdated(
lastKnownUserStoreVersion: Long,
lastActivationTimeInMillis: Long): RangerUserStore
```
Patching RangerLocalClient.scala
```
override def getUserStoreIfUpdated(
lastKnownUserStoreVersion: Long,
lastActivationTimeInMillis: Long): RangerUserStore = {
val userGroupsMapping = new util.HashMap[String, util.Set[String]]()
userGroupsMapping.put("bob", Set("group_a", "group_b").asJava)
val groupAttrMapping = new util.HashMap[String, util.Map[String, String]]()
val userAttrMapping = new util.HashMap[String, util.Map[String, String]]()
val userAttr1 = new util.HashMap[String, String]()
userAttr1.put("city", "guangzhou")
userAttrMapping.put("bob", userAttr1)
val userStore = new RangerUserStore()
userStore.setUserGroupMapping(userGroupsMapping)
userStore.setGroupAttrMapping(groupAttrMapping)
userStore.setUserAttrMapping(userAttrMapping)
userStore
}
```
Adding AccessRequestSuite.scala
```
package org.apache.kyuubi.plugin.spark.authz.ranger
// scalastyle:off
import org.apache.hadoop.security.UserGroupInformation
import org.scalatest.funsuite.AnyFunSuite
import org.apache.kyuubi.plugin.spark.authz.{ObjectType, OperationType}
import org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin.getFilterExpr
class AccessRequestSuite extends AnyFunSuite {
// scalastyle:on
test("[[KYUUBI #3300]] overriding userGroups with UserStore") {
SparkRangerAdminPlugin.init()
val resource1 =
AccessResource(ObjectType.COLUMN, "my_db_name", "my_table_name", "my_col_1,my_col_2")
val ugi1 = UserGroupInformation.createRemoteUser("anonymous")
val art1 = AccessRequest(resource1, ugi1, OperationType.QUERY, AccessType.SELECT)
assert(art1.getUserGroups.isEmpty)
val ugi2 = UserGroupInformation.createRemoteUser("bob")
val art2 = AccessRequest(resource1, ugi2, OperationType.QUERY, AccessType.SELECT)
assert(!art2.getUserGroups.isEmpty)
assert(art2.getUserGroups.contains("group_a"))
assert(art2.getUserGroups.contains("group_b"))
val are = AccessResource(ObjectType.TABLE, "default", "src_group_row_filter", null)
val art3 = AccessRequest(are, ugi2, OperationType.QUERY, AccessType.SELECT)
val maybeString = getFilterExpr(art3)
assert(maybeString.get === "city == 'guangzhou'")
}
}
```
Patching ranger-spark-security.xml
```
<property>
<name>ranger.plugin.spark.use.usergroups.from.userstore.enabled</name>
<value>true</value>
</property>
<property>
<name>ranger.plugin.spark.enable.implicit.userstore.enricher</name>
<value>true</value>
</property>
<property>
<name>ranger.plugin.hive.policy.cache.dir</name>
<value>target/tmp</value>
</property>
```
Patching sparkSql_hive_jenkins.json
```
{
"service": "hive_jenkins",
"name": "row_filter_with_user_attr",
"policyType": 2,
"policyPriority": 0,
"description": "",
"isAuditEnabled": true,
"resources": {
"database": {
"values": [
"default"
],
"isExcludes": false,
"isRecursive": false
},
"table": {
"values": [
"src_group_row_filter"
],
"isExcludes": false,
"isRecursive": false
}
},
"policyItems": [],
"denyPolicyItems": [],
"allowExceptions": [],
"denyExceptions": [],
"dataMaskPolicyItems": [],
"rowFilterPolicyItems": [
{
"rowFilterInfo": {
"filterExpr": "city == '${{USER.city}}'"
},
"accesses": [
{
"type": "select",
"isAllowed": true
}
],
"users": [],
"groups": ["group_b"],
"conditions": [],
"delegateAdmin": false
}
],"id": 33,
"guid": "e1caa15f-3280-4435-8dbd-f5c664ab9f0f",
"isEnabled": true,
"version": 1
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org