You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/08/22 14:09:04 UTC

[GitHub] [incubator-kyuubi] bowenliang123 opened a new pull request, #3308: [KYUUBI 3300] [FEATURE] [AUTHZ] WIP: Support overriding usergroup with UserStore in AccessRequest

bowenliang123 opened a new pull request, #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308

   <!--
   Thanks for sending a pull request!
   
   Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
     2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
   -->
   
   ### _Why are the changes needed?_
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you add a feature, you can talk about the use case of it.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   Fix https://github.com/apache/incubator-kyuubi/issues/3300
   
   User group based policies from Ranger are not working as expectedly.
   User can be binded to multiple user groups in Ranger Admin. But in privileage checking, user group is identified by plugin it self.
   In Authz, User group param setting in AccessRequest currently relies on UserGroupInformation returned by getAuthzUgi method of AuthZUtils. RangerPlugin is plainly use the value of it to check policies conditions.
   
   I suggest,
   
   support overriding usergroup in AccessRequest by fetching user-group relation mapping from UserStore (which introduced since Ranger 2.3) in RangerBasePlugin. UserStore is loaded and refreshed by base plugin with user-group mapping and user/group attributes and used for policy evaluation.
   add a switch config for it, disabled by default.
   
   ### _How was this patch tested?_
   - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042973015


##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerLocalClient.scala:
##########
@@ -45,6 +48,24 @@ class RangerLocalClient extends RangerAdminRESTClient with RangerClientHelper {
       lastKnownVersion: Long): ServicePolicies = {
     policies
   }
+
+  override def getUserStoreIfUpdated(

Review Comment:
   I have removed the ut which needs impleation for `getUserStoreIfUpdated` that would break compilation compatibility for Ranger 2.0 or below. And I will put the removed ut and necessary resources in rule json in this PR as remark, after it ismerged.
   @yaooqinn 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] yaooqinn closed pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest 
URL: https://github.com/apache/incubator-kyuubi/pull/3308


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1040299932


##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerLocalClient.scala:
##########
@@ -45,6 +48,24 @@ class RangerLocalClient extends RangerAdminRESTClient with RangerClientHelper {
       lastKnownVersion: Long): ServicePolicies = {
     policies
   }
+
+  override def getUserStoreIfUpdated(

Review Comment:
   `RangerLocalClient` is put in `ranger-spark-security.xml` as implementation of `RangerAdminRESTClient` interface, and it must override `getUserStoreIfUpdated` for Ranger 2.1+ but it will comes to error while overriding nothing for Ranger 2.0 and below.
   
   I have difficulty in shim the version difference. What shall we do here? Make authz support 2.1+ only, or just remove support this use case ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1342573919

   thanks, merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042971769


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(

Review Comment:
   addressed.



##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(
+      s"ranger.plugin.${SparkRangerAdminPlugin.getConfig.getServiceType}" +
+        ".enable.override.usergroup.from.userstore",
+      false);
+    if (!enableOverrideUserGroupFromUserStore) {
+      user.getGroupNames.toSet.asJava
+    }
+
+    try {
+      val userGroupsFromUserStore: util.Set[String] = {
+        val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+          "getUserStoreEnricher")
+        getUserStoreEnricher.setAccessible(true)
+        val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+        val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+        getRangerUserStore.setAccessible(true)
+        val userStore = getRangerUserStore.invoke(storeEnricher)
+
+        val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+        getUserGroupMapping.setAccessible(true)
+
+        val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+          mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+            .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+        userGroupMappingMap.get(userName).orNull

Review Comment:
   addressed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042886117


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(

Review Comment:
   how define this in SparkRangerAdminPlugin like 
   
   ```
   def authorizeInSingleCall: Boolean = getRangerConf.getBoolean(
       s"ranger.plugin.${getServiceType}.authorize.in.single.call",
       false)
   
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042886117


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(

Review Comment:
   how define this in SparkRangerAdminPlugin like 
   
   ```scala
   def authorizeInSingleCall: Boolean = getRangerConf.getBoolean(
       s"ranger.plugin.${getServiceType}.authorize.in.single.call",
       false)
   
   ```
   
   As getConfig breaks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042944206


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(

Review Comment:
   OK, change to used `getRangerConf` instead.
   ```
       val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getRangerConf.getBoolean(
         s"ranger.plugin.${SparkRangerAdminPlugin.getServiceType}" +
           ".enable.override.usergroup.from.userstore",
         false)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042884950


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,43 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroups(user: UserGroupInformation, userName: String): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getConfig.getBoolean(
+      s"ranger.plugin.${SparkRangerAdminPlugin.getConfig.getServiceType}" +
+        ".enable.override.usergroup.from.userstore",
+      false);
+    if (!enableOverrideUserGroupFromUserStore) {
+      user.getGroupNames.toSet.asJava
+    }
+
+    try {
+      val userGroupsFromUserStore: util.Set[String] = {
+        val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+          "getUserStoreEnricher")
+        getUserStoreEnricher.setAccessible(true)
+        val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+        val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+        getRangerUserStore.setAccessible(true)
+        val userStore = getRangerUserStore.invoke(storeEnricher)
+
+        val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+        getUserGroupMapping.setAccessible(true)
+
+        val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+          mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+            .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+        userGroupMappingMap.get(userName).orNull

Review Comment:
   nit: user.getShortUserName



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1043051325


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,48 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroupsFromUgi(user: UserGroupInformation): util.Set[String] = {
+    user.getGroupNames.toSet.asJava
+  }
+
+  private def getUserGroupsFromUserStore(user: UserGroupInformation): Option[util.Set[String]] = {
+    try {
+      val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+        "getUserStoreEnricher")
+      getUserStoreEnricher.setAccessible(true)
+      val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+      val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+      getRangerUserStore.setAccessible(true)
+      val userStore = getRangerUserStore.invoke(storeEnricher)
+
+      val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+      getUserGroupMapping.setAccessible(true)
+
+      val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+        mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+          .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+      userGroupMappingMap.get(user.getShortUserName)
+    } catch {
+      case _: NoSuchMethodException =>
+        None
+    }
+  }
+
+  private def getUserGroups(user: UserGroupInformation): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getRangerConf.getBoolean(

Review Comment:
   Good point.
   Moved to SparkRangerAdminPlugin, and renamed to `ranger.plugin.spark.use.usergroups.from.userstore.enabled`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3308: [KYUUBI #3300] [FEATURE] [AUTHZ] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1225069992

   cc @yaooqinn 
   Please have a look at this feature. The WIP tag removed and it should be ready for review. 
   Unit test is not added which will not increate much code coverage as it requires methods in Ranger 2.3. (realted to bump of ranger version : https://github.com/apache/incubator-kyuubi/pull/3282)
   Docs could be enhanced after code review, for showing usage of the switch config.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1338552867

   cc @yaooqinn @pan3793 .
   
   This PR passes the CI with ut added over Ranger 2.1+. 
   And I need some help to solve the compatibility for Ranger 2.0 and below. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1043051325


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,48 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroupsFromUgi(user: UserGroupInformation): util.Set[String] = {
+    user.getGroupNames.toSet.asJava
+  }
+
+  private def getUserGroupsFromUserStore(user: UserGroupInformation): Option[util.Set[String]] = {
+    try {
+      val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+        "getUserStoreEnricher")
+      getUserStoreEnricher.setAccessible(true)
+      val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+      val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+      getRangerUserStore.setAccessible(true)
+      val userStore = getRangerUserStore.invoke(storeEnricher)
+
+      val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+      getUserGroupMapping.setAccessible(true)
+
+      val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+        mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+          .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+      userGroupMappingMap.get(user.getShortUserName)
+    } catch {
+      case _: NoSuchMethodException =>
+        None
+    }
+  }
+
+  private def getUserGroups(user: UserGroupInformation): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getRangerConf.getBoolean(

Review Comment:
   Moved to SparkRangerAdminPlugin, and renamed to `ranger.plugin.spark.use.usergroups.from.userstore.enabled`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3308: [KYUUBI #3300] [FEATURE] [AUTHZ] WIP: Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1229166295

   The logic change is ready and works correctly in practices. But ut is required some more investigation to activate UserStoreRefresher with RangerLocalClient to demostrate sample user groups mapping info.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #3308: [KYUUBI 3300] [FEATURE] [AUTHZ] WIP: Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1222509782

   # [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3308](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5d035c7) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/bc168527177899e2a72c50cb8f4477c15b4f9452?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bc16852) will **decrease** coverage by `0.05%`.
   > The diff coverage is `20.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3308      +/-   ##
   ============================================
   - Coverage     51.45%   51.40%   -0.06%     
     Complexity       13       13              
   ============================================
     Files           470      470              
     Lines         26242    26256      +14     
     Branches       3629     3630       +1     
   ============================================
   - Hits          13504    13498       -6     
   - Misses        11462    11476      +14     
   - Partials       1276     1282       +6     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...uubi/plugin/spark/authz/ranger/AccessRequest.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L3Jhbmdlci9BY2Nlc3NSZXF1ZXN0LnNjYWxh) | `69.56% <20.00%> (-24.19%)` | :arrow_down: |
   | [...apache/kyuubi/engine/JpsApplicationOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvSnBzQXBwbGljYXRpb25PcGVyYXRpb24uc2NhbGE=) | `77.41% <0.00%> (-3.23%)` | :arrow_down: |
   | [...pache/kyuubi/engine/YarnApplicationOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvWWFybkFwcGxpY2F0aW9uT3BlcmF0aW9uLnNjYWxh) | `64.81% <0.00%> (-1.86%)` | :arrow_down: |
   | [...ache/kyuubi/operation/KyuubiOperationManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uTWFuYWdlci5zY2FsYQ==) | `80.82% <0.00%> (-1.37%)` | :arrow_down: |
   | [...ain/scala/org/apache/kyuubi/engine/EngineRef.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvRW5naW5lUmVmLnNjYWxh) | `74.07% <0.00%> (-0.93%)` | :arrow_down: |
   | [...che/kyuubi/server/KyuubiTHttpFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvS3l1dWJpVEh0dHBGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `64.23% <0.00%> (-0.73%)` | :arrow_down: |
   | [...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvUHJvY0J1aWxkZXIuc2NhbGE=) | `82.60% <0.00%> (-0.63%)` | :arrow_down: |
   | [...a/org/apache/kyuubi/service/TFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL1RGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `91.17% <0.00%> (-0.30%)` | :arrow_down: |
   | [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3308/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `96.97% <0.00%> (-0.09%)` | :arrow_down: |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1042973015


##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerLocalClient.scala:
##########
@@ -45,6 +48,24 @@ class RangerLocalClient extends RangerAdminRESTClient with RangerClientHelper {
       lastKnownVersion: Long): ServicePolicies = {
     policies
   }
+
+  override def getUserStoreIfUpdated(

Review Comment:
   I have removed the ut which needs impleation for `getUserStoreIfUpdated` that would break compilation compatibility for Ranger 2.0 or below. And I will put the removed ut and necessary resources in rule json in this PR as remark, after it is merged.
   @yaooqinn 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#discussion_r1043012175


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessRequest.scala:
##########
@@ -72,4 +74,48 @@ object AccessRequest {
     }
     req
   }
+
+  private def getUserGroupsFromUgi(user: UserGroupInformation): util.Set[String] = {
+    user.getGroupNames.toSet.asJava
+  }
+
+  private def getUserGroupsFromUserStore(user: UserGroupInformation): Option[util.Set[String]] = {
+    try {
+      val getUserStoreEnricher = SparkRangerAdminPlugin.getClass.getMethod(
+        "getUserStoreEnricher")
+      getUserStoreEnricher.setAccessible(true)
+      val storeEnricher = getUserStoreEnricher.invoke(SparkRangerAdminPlugin)
+
+      val getRangerUserStore = storeEnricher.getClass.getMethod("getRangerUserStore")
+      getRangerUserStore.setAccessible(true)
+      val userStore = getRangerUserStore.invoke(storeEnricher)
+
+      val getUserGroupMapping = userStore.getClass.getMethod("getUserGroupMapping")
+      getUserGroupMapping.setAccessible(true)
+
+      val userGroupMappingMap: mutable.Map[String, util.Set[String]] =
+        mapAsScalaMap(getUserGroupMapping.invoke(userStore)
+          .asInstanceOf[util.HashMap[String, util.Set[String]]])
+
+      userGroupMappingMap.get(user.getShortUserName)
+    } catch {
+      case _: NoSuchMethodException =>
+        None
+    }
+  }
+
+  private def getUserGroups(user: UserGroupInformation): util.Set[String] = {
+    val enableOverrideUserGroupFromUserStore = SparkRangerAdminPlugin.getRangerConf.getBoolean(

Review Comment:
   how about moving this to `SparkRangerAdminPlugin`, so that we can group all possible configurations there from the present to the future



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3308: [KYUUBI #3300] [Authz] Support overriding usergroup with UserStore in AccessRequest

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3308:
URL: https://github.com/apache/incubator-kyuubi/pull/3308#issuecomment-1342675622

   Remarks with removed unit test, which requires Ranger 2.1 or above, for further reference.
   
   
   Patching RangerClientHelper trait
   
   ```
     def getUserStoreIfUpdated(
         lastKnownUserStoreVersion: Long,
         lastActivationTimeInMillis: Long): RangerUserStore
   ```
   
   Patching RangerLocalClient.scala
   ```
     override def getUserStoreIfUpdated(
         lastKnownUserStoreVersion: Long,
         lastActivationTimeInMillis: Long): RangerUserStore = {
       val userGroupsMapping = new util.HashMap[String, util.Set[String]]()
       userGroupsMapping.put("bob", Set("group_a", "group_b").asJava)
       val groupAttrMapping = new util.HashMap[String, util.Map[String, String]]()
       val userAttrMapping = new util.HashMap[String, util.Map[String, String]]()
       val userAttr1 = new util.HashMap[String, String]()
       userAttr1.put("city", "guangzhou")
       userAttrMapping.put("bob", userAttr1)
   
       val userStore = new RangerUserStore()
       userStore.setUserGroupMapping(userGroupsMapping)
       userStore.setGroupAttrMapping(groupAttrMapping)
       userStore.setUserAttrMapping(userAttrMapping)
       userStore
     }
   ```
   
   
   Adding AccessRequestSuite.scala
   ```
   package org.apache.kyuubi.plugin.spark.authz.ranger
   
   // scalastyle:off
   import org.apache.hadoop.security.UserGroupInformation
   import org.scalatest.funsuite.AnyFunSuite
   
   import org.apache.kyuubi.plugin.spark.authz.{ObjectType, OperationType}
   import org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin.getFilterExpr
   
   class AccessRequestSuite extends AnyFunSuite {
   // scalastyle:on
     test("[[KYUUBI #3300]] overriding userGroups with UserStore") {
       SparkRangerAdminPlugin.init()
   
       val resource1 =
         AccessResource(ObjectType.COLUMN, "my_db_name", "my_table_name", "my_col_1,my_col_2")
   
       val ugi1 = UserGroupInformation.createRemoteUser("anonymous")
       val art1 = AccessRequest(resource1, ugi1, OperationType.QUERY, AccessType.SELECT)
       assert(art1.getUserGroups.isEmpty)
   
       val ugi2 = UserGroupInformation.createRemoteUser("bob")
       val art2 = AccessRequest(resource1, ugi2, OperationType.QUERY, AccessType.SELECT)
       assert(!art2.getUserGroups.isEmpty)
       assert(art2.getUserGroups.contains("group_a"))
       assert(art2.getUserGroups.contains("group_b"))
   
       val are = AccessResource(ObjectType.TABLE, "default", "src_group_row_filter", null)
       val art3 = AccessRequest(are, ugi2, OperationType.QUERY, AccessType.SELECT)
       val maybeString = getFilterExpr(art3)
       assert(maybeString.get === "city == 'guangzhou'")
     }
   }
   ```
   
   Patching ranger-spark-security.xml
   ```
     <property>
       <name>ranger.plugin.spark.use.usergroups.from.userstore.enabled</name>
       <value>true</value>
     </property>
   
     <property>
       <name>ranger.plugin.spark.enable.implicit.userstore.enricher</name>
       <value>true</value>
     </property>
   
     <property>
       <name>ranger.plugin.hive.policy.cache.dir</name>
       <value>target/tmp</value>
     </property>
   ```
   
   Patching sparkSql_hive_jenkins.json
   ```
   {
         "service": "hive_jenkins",
         "name": "row_filter_with_user_attr",
         "policyType": 2,
         "policyPriority": 0,
         "description": "",
         "isAuditEnabled": true,
         "resources": {
           "database": {
             "values": [
               "default"
             ],
             "isExcludes": false,
             "isRecursive": false
           },
           "table": {
             "values": [
               "src_group_row_filter"
             ],
             "isExcludes": false,
             "isRecursive": false
           }
         },
         "policyItems": [],
         "denyPolicyItems": [],
         "allowExceptions": [],
         "denyExceptions": [],
         "dataMaskPolicyItems": [],
         "rowFilterPolicyItems": [
           {
             "rowFilterInfo": {
               "filterExpr": "city == '${{USER.city}}'"
             },
             "accesses": [
               {
                 "type": "select",
                 "isAllowed": true
               }
             ],
             "users": [],
             "groups": ["group_b"],
             "conditions": [],
             "delegateAdmin": false
           }
         ],"id": 33,
         "guid": "e1caa15f-3280-4435-8dbd-f5c664ab9f0f",
         "isEnabled": true,
         "version": 1
       },
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org