You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "AngersZhuuuu (via GitHub)" <gi...@apache.org> on 2023/09/11 10:46:54 UTC
[GitHub] [kyuubi] AngersZhuuuu opened a new pull request, #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
AngersZhuuuu opened a new pull request, #5272:
URL: https://github.com/apache/kyuubi/pull/5272
### _Why are the changes needed?_
Since AnalyzeTableCommand also update table's metadata, since alter command also need table write privilege, AnalyzeTableCommand need too
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn commented on pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#issuecomment-1715086639
Probably, this will make the behavior inconsistent with Apache hive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] AngersZhuuuu commented on a diff in pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#discussion_r1321443828
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -434,7 +434,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(out.size === 0)
+ assert(out.size === 1)
+ val po1 = out.head
+ assert(po1.actionType === PrivilegeObjectActionType.OTHER)
+ assert(po1.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
+ assertEqualsIgnoreCase(reusedDb)(po1.dbname)
+ assertEqualsIgnoreCase(reusedPartTableShort)(po1.objectName)
+ // ignore this check as it behaves differently across spark versions
+ assert(po1.columns.isEmpty)
+ checkTableOwner(po1)
+ val accessType1 = ranger.AccessType(po1, operationType, isInput = true)
+ assert(accessType1 === AccessType.SELECT)
Review Comment:
> Should be ALTER?
Hmm. the operation type here is ANALYZE_TABLE, so the access type is select.
One query only one operation type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn commented on a diff in pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#discussion_r1321427887
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -434,7 +434,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(out.size === 0)
+ assert(out.size === 1)
+ val po1 = out.head
+ assert(po1.actionType === PrivilegeObjectActionType.OTHER)
+ assert(po1.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
+ assertEqualsIgnoreCase(reusedDb)(po1.dbname)
+ assertEqualsIgnoreCase(reusedPartTableShort)(po1.objectName)
+ // ignore this check as it behaves differently across spark versions
+ assert(po1.columns.isEmpty)
+ checkTableOwner(po1)
+ val accessType1 = ranger.AccessType(po1, operationType, isInput = true)
+ assert(accessType1 === AccessType.SELECT)
Review Comment:
Should be ALTER?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] AngersZhuuuu commented on pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#issuecomment-1715052232
@yaooqinn @bowenliang123 How about current?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#discussion_r1322246701
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -434,7 +434,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(out.size === 0)
+ assert(out.size === 1)
+ val po1 = out.head
+ assert(po1.actionType === PrivilegeObjectActionType.OTHER)
+ assert(po1.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
+ assertEqualsIgnoreCase(reusedDb)(po1.dbname)
+ assertEqualsIgnoreCase(reusedPartTableShort)(po1.objectName)
+ // ignore this check as it behaves differently across spark versions
+ assert(po1.columns.isEmpty)
+ checkTableOwner(po1)
+ val accessType1 = ranger.AccessType(po1, operationType, isInput = true)
+ assert(accessType1 === AccessType.SELECT)
Review Comment:
+1 for ALTER.
Although the stat is just stats with no guarantee or accuracy or must-refreshing crossing different engines, they are still part of the table and should not be changed without any proper privilege checking. ANALYZE TABLE causes the change in metadata or properties to the table. As we have ALTERTABLE_PROPERTIES in OperationType, it's a proper operation type for such metadata privilege checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#discussion_r1322246701
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -434,7 +434,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(out.size === 0)
+ assert(out.size === 1)
+ val po1 = out.head
+ assert(po1.actionType === PrivilegeObjectActionType.OTHER)
+ assert(po1.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
+ assertEqualsIgnoreCase(reusedDb)(po1.dbname)
+ assertEqualsIgnoreCase(reusedPartTableShort)(po1.objectName)
+ // ignore this check as it behaves differently across spark versions
+ assert(po1.columns.isEmpty)
+ checkTableOwner(po1)
+ val accessType1 = ranger.AccessType(po1, operationType, isInput = true)
+ assert(accessType1 === AccessType.SELECT)
Review Comment:
+1 for ALTER.
Although the stat is just stats with no guarantee or accuracy or must-refreshing crossing different engines, they are still part of the table and should not be changed without any proper privilege checking. ANALYZE TABLE causes the change in metadata or properties to the table. As we have ALTERTABLE_SERDEPROPERTIES in OperationType, it's a proper operation type for such metadata privilege checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#discussion_r1322246701
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -434,7 +434,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(out.size === 0)
+ assert(out.size === 1)
+ val po1 = out.head
+ assert(po1.actionType === PrivilegeObjectActionType.OTHER)
+ assert(po1.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
+ assertEqualsIgnoreCase(reusedDb)(po1.dbname)
+ assertEqualsIgnoreCase(reusedPartTableShort)(po1.objectName)
+ // ignore this check as it behaves differently across spark versions
+ assert(po1.columns.isEmpty)
+ checkTableOwner(po1)
+ val accessType1 = ranger.AccessType(po1, operationType, isInput = true)
+ assert(accessType1 === AccessType.SELECT)
Review Comment:
+1 for ALTER.
Although the stat is just stats with no guarantee or accuracy or must-refreshing crossing different engines, they are still part of the table and should not be changed without any proper privilege checking. As we have ALTERTABLE_SERDEPROPERTIES in OperationType, it's a proper operation type for such metadata privilege checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn closed pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
URL: https://github.com/apache/kyuubi/pull/5272
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] codecov-commenter commented on pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#issuecomment-1714991835
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5272?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#5272](https://app.codecov.io/gh/apache/kyuubi/pull/5272?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (cdd8100) into [master](https://app.codecov.io/gh/apache/kyuubi/commit/93021109db4bdb6dc6db9464319b13250a792de9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (9302110) will **not change** coverage.
> Report is 6 commits behind head on master.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #5272 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 590 590
Lines 33345 33345
Branches 4406 4406
======================================
Misses 33345 33345
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn commented on pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#issuecomment-1717477563
Thanks, merged to master and 1.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn commented on a diff in pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#discussion_r1322506090
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/TableCommands.scala:
##########
@@ -181,24 +181,24 @@ object TableCommands {
val cd2 = cd1.copy(fieldExtractor = classOf[StringSeqOptionColumnExtractor])
val td1 = tableIdentDesc.copy(columnDesc = Some(cd1), isInput = true)
val td2 = td1.copy(columnDesc = Some(cd2))
- TableCommandSpec(cmd, Seq(td1, td2), ANALYZE_TABLE)
+ TableCommandSpec(cmd, Seq(tableIdentDesc, td1, td2), ALTERTABLE_PROPERTIES)
}
val AnalyzePartition = {
Review Comment:
Let's add some comments here to explain why ALTERTABLE_PROPERTIES instead of ANALYZE_TABLE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] AngersZhuuuu commented on a diff in pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#discussion_r1322621464
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/TableCommands.scala:
##########
@@ -181,24 +181,24 @@ object TableCommands {
val cd2 = cd1.copy(fieldExtractor = classOf[StringSeqOptionColumnExtractor])
val td1 = tableIdentDesc.copy(columnDesc = Some(cd1), isInput = true)
val td2 = td1.copy(columnDesc = Some(cd2))
- TableCommandSpec(cmd, Seq(td1, td2), ANALYZE_TABLE)
+ TableCommandSpec(cmd, Seq(tableIdentDesc, td1, td2), ALTERTABLE_PROPERTIES)
}
val AnalyzePartition = {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] AngersZhuuuu commented on pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#issuecomment-1713634660
@pan3793 @bowenliang123 @yaooqinn Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn commented on a diff in pull request #5272: [KYUUBI #5271][Bug] AnalyzeTableCommand should also add table write privilege
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5272:
URL: https://github.com/apache/kyuubi/pull/5272#discussion_r1321510652
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -434,7 +434,17 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
val accessType0 = ranger.AccessType(po0, operationType, isInput = true)
assert(accessType0 === AccessType.SELECT)
- assert(out.size === 0)
+ assert(out.size === 1)
+ val po1 = out.head
+ assert(po1.actionType === PrivilegeObjectActionType.OTHER)
+ assert(po1.privilegeObjectType === PrivilegeObjectType.TABLE_OR_VIEW)
+ assertEqualsIgnoreCase(reusedDb)(po1.dbname)
+ assertEqualsIgnoreCase(reusedPartTableShort)(po1.objectName)
+ // ignore this check as it behaves differently across spark versions
+ assert(po1.columns.isEmpty)
+ checkTableOwner(po1)
+ val accessType1 = ranger.AccessType(po1, operationType, isInput = true)
+ assert(accessType1 === AccessType.SELECT)
Review Comment:
TBH, ANALYZE TABLE for altering table parameters is not very necessary to invoke privilege checking. Because,
- Unlike modifying critical table parameters, these are just stats
- The stats are correct and have introduced an execution that is better and worthy of saving it.
- The stats are modified automatically by spark internal, not a user-specified command
- A generic ANALYZE TABLE statement might not have to update the table meta, it's more likely an implementation-dependent choice.
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org