You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "AngersZhuuuu (via GitHub)" <gi...@apache.org> on 2023/11/29 09:44:13 UTC
[PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
AngersZhuuuu opened a new pull request, #5787:
URL: https://github.com/apache/kyuubi/pull/5787
# :mag: Description
## Issue References ๐
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #5594
## Describe Your Solution ๐ง
For case
```
def filter_func(iterator):
for pdf in iterator:
yield pdf[pdf.id == 1]
df = spark.read.table("test_mapinpandas")
execute_result = df.mapInPandas(filter_func, df.schema).show()
```
The logical plan is
```
GlobalLimit 21
+- LocalLimit 21
+- Project [cast(id#5 as string) AS id#11, name#6]
+- MapInPandas filter_func(id#0, name#1), [id#5, name#6]
+- HiveTableRelation [`default`.`test_mapinpandas`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#0, name#1], Partition Cols: []]
```
When handle `MapInPandas`, we didn't match its input with `HiveTableRelation`, cause we miss input table's columns. This pr fix this
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklists
## ๐ Author Self Checklist
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## ๐ Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#discussion_r1409250280
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -987,7 +987,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
}
test("Query: Literal") {
- checkColumns(s"select 1 from $reusedTable", Nil)
+ checkColumns(s"select 1 from $reusedTable", Seq("key", "value"))
Review Comment:
We only change the behavior here since it's a similar behavior like
```
SELECT count(1) from table
```
In current code, this count query return privilege of all columns of the table.
But `test("Query: Literal")` won't return any column, the behavior is not different.
In this pr, we unify the behavior.
If want to fallback the origin behavior of `test("Query: Literal")`
we can write `buildQuery()` as below
```
// If current plan's references don't have relation to it's input, have two case
// 1. Such as `MapInPandas`, `ScriptTransformation`
// 2. Project output only have constant value
// For case 1, we should pass it's input as projection list
// since it's children's columns was pruned.
// For case too, we just skip.
if (columnPrune(p.references.toSeq ++ p.output, p.inputSet).isEmpty) {
// If plan is project and output don't have relation to input, can ignore.
if (!p.isInstanceOf[Project]) {
buildQuery(
child,
privilegeObjects,
p.inputSet.map(_.toAttribute).toSeq,
Nil,
spark)
}
} else {
buildQuery(
child,
privilegeObjects,
columnPrune(projectionList ++ p.references.toSeq, p.inputSet).distinct,
conditionList ++ p.references,
spark)
}
```
If write code like this, also all `count(1)` `count(*)` query won't return privilege.
cc @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#issuecomment-1925638844
Thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#discussion_r1409250280
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -987,7 +987,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
}
test("Query: Literal") {
- checkColumns(s"select 1 from $reusedTable", Nil)
+ checkColumns(s"select 1 from $reusedTable", Seq("key", "value"))
Review Comment:
We only change the behavior here since it's a similar behavior like
```
SELECT count(1) from table
```
In current code, this count query return privilege of all columns of the table.
But `test("Query: Literal")` won't return any column, the behavior is not different.
In this pr, we unify the behavior.
If want to fallback the origin behavior of `test("Query: Literal")`
we can write `buildQuery()` as below
```
// If current plan's references don't have relation to it's input, have two case
// 1. Such as `MapInPandas`, `ScriptTransformation`
// 2. Project output only have constant value
// For case 1, we should pass it's input as projection list
// since it's children's columns was pruned.
// For case too, we just skip.
if (columnPrune(p.references.toSeq ++ p.output, p.inputSet).isEmpty) {
// If plan is project and output don't have relation to input, can ignore.
if (!p.isInstanceOf[Project]) {
buildQuery(
child,
privilegeObjects,
p.inputSet.map(_.toAttribute).toSeq,
Nil,
spark)
}
} else {
buildQuery(
child,
privilegeObjects,
columnPrune(projectionList ++ p.references.toSeq, p.inputSet).distinct,
conditionList ++ p.references,
spark)
}
```
If write code like this, also all `count(1)` `count(*)` query won't return privilege.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #5787: [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input
URL: https://github.com/apache/kyuubi/pull/5787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#issuecomment-1835396055
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5787?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`30c9c16`)](https://app.codecov.io/gh/apache/kyuubi/commit/30c9c1657fe5dd6bf6bc58b5b51c44707c69c929?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.37% compared to head [(`490eb95`)](https://app.codecov.io/gh/apache/kyuubi/pull/5787?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.36%.
> Report is 3 commits behind head on master.
> :exclamation: Current head 490eb95 differs from pull request most recent head db95941. Consider uploading reports for the commit db95941 to get more accurate results
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #5787 +/- ##
============================================
- Coverage 61.37% 61.36% -0.01%
Complexity 23 23
============================================
Files 607 607
Lines 35940 35932 -8
Branches 4936 4931 -5
============================================
- Hits 22057 22051 -6
- Misses 11491 11495 +4
+ Partials 2392 2386 -6
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/5787?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#issuecomment-1831612073
ping @yaooqinn This is a new solution about https://github.com/apache/kyuubi/pull/5627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#issuecomment-1925597320
ping @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#discussion_r1461687540
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/V2CommandsPrivilegesSuite.scala:
##########
@@ -127,7 +127,7 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
assert(po0.catalog.isEmpty)
assertEqualsIgnoreCase(reusedDb)(po0.dbname)
assertEqualsIgnoreCase(reusedTableShort)(po0.objectName)
- assert(po0.columns.take(2) === Seq("key", "value"))
+ assert(po0.columns === Seq("a", "key", "value"))
Review Comment:
Added by test `test("AlterTableAddColumnsCommand")` cc @yaooqinn
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/V2CommandsPrivilegesSuite.scala:
##########
@@ -186,7 +186,7 @@ abstract class V2CommandsPrivilegesSuite extends PrivilegesBuilderSuite {
assert(po0.catalog.isEmpty)
assertEqualsIgnoreCase(reusedDb)(po0.dbname)
assertEqualsIgnoreCase(reusedTableShort)(po0.objectName)
- assert(po0.columns.take(2) === Seq("key", "value"))
+ assert(po0.columns === Seq("a", "key", "value"))
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#discussion_r1409271091
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -125,7 +101,31 @@ object PrivilegesBuilder {
case p =>
for (child <- p.children) {
- buildQuery(child, privilegeObjects, projectionList, conditionList, spark)
+ // If current plan's references don't have relation to it's input, have two cases
+ // 1. `MapInPandas`, `ScriptTransformation`
+ // 2. `Project` output only have constant value
+ if (columnPrune(p.references.toSeq ++ p.output, p.inputSet).isEmpty) {
+ buildQuery(
+ child,
+ privilegeObjects,
+ p.inputSet.map(_.toAttribute).toSeq,
+ Nil,
+ spark)
+ } else {
+ buildQuery(
+ child,
+ privilegeObjects,
+ // Here we use `projectList ++ p.reference` do column prune since:
+ // For `Project`, project's output is contained by plan's referenced
+ // For `Aggregate`, aggregation's output also in it's reference.
+ // For `Filter`, `Sort` etc... it rely on upper `Project` node,
+ // so we wrap a `Project` before call `buildQuery()`.
+ // So here we use upper node's projectionList and current's references
+ // to do column pruning can get the correct column.
+ columnPrune(projectionList ++ p.references.toSeq, p.inputSet).distinct,
Review Comment:
Add comment here to explain why we do this in this pr.
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala:
##########
@@ -125,7 +101,31 @@ object PrivilegesBuilder {
case p =>
for (child <- p.children) {
- buildQuery(child, privilegeObjects, projectionList, conditionList, spark)
+ // If current plan's references don't have relation to it's input, have two cases
+ // 1. `MapInPandas`, `ScriptTransformation`
+ // 2. `Project` output only have constant value
+ if (columnPrune(p.references.toSeq ++ p.output, p.inputSet).isEmpty) {
+ buildQuery(
+ child,
+ privilegeObjects,
+ p.inputSet.map(_.toAttribute).toSeq,
+ Nil,
+ spark)
+ } else {
+ buildQuery(
+ child,
+ privilegeObjects,
+ // Here we use `projectList ++ p.reference` do column prune since:
+ // For `Project`, project's output is contained by plan's referenced
+ // For `Aggregate`, aggregation's output also in it's reference.
+ // For `Filter`, `Sort` etc... it rely on upper `Project` node,
+ // so we wrap a `Project` before call `buildQuery()`.
+ // So here we use upper node's projectionList and current's references
+ // to do column pruning can get the correct column.
+ columnPrune(projectionList ++ p.references.toSeq, p.inputSet).distinct,
Review Comment:
Add comment here to explain why we do this in this pr. cc @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node's input [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5787:
URL: https://github.com/apache/kyuubi/pull/5787#discussion_r1411639948
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilderSuite.scala:
##########
@@ -987,7 +987,7 @@ abstract class PrivilegesBuilderSuite extends AnyFunSuite
}
test("Query: Literal") {
- checkColumns(s"select 1 from $reusedTable", Nil)
+ checkColumns(s"select 1 from $reusedTable", Seq("key", "value"))
Review Comment:
This is wrong to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org