You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "AngersZhuuuu (via GitHub)" <gi...@apache.org> on 2023/11/27 07:21:26 UTC
[PR] [KYUUBI #5780][AUTHZ] Kyuubi tread PVM ass LeafNode to make logic more simple [kyuubi]
AngersZhuuuu opened a new pull request, #5781:
URL: https://github.com/apache/kyuubi/pull/5781
# :mag: Description
## Issue References 🔗
<!-- Append the issue number after #. If there is no issue for you to link create one or -->
<!-- If there are no issues to link, please provide details here. -->
This pull request fixes #5780
## Describe Your Solution 🔧
Currently we convert persist view to PermanentViewMaker, but after optimizer, it changed it's child, make hard to do column prune and get right column privilege object.
After make PVM as a LeafNode, we won't change it's child plan, treat it as a HiveRelation, make code more simple.
## Types of changes :bookmark:
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklists
## 📝 Author Self Checklist
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [ ] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
## 📝 Committer Pre-Merge Checklist
- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested
**Be nice. Be informative.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Kyuubi tread PVM ass LeafNode to make logic more simple [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1827260048
@yaooqinn WDYT about this solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#discussion_r1405934904
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/permanentview/PermanentViewMarker.scala:
##########
@@ -19,20 +19,15 @@ package org.apache.kyuubi.plugin.spark.authz.rule.permanentview
import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, UnaryNode}
+import org.apache.spark.sql.catalyst.plans.logical.{LeafNode, LogicalPlan}
import org.apache.kyuubi.plugin.spark.authz.util.WithInternalChild
-case class PermanentViewMarker(
- child: LogicalPlan,
- catalogTable: CatalogTable,
- outputColNames: Seq[String],
- isSubqueryExpressionPlaceHolder: Boolean = false) extends UnaryNode
+case class PermanentViewMarker(child: LogicalPlan, catalogTable: CatalogTable) extends LeafNode
with WithInternalChild {
override def output: Seq[Attribute] = child.output
override def withNewChildInternal(newChild: LogicalPlan): LogicalPlan =
Review Comment:
Yea, inherit from `UnaryLike`. Can remove.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1831132483
Let's fulfill the PR desc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1831076148
ping @yaooqinn GA passed and conflict resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#discussion_r1405920920
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/permanentview/PermanentViewMarker.scala:
##########
@@ -19,20 +19,15 @@ package org.apache.kyuubi.plugin.spark.authz.rule.permanentview
import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, UnaryNode}
+import org.apache.spark.sql.catalyst.plans.logical.{LeafNode, LogicalPlan}
import org.apache.kyuubi.plugin.spark.authz.util.WithInternalChild
-case class PermanentViewMarker(
- child: LogicalPlan,
- catalogTable: CatalogTable,
- outputColNames: Seq[String],
- isSubqueryExpressionPlaceHolder: Boolean = false) extends UnaryNode
+case class PermanentViewMarker(child: LogicalPlan, catalogTable: CatalogTable) extends LeafNode
with WithInternalChild {
override def output: Seq[Attribute] = child.output
override def withNewChildInternal(newChild: LogicalPlan): LogicalPlan =
Review Comment:
is this still necessary？
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/permanentview/PermanentViewMarker.scala:
##########
@@ -19,20 +19,15 @@ package org.apache.kyuubi.plugin.spark.authz.rule.permanentview
import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, UnaryNode}
+import org.apache.spark.sql.catalyst.plans.logical.{LeafNode, LogicalPlan}
import org.apache.kyuubi.plugin.spark.authz.util.WithInternalChild
-case class PermanentViewMarker(
- child: LogicalPlan,
- catalogTable: CatalogTable,
- outputColNames: Seq[String],
- isSubqueryExpressionPlaceHolder: Boolean = false) extends UnaryNode
+case class PermanentViewMarker(child: LogicalPlan, catalogTable: CatalogTable) extends LeafNode
with WithInternalChild {
override def output: Seq[Attribute] = child.output
override def withNewChildInternal(newChild: LogicalPlan): LogicalPlan =
Review Comment:
is this still necessary？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#discussion_r1408681485
##########
extensions/spark/kyuubi-spark-authz/src/main/resources/database_command_spec.json:
##########
@@ -215,4 +215,4 @@
} ],
"opType" : "SWITCHDATABASE",
"uriDescs" : [ ]
-} ]
+} ]
Review Comment:
I don't know why this pr needs to change this...But without this change, GA failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1829034264
Any more suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #5781: [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object
URL: https://github.com/apache/kyuubi/pull/5781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1831299431
Thank you, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1831153128
> The PR title is for refactoring, but actually what we do here is a bugfix, can we make it more precisely.
How about current, desc also full fill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1827373376
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5781?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `4 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`7f02809`)](https://app.codecov.io/gh/apache/kyuubi/commit/7f02809e54686fa43ae0bffcf7c546d5374458eb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.38% compared to head [(`57bf5ba`)](https://app.codecov.io/gh/apache/kyuubi/pull/5781?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 61.36%.
| [Files](https://app.codecov.io/gh/apache/kyuubi/pull/5781?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../authz/rule/RuleEliminatePermanentViewMarker.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5781?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L3J1bGUvUnVsZUVsaW1pbmF0ZVBlcm1hbmVudFZpZXdNYXJrZXIuc2NhbGE=) | 70.00% | [0 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/5781?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...authz/rule/permanentview/PermanentViewMarker.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5781?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L3J1bGUvcGVybWFuZW50dmlldy9QZXJtYW5lbnRWaWV3TWFya2VyLnNjYWxh) | 0.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/5781?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #5781 +/- ##
============================================
- Coverage 61.38% 61.36% -0.03%
Complexity 23 23
============================================
Files 607 607
Lines 35944 35936 -8
Branches 4936 4936
============================================
- Hits 22066 22051 -15
- Misses 11486 11496 +10
+ Partials 2392 2389 -3
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/5781?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1829321850
please take care of the tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#discussion_r1408682337
##########
extensions/spark/kyuubi-spark-authz/src/main/resources/database_command_spec.json:
##########
@@ -215,4 +215,4 @@
} ],
"opType" : "SWITCHDATABASE",
"uriDescs" : [ ]
-} ]
+} ]
Review Comment:
cc @yaooqinn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1831151044
The PR title is for refactoring, but actually what we do here is a bugfix, can we make it more precisely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5780][AUTHZ] Simplification by treating PermanentViewMarker as LeafNode [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on PR #5781:
URL: https://github.com/apache/kyuubi/pull/5781#issuecomment-1829351456
> please take care of the tests
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org