You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "AngersZhuuuu (via GitHub)" <gi...@apache.org> on 2023/10/18 05:55:17 UTC
[PR] [KYUUBI #5452][AUTHZ] Support Compaction table/path related command [kyuubi]
AngersZhuuuu opened a new pull request, #5454:
URL: https://github.com/apache/kyuubi/pull/5454
### _Why are the changes needed?_
To close #5452
Support Compaction table/path related command. The SQL grammar is https://github.com/apache/hudi/blob/release-0.14.0/hudi-spark-datasource/hudi-spark/src/main/antlr4/org/apache/hudi/spark/sql/parser/HoodieSqlCommon.g4
- CompactionHoodiePathCommand: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CompactionHoodiePathCommand.scala
- CompactionHoodieTableCommand :https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CompactionHoodieTableCommand.scala
- CompactionShowHoodiePathCommand: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CompactionShowHoodiePathCommand.scala
- CompactionShowHoodieTableCommand: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/CompactionShowHoodieTableCommand.scala
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5452][AUTHZ] Support Compaction table commands for Hudi [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5454:
URL: https://github.com/apache/kyuubi/pull/5454#issuecomment-1768109721
Thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5452][AUTHZ] Support Compaction table/path related command [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5454:
URL: https://github.com/apache/kyuubi/pull/5454#discussion_r1363232914
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/HudiCommands.scala:
##########
@@ -127,6 +127,28 @@ object HudiCommands {
TableCommandSpec(cmd, Seq(tableDesc), TRUNCATETABLE)
}
+ val CompactionHoodieTableCommand = {
+ val cmd = "org.apache.spark.sql.hudi.command.CompactionHoodieTableCommand"
+ val tableDesc = TableDesc("table", classOf[CatalogTableTableExtractor])
+ TableCommandSpec(cmd, Seq(tableDesc, tableDesc.copy(isInput = true)), CREATETABLE)
+ }
+
+ val CompactionHoodiePathCommand = {
+ val cmd = "org.apache.spark.sql.hudi.command.CompactionHoodiePathCommand"
+ TableCommandSpec(cmd, Seq.empty, CREATETABLE)
Review Comment:
For path-related commands, we just ignore or handle them in another way? Is there any example? cc @yaooqinn @bowenliang123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5452][AUTHZ] Support Compaction table commands for Hudi [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5454:
URL: https://github.com/apache/kyuubi/pull/5454#discussion_r1363340813
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/HudiCatalogRangerSparkExtensionSuite.scala:
##########
@@ -264,4 +264,35 @@ class HudiCatalogRangerSparkExtensionSuite extends RangerSparkExtensionSuite {
doAs(admin, sql(truncateTableSql))
}
}
+
+ test("CompactionHoodieTable/CompactionShowHoodieTable Command") {
Review Comment:
Done
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/HudiCommands.scala:
##########
@@ -138,5 +150,7 @@ object HudiCommands {
CreateHoodieTableAsSelectCommand,
CreateHoodieTableLikeCommand,
DropHoodieTableCommand,
- TruncateHoodieTableCommand)
+ TruncateHoodieTableCommand,
+ CompactionHoodieTableCommand,
+ CompactionShowHoodieTableCommand)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5452][AUTHZ] Support Compaction table commands for Hudi [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #5454: [KYUUBI #5452][AUTHZ] Support Compaction table commands for Hudi
URL: https://github.com/apache/kyuubi/pull/5454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5452][AUTHZ] Support Compaction table/path related command [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5454:
URL: https://github.com/apache/kyuubi/pull/5454#discussion_r1363278016
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/HudiCommands.scala:
##########
@@ -127,6 +127,28 @@ object HudiCommands {
TableCommandSpec(cmd, Seq(tableDesc), TRUNCATETABLE)
}
+ val CompactionHoodieTableCommand = {
+ val cmd = "org.apache.spark.sql.hudi.command.CompactionHoodieTableCommand"
+ val tableDesc = TableDesc("table", classOf[CatalogTableTableExtractor])
+ TableCommandSpec(cmd, Seq(tableDesc, tableDesc.copy(isInput = true)), CREATETABLE)
+ }
+
+ val CompactionHoodiePathCommand = {
+ val cmd = "org.apache.spark.sql.hudi.command.CompactionHoodiePathCommand"
+ TableCommandSpec(cmd, Seq.empty, CREATETABLE)
Review Comment:
Maybe ignoring path based table first? There's another issue for path-based table #5368. We can align the processing way there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5452][AUTHZ] Support Compaction tablerelated command [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5454:
URL: https://github.com/apache/kyuubi/pull/5454#discussion_r1363299384
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/HudiCommands.scala:
##########
@@ -127,6 +127,28 @@ object HudiCommands {
TableCommandSpec(cmd, Seq(tableDesc), TRUNCATETABLE)
}
+ val CompactionHoodieTableCommand = {
+ val cmd = "org.apache.spark.sql.hudi.command.CompactionHoodieTableCommand"
+ val tableDesc = TableDesc("table", classOf[CatalogTableTableExtractor])
+ TableCommandSpec(cmd, Seq(tableDesc, tableDesc.copy(isInput = true)), CREATETABLE)
+ }
+
+ val CompactionHoodiePathCommand = {
+ val cmd = "org.apache.spark.sql.hudi.command.CompactionHoodiePathCommand"
+ TableCommandSpec(cmd, Seq.empty, CREATETABLE)
Review Comment:
> Maybe ignoring path based table first? There's another issue for path-based table #5368. We can align the processing way there.
Yea, will remain a sub task for compaction path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5452][AUTHZ] Support Compaction tablerelated command [kyuubi]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #5454:
URL: https://github.com/apache/kyuubi/pull/5454#discussion_r1363301654
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/HudiCommands.scala:
##########
@@ -127,6 +127,28 @@ object HudiCommands {
TableCommandSpec(cmd, Seq(tableDesc), TRUNCATETABLE)
}
+ val CompactionHoodieTableCommand = {
+ val cmd = "org.apache.spark.sql.hudi.command.CompactionHoodieTableCommand"
+ val tableDesc = TableDesc("table", classOf[CatalogTableTableExtractor])
+ TableCommandSpec(cmd, Seq(tableDesc, tableDesc.copy(isInput = true)), CREATETABLE)
+ }
+
+ val CompactionHoodiePathCommand = {
+ val cmd = "org.apache.spark.sql.hudi.command.CompactionHoodiePathCommand"
+ TableCommandSpec(cmd, Seq.empty, CREATETABLE)
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #5452][AUTHZ] Support Compaction tablerelated command [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5454:
URL: https://github.com/apache/kyuubi/pull/5454#discussion_r1363319744
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/HudiCommands.scala:
##########
@@ -138,5 +150,7 @@ object HudiCommands {
CreateHoodieTableAsSelectCommand,
CreateHoodieTableLikeCommand,
DropHoodieTableCommand,
- TruncateHoodieTableCommand)
+ TruncateHoodieTableCommand,
+ CompactionHoodieTableCommand,
+ CompactionShowHoodieTableCommand)
Review Comment:
Sort the list in alphabet order for better maintainability.
##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/HudiCatalogRangerSparkExtensionSuite.scala:
##########
@@ -264,4 +264,35 @@ class HudiCatalogRangerSparkExtensionSuite extends RangerSparkExtensionSuite {
doAs(admin, sql(truncateTableSql))
}
}
+
+ test("CompactionHoodieTable/CompactionShowHoodieTable Command") {
Review Comment:
```suggestion
test("CompactionHoodieTableCommand / CompactionShowHoodieTableCommand") {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org