You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "zml1206 (via GitHub)" <gi...@apache.org> on 2023/11/06 13:29:43 UTC

[PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

zml1206 opened a new pull request, #5636:
URL: https://github.com/apache/kyuubi/pull/5636

   <!--
   Thanks for sending a pull request!
   
   Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
     2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
   -->
   
   ### _Why are the changes needed?_
   To close #5534 .
   Support optimize into table command for Delta Lake. This feature is available in Delta Lake 1.2.0 and above.
   https://docs.delta.io/latest/optimizations-oss.html#optimize-performance-with-file-management
   
   
   ### _How was this patch tested?_
   - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
   
   
   ### _Was this patch authored or co-authored using generative AI tooling?_
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384530068


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   `ALTER` is used to modify metadata data of object, `UPDATE` is used to modify physical data of object. `OPTIMIZETABLE` merged the files to generate a new version. Although the data results have not changed, the data files have changed, so is UPDATE more appropriate?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1801717669

   > There is a problem with the merge
   
   Fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384540274


##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",

Review Comment:
   https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/commands/OptimizeTableCommand.scala



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384573773


##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",
+  "tableDescs" : [ {
+    "fieldName" : "tableId",

Review Comment:
   I couldn't found the field named tableId



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384297676


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   IIUC, OPTIMIZETABLE only changes data layout?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384389080


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   Alter access type? Does `VacuumCommand`  also is alter type?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384332785


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   It seems that MSCK is approaching.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1798494223

   > Thank you, I'm OK to separated it. Would you mind to attach the plan for `OPTIMIZE '/path/to/delta/table'`?
   
   OPTIMIZE 'tmp/dt1'
   == Analyzed Logical Plan ==
   path: string, metrics: struct<numFilesAdded:bigint,numFilesRemoved:bigint,filesAdded:struct<min:bigint,max:bigint,avg:double,totalFiles:bigint,totalSize:bigint>,filesRemoved:struct<min:bigint,max:bigint,avg:double,totalFiles:bigint,totalSize:bigint>,partitionsOptimized:bigint,zOrderStats:struct<strategyName:string,inputCubeFiles:struct<num:bigint,size:bigint>,inputOtherFiles:struct<num:bigint,size:bigint>,inputNumCubes:bigint,mergedFiles:struct<num:bigint,size:bigint>,numOutputCubes:bigint,mergedNumCubes:bigint>,numBatches:bigint,totalConsideredFiles:bigint,totalFilesSkipped:bigint,preserveInsertionOrder:boolean,numFilesSkippedToReduceWriteAmplification:bigint,numBytesSkippedToReduceWriteAmplification:bigint,startTimeMs:bigint,endTimeMs:bigint,totalClusterParallelism:bigint,totalScheduledTasks:bigint,autoCompactParallelismStats:struct<maxClusterActiveParallelism:bigint,minClusterActiveParallelism:bigint,maxSessionActiveParallelism:bigint,minSessionActiveParallelism:bigint>,deletio
 nVectorStats:struct<numDeletionVectorsRemoved:bigint,numDeletionVectorRowsRemoved:bigint>,numTableColumns:bigint,numTableColumnsWithStats:bigint>
   OptimizeTableCommand DeltaOptimizeContext(false,None,None,false)
   +- ResolvedTable org.apache.spark.sql.delta.catalog.DeltaCatalog@608e34bf, delta.`tmp/dt1`, DeltaTableV2(org.apache.spark.sql.SparkSession@3a4af228,tmp/dt1,None,None,None,Map(),org.apache.spark.sql.util.CaseInsensitiveStringMap@1f), [id#818L]
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384389080


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   Alter access type? Does `VacuumCommand`  also is alter type?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1798367571

   > For the test part, I think we shall cover all these cases.
   > 
   > ```sql
   > OPTIMIZE '/path/to/delta/table' -- Optimizes the path-based Delta Lake table
   > 
   > OPTIMIZE delta_table_name;
   > 
   > OPTIMIZE delta.`/path/to/delta/table`;
   > 
   > -- If you have a large amount of data and only want to optimize a subset of it, you can specify an optional partition predicate using `WHERE`:
   > OPTIMIZE delta_table_name WHERE date >= '2017-01-01'
   > ```
   > 
   > If you don't mind, you can also provide the analyzed logical plans in PR comments for better review experiences, Thank you.
   
   Analyzed logical plans added.
   I briefly tried it before. Delta Lake table with path is not supported in Authz. I plan to do it separately later. What do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1798393766

   Thank you, I'm OK to separated it. Would you mind to attach the plan for `OPTIMIZE '/path/to/delta/table'`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1798179512

   For the test part, I think we shall cover all these cases.
   
   ```sql
   OPTIMIZE '/path/to/delta/table' -- Optimizes the path-based Delta Lake table
   
   OPTIMIZE delta_table_name;
   
   OPTIMIZE delta.`/path/to/delta/table`;
   
   -- If you have a large amount of data and only want to optimize a subset of it, you can specify an optional partition predicate using `WHERE`:
   OPTIMIZE delta_table_name WHERE date >= '2017-01-01'
   ```
   
   If you don't mind, you can also provide the analyzed logical plans in PR comments for better review experiences, Thank you.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1798691590

    Only on Delta Lake 3.0/Spark 3.5, the analyzed logical plan of OPTIMIZE delta.`/tmp/t1` and OPTIMIZE '/tmp/t1' are the same.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1803055730

   Thank you. merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1798685605

   ```
   OPTIMIZE delta.`/tmp/t1`
   before Delta Lake 3.0/Spark 3.5
   == Analyzed Logical Plan ==
   path: string, metrics: struct<numFilesAdded:bigint,numFilesRemoved:bigint,filesAdded:struct<min:bigint,max:bigint,avg:double,totalFiles:bigint,totalSize:bigint>,filesRemoved:struct<min:bigint,max:bigint,avg:double,totalFiles:bigint,totalSize:bigint>,partitionsOptimized:bigint,zOrderStats:struct<strategyName:string,inputCubeFiles:struct<num:bigint,size:bigint>,inputOtherFiles:struct<num:bigint,size:bigint>,inputNumCubes:bigint,mergedFiles:struct<num:bigint,size:bigint>,numOutputCubes:bigint,mergedNumCubes:bigint>,numBatches:bigint,totalConsideredFiles:bigint,totalFilesSkipped:bigint,preserveInsertionOrder:boolean,numFilesSkippedToReduceWriteAmplification:bigint,numBytesSkippedToReduceWriteAmplification:bigint,startTimeMs:bigint,endTimeMs:bigint,totalClusterParallelism:bigint,totalScheduledTasks:bigint,autoCompactParallelismStats:struct<maxClusterActiveParallelism:bigint,minClusterActiveParallelism:bigint,maxSessionActiveParallelism:bigint,minSessionActiveParallelism:bigint>,deletio
 nVectorStats:struct<numDeletionVectorsRemoved:bigint,numDeletionVectorRowsRemoved:bigint>,numTableColumns:bigint,numTableColumnsWithStats:bigint>
   OptimizeTableCommand `delta`.`/tmp/t1`, DeltaOptimizeContext(false,None,None)
   
   Delta Lake 3.0/Spark 3.5
   == Analyzed Logical Plan ==
   path: string, metrics: struct<numFilesAdded:bigint,numFilesRemoved:bigint,filesAdded:struct<min:bigint,max:bigint,avg:double,totalFiles:bigint,totalSize:bigint>,filesRemoved:struct<min:bigint,max:bigint,avg:double,totalFiles:bigint,totalSize:bigint>,partitionsOptimized:bigint,zOrderStats:struct<strategyName:string,inputCubeFiles:struct<num:bigint,size:bigint>,inputOtherFiles:struct<num:bigint,size:bigint>,inputNumCubes:bigint,mergedFiles:struct<num:bigint,size:bigint>,numOutputCubes:bigint,mergedNumCubes:bigint>,numBatches:bigint,totalConsideredFiles:bigint,totalFilesSkipped:bigint,preserveInsertionOrder:boolean,numFilesSkippedToReduceWriteAmplification:bigint,numBytesSkippedToReduceWriteAmplification:bigint,startTimeMs:bigint,endTimeMs:bigint,totalClusterParallelism:bigint,totalScheduledTasks:bigint,autoCompactParallelismStats:struct<maxClusterActiveParallelism:bigint,minClusterActiveParallelism:bigint,maxSessionActiveParallelism:bigint,minSessionActiveParallelism:bigint>,deletio
 nVectorStats:struct<numDeletionVectorsRemoved:bigint,numDeletionVectorRowsRemoved:bigint>,numTableColumns:bigint,numTableColumnsWithStats:bigint>
   OptimizeTableCommand DeltaOptimizeContext(false,None,None)
   +- ResolvedTable org.apache.spark.sql.delta.catalog.DeltaCatalog@13553b2e, delta.`/tmp/t1`, DeltaTableV2(org.apache.spark.sql.SparkSession@18f9745a,/tmp/t1,None,None,None,Map()), [id#82, name#83, gender#84, birthDate#85]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1794987260

   ## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5636?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#5636](https://app.codecov.io/gh/apache/kyuubi/pull/5636?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (112bc65) into [master](https://app.codecov.io/gh/apache/kyuubi/commit/e45f4d950bb95841350e23ba1c576d57840e542b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (e45f4d9) will **increase** coverage by `0.01%`.
   > The diff coverage is `85.71%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #5636      +/-   ##
   ============================================
   + Coverage     61.40%   61.42%   +0.01%     
     Complexity       23       23              
   ============================================
     Files           603      603              
     Lines         35635    35638       +3     
     Branches       4874     4874              
   ============================================
   + Hits          21883    21891       +8     
   - Misses        11372    11377       +5     
   + Partials       2380     2370      -10     
   ```
   
   
   | [Files](https://app.codecov.io/gh/apache/kyuubi/pull/5636?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...ache/kyuubi/plugin/spark/authz/OperationType.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5636?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L09wZXJhdGlvblR5cGUuc2NhbGE=) | `100.00% <100.00%> (ø)` | |
   | [...ubi/plugin/spark/authz/serde/tableExtractors.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5636?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L3NlcmRlL3RhYmxlRXh0cmFjdG9ycy5zY2FsYQ==) | `92.40% <100.00%> (+0.07%)` | :arrow_up: |
   | [.../kyuubi/plugin/spark/authz/ranger/AccessType.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5636?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZXh0ZW5zaW9ucy9zcGFyay9reXV1Ymktc3BhcmstYXV0aHovc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvcGx1Z2luL3NwYXJrL2F1dGh6L3Jhbmdlci9BY2Nlc3NUeXBlLnNjYWxh) | `12.50% <0.00%> (ø)` | |
   
   ... and [7 files with indirect coverage changes](https://app.codecov.io/gh/apache/kyuubi/pull/5636/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/codecov/) today!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384523411


##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",

Review Comment:
   can you provide us the link to this class？



##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",

Review Comment:
   can you provide us the link to this class？



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1386438798


##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DeltaCommands.scala:
##########
@@ -53,8 +54,16 @@ object DeltaCommands extends CommandSpecs[TableCommandSpec] {
     TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
   }
 
+  val OptimizeTableCommand = {
+    val cmd = "org.apache.spark.sql.delta.commands.OptimizeTableCommand"
+    val childDesc = TableDesc("child", classOf[ResolvedTableTableExtractor])
+    val tableDesc = TableDesc("tableId", classOf[TableIdentifierOptionTableExtractor])
+    TableCommandSpec(cmd, Seq(childDesc, tableDesc), OPTIMIZETABLE)

Review Comment:
   How about ALTERTABLE_COMPACT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1803075894

   Thanks, @yaooqinn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1386270305


##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DeltaCommands.scala:
##########
@@ -53,8 +54,16 @@ object DeltaCommands extends CommandSpecs[TableCommandSpec] {
     TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
   }
 
+  val OptimizeTableCommand = {
+    val cmd = "org.apache.spark.sql.delta.commands.OptimizeTableCommand"
+    val childDesc = TableDesc("child", classOf[ResolvedTableTableExtractor])
+    val tableDesc = TableDesc("tableId", classOf[TableIdentifierOptionTableExtractor])
+    TableCommandSpec(cmd, Seq(childDesc, tableDesc), OPTIMIZETABLE)

Review Comment:
   Let’s use MSCK here?



##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DeltaCommands.scala:
##########
@@ -53,8 +54,16 @@ object DeltaCommands extends CommandSpecs[TableCommandSpec] {
     TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
   }
 
+  val OptimizeTableCommand = {
+    val cmd = "org.apache.spark.sql.delta.commands.OptimizeTableCommand"
+    val childDesc = TableDesc("child", classOf[ResolvedTableTableExtractor])
+    val tableDesc = TableDesc("tableId", classOf[TableIdentifierOptionTableExtractor])
+    TableCommandSpec(cmd, Seq(childDesc, tableDesc), OPTIMIZETABLE)

Review Comment:
   Let’s use MSCK here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1386446241


##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DeltaCommands.scala:
##########
@@ -53,8 +54,16 @@ object DeltaCommands extends CommandSpecs[TableCommandSpec] {
     TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
   }
 
+  val OptimizeTableCommand = {
+    val cmd = "org.apache.spark.sql.delta.commands.OptimizeTableCommand"
+    val childDesc = TableDesc("child", classOf[ResolvedTableTableExtractor])
+    val tableDesc = TableDesc("tableId", classOf[TableIdentifierOptionTableExtractor])
+    TableCommandSpec(cmd, Seq(childDesc, tableDesc), OPTIMIZETABLE)

Review Comment:
   Nice choice！



##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DeltaCommands.scala:
##########
@@ -53,8 +54,16 @@ object DeltaCommands extends CommandSpecs[TableCommandSpec] {
     TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
   }
 
+  val OptimizeTableCommand = {
+    val cmd = "org.apache.spark.sql.delta.commands.OptimizeTableCommand"
+    val childDesc = TableDesc("child", classOf[ResolvedTableTableExtractor])
+    val tableDesc = TableDesc("tableId", classOf[TableIdentifierOptionTableExtractor])
+    TableCommandSpec(cmd, Seq(childDesc, tableDesc), OPTIMIZETABLE)

Review Comment:
   Nice choice！



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384578014


##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",
+  "tableDescs" : [ {
+    "fieldName" : "tableId",

Review Comment:
   In old version. https://github.com/delta-io/delta/blob/v2.2.0/core/src/main/scala/org/apache/spark/sql/delta/commands/OptimizeTableCommand.scala



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384661235


##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",
+  "tableDescs" : [ {
+    "fieldName" : "tableId",

Review Comment:
   Thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #5636: [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake
URL: https://github.com/apache/kyuubi/pull/5636


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384567260


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   UPDATE is related to record changing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384638915


##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",
+  "tableDescs" : [ {
+    "fieldName" : "tableId",

Review Comment:
   I think we shall target the latest delta release first, then backward compatibilities for elderlies.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1386314783


##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DeltaCommands.scala:
##########
@@ -53,8 +54,16 @@ object DeltaCommands extends CommandSpecs[TableCommandSpec] {
     TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
   }
 
+  val OptimizeTableCommand = {
+    val cmd = "org.apache.spark.sql.delta.commands.OptimizeTableCommand"
+    val childDesc = TableDesc("child", classOf[ResolvedTableTableExtractor])
+    val tableDesc = TableDesc("tableId", classOf[TableIdentifierOptionTableExtractor])
+    TableCommandSpec(cmd, Seq(childDesc, tableDesc), OPTIMIZETABLE)

Review Comment:
   Do you mean to remove the new OperationType of OPTIMIZETABLE and use MSCK directly? Is it better to make a difference? After all, they are two kinds of operations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384521900


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   It depends on the impact of Vacuum on data or meta



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384312173


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   Yes, what type should be used?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384578626


##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AccessType.scala:
##########
@@ -69,7 +69,7 @@ object AccessType extends Enumeration {
               SHOWTABLES |
               SHOWFUNCTIONS |
               DESCFUNCTION => USE
-          case TRUNCATETABLE => UPDATE
+          case TRUNCATETABLE | OPTIMIZETABLE => UPDATE

Review Comment:
   I understand, change it immediately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384652188


##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",
+  "tableDescs" : [ {
+    "fieldName" : "tableId",

Review Comment:
   Updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1384578014


##########
extensions/spark/kyuubi-spark-authz/src/main/resources/table_command_spec.json:
##########
@@ -2056,6 +2056,30 @@
     "fieldExtractor" : "LogicalPlanQueryExtractor"
   } ],
   "uriDescs" : [ ]
+}, {
+  "classname" : "org.apache.spark.sql.delta.commands.OptimizeTableCommand",
+  "tableDescs" : [ {
+    "fieldName" : "tableId",

Review Comment:
   In old version. https://github.com/delta-io/delta/blob/v2.0.2/core/src/main/scala/org/apache/spark/sql/delta/commands/OptimizeTableCommand.scala



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#discussion_r1386427937


##########
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/gen/DeltaCommands.scala:
##########
@@ -53,8 +54,16 @@ object DeltaCommands extends CommandSpecs[TableCommandSpec] {
     TableCommandSpec(cmd, Seq(tableDesc), queryDescs = Seq(queryDesc))
   }
 
+  val OptimizeTableCommand = {
+    val cmd = "org.apache.spark.sql.delta.commands.OptimizeTableCommand"
+    val childDesc = TableDesc("child", classOf[ResolvedTableTableExtractor])
+    val tableDesc = TableDesc("tableId", classOf[TableIdentifierOptionTableExtractor])
+    TableCommandSpec(cmd, Seq(childDesc, tableDesc), OPTIMIZETABLE)

Review Comment:
   Let's keep the enum as a subset of https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org

Re: [PR] [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake [kyuubi]

Posted by "zml1206 (via GitHub)" <gi...@apache.org>.
zml1206 commented on PR #5636:
URL: https://github.com/apache/kyuubi/pull/5636#issuecomment-1801704284

   
   There is a problem with the merge


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org