You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "malinjawi (via GitHub)" <gi...@apache.org> on 2026/05/09 10:10:30 UTC

[PR] [VL][Delta] Support native OPTIMIZE ZORDER expressions [gluten]

malinjawi opened a new pull request, #12062:
URL: https://github.com/apache/gluten/pull/12062

   What changes are proposed in this pull request?
   This PR follows #12024 and adds native support for Delta OPTIMIZE ZORDER expression execution in the Velox backend.
   
   The change:
   
   - adds Velox native functions for Delta ZORDER expressions:
     - interleave_bits for InterleaveBits
     - range_partition_id for RangePartitionId / PartitionerExpr
   - converts supported Delta ZORDER expressions in ExpressionConverter
   - allows supported OPTIMIZE ... ZORDER BY commands through GlutenOptimisticTransaction
   - keeps unsupported OPTIMIZE variants on the existing Delta command path
   - adds Delta 3.3 and Delta 4.0 coverage for path-based ZORDER and partition-predicate ZORDER
   
   Why are the changes needed?
   #12024 enabled plain OPTIMIZE compaction command offload. OPTIMIZE ZORDER still needed native expression coverage for InterleaveBits and RangePartitionId to keep the supported command execution native.
   
   Does this PR introduce any user-facing change?
   No public API change. It extends native Delta OPTIMIZE ZORDER support in the Velox backend.
   
   How was this patch tested?
   Built and ran locally:
   
   - Spark 3.5 test-compile with delta profile
   - Spark 4.0 test-compile with delta profile
   - C++ Velox backend build
   - focused Spark 3.5 DeltaNativeWriteSuite ZORDER tests
   - focused Spark 4.0 DeltaNativeWriteSuite ZORDER tests
   
   Performance
   I ran a targeted local benchmark for OPTIMIZE ZORDER on Spark 3.5. Workload: 2,000,000 rows, 14 columns, 128 input files, 1 warmup, 3 measured runs. The benchmark measures only:
   
   OPTIMIZE delta.`path` ZORDER BY (z1, z2)
   
   Table setup time is excluded.
   
   Compared with native Delta write disabled:
   
   | mode | avg ms | median ms | files removed | files added |
   |---|---:|---:|---:|---:|
   | native ZORDER | 5519.0 | 5361.2 | 128 | 1 |
   | fallback ZORDER | 7459.8 | 7116.3 | 128 | 1 |
   
   Compared with local vanilla Spark/Delta using spark.gluten.enabled=false:
   
   | mode | avg ms | median ms | files removed | files added |
   |---|---:|---:|---:|---:|
   | native ZORDER | 5762.7 | 5756.6 | 128 | 1 |
   | vanilla Spark/Delta | 5293.5 | 5320.6 | 128 | 1 |
   
   The patch improves Gluten native ZORDER posture and is faster than the existing Gluten fallback path on this workload. It does not yet beat local vanilla Spark/Delta; remaining overhead is likely in Delta command planning/log/listing/commit work plus Gluten planning/listener and small terminal job overhead.
   
   Related issue: #10215
   Tracked by #12025
   
   Was this patch authored or co-authored using generative AI tooling?
   Generated-by: IBM BOB


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL][Delta] Support native OPTIMIZE ZORDER expressions [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12062: [VL][Delta] Support native OPTIMIZE ZORDER expressions
URL: https://github.com/apache/gluten/pull/12062


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL][Delta] Support native OPTIMIZE ZORDER expressions [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12062:
URL: https://github.com/apache/gluten/pull/12062#issuecomment-4785298067

   This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL][Delta] Support native OPTIMIZE ZORDER expressions [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12062:
URL: https://github.com/apache/gluten/pull/12062#issuecomment-4884533174

   This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [VL][Delta] Support native OPTIMIZE ZORDER expressions [gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12062:
URL: https://github.com/apache/gluten/pull/12062#issuecomment-4412254282

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org