You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 04:34:40 UTC
[I] Support optimize data layout by sql and make the build more fast [hudi]
hudi-bot opened a new issue, #14930:
URL: https://github.com/apache/hudi/issues/14930
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-2873
- Type: New Feature
- Epic: https://issues.apache.org/jira/browse/HUDI-2100
---
## Comments
12/Jan/22 17:58;alexey.kudinkin;[~xiaotaotao] can you please elaborate what exactly you're planning to do as part of this task?;;;
---
17/Jan/22 12:18;shibei;[~xiaotaotao] Can you describe your idea in detail?;;;
---
17/Jan/22 13:34;mengtao;[~alexey.kudinkin] [~shibei]
1) support optimize data by sparksql, just like dela lake : OPTIMIZE xx_table ZORDER/HILBERT by col1, col2;
2) introduce a new write operation to rewrite table data directly , At present, The performance of clustering operation is slightly worse than that of direct overwrite
;;;
---
18/Jan/22 03:23;shibei;[~xiaotaotao] Two things need to be clarified:
1) Compaction is closer to the semantics of optimize in the data world. At present, compaction doesn't have the ability to sort data when merging log into base file, but the example given above means the optimize command should trigger data sorting, which implies the optimize command should be implemented based on clustering, correct?
2) Is there any possibility to optimize clustering operation instead of introduce a new write operation?
;;;
---
18/Jan/22 10:26;mengtao;[~shibei] do you have wechat, pls add me 1037817390;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support optimize data layout by sql and make the build more fast [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14930: Support optimize data layout by sql and make the build more fast
URL: https://github.com/apache/hudi/issues/14930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support optimize data layout by sql and make the build more fast [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #14930:
URL: https://github.com/apache/hudi/issues/14930#issuecomment-3593823987
This issue belongs to a closed JIRA Epic: https://issues.apache.org/jira/browse/HUDI-2100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support optimize data layout by sql and make the build more fast [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14930:
URL: https://github.com/apache/hudi/issues/14930#issuecomment-4795285043
Triage update: this looks delivered via the SQL-callable `run_clustering` procedure, which supports z-order/hilbert layout optimization. Example:
```sql
CALL run_clustering(table => 'tbl', order => 'col1,col2', order_strategy => 'z-order');
-- or 'hilbert', 'linear'
```
The procedure (`RunClusteringProcedure.scala`) sets `hoodie.layout.optimize.strategy` and `hoodie.clustering.plan.strategy.sort.columns` under the hood, and supports `WHERE`-style partition predicates and partition selection. Z-order/Hilbert curve implementations live under `hudi-client-common` (e.g. `HilbertCurveUtils`), with coverage in `TestLayoutOptimization` and `SpaceCurveOptimizeBenchmark`. The linked epic (HUDI-2100) is also closed.
If the original request was specifically for a Delta-style `OPTIMIZE table ZORDER BY ...` DDL keyword (rather than `CALL`-based invocation), that exact syntax isn't wired up — please open a more focused ticket for that. Otherwise, this can likely be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org