You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:09:37 UTC
[I] Create a new clustering strategy to execute parquet tools commands during clustering [hudi]
hudi-bot opened a new issue, #16034:
URL: https://github.com/apache/hudi/issues/16034
Create a new clustering strategy to execute parquet tools commands during clustering.
If there is a use case of pruning some columns to save storage memory, current approach of clustering will iterate over every record and remove the unused column, this is so much time consuming. By directly using ParquetTools we can achieve this by running a command within the clustering strategy.
Here, the logic goes through the process of creating marker files that on event of failures we could use the existing rollback mechanism to remove the inflight files and parquet files.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6404
- Type: New Feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create a new clustering strategy to execute parquet tools commands during clustering [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #16034:
URL: https://github.com/apache/hudi/issues/16034#issuecomment-3630135530
### Linked PR(s)
- https://github.com/apache/hudi/pull/9006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create a new clustering strategy to execute parquet tools commands during clustering [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16034:
URL: https://github.com/apache/hudi/issues/16034#issuecomment-4795521181
This feature has landed on master via PR #18409 (commit `a649188b6460`, *feat(spark): refresh parquet tools clustering strategy for current master*). It introduces:
- `SparkExternalFileClusteringExecutionStrategy` (in `hudi-spark-client`) — a clustering execution strategy that rewrites files via an external file-to-file hook instead of iterating records.
- `ExternalFileClusteringWriteHandle` and `FileMetadataWriteStatusConverter` (in `hudi-client-common`) — supporting plumbing that builds a `WriteStatus` from rewritten file metadata while preserving the marker/rollback flow described in this issue.
The original linked PR (#9006) had drifted from current Hudi internals; #18409 is the refreshed, merged implementation. This issue / HUDI-6404 can likely be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create a new clustering strategy to execute parquet tools commands during clustering [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16034: Create a new clustering strategy to execute parquet tools commands during clustering
URL: https://github.com/apache/hudi/issues/16034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org