You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/08/19 23:22:23 UTC
[I] Add a Spark bundle that packages Apache DataFusion Comet [hudi]
yihua opened a new issue, #19673:
URL: https://github.com/apache/hudi/issues/19673
### Problem
Hudi has no bundle that ships a native execution engine. Anyone wanting
[Apache DataFusion Comet](https://datafusion.apache.org/comet/) alongside Hudi has to assemble and
version-match the jars themselves, against a Comet release matrix that does not line up with
Hudi's Spark profiles: Comet publishes one artifact per Spark minor version and, at 1.0.0, covers
only Spark 3.4, 3.5, 4.0 and 4.1.
### Proposal
Publish `hudi-native-spark-bundle`, the contents of `hudi-spark-bundle` plus Comet, for the Spark
versions Comet actually releases for.
Scope note: Comet does not read Hudi tables natively (it rejects Hudi's file format with
`Unsupported file format HoodieFileGroupReaderBasedFileFormat`), so the Parquet decode stays on
Spark. What the bundle enables is native execution of the rest of the plan: joins, shuffles, sorts
and aggregates over Hudi tables. Making the Hudi scan itself native is separate work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a Spark bundle that packages Apache DataFusion Comet [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #19673:
URL: https://github.com/apache/hudi/issues/19673#issuecomment-5349197959
Opened #19674 (draft) implementing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add a Spark bundle that packages Apache DataFusion Comet [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed issue #19673: Add a Spark bundle that packages Apache DataFusion Comet
URL: https://github.com/apache/hudi/issues/19673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org