You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/04/28 04:30:40 UTC
[I] Vortex File Format support in Hudi [hudi]
yihua opened a new issue, #18623:
URL: https://github.com/apache/hudi/issues/18623
### Feature Description
**What the feature achieves:**
Adds [Vortex](https://github.com/spiraldb/vortex) as a pluggable base file format in Hudi, alongside Parquet, ORC, and Lance.
**Why this feature is needed:**
Vortex is a Linux Foundation columnar format (Apache 2.0) offering 100x faster random access and 10-20x faster scans vs Parquet through cascading compression. Java bindings and a Spark DataSource V2 connector are [published on Maven Central](https://central.sonatype.com/artifact/dev.vortex/vortex-jni) (`dev.vortex:vortex-jni`, `dev.vortex:vortex-spark_2.12/2.13`, v0.69.0). The integration follows the same architecture established by Lance ([RFC-100, #14127](https://github.com/apache/hudi/issues/14127)), using Arrow C Data Interface for JNI data transfer.
**Scope:** Spark-only initially. Standard Hudi data types. No filter pushdown or data skipping (same starting point as Lance).
### Task List
- [ ] Sub-task 1: Core plumbing -- `VORTEX` enum, storage configs, Maven deps
- [ ] Sub-task 2: Factory wiring -- reader/writer dispatch, `VortexUtils` stub
- [ ] Sub-task 3: Writer -- base Arrow-buffered writer + Spark writer _(parallel with 4)_
- [ ] Sub-task 4: Reader -- Spark reader, iterator, resource closer _(parallel with 3)_
- [ ] Sub-task 5: Utilities -- `VortexUtils` impl, Hive InputFormat stub
- [ ] Sub-task 6: Spark DataSource -- query-time reader, SparkAdapter wiring
- [ ] Sub-task 7: Tests -- unit + functional (COW/MOR)
```
1 -> 2 -> 3 & 4 (parallel) -> 5 -> 6 -> 7
```
Follow-ups: schema evolution, log file format, columnar batch reading, Hive support, BLOB/VECTOR types, filter pushdown.
### Detailed Implementation
<details>
<summary>Click to expand full implementation plan</summary>
#### Sub-task 1: Core Enum, Config, Maven Dependencies (S ~150 lines)
| Action | File |
|--------|------|
| Modify | `hudi-common/.../model/HoodieFileFormat.java` -- add `VORTEX(".vortex")` + `VORTEX_SPARK_ONLY_ERROR_MSG` |
| Modify | `hudi-common/.../config/HoodieStorageConfig.java` -- add `hoodie.vortex.max.file.size`, allocator sizes, flush watermark |
| Modify | `pom.xml` -- version properties, `<dependencyManagement>`, per-Spark-profile artifact overrides |
#### Sub-task 2: Factory Dispatch + VortexUtils Stub (M ~300 lines)
| Action | File |
|--------|------|
| Modify | `hudi-common/.../io/storage/HoodieFileReaderFactory.java` -- `case VORTEX` + `newVortexFileReader()` |
| Modify | `hudi-common/.../io/storage/HoodieFileWriterFactory.java` -- `case VORTEX` + `newVortexFileWriter()` |
| Modify | `hudi-common/.../io/storage/HoodieIOFactory.java` -- `.vortex` extension dispatch |
| Create | `hudi-common/.../util/VortexUtils.java` -- stub extending `FileFormatUtils` |
| Modify | `hudi-hadoop-common/.../hadoop/HoodieHadoopIOFactory.java` -- `case VORTEX` |
| Modify | `hudi-hadoop-common/pom.xml` -- add `dev.vortex:vortex-jni` |
#### Sub-task 3: Base Writer + Spark Writer (L ~650 lines)
Template: `HoodieBaseLanceWriter`, `HoodieSparkLanceWriter`
| Action | File |
|--------|------|
| Create | `hudi-hadoop-common/.../io/vortex/HoodieBaseVortexWriter.java` -- Arrow batch buffering, `dev.vortex.api.VortexWriter`, bloom filter metadata |
| Create | `hudi-client/hudi-spark-client/.../io/storage/HoodieSparkVortexWriter.java` -- InternalRow writer, builder pattern, meta field population |
| Modify | `hudi-client/hudi-spark-client/.../io/storage/HoodieSparkFileWriterFactory.java` -- override `newVortexFileWriter()` |
| Modify | `hudi-client/hudi-spark-client/pom.xml` -- add `dev.vortex:${vortex.spark.artifact}` |
#### Sub-task 4: Spark Reader + Iterator + Resource Closer (L ~650 lines)
Template: `HoodieSparkLanceReader`, `LanceRecordIterator`, `LanceResourceCloser`
| Action | File |
|--------|------|
| Create | `hudi-client/hudi-spark-client/.../io/storage/HoodieSparkVortexReader.java` -- `dev.vortex.api.Session`/`DataSource`/`Scan`, bloom filter + min/max from Arrow metadata |
| Create | `hudi-client/hudi-spark-client/.../io/storage/VortexRecordIterator.java` -- Arrow batch to UnsafeRow |
| Create | `hudi-client/hudi-spark-client/.../io/storage/VortexResourceCloser.java` -- ordered cleanup |
| Modify | `hudi-client/hudi-spark-client/.../io/storage/HoodieSparkFileReaderFactory.java` -- override `newVortexFileReader()` |
#### Sub-task 5: VortexUtils Implementation + Hive Stub (M ~300 lines)
| Action | File |
|--------|------|
| Modify | `hudi-common/.../util/VortexUtils.java` -- implement key iteration, schema reads, row counts |
| Create | `hudi-hadoop-mr/.../hadoop/HoodieVortexInputFormat.java` -- unsupported stub |
#### Sub-task 6: Spark DataSource Reader + SparkAdapter Wiring (L ~550 lines)
Template: `SparkLanceReaderBase`
| Action | File |
|--------|------|
| Create | `hudi-spark-datasource/hudi-spark-common/.../datasources/vortex/SparkVortexReaderBase.scala` |
| Modify | `SparkAdapter.scala` -- add `createVortexFileReader()` |
| Modify | Spark adapters (3.3, 3.4, 3.5, 4.0, 4.1) -- override per version |
| Modify | `HoodieFileGroupReaderBasedFileFormat.scala` -- add `VORTEX` dispatch |
| Modify | `MultipleColumnarFileFormatReader.scala` -- add `VORTEX` case |
#### Sub-task 7: Tests (L ~800 lines)
| Action | File |
|--------|------|
| Create | `.../test/.../io/storage/VortexTestUtils.java` |
| Create | `.../test/.../io/storage/TestHoodieSparkVortexWriter.java` |
| Create | `.../test/.../io/storage/TestHoodieSparkVortexReader.java` |
| Create | `.../test/.../functional/TestVortexDataSource.scala` |
All gated by `@DisabledIfSystemProperty(named = "vortex.skip.tests", matches = "true")`
</details>
### Risks
1. **Arrow schema metadata roundtrip** -- must verify Vortex preserves custom metadata (bloom filters, min/max keys)
2. **JNI API stability** -- v0.69.0 APIs may evolve
3. **Spark version matrix** -- confirm connector works with Spark 3.5 and 4.x
4. **Native library in CI** -- `libvortex_jni` availability; `vortex.skip.tests` toggle provided
### User Experience
**How users will use this feature:**
- Set `hoodie.table.base.file.format=VORTEX` when creating a table
- Standard `spark.write.format("hudi")` / `spark.read.format("hudi")` -- transparent
- New configs: `hoodie.vortex.max.file.size`, `hoodie.vortex.write.allocator.size.bytes`, etc.
### Hudi RFC Requirements
**RFC PR link:** Builds on [RFC-100](https://github.com/apache/hudi/issues/14127)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? Yes -- new enum value and configs
- Does this change storage format? Yes -- `.vortex` as base file format option
- Justification: Plugs into the existing file format architecture from RFC-100. No new architectural changes needed.
### References
- Vortex: https://github.com/spiraldb/vortex
- Vortex Java/Spark: https://github.com/spiraldb/vortex/tree/develop/java
- Maven: `dev.vortex:vortex-jni` / `dev.vortex:vortex-spark_2.12`
- Lance RFC-100: https://github.com/apache/hudi/issues/14127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Vortex File Format support in Hudi [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on issue #18623:
URL: https://github.com/apache/hudi/issues/18623#issuecomment-4337577818
Nice initiative @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Vortex File Format support in Hudi [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed issue #18623: Vortex File Format support in Hudi
URL: https://github.com/apache/hudi/issues/18623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org