You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/06/23 12:02:02 UTC
[I] [FEATURE] Support reading and writing VECTOR columns for Flink with the Lance base file format [hudi]
cshuo opened a new issue, #19053:
URL: https://github.com/apache/hudi/issues/19053
---
## Feature Description
**What the feature achieves:**
Add VECTOR column read/write support to the Flink **Lance** base-file path.
Hudi already supports VECTOR columns end-to-end on **Spark + Lance** and on **Flink + Parquet**. However, **Flink + Lance** currently supports neither writing nor reading VECTOR columns (in fact, not even plain `ARRAY` columns). The gap is entirely inside `HoodieFlinkLanceArrowUtils`
(`hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieFlinkLanceArrowUtils.java`):
- On write, `toArrowType` / `writeValue` only handle scalar types and throw `HoodieNotSupportedException` for `ARRAY`.
- On read, `toLogicalType` / `readValue` likewise throw for non-scalar Arrow types (e.g. `FixedSizeList`), so the reader fails while converting the Arrow schema.
This feature implements the missing encode/decode so that a VECTOR column can be written to and read from a Flink Lance base file, and—critically—keeps the on-disk encoding **strictly aligned with the Spark Lance path** so files are cross-engine readable.
**Why this feature is needed:**
- Vector-search / embedding workloads increasingly land through Flink streaming pipelines; Lance is the columnar format optimized for vector data, but Flink users currently cannot use VECTOR columns with Lance.
- It closes a parity gap: the same table written by Spark+Lance (or Flink+Parquet) cannot be written/read by Flink+Lance today.
- Cross-engine interoperability: a Lance file with a VECTOR column written by Flink should be readable by Spark and vice-versa.
## User Experience
**How users will use this feature:**
- **DDL / config:** declare the vector column as `ARRAY<FLOAT>` or `ARRAY<DOUBLE>`, mark it via the existing vector-columns option (`FlinkOptions.VECTOR_COLUMNS`, e.g. `embedding:128,features:64`), and select the Lance base file format:
```sql
CREATE TABLE t (
id STRING,
embedding ARRAY<FLOAT>, -- VECTOR(128)
ts BIGINT,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'hudi',
'path' = '...',
'vector.columns' = 'embedding:128',
'hoodie.table.base.file.format' = 'LANCE'
);
```
- **Behavior:** insert / read / column projection / MOR upsert behave the same as on the Parquet base file format (matching the existing `ITTestVectorDataSource` coverage).
- **Element types:** `FLOAT` and `DOUBLE` are supported. `INT8` (`ARRAY<TINYINT>` vector) is **rejected with a clear error**, matching the current Spark Lance writer limitation (the lance-spark connector only treats FLOAT/DOUBLE as fixed-size-list, and the Spark reader only restores FLOAT/DOUBLE). INT8 over Lance is a coordinated cross-engine follow-up.
- **API changes:** none. Existing options are reused.
**On-disk encoding (aligned with Spark Lance):**
- A `VECTOR(dim, FLOAT|DOUBLE)` column is stored as an Arrow `FixedSizeList<Float32|Float64, dim>` with a non-nullable child.
- The Lance file footer carries `hoodie.vector.columns` (`HoodieSchema.VECTOR_COLUMNS_METADATA_KEY`), identical to the Spark writer, so any engine can identify and reconstruct VECTOR columns.
## Hudi RFC Requirements
**RFC PR link:** N/A
**Why RFC isn't needed:**
- Does this change public interfaces/APIs? **No** — reuses existing `vector.columns` option and the existing VECTOR data model.
- Does this change storage format? **No new format** — it reuses the exact Lance `FixedSizeList` encoding and footer metadata already produced by the Spark Lance writer; the change only adds the Flink-side encode/decode to match.
- Justification: scoped, additive engine parity with an already-shipped on-disk contract; no new public surface or format.
## Implementation notes (for reference)
- Write: thread the vector-column map (`HoodieVectorUtils.detectVectorColumns`) from `HoodieRowDataFileWriterFactory.newLanceFileWriter` into `HoodieRowDataLanceWriter`; emit `FixedSizeList` fields in a vector-aware `toArrowSchema`; write the footer via the existing `HoodieBaseLanceWriter.additionalSchemaMetadata()` hook.
- Read: use the already-passed `requestedSchema` in `HoodieRowDataLanceReader.getRowDataIterator` to detect vectors; decode `FixedSizeListVector` → `ArrayData`; map `FixedSizeList` → `ARRAY` in `toLogicalType`; restore VECTOR metadata from the footer in `getSchema()`.
- Reuse: `HoodieSchema.Vector` model + footer helpers, `HoodieVectorUtils.detectVectorColumns`, Arrow `FixedSizeListVector` / `Float4Vector` / `Float8Vector`.
## Out of scope
- INT8 vectors on Lance (blocked by lance-spark; revisit cross-engine).
- Schema evolution on Flink Lance (already explicitly unsupported).
- Nested VECTOR (inside struct/array/map) — stays rejected, consistent with Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support reading and writing VECTOR columns for Flink with the Lance file format [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo closed issue #19053: Support reading and writing VECTOR columns for Flink with the Lance file format
URL: https://github.com/apache/hudi/issues/19053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org