You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "hemanthsavasere (via GitHub)" <gi...@apache.org> on 2025/12/30 09:48:23 UTC
[PR] [lake/lance] Array support for Lance [fluss]
hemanthsavasere opened a new pull request, #2288:
URL: https://github.com/apache/fluss/pull/2288
### Purpose
Linked issue: close #2186
### Brief change log
This PR implements comprehensive support for Apache Arrow `ARRAY` data types in the Lance data lake integration, enabling Fluss to write array columns (e.g., `ARRAY<INT>`, `ARRAY<STRING>`, nested arrays) to Lance datasets.
Previously, the Lance integration only supported primitive types and strings. Users couldn't write tables with array columns to Lance, limiting the types of data that could be stored in the Lance data lake format. This is a critical gap for users working with complex data structures like:
- Log events with tag arrays
- User profiles with list-type fields
- Time series data with array-valued metrics
- Any schema requiring nested or repeated data
**Key Implementation Notes**
1. **Separate element writers per type:** Following the existing pattern in `fluss-common`, each primitive type has its own array element writer. This provides type safety and mirrors the Arrow type system.
2. **Generic `ArrowFieldWriter<InternalArray>`:** Array element writers extend `ArrowFieldWriter<InternalArray>` to work with array data, distinct from row-based writers.
3. **Count-based positioning:** Element writers use `getCount()` for vector position tracking, which auto-increments via `write()` method. This is critical for correct array element placement.
4. **Nested array support:** `ArrowNestedListWriter` composes element writers recursively, enabling arbitrary nesting depth.
**Type System Integration:**
- Extended `LanceArrowUtils.toArrowField()` to handle `ArrayType` and generate proper Arrow `List` fields with element type children
- Added `ArrayType` visitor in `DataTypeToArrowTypeConverter` to map to `ArrowType.List.INSTANCE`
- Updated `createFieldWriter()` to recognize `ListVector` and create appropriate array writers
**Array Field Writers:**
- **`ArrowListWriter`** - Main writer for array fields, handles array iteration and delegates to element writers
- **`ArrowNestedListWriter`** - Specialized writer for nested arrays (e.g., `ARRAY<ARRAY<INT>>`)
- **15 Array Element Writers** in `writers/array/` package supporting all Fluss primitive types:
- Numeric: `ArrowArrayIntWriter`, `ArrowArrayBigIntWriter`, `ArrowArraySmallIntWriter`, `ArrowArrayTinyIntWriter`, `ArrowArrayFloatWriter`, `ArrowArrayDoubleWriter`
- String/Binary: `ArrowArrayVarCharWriter`, `ArrowArrayVarBinaryWriter`, `ArrowArrayBinaryWriter`
- Temporal: `ArrowArrayDateWriter`, `ArrowArrayTimeWriter`, `ArrowArrayTimestampLtzWriter`, `ArrowArrayTimestampNtzWriter`
- Other: `ArrowArrayBooleanWriter`, `ArrowArrayDecimalWriter`
**Implementation Details:**
- Array element writers use `getCount()` for vector position tracking
- Proper handling of null arrays, empty arrays, and arrays with null elements
- Support for nested arrays via recursive writer composition
- Fixed critical bug: Element writers must call `write()` (not `doWrite()`) to properly increment position count
### Tests
**Integration Tests (5 new tests in `LanceTieringTest`):**
1. `testArrayTypeInt` - Tests `ARRAY<INT>` with 10 rows of varying-length arrays
2. `testArrayTypeString` - Tests `ARRAY<STRING>` with BinaryString arrays
3. `testArrayTypeNullable` - Tests null arrays, empty arrays, and arrays containing null elements
4. `testNestedArrayType` - Tests `ARRAY<ARRAY<INT>>` with 2-level nesting: `[[1,2], [3,4,5], [6]]`
5. `testMultiplePrimitiveArrayTypes` - Tests 5 different array types in single schema (INT, BIGINT, DOUBLE, BOOLEAN, DATE)
**Unit Tests (3 new tests in `ArrowListWriterTest`):**
1. `testWriteSimpleArray` - Tests writing `[1, 2, 3]`
2. `testWriteNullArray` - Tests null array handling
3. `testWriteEmptyArray` - Tests empty array `[]` handling
### API and Format
None
### Documentation
Yes - This Introduces a Major New Feature
Feature: Array Type Support for Lance Data Lake
What's New:
Before this change:
- Users could NOT write tables with ARRAY columns to Lance
- Attempting to use ARRAY<INT>, ARRAY<STRING>, etc. would throw UnsupportedOperationException
- Only primitive types (INT, STRING, DATE, etc.) were supported
After this change:
- Users CAN write tables with ARRAY columns to Lance
- Support for all array types: ARRAY<INT>, ARRAY<STRING>, ARRAY<DATE>, etc.
- Support for nested arrays: ARRAY<ARRAY<INT>>
- Proper handling of null arrays, empty arrays, and arrays with null elements
Feature Scope:
Supported Array Types (15 total):
- Numeric arrays: INT, BIGINT, SMALLINT, TINYINT, FLOAT, DOUBLE
- String/Binary arrays: VARCHAR, VARBINARY, BINARY
- Temporal arrays: DATE, TIME, TIMESTAMP_LTZ, TIMESTAMP_NTZ
- Other: BOOLEAN, DECIMAL
- Nested: ARRAY<ARRAY>
Example Use Cases Enabled:
// User event logging with tags
Schema.newBuilder()
.column("event_id", DataTypes.INT())
.column("tags", DataTypes.ARRAY(DataTypes.STRING()))
.build();
// Time series with array metrics
Schema.newBuilder()
.column("timestamp", DataTypes.TIMESTAMP_LTZ())
.column("sensor_readings", DataTypes.ARRAY(DataTypes.DOUBLE()))
.build();
// Complex nested data
Schema.newBuilder()
.column("user_id", DataTypes.INT())
.column("session_paths", DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.STRING())))
.build();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "hemanthsavasere (via GitHub)" <gi...@apache.org>.
hemanthsavasere commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3700128518
Hi @xx789633 and @wuchong, Can you please review the PR.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "hemanthsavasere (via GitHub)" <gi...@apache.org>.
hemanthsavasere commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3702527384
Agreed @wuchong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "XuQianJin-Stars (via GitHub)" <gi...@apache.org>.
XuQianJin-Stars commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3765970731
hi @hemanthsavasere #1569 This PR has been merged; this PR can now be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3702131985
Thanks for the contribution, @hemanthsavasere!
However, the current implementation duplicates a significant amount of `ArrowWriter` logic that’s already implemented in `fluss-common`. This duplication creates substantial maintenance overhead for the community: as we add support for more data types or fix bugs in the `ArrowWriter` within `fluss-common`, those improvements and fixes won’t automatically propagate to the `fluss-lance` module, making it hard to keep both in sync.
To ensure long-term maintainability and consistency, could we **first refactor the Arrow writer in `fluss-lance` to directly reuse the Arrow utilities from `fluss-common`**? Once that’s in place, we can safely proceed with adding complex type support for Lance on a solid, shared foundation.
This approach will reduce redundancy, minimize future drift, and make contributions easier to sustain across modules.
What do you think? @hemanthsavasere @xx789633 @luoyuxia @leonardBang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "XuQianJin-Stars (via GitHub)" <gi...@apache.org>.
XuQianJin-Stars commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3805070069
> Sure @XuQianJin-Stars, I will update the PR based on the changes. Thanks
hi @hemanthsavasere Has there been any progress on this PR (Pull Request) yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "hemanthsavasere (via GitHub)" <gi...@apache.org>.
hemanthsavasere commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3805131609
Hi @XuQianJin-Stars, I'm working on it. Will be raising the PR shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "hemanthsavasere (via GitHub)" <gi...@apache.org>.
hemanthsavasere closed pull request #2288: [lake/lance] Array support for Lance
URL: https://github.com/apache/fluss/pull/2288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3702241285
> Thanks for the contribution, @hemanthsavasere!
>
> However, the current implementation duplicates a significant amount of `ArrowWriter` logic that’s already implemented in `fluss-common`. This duplication creates substantial maintenance overhead for the community: as we add support for more data types or fix bugs in the `ArrowWriter` within `fluss-common`, those improvements and fixes won’t automatically propagate to the `fluss-lance` module, making it hard to keep both in sync.
>
> To ensure long-term maintainability and consistency, could we **first refactor the Arrow writer in `fluss-lance` to directly reuse the Arrow utilities from `fluss-common`**? Once that’s in place, we can safely proceed with adding complex type support for Lance on a solid, shared foundation.
>
> This approach will reduce redundancy, minimize future drift, and make contributions easier to sustain across modules.
>
> What do you think? @hemanthsavasere @xx789633 @luoyuxia @leonardBang
big +1 for that. And i think `paimon-lance` is a good reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "hemanthsavasere (via GitHub)" <gi...@apache.org>.
hemanthsavasere commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3766368508
Sure @XuQianJin-Stars, I will update the PR based on the changes. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "xx789633 (via GitHub)" <gi...@apache.org>.
xx789633 commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3704417565
> Thanks for the contribution, @hemanthsavasere!
>
> However, the current implementation duplicates a significant amount of `ArrowWriter` logic that’s already implemented in `fluss-common`. This duplication creates substantial maintenance overhead for the community: as we add support for more data types or fix bugs in the `ArrowWriter` within `fluss-common`, those improvements and fixes won’t automatically propagate to the `fluss-lance` module, making it hard to keep both in sync.
>
> To ensure long-term maintainability and consistency, could we **first refactor the Arrow writer in `fluss-lance` to directly reuse the Arrow utilities from `fluss-common`**? Once that’s in place, we can safely proceed with adding complex type support for Lance on a solid, shared foundation.
>
> This approach will reduce redundancy, minimize future drift, and make contributions easier to sustain across modules.
>
> What do you think? @hemanthsavasere @xx789633 @luoyuxia @leonardBang
Totally agree. I don't quite remember why we did not that in our initial code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake/lance] Array support for Lance [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
wuchong commented on PR #2288:
URL: https://github.com/apache/fluss/pull/2288#issuecomment-3707555914
Thank you all, Let's move the discussion to #1569. Let's see if we have any problems when reusing `ArrowWriter`s from `fluss-common`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org